Download: Report on Interviews with TrainersField-Defining Organizations March 2017
Background
Following the development of a literature review last year on measuring the impact of artistic/creative activism, The Center for Artistic Activism (C4AA), conducted interviews with artistic activism trainers and other “field-defining” organizations to expand the learning for this part of the Æfficacy project. This project has been underwritten by the Compton and Open Society Foundations.
Objectives
The objectives for this research were to:
- Build on the learning from the literature review
- Understand the challenges and opportunities to developing a culture of measuring impact in the field
- Learn what tools and techniques for assessment are currently being used in the field (or not) and how effective they are
- Identify potential tools that would be helpful to the field and what needs to be done to develop them
Methodology
George Perlov Consulting conducted twelve phone/Skype interviews with a mix of artist/practitioner/trainers from field–defining organizations. Individuals were selected by C4AA for their diverse views, experience and range of work. Interviews lasted approximately 45 minutes. A copy of the discussion guide is included at the end of the report. Interviewees who graciously shared their time with us included:
- Andrew Boyd, Beautiful Trouble
- Nadine Block, Beautiful Trouble
- Beka Economopoulos, Not an Alternative
- Jessica George, Revolutions Per Minute
- Gan Golan, Beautiful Trouble
- Adam Horowitz, US Department of Arts and Culture
- Andy Hsiao, Culture Strike
- Jason Jones, Not an Alternative
- Terry Marshall, Intelligent Mischief
- Bill Moyer, Backbone Campaign
- Leo Martin Saura, En Medio
- Jacques Servin, Yes Men
- Igor Vamos, Yes Men
Caveat
Qualitative research, such as these interviews conducted, can provide a broad view of the opinions of a small number of participants. The value of this type of inquiry lies in revealing the range of opinions held and the intensity with which some participants hold those opinions. Data should not be used to project to a larger population or other audiences. Some verbatim quotes have been gently edited for clarity.
Executive Summary
As part of its exploration of the “aefficacy” of artistic or creative activism projects, the Center for Artistic Activism (C4AA) and George Perlov Consulting conducted a series of interviews with 13 artist/practitioner/trainers from field–defining organizations. The purpose of the research was to build on learning from the previously conducted literature review and to identify what tools and information activists, artists, trainers, funders and others in the field need to be more successful in these endeavors. Overall, interviewees provided a wide range of beliefs, attitudes and practices. They also confirmed our hypothesis that more tools and resources to help plan and measure impact of interventions would be beneficial to the field.
There was little consensus among those interviewed on the idea of a standard theory-of-change or framework behind artistic activism efforts. Some believe that they should be identical to advocacy or organizing frameworks. Others believe that they need to be different due to the artistic or creative elements employed. New ideas and models based on existing models seem to be developing in the field. Challenges in setting realistic goals for efforts also add to the difficulty of planning initiatives strategically.
How impact is defined was also an area of conflicting views. Some participants saw the legislative or policy goal to be changed as the ultimate definition of impact. Yet others could see impact in more ancillary effects like reaching new audiences or media coverage of the issue. There was also some debate over whether impact could or should be a defining metric for this kind of work, and how a focus on impact might stifle the creative process or not be reflective of the non-linear nature of social change.
Trainers who specialize in helping groups or organizations use artistic or creative elements in their activism report they sometimes get feedback on their trainings from the trainees. But they often do not stay engaged with them long enough to get reports back on the impact of their work or have time to help them plan for assessment or evaluation. There is an overall sense in the field that more focus is being placed on assessment and evaluation, some of this coming from a self-induced desire for improvement, others feel it is being imposed by those who see nonprofit program evaluation as a gold standard that should be utilized by artistic activists as well.
There is a broad range of thinking about metrics for measuring the impact of interventions. Process metrics are an area that many feel are critically important but often overlooked, possibly because they are believed to be hard to measure. Similarly, metrics to help measure cultural change are greatly welcomed by the field. Media metrics, especially social media metrics, are seen by many as providing a fantastic wealth of information, but few expressed having the competency or time to plan and assess them. There are also concerns that while media buzz for initiatives is critically important in our hyper-connected world, the metrics were becoming more of an end than a means.
A few participants reported utilizing or considering to utilize assessment methods from similar fields that promote behavior change, like social marketing and public relations. The language surrounding impact measurement – evaluation, assessment, metrics, etc., — also feels foreign or alienating to many in the field.
Most report not incorporating evaluation or assessment into their projects due to lack of money, time and knowledge, as well as a focus on up-front strategy and not back-end results. Lastly, there is interest in simple tools that could be utilized without need for too much training. Those that combine objective (metrics) and subjective (story) outcomes are of most interest.
Key Findings
- No standardized thinking about theory-of-change.
When asked if their work followed a specific theory-of-change or logic model, participants expressed a range of thoughts and ideas. Some could point to specific models that have been used by grassroots organizers for years, others reported that they don’t rely on them and felt that they were creating models as they went along. Some viewed artistic activism as a subset of community-based activism and organizing. And some, reflecting the sometime contrasting natures of creating art and behavioral change, felt pigeonholed into using models imposed on them by others.
Overall, there appears to be a nascent “mashup” trend regarding theory-of-change — practitioners building from existing stories and actions by noticing what’s emerging, amplifying those existing narratives through symbolic actions and storytelling to make them more legible, and spreading those shifted narratives to change collective mindsets.
There is no model where it’s modeled in that way. I think there are a few indirect processes. One is the method between the local and national to sort of stir it up with local grassroots organizing conversations, seeing what’s emerging across the many sites, and then collating that into kind of a national dialogue…So the feedback mechanism of that accord is the local and national…We’re drawing from a longer history of community arts organizing or community culture, and then we draw out the body of knowledge and practitioners who have been in the field for quite a long time.
The Beautiful Trouble book has a strong framework and through lines and multiple threads of tradition in there. I would say my big influences historically would be…Abbie Hoffman, the Greenpeace media stunts…the fierceness and radical messaging… ACT Up, Gran Fury. Barbara Kruger type of messaging. And then, just a lot you learn along the way because every generation reinvents the stuff, so I learned a lot from colleagues all along the way. I would say those were in terms of the cultural practice those would be…and Madison Avenue, to be honest… I came up in the first antinuclear movement, so the early ‘80s, and that was very influenced by sort of the Seabrook Affinity Group, which created a direct-action kind of model.
And in terms of our theory of practice, or theory of change, part of it is that we need to build power in an era of communicative capitalism and fragmentation. The left has historically been allergic to power, had a very fraught relationship to it, so how can we occupy existing infrastructures and do it not as branding from above or coalition politics where we have to sit on a conference call and agree on what the fuck we’re doing. But rather look at what’s happening within movements. What are the songs, the stories, the memes, the hashtags, the aesthetics, etc., that are being used, and how can we help to make that more legible and can teach us and signal a counter power?
One thing that is very important to us is about building from what already exists rather than creating something new… which includes showing people what they’re doing, the value, and making connections between groups that are already doing things rather than creating new groups. And so, in this sense, we resist a lot of the artisans, the individualistic, artistic production or even “creativity” as a term. Rather, we point to existing forms, existing groups, who have creativity already built into their practice that wouldn’t be recognized by artists who were looking for something much more individualistic, and the thing that stands out from the crowd. We seek to not stand out from the crowd, but to create a crowd based on collectivity… So, it’s basically union organizing. You go inside the sector, like you want to enter a factory. You get a job at the factory as a labor organizer and you organize the workers on the inside. That’s exactly what we’re doing. And we speak about it in those terms so it’s recognizable to those union groups.
If we bring the right constellation of scientists and activists and museum people together, the museum sector is going to respond positively and we’ll get enough signatures that there will be enough attention on us, and the museum sector will then embrace us. Our measure of success is not just that we get the signatures, but then we can make the next move because the museum sector has accepted us as a museum.
I personally kind of despise the whole theory of change rhetoric. We’ve certainly written plenty of versions of our theory of change in different grant proposals when we needed to have one. What we try to do is symbolic action in the service of social movements because we do think that social movements succeed. Historically they do. We usually try to think of what we’re doing with the existing campaigns that are part of movements… what we’re trying to do is model our behavior after successful symbolic actions that happened in the past and that were clearly part of the collective consciousness when it comes to the success of the movement, like the Salt March, which has a sort of narrative and mediagenic element that got retold and became a part of the legacy of that movement. I don’t know if anybody ever assessed how that was successful or not. I think in the context of the movement, I think it’s generally seen as a success. But whether it actually did anything is, I don’t know, maybe you’ll be able to see now.
Our main theory is that culture shifting is the most valuable and important thing to transforming… socially transforming the world. The culture is the glue between everything else…shift culture, which is shifting people’s behavior in practice and mindset…shifting the social norms. We like using narrative and design… stories and aesthetics…How do you make things appealing and pleasing to people that creates an effect, and then what stories they need to develop…people’s stories develop to make sense of the world. You have to change people’s stories. That’s what we do.
We live in democracies. The way to change is through mass attention to things. One avenue to that is through the mainstream media. We basically hop on issues that have a certain momentum and we just try to add some more attention to it. Where journalists are already primed to consider an issue, we can sometimes come up with a funny action that then the journalist can relate in a different way than they usually report on the issue…The more people are outraged and the more people take action of various sorts, the more likely things are going to change. When you have a large number of people, percentage-wise a very small part of the population, but in numbers, large, thousands and thousands of people, when you have them taking concerted radical action, it usually has a big effect.
- Goal setting is a moving target
One challenge that activist/trainers find is around setting goals, as issues and problems change and evolve. Setting goals with client organizations also can be challenging as the desire for change may not be clearly articulated by these organizations, and they may not realize how long it takes to effect long-term change. Conversely, when goals are narrowly defined, there is more opportunity for success. There are also issues when seemingly good ideas are developed into interventions only to later be recognized as non-strategic. Although seemingly simple, aligning strategies and tactics with goals can be a challenge for the field.
Right now, there’s a big deal in the northeast where so many pipeline projects have been postponed or cancelled. Our activists have been fighting really hard, but they don’t really know if their fighting is making a difference, because it’s also coming at a time when all choices are collapsed, and companies aren’t finding the financing for these pipeline projects, and the fossil fuel infrastructure.
What is your goal? Are you trying to uplift people? Are you trying to shock people? Are you trying to educate? Are you doing this because you want to give your membership a chance to experiment with self-organization and creating a small project that will be part of a bigger whole so that they feel they’re part of the bigger whole? What is the actual strategic goal or bottom line objective of this? Is it just to beautify street corners? That’s fine. Is it because you’re really angry and you just need to yell at someone? Also, fine. Let’s be clear about what their goal is, and that is definitely a process, whether they bring me on to help coordinate a bunch of artistic stuff for a particular mobilization, or whether they bring me on to run a training, or to integrate culture into a conference. So, I can help them identify those goals.
There are exercises (for setting goals). We actually use the Steves’ (Center for Artistic Activism’s) visioning utopia exercise. But other exercises as well, like the magic newspaper headlines that you might want to see, and so on.
It’s like with all the different elements to the movement or people who are taking up that case, how do you tease out which parts are having what affects. Obviously, if we knew that it would allow us to better strategize and better focus and better train and better fund.
Our goals are narrowly defined as getting as much attention as possible. And so, what we do is typically seen as wildly successful for these organizations. You know, they’ll say things like we haven’t gotten a single bit of coverage in the mainstream press until we did this.
Activists make such basic simple errors. Like we’ll come up with some kind of an action where – hey, let’s do this die-in during the protest blah, blah, blah. We’ll have these puppets that represent good and bad; we’ll have this little skit going on. And it’s all very beautiful and exciting and full of puppets and music and so on, but then nobody actually bothers to step back and go, and who’s going to see this? And what are they going to think when they see it? And how is it going to work, you know?
I think the biggest challenge is the tensions between short-term and long-term, and so cultural strategies are really long-term strategies.
- The concept of impact is broadly and diversely defined
When asked how they defined impact of their work, participants expressed a range of thinking. For some, it was clearly and succinctly linked with the legislative or advocacy goal they were hoping to change. Others viewed their impact in various ways, like more/different stories told, new audiences engaged, communities built, advocates inspired, stories in the media, funds raised for organizations, building power, creating social cohesion, etc. Some were skeptical of being able to measure impact at all, which may reflect their own lack of skill or competence in this area.
For us, it’s the widest array of whoever we’re able to bring together. I just received a call from our agent yesterday and could not believe the diversity of who they were able to bring together.
A lot is going on. We’re not exactly sure how to measure it.
It’s very hard to really put a finger on it. The Billionaires are using a very engaging, cultural, sort of meme-based, humor-based, approach. Catching a lot of attention and interest and even joy and a little bit of mayhem that breaks out of the usual sort of narrow messaging. It’s not so much in the primary stated purpose of the organization, which might be to persuade voters, but in all of the secondary aspects …inspiring the troops, if you will, like giving people something to point to, changing the conversation, adding a meme that journalists can play off of, giving the rest of us who are doing the door knocking and phone calling work, something to feel fierce about and feel like we’re pulling a David and Goliath move when a prank goes off well, like giving us more…in the ether, in the sort of culture suit, talking points.
And then I think the last sort of measure of success is yeah, things should change. We’ve gotten eight institutions in the last year or two who cut ties to fossil fuel interests by either divesting from fossil fuels, dropping a board member from the industry, or canceling a sponsorship contract with a fossil fuel company. Those are traditional activist metrics for success.
I really think when you talk about artistic activism – that artistic activism is the nexus between creating meaning and building power. These two very different practices that, when they intercept, can become very powerful. And it’s really about channeling collective energy on a massive level, which is different than some of the individualized ways of personal transformation, or shifting someone’s opinion, or all of those things which we sometimes try to look for impact.
One (type of impact) is that it reaches audiences – stories reach audiences that might get to know about an issue that they otherwise wouldn’t be thinking about it. And the other primary impact for us is that people in the movement are energized by being in a story that is basically fun for them to read. It’s like cheerleading…I think a lot of what we do is about, at least for the younger people, is about overcoming the fear that they have of acting, of doing things. At least that’s what we get in informal reporting. Most of our methods are like anecdotal and informal, and people will write us emails saying, “Wow, I saw your movie in high school and it totally opened my eyes and now I’m out protesting.”
We did one project with this group called the Action Traction, where we impersonated the U.S. Chamber of Commerce and announced that they were changing their position on climate legislation. And two weeks later they actually did. So that’s an example of feeling like the project was successful. It’s not a causal relationship – we didn’t call them to do it –we played a prank. There was a lot of activist pressure that we were a part of, that played a role in them changing their position.
I would say that their money raised is a particular quantitative tool we use. We work with nonprofits to have more of a qualitative approach to judging the impact. After between five and eight months we go back to the organization and talk to them about how our intervention helped and we can look at social media impact…how have things changed in terms of your social media followers, have you recruited new members. We’ll ask them to help us to understand how the partnership impacted their work because ultimately our goal is to impact a particular piece of work and support an organization to reach more people and make more change.
[On the impact of one of their actions] So it’s not stopping yet, but for some, it brought about a change in people’s minds. It’s like, “Oh, I’m not alone in this…thousands of people are suffering the same as me. That means that maybe it’s not only my problem…” that created a big debate and it still is a big debate in Spain.
We have no idea how to measure our impact. It’s purely guesswork. When I see organizations that we’ve worked with talking about measuring impact, I’m not sure they do a good job at all. I think they’re just convincing themselves of things.
- Some questions raised about whether measurement and evaluation is needed for artistic activism projects
A number of the activists/trainers interviewed expressed concerns about whether artistic activism interventions can or even should be subject to any sort of post-analysis as to effect. These opinions recognize the difficulty of this work and in some cases, make the argument that artistic interventions should be exempt from any analysis because of beliefs that social change does not happen in linear, measurable ways. There is also a worry that measurement and criteria of evaluation can get in the way of imagining and experimenting with seemingly impossible goals and outcomes, especially during the formative stages of interventions . An underlying resentment towards funders or others who require such data or information was clearly perceived.
It’s too difficult to get meaningful data. It wouldn’t be “needed” because we actually can’t do it in a meaningful way. The second thing is even if we could do it in a meaningful way it’s possible that the practitioners are just not going to be inspired by it. That’s just not how they operate… there’s an ideological resistance to being so instrumental or even if they’re open to it intellectually or ideologically there is no mechanism for them to actually be guided by it. I am not saying “artists just do it…go off half-cocked and follow your visions.” Artists can be extremely strategic… Some of the more anarchist street theater Bread and Puppet types are not going to care about that or be influenced by it or feel that it’s a betrayal of their pureness of mission and purpose. Others who tend towards more of the deliberate campaigning with a mission and focused targets and things like that would be more so…There are best practices independent of hard numbers that I think people just know and understand through winning and losing enough times. That’s where I think Beautiful Trouble comes in. It’s not really metrics in the normal sense of the term.
I do think it matters that it works. I think artists are often accused of doing things that don’t matter because they are intuitively doing things that are meaningful and make sense to them. Sometimes they may be plugging into something that is incredibly powerful and important. So, I would encourage people to keep doing things that don’t matter to others the way that we define them, because we’re reaching out into that dark, undefinable place in which so much of human experience actually happens.
We’re creating stories that use the media as a vehicle to get them out there and to make them stick with audiences. And I think it’s really hard to assess the impact of stories. I feel for you, because it seems like a really difficult thing to do. It’s typical that we have to write reports where we validate our actions, and where we have to try to make the organization that funded us feel like their money was well-spent.
I think sometimes it’s just really obvious what to do, and those are the moments when it really does no good to think about measurements of impact. It can actually get in the way. Take the Freedom Riders, as an example. When they set off to do what they were doing, they were counseled not to do it. And the impact was evaluated by many groups along the way and they said, “don’t do it, don’t do it. Stop, this is backfiring.” It went all the way to the White House, and the White House was calling these people and saying don’t do this. Martin Luther King was against it. Everybody was against it. And yet, in retrospect, it’s clear it was a massive triumph and moved the whole civil rights movement forward in a really dramatic way, including moving Martin Luther King’s organization forward.
Change happens, some things happen in very unexpected ways that nobody predicts. I really don’t think that I want to waste any of my time trying to figure out what’s going to have the biggest impact, because I’m pretty sure I don’t know…I think that might be the biggest flaw in the impact assessment model. It supposes that people are going to do something different from what they want to do based on some mysterious number magic… (on Occupy Wall Street) I don’t think very many of them realized what would happen or how it would work out. So, I don’t think it would have helped for people to be thinking about that at the beginning. Even if you have a perfect model for the whole thing, I don’t think it would have changed anything.
- Evaluation as part of training and strategy development
Whether training others or developing their own work, evaluation is not often an up-front consideration. Some who work with organizations in the up-front planning and strategic stages found they did not have the opportunity to follow through with these groups regarding impact. Or not think about impact until later in the process. Some did utilize process evaluation tools to get feedback and learning along the way in their trainings.
We are not as strong on inventorying the impact of our work and when we do our strategic planning trainings and work with groups we train around the importance of ongoing assessment. We’ve been doing more trainings…we haven’t had long-term campaign clients where we needed to do that…I don’t think we emphasize (evaluation) because we join groups more at the frontend than at the backend…We’re the creative people they pull in more when they’re planning or thinking or brainstorming. That’s where we can intervene with good strategic planning, which is more on the tactic side of things rather than evaluation, assessment, et cetera. We do emphasize the importance of evaluation, and that it needs to be built into their plan, and that you need to allocate resources, and that it should not be just something that happens at the very end but is used as a constant course correction. I would again say that we’re probably weak on the quantifying of that.
We just throw our energy behind stuff and then along the way or after the facts we measure. But we’re trying to get better about that now.
At the end of every training, or if it’s a multi-day training we integrate it in different ways – we do things that we call cute little names like check-ins or debriefs or use tools called head-heart-hands, or thorns-and-buds, or whatever, to try to assess what people are taking away from it. So, on a very small, local level, we do evaluation. Sometimes, not every time because of time constraints and because it’s not a priority for people, we do written evaluation and we collect them.
When we finish the product or training, that’s when we can say our job is done…we completed that, that was successful. We’ve wanted to be more involved with the actual implementation of the projects because then we may have given them a training, but it’s not enough. Some of the groups have to change their whole culture or may need a new structure to really implement what we’ve giving them, like how to effectively use culture in organizing…In terms of client work, we haven’t figured out what is the exact impact of what we did for them or worked with them on. We’ve been grappling with that.
We do a survey about the training afterwards with those types of questions, and then engage one-on-one with the participants.
Trainers raised a few additional points about the relationship of their training work to the impact of their trainees’ efforts:
We’ve done all these trainings. People have asked for them, they clearly need them and think they need them, but we don’t actually know if a particular training led to a campaign winning or X number of additional chapters being formed or whatever secondary metrics they were going after. I know I can’t back up our work as trainers quantitatively or with a perfectly great narrative or numbers, except say 12,000 copies of the book were sold to people who clearly thought it was worth spending money on, and it’s translated into seven languages.
One thing the trainees did that we thought was really impressive was they started doing test protests. They would do a protest but they didn’t call media, it was just like testing how far they could go with the target.
The more we can give our agents (trainees) the tools to create the intervention that is right for the culture of the debate that they’re in, the better. And so, yes, of course, they have to be effective, and they have to then be effective in terms of how it affects the culture shift.
- No consensus on who’s driving need for assessment
While most feel there is increased focus on evaluation and assessment in the field, there’s a variety of opinion on who is driving this thinking. Some see it as self-induced — organizations just wanting to do a better job of understanding the effect of their work. Others feel that the evaluation of programmatic work of NGOs has creeped into the world of advocacy where different rules and outcome apply. And more esoteric questions about who is trying to convince whom of what.
Our initiative is unusual enough that for one, it’s very hard to find funders, and two, the ones who we do find understand it at a more intuitive level of what we’re doing. And of course, there are metrics that we support, but it doesn’t feel like a heavy-handed mandate.
It’s hard to have traditional measuring tools. I think people have settled on the idea that there is no measurement, you can’t measure these things. There are foundations or other institutions demanding these tools that don’t work.
[On who’s driving need for assessment] I think it’s coming from our desire, but we’ve been pushed a little – (1) We recognize that we’ll be more effective fundraisers if we can build the case, and demonstrate results. So that’s why I make a concerted effort to raise money that’s sustainable. (2) It comes from our own desire to know more about how effective we’re being or whether we need to course correct. And (3), it’s also further propelled by the fact that we are one of four nonprofits that were selected for two years by a nonprofit incubator called Beespace, that has given us a bunch of seed money as well as a ton of in-kind support through hiring consultants and coaches.
There’s also a question of who are we trying to convince, and of what? Are we trying to convince funders? Are we trying to convince economists? Are we trying to convince advocacy organizations? And to do what? To spend more money on this stuff? You know, to respect the role of art and culture? Those are my questions.
There’s some focus on impact but coming from just internally…We just self-reflected and made some adjustments or looked at the things that changed. We haven’t gone to a level of defining our measuring tools, but it’s been more like our self-evaluation.
I think it’s the whole NGO system, it’s not the funders. It’s more the NGOs who want funding saying,” we’ll guarantee your investment by measurement. We can measure our impact.” I think it’s as simple as that. Though I think it doesn’t do any good for activists…
- The role of process or interim metrics
Some participants raised the issue of the importance of process or interim metrics involved in the development of interventions and how these might be assessed. Because a lot of what is desired is in these interventions is affective, there is a need to look more closely at process than most traditional forms of evaluation.
The critical part at the end of the journey in understanding what we’ve learned, and how do we take this all-volunteer energy that is capitalized and turned into something more focused or impactful.
Did you actually have an artist sitting at your strategy table? Did you have a cultural worker there? Did you do an assessment of what kind of cultural pieces are within your community group to speak to the people that you’re trying to outreach to, or to involve them, or to build on your power internally? All that stuff can happen, but it’s not in any kind of organized fashion. And it’s very much part of the DNA of the movement, and that is the case.
I think one of the basic functions that cultural work serves is that it allows people to work together and figure out how to work together. They’ve got a basic function that we need to move towards, how do we work together better? How do we make decisions collectively better? And so that’s a function that arts and culture helped with. How do you measure that?
- Love the media metrics, hate the media metrics
There is a love/hate relationship with media metrics. Most interviewees recognize that media metrics are interim measures of their interventions’ impact that can provide good value and demonstrate visibility. And, as many artistic activism interventions rely on media pick-up to spread the word of the intervention, and because they can be relatively easily measured, they play an important part in many efforts. There is a critical recognition that media attention as a means can become an end in itself. Social media metrics also provide immediate response and great data on audiences, but practitioners often don’t have the time, resources or knowledge to fully measure, analyze and apply learnings.
On the flip side, because it is often hard to measure ultimate outcomes, in many cases media metrics have become a proxy for outcome metrics. Trainers/practitioners are recognizing that media metrics need to be assessed for the quality of engagement they provide. And many long for better ways to measure the social or political change they seek to effect. Media metrics also raise the question of whether a separate communications plan is needed in addition to the actual intervention.
I wouldn’t call it the metric (we’re looking for). Visibility is always a goal. Of course, we give all our agents press templates and everything like that, but you’re not always sure you get the three local media pieces.
It’s harder to get the primary metrics, like how has it (the intervention) affected the hearts and minds of America or the world. Then you fall back on the secondary metrics, but then you can franchise these metrics. These secondary metrics become a stand-in as you’re just wondering am I being effective, am I changing anything… so you fall back on friends on your Facebook page, likes for this thing, comments to this…YouTube clicks, et cetera. In some ways that is an indication of whether you’re striking a chord, whether you’re hitting your audience, but it’s not always a good indication. So much happens online. There are these strange ephemeral numbers that give you a little endorphin boost every time it clicks. Same thing goes for fundraising. Every time you set up a fundraising email and the dollars come back in you think that you’re being successful. The point is that it’s not how many people watch your thing but if you’re really clear on your audience…you could have a thousand influencers in this particular space, whether they’re state legislators or scientists in some kind of professional association and are moved and influenced…That’s better than a million hits on some kind of wacky viral video who are going to be amused by it and send it to friends but are not going to have any bearing on the issue. Sometimes a video with 500 views is better than one with two million views. It all depends on being precise and targeting and knowing how the outreach channels, which is more important than the hilarity of your content. We didn’t have the resources or the precision to be able to do that pre-poll, post-poll, targeted outreach thing to know if we were persuading any voters. We had to rely on column inches and number of chapters formed and all that kind of stuff.
With the Natural History Museum, and the letter that we’re doing right now around Standing Rock, we grew our email list to 100,000 in one year from scratch. We got 552,000 petition signatures to get David Koch off the board of the American Museum of Natural History. We built a network of 150 of the world’s top scientists, including several Nobel laureates. We now have about 800 museum professionals, directors, senior staff signed on to a political letter. That’s the “by-the-numbers” stuff, but then how do those numbers translate into impact? Well, we look at media hits and then assess them not just for reach and number of hits. Around the sciences letter we released, it was something like 150 news hits around the world in the biggest outlets, and covered by news, science, climate, higher education, philanthropy, and arts and culture reporters. That’s impact! But is just getting news coverage enough to change the game?… You can suggest that it’s changing the conversation, but then we look for some qualitative assessment of that news coverage. Is it good? Is it on message? Is it communicating something different? Then we dug deeper to see if that coverage and these petition signatures actually impacted the conversation in the sector we’re trying to transform. Now we’re in the middle of a several month-long curator stakeholder research phase, conducting a ton of interviews with museum professionals, scientists, and environmental justice groups. We want to get a sense of what’s changed over the last two years since we launched the projects, and to inform our strategies moving forward.
Those “by-the-numbers” metrics around this notion of are we building power, a counter-power, and are we changing and transforming understanding of the goals or institutions that we’re intervening upon? I think they can be distilled into measurable, quantifiable things. But that’s challenging, and I think it requires an investment in time and a particular expertise in measuring and in evaluation, that most artists and cultural producers engaged in activism either aren’t trained in or don’t fucking have time to do.
[re the NYC Climate March] You could see the proliferation of hashtags for more complex narratives around climate change. I would love to see people do studies around these new areas that were introduced during the Climate March that had previously not been talked about, stuff like the relation between climate change and immigration, for example.
Social media is not getting at the heart of why these issues are meaningful and powerful, and mobilize human labor on a massive scale towards a particular goal.
I think social media is actually a stepping stone metric. It’s the first time we’ve actually able to measure the way in which people are responding to something on a massive level. Some people are just saying, “well how can we get more hits, how can we get more views?” And then there are all kinds of artificial ways to create that. So, people are going after the metric rather than the thing that actually causes the metric. We’re tricking ourselves in the lab, so to speak. I don’t think we have yet found metrics that really show meaning, but I do think that when you see massive levels of people willing to undertake risk, willing to exert their energy, particularly when they’re not being compensated by anything else, that’s when you start to see human meaning being actualized in concrete form.
Social media pickup is a big one. Sometimes you’ll get how the story is reported, like what the messaging is and whether it stays on message or not. And then you have this other metric. It’s just basically like what other people are doing. Like if people are laughing a lot, that’s how we know it worked. Or, if they’re very moved somehow emotionally, that might show it also worked. Also email feedback is another one that we like. We might get an email that says, “Thank you! This is amazing.” We used to get a lot more of that than we do now.
There are a couple different measurements. One is simply number of articles or amount of media attention and the quality of media attention. For the BBC thing we did, there were 600 articles in the U.S. press that wouldn’t have been there otherwise.
The Chamber of Commerce thing that begins our third movie resulted in a huge amount of press, and two weeks later the Chamber of Commerce actually caved and changed positions. It wasn’t because of our action, but it didn’t hurt. It was a drop in the bucket, at least. We can’t judge and say that actually resulted in that change, but it did add a lot of embarrassment to the Chamber of Commerce and a lot of attention paid to the Chamber of Commerce, and they became more worried about pressure.
People forget media stories very, very quickly. We went around a year after our BBC announcement, which was by far our most successful action ever and still is. We asked people in London if they remembered it, and nobody did. I think a couple of people did have vague memories of it. But it really doesn’t stick around. Whereas with our movies, I think our actions do make a difference. I think people see them and go, “oh, I could do something, too” … You have time to build an argument and to really let it sink in, develop characters and the whole thing. People get involved in the story. With the media action, they don’t get involved in the story.
And so, then the evaluation becomes the media roundup. And if you have a big enough organization or you have enough time – i.e., persons who track media – that will make it easy enough.
- Other metrics can be more engaging
Participants noted a number of other ways their interventions and activities could be measured and evaluated aside from media metrics. Many talked about how the narrative of an issue had changed, or new narratives were started, how new players entered the conversation, how morale was raised among advocates, what relationships were formed, how quickly ideas spread, etc. They also talked about the numerous unexpected (positive) consequences that occurred as a result of interventions.
Most found these metrics as being much more difficult to measure over time. Some also raised the issue of culture shifting. Many felt the actual law, policy or activity that they were trying to change was the ultimate metric. There were also jabs taken at the “nonprofit industrial complex” for their embrace of a culture of metrics and assessment that some feel should not be applied to this kind of advocacy work.
Our campaign was really changing the environment in which the campaign was happening. We were reaching “persuadables” in the right districts and the right states. We were getting these memes circulating and making our side feel sexier, raising the morale, giving journalists some more useful ways to talk about the issues and bringing up issues that hadn’t been brought up before, even if we ended up losing the election. Sometimes these secondary metrics are kind of what a particular effort is about.
There are intangible best practices and some are more tangible, like how do you win, how do you force the target to cry uncle and do what you want, and also how do you stay healthy while doing this work. Everything along that spectrum…how do you build community, how do you give people a meaningful experience, how do you operate ethically? All of these things, as well as how do you come away with the victory, whether that’s passing a piece of legislation or forcing a company to stop a bad practice or whatever. I think one of our metrics is to turn young people on to activism broadly, but also to instigate or foster or contribute to a culture of analysis, even though our version of it is not particularly quantitative. It is very analytical in terms of tactics and practices and strategic principles and how to make the theory shit that we learn…how to bring it to bear and make it relevant, why does it matter? How do you quantify that shit? That’s hard.
What’s forefront are relationships, and from that, I think we can have very meaningful dialogue around impact
How you measure culture shift is really hard, but it has to do with how people are treating each other, how a community sees itself, how the story of the community is changing and scripted by a community…I feel we have great, great qualitative press, we do a lot of storytelling about our work.
If we want to see how a new narrative was injected into the popular discourse, that’s a really interesting question, and I think there are answers to some degree… The economic crisis started in 2008. And really, there wasn’t a whole lot of talk about it, but suddenly Occupy happened, and there’s massive proliferation of the term “income equality” through the media that dovetails perfectly to the day of the kind of public life of the Occupy movement.
I think a lot of the conversation on metrics has been damaging because the things that are the most measurable are sometimes the least important. And I think that the most important ones are often the least measurable. As a result, the metrics and the money and support and the institution building all go in the other direction… I actually think this explains a huge amount of the diversion of the left for the last 30 years into professional forms of organizing that have actually depleted the things that are effective and meaningful. And that’s why most of the collective organizing that I see, that is really shifting the discourse, is coming almost totally outside of the professional vendors. We built up a nonprofit industrial complex for 30 years, and it coincided with total increase in the power of the left. And then these breakthroughs are happening from almost totally outside the system that we created in order to address the problem.
If we look at massive occupations, like Black Life Matters, that activists are doing, then we start to see a much more concrete manifestation of how meaning is motivating people to undertake these forms of action. And if there are ways to get at that, then I think we’re going to get a lot closer to finding things to help increase the accuracy and power of the kind of work that we’re all doing.
If we don’t get a lot of media attention for things, we don’t despair. We think of a secondary use for the material. Like this gun control thing we recently did, it got a certain amount of attention because the NRA reacted a certain way and took down thousands of websites, and it turned into a story. But what we’re really doing with that piece is turning it into a television pilot.
I think the most important thing is to build movements and bring new people into them, make people more activated. That can be done with or without media attention. It just happens to be what we’ve enjoyed doing and are good at and do. It’s not necessarily the best way to do it.
I really saw New York change, its people, after Occupy. They were different. They didn’t go after jobs as they did before. They started to think changing some things in their own lives because they didn’t work for them anymore.
So even with our Natural History Museum project where we have sort of traditional metrics for success, we’re still aiming for something beyond that. The world of metrics lives down here, but it’s like above this level of getting museums to do X, Y, Z, there’s this level. And above that there’s this level, and above that there’s universal egalitarian emancipation? I don’t know how you fucking measure that. How do you measure culture change? I think it is measurable. You can look at things like how much are people talking more about something, how much are things being initiated independent of you. Is it just this stuff that you’re doing as a heavy lift, or has it become contagious? Is it manifesting in ways that you hadn’t even imagined, like when the Black Congressional Caucus stages a Black Lives Matter and walks out themselves. Or the American Anthropological Association has a die-in with 1000 anthropologists in the middle of their convention. Wow. We didn’t plan on that, so really those are things to think about in tracking and communicating.
I think this is where the question of counter power and contagiousness come together, because if you’re producing things that are not associated with an individual, they can be contagious because they become signifiers of a collectivity. The measure of success is that they become something that other people pick up and use. So, for example, a hashtag is an example of that. For years, we were producing all these props. We would organize a demonstration and we would lay them out as people used them, then that was a measure of success. They started to be reproduced in other cities or around the world, then that was another measure of success.
- Limited use of other methodologies for measurement
Overall, there was little reported use of methodologies from other fields to help assess artistic activism interventions. Social marketing was mentioned by a few. The design field and PR were also cited. To put things in perspective, one practitioner suggested that all fields struggle with measuring impact.
Social marketing might be a tool that could be useful for us.
We look a lot at design but not exactly from a measurement point of view, but use design as a way to assess if the product or tool that we’re creating actually has an effect on someone, is it useful. That’s the only other field we’ve looked at.
(on PR) Corporations send out press releases all the time, news videos. They never reveal that they’re hoaxes. We do. But we use exactly the same kinds of techniques except that we reveal the hoax.
Measuring impact is not done well across pretty much every field.
- On language
Many pet peeves about the language used in artistic activism, and especially the language regarding measurement and assessment, surfaced during the interviews. Some feel that language issues often block great ideas from developing or being able to fully describe the impact practitioners are seeking. “Alienating” was a commonly repeated phrase for this language, and the need for different words or grammar of assessment was called for.
I guess” strategic” and “creative” are handmaidens to being more effective, possibly…though they may also be useful in their own way…I think “impact” is something that we know is important, and know that it matters to funders. We want it to matter to funders. We want the money to go where it could be most practical, but I would say we have a little bit of trepidation around that word.
I think that there is this NGO-ish strain in artistic activism that uses the word “culture” more than “art”. And “impact producers.” When you get into documentary film or media projects or video, there’s this whole ancillary industry of consultants or contractors that try to translate the arts or the media into “impact and upscale responsibility.” But in the world of art, which is where we’ve historically been a little more ensconced, people say “socially engaged art.” I think that these metrics for success, or ROI, or whatever you call them, is sort of alien and I completely understand why it annoys people.
I think we’ve internalized a lot of professional language. There are many historical reasons why we did that. I don’t think it’s helped us. These are all arbitrary languages. None of them have any inherently deeper grasp on the problems, and when we use the language of a particular community we’re communicating to people, “Yes, you are part of this, or no, you are not.” So, I think some of that language could be alienating.
If you use “measurement” and “evaluation” with my local activist group, they’re going to be like, yeah, whatever, and give you the glazed-over look.
Conflicts like climate change need something different in terms of language, something that we have to believe in. We don’t have the words, we don’t have the grammar we need. We can only create the right words if we really change ourselves. We need to break ourselves apart from our capitalistic selves to do this.
- What would be useful?
Participants cited a few areas where new tools and thinking would be helpful for them. Simple assessment tools that were not reliant on a lot of number crunching was one area suggested. A better understanding of how relationships form between people and groups in the organizing process was another area mentioned, as well as understanding how collective identities work and how they attract people to a movement or action. There was also interest in looking at how iconic movements or actions, like Rosa Parks sitting in the white section of the bus, actually inspired social change. Or knowing what the tools are to analyze such a change. One participant mentioned that having a better understanding of how artistic activism works could be useful in convincing large unions or other groups to utilize artistic activism more often. Easy to use assessment tools and tools that are human scale were called for.
If you can vet some easy to use assessment tools, but also roundup some of the best stuff out there, that would help. Assessment tools to understand your audience. There’s a lot of academic assessment tools, things that are going to matter for just multi-million dollar grants or obscure journals. They may be important but we’re not in a position to make use of that. I would imagine 99 percent of the work-a-day activists and artists aren’t either. Tools that are human scale that we can lay our hands on and actually use in the course of own practice, where people are really busy and have limited capacity, resources and training in this area. Something where there aren’t too many numbers…numbers that matter in terms of really being able to evaluate things…besides just list shit.
[re 2014 Climate March] What we were able to see was how this was operating collectively at a very large scale, and creating new webs of relationships that many of them are doing to this day – a number of organizations came out of that effort, as well as things like shifting the popular narrative around these issues. I guess the short answer is I think that we need more metrics around collaborative mobilization about relational density, looking at the densification of relationships within certain communities that are engaged in the creative practice, as well as a marriage just on a large scale.
It would be really interesting to see a focus on the way that new identities – collective identities, political identities – are created. I think a lot of that has to do with artistic practice… when you look at Occupy as an identity, Black Lives Matter as an identity, Creative Defenders as an identity, you see that people are highly motivated to act because they have internalized that identity. It has become meaningful to them. But it’s something that is shared with other people… if you look at the history of the labor movement, so much of the art and culture that accompanied it was about who we are as a labor movement and as a working-class people. There was a class identity. It was about union membership and it wasn’t just about contract fights and negotiations and policy, it was about redefining who is “we” and who is the “they,” and creating that narrative that people, that story that people were then living with the activism.
I was thinking of those historical examples that are really large in our imagination as having worked. Certain examples of some symbolic action that caused a social movement – I would be really curious to try to understand the relative success or value of those actions. I’m curious about it because I think a lot of times you think of creative actions as something that’s kind of new, even though it’s been going on forever… Was Rosa Park sitting on the bus effective? It was important in the civil rights movement. I think if you asked most people they would say yes, it was. But I don’t know how you would actually measure it. I’m sure there’s a way. These symbolic moments have played such a huge role in our understanding of these histories, we assume that they’re important.
Take the unions, for example. They do a lot of planning. They have the resources to do it. They have professional researchers on their staff, there is a lot of money on the line and there’s a lot of economic impact. They want to know how will this benefit my members; how will this impact the local economy if we’re able to unionize this plant. They are also sort of sketchy about the value of creativity and arts. If you could really develop a tool or tell some stories where you make the case that this is valuable and important and strategic, and you make the case with numbers or hard data of some kind in the kind of language that they respect…where artists were brought in on the ground floor and were able to help craft the messaging and designed a good website as opposed to a crappy union made website. That would be really powerful and that’s where someone who gets it, who has that instinct, can then use this truth, if you will, or data, to make the case up the chain. That’s where this could be really useful, and it’s not just in unions but could be in art NGOs and elsewhere.
- On why they’re not doing more evaluation
Time, money and lack of skills or knowledge are typical reasons why activist/practitioners haven’t done more evaluation of their work, as well as more of a focus on strategy rather than a comprehensive program that includes evaluation.
It’s something that requires different skill sets than what we have. Things like polling, focus groups, etc. We were looking into prices of doing research through a focus group company, and the prices were so exorbitant that there was no way we could afford to do it anyway… A lot of groups test their emails to see how people respond and pick up and stuff like that. We’ve never done that. There are only a couple of us and we’re all part-time. (laughs) We try to write a good email and sometimes it doesn’t really work the way it’s supposed to. But it’s probably our fault for not actually testing it with a small sample…If we had to hire somebody to do it, we probably wouldn’t because we’re pretty strapped for cash and time.
Measuring impact is not only having things in place to measure –we very rarely have any money to do it… the whole field is dominated by strategic methodology pieces, and so funders and organizations and people sitting down to do strategy don’t pay enough attention to evaluation.
- Thoughts on a standard framework
Participants were intrigued by the idea of a standard framework for evaluating artistic activism interventions. They would be willing to give it a try if they felt it could offer value for understanding how the intervention works. A methodology that included both objective (numbers and metrics) and subjective (story) elements would be of most interest.
I have a feeling it would be hard to come up with something that would be one-size fits all, but it would still be very useful…there’s no standard for reporting in the media, but you can choose one side or another side, and I think metrics, how you read metrics, are the same thing. So basically, there’s a radical way of reading metrics and then there’s a conservative way of reading metrics. And I think that basically this is like the argument for the story that the metrics tell, so I’d be more concerned about the story that people tell about the metrics than the metrics themselves.
Yes, if it can help us see into some of those black boxes, some of the dark areas. We’re trying to see the dark matter that’s out there and detect it and really understand it. If we can somehow observe the process that happens between a work of art and an idea reaching a community, and then that community taking a defiant form of action, that’s the black box.
Implications/Recommendations
Implications
These interviews suggest that there is no standard thinking about the topic of aefficacy in artistic activism projects. While there are some individuals and groups that follow very focused and regimented methods for planning their interventions, there appears to be a looser need for formalization. That said, much more energy is put into the planning phases of interventions than to the assessment/evaluation phase. As some trainers only work with groups through implementation, the assessment/evaluation piece is sometimes an afterthought to them and/or their trainees. Pressure from funders often instigates formal assessment reporting.
Despite some resistance to or resentment of this pressure from funders, most interviewees would like to have better assessment tools. For many, press coverage/media metrics become their de-facto evaluation method. While these metrics may provide a lot of information and data, most feel that they are intermediary to the larger social change or advocacy goals they have set for their interventions. Participants feel they often don’t know enough about media metrics and don’t have the time or resources to bring other measures or resources into their processes. Interestingly, many of the measures they mentioned they would like to bring into their processes are not that difficult to assess. They do require more thought during the up-front planning phase of interventions.
The majority of those interviewed would be interested in tools that could help them better assess their work. The tools need to be easy to follow, and need to have the right balance of hard data (metrics) and soft information (storytelling). Some feedback loops with these practitioners would be helpful as these tools are developed to ensure they are on track in terms of content, and that they would actually be utilized.
Recommendations
Based on these findings and implications, it is suggested that C4AA take the following steps as we proceed with this project:
- Investigate the organizations/individuals whom interviewees think are doing good work to add to the learning.
- Ensure that toolkits for both practitioners and funders are easy-to-follow and implement, and make evaluation/assessment part of the planning process.
- As part of the toolkit, draft a theory-of-change model that could be used on a turnkey basis by groups and organizations.
- Investigate ideas for including easy-to-plan outcome metrics and data collection methods in the toolkit.
Discussion Guide for Trainers/Field Defining Organizations
Revised Draft May 23, 2016
Background
Following the recently conducted literature review, C4AA is interested in conducting interviews with artistic activism trainers and other “field-defining” organizations to expand the learning for this part of the AEfficacy project.
Objectives
The objectives for this research are to:
- Build on the learning from the lit review
- Understand the challenges and opportunities to developing a culture of measuring impact in the field
- Learn what tools and techniques for assessment are currently being used in the field (or not) and how effective they are
- Identify potential tools that would be helpful to the field and what needs to be done to develop them
Methodology
Participants were sent a copy of the lit review report and asked to read it before the interview to provide them with some context. Where possible, multiple members of an organization will be interviewed together. Interviewees will be told up front that they will be audiotaped and that their comments (with attribution) may be used in a report that will be developed as a final stage of the project. The interviewer will also provide a brief summary of the project and its work to date at the beginning of the conversation.
Key Questions
Background
- What is the service your organization provides and with what audiences do you work?
- What is the basic process you use for your training or project development?
- What theories of change or frameworks underlie the work you do?
Issues around measuring impact
- Overall thoughts about measuring impact of artistic activism
- How do you define impact?
- How well does the field do this?
- How well do you do this?
- Is there a focus on measuring impact in the field now? If not, how can it be developed – or is it needed at all?
- Anyone in the field doing a good job?
- Who’s driving the process? Funders? Academics? Others?
- Were any of the tools or ideas mentioned in the lit review relevant to your work?
- Challenges in regard to assessment and evaluation
- Is it part of your planning/training process?
- Do you set goals/objectives? Is there a specific behavior, law or practice you are looking to change?
- Are there models or frameworks you utilize? Are they useful? What’s missing?
- Are there specific metrics you use?
- Outreach, new audiences, media pick up, social media buzz, attitude, behavior or policy change, etc.
- Do you utilize models from other fields (which) and if so, what works best?
- What are the specific issues you have regarding measurement and assessment?
- And what do you think of the language used? Measurement, Impact, Evaluation, Assessment, Effect?
- Does it matter if artistic activism projects are effective?
- Who measures such work currently for you or other organizations?
What’s needed?
- Would a standard framework, model, or theory of change for impact measurement in the field be helpful to you? How might you use it?
- What would need to be part of it?
- How should it be developed?
Anything else you would like to share about the topic?
