Paper summary - A Systematic Review and Thematic Analysis of Community-Collaborative Approaches to Computing Research, by Ned Cooper, et al

What I read

In this paper, the authors conduct a literature review of computing papers to understand and reflect on the nature of participatory research with communities. They identify that power dynamics are central to the corpus of papers selected.

First, the authors describe the slow but steady change in computing research towards participatory design approaches and action research, in order to better be inclusive and supportive of communities. This is broadly labeled Community-Based Participatory Research. They indicated that this paper was intended to systematically “identify, review, and reflect on findings” that emerged from this type of research; they introduce a new phrase (“Community-collaborative approaches”) as a container for these types of studies.

The researchers pursued these questions in this work: how is participatory work described, in what types of projects is it used, what elements of the work are described in published papers, and how does the content of the papers align with or diverge from typically accepted norms related to participatory research.

Next, the authors provide a history of the relationship of HCI with participatory approaches. This was a progression from a focus on “sociotechnical systems” in Europe, and research in organizations, through democratized workplace planning (in the context of unionized workers), into the world of CHI and computing research. Many of these examples (in CHI) were focused on being inclusive of an individual, rather than a group at large. By 2010, papers in CHI began to reference a more group or community focused form of participatory research; simultaneously, academic conversations turned towards the nature of participants as “subjects”, where research was intended to benefit a researcher’s agenda rather than enabling the community to develop their own agenda and enact their own social change. The authors note that this conversation is new to the HCI community, but not to the world of social science. Design justice (and design for social justice) is an evolution of this approach, where research is conducted for, with, and by the community.

The authors recognize that the community has increasingly been engaged in debates about the ways to engage with communities in research. For example, “identity politics” may be at play when working with participants with disabilities, and their participation in a research program may be seen as tokenizing, or inadvertently “outing” them to the world through authorship. They conclude this section by further explaining the goals of their research and analysis—identifying how some of these issues and concerns have, or have not, been historically considered and addressed.

Next, the authors explain in great detail the method used for the paper selection and analysis. In identifying the papers to review, the authors used the “PRISMA” approach for selection, keyword searching across databases of peer-reviewed materials, down-selection through screening and eligibility, and discussion. For their deep investigation, the authors identified a set of content themes to extract that would be present in each (or in the vast majority) of papers. The result of this was a dataset of content from 47 articles, which could then be interpreted.

The main analysis and interpretation process followed Braun and Clarke’s reflexive thematic analysis. As they describe, this recognizes “the influence of researchers in interpreting data, and encouraging researchers to reflect on that influence as they develop and refine codes.” The authors used an inductive approach to identify the themes.

Next, the authors explicitly describe that influence as referenced above, in a section called Positionality of the Research. In this section, they describe their perspectives and viewpoints, as separate than but as a lens for interpreting the research data. This is a reflection of their experiences, and also their feelings about these experiences and the context in which they find themselves. For example, they explain that “we feel a kind of dissonance representing sometimes conflicting roles” as scholars who are also activists.

The authors conclude the methodology section by acknowledging limitations in their approach, particularly the idea that a reflexive and reflective perspective is interpretative. This limited the depth of analysis, because of the amount of content available and amount of practical time necessary to evaluate the material.

Next, the authors describe their findings.

First, they describe statistical findings related to publication year, venue, location, community, research approach, technology developed, and timeframe. Next, and in much further depth, the authors describe the dimensions that resulted from the thematic analysis. These are presented in large groups (Establishing a Partnership, Enabling Participatory Models for Decision-Making and Collaborative Research and Sustaining Results of a Project), and then in subthemes.

In the first theme, it became apparent that authors establish community partners, although in different ways. Some directly engaged with a new community, while others already had a community connection. Only a few included the community members as parts of the research team itself. The majority of papers described building some sort of relationship and trust with the participants, often before beginning the project. There were different ways of building that relationship, including relying on the community itself to identify a problem space, collaborating to identify problems, and situating the team itself in the community for at least part of the project.

In the second theme, the authors describe that they found a mix of ways to incorporate shared community knowledge into the published work. Nearly all researchers engaged in “collaborative ideation” activities in solving a problem, but few examples of researchers conceptualizing the problem to solve with the communities, in the context of the study (as compared to the work done before the study began.) About half of the researchers included participants in prototyping and critiquing prototyping. Many of the researchers and communities shared control over the methodology that was to be used, and the decision making that was used, although only four described analyzing gathered data in a shared manner. Most of the articles explicitly referenced the power dynamic at play, and the steps taken to mitigate it. The majority of papers indicated that the work was multi-phased and iterative, and highly adaptive to the way the work played out over time. But, few indicated the burden this form of iteration might place on the community being engaged with.

In the final theme, the authors explored the way researchers describe the results of their projects. Most of the papers discussed the attempts that were made to provide benefit not only to researchers but also to the community, and many identified those benefits. “Capacity building” was one of the most described benefits, as participants learned and used new skills throughout the project. For researchers, a benefit was understanding the community and its context better. All of these benefits are often deferred, with value being realized in a longer term. There were few references to the measurement of benefit for the community. Many of the articles described the value of sharing the research results “alongside the community.” And, many of the articles described the way in which the output of the research would actually be used in the community after the research was concluded.

The authors go on to discuss the implication of the findings.

A primary finding was the limited engagement of the community, and the authors hypothesize that this is because of the pragmatics of publishing a paper. The academic culture values “tightly-scoped” projects, which then limits the ambiguity of allowing the method to emerge by and with the community itself. The authors challenge the communities of HCI researchers to think of new funding models and publication models that allow for and encourage a more fluid, community-led approach to the research itself.

Next, the authors propose four opportunities for the research community to change, in order to better support community-driven work. First, the authors recommend a richer view of what constitutes a community. Communities are not necessarily formed by agreement or like traits, but research often focuses on that commonality. Next, the authors recommend “contestability” in the research community. Instead of working to triangulate on shared and accepted approaches and attitudes, the historic “dissensus” of the HCI community can be used with communities to stray away from the accepted norms of publishing. Third, the authors indicate that few papers described reorienting the entire process to be community-out, rather than HCI-in. Researchers adapt existing HCI methods to the community, rather than the other way around. Finally, the authors indicate that it’s unclear how long-term benefits of community engagement are measured, or even if those benefits exist. Metrics should shift to include qualitative measures of success, long-term stability of relationships between researchers and community members, and more longitudinal post-study engagement.

In the last part of the paper, the authors reflect on a reflexive practice itself. Researchers should not approach their community work as scientists, and should not seek objectivity. Instead, the role of the researcher in this form of work is as a reflective participant. This means reflecting on the reflective nature itself: thinking about the impact of the researcher in the community, and describing that reflection—the tensions and conflicts that emerge during a project—as a part of the research itself.

What I learned and what I think

This paper was a recommendation from Katie, based less on the subject matter and more on my request for examples of how reflexive thematic analysis is used and described in peer-reviewed papers. I find the subject matter interesting, but I find the way it’s presented more interesting.

The authors describe and cite reflexive thematic analysis, indicating that it’s bottom-up. It’s only a brief description and reference, and that may mean that it’s becoming more generally accepted as a strong and “acceptable” approach for synthesizing data. They separate their themes into three major sections, and then into minor sections, and that organizational method leads to a narrative-style discussion which was easy to read and understand.

The theme names were formed in a different way than I’m familiar with. For example, one was Establishing a Partnership, with a sub-theme named Defining a Community Partner. These are actually the inactive style of labeling that I discourage my students from using during design synthesis, and that we avoid on client projects. Instead, I might write these as “I struggle to identify a partner to work with if I haven’t already built trust in that community,” or “Researchers struggle to identify partners to work with if they haven’t already built trust in that community.” I wonder if that assertiveness is avoided because it’s not 100% accurate; it may be that if even one paper contradicted that, it would need qualifiers (like “Some researchers” or “Researchers often”) to the point of being silly or overly timid.

Our goal with active themes is to provoke design, and to provoke a sense of urgency with our clients. There is something missing in their product roadmap, or in the way they are viewing the customers, or in the way they aren’t addressing a market need, and they need a bias towards action of addressing that miss. I would think academic researchers would want the same form of urgency, but I also see that it wouldn’t hold up to scrutiny in a very literal peer-review process; I don’t think there’s a lot of “read between the lines of what I wrote to see what I meant” going on here. It’s another case where I am not there in person to explain my assertion and address concerns like “Well, not everyone does…”, and so the paper needs to address it directly, or avoid it by using passive themes.

I noticed the section “Positionality of the research” as a non-standard part of the paper writing that I’ve seen so far, and as one of the sections that Braun and Clarke recommend including explicitly. I like this, both because it justifies the method and almost “speaks away” criticism, but mostly because it makes my interpretive lens clear and strong. Qual research isn’t objective, and it shouldn’t try to hide behind that objectivity in order to fit into a culture that is used to scientific “prove or disprove” approaches. This section is a way to acknowledge my bias as being valuable, not as a deficiency.

I want a more detailed guide on how to do the reflexive thematic analysis in the “right” way, so I’ll try to track down original content from Braun and Clarke to read next.