Paper summary - Requirements Framing Affects Design Creativity, by Rahul Mohanani, Burak Turhan, Paul Ralph

What I read

In this paper, the authors describe a study that explored the relationship between how software requirements are presented and the creativity of the result. They conclude that when software capabilities are presented as formal and structured, design solutions are less innovative; small changes in how requirements are presented have large influence on design results.

First, the authors describe the context of requirements in software engineering, and present their research question and focus—does the framing of desiderata affect design creativity? They define desiderata as a property that is “wanted, needed, or preferred” by stakeholders. They define creativity as “a cognitive process of generating one or more ideas that are not only novel but also feasible.” They indicate that this work is an extension of work previously published by the authors.

Next, the authors describe the ideas of fixation and framing, discussing previous research outside of the context of software engineering. The framing effect shows that people respond differently to problems based on how the problems are presented, even when the questions are objective and the two contexts are equivalent. Fixation and specifically design fixation is the “tendency for designers to generate solutions very similar to given examples or existing artifacts”—designers focus on and embrace things that have already been done.” They present this as “task framing causes fixation.”

The authors revisit the definition of creativity, and focus on the idea of “novel” and “practically useful” as being criteria for a creative output; the output can be assessed to determine creativity based on judging originality and practicality. They note that the role of creativity in the context of software requirements is controversial, as some feel that requirements are discovered, while others feel that requirements are extracted through various workshop-style activities.

Next, the authors describe their research method. In two related experiments, the researchers presented the same software capabilities in different ways—as ideas vs requirements. They began with four hypotheses, related to the level of originality of ideas as compared to the way software capabilities were presented. They designed a set of desiderata for a hypothetical mobile health application; various versions of the list presented the data in different ways. One list showed the items as “ideas” and used the word “might.” One showed the items as “requirements” and used the word “shall.” One version prioritized the items into five levels. The documents were given to participants, who then designed their solutions. These were judged by two “expert judges,” who graded the output on a scale of 1-5 for both originality and practicality.

The authors analyze their results. I will skip a summary of the detailed statistical analyses performed on the data set.

The authors then discuss their findings. Their meta-conclusion is that “presenting the same collection of desiderata in slightly different ways has a profound effect on design performance.” As a result, “when a project calls for innovation—significant departures from current approaches—desiderata should not be presented as requirements. In contrast, when a project calls for more straightforward, archetypal solutions, presenting desiderata as requirements is more appropriate.” They then provide educated guesses to the question “Why would designers produce design concepts that are less original and more practical in response to more formal desiderata framing?” One reason is that the formality of the word requirement implies that the problem is well understood and solved, and exploration isn’t needed. This is a form of fixation on the requirements.

They offer implications for further research, practice, and education. They recommend additional set of trials to expand the knowledge that they have generated, and for practicing software professionals to think about the way this knowledge impacts their requirement presentation—to “encourage them to critically evaluate and recognize real requirements from the dubious ones to produce more creative solutions.” They also recommend teaching software engineering students to take on more abstract projects, and to present them with a deeper understanding of some of the cognitive phenomenon that underly design.

The authors briefly indicate the limitations of their work. These include: the population was not randomly selected; originality and practicality are constructs with no objective scales; there are other dimensions of performance; a focus was placed on output, rather than the people making the output.

They conclude that “this paper highlights the innate tension between creativity and software requirements. It reveals that the way a specification is presented might be just as important as its contents.”

What I learned and what I think

The idea that changes in presentation of requirements impacts the output makes intuitive sense to me, and in some respect matches what I’ve seen in practice. It’s intriguing to me that this can come through with simple word changes. I’ve certainly seen it based on different styles of desired capabilities, which are often based on the culture of the company and the demeanor of the product manager.

This is related to the “who goes first” aspect of engineering. If design is leading with concept and vision work, based on close discussion and iteration with product, the results tend to be more exploratory (although not necessarily “more creative” as I don’t really know what that means.) When product has articulated a set of story stubs and presented these in a meeting in some sort of spreadsheet or Powerpoint, designers work to those stubs. And when a story has been authored in full, or presented in an arduous PRD, designers just do as they are told.

There’s a certain type of designer that flourishes in the later contexts, but most that I know hate it. There’s a similar distinction in engineering, and I’ve noticed this seems to related to age/experience. Older developers seem grouchier about not knowing what to build, while newer ones seem okay with much more ambiguity. Outside of startup land, I wouldn’t say they particularly like the ambiguity, but they don’t just dig in and wait.

I would like to read and learn more about how “permission” to “reframe” requirements impacts the output.

The authors mention that requirements can be presented in different forms (lists, use cases, stories) and that wasn’t a focus of this study, but I would like to understand how those forms impact output, too. Many people I’ve worked with aren’t hotshots with language, so hoping for attention to subtle changes in wording will probably be a non-starter, but they can probably be convinced to change the vehicle in which requirements are provided.

I also want to see some form of study on sentiment of designers and engineers based on the controls here, and trying to find a breakdown or discussion of “types” of people drawn to each form of work (defined vs loose.)

As with many of the other papers I’ve been reading, I take issue with the definition of creativity. It is, and isn’t, the pursuit of uniqueness and usefulness. Moving stupid components around in Figma is a creative activity, even if I don’t like it, and the output is probably not unique and arguable on the useful scale. A lot of software that’s designed isn’t useful, but the design of it is still creativity at play. Bad designers make things that are both un-useful and unusable, but their work is still creative.

The authors recognized that there’s no objective measure of creativity, yet they had experts “score” it with an objective measure anyway. I understand the methodological challenge of conducting experiments around this, and I keep returning to the second half of all of these studies and activities. The researchers conduct a thorough and thoughtful set of actions with participants and generate a lot of data, and then they push the data through rich statistical analyses and models, and that makes no sense. The results of the study can be extremely valuable without pursuing causality; and sure, they may be “wrong” and then “disproved” later, but for now, if they pass the sniff test and make people happier, more productive, more successful, whatever, who cares? Design, creativity, software engineering—they aren’tWorld Is Flat And Now It’s Round things. We shouldn’t treat them as if they are.

In this article, their stated goal was to answer the question, “does the framing of desiderata affect design creativity?” Why does it have to be answered in the context of a traditional empirical study, with attempts to isolate variables, add control, and then formalize p values? “Framing of desiderata” exists in such a convoluted space: a business. It makes no sense to try to remove the situational and contextual externalities, even if one could, because they are always there in practice.

All of that said; there’s a pretty clear connection between this work and the things I’m interested in, in the context of design systems. Using a system is a massive set of requirements, internally provided from designers to designers. Its purpose is to normalize, but if you ask designers of the systems, I doubt they’ll tell you it is to constrain creativity. But I’m pretty convinced it does just that, and for the same reasons here. There is a fixation on using the pieces and parts of the system (and avoiding making new Lego pieces), and the perception that things are immutable and have already been decided is present. Decided by whom? It’s a power conversation, too; and it’s also a risk management issue, with “vetted” parts theoretically minimizing risk of newness; and it’s also a resources issue, with components theoretically saving time and money.

I liked this work. I’m going to pull more from these authors, and from many of the citations they provide.