How Templated Requirements Specifications Inhibit Creativity in Software Engineering

What I read

In this paper, the authors performed a study with software designers to understand the role of requirements in designing creative solutions. They determined that designers fixate on “desiderata” when they are presented as requirements, and this limits the creativity of their solutions.

First, the authors describe the different ways that researchers think of software requirements, with one extreme viewing them as being discoverable and then fixed, and the other viewing them as created and vague. The authors propose a research question: “How do fixation and critical thinking explain reduction in design creativity when desiderata are presented as templated requirements specifications?” They define some of the language that will be used in the paper, and explain that “Design creativity… denotes the originality and practicality of new product systems.”

The authors next describe the research backdrop upon which this work was grounded.

First, they revisit desiderata, which they presented in previous work, as the properties of a system that are wanted, needed, or preferred. Successful articulation of the properties as requirements means balancing the specifications accepted as truth, and the “naturalistic approach where requirements are a product of the conflicts and contradictory viewpoints of the stakeholders involved.”

They then compare well-structured problems with ill-structured problems, and reference previous studies that show that “Over-concentration (over structuring) on problem definition does not necessarily lead to successful design outcomes.” One of the reasons is a fixation on requirements—the “tendency [for designers] to attribute undue confidence and importance to desiderata presented as [fixed requirements].” The authors observe that expert designers tend to view all problems as ill-structured, even when the problem is presented as fixed, but novice designers are “less fixated on the problem domain and, hence, are able to generate highly creative solutions.”

The authors then describe creativity in the context of requirement definition. They discuss how various forms of design workshops are used to elicit and clarify requirements. A “Six P” model for thinking about how creativity shows up is described, organizing creativity as: 1) creativity’s underlying cognitive process; 2) the creative product; 3) the person (or personality) doing the creative work; 4) the place (or context) of the creative work; 5) stimulating creative thinking (or persuasion) and 6) improving creative potential.

The last part of existing research that is described is fixation—the bias where designers have “blind adherence” to existing concepts. This fixation is moderated by several factors, including the commonness of examples that exist (common, as compared to rare.) When requirements conflict with one-another, designers may delay committing to (fixating on) a solution.

The authors explain the research approach used in this study. In pairs, designers were tasked with designing a new, hypothetical application. They were provided with a short list of requirements, and in 60 minutes, each pair worked to solve the problem. Because the designers worked in pairs, they spoke to each other, and that dialog generated data for analysis. The experiment was repeated twice, once with professional designers in a design consultancy, and one with students. The generated data was transcribed, and then explored in two ways. First, inductive coding was used to explore the way participants worked through the design problem, with a focus on “words ending with -ing.” Next, closed coding was used to analyze the data against the ideas of fixation and critical thinking.

The results of the analysis led to seven categories of utterances. These included making design moves, uncritically accepting, rejecting, grouping, questioning, assuming, and considering quality criteria. Some behaviors were most notable. Nearly all participants uncritically accepted their design ideas as they emerged. They also rejected requirements, but primarily rejected those that were described as low-priority. Some parts of the task that was provided were universally accepted, while others were questioned.

During the closed-coding mapping of utterances against fixation and critical thinking, several behaviors emerged. Some requirements were uniformly accepted without discussion or hesitation. There were many occurrences where teams fixated on an initial solution and never returned to question it. And, some designers fixated on existing popular solutions, and were either informed by those solutions or copied them. There were very few instances of designers critically thinking about and challenging the parameters of their task, or the designs they had generated.

Next, the authors discuss the implications of the work. The overall finding was that “providing designers with templated requirements induces requirement fixation and hinders critical thinking, thereby negatively affecting creativity.” In practice, this means that “When creativity is a priority, avoid templated requirements and over-structuring, over-simplifying, and over-rationalizing problem statements.” Designers should receive desiderata in a way that encourages critical thinking. Researchers should “abandon the naive view that analysts elicit requirements and that design transforms them into appropriate system features.” Instead, they should view requirements as an in-situ process. “Design is a creative, improvised, non-deductive process in which designers imagine new systems rather than rearrange old ideas.” And educators should stop teaching legacy techniques like waterfall models, and should increase open-ended and ambiguously structured assignments.

The authors describe an underlying philosophy of their research, analysis and synthesis, one that is called Critical Realism, which is different from the commonly accepted extremes of either Positivism or Constructivism. In this way of thinking, “critical realism assumes that the phenomena that scientists study are real whether they can be directly observed (e.g. people, length, Mars) or not (e.g. electrons, creativity, quasars). But because social reality resists experimental closure and is rich in unobservable properties, reality is only imperfectly and ‘probabilistically apprehensible’. Rather than discovering causal laws, scientists therefore construct explanations based on ‘generative mechanisms’—the powers objects have to influence each other.”

They introduce this philosophy because it supports their methodology and provides a different set of criteria for a reader to evaluate the work. In critical realism, validity includes “ontological appropriateness”, “contingent validity”, multivocality, “trustworthiness”, “analytic generalization” and “construct validity.” They then indicate some potential pragmatic limitations to their study and to generalizing their findings.

Finally, the authors conclude by summarizing their process, and describing their main contributions: “(1) it advances a theory that explains the (previously established) relationship between templated requirements specifications and design creativity; (2) it elaborates the concept of requirements fixation; (3) it presents a simple taxonomy of software design actions.”

What I learned and what I think

I appreciate the topic, as I’ve observed that flexibility around requirements is one of the biggest predictors of software project success (the other being talent of the team members.)

“Requirements” has consistently bothered me as a term and a concept. Required? By whom? Why? In order to do what? The separation of “functional requirements” and “experience requirements” is even worse, as if they two can be or should be separated. I understand when things actually are rules, set by a government agency (like FedRAMP compliance or conformity with 508) in order to “get something”, like certification. But experience-based and capability-based requirements are non-sensical. And attempts to formalize them in a consumable way (a way that will actually be read by a developer or a designer) means reducing the things someone has decided are important into concise statements, and so they need to be made explicit—and so they become prescriptive.

There’s something in here about the relationship between requirements and constraints. Design benefits from constraints, and the type or source of the constraint doesn’t matter; it might come from the material or the process of designing or from a user or from an executive. But a constraint shapes a solution space, not a problem space. It contextualizes solutions in a sense of right-ness, and becomes a subjectively stood up objective way of judging success. And constraints are flexible, again based on the “rightness” of how the design emerges. I think requirements get conflated with constraints, particularly by junior designers and in cultures that are executionally-oppressive (overindexing on speed and delivery as compared to quality and craft.)

I really, really appreciate the introduction (to me) of critical realism as a distinction from positivism and constructivism. I’ve always believed that there is an objective reality, but we can’t see it because we’re in it. This is supporting that, at least broadly. And it helps me rationalize the want to quantify things when researching things that can’t be quantified. The paper again confuses me in having Important Counting Of Things and Important P Values, but maybe it’s a way to get a good thing published: “we have good content, and if the way we can get it in the world is by playing by these rules, we’ll do it.”

I do wonder about the “replication package” that helps another researcher perform the same study. Super useful, and I wish more papers did this. But to what end—is the goal to confirm the findings? Why? And what does confirm even mean here?

I’m a little confused by the way the results are presented. The findings as stated in the conclusion are “These results suggest that presenting desiderata more restrictively as templated requirements specifications is associated with less critical evaluation of task structure and less critical thinking. In other words, templated requirements specifications inhibit design creativity because designers get fixated on desiderata presented (i.e. written) restrictively, well-structured and constrained language, hindering critical thinking.” I believe that. But that’s not what I gathered from the previous analysis, which seemed to indicate that there was not a great deal of uniformity across pairs.

A big part that’s missing from this discussion is the role of culture and pressure and power on the way software is built. It makes sense they excluded it, because it’s a totally separate hairball (it’s literally the hairball in Orbiting the Giant Hairball). Companies that are presenting requirements as fixed have a certain type of culture. It attracts a certain type of product manager, a certain type of design and development talent, and a certain type of management. There’s a different level of maturity for companies like this, and I don’t mean in a good way; when some companies reach this level of prescriptiveness, it’s because they’ve grown to a point of revenue or market-share saturation where innovation isn’t going to do anything valuable. Instead, it’s crank out features, probably related to in-product marketing (I’m looking at you Intuit). Maybe “innovation” and “creativity” has no place in these companies, and they should be blown up/disrupted/whatever.

I’m also back to the definition of creativity, which is at the heart of the initial research question (“How do fixation and critical thinking explain reduction in design creativity…”). Sometimes creativity means originality and practicality, as they describe. And sometimes it just means practicality. This statement is really problematic: Design is a creative, improvised, non-deductive process in which designers imagine new systems rather than rearrange old ideas. The use of a design system is literally rearranging old ideas, and nothing more. We can call that good or bad, but it’s creative, and it’s highly constrained. If that “counts” as creativity, and I think it does, then a research study that looks at, measures, and equates newness to creativity is fundamentally missing a huge part of actual software development.

Some old-school designers just don’t like the Figma Moving Things Around reality, but I’ve seen junior designers absolutely love it, and it’s highly, highly valued in large organizations. It’s different than the Figma = Design problem, although related; it’s “design language systems are requirements,” as used here, and are actually liberating to Kids Today. It’s a weird smushing of requirements as framing a problem and constraints as framing a solution, where the design language system is both.

I’m benefiting a lot from these researchers, both from their content and their communication approach. I will keep looking for their work.