Paper summary - Understanding the Software Development Industry’s Perspective on Coding Boot Camps versus Traditional 4-year Colleges, by Quinn Burke, et al

What I read

In this paper, the authors describe a set of interviews and focus groups intended to better understand hiring attitudes about developers who emerge from bootcamp programs, and those that emerge from undergraduate programs. The results indicate differences, but make no strong conclusive statements.

First, the authors describe the need for more software developers in the United States. Citing both the Bureau of Labor Statistics and Code.org, they indicate that the US will have a gap of hundreds of thousands of people who can code. Short-form bootcamps have emerged, and can be used to fill this gap. The authors describe that most popular articles about the bootcamps describe the business models, rather than the broad learning efficacy of the program (focusing on extreme examples of success or failure.) They indicate that their research was focused on identifying “hard” and “soft” skills for developers, informed by interviews and focus groups with hiring managers.

The authors describe generally accepted skills for developers, noting that in industry, there is a distinction made between “hard” skills—the ability to code, or solve coding problems—and “soft” skills—the ability to work well with others, and be a team player. Another way to separate skills is through “routine” and “adaptive” skills, which correspond to the rote usage of a method or process, as compared to leveraging knowledge in new ways in order to solve problems.

Next, the authors describe the research methodology.

The overall research questions were:

What kind of learners are attracted to different learning opportunities, and why?

What kinds of software development learning opportunities are offered by undergrad programs, and by bootcamps?

How well do the settings align with regional software development industry’s needs?

Fifteen hiring managers from more traditional companies engaged in three focus groups and four interviews. Questions focused on recruitment, retention, expectations of skills, and expectations of educational institutions. Eleven representatives from boot camps, and nine professors, engaged in interviews. These interviews focused on implementation of curriculum, recruitment and admissions, student profiles, and skills taught. Data was coded and themed.

The authors then describe the findings of their research.

Industry representatives uniformly value soft skills. These include the ability to continue learning, teamwork and communication, and things like empathy, creativity, maturity, and being career driven. They also nearly uniformly value routine skills.

The research probed on the requirements for hire; there was no strong grouping of requirements. Some looked for four-year degrees, both in and out of computer science majors. Some are looking for theoretical knowledge. Some don’t consider bootcamp hires. However, some do consider bootcamp hires, particularly for more experience with hands-on projects, more efficient problem-solving skills, and familiarity with more recent development languages and approaches. This is compared to computer science graduates, who may lack the more practical, hands-on coding abilities.

The research with faculty members showed that hard skills are a formal part of their educational offerings, while soft skills showed up in a variety of ways (or didn’t show up at all.) Additionally, educational programs differed in their interest in “training” as compared to teaching. However, all of the bootcamp participants characterized their programs as training, with a focus on experiential learning. Additionally, the bootcamp programs went out of their way to provide vocational preparation.

The authors then discuss their findings. First, they conclude that bootcamps are not a threat to computer science programs. Some companies will hire bootcamp graduates. Hard skills are not sufficient for finding a job. Presence of bootcamps may “impact future graduate school admissions.” They describe that their findings do not indicate how well hard and soft skills are taught in either program. They indicate that their future work will attempt to answer the questions related to technical skills gained, soft skills gained, and preparation for a job.

What I learned and what I think

Ouch, this is rough; it’s another one of these papers that makes me wonder how it got accepted to be published. It’s in SIGCSE, which is peer-reviewed… but the method and findings are so thin.

Conducting a focus group seems crazy here, particularly around a topic where participants will have strong feelings. I would throw out that data entirely, given the group think, and the likelihood that participants didn’t get to participate equally. The inclusion of professors seems valuable and valid. The “representatives” from bootcamps, with lack of a better description (later in the paper, they are described as “instructors and administrators”) makes me think that it’s very likely that many are in marketing or public relations, not the actual instructors. I don’t understand why, if this is a foundational or initial research effort by the group, they didn’t just focus on one set of participants instead of mixing three sets of stakeholders.

The paper does use narrative-based descriptions of some of their findings, which I appreciate, but I question the value of the narratives themselves.

The definition of soft skills includes creativity, which is objectively wrong, although I’m not sure how much that matters.

The research questions the authors state aren’t addressed in the findings. I don’t know the types of learners that are attracted to bootcamps vs 4-year programs (there is a brief mention of age, but not related to this goal.) Software development learning opportunities are touched on only vaguely, in a reference to a prioritization on languages in bootcamps. And, there’s no conversation of the third goal, related to regional software development needs. I would have assumed that the discussion would tie the research back to these, even if there was nothing to report. But, they are entirely ignored.

If I get over the method critique above (particularly my assumption on the bootcamp “representatives” representing marketing), this is what I gather as a contribution of knowledge;

Some bootcamp programs focus on languages, while some CS programs don’t

Some bootcamp programs offer vocational support; nothing is said about CS programs

Some bootcamp graduates are hired based on their ability to write code; some CS graduates are not hired because their problem-solving abilities have been prioritized over code development abilities

Is this really a sufficient contribution to be publishable? If not, why is it published? If so, why is the body of knowledge about this topic so incomplete? Maybe it is that poorly explored, which is a “shame on us” to the academic community for ignoring it.

This is NSF grant 1561705, which was for $215,267.00.

The grant abstract said it would yield (1) models of the attributes of effective software developers from the perspective of universities, coding boot campus [SIC], and software development companies; (2) a conceptual framework of how software development workforce needs align with training preparation, and how variations in the preparation of adaptive expertise can inform efforts to broaden participation; and (3) longitudinal case studies of students from underrepresented groups from the two types of training settings.

The outcome report said they spoke with 49 students, and that’s not included in this report; it appears to be from a parallel grant 1561717 (which seems to be for another $277,920.00?!), that led to the summary "Should Your College Computer Science Program Partner with a Coding Boot Camp", but that is represented only in a poster, and the poster is not available online.

Update; I found and read the article. Thoughts here.

I can’t believe I actually agree with DOGE on anything, but if this was the outcome of $492,000 of taxpayers money, that’s appalling. Even if it is within that ballpark – even if the study was 1/4 of that – I can see why academia is viewed with disdain by a lot of the US populous. If we did this study at Narrative, we would charge, at most, $100k and that would include design solutions, too.

Woof.