In industry surveys, more than 80% of embedded systems software developers report using either C or C++ as their primary programming language. Yet as a group, they earned a failing grade on a multiple-choice evaluation of their firmware-related C knowledge.
In December of 2007, an Embedded C Quiz was launched online as part of a redesigned Netrino website. In the 25 months since, over 4,000 data points have been collected regarding the C language skills of embedded systems programmers. The aggregate quiz results, as well as those from specific countries and regions, tell an interesting tale.
In a nutshell, the world's embedded systems programmers merit only a failing grade of D- in the C language. A rather scary result, considering that embedded software inside the medical devices, industrial controls, anti-lock brakes, and cockpits they design risks human lives every day. In an earlier column I discussed converging trends that suggest this problem will only get worse.
The Embedded C Quiz consists of 10 questions. Each question has four multiple-choice answers, including an occasional “None of the above”-styled choice.1 The whole process takes just a few minutes, but no time limit is enforced. Each question is entirely independent of the others. The order of the questions is the same for all quiz takers.
The set of 10 questions was developed by a small team of firmware experts, including myself, to test knowledge of the C language features that are most specific to work in embedded software development. Examples of covered topics include proper use of C's 'const', 'volatile', and 'static' keywords; writing portable C code using ANSI/ISO-compatible compilers and preprocessors; and the use of bitwise operators.
For example, one question asks that the type of 'x' in the declaration “int * const x;” be properly chosen from a list of descriptions. Another asks that the one line of C code out of four options that will “reset the least significant bit of x” be identified. No individual code snippet is longer than one line. Most of the tested concepts and techniques are well explained in free technical articles on the same website.
The quiz may only be taken by a registered user of the website and each user may only take the quiz once.2 His or her final score, as well as their answers and the correct answer to each question, is presented only upon completion; these remain available for them to review again later. Individual answers and scores are kept in a private database and not displayed to anyone other than the registered user.
From time to time, we have offered incentives (such as a drawing for free books or training courses) to encourage participation. But these offers have only been made within the online community of embedded systems software developers and all prizes but a few MP3 players would be of interest only within that community. Still, I suppose, it is likely that some quiz takers have come from other software development communities who happened upon our website; that said, I have every reason to believe that the vast majority of quiz-takers are employed as firmware engineers.
All of that is to say that we've worked hard to make the results of our embedded C quiz as meaningful as possible. However, it should be understood that this was not a scientific survey, in which embedded systems programmers should have been randomly chosen by us not themselves. But with more than half a million website visitors offered a link to the quiz over more than two years and fewer than 1% actually taking the quiz, we hope that the law of large numbers has brought us a mostly representative sample.
Upon a review of the aggregate results, we noted that some quiz takers (about 3%) started the quiz but didn't answer all 10 questions. The following analysis of the results is based only upon the 4,073 completed quizzes.
Anyone with a web browser can take the quiz, and so it has been taken from all corners of the globe. But given that the quiz and the Netrino website are entirely in English, there is an inherent selection bias in favor of English takers. I don't know what sort of masochist would seek out an “embedded C quiz,” but if someone does that all three major search engines currently direct them to our quiz at http://www.netrino.com/Embedded-Systems/Embedded-C-Quiz.
Country data is self-reported at website registration, from a list of 46 countries (provided as a default set by our content management system). Four countries provided at least 100 participants each: the United States, India, Canada, and the United Kingdom. In addition, the quiz was completed by 80 people from the great land of Other. A set of nine additional countries provided at least 30 results each: Turkey, Germany, Brazil, China, Australia, Romania, Italy, France, and Sweden.
The three largest geographic groupings each contain over a thousand data points:
• United States: 1,465
• India: 1,361
• Rest of World: 1,247
The three most significant subgroups within the Rest of World grouping each contain over one hundred data points:
• Western Europe: 446
• Non-Indian Asia: 304
• Middle East: 101
In the next section I'll present the aggregate data in various ways. But I don't want to draw any conclusions about countries or regions with sparse representation. So we'll mostly be talking about the countries or regions so far named. If you are interested in the raw data for other countries, please e-mail me for the Excel file.
Worldwide, the mean score was 61.0%. The average embedded systems programmer incorrectly answered four of 10 multiple-choice questions about their primary programming language! A rather scary fact, given that C is the language of choice for most embedded projects and that C++ is even harder to master. And a grade of D-, academically speaking.
Programmers in the United States scored slightly above average. But they still earned a failing grade of 62.0%. Programmers in India scored below the worldwide average, at 59.3%. As a group, the Rest of World programmers scored 61.7%. Table 1 provides this data along with the scores for several other countries and regions.
Table 1. Embedded C Quiz Results by Country/Region
Worldwide, there were 238 perfect scores. Every country with at least 30 quiz takers had at least one perfect score. At the other end of the scale, 209 programmers scored below the expected outcome for guessing (25%). Sweden was the only country with at least 30 data points that managed to avoid such a score.
The full distribution of scores across all countries is shown in Figure 1. The underlying data shows that fewer than one in three of the world's embedded systems programmers “passed” with a score of 80% or above. Sadly, another one in four scored below 50% despite an expected minimum score (for guesswork) of 25%.
Figure 1. Distribution of Embedded C Quiz Scores (Entire World)
When I find time, I aim to analyze the same dataset along a different dimension–by question. Such an analysis should reveal the largest areas of programmer misunderstanding with respect to the C programming language–or a weakness in the wording of one or more questions. I'll be on the lookout for both.
It would be interesting also to develop a larger set of multiple-choice questions about embedded systems programming that could be used as a routine part of the interview process for prospective firmware engineers. A larger set of questions should test some of the basics of C as well as additional subjects specific to software development for real-time and safety-critical systems.
In September of 2009, we added a new Embedded C++ Quiz to our website. The early results look worse than those presented above for C. And the dropout rate is higher. But the main thing we need right now is a much larger sample size. So if you use C++ in your work, would you please take just a few minutes to test your skills at http://www.netrino.com/Embedded-Systems/Embedded-C++-Quiz.
And be sure to tell your colleagues about both of these quizzes. Additional online quizzes for embedded systems programmers are in the planning stages.
Michael Barr is the author of three books and over 50 articles about embedded systems design, as well as a former editor-in-chief of Embedded Systems Design magazine. Michael is also a popular speaker at the Embedded Systems Conference, a former adjunct professor at the University of Maryland, and the president of Netrino. He has assisted in the design and implementation of products ranging from safety-critical medical devices to satellite TV receivers. You can reach him via e-mail at or read more of what he has to say at his blog ().
1. On this basis, a random set of answers should be expected to score either 20% or 30%.
2. A person with two e-mail addresses could register twice and thereby take it twice. But the psychological incentives to do strike me as small.