Modern unit testing in C with TDD and Ceedling

Back when I first tried to unit test some code, I didn't really get it.

A colleague of mine was really excited about unit testing, and suggested that I should try it on my project. So I wrote some code and picked out a shiny, new unit test framework.

I did manage to write a few tests, but it was painful. At the time, writing the code itself was challenging enough let alone figuring out how to write tests for it. In addition, each test needed extra boilerplate code to get it running, and I had an awkward setup with #ifdefs to control the running of the tests.

And I wasn't particularly convinced these tests were worth the effort.

Since that time though, I've come to value unit testing as critical to the way I develop embedded software.

What changed my mind? Well looking back, I've since discovered a few things that were missing from my original approach. I'll get to these in bit, but first let's make sure we're on the same page.

What's a unit test?
A unit test is just some code that calls some other code, used to test that it behaves as you expect:

void this_is_a_unit_test(void) {

      int next = get_next_fibonacci(5);

      ASSERT_EQUAL(next,8);

}

In this example we're testing the get_next_fibonacci function. We call the function with an input of 5 and we expect to get an 8 back.

The get_next_fibonacci function is likely part of a module of code with other Fibonacci-related functions. In this case the Fibonacci module is the unit under test.

This test verifies that our get_next_fibonacci function does what we want, and it doesn't require us to run the entire application to do so. We just execute this unit test function and get the results.

The unit tests can be run during development of the Fibonacci module to make sure we did it right, and then at any later time (like when we make changes to the code) to make sure we haven't broken anything.

What's a unit test framework?
A unit test framework is just some code that makes it easier to run and record the results of unit tests.

What was that ASSERT_EQUAL(next, 8) in the example above? It's a macro I just invented for comparing two integer values – but it's the sort of thing you'd expect to have with a unit test framework. If next == 8 the test will pass and if not the test will fail .

There is also usually a way to run each test like: 

void main(void) {

      RUN_TEST(this_is_a_unit_test);

}

When the test runs, this will print the results of test. If a test fails, we'll typically get more specific information about the failure like a line number.

There are usually a few other features (like fixtures or suites with set up and tear down code) but this is pretty much all there is to a unit test framework.

There are many, many unit test frameworks available for C. In fact, it's actually easy to write a simple one for yourself. A few popular ones are Unity, CppUTest, and GoogleTest.

But this is where I started my unit testing adventure. I knew I wanted to unit test so I needed a unit test framework, right? Well it turns out that a unit test framework makes it a little easier to write unit tests… but that's not the end of the story.

Testing real code
Most real code is not like the over-simplified examples found in articles on the Internet. It's certainly not like that earlier Fibonacci example. Real code calls other functions, and real embedded code interacts with the hardware.

For (a slightly less simplified) example, imagine a function that updates an LED display to show our Fibonacci numbers: 

void update_display(int previous) {

      int next = get_next_fibonacci(previous);

      led_display_set_number(next);

}

Armed only with my unit test framework, I didn't know how to test this. In this case, our update_display function in the Display module depends on both the LED module and the Fibonacci module:


Figure 1. A module with dependencies. (Source: Author)  

This function takes no arguments and returns no value, so we can't test it like get_next_fibonacci. How do we know when this is working correctly? It's working when led_display_set_number is called with whatever value is returned from get_next_fibonacci .

How do you test these sorts of interactions with other functions and modules? This was my first discovery: mocks .

Continue to page 2 >>

Discovery #1: Mocks
Unit tests should test each of your units in isolation. This keeps the tests simpler because there is less behavior to test.

However, like in the previous example, some modules have dependencies on other modules. Mocks are what allow you to unit test in these situations.

Mocks are stand-ins for your dependencies. They take the place of your real functions, let you simulate different return values and verify that particular functions are called.

To test the update_display example, you could use a mock to simulate a particular return value from get_next_fibonacci and then verify that led_display_set_number is called with that value:

void unit_test_with_a_mock(void){

      //Expect our function to get called with a 5, and return 8.

      get_next_fibonacci_ExpectAndReturn(5,8);

      //Expect this function to get called with an 8.

      led_display_set_number_Expect(8);

      //Call the function under test.

      update_display(5);

}

This is how the test would look using CMock, a popular mocking framework for C. Both the LED and Fibonacci modules have been mocked: 


Figure 2. Mocking dependencies to make unit testing possible. (Source: Author)  

Using mocks can be challenging in C, because it has to be done at build time. This is done by generating mock versions of the dependencies, and compiling and linking them in place of the real modules. Jordan Schaenzle describes this technique well in the mock object approach to test-driven development.

CMock will actually generate these “mock objects” for you from your C header files. This is where the get_next_fibonacci_ExpectAndReturn and led_display_set_number_Expect functions come from.

Although CMock is the first framework I experimented with, there are a few others like the Fake Function Framework and GoogleMock (now part of GoogleTest).

So now with my mocking framework, I could finally test some real code with dependencies. Even with mocks though, I still managed to write code that was hard or even impossible (at least I couldn't figure it out) to test.

What did I do about this untestable code? I've since found that the best strategy is to prevent it from ever getting written in the first place. The way I do this is with test-driven development (TDD) .

Discovery #2: Test-driven development
When test-driving , I write the unit tests as I write my code. First I write a test that fails and then I write the code to make the test pass. Then I might refactor the code to clean things up. I repeat this cycle over and over.


Figure 3. The test-driven development cycle. (Source: Author)  

Because I'm writing the tests as I write the code, I inherently avoid problems that make my code more difficult to test.

In particular, the dependencies of the code under test are obvious. If the dependencies make the code too hard to test, I know right away and can fix it. I even find myself preferring designs that avoid dependencies altogether.

For example, in our earlier update_display example instead of calling the get_next_fibonacci function I might just pass in the next Fibonacci number instead:

void update_display(int next) {

      led_display_set_number(next);

}

This makes update_display easier to test, because there is one less mock to use. What I've really done though is create a more loosely-coupled design by completely removing the dependency of the display module on the Fibonacci module.

Another thing that's great about TDD is the test coverage (that's the amount of your code that's actually tested by the unit tests).

Recently I worked on project where some code was test-driven whileother code was not. When I ran a test coverage analysis, some modulesshowed significantly higher coverage. When I dug a little deeper I foundthat it was the test-driven modules which had the higher coverage.

This makes sense, because you only write code to pass a test you'vealready created. So almost every bit of code has a test before you evenwrite it.

But do these tests really find problems? Yes they certainly do forme, and they do it all the time. Sometimes I catch just simple mistakes,but a failing test makes it obvious. Other times I know I'm working on ahard problem, and the tests let me experiment quickly until I come upwith the right solution.

Each time I expect a test to pass – but it doesn't – I know I've justsaved myself from a bug. Each time a passing test breaks with a newchange I know I've found another problem I can fix right away.

All this helps me answer one of my original questions: are these tests worth the effort? The answer is an unequivocal yes.

Writing the tests up front encourages me to pursue a more modulardesign. Having the tests exercise so much of the code gives meconfidence that my code is going to work reliably. Finding and fixingproblems while I'm writing the code (before I've even run the fullapplication on the target!) saves me the time of chasing down those bugslater.

Note that this is really just a short intro to the benefits of TDD. For a more persuasive case, check out Jack Ganssle's interview with James Grenning.

Doing TDD however means I'm writing lots of unit tests and runningthem often. It also means I need to create a bunch of mocks on-the-fly.To keep the TDD cycles moving smoothly I need all of this to be quickand easy. Using just a test or a mocking framework isn't going to cut itanymore.

What I really need is a build system that automates all ofthis for me. Sure I could build my own but who has the time for that?And, chances are it's still not going to be as good as my finaldiscovery: Ceedling .

Discovery #3: Ceedling
Ceedlingis a build system specifically designed for running unit tests in C. Itincludes a test framework (Unity) and a mocking framework (CMock).

Ceedling provides some killer features:

  • Automatic test discovery and run.
  • Automatic mock generation.

These are the unit testing features that really make creating and running tests easier.

During my original unit test experience (with just a test framework),each time I created a new test function I needed to manuallycreate a test runner. This is the code that actually calls the testfunction to execute the test:

void this_is_a_unit_test(){

}

void here_is_another_one(){

}

//Here's my test runner.

Void main(void){

      RUN_TEST(this_is_a_unit_test);

      RUN_TEST(here_is_another_one);

}

This might sound trivial, but the extra work of having to add a newcall for each new test is a pain. When I'm just trying to write thecode, I don't want to waste time messing with the test infrastructure. Ijust want to crank out tests as fast as I can.

Ceedling automatically finds the tests in your code and generates the test runners for them.

You don't need to write anymore boilerplate code to run your tests!It does this by using some conventions (which are all configurable).

By default, if you start a function name with test_ and put it in a source file with a name that starts with test_ , Ceedling will automatically find this test and run it.

The other issue where a build system really helps is this wholebusiness with mocking. As we discussed earlier, the way to do this in Cis by substituting mock modules at link time.

But different tests will use different mocks. This means that we needto create different binaries for different tests depending on whichreal modules and which mock modules are included. This is in addition toactually creating each mock. Whoa, this sounds like a lot work.

But wait, Ceedling does all of this!

For each test file, Ceedling creates a separate test binary thatlinks in the right combination of real and mock modules. And, each ofthose mock modules is automatically generated with CMock.

All this is managed by convention as well. When you create a testmodule Ceedling knows what to link into the test by what header filesyou #include . Whenyou include a plain-old header file, Ceedling knows to find and link inthe corresponding source file. But if you include a header file namethat starts with mock_ , Ceedling knows it's time to generate and link in a mock. For example, if you #include “mock_led.h” Ceedling will create a mock LED module from the led.h header file. Pretty slick!

Ceedling is reusable solution to the build problems that come up whentrying to unit test in C. It has saved me the time of creating my ownbuild system, and it saves me time with every test that I write. It's atool that removes the friction to TDDing an embedded project.


Figure 4. Ceedling is a combination of unit test and mocking frameworks into a build system. (Source: Author)  

I haven't come across another tool for C that works similarly. Morethan just a test or a mocking framework, Ceedling is the glue that putsthem together and makes them easier to use. It brings to C the unit testfeatures that you'd expect from higher-level languages and moreintegrated development environments.

Ceedling is built around Rake (a build automation tool used likeMake) so you operate it through Rake tasks. From the command line, you'drun rake test:all to execute all of the tests and get a report. To run just the tests for our Display module, you'd use rake test:display

It all works well once you get the hang of it. For help getting started, you might want to see my articles on test-driven development with Ceedling and mocking embedded hardware interfaces.

In summary, I've learned that there's more to unit testing than justpicking a unit test framework and trying to write some tests. Mocks helpme test code with dependencies. TDD is a change in mindset, one whichhelps me write code that's more testable. And finally Ceedling is thebuild system that makes it all a little bit easier.

5 thoughts on “Modern unit testing in C with TDD and Ceedling

  1. “Great article. I was inspired a few years ago by Grenning and recently drove a non-trivial embedded firmware project using TDD with CppUtest and my own custom mocks/build environment for the tests. It was a great experience. I certainly need to check out

    Log in to Reply
  2. “Thanks Matthew, glad you liked it. It's great hear about others doing TDD in the real (embedded) world. In my experience, you learn *so much* and get better at it by actually doing it. Much respect for creating your own custom mocks and build environment!

    Log in to Reply
  3. “Fantastic article, Matt. However, I am wondering how you remove the test code in order to burn the actual application code on target. Is this done manually?nnWhere I used to work, we used software called RTRT by IBM to perform unit testing on our embedd

    Log in to Reply
  4. “Thanks Sherrah, long comments are great!nnBy convention Ceedling tests go in C source files but they're not really “embedded in the code.” You put your tests in separate source files in separate test folder within your project. Ceedling knows to go in

    Log in to Reply
  5. “Thank you for this great article, Matt!nI am driving alone into a fairly big project and I have to figure how to test my software, this article inspired me to start with TDD.nI've previoulsly worked (for a short period) in a team where we used unit test

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.