Do-it-yourself embedded Linux development tools -

Do-it-yourself embedded Linux development tools

These key tools can be fine-tuned to fit your needs and your style of work.

Development tools are important. They save development and debug time. But most importantly, they make developers more happy and productive by automating many routine, boring, and time-consuming tasks. It's painful to see programmers spend a significant percentage of their valuable time on such routine tasks as downloading their code to the embedded target. This situation is not uncommon even with traditional embedded systems, but it's far worse with embedded Linux, where the lack of good development tools is evident.

The perception that there are no good embedded Linux tools is not entirely true. First, a few commercial tools are available that are worth considering. However, contrary to the traditional embedded systems development world, commercial tools aren't the only option. A lot can be done with relatively little effort using freely available open-source tools. In fact, I would argue that the do-it-yourself development environment can be far superior to any commercial offering, as it will allow you the level of customization, flexibility, and agility that none of the off-the-shelf products can match.

Commercial tools
Embedded Linux integrated development environment (IDE) software suites are usually available from the same companies that sell embedded Linux. Wind River, MontaVista, TimeSys, LynuxWorks, and a dozen other vendors come to mind. Although these companies will try to sell you both the operating system and the accompanying IDE, this IDE may or may not be tied to that particular distribution. Most of these tools are Eclipse-based and offer similar functionality, which at closer look turns out to be no more than a clumsy editor, compiler wrapper, and debugger. The only exception I'm aware of is Wind River Workbench, which is actually a commercial grade product–not surprising considering Wind River's experience with its Tornado IDE for VxWorks.

The major problem with off-the-shelf IDE suites isn't the software itself, but rather the nature of embedded systems development. To unleash the full power of an IDE, you must run an agent on the target embedded platform, as illustrated in Figure 1 . This agent, however, may not be available if you're working on a customer system, or you may not have enough time to integrate it if you're doing a relatively short-term project. However, this agent typically fails to run because it relies on some kernel functionality that may not be available, as all embedded platforms are different and embedded systems programmers love to tweak system internals, often breaking some functionality the IDE agent relies on.

Do it yourself
The do-it-yourself approach has many advantages, such as the resulting tool is free, not tied to any particular Linux distribution, customized for your needs, and most importantly, it's modular. This lets you quickly port to a new platform only the functionality that you really need.

Now, let's take a look at the most common development tasks and see how they can be automated and simplified using open-source tools. We'll also try to bind it all together to form something similar to IDE, although it's not really necessary–every tool described here can be used by itself. Note that all network-related examples assume that all networking components reside on the same LAN with a subnet; is assumed to be host address and the target.

Host platform
Although everything I will describe can be also done on a Windows host, I recommend using Linux. It's more convenient, and more tools and utilities are available. And if you rely on a few Windows applications such as Word and Outlook, you can still run them on Linux in emulation using VirtualBox, Wine, or other commercial package. If you're new to Linux, using a Linux host will also force you to learn the new platform faster.

Writing code
Many developers have definite preferences with regard to programming editors, and there are many open-source and commercial Linux packages to choose from. Emacs is my favorite, as it has all the required features (and much more) and most probably will be installed on every Linux host.

Emacs is a powerful editor, but it requires a bit of configuration to unleash it's power. I recommend enabling at least the following options: global font-lock mode for syntax highlighting, imenu-add-menubar-index to show a list of C functions defined in the current file as a drop-down menu, and cscope (which is a separate package with Emacs integration support) tags for fast search and code browsing. You can either change individual options through Emacs' Options menu, or just cut and paste the example in Listing 1 into your .emacs file.

View the full-size image

I recommend running the compilation process inside Emacs. Having the following keyboard shortcut will make it more convenient:

(global-set-key 'f5 'compile)   

Emacs has many other features that a programmer would find handy, such as version control integration, auto-completion, matching parentheses, auto-indentation, C macro expansion, gdb debugger integration, and code folding (for further reading, I suggest “Emacs for programmers,” and “Using Cscope and SilentBob to analyze source code,” sid=07/03/05/1715201). And don't forget to print the Emacs reference card ( that came with your Emacs installation.

Running your app
Downloading and running your code on the embedded target is probably one of the most frequent operations you'll do, so it's important to make this process as fast and easy as possible. This usually done using TFTP or JTAG, but Network File System (NFS) can make this process easy and transparent. NFS can be used by running the NFS server on your host and having makefile rule to copy the compilation results to the NFS root directory and mounting this directory via NFS from the embedded target. This way, the compiled application will appear immediately at the target without any intervention by the programmer, as illustrated in Figure 2 .

When implementing the host configuration, make sure that the NFS server is installed, and add the following line to /etc/exports (after modifying IP and directory):

/home/user/nfsroot 192.168.0.*    (rw, no_root_squash, sync)   

then restart the NFS server. This gives NFS clients from subnet full read/write access to your nfsroot directory.

In the target configuration, make sure that the kernel is compiled with NFS support. In other words, CONFIG_NFS_FS, CONFIG_NFS_V3 and CONFIG_NFS_V4 options are enabled and the mount command supports NFS (if you are using busybox, check that the CONFIG_FEATURE_MOUNT _NFS option is enabled). You'll have to issue the following command on the target machine (probably as part of initialization script):

mount     nfsroot /mnt/nfs.   

If you can't use NFS, whether because you can't modify the kernel or because your short term project doesn't justify the effort to configure it, you still don't have to copy your application manually from host to target. This task can be automated using expect or minicom scripts, as shown below.

Many tasks can be automated using shell scripts. Note that you can run scripts on the host as well as on an embedded target (most embedded Linux distributions include shell with scripting functionality, although it's more limited). An introduction to bash scripting can be found at, but remember that you won't be able to use many of the advanced scripting options on your embedded target, as it'll likely have one of the less powerful (and less memory hungry) shells, such as BusyBox ash.

Assuming you're capable of writing a simple bash script, we'll dive into a bit more complex but useful topic, interactive scripts. Using interactive expect(1) scripts, you can automate such tasks as image download and flash programming. But you can't use simple bash scripts to automate interactive tasks because of branching and timing issues.

Basic expect(1) scripts consist of a “spawn” command that executes the utility that requires interactive automation, such as telnet or minicom and a series of “expect” and “send” commands, as illustrated by the example in Listing 2 .

View the full-size image

The expect command waits until one of the patterns matches the output of a spawned process. Send sends string to the current process. The script in Listing 2 downloads and copies a new bootloader image to flash. It's written for Das U-Boot, but can be easily modified for any other bootloader or other environment. Most expect(1) distributions include a handy autoexpect script that automatically remembers the commands you type and creates an expect script for you. However, I encourage you to use automatically generated scripts only as a template for writing your own, which will be more readable and easier to maintain.

Debugging embedded Linux is tricky, because the technique can be different depending on whether you are debugging applications, drivers, or kernel code. The only common element is the gdb client front end, so we'll start there. A gdb client will usually run on the host platform, although technically you can run it on the target, too. It can connect to the gdb server running on the target (more on that later) through a serial port or tcp/udp protocol. Note that you can't use the x86 gdb client that comes with your Linux distribution. You'll need the one from the cross-compiler toolchain for your embedded CPU.

Simple gdb debugging session is illustrated as:

gdb>file vmlinuxtarget remote   

This example loads vmlinux image symbols, connects to a remote target, interrupts the running code, and prints backtrace (for more information about gdb commands, see the gdb user manual at Using a command-line interface is handy for quick tasks, but for serious debugging, most users prefer a graphical interface. I recommend two graphical gdb wrappers, DDD and Insight.

Insight has a slicker user interface, but because it incorporates gdb, you'll need a different binary for every embedded CPU architecture you work with. DDD is a bit more clumsy as it's a GUI wrapper, but it can work with an external gdb executable (using the -debugger parameter) allowing you to use a gdb binary from your cross compiler toolchain. The gdb server will vary depending on what you're debugging. For applications, you'll need to run a gdbserver executable on the target in the following way:

gdbserver     your_application   

gdbserver will run the binary your_application that you're going to debug and wait for gdb client to connect on the specified port. It can also attach to a running process if executed with -attach command-line argument.

Kernel debugging is trickier for various reasons, not the least of which is the fact the kgdb (kernel gdbserver equivalent) may not be integrated into your kernel, so you may need to download a kgdb patch from, apply the patch, and recompile the kernel. Doing this will allow you debug the kernel through a serial port or over Ethernet. When debugging kernel loadable modules using kgdb version 1.8 and earlier, you'll have to load a module object file into gdb manually using the script or add-symbol-file gdb command so that gdb will be aware of your module's the symbol table.

If you're creating a board support package or doing some low-level kernel programming, you'll probably need a JTAG probe. When choosing a probe, ensure that it supports Linux, although most probes nowadays do. A Linux-friendly JTAG probe should support the Linux target and host; support a remote gdb protocol for debugging; support the debugging of kernel code, applications, and dynamically loadable modules; and support a Linux MMU.

The latter one is tricky because during debugging, you may want to access a virtual memory page that's not currently mapped. The probe will have to either know how to extract this mapping information from Linux internal data structures or make Linux remap this virtual address. Linux host and remote gdb support is not essential, but it is convenient, as it lets you debug with JTAG using the same gdb client frontend.

Debugging lets you to catch simple bugs, but unfortunately the hard ones are usually affected by timing and probably won't show up in the debugger. If this is the case, you consider using the Linux Trace Toolkit (LLT), which can be downloaded from It allows tracing various event types for multiple processes and the kernel itself, and it can present this information graphically to help debug complex multiprocess systems. To use LTT, you'll have to patch the kernel and install LTT daemon and utilities. Note that LTT daemon requires read-write filesystem access to store the trace file. I don't recommend using the jffs2 filesystem to store such information in flash as this could severely effect system performance and timing. A RAM disk is the best option, provided you have enough memory. If this isn't the case I suggest using NFS.

The most simple LTT session looks like:

trace 60 trace_filetraceview trace_file   

Both commands are a helper scripts. The first enables tracing for 60 seconds and uses trace_file as a base name for the trace results file. The second executes the graphical trace visualization tool (for more information, read the LTT reference manual at dox/ltt-online-help/index.html).

Eventually all embedded systems projects reach the optimization phase where a good profiler can come in handy. OProfile is a systemwide profiler for Linux that's capable of profiling user space applications, kernel code, and loadable modules. The OProfile software package consists of kernel code, userspace daemon, and utilities. Fortunately, no kernel patching is required as OProfile is part of the stock Linux 2.6 kernel.

A typical OProfile session looks like:

opcontrol --vmlinux     =/path/to/vmlinuxopcontrol --start#execute your codeopreport -l /path/to/mybinaryopcontrol --reset   

The first two commands start the profiler, opreport prints the results, and the last command resets the data. OProfile is a powerful profiler with many features, all of which are outside the scope of this article, but I encourage you to read more at dex.html .

An important tip for embedded developers is that they can filter profiling results by modules, which can be a driver, an application, or the kernel itself. Note that OProfile is a statistical profiler, which means it samples program-counter values on interrupt (usually timer interrupt), so you must run the code for some minimal time period to get meaningful results.

If your hardware is like most, and has performance counters that can generate an interrupt on such events as cache miss, you can find not only the most CPU-intensive pieces of code, but also the most cache inefficient. Unless you use an exotic CPU, OProfile will already have support for performance counters of that CPU.

Binding it all together
Although it's not necessary and can even make things more complex, if you're more comfortable with a monolithic IDE-like software package that can do all your daily tasks using keyboard shortcuts and menus, the simplest way to do this is to create a new Emacs menu called “Embedded” and add all your scripts to that menu. The example in Listing 3 (Emacs Lisp code that should be added to your .emacs file) illustrates this.

View the full-size image

Full explanation of this code requires some Lisp knowledge (and beyond the scope of this article), but a brief description of each command will be given, so you can customize the code and add new functionality without a full understanding of Lisp and Emacs internals. The first set of commands creates the “Embedded” drop-down menu. The second one adds a new entry “Program flash” to the menu that executes the “program-flash” Lisp function. The third set binds that command to “Ctrl-c” followed by a “pf” key combination. The final one is a Lisp function that executes a “” shell script redirecting its output to a newly created Emacs buffer.

Creating an embedded Linux development environment by yourself may look like a serious task. But keep in mind that this is a good investment. This task can be easily outsourced to an embedded Linux consultant, in which case it'll probably cost less than an off-the-shelf embedded Linux IDE.

Note that in the Linux world, there many different ways to accomplish the same task. This article shows just one of those alternatives.

Alexander Sirotkin works for Metalink Broadband as a software architect. For more than 10 years, he's been dealing with software, operating systems, and networking, and holds M.Sc. and B.Sc. degrees in applied statistics, computer science and physics from Tel-Aviv University. Sirotkin can be reached at











Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.