How containers improve the way we develop software - Embedded.com

How containers improve the way we develop software

To understand containers is to understand the future of software development. Here’s how to best leverage this software innovation.

Containers are known as an amazing tool for cloud development, allowing software to run reliably when moved from one computing environment to another. But there are many other ways in which they can improve our lives as developers. This article looks at how to use containers to their greatest advantage and how to work around problem areas when they arise. We’ll be following up this article with another that looks at containers specifically from the embedded developer’s perspective.

What are containers?

Containers are gaining rapid acceptance in the application development scene largely because they solve the problem of portability. Simply put, containers componentize whole systems, abstracting them from a physical platform, allowing them to move around from platform to platform, platform to cloud, or cloud to cloud. What many people don’t know, is that portability strength also allows them to do so much more.

To understand containers is to understand the future of software development. Here’s how to best leverage this software innovation.

Moving projects between environments

If you have multiple workstations (such as a desktop and laptop, or multiple laptops), containers make it trivial to move projects between environments. Instead of worrying about having all the right tools, files, or directories you need on both computers, it’s easy to put the whole mess in a container. Then just copy the container to any new environment where you need to work, and you have a setup that’s certain to be right every time.

I hear you saying, “I just keep all my tools in git.” That might work for smaller, self-contained tools, but git isn’t terribly binary friendly, and you aren’t likely to create a git repository at the root of your filesystem to catch files installed into /bin, /usr/lib, or /etc. Containers let you corral big or invasive tool installations, making containers a more widely applicable solution.

Avoiding dependency hell

Multiple projects that share components can be a hassle. If those projects are built with different versions of shared libraries, you’ll often find that you’re managing conflicting dependencies unless you can bring all projects to the same baseline set. That’s not always possible, especially if you’re working on software that’s outside of your organization’s control like open-source projects or third-party components.

However, if each project puts all its files – including any dependent libraries – into a container, these projects can live side-by-side without inflicting dependency hell on its developers. This can be particularly handy if your company works on software projects for multiple clients. Every client’s environment goes into a separate container, allowing clients to use whatever library, compiler, or tool chain they need without impacting any other environments the developer manages.

Experimenting with new software

Dying to try out that new tool update? Nobody wants to find out a new package you just installed overwrote files that your project depends on, like shared libraries with a later (and API incompatible) version. Containers allow you to download and try out new software without fear that you’re going to mess things up. With a properly containerized environment, it’s easy to bring your computer back to exactly where you were before you started trying out experimental code.

Using WINE to run Windows apps in Linux is a perfect case-in-point. A WINE installation is notorious for needing tons of assorted files. To get rid of WINE to go back to a known system state is very challenging, since you need to remove not just the app, but anything else it may have installed. Putting a WINE app into a container gives you the confidence that you can just delete the app and guarantee you’re back to normal, with no dependency hangover.

Adding new members to a project

Your project is scaling up, and you need to include other developers, but it takes them more than a day to be productive just because of the development environment setup. That’s why you need to put your development environment in a container. Better yet, if you keep all the team’s development containers in a central registry, you can make it simple to add new members to a project, even when they’re external or outsourced resources.

A containerized development environment also makes sense for small, isolated projects that don’t need to scale. What if the only person who builds your company’s device drivers suddenly goes to the hospital or takes parental leave? Containerized development environments can help others take over in the interim, and a well-organized corporate container registry can help your company maintain proper succession planning.

Maximizing resources through continuous integration (CI)

A lot of tools that software engineers use during active development are overkill for the build and test environments. Why does that matter? Trimming out the fat lets you spin up more test or build instances to cut cycle times, and lets you run them on less capable machines.

Containers are the perfect solution to maximizing your resources since containers can build upon other containers to create concentric rings of capability. Leaving tools out of the development stage that doesn’t need them ensures that every part of the development workflow is as efficient as it can be.


Figure 1: Using concentric containers for stages of development lets you layer on the
functionality you need without duplicating the container contents. (Source: KDAB)

Ferreting out dependencies

Have you ever created a project and found it wouldn’t build on another machine because you already had half of what you needed installed on your main development system? (Oh yeah, I forgot I needed to install Boost first…)

Containers can help you ferret out any dependencies that you need to build or run your software. Start with a blank container and add things piece by piece until you’re sure that everything you need to run the application is clearly identified without pulling in anything unnecessary. This also helps you document what pieces are needed since you need to know exactly what is going into the environment.

Containers for large embedded systems (and desktop too)

What specific ways can containers be used in fully featured embedded systems or in desktop application development?

Running graphical apps

Running command-line applications are a container’s strength. That said, it doesn’t seem possible to run GUI applications within a container. After all, containers encapsulate applications so that they are isolated from the hardware they’re running on, and the screen definitely qualifies as hardware. However, by sharing X11, VNC, or Wayland sockets between the container and the host, you can run an application within a container and still use the screen.

Microservice architecture

Although a microservice architecture is a recent trend in software organization, not all development teams are using it since it comes with some drawbacks around speed and complexity. However, one of its main benefits is scalability – something that’s an ideal match with containers. Placing microservices in containers allows you to spin up as many instances as you need to handle your load. This can be very handy in situations where your company has a product line with varying capabilities based on processor speed, cores, and/or RAM. One firmware load can scale up to the more capable system by spawning more microservices to process images, lay out page graphics, or handle socket connections faster.

Emulating a web hosting environment

Developing web sites is a natural fit for containers since you can easily move between local interim environments to the hosted version. Create a container that emulates the web hosting environment – with your own Apache, PHP, MySQL, and other server-side tools – and your browser won’t be able to tell the difference. Since the browser provides the platform integration needed to run client-side software, it doesn’t need any special access outside the container. Another benefit: isolating the web server environment for an embedded system into a container allows you to develop, build, and test within the convenience of a desktop system without requiring the embedded hardware.

Containers for packaging

There are hundreds of Linux distributions out there, and you can’t always find your favorite application for your preferred distribution. While installing programs from source is an option, it’s time consuming at best. At worst, it doesn’t always work if autoconf gets something wrong, if critical build files are missing, or if cross-dependencies cause the make to mysteriously fail.

Flatpak and Snap

Flatpak is a packaging system for Linux that places all the runtime requirements for an application into a container along with the application to help both Linux users and developers. By providing all dependencies along with the app, Flatpak supports executables that always run as intended on every Linux distribution.

 Like Flatpak, Snap provides a containerized application packaging environment that works with any Linux distribution, making it much simpler to deploy and use apps. Both Snap and Flatpak provide their own container environment through standard Linux APIs. That is, neither is built on top of Docker, the industry-leading container platform.


Figure 2: Flatpak architecture allows one runtime setup to be inserted into multiple apps. (Source: KDAB)

What are the biggest differences between Flatpak and Snap? Snap provides a centralized and controlled app store and better supports embedded development, while Flatpak supports distributed application repositories and better supports desktop apps.

Benefits of containerized packaging

Using Snap or Flatpak means you get to build, test, and debug the same image that users are using. These environments provide already created base images that let you start with basic configurations (such as toolkits like Qt, KDE, or GNOME, and languages like C++, Python, Electron, Ruby, or Rust). They also offer application repositories – in other words, a Linux application store that users can browse. This also provides a central point to release maintenance builds.

Containers inside apps

We’ve so far talked about containers as a developer tool and an app distribution mechanism. However, you can use container technology as part of your application implementation too.

Gaining extra security

Let’s say you want to add plug-in support to your application. If those plug-ins come from third-party developers, you might want to protect the application and your user by running  plug-ins within their own containers. As an example, the Nautilus file browser for GNOME isolates the thumbnail applications that it launches to generate thumbnail images. (Nautilus thumbnailers are run using bubblewrap, which uses Linux namespace technology to implement a container-like application sandbox.)

Consistent execution

What about launcher applications? If your primary application launches many different sub-applications, what happens if two different apps use different versions of the same library?

To ensure all apps run properly, you’ll need to tightly control the environment for each launched application by putting each app into a container. Game distribution systems might be an ideal candidate for this, like Valve’s Steam. (Valve is currently transitioning their Steam engine to use containers for all games.) In this case, the Steam engine itself runs normally on the host, but each game runs in a separate container to ensure it has the same environment it depended on when it was written. This lets games that are several years old still work properly, even when most of the other libraries have been updated several times in the interim. Games are also a good candidate for containerization since they primarily have screen and audio output and keyboard/mouse/joystick input – a few things that are centrally controlled.

Controlled environments

A browser-based laptop like chromebook works great for some people – families, students, and non-power-users – and can be a good way for a limited IT staff to manage many machines. However, many people might appreciate an environment that can run non-browser apps without their having to become an administrator or programmer to install or manage their system. An example is Endless OS. It lets you install apps that are based on Flatpak, and it automatically pulls the latest releases of each app, ensuring everyone has access to the best features and is using the most secure and least buggy code.

Problem areas for containers

Containers are great – but they’re not perfect. Let’s explore some problem areas in using containers that either require workarounds, special handling, or can be showstoppers.

Conversion difficulties with multi-process

Since Docker runs each process in one container, applications that use multiple processes may have problems talking to each other across container boundaries. This isn’t a problem for newer applications that can be built as microservices with controlled entry points. However, apps that use a wide array of inter-process communication mechanisms between processes can be difficult to convert to a containerized environment.

Working with IDEs

Most developers use an IDE to write code, but how do you use one to develop an application that’s intended to live in a container? If your IDE is something like Qt Creator, you must put it inside the container itself. There is an advantage to this, since you can then work on two (or more) applications using different versions of Qt. But the big disadvantage is the management effort and excessive size of the many Qt Creator instances installed into your application containers.


Figure 3: VSCode Remote Container architecture lets you use the full power of an IDE to develop a containerized application. (Source: KDAB)

If, on the other hand, you’re using VSCode, you’re in luck. VSCode is very well-integrated with containers and can manage them all automatically if you install the Remote Containers plug-in. To start, you create a small json file to describe how you want to build and start your development container. From there, VSCode builds and spawns the container as needed, automatically connecting the runtime to the IDE debugging services through the container. With VSCode, you don’t even realize you’re running the application within a container.

//devcontainer.json
{
  "name": "Node.js Sample",
  "dockerFile": "Dockerfile",
 
  "settings": {
    "terminal.integrated.shell.linux": "/bin/bash"
  },
 
  "extensions": ["dbaeumer.vscode-eslint"],
 
  "forwardPorts": [3000],
 
  "postCreateCommand": "yarn install",
 
  "remoteUser": "node"

Example 1: Sample devcontainer.json gives you some simple options to use VSCode with the Remote Containers plug-in.

Not a true cross-OS technology

You may have noticed that most of our examples refer to Linux. What about Windows or MacOSX? Both OSes support Docker (and other) containers. However, most of the container-based solutions are being actively developed on Linux. Partly this is because containers have a heritage that Linux shares with the cloud. It also could be because Microsoft and Apple provide their own application stores and control the development environment. As such, it becomes less valuable and more difficult to build effective containerized packaging options for them.

Regardless of the reason for a dearth of non-Linux container solutions, another problem is that containers aren’t a true cross-OS technology. Unlike a virtual machine, a container depends on the underlying OS, which means it can only run containers that were built for that OS. If you want to run Linux containers, you must run them on a Linux machine or a Linux virtual machine.

Integration complexities

Containers keep applications within their confines, which can be a problem if they need to access anything external to the container. We’ve already talked about how graphical applications can get around this through X11, VNC, or Wayland sockets. But what about anything else the application may need to access, such as sound, printing services, file selection dialogs, clipboards, PDF viewers, or the like?

Environments like Flatpak and Snap try to accommodate this limitation in several ways. For example, they expose D-Bus, which allows applications to send and receive messages to and from services that are running on the host machine. Both Snap and Flatpak further help application developers by incorporating xdg-desktop-portal, which uses these D-Bus messages between the container and host to provide various desktop services, such as choosing files, opening a URI, or printing.

However, any hardware resources or software services that the container or container environment doesn’t explicitly support will need to have a workaround that allows the containerized application to use it. Just like xdg-desktop-portal does, these services require an in-client service that can forward requests to an authorized service running on the host that accomplishes the task. Whether this is problematic or not depends on how much of these features your application requires and whether the container in question provides a means to implement them.

Conclusion

Containers are considered one of the truly defining trends of the last decade. Once you understand all that can be accomplished with them, it’s easy to see why.


Andrew Hayzen is a Qt developer who works with QML, Rust, and Linux systems at KDAB, a global software engineering services firm. He has experience with all aspects of developing QML applications and has a keen interest in improving developer workflows. Andrew holds a BSc in Software Engineering.

Related Contents:

For more Embedded, subscribe to Embedded’s weekly email newsletter.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.