Distributed system code development needs effective build management tools - Embedded.com

Distributed system code development needs effective build management tools

With the advent of distributed computing in many mobile and embedded designs in consumer electronics, most applications to some degree make use of Java code and virtual machines to handle interactions amongst diverse computing nodes.

In this predominantly client/server-based environment, not only must software developers concern themselves with diverse code builds performed on a number of nodes, but with multiple design teams whose code output must be managed, reconciled and debugged.

The key to successful build management in such distributed environments is a foundational commitment to consistency, repeatability and portability. This is just as true for small, homogeneous development environments using in-house, scripted build systems, as it is for large, complex environments, where a new class of non-scripted, distributed build-management tools are now available.

In small- and medium-sized environments, using a properly implemented in-house build system can mitigate many of the risks and challenges associated with builds. Additionally, with a few simple steps, the burdensome task of developing and maintaining build scripts can be significantly reduced, using build tools such as Make and Ant.

Build Management Evolution

Application builds have traditionally been managed using a rules-based program derived from Make, the world's oldest, best-known build tool. Make controls the generation of executables and other non-source files from a program's source files. There are many Make versions, each with a unique file syntax that precludes portability between development tools, operating systems, or even compilers within the same operating environment.

But Java development requires a new, platform-independent build tool, which led to the creation of Apache Software Foundation's Java-based Ant, so-named by developer James Duncan Davidson because, “it is a little thing that can build big things.” Ant eliminates Make's platform-dependent wrinkles, and is extended not with shell-based commands but Java classes. Configuration files are XML-based, calling out a target tree where various tasks get executed. Each task is run by a Java class that implements a particular task interface.

Ant is powerful, but the XML configuration scripts can create limitations. XML does not handle conditional logic effectively and it is, therefore, difficult to use Ant to write “intelligent” build scripts that support robust batch processing.

Additionally, many development projects include Java and non-Java components that require both Ant and Make, as neither handles both languages. Scripting for the two is very different. Make scripts are the input to Make programs and dictate how to build each software component.

A Make file tells the Make program which binaries are to be created from which source modules. Make rules are then “fired” based on out-of-date conditions between source and object. In contrast, Ant/XML scripting uses serial batch processing. Rules for creating Java binaries such as .jar, .war, and .ear are handled statically for each step or “task” in the XML script. Figure 1 below shows the differences between Make and Ant scripts for a similar type of build task.


FIGURE 1

GENERIC MAKE BUILD SCRIPT
# =================================
# Builds the application executable
# =================================
application: application.c lib_a.o lib_b.o lib_c.o
@cc g -qcpluscmt -qidirfirst -I. -I/sys_apps/ref_dir/release/include- I/usr/include -o./exe/application $? –
bE:/sys_apps/ref_dir/release/include/application.imp

lib_a.o: lib_a.c
@cc g -qcpluscmt -qidirfirst -I. -I/sys_apps/ref_dir/release/include -I/usr/include “o
lib_a.o -c $?

lib_b.o: lib_b.c
@cc g -qcpluscmt -qidirfirst -I. -I/sys_apps/ref_dir/release/include -I/usr/include “o lib_b.o -c $?

lib_c.o: lib_c.c
@cc g -qcpluscmt -qidirfirst -I. -I/sys_apps/ref_dir/release/include -I/usr/include -o lib_c.o -c $?

GENERIC ANT SCRIPT
====================================================>
Compiles source code and packages application.jar>
=====================================================>
target name=”compile” depends=”prepare”>
javac srcdir=”./src”
includes=**/*.java”
destdir=”./build”
debug=”off”
deprecation=”off”
optimize=”on”
/target>
target name=”application_jar” depends=”compile”>
jar jarfile=”./build/application.jar”
basedir=”./build/classes”
compress=”false”
includes=”com/**”/>
/target>


For either approach, the programmer must understand not only how the application is constructed, but also the specific syntax requirements of the build scripting language they are using. Additionally, Make and Ant/XML scripts are not re-usable because static application information is coded into the script.

Make and Ant/XML Challenges

As Client/Server and Java development has evolved, so has build complexity, especially with Make. When Make is used recursively (one Make file per final target executable, or binary ” the most common method of managing large build processes), an application with 50 binaries would require 50 Make files plus a “driver” Make file.

The system build is completed by calling the Make program repeatedly and passing a different Make file each time. Dependency checking between the individual Make files is impossible, which means large application Make files can't be managed by a single Make file. Developers get around Make challenges through clever file-ordering to track dependencies, along with object-borrowing and multi-system parallel building techniques to reduce the associated long system build times.

Although most Ant build systems do not appear to be as complex as many Make-based build systems, it is only a matter of time. As Ant scripts suffer from being passed through different developer's hands, as new technology emerges that effects the way Ant scripts are coded or used, and as applications grow more complex, Ant will encounter many of the problems associated with a Make-based system.

The key to avoiding this is to implement best-practices for manual scripting starting with an in-house build system, while monitoring factors that would signal the need to move to an automated, non-scripting approach.

Solving Typical Scripting Problems

Scripting challenges are easier to solve in small, homogeneous development environments confined to a single language and target operating system. The first step is to shift from a developer-centric view of builds to a team-centric view, and from the notion of scripting “my build solution” to scripting “our build solution.”

Build inconsistency is the toughest problem. If developers use their own build scripts in the language or tool of choice, it can be difficult to know whether problems result from bad code or a bad build. Build administrators must standardize on a single scripting approach that best suits the language being used.

The first step to reducing build inconsistency between individual developer's build scripts is to develop build script templates that can be used as a basis for all build scripts. All builds require the same basic information: source code, compiler, and final target. Individual developers can populate the build script templates with their own build specifics; i.e., the source code location, compiler location, and a final-target description. Build script templates should be well-commented and clearly organized to ease the process of populating the template with build specific information.

A major contributing factor to build inconsistency is a lack of compiler and third-party library standardization. In a disparate and distributed build environment, developers frequently build against different versions of compilers and third-party libraries. This makes it difficult to re-create builds and diagnose problems.

To promote standardization, all compilers should be centralized on a network drive accessible by all developers, on a clean and “locked down” machine. The build script templates should specifically reference the standard compiler versions on the mapped network drives, ensuring that all builds occur against consistent compiler versions. All third-party libraries should similarly be consolidated on a shared network drive so the latest, approved versions are used.

Another commonly faced problem is lack of build ortability. Builds often work only on an individual developer's machine, which by default becomes the “production” build machine. This approach can cause severe problems when trying to track down bugs that are discovered once an application has been released to Production.

To solve this problem, development teams should standardize their directory structure. All developers should work on code in the same directory structure. If a versioning or CM tool is used, pull the directory structure from it; if not, enforce strong directory conventions for all developers.

Portability problems also can be mitigated by using global variables in the build script templates that identify the root location for all source code, compilers and common libraries. By setting environment variables such as SRC_HOME, COMPILER_HOME, and COMMON_HOME, the same build scripts should work on all machines. Using global variables in the build script templates also reduces the amount of template editing that is required by developers.

Finally, isolate the build scripts to just that: builds. Too often, “build” scripts include substantial pre- and post-build logic unrelated to the build. Pre- and post-build logic can be extremely complex, especially as an application matures and development is being performed on multiple versions simultaneously. The Ant script in Figure 2 below demonstrates a build script with a very basic and generic deployment portion.


FIGURE 2

==================================================
Setup properties. Values will be edited by developers
in order to make scripts run on their on workstations.
==================================================
property name=”src.dir” value=”./src”/>
property name=”images.dir” value=”./images”/>
property name=”doc.dir” value=”./doc”/>
property name=”build.dir” value=”./build”/>
property name=”classes.dir” value=”${build.dir}/classes”/>
property name=”deploy.dir” value=”./deploy”/>
property name=”deploy.doc.dir” value=”${deploy.dir}/doc”/>
property name=”deploy.images.dir” value=”${deploy.dir}/images”/>
property name=”deploy.jars.dir” value=”${deploy.dir}/jars”/>
property name=”debug” value=”off”/>
property name=”optimize” value=”on”/>
property name=”deprecation” value=”off”/>
property name=”appjar.name” value=”application.jar”/>
property name=”commjar.name” value=”common.jar”/>

==================================================
Compiles source code
==================================================
target name=”compile” depends=”prepare”>
javac srcdir=”${src.dir}”
includes=**/*.java”
destdir=”${classes.dir}”
debug=”${debug}”
deprecation=”${deprecation}”
optimize=”${optimize}”
/target>

=============================================
Packages application.jar
=============================================
target name=”common_jar” depends=”compile”>
jar jarfile=”${build.dir}/${appjar.name}”
basedir=”${classes.dir}”
compress=”false”
includes=”com/company/application/**”/>
/target>

==================================================
Packages common.jar
==================================================
target name=”application_jar” depends=”common_jar”>
jar jarfile=”${build.dir}/${commjar.name}”
basedir=”${classes.dir}”
compress=”false”
includes=”com/company/common/**”/>
/target>

==================================================
Prepares the objects for distribution.
==================================================
target name=”prep-deploy” depends=”application_jar”>
mkdir dir=”${deploy.dir}”/>
mkdir dir=”${deploy.doc.dir}”/>
mkdir dir=”${deploy.images.dir}”/>
mkdir dir=”${deploy.jars.dir}”/>
p> copy todir=”${deploy.doc.dir}” >
fileset
dir=”${doc.dir}”
includes=”**/*.html”
excludes=”**/dev/*,**/qa/*”/>
/copy>

copy todir=”${deploy.images.dir}” >
fileset
dir=”${images.dir}”
includes=”**/*.jpg,**/*.gif”
excludes=”**/dev/*,**/qa/*”/>
/copy>

copy todir=”${deploy.jars.dir” >
fileset
dir=”${build.dir}”
includes=”*.jar”/>
/copy>

zip zipfile=”deployable.zip”
basedir=”${deploy.dir}”/>

tar tarfile=”deployable.tar”
basedir=”${deploy.dir}”
/target>


Rather than writing pre- and post-build logic within a build script (where the functionality is often limited by the scripting language or tool), place the non-build logic in external scripts. The external scripts should be written in a scalable, lightweight, and cross-platform language such as PERL or PYTHON. Tightly focused build scripts can then have built-in hooks to the external build utility. Figure 3 below takes the overly complex build script of the prior example and replaces it with a call to an external script.


FIGURE 3

====================================================
Setup the properties. Values edited by developers
in order to make the scripts run on their workstations.
====================================================
property name=”src.dir” value=”./src”/>
property name=”buildutility.dir” value=”./scripts”/>
property name=”build.dir” value=”./build”/>
property name=”classes.dir” value=”${build.dir}/classes”/>
property name=”debug” value=”off”/>
property name=”optimize” value=”on”/>

property name=”deprecation” value=”off”/>
property name=”appjar.name” value=”application.jar”/>
property name=”commjar.name” value=”common.jar”/>

==================================================
Compiles source code
==================================================
target name=”compile” depends=”prepare”>
javac srcdir=”${src.dir}”
includes=**/*.java
“destdir=”${classes.dir}
“debug=”${debug}”
deprecation=”${deprecation}”
optimize=”${optimize}”
/target>

==================================================
Packages application.jar
==================================================
target name=”common_jar” depends=”compile”>
jar jarfile=”${build.dir}/${appjar.name}”
basedir=”${classes.dir}”
compress=”false”
includes=”com/company/application/**”/>
/target>

============================================
Packages common.jar
============================================
target name=”application_jar” depends=”common_jar”>
jar jarfile=”${build.dir}/${commjar.name}”
basedir=”${classes.dir}”
compress=”false”
includes=”com/company/common/**”/>/target>

=================================================
Executes the deployment script that handles preparing
deployable objects and actually deploys them.
==================================================
target name=”post-build” depends=”application_jar”>
exec executable=”post-build.pl” dir=”${ buildutility.dir }”>
arg line=”/package” />
arg line=”/deploy” />
/target>


By partitioning the build scripts in this way, developers (or build masters) who encounter build problems can drill down to the root cause very rapidly. Additionally, as development grows in complexity and new languages or target Operating Systems are added, the in-house build utility can scale more effectively.

For example, consider a C and C++ development shop that uses an entirely Make based build system with all pre- and post-build logic written in the Make scripts. When the development shop decides to add a Java component to their application, they are faced with writing an Ant component (equivalent to their existing Make scripts) that manages all of Java-related pre-build, build, and post-build logic.

However, if the development shop has a build utility, written in PERL, that executes Make scripts limited to build execution, they only have to write Ant scripts that handle the Java builds, and can use the existing PERL framework as a basis for all of the non-build functionality.

Dealing with multiple languages

Another common problem in complex distributed environments is build scripting inconsistency resulting from development in multiple languages. Build administrators can either force a single scripting language, or maintain different build scripts for different teams (Make, for example, works for C and C++, but is not particularly well suited for Java).

The best approach is to maintain different scripts with isolated build functionality, using a consistent, cross-platform, lightweight scripting language for all non-build functionality (e.g., retrieving code from a CM tool, moving files around, deploying binaries etc.).

Separating build functionality from all non-build functionality limits variances. There is no reason to be using Make or Ant scripts to copy files around or make logical decision during batch processing.

A common problem in such complex environments is the lack of an effective audit trail. Log all build script templates and “non-build” script components, and make sure audit trails track source code to executable. For each action that touches source code (check-out, move, compile etc.), embed a logging message into the script templates. This is facilitated by adding a basic Bill Of Materials report to the in-house build solution, including:

1. Name of the final target being built
2. Build machine environment information
3. Compiler version information
4. Version information from the CM tool for every dependency included in the build

Identifying Breaking Points

There are a number of critical “breaking points” that cause in-house build systems to become cost- and/or resource-prohibitive. When they occur, development teams generally begin to consider an automated, non-scripting environment.

One of the first breaking points occurs when the amount of time it takes for an application to build begins to limit unit- and integration-testing effectiveness. Only the items that need to be built should be built, in a true incremental approach.

Another breaking point is excessive problem-resolution turnaround, because the development environment scales beyond the capabilities of the in-house scripted manual build system. Developers find themselves spending most of their time tracking down what source code and common libraries went into a built object rather than resolving coding problems.

A sure sign that developers are reaching the limits of manual scripting efficiency is when they find themselves consistently spending as much as an hour a day working on build problems (either their own, or debugging build problems of a centralized CM team). Some companies actually assign a dedicated CM team whose sole responsibility it is to execute builds.

Developers find themselves waiting for the CM team to build their applications before they can move on to the next development effort. It can reach the point where the centralized CM team simply cannot keep up with the demand, especially when builds are cross-language, cross-platform and incredibly complex.

Migrating to Automated Build-Management

To solve the problems described above, teams within medium- to large-sized development environments are now turning to tools based on a true Client/Server architecture with a central build knowledge base.

Introduced over the past five years, this new class of build tool provides a standardized method for creating and managing Build Control files that replace Make and Ant/XML manual scripting. This approach eliminates the portability issues of rule-based programs derived from Make, while resolving the standardization challenges associated with scripted build processes based on Ant/XML.

One example of this approach is Openmake, a build management tool that weaves together human and machine intelligence to automate and standardize the enterprise build process.

It incorporates a browser-based user interface and a Tomcat or WebSphere Application Server to provide access to an Openmake Knowledge Base Server. Enterprise-based features allow for the connection to multiple remote build servers. Simple Object Access Protocol (SOAP) is used as the communication layer between the browser and the application servers.

Developers interface with Openmake through a web client, a command line interface, or indirectly through IDE plug-ins. Build meta-data is stored and managed via the central Knowledge Base Server and reused by multiple developers to generate Ant/XML scripts for Java support, or to generate “Make”-like scripts for traditional build requirements. Build Control files can be generated to build a single object (supporting developer daily compile activities) or a complete application (containing hundreds of inter-dependent modules).

When a complete application Build Control file is generated, it eliminates the problem of recursive Make and ensures the accuracy of incremental builds. Builds can be managed from an empty build directory pulling source code from a pre-defined search path, or by retrieving source code from a version management tool. Openmake also allows control over environment variable settings such as LIB, INCLUDE and CLASSPATH so that, regardless of the build machine, the build results are the same.

Openmake does not replace Ant for completing Java builds, but rather extends the use of Jakarta Ant without the need for manually coding XML scripts. In the place of hard-coded Make and Ant/XML scripts, for instance, its rules engine takes advantage of a knowledge base of build meta-data, such as Target Name and Dependency information, to dynamically generate portable, PERL-based build processes at build time that can be referenced by multiple development teams.

Steve Taylor is chief technology officer and Matt Gabor is senior developer atCatalyst Systems Corp.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.