Is Parallel Programming In Your Future? - Embedded.com

Is Parallel Programming In Your Future?

“Sun is only partly right with its marketing slogan 'The Network Is The Computer,'” a software developer remarked at a recent conference. “What I think is more true is that the network is a dataflow processor, and that's what will fundamentally change the computer industry at all levels and in all segments.”

His comment got me thinking about what architectural model will be in the mainstream of net-centric computing in the future. It also got me thinking about issues related to sequential versus parallel programming and whether or not we can transition to the latter, if necessary, and if so, how. Will the shift be relatively seamless, through some extension of today's C/C++/Java languages, or will it require that we learn totally new programming languages and techniques?

In some segments of the computer industry, servers, routers and switches, for example, it is literally true that “the network is a dataflow processor.” In other segments, such as PCs and information appliances, the infrastructure and architectures will remain the same, but the way in which we use them and the software we use will change from processing information to mainly moving it in and out, from here to there.

While not fitting the specific definition of dataflow, we are making the change from using our personal computer as an isolated word or database processor with occasional connection to using it as a data mover. Now, on a PC, we spend a substantial part of our time sending e-mails (moving data out) and receiving e-mails and Web pages (moving data in). In the new breed of Internet-enabled information appliances, e-mail and Web access are all there is, except for an occasional game.

A data flow processor, according to one description I have read, generally consists of many processors operating as a single multiprocessor array, using a packet routing network to connect any processor to any other, and memory array units for holding large databases of information.

This definition is remarkably similar to how the new generation of embedded and highly deterministic network processors is deployed in many of the new switches and routers required by the 1-100 gigabit/s optical fiber networks. It is also similar to the server clusters and server area networks made possible by the gigabit/s plus data rates of switched fabric and point-to-point to interconnection schemes such as Infiniband and SCSI over IP (iSCSI). As the backbone of the Internet shifts to terabit/s bandwidths, it's likely that dataflow multiprocessors based on cooperating nodes on this high-bandwidth inter-network will become more common.

Computer system architects tell me that the emergence of the dataflow and other parallel computing architectures into the mainstream is apt to present problems to programmers trained on the concurrency models and languages used on the traditional Von Neumann-based CPUs.

Unlike the control flow environments of existing mainstream processors where the focus of execution is determined by instruction pointers that identify the operation to be performed next, the operations of a dataflow machine are determined by the availability of the data needed for these actions. Computations are expressed and executed in dataflow terms determined by the arrival of data.

The difficulty programmers will face has to do with the concurrency models used on most conventional processor architectures. Although the performance and architectural enhancements of embedded and desktop processors have advanced significantly over the past decade, the concurrency models used in most of their operating systems have not. The concurrency models were conceived when multitasking was expensive and programmers had to use threads sparingly. In virtually every OS I know about, threads are associated with one of two things: specific pieces of code, or with instances of large objects. The result is software with many threads that are interdependent and which spend a lot of time waiting on each other.

The problem for programmers is that the dataflow environments found in routers and switches using network processors are event driven, deterministic and real-time to a degree that would give most embedded developers nightmares, full of asynchronous and concurrent interactions.

However, the programming languages and techniques in common use now, and the code that results, are sequential. Developers writing in C and C++, for example, must take highly parallel operations and convert them all into a sequential series of statements: functions, call backs, control blocks, data structures, and linked lists. Not only is this process excruciatingly time consuming, the resulting code doesn't use computing resources efficiently, to the detriment of the overall throughput of the system.

The options open to developers in dataflow-based net-centric computing environment are to (1) accept inefficiencies and try to make up the difference in faster processors, memory and interconnect; or (2) switch to programming languages based on alternative concurrency models.

There are any number of specialized programming languages developed for specific dataflow and parallel computing architectures: the Value-oriented Algorithmic Language (VAL); the Streams and Iteration in a single Assignment Language (SISAL); and the Irvine Dataflow language (Id). A language that has received attention and seems to be architecture-independent is Linda, which allows the easy development of parallel code using a manager-worker model. Also, a number of data parallel extensions have been proposed for languages such as Fortran, Modula 2, and fortunately for all of us, C.

But all of these approaches require significant investment in developing the appropriate compiler technology. Two alternatives that seem likely possibilities and that would not require a difficult transition for developers are: (1) an approach called Single Program, Multiple Data (SPMD); and (2) a standard message-passing library called MPI developed in the early '90s. The first is a low-level approach based on message passing to generate parallel programs with processes that can be mapped to the CPUs of a single parallel computer, such as in an Infiniband-based server, or that maybe distributed across a network. The second is a library of more than 125 functions that can be called from C programs to execute parallel operations.

There may be other, much simpler alternatives I'm not familiar with. If so, I would like to hear about them.

Reader Feedback

To respond whether classical programmers will shift to parallel programming is first to know that human being is naturally capable of thinking and working following a parallel model. Let's look a company management. It is a parallel running!So, if manufacturors create attractive parallel tools which seduced programmers and believed in the programmer's capability to programm current multiprocessors like multi-DSP or multi-microcontrollers, I am sure it will be a success. And I believe that CSP-based language will be the easiest to focalize on the way to parallelize an algorithm and not to be forced to spend dummy time to learn a complicated language. Why not OCCAM ? It is simple, and directly turn to parallel model.Best Regards from France…

Nicolle
Manager
RE.C.A


I agree with the article that languages with good support for parallel architectures are limited. However after mentioning a number of interesting languages(many I haven't heard of before)the author selects ADA as good language for this sort of application. I am not convinced ada offers much more than such things a parallel C. Personally I can think of two languages with much better support. One current and one unfortunately probably not now widely supported. Firstly OCCAM which was developed intially for transputer, was a very pure parallel language, having very high level support for message passing, spawning new processes, etc. This could be achieved as easily as assigning new variables in C. Unfortunately I am not sure if there exists many present ports.

Secondly I would suggest Erlang (www.erlang.org). Again this has concurrency built in as part of the language. It has a number of additional advantages such as the ability to move processes dynamically across processors to facilitate load balancing or fail overs. In fact it was the language used to produce a 160 GByte switch.

Actually the problem is not so much the languages used(although a good choice is a help). But the tools given to debug the software. Debugging tasks running over a number of processors is far more difficult than debugging on one.

One thing that helped with Occam and Erlang was that we could write applications and simulate running them on one processor and then seamlessly move them across a network.

Tony Pedley
Snr SW Engineer
Alsthom Power Conversion


Having done parallel programming in graduate school, I feel I can offer some insight into your question. Parallism brings another level design for the sequential programmer. Say for example a sequential designer must parallize and existing sequential program. The sequential programmer must identify which parts of the are parallelizble, and which are not. The sequential programmer or someone must determine the correct message passing architecture to use between processors, to clarify you just can't pass messages from one computer to another, there must be links setup between processors in a chosen message passing architecture. As tools go, we used a C library package called PVM, (PARALLEL VIRTUAL MACHINE) for our message passing constructs. It was rather straight forward use, very similar to a com library. Once an architecture is defined, and the skill of identifying parallelizble parts of code is mastered, which is a manageable skill, I think sequential programmers will readily accept parallel programming.

Jeffrey Fann
Electrical Engineering
Kemet Electronics


Ada95 and its predecessor Ada83 both have constructs built into the language for multitasking programs. In fact I write my multitasking code on linux and then can move it to windows without changin any code and it works fine.

e.g:

with Ada.Text_Io;
use Ada.Text_Io;
procedure Main is

task A;
task body A is
begin
loop
Put_Line(“this is a”);
end loop;
end A;

task B;
task body B is
begin
loop
Put_Line(“this is b”);
end loop;
end B;

begin

loop
Put_Line(“this is main”);
end loop;
end Main;

Compile this on windows/linux and you will have 3 threads running. One will print 'this is a', the other will print 'this is b' and the last will print 'this is main' on stdoutput. In fact I have used Ada multitasking on windows programs with GUI and all the other happy stuff. Try Ada95. You won't regret it. There is absolutely no need to continue struggling with multitasking using C/C++. Even Java's support for multitasking is not very strong. Ada provides extremely powerful language constructs to do parallel computing.

There is an extremely powerful free Ada compiler available on the web called GNAT and is available on www.adapower.com. Also, all bindings for Ada to program in windows/linux are available there. There is also a very strong and dedicated Ada community that you can join and get information from.

Sid Ray
Software Engineer
Tucson Embedded Systems

Bernard Cole responds:

This is interesting to know and is something I will haveto look into further. I do know that a number of languageshave some degree of multitasking, but more importantly,parallelism built in, including Java. The big question I wasposing was not whether or not there was an appropriatelanguage or environment in which to do parallel programming,but whether programmers would be willing to make the shiftfrom the sequential programming model. If not, what willtool vendors have to do: perform the parallel programming inthe background, allowing the programmer to worksequentially, using compilers to parallelize the code? Orwill they have to create environments in which programmerscan be seduced into a more parallel frame of mind?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.