Python NetWorkSpaces and Parallel Programs -

Python NetWorkSpaces and Parallel Programs


Parallel programming has the reputation of being an exotic field,pursued by experts using extremely large and expensive machines.Unfortunately, due in part to its history, parallel programminglanguages and tools still mostly focus on “big iron” and olderlanguages such as C and Fortran. Performance improvement viaparallelism should be of interest to anyone whose codes run too slowly.This requires a shift in focus.

Today, many new computers are multicore and most users have accessto multiple computers. Many developers work with newer dynamiclanguages like Python and R.

To meet the needs of these users, we'vedeveloped a Python-based coordination system called “NetWorkSpaces”(NWS) that is easy to learn, accessible via almost all developmentenvironments (including R, Java, octave, Python, Perl, and Ruby), anddeployable on ad hoc collections of spare CPUs.

But while itssimplicity makes it a good choice for pedagogical examples, it's not atoy system. We've used NWS to run parallel programs on hundreds ofprocessors, producing many CPU years of useful computation.


NetWorkSpaces ( wasdeveloped at Scientific Computing Associates and is available atSourceForge (

You must install both the server (on one machine) and a client (onall machines involved in the computation). The server is implementedusing Python and Twisted, which are required. Even though NWS isimplemented in Python, we have NetWorkSpace client APIs for a varietyof languages. While we describe the Python client here, the ideastransfer to other language clients.

NWS is based on the concept of a set of bindings, which inprogramming languages are sometimes known as “environments,””namespaces,” “workspaces,” and the like. Generally in programminglanguages, a binding maps a name to a value. Because this is a conceptfamiliar to programmers, it is a good foundation for building acoordination system.

A given language has rules about allowable names, allowable valuesfor a given name, and the context in which the binding is valid (thebinding's scope). The language also provides operations forestablishing a binding and for retrieving the value of a bound name.

Often these operations are implied by the lexical structure of code,asis the intended binding set. So, for example, in x = y the y implies a look-up of the value bound to y , while the x is the target of the assignment. Scoping rules determine which x and y are meant.

NWS provides a particular encapsulation of binding semantics. Usingthis encapsulation, we explicitly specify the look-up (fetch ),the association of the name x with the retrieved value (store ),and the intended binding set (indicated by the ws object).Thus, a simple assignment looks like this in NWS:'x', ws.fetch('y'))

So far, we've succeeded in making a fairly routine construct moreverbose. The key point is that the NWS encapsulation is amenable to anetwork-based implementation, which lets different processes exchangedata and synchronize via NWS bindings.In many languages, including Python, we could have used syntax similarto that of normal bindings:

ws.x = ws.y

However, the semantics of these NWS variables differ in important waysfrom that of normal variables. In our opinion, it's a mistake to createa false illusion of similarity when, in fact, there are importantdifferences.NWS is designed to be a coordination facility that is language neutral.The advantages this neutrality offers include:

  • NWS coordination patterns and idioms can be recycled from onelanguage environment to the next.
  • NWS can be used to coordinate heterogeneous ensembles of codewritten in different languages.

To facilitate interlanguage coordination, NWS variable names are ASCIIstrings and don't need to conform to the variable naming rules of anygiven language. The values can be any native type in the clientlanguage for which that language has a workable serialization.

Mostvalues in most of the languages mentioned can be automaticallyserialized (the serialization is done behind the scenes by NWS, and isnot of direct concern to programmers). For example, Python NWS canautomatically handle composite data structures:

>>> from nws.client import NetWorkSpace
>>> ws=NetWorkSpace('test')
>>> l=['a','b','c']
>>> t=(1,2,3)
>>> d={'list':l, 'tuple':t}
>>>'dict example', d)
>>> ws.fetch('dict example')
{'list': ['a', 'b', 'c'], 'tuple': (1, 2, 3)}

Finally, ASCII strings used as values are treated in a special way(they are not subject to the client language serialization protocol)that makes it possible for them to be exchanged across clientlanguages. In Example 1 below ,for instance, you can use NWS to move datafrom Python to R encoded as an ASCII string.

>>> from nws.client import NetWorkSpace
>>> ws=NetWorkSpace('tickets')
>>>'ticket', 'ticket string')

> library(nws)
> ws<-netWorkSpace('tickets')
> nwsFetch(ws, 'ticket')
[1] "ticket string"
(Clickhere toread more)

Robert and Nicholas are members ofthe Department of Computer Science and W.M. Keck FoundationBiotechnology Resource Laboratory at Yale University. Nicholas andStephen are researchers at Scientific Computing Associates.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.