Rapid prototyping of embedded and mobile device user interfaces - Embedded.com

Rapid prototyping of embedded and mobile device user interfaces

Editor's note: Qt Quick is a free software application framework developed and maintained by the Qt Project. It provides a declarative way of building custom, highly dynamic user interfaces with fluid transitions and effects, which are becoming more common especially in mobile devices. Qt Quick includes a declarative scripting language called QML.

The market requirements for embedded device user interfaces have dramatically changed over late years into touch-based, interactive, rich and fluid user experiences. At the same time, agility and reactivity of modern software processes have influenced embedded software development. Rapid UI prototyping is something that enables the incremental development of a UI parallel to the application back-end.

With Qt Quick (one of the UI approaches of the full Qt environment), rapid UI prototyping can be done without the actual embedded target hardware. This is because Qt runs cross-platform on desktop, embedded and mobile operating systems. With Qt Quick the UI development is done using a dynamically replaceable declarative UI file that allows rapid deployment cycles of the UI layer.

Qt is a software framework for creating cross-platform applications and user interfaces for C++. Originally Qt was created for desktop platforms, to have the same application code work across different desktop operating systems without modifications. Through the 20 years of Qt’s history it has also been widely adopted into the creation of embedded devices. Qt is being used as a software framework in over 70 industries by thousands of companies and hundreds of thousands of individual developers.

Where the Qt developer story on desktop platforms has focused on its cross-platform capabilities, on embedded devices Qt has been mainly used for creating fluid, scalable and high-performing native user interfaces. The cross-platform story certainly works on embedded platforms as well: The developer can do prototyping and development on his desktop host computer without the actual target hardware in place and then later deploy it to the device.

Qt is owned and maintained as a product by Digia which is driving core Qt activities including product development with the Qt Project under the open governance model. Qt is widely available for everyone either through a commercially licensed package or by an open source licensed (LGPL) distribution.

Qt Quick
Originally all the Qt user interfaces were written completely with C++ using a library of pre-built but customizable and extendable UI controls, the QWidgets. That approach is still part of the UI developer offering of the full Qt development environment. But around 2008 when the general trend of UIs was dramatically changing through touch-based smart phones being introduced, the traditional desktop-oriented UI paradigms were not flexible enough to be used for fluid user experiences. The requirements for embedded and mobile UIs were quickly changed. To meet these requirements a new way for writing UIs was introduced and named Qt Quick (Qt User Interface Creation Kit).

Qt Quick separates the declarative UI design and the imperative programming logic. The presentation layer of the UI is written with Qt’s own declarative language called QML. QML is often compared to JSON and parts of the UI logic can directly be written with JavaScript. Even though the UI is written with QML it still has a C++ backend that takes care of rendering and allows direct native device access from the QML as well as extending QML with C++ written graphics, like OpenGL objects. A break-down of the software stack of a Qt Quick application is visualized in Figure 1 . QML was designed to bridge the gap between the UI designers and actual developers. Instead of writing UI prototypes with an alternative technology the designers could actually work on something that can be directly used in the end product. This is also supported by tooling, for instance with a drag’n’drop WYSIWYG editor and the possibility to directly export QML out from Adobe Photoshop or GIMP via a separate plugin.

Figure 1: QtQuickSoftwareStack

Like all Qt code, the same Qt Quick application works as-is across all the supported platforms. By default the same UI layout would be just scaled across different screen sizes on various devices or platforms. Sometimes this is exactly what is wanted and no additional work is required to make the application cross-platform.

Quite often it is desired that the UI layout would be different across different platforms like between mobile phone, car IVI system and a desktop application. In this case the application would be taken through this N-screen approach by replacing just the thin QML written presentation layer of the application re-using the Qt/C++ written backend with most of the functionality within completely. Even though one would need to re-write the QML presentation of the UI layout for all the different deployments of the application it is still very efficient to work with QML due its high abstraction level and easiness to write.

Writing UI Components with QML
The QML as a language only contains basic UI controls, or elements, such as Rectangle, Image, Text, MouseArea and BorderImage. Using these UI elements one can create the desired UI look-and-feel directly or preferably by creating one’s own hierarchical compound elements.

Figure 2: The main QML UI layout, using a new self-created element Figure 3, First, rough implementation of MyButton. (This will later be re-written using imported graphics and the BorderImage QML element).

Figure 3: F i rs t rough i mp l emen t a ti on of My B u tt on. ( This will l a t er be re-wr itt en us i ng i mpor t ed graph ic s and t he Border I mage QML e l emen t) .

Creating new compound elements is rather straightforward using QML. Each QML file defines a new data type named, or element, by the file name that can then be re-used in other files. Thus one could create a MyButton.qml file that would contain a Rectangle with a Text and a MouseArea element inside and then re-use this new MyButton element in their actual UI layout within other QML files. An example code demonstrating this is shown in Figure 2 and Figure 3 above with their output shown in Figure 4 below.

As elements are created hierarchically and abstracted to new data types it is easy to later replace the implementation of one of these types with a refreshed increment. The abovementioned MyButton element could be re-done at a later phase using a BorderImage element with imported high-quality graphics. At the same time the actual UI layout file would remain untouched.

Figure 4: The output of the QML example code

In addition to creating new compound data types from existing ones it is also possible to extend the QML language through its C++ back-end. All the QML elements are rendered using Qt’s C++ rendering APIs and in a same way than the actual language built-in components are done it is possible to write new custom data types using a plugin-based approach. This is often used for instance if one wants to add OpenGL based UI components.Workflow for Rapid UI Prototyping
As QML consists of onlybasic UI elements, it doesn’t provide a real UI framework. So in casethe UI would have different views, navigation and view transitions somesort of UI framework needs to be done manually. It is not necessarily avery demanding task and can be fine-tuned later on, but some thoughtneeds to be done in the beginning to avoid having to re-write the actualcontents: Does the UI consist of different views or pages, does the UIcontain some static parts, like a tab navigator and a status bar or isit just a plain view with pop-up dialogs opening? After this it is easyto know what to start working with–the individual views or dialogs oneby one.

It is rather straightforward to start forming the UI withQML. A good way to start is to use Rectangle elements for the main UIlayout as placeholder for the actual contents and then, one-by-one,create these re-usable UI components.

In the beginning it mightbe easier to leave all graphics out and just use rectangles, circles,texts and the basic layout managers for the UI contents. This alsoallows the UI developer to start working with the actual product evenwith the possible separate UI designer still working with the graphics.Or, one can even take it further: The UI designer doesn’t need to startworking on the graphics at all before a good overall idea of what kindof an UI would work in practice has been achieved.

To get betterunderstanding on how certain UI iteration might work one can conductusability tests also for certain UI iterations. The cross-platformfeature of all Qt code allows the developer to work on the UI in hisdesktop computer and then take the same code and deploy for instance toan Android tablet or a mobile phone. The real life end users can thentest the UI on this device simulating more real-life situation of anembedded touch-based device.

After collecting feedback from theend user the developer can make quick alterations on his desktop andproduce a new version to the device. The QML file of a Qt Quick UI canbe replaced dynamically without needing to recompile. This makes bothtesting on the development environment and deploying to the device veryconvenient and encourages for rapid, agile development. Following is oneexample of the whole iteration of an UI..

Step 1. The first UI mockup (Figure 5 ) is made with QML rectangles and layout managers acting as placeholders for the actual elements.

 

Figure 5: The first UI mock-up made with QML Rectangles and layout managers)

Step 2.  In the second iteration (Figure 6 ),the custom slider element has been implemented. It is done quicklyusing on Rectangles and not graphics but works and could be used to testthe UI. Some graphics have also been brought in.

As the purposeof the application would be to rotate the 3D element ( a Globe), inthis case the sliders would only rotate the blue rectangle.

Figure 6: Custom slider element implemented

Step 3. The third iteration (Figure 7 )now contains the 3D element as well. With the usability tests it wasrealized that two dimensions are enough to rotate a globe and the thirdslider was removed. Instead of the engineer-type approach using X and Yaxes, the sliders were place at the corresponding edges for more naturalbehavior. The menu was also implemented.

Figure 7: A 3D element is implemented in the third iteration

Step 4. Final application While doing tests with the previous iteration using atouch-based tablet, someone asked: “Why are sliders needed at all? Whynot just drag the globe itself around? Not having a logic answer tothis, the drag functionality was quickly implemented for the 3D elementand the can be taken away completely, resulting in a more simplistic UI.

Figure 8: Drag functionality implemented and sliders to create the finished, simplified user interface with natural interaction

Conclusion
QtQuick is a relatively new UI approach within the cross-platformsoftware framework Qt. Qt Quick separates the declarative QML-written UIdesign from the imperative C++ based application logic and back-end.This allows and naturally encourages the developer to work incrementallywith the UI design making rapid prototypes and deployments of the userinterface independently from the possible UI designer as well as theback-end applications development progress.

The possibility ofdynamically replacing the QML UI brings a new aspect to post-marketdevelopment and serviceability as well. By creating a suitableinfrastructure it is possible to dynamically replace the user interfaceof an already sold device either by connectivity or through service.Parts of the UI, or even the whole UI can be dynamically loaded from thebeginning, having the actual QML files reside in a remote serverresulting into a (possibly partially) remote UI, or RUI.

A RemoteUser Interface with Qt Quick allows the developer of the actual deviceto create a framework or a platform for external developers and theirservices. The actual device runs a “main menu” of services that launchthe RUIs of external service providers. This approach has been used forinstance in Consumer Electronics and set-top boxes that allow the enduser to load a network provider’s own submenu, or RUI, if the user hassubscribed to those channels.

Juha Turunen is aPrincipal Consultant working at Digia, Qt. He has years of experience inQt on Embedded devices and is a pioneer of real-world applicationdevelopment with Qt Quick. Juha holds a diploma from LappeenrantaUniversity of Technology (LUT). Juha is based in Santa Clara, CA, USA.

Tuukka Ahoniemi works with the product marketing of Qt, Digia. For the past years hehas traveled around the world spreading the message of Qt supremacy inthe form of having held over 100 Qt training courses, speaking invarious conferences and trade shows. For past two years he worked inadopting Qt into the various use cases of Qt Enterprise customers as aSales Engineer in the Qt Americas Sales team. Still trying to “finish”his PhD he holds a M.Sc (tech) in Computer Sciences with the backgroundin academic teaching of programming and research around it. Tuukka isbased in Tampere, Finland.

This article was part of thematerials for a class on “Rapid Embedded UI Prototyping with Qt Quick(ESC-301)” taught by the authors at the Embedded Systems Conference.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.