Designing An Arm-Based Multithreaded Audio/Visual/Motion Recording System: Part 1 -

Designing An Arm-Based Multithreaded Audio/Visual/Motion Recording System: Part 1


Although it is a generic case study design – not an actualimplementation – a real time video/ audio/ motion (VAM) recordingsystem highlights a number of key operating system features andservices that must be considered in any embedded system design,including:

* application timers
* threads
* message queues
* mutexes
* memory byte pools

Designed for use in motorized vehicle fleets this case study dependsheavily on the use of application timers. One reason for using so manytimers is because we need to schedule the copying of data from thetemporary memory to the protected memory whenever any one of fourevents occurs.

Our design provides the ability to record several events within each24-second time frame, rather than just one. Application timers play amajor role in providing this feature. We also used application timersto simulate interrupts that signify the occurrence of events, and weused one timer to display periodic system statistics.

How the VAM works
The VAM system, based on the ThreadXRTOS and ARM processor,features a small recording device that could be attached to a vehicle'swindshield directly behind the rear-view mirror to avoid intrusion intothe driver's field of vision.

When triggered by an accident or unsafe driving, the VAM systemautomatically records everything the driver sees and hears in the 12seconds preceding and the 12 seconds following the event. Events arestored in the unit's digital memory, along with the level of G-forceson the vehicle. In the event of an accident, unsafe driving, warning,or other incident, the VAM system provides an objective, unbiasedaccount of what actually happened.

To complete the system, there should be a driving feedback system that downloadsthedata from the VAM system unit and provides playback and analysis. Thissystem could also be used to create a databaseof incidents for all the drivers in the vehicle fleet. We will notconsider that system; instead, we will focus on the capture ofreal-time data for the VAM system unit.

As noted earlier, most of the VAM unit could be located behind therear view mirror, so it would not obscure the vision of the driver. Theunit would have to be installed so that the lenses have a clear forwardview and rear view. The system includes a readily accessible emergencybutton so the driver can record an unusual, serious, or hazardousincident whenever necessary. The VAM system is constantly recordingeverything the driver sees, hears, and feels. That is, the VAM systemrecords all visual activities (front and rear of the vehicle), audiblesounds, and G-forces.

As illustration of how the VAM system could be used, consider thefollowing scenario: A driver has been somewhat inattentive and hasfailed to stop at a red traffic light. By the time the driver realizesthe error, the vehicle has already entered the intersection and isheaded toward an oncoming vehicle.

The driver vigorously applies the brakes and swerves to the right.Typical G-forces for thisincident are about “0.7 (forward) and +0.7 (side). Thus, the VAM systemdetects this incident and records it as an unsafe driving event in theprotected memory.

When we download and analyze the data from the VAM system, we shouldbe able to clearly see that the driver ran a red light and endangeredpassengers and other people on the highway, as well as the vehicleitself. We would have been legally liable for the driver's actions ifthis incident had resulted in a collision.

In this scenario, no collision resulted from this incident. However,this recording would show that this driver was clearly at fault andperhaps needs some refresher training. Figure1 below illustrates the G-forces that can be detected, where thefront of the vehicle appears at the top of the illustration.

Figure1: Directions of G-forces.

The system stores the 24 seconds of video, audio, and motionrecording that surround the time of this incident in protected memoryand illuminates a red light that indicates a driving incident hasoccurred. This light can be turned off only when the specialdownloading process has been performed; the driver cannot turn it off.For simplicity, we will omit certain details that are not important tothe development of this system, such as file-handling details.

Statement of Problem
The VAM system is based on a set of sensors that measure G-forcesexperienced by a driver in a motorized vehicle. The system uses twosets of measurements. One set indicates forward or backward motion ofthe vehicle. Negative forward values indicate deceleration, or G-forcespushing against the driver's front side, while positive forward valuesindicate acceleration, or G-forces pushing against the driver's back.

The other set of measurements indicates sideways motion of thevehicle. Negative side values indicate acceleration to the right, orG-forces pushing against the driver's left side, while positive sidevalues indicate acceleration to the left, or G-forces pushing againstthe driver's right side. For example, if a vehicle makes a hard leftturn, then the sensors produce a positive side value.

Figure2: Events and corresponding priorities

The VAM system detects and reports four categories of events. Weassign each category a priority, indicating the importance of theevent. Figure 2 above liststhe event categories and their corresponding priorities. (This eventpriority is not the same as a ThreadX thread or interrupt priority. Weuse event priorities to classify the relative importance of the events;we do not use them to affect the time when the events are processed.)

Event priorities serve two primary purposes. First, a priorityindicates the severity of an event. Second, an event prioritydetermines whether the current event can overwrite a previously storedevent in the protected memory.

For example, assume that the protected memory is full and the driverhits the emergency button, thereby creating a manually triggered event.The only way that this event can be saved is if a previous manuallytriggered event has already been stored. Thus, a new event canoverwrite a stored event of the same or lower priority, but it cannotoverwrite a stored event with a higher priority.

If the G-force sensors detect an accident, unsafe driving, orwarning, the VAM system generates an interrupt so that ThreadX can takeappropriate action and archive that event. Figure 3 below contains a graphicalrepresentation of the G-forces in this system, in which the eventlabeled by the letter “W” is a warning event.

Figure3: Graphical classification of events by G-forces.

The driver may hit the emergency button at any time to generate aninterrupt, signifying a manually triggered event. Figure 4 below   contains the actualG-force values that are used to detect and report these events. Weassume symmetry in how we classify forward and side G-forces, but wecould easily modify that assumption without affecting our design.

Figure4: G-Forces and event classifications.

To add some perspective about G-forces, consider a vehicleaccelerating from zero to 60 miles per hour (0 to 96 kilometers/hour)in six seconds. This produces a G-force of about 0.4—not enough totrigger an unsafe incident report, but enough to trigger a warningevent.

However, if a driver is applying “hard braking” to a vehicle, itcould produce a Gforce of about 0.8, which would trigger an unsafedriving event. If a vehicle crashes into a solid wall while travelingat 62 mph (100 km/hr), this produces a G-force of almost 100!

The VAM system uses two non-volatile memory systems: a temporarymemory system and a protected memory system. The protected memorysystem stores only detected or manually triggered incidents, while theother system is a temporary memory that records video, audio, andG-forces. It's not necessary to retain ordinary driving activities, sothe temporary memory system is overwritten after some period of time,depending on the size of the temporary memory.

As noted previously, the protected memory system stores all crashevents, unsafe driving events, warnings, and manually triggered events,plus associated audio and video, as long as memory space is available.The protected memory system could be available in several differentsizes, and our design will be able to accommodate those differentmemory sizes.

Figure 5 below  illustrates the temporary memory system used for continuous recording.This is actually a circular list where the first position logicallyfollows the last position in the list.

Figure5: Temporary memory (Circular list).

This system provides temporary storage that is overwrittenrepeatedly. Its main purpose is to provide data storage for an eventthat needs to be saved in the protected memory.When an event occurs,the 12 seconds preceding the event have already been stored in thetemporary memory. After the 12 seconds of data following the event havebeen stored, the system stores this 24 seconds of data in protectedmemory.

The actual size of the temporary memory can be configured to theneeds of the user. Figure 6 below illustrates the protectedmemory that is used to store the automatically detected or manuallytriggered events.

Figure6: Protected memory.

The size of the protected memory can also be configured according tothe needs of the user. We arbitrarily assume that this memory can store16 events, although we can change this value without affecting ourdesign. In addition to the actual data for the event, the protectedmemory must store the priority and time of the event's occurrence.

This information is essential in the case where the protected memorybecomes full and another event is detected or manually triggered. Anevent can never overwrite a higher priority event, but it can overwritean event with the same or lower priority. Figure 7 below summarizes the event overwriterules.

Figure7: Event overwrite rules.

As stated previously, when the G-force sensors detect a crash, thesystem generates an interrupt with event priority 1. Unsafe drivingevents are logged as event priority 2, warnings as event priority 3,and manual events (pushing the emergency button) as event priority 4.Our objective is to respond to these events (which will appear to thesystem as interrupts), as well as to handle initialization and toprocess routine data.

Our design will be simplified and will concentrate on controlissues. Next we will consider thread design issues, and publicresources design. We will assume that other processes will handle theactual storing and copying of data. (These data handling tasks could beperformed by the software package FileX, which is a companion productto ThreadX.)

Thread Design
We need a thread to performvarious initialization duties such as setting pointers and variables, opening thefiles for the temporary memory and the protected memory, andestablishing communication paths and buffers. We will assign the nameinitializer to thisthread. 

We need a thread to coordinate the capture and storage of data fromthe VAM system unit to the temporary memory. This thread manages thetransfer of audio, video, and G-force data from the VAM system unit toan internal buffer, and then to the temporary memory. We will assignthe name data_capture to thisthread. Figure 8 below, illustrates this thread'soperation.

Figure8: Capturing data from the VAM unit.

We need four ISRs to handle the detected and triggered events. Weneed one message queue called event_notice to store information about an event until the copying process begins.We also need one thread to process the copying of data from thetemporary memory to the protected memory. We will assign the nameevent_recorder to this thread.

This thread is activated 12 seconds after an event has beendetected. At this time, the temporary memory contains the 12 seconds ofdata preceding the event and 12 seconds of data following the event.

The event_recorder thread then takes identifying information fromthe event_notice queue andthen copies the audio, video, G-force, and timing data for this eventfrom the temporary memory to the protected memory, including the eventpriority. If protected memory is full, the event recorder threademploys the overwrite rules shown in Figure7, earlier .

We will use four timers to simulate the arrival of externalinterrupts for the detectable and triggered events. To simulate such aninterrupt, the timer in question saves the thread context, invokes thecorresponding ISR, and then restores the thread context.

We will assign the names crash_interrupt ,unsafe_interrupt,warning_interrup t, and manual_interrupt to these four timers. Figure 9 below presents an overview of the interaction between the timers thatsimulate interrupts, their associated ISRs, the scheduling timers, andthe event recorder thread that actually performs the copying.

Figure9: Overview of event interrupts and data recording.

Public Resources Design
We will use one mutex to provide protection for the protected memorywhile copying data from the temporary memory. To begin copying data,the event_recorder thread must obtain ownership of this mutex. We willassign the name memory_mutex to this mutex.

We need four application timers to schedule the copying of data fromtemporary memory to protected memory. When an event is detected, thecorresponding application timer is activated and expires precisely 12seconds later. At this point in time, the event_recorder thread beginscopying data. We will assign the names crash_timer, unsafe_timer, warning_timer, and manual_timer to thesefour application timers. Figure 10,below   illustrates how these applicationtimers generate interrupts and schedule the processingof events.

Figure10: Event processing.

As illustrated in Figure 10 above ,an event interrupt is simulated by one of the four timers created forthis purpose. One of four corresponding ISRs is invoked and it sendsevent information to the message queue, activates one of fourcorresponding scheduling timers, and sets it to expire in 12 seconds.When that timer expires, it resumes the event_recorde r thread, which shouldbe in a suspended state unless it is in the process of copyinginformation from temporary memory to protected memory.

When the event_recorder thread resumes execution, it takes one message from the queue. Thismessage contains two pieces of information about an event: the frameindex and the event priority. The frame index is a location intemporary memory that specifies where information about the eventbegins.

The event_recorder thread then obtains the memory_mutex and if successful,proceeds to copy information from temporary memory to protected memory.When the thread completes copying, it releases the mutex and suspendsitself. Figure 11 below contains a summary of the public resources needed for this case study.

Figure11: Summary of public resources used for the VAM system.

Figure 12 below contains asummary of the definitions, arrays, and variables used. We willsimulate the temporary memory file system and the protected memory filesystem with arrays.

Figure12: Definitions, arrays, and counters used in the VAM system.

The event counters in Figure 12,above are used when printing periodicstatistics about the state of the system. We also need a collection offunctions to perform the actions of the timers and the threads. Figure 13 below summarizes allthread entry functions and timer expiration functions used in thissystem.

Figure13: Entry and expiration functions used in the VAM system.

Next in Part 2on Implementing the VAM system, wewill develop the complete program for our system as well as for theindividual components of our system.

Thisseriesis reprinted in two parts from Chapter 14 of “ Real-TimeEmbedded Multithreading, Using ThreadX and ARM ” with permission ofElsevier.

Dr. Edward L. Lamie is professoremeritus and former department chair of California State University'sComputer Science Department at the Stanislaus Campus. He directseducational services at Express Logic, where he is responsiblefor development and delivery of customer training.

Editor's note:A complete listing of the system described here is included in Lamie'sbook as well as on the CD in the back of the book.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.