How I test software, again and again
Validation?
All my career, I've had people tell me their software was "validated." On rare occasions, it actually is, but usually not. In my experience, the claim is often used to intimidate. It's like saying, "Who do you think you are, to question my software, when the entire U.S. Army has already signed off on it?"
Usually, the real situation is, the fellow showed his Army customer one of his outputs, and the customer said, "Yep, it looks good to me." I've worked with many "validated" systems like that and found hundreds of egregious errors in them.
Even so, software absolutely can be validated and should be. But trust me, it's not easy.
A lot of my projects involve dynamic simulations. I'm currently developing a simulator for a trip to the Moon. How do I validate this program? I sure can't actually launch a spacecraft and verify that it goes where the program said it would.
There's only one obvious way to validate such software, and that's to compare its results with a different simulation, already known to be validated itself. Now, when I say compare the results, I mean really compare them. They should agree exactly, to within the limits of floating-point arithmetic. If they don't agree, we have to go digging to find out why not.
In the case of simulations, the reasons can be subtle. Like slightly different values for the mass of the Earth. Or its J2 and J4 gravity coefficients. Or the mass of its atmosphere. Or the mass ratio of the Earth/Moon system. You can't even begin to run your validation tests until you've made sure all those values are identical. But if your golden standard is someone else's proprietary simulation, good luck getting them to tell you the constants they use.
Even if all the constants are exactly equal and all the initial conditions are identical, it's still not always easy to tell that the simulations agree. It's not enough to just look at the end conditions. Two simulations only agree if their computed trajectory has the same state values, all the way from initial to final time.
Now, a dynamic simulation will typically output its state at selected time steps—but selected by the numerical integrator way down inside, not the user. If the two simulations are putting out their data at different time hacks, how can we be sure they're agreeing?
Of course, anything is possible if you work at it hard enough, and this kind of validation can be done. But, as I said, it's not easy, and not for the faint-hearted.
Fortunately, there's an easier and more elegant way. The trick is to separate the numerical integration scheme from the rest of the simulation. You can test the numerical integrator with known sets of differential equations, that have known solutions. If it gets those same solutions, within the limits of floating-point arithmetic, it must be working properly.
Now look at the model being simulated. Instead of comparing it to someone else's model, compare its physics with the real world. Make sure that, given a certain state of the system, you can calculate the derivatives of the states and verify that they're what the physics says they should be.
Now you're done, because if you've already verified the integrator, you can be sure that it will integrate the equations of motion properly and get you to the right end point.
Is this true validation? I honestly don't know, but it works for me.
Hand checks
Now we get to the real meat of the issue, which is how to do hand checks. I've been telling you for some time now that I do a lot of testing, including testing every single line of code, usually as I write it. When I say testing, I mean give the code under test some inputs and verify that it gives the right outputs.
But where do those inputs come from? From a test case, of course.
Now, almost all of my software tends to be mathematical in nature—old-time readers know that. So my test cases tend to be numeric, and so are the expected results. But the nature of the software really doesn't matter. Regardless of the type of data, your code should still process it correctly.
Remember the Adams dictum for software testing:
Never ask the computer a question you don't already know the answer to.
Just picking inputs at random (that word again), looking at the results and saying "Yep, that looks right" won't get it done. You have to know it's right.
In the good old days, I used to generate the expected outputs by laborious hand calculation. Typically, the only aid I had was a pocket calculator or, worse yet, a Friden (www.oldcalculatormuseum.com/fridenstw.html). Sometimes, I even had to compute the expected results in octal, because I was dependent on core dumps to see the internal computations. It was not fun, but it had to be done.
Fortunately, today there's a much better approach. Today, we can choose between math-oriented applications such as MATLAB, Mathcad, Mathematica, and Maple. I usually find myself using Matlab or Mathcad, or both.
Enter Mathcad
Long-time readers might remember my heated rants against Mathcad (now sold by PTC). I complained loudly and longly how frustrating it was to use, and how much it was plagued with bugs.
I haven't changed my mind. But a funny thing happened. One day I realized that, despite my frustration with Mathcad, I use it almost daily. It's by far the most oft-used application in my computer. The reason is simple. Despite its faults and my complaints, it's still the best tool of its type, bar none. It's like the old joke: it's the only game in town.
Yes, I know, the other tools I mentioned tend to be much more powerful (and a whole lot more expensive) than Mathcad. But only Mathcad lets me both enter math equations in a WYSIWG format and get the results back the same way. When I'm doing math analysis, either as a pure analysis or to generate test cases, I like to use the application to both do the calculations and document the process. With Mathcad, I can interleave the math equations, text, and results to get a worksheet that looks like a page out of someone's textbook.
And yes, I still curse Mathcad sometimes. It's a love-hate relationship. I hate the frustrations, but love the final results. Come to think of it, it's the same relationship I have with Microsoft Word or Windows.
Loading comments... Write a comment