Why least squares?
After all is said and done, you might be wondering if the definition of the penalty function of Equation 11 is the best one. How did we come to decide that the least squares criterion is the right one for a "best fit," anyhow?

The answer is both simple and incredibly profound. It involves the secret to Life, the Universe, and Everything.

Albert Einstein is widely quoted as saying, "Simplify as much as possible, but no more." Ockham's razor says that, given two competing theories, the simpler one is probably the correct one. And indeed, the universe does seem to be a simple place, in the sense that most of the equations describing it are themselves simple, usually involving linear relationships. Acceleration is proportional to force. Voltage is proportional to the rate of change of current.

If a linear relationship doesn't do, a quadratic or inverse-square relationship usually does. The electrostatic force is inversely proportional to the square of the distance. Einstein himself was struck by the apparent simplicities in nature and noted that we had no right to expect such a result. But we have it just the same.

So as we look for a definition of "best fit," we can feel justified in seeking a simple definition. There are candidate criteria even simpler than the sum-of-squares criterion. For example, we could try minimizing the sum of the errors instead of their squares. Or we could minimize the sum of their absolute values.

Neither of these criteria work. If I define M as the sum of errors and then move the fitting line up and down by changing a, I simply add na to M. There is no minimum value in M. On the other hand, if I use absolute values, moving the line upward increases errors below the line, and decreases the ones above it. Again, there's no direct relationship between a and M that can lead to a minimum.

There is one possible candidate that could work. This is the so-called minimax solution, in which we seek to minimize the magnitude of the largest error(s). Such an approach usually gives a solution in which two or more of the largest errors are equal in magnitude.

The problem with the minimax approach is that it's not continuous. We got good results from the use of the partial derivatives, by assuming M was a smooth function with continuous derivatives in the vicinity of the minimum. That won't be true for the minimax case. As we move the fitting line around, the points with the largest errors will tend to jump around also, and in somewhat unpredictable ways. A computerized search could still produce a minimum, but a calculus-based minimization won't work.

So here's the big secret behind the least squares fit, and it's also the secret behind Life, the Universe, and Everything:

It's the simplest definition that works.

Simplify as much as possible, but no more. Words to live by.

Jack Crenshaw is a systems engineer and the author of Math Toolkit for Real-Time Programming. He holds a PhD in physics from Auburn University. E-mail him at jcrens@earthlink.net. For more information about Jack click here