Updating parameters in linear models

Abstract: It is sometimes desired to update solutions to systems of equations or other
problems as new information is to be appended. Also, a system that is too
large to solve directly can often be managed by first solving a part of the
system, and then updating the solution with the rest of the system. This
updating procedure is often required to be both efficient and stable, and
recomputing the solution from scratch may be too costly. Beside efficiency
and stability, factors such as storage requirement, simplicity, and
applicability are often important.

Updating the least squares solution to an over determined system of linear
equations can be done in many ways. The method of Recursive Least Squares is
simple and efficient, and works in most cases, although it is a bit sensitive
to round-off errors. Updating the QR Decomposition, the Cholesky
Factorization, or the Singular Value Decomposition is in general more stable,
but these tasks are often a bit more complex than the Recursive Least
Squares. Updating problems with constraints is possible when using
Constrained Recursive Least Squares.

These methods all give a single solution as result, for example the least
squares solution, but errors and uncertainty are not handled.

The Bayesian Inference offers a different type of updating. Here the answer
is given in form of a probability distribution, with which it is possible to
study the reasonableness of different solutions. This approach handles
measuring errors, and information about the uncertainty in the answer is
available.

Bayesian Inference can be applied to both linear and non-linear models, and
one can incorporate additional information about the solution. When the noise
and other errors are assumed Gaussian, the calculations are particularly
simple and closely related to ordinary least squares problems. Some
connections to non-Bayesian methods are pointed out, as well as a
non-recursive property of the Bayesian Inference in linear problems with
independent observation series.

These two different approaches are discussed, and examples are given at the
end to give concrete form to the theory.