Optimal Control

Following [1], let $[0, T]$ be a finite and fixed time horizon and suppose $$\begin{align*} A: [0, T] \rightarrow \mathbb{R}^{n \times n}&, \;\; t \mapsto A(t) \\ B: [0, T] \rightarrow \mathbb{R}^{n \times m}&, \;\; t \mapsto B(t) \\ Q: [0, T] \rightarrow \mathbb{R}^{n \times n}&, \;\; t \mapsto Q(t) \\ R: [0, T] \rightarrow \mathbb{R}^{m \times m}&, \;\; t \mapsto R(t) \end{align*}$$ are continuous matrix-valued functions defined on $[0, T]$. We assume that the matrices $Q(t)$ and $R(t)$ are symmetric and in addition that $Q(t)$ is positive semidefinite and $R(t)$ is positive definite for all $t \in [0, T]$. Furthermore, let $S_T \in \mathbb{R}^{n \times n}$ be a constant, symmetric, and positive semidefinite matrix.

Linear-Quadratic Regulator

The linear-quadratic regulator then is the following optimal control problem [LQ]: Find a continuous function $u: [0,T] \rightarrow \mathbb{R}^m$, the control, that minimizes a quadratic objective of the form $$\begin{equation} \begin{split} { J(u) = \frac{1}{2}\int_0^T \left[ x^\top(t)Q(t)x(t) + u^\top(t)R(t)u(t) \right] dt } \\ { + \frac{1}{2}x^\top(T)S_Tx(T) } \label{eq:cost_fcn} \end{split} \end{equation}$$ subject to the linear dynamics $$\begin{equation*} { \dot{x}(t) = A(t)x(t) + B(t)u(t), \quad x(0) = x_0 } \end{equation*}$$ Theorem. The solution to the linear-quadratic optimal control problem [LQ] is given by the linear feedback control $$\begin{equation} u_*(t,x) = -R(t)^{-1}B(t)^\top S(t) x, \label{eq:lqr_ctrl} \end{equation}$$ where $S$ is the solution to the Riccati terminal value problem $$\begin{align} \begin{split} \dot{S} + SA(t) + A^\top(t)S - SB(t)R(t)^{-1}B^\top(t)S + Q(t) &= 0 \\ S(T) &= S_T. \end{split} \label{eq:dynamic_riccati} \end{align}$$ Proof. Let $u: [0,T] \rightarrow \mathbb{R}^m$ be any continuous control and let $x: [0,T] \rightarrow \mathbb{R}^n$ denote the corresponding trajectory. Dropping the argument $t$ from the notation, we have for any differentiable matrix function $S \in \mathbb{R}^{n \times n}$ that $$\begin{align*} \frac{d}{dt} \left( x^\top S x \right) &= \dot{x}^\top S x + x^\top \dot{S} x + x^\top S \dot{x} \\ &= \left( Ax + Bu \right)^\top S x + x^\top \dot{x} x + x^\top S \left( Ax + Bu \right) \end{align*}$$ and thus, by adjoining this quantity to the Lagrangian in the objective, we can express the cost equivalently as $$\begin{equation*} { J(u) = \frac{1}{2} \int_0^T \left[ x^\top \left( Q + A^\top S + \dot{S} + SA \right)x + x^\top SB u + u^\top B^\top S^\top x + u^\top R u \right] dt } \\ { + \frac{1}{2}x^\top(T) \left[ S_T - S(T) \right]x(T) + \frac{1}{2}x_0^T S(0) x_0 }. \end{equation*}$$ For the moment, let us assume that there exists a solution $S$ to the matrix Riccati equation $\eqref{eq:dynamic_riccati}$ over the full interval $[0,T]$. Then the objective simplifies to $$\begin{equation*} J(u) = \frac{1}{2}\int_0^T \left[ \left( u + R^{-1}B^\top Sx \right)^\top R \left( u + R^{-1}B^\top Sx \right) \right] dt + \frac{1}{2}x_0^T S(0)x_0 \end{equation*}$$ Since the matrix $R$ is continuous and postivie definite over $[0,T]$, th eminimum is realized if and only if $$\begin{equation*} u(t) = -R^{-1}(t)B^\top(t)S(t)x(t), \end{equation*}$$ and the minimum value is given by $$\begin{equation*} J_*(u) = \frac{1}{2}x_0^\top S(0)x_0. \end{equation*}$$ For this argument to be valid, it remains to argue that such a solution $S$ to the initial value problem $\eqref{eq:dynamic_riccati}$ indeed does exist on all of $[0,T]$. It follows from general results about the existence of solutions to ordinary differential equations that such a solution exists on some maximal interval $(\tau, T]$ and that as $t \searrow \tau$ (i.e., $t \rightarrow \tau$ and $t > \tau$), at least one of the components of the solution $S(t)$ needs to diverge to $+\infty$ or $-\infty$. For if this were not the case, then by the local existence theorem on ODEs, the solution could be extended further onto some small interval $(\tau-\epsilon, \tau+\epsilon)$, contradicting the maximality of the interval $(\tau, T]$. In general, however, this explosion time $\tau$ could be nonnegative, invalidating the argument above. That this is not the case for the linear-quadratic regulator problem is a consequence of the positivitiy assumptions on the objective, specifically, the definiteness assumptions on the matrices $R$, $Q$, and $S_T$.

$\blacksquare$

Corollary. If $A(t) \equiv A$, $B(t) \equiv B$, $Q(t) \equiv Q$, and $R(t) \equiv R$ and $T \rightarrow \infty$, the same conclusion holds with the algebraic Riccati equation $$\begin{equation} SA + A^\top S - SBR^{-1}B^\top S + Q = 0 \label{eq:algebraic_riccati} \end{equation}$$

Solving the Algebraic Riccati Equation

It is possible to find the solution to the algebraic Riccati equation by finding the eigendecomposition of a larger system. We define the Hamiltonian matrix $$\begin{equation*} Z = \begin{bmatrix} A & -BR^{-1}B^\top \\ -Q & -A^\top \end{bmatrix} \end{equation*}$$ Since $Z$ is Hamiltonian, if it does not have any eigenvalues on the imaginary axis, then exactly half of its eigenvalues have a negative real part. If we denote the $2n \times n$ matrix whose columns form a basis of the corresponding subspace, in block-matrix notation, as $\begin{bmatrix} U_1^\top & U_2^\top \end{bmatrix}^\top$ then $$\begin{equation*} S = U_2U_1^{-1} \end{equation*}$$ is a solution of the algebraic Riccati equation $\eqref{eq:algebraic_riccati}$; furthermore, the eigenvalues of $A - BR^{-1}B^\top S$ are the eigenvalues of $Z$ with negative real part.