Fréchet Derivative

If F\colon X\to Y is a function between normed linear spaces the Fréchet derivative {DF\colon X\to\mathcal{B}(X,Y)} is defined by F(x + h) = F(x) + DF(x)h + o(h) for x,h\in X where \mathcal{B}(X,Y) is the set of bounded linear operators from X to Y. The “little o” notation means \|F(x + h) - F(x) - DF(x)h\|/\|h\| \to 0 as \|h\| \to 0. The function F can be approximated near x by the linear operator DF(x).

A norm is a function \|\cdot\|\colon X\to\boldsymbol{R} satisfying \|ax\| = |a|\|x\|, \|x + y\| \le \|x\| + \|y\|, \|x\|\ge0 and \|x\| = 0 implies x = 0 for a\in\boldsymbol{R}, x,y\in X.

If X = Y = \boldsymbol{R} then DF\colon\boldsymbol{R}\to\mathcal{B}(\boldsymbol{R},\boldsymbol{R}) where DF(x)h = F'(x)h.

If X = \boldsymbol{R}^n and Y = \boldsymbol{R}^m then DF(x)\colon\boldsymbol{R}^n\to\boldsymbol{R}^m is the Jacobian.

Exercise. If T\colon X\to X is a bounded linear operator then DT = T.

If G\colon Y\to Z then D(G\circ F)(x) = DG(F(x))DF(x)

Exercise: Prove the chain rule for Frechet derivatives.

Hint: G\circ F(x + h) = G(F(x + h)) = G(F(x) + DF(x)h + o(h)) = G(F(x)) + DG(F(x))DF(x)h + o(h).

Define the dual of a normed linear space X by the vector space of bounded linear functionals X^* = \mathcal{B}(X,\boldsymbol{R}). If {X^I = \{x\colon I\to\boldsymbol{R}\}} is the set of functions from the set I to \boldsymbol{R} we can define an inner product (\cdot,\cdot)\colon X\times X\to\boldsymbol{R} by (x, y) = \sum_{i\in I}x(i) y(i) if I is finite. We also write x_i for x(i).

Exercise. Show the inner product is bilinear.

Hint: Show (ax + y, z) = a(x,y) + (y,z) and (x, ay + z) = a(x,y) + (x,z) for a\in\boldsymbol{R}, x,y,z\in X using (x,y) = \sum_{i\in I}x_i y_i.

The inner product defines a norm by \|x\| = \sqrt{(x, x)}.

Exercise. Show |(x,y)| \le \|x\| \|y\| for x,y\in X.

Hint: Use 0\le\|ax + y\| for a\in\boldsymbol{R}, x,y\in X and minimize over a.

Exercise. Show if |(x,y)| = \|x\| \|y\| and x\not=0 then y = ax for some a\in\boldsymbol{R}.

Exercise. Show \|\cdot\| is a norm.

The canonical basis of \boldsymbol{R}^I is e_i\in\boldsymbol{R}^I, i\in I, defined by e_i(j) = \delta_{ij}, where \delta_{ij} = 1 if i = j and is zero otherwise.

Exercise: Show x = \sum_{i\in I} x(i)e_i for x\in\boldsymbol{R}^I.

Hint: Compute x(j), j\in I.

Define the dual basis e_j^*\in(\boldsymbol{R}^I)^*, j\in I, by e_j^*(e_i) = \delta_{ji}.

Exercise: Show x^* = \sum_{j\in I} x^*(e_j) e_j^* for x^*\in X^*.

Hint: Compute x^*(e_i), i\in I.

The dual map *\colon\boldsymbol{R}^I\to(\boldsymbol{R}^I)^* defined by e_i\mapsto e_i^* allows us to identify (\boldsymbol{R}^I)^* with \boldsymbol{R}^I.

Exercise. Show the dual map is an isometric isomorphism.

Exercise: _Show D\|x\|^p = p\|x\|^{p-1}x^* for x\in\boldsymbol{R}^I.

Hint: Compute D\|x\|^2 and use \|x\|^p = \exp(p/2 \log\|x\|^2).

Exercise: Show (Dx^2)h = hx + xh if x is a linear operator in \mathcal{B}(V,V).

For x\in\mathcal{B}(V,V) define right and left multiplication by R_xy = xy and L_xy = yx.

Exercise: Show Dx^n = \sum_{j=0}^{n - 1} R_x^{n - j} L_x^j.

Hint: The previous exercise establishes the case n = 2. Use induction.