Lagrange multipliers

"The method of Lagrange multipliers" is one of indispensable theorem for my life. Therefore I will share what the "Lagrange multipliers" is and some intuitive understanding about it.

0. Lagrange multipliers technique¶

First of all, let's wrapp you head around lagrange multipliers technique. To make it simple, we're gonna deal with only 2 dimention in this article. Let's say you wanna maximize multivariate function $f(x,y)$. subject to the constraint that another multivariate function equals a constant, $g(x,y) = C$
In that situation we can apply lagrange multiplier technique. At first introduce new variable $\lambda$. And define new function $L$ which takes form

$$L(x,y,\lambda) = f(x,y) - \lambda \varphi(x,y)\ \ where\ \ \varphi(x, y) = g(x,y) -C$$

$L$ is called "Lagrange function", $\lambda$ is "Lagrange multiplier".
Suppose $(x_0, y_0)$ maximaize (or minimize) $f(x,y)$ subject to $\varphi(x,y) = 0$, the following equality holds,

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} = \frac{\partial L}{\partial \lambda} = 0$$

1. Intuitive understanding¶

From above equality,

$$\begin{eqnarray}\frac{\partial \left\{f(x,y) - \lambda \varphi(x,y)\right\}}{\partial x}&=&0 \\ \frac{\partial f(x)}{\partial x} &=& \lambda \frac{\partial g(x)}{\partial x}\end{eqnarray}$$ 　　 In the same way, $$\begin{eqnarray}\frac{\partial f(x)}{\partial y} &=& \lambda \frac{\partial g(x)}{\partial y}\end{eqnarray}$$ 　　

Therefore,

$$\nabla f = \lambda \nabla \varphi$$

You can tell gradient of $f(x,y)$ and $g(x,y)$ are parallel at the point of $(x,y)$ which maximaize (or minimize) $f(x,y)$ subject to $g(x,y)$.

2. What the $\nabla f(x,y)$ is ??¶

Let's think about contour of $f(x,y)$ where $f(x,y) = C$. Suppose moving the point along contour by $(\varDelta x, \varDelta y)$. At the time, total derivative can be captured as below,

$$\frac{\partial f}{\partial x}dx + \frac{\partial y}{\partial y} dy = \nabla f(x) \cdot \varDelta x= 0$$

Therefore gradient $\nabla f$ is gonna be normal vector to tangent vector of contour.

Reference :
Interpretation of Lagrange multipliers
ラグランジュの未定乗数法の解説と直感的な証明