When I worked on "variational inference" of "Bayesian machine learning" using ''mean-field approximation", because of inadequate understanding "expected value", I was stuck for several hours... Then I thought, there might be someone in the world who has the same adversity. Hence I will share with you regarding properties of "expected value".

Properties of "expected value "¶

Just in case, before getting into properties of expected value we should be on the same page on what the expected value is .
Let's say that we have random variable $x$, and $f(x)$ as a function of $x$. Expected value of random variable (continuous) is presented as below.
$$< f(x)> _{p(x)} = \int f(x)p(x)dx$$

1. Linearity¶

When it comes linearity, you can derive as below with comparative ease :) $$\begin{eqnarray}<f(x) + g(x)>_{p(x)} &=& \int \{f(x)+g(x)\}p(x)dx\\ &=& \int f(x)p(x)dx + \int g(x)p(x)dx\\ &=& <f(x)>_{p(x)} + <g(x)>_{p(x)}\end{eqnarray}$$ $$\begin{eqnarray}<cf(x)>_{p(x)} &=& \int cf(x)p(x)dx\\ &=& c\int f(x)p(x)dx\\ &=& c<f(x)>\ \ \ where \ c \in \mathcal{R} \\ \end{eqnarray}$$

2. Multiple random variable 1¶

If there are maultiple random variable ($z_1, z_2, z_3$), expected value can be calculated as below. $$\begin{eqnarray} <f(z_1, z_2, z_3)>_{p(z_1)p(z_2)p(z_3)} &=& <\int p(z_1)f(z_1, z_2, z_3)dz_1>_{p(z_2)p(z_3)}\\ &=&<(f\ of\ z_2\ and\ z_3) >_{p(z_2)p(z_3)}\\ &=&<\int p(z_2)(f\ of\ z_2\ and\ z_3)dz_2 >_{p(z_3)}\\ &=&<(f\ of\ z_3) >_{p(z_3)}\\ &=& \int p(z_3)(f\ of\ z_3)dz_3\\ &=& Final\ expected\ value \end{eqnarray}$$

3. Multiple random variable 2¶

This is a pattern where $f$ is a function of only $z_1$, whereas ramdom variables are $z_1,z_2,z_3$, $$\begin{eqnarray} <f(z_1)>_{p(z_1)p(z_2)p(z_3)} &=& <\int p(z_2)f(z_1)dz_2>_{p(z_1)p(z_3)}\\ &=&<f(z_1)\int p(z_2)dz_2>_{p(z_1)p(z_3)}\\ &=&<f(z_1)>_{p(z_1)p(z_3)}\\ &=&<\int p(z_3)f(z_1)dz_3>_{p(z_1)}\\ &=&<f(z_1)\int p(z_3)dz_3>_{p(z_1)}\\ &=& <f(z_1)>_{p(z_1)}\\ \end{eqnarray}$$