Consider an event with two possible outcome, + and -. If a prediction of $p_+ = p_\mathrm{p}$ is made, how much surprise will be there when the actual outcome is known?
It should be $-\log_2 p_\mathrm{p}$ if the outcome is + and $-\log_2 \left(1-p_\mathrm{p}\right)$ if the output is -.
Consider an unknown spin in a system with known effective local field $h^\mathrm{eff}$ at the temperature $T=\beta^{-1}=1$. The probability of the spin $s$ being positive ($s=+1$) is given by $$ \frac{e^{h^\mathrm{eff}}}{e^{h^\mathrm{eff}}+e^{-h^\mathrm{eff}}} = \frac{1}{2}\left(1+\tanh h^\mathrm{eff}\right). $$ Similarly, the probability for negative spin $s=-1$ is $$ \frac{1}{2}\left(1-\tanh h^\mathrm{eff}\right). $$ Thus, the surprise when learning the outcome of $s$ is generally $$ -\log_2\frac{1+s\tanh h^\mathrm{eff}}{2}. $$
We can use this to check how well the prediction is working for a model that can estimate the effective field $h^\mathrm{eff}$.