From likelihood to posterior to MCMC

What the sampler does, one step at a time. It wanders over values of $p$, and the values it visits pile up into the posterior. Each step:

1. Propose. From the current $p$, jump to a nearby candidate $p' = p + \text{(a random nudge of size "step")}$.
2. Score. Compute the posterior ratio $r = \dfrac{\text{posterior}(p')}{\text{posterior}(p)}$. Because the unknown normalizing constant cancels, this is just $\dfrac{p'^{\,k}(1-p')^{\,n-k}}{p^{\,k}(1-p)^{\,n-k}}$ — likelihood × (flat) prior.
3. Accept or reject. If the candidate is more probable ($r \ge 1$), move there. If it is less probable, move there only with probability $r$ (flip a biased coin); otherwise stay put and record the current value again.

That is the entire algorithm. Downhill moves are sometimes accepted, which is what lets the chain explore the whole posterior instead of just climbing to the peak. Run it and watch the histogram fill in the exact posterior curve — without ever computing the integral in Bayes' rule.

1. The peak is the MLE. Drag p in panel 1 to the top of the likelihood curve — it lands on k/n.
2. MCMC recovers the posterior. Hit Run and watch the orange histogram in panel 2 fill in the dashed black posterior curve. The MCMC never computes that curve — it just wanders and the samples pile up in the right shape.
3. Step size matters. Set the step very small: the chain crawls and the acceptance rate is high but mixing is slow. Set it very large: most proposals are rejected (low acceptance) and the chain gets stuck. A middling step (acceptance ≈ 20–45%) works best.
4. More data → tighter posterior. Raise n (keeping k/n fixed) and both the likelihood and the posterior get narrower.