The Bayesian approach to the classical Binomial test has been discussed in a few different places (see, here, here, here, and here ). Here, I thought I could provide a brief synopsis of the methodology and then share a javascript - d3 implementation that allows a user to supply a comma separated list of observations and then receive the results of the analysis through an explanatory visualization.
Given some data that represent samples drawn from two binary processes, we would like to understand if these two processes are equivalent. These processes could represent the results of flipping coins, users clicking on web ads, or people responding to email campaigns.
Mathematically, we can model each process using a Binomial distribution, \[ Binomial(n,k) = {n \choose k} p^k (1-p)^{n-k} \] where n corresponds to the number of trails, k the number of positive observations (or successes), and p is the probability of observing a positive observation.
Being a Bayesian approach, we can then place prior over the data. For simplicity, we'll choose an uninformative one. The Binomial's conjugate prior is the Beta distribution, \[ Beta(\alpha,\beta) = \frac{p^{\alpha-1}(1-p)^{\beta -1}}{B(\alpha,\beta)} \] where \(\alpha \) and \( \beta\) represent pseudo counts of prior observations. When both parameters are equal to 1, the distribution is uniform and thus, uninformative.
Since the Beta distribution is the conjugate of the Binomial, it is simple to obtain an analytical solution to the posterior for each process, we simply add up the observations and pseudo counts and use them as parameters for a new Beta distribution, \[ Beta(k+\alpha,n-k+\beta). \]
Thus far, we have only described a single process, but we are really interested in the difference between the two. To get a posterior that represents this difference, we can simply sample from each posterior, take the difference between samples, and study the structure of the resulting posterior.
To quantity uncertainty, Bayesians use credible intervals, which are practically similar (but by no means are they philosphically the same) to how Frequentists use confidence intervals. A 95% credible interval will contain those domain values that fall between the 0.0275 and 0.975 quantiles.
Now that we have reviewed the foundations of the Bayesian equivalent to the Binomial test and discussed how we can quantify the uncertainty in our estimate, we can use the test. For completeness, the code is available on Github here.
To use the tool, simply enter a comma separated list of observations for each group into the appropriate text area below and then click the Run Test button to see the results. A smoothed posterior distribution will be plotted, along with the bounds for a 95% credible interval.