The statistics of A/B testing results can be confusing unless you know exact formulas. Earlier, we had published articles related to mathematics of A/B testing and also have a free A/B testing calculator on the site to see if your results are significant or not. The articles provides an introduction and calculator simply provides an interface; the real formulas used for calculate statistical significance of split testing results are still missing.
Excel sheet with A/B testing formulas
So, we have come up with a FREE spreadsheet which details how exactly the significance is calculated. You just need to provide the number of visitors and conversions for control and variations. The spreadsheet will automatically calculate for you significance, p-value, z-value and other relevant metrics for any kind of split testing (including Adwords). Of course, you can see the relevant formulas in the spreadsheet. Click the screenshot below to download the calculator (spreadsheet):
Please feel free to share the file with your friends and colleagues or post it on your blog / twitter.
PS: By the way, if you want to do quick calculations, we have a version of this calculator hosted on Google Docs (please make a copy of the Google Doc sheet into your own account before you make any changes to it).
Update: Thanks to Jai (in the comments below), we had noticed a minor error in conversion rate range calculations (though significance results were unaffected). The error in fixed in the latest version of spreadsheeet.