Is my A/B test result significant? A plain-English guide with a worked example
Version B of your signup page converted at 7% and version A at 5%. B looks like the winner. The question an A/B test has to answer is whether that gap is real, or whether you'd see one like it by chance even if both pages were identical. That's what statistical significance measures.
What "significant" means
Imagine A and B truly convert at the same rate. Visitors still arrive at random, so the two observed rates will rarely match exactly. The p-value is the chance of seeing a gap at least as big as yours under that assumption. A small p-value means "a gap this large would be unusual if nothing changed". The common threshold is 0.05, often described as 95% confidence.
It doesn't mean there's a 95% chance B is better, and it says nothing about how much better. It's a guard against fooling yourself with noise.
A worked example
Each version gets 1,000 visitors. A has 50 conversions (5%), B has 70 (7%).
- Pooled rate: 120 conversions ÷ 2,000 visitors = 6%.
- Standard error of the difference: √(0.06 × 0.94 × (1/1,000 + 1/1,000)) ≈ 0.0106.
- z = (0.07 − 0.05) ÷ 0.0106 ≈ 1.88.
- Two-sided p-value ≈ 0.06.
So a two-point lift, 40% in relative terms, is not significant at 95% with this much traffic. It's promising, and it might well be real, but a gap like this turns up by chance about 6 times in 100 when there's no difference. The A/B test significance calculator runs this same two-proportion z-test for your numbers.
Mistakes that create false winners
- Stopping as soon as it looks significant. If you check every day and stop the first time p dips below 0.05, you'll declare far more false winners than 5%. Decide the sample size or duration in advance and check once.
- Tiny samples. With a handful of conversions per version, the result swings wildly. The z-test's approximation is also unreliable below roughly 10 conversions per version; treat those results as a hint.
- Changing several things at once. A new headline, button and price together can win, but you won't know which change did it.
- Ignoring the calendar. Weekday and weekend visitors behave differently. Run tests for whole weeks so each version sees the same mix.
- Unequal traffic. If one version got the newsletter blast and the other didn't, you're comparing audiences, not pages.
When you don't have much traffic
Small sites often can't reach significance on small changes in any reasonable time. That's useful to know: test bolder changes (a different offer, a much shorter form), where the effect is big enough to detect, rather than button colours. And measure the steps before the conversion, which have more traffic. A funnel shows which step loses the most people, and that's usually the best place for the next test.
To compare versions in measuremy.site, send each variant's conversion as an event with the variant as a property, and ask your coding agent for conversions by variant. Then put the numbers into the calculator before you call it.
See it on your own site
measuremy.site is cookieless analytics your coding agent can read. 30-day free trial, no card. Plans from $9/month.
Start free trial See the live demoAdd measuremy.site analytics to this project: connect the measuremy.site MCP server at https://a.measuremy.site/mcp for all my projects (claude mcp add --scope user; it signs in with OAuth, no key, and one sign-in covers every site), then tell me how to sign in. Details: measuremy.site/llms.txt
Free tools
- Free conversion rate calculator: Conversion rate from visitors and signups, plus the traffic needed for a goal.
- Free A/B test significance calculator: Is your A/B test result real or luck? Check significance.
- UTM link checker: Find broken UTM tags and see which GA4 channel a link lands in.
More from the blog
- How to calculate your website conversion rate, and what to compare it to
- What is bounce rate, how is it calculated, and is yours a problem?
- How to add a visitor counter to your website or GitHub README
- How much traffic does a website get? How to estimate any site's visits
- How to track QR code scans: count them, and see which poster worked
- How to see how many people clicked your link (without a paid shortener)
- How to check if your Google Analytics 4 tag is installed (and only counting once)
- Why GA4 shows your campaign traffic as Unassigned (and how to fix the links)
- Does your redirect keep your UTM tags? How to check before a campaign goes out
- How I built a self-hosted analytics MCP server on Cloudflare Workers and D1
- UTM tracking explained: what the parameters mean and how to use them
- Conversion funnel analysis for small sites: a practical guide
- Sudden drop in website traffic? A checklist for finding the cause
- What is an MCP server? A plain-English guide for people who run websites
- Website Data Analytics for Owners: A Practical Guide
- Vanity Metrics vs Useful Metrics: What to Track
- How to Compare Cookieless Website Tracking Tools
- Why Your Analytics Numbers Differ Between Tools
- When GA4 Is Too Much for a Small Website
- How to keep historical data when switching analytics tools
- How to give an AI agent read-only access to your analytics
- Why developers are connecting agents to production data
- How to recover from a sudden traffic drop
- Website analytics for indie hackers: what to watch and what to ignore
- Direct Traffic Explained: Why So Much Is Unknown
- Why a high bounce rate is not always bad
- High bounce rate: 8 causes and practical fixes
- How to Set Up Traffic Alerts for Sudden Drops
- How to Track Traffic From a Podcast
- How to measure the return on a paid ad campaign
- How to use analytics to decide what to write next
- How the Model Context Protocol Works
- Track File Downloads Without Guesswork
- Adding analytics to a Next.js app without slowing it down
- Core Web Vitals and Traffic: Is There a Link?
- Analytics After Cookies: What Actually Changes
- GDPR and website analytics: a practical guide for small sites
- How to Compare Website Analytics Tools by Privacy
- GA4 Events Explained for Beginners
- How to measure a newsletter send with UTM links
- How to track Hacker News traffic
- Adding Analytics to a Framer Site
- How to ask your coding agent about your analytics
- What Is Bounce Rate and How Is It Calculated?
- A practical website analytics retention policy
- Map your site's funnel from first visit to signup
- What Is a Good Bounce Rate?
- How to automate a weekly traffic report
- A Practical Privacy-Friendly Analytics Checklist
- Adding an MCP server to Claude Code
- How long should you keep analytics data?
- How to Build a Small MCP Server That Actually Helps
- How to Find the Biggest Drop Off in a Funnel
- What Is the Urchin Tracking Module?
- What Microsoft MCP servers actually do
- UTM parameters for email newsletters
- Track Button Clicks Without Google Tag Manager
- How to track the results of a social media post without cookies
- How to track traffic from a GitHub README
- How to measure a product launch without guessing