FAQ
What does "significant" mean?
At 95% confidence (the standard), the observed difference is unlikely to have occurred by chance if the true conversion rates were identical. The p-value must be below 0.05.
How long should I run the test?
Long enough to reach a meaningful sample (typically thousands per arm) AND covering at least one full business cycle (often a week). Don't stop the test the moment you see significance.