Why you need the number before the test starts
Say a clothing store in Tashkent tests two Instagram banners. By the evening of day one, one has six Direct messages and the other has nine. It is tempting to switch off the loser. But at numbers this small, the picture can flip tomorrow, because chance, not the creative, produced the gap.
Sample size is the amount of data per variant at which a difference stops being a coin toss. Calculating it up front also tells you whether your traffic and budget can support an honest test, or whether you should make the decision another way.
The four inputs behind every calculator
Every sample size calculator asks for the same things. You do not need to memorise the formula, but you do need to know what each input means.
- Baseline conversion: the share of people who currently take the target action. The lower it is, the more people you need to collect enough conversions to compare.
- Minimum detectable effect: how different a variant must be for the difference to matter to you. Spotting a small gain costs far more traffic than spotting a big one.
- Significance level: how much risk you accept of seeing a difference that is not real. You choose it in advance.
- Statistical power: the chance of catching a real difference when one exists. Higher power means a larger sample.
How to calculate A/B test sample size, step by step
Calculators usually ship with sensible defaults for significance and power, and for ad tests most people leave them alone. The first two inputs are the real levers.
- Take your baseline conversion from past data, for example the share of clicks that became enquiries.
- Decide the smallest improvement that would change your decision.
- Open any free sample size calculator for A/B tests.
- Enter the baseline and the minimum effect, and keep the default significance and power.
- Write down the result: it is the number of visitors or clicks each variant needs.
Choosing a conversion that reflects real sales
A common mistake is to compare variants by likes or cheap clicks, while money comes from other events: a Click or Payme payment, an order in a Telegram bot, a phone call, a sale recorded in your CRM. The closer an event is to revenue, the rarer it is, and the larger the sample you need.
This is where end-to-end analytics earns its place. When events from your website, bot and calls are tied back to the ad account, you can see which variant brought buyers, not just enquiries. Without it, the winner may be the ad that collects the most empty leads. Building that kind of reporting is part of what markdev does for businesses in Uzbekistan.
What to do when traffic is limited
For a small business in the regions or a narrow niche, a full test on sales can take impractically long. A few compromises help.
- Test big differences: a different offer or angle, not a button shade.
- Compare on a more frequent event, such as a click or starting a form, and remember the conclusion is weaker.
- Group similar ideas into one test period rather than running many tiny tests.
- Extend the test without changing conditions instead of adjusting the success criterion afterwards.
Mistakes that quietly ruin results
The most dangerous one is checking numbers daily and stopping the moment a variant pulls ahead. Early swings are normal, and stopping on them sharply raises the chance of a false conclusion. Fix the sample size or duration in advance and draw no conclusions before you reach it.
Ads add their own quirks. The algorithm needs a few days to learn, so early results are unstable. Run whole weeks so weekdays and weekends are represented equally. Avoid launching just before Navruz, Ramadan or a big sale, when demand jumps. And do not change several elements in one variant, or you will not know what worked.
For important decisions, use the built-in experiment tools in Meta Ads and Google Ads. They split the audience without overlap and report a confidence estimate, which a manual comparison of two overlapping groups with uneven budgets cannot match.
Write the test down
Before launch, record the hypothesis, variants, metric, minimum effect, calculated sample size, duration and budget. Afterwards, add the actual data, the conclusion and the decision. That log turns tests into accumulated knowledge instead of a pile of impressions.
If you want your tests to rest on real sales, talk to markdev about your task. We will look at your events, traffic and a realistic experiment plan together.
FAQ
Can I skip the calculator?
You can, but you risk stopping too early. A calculator is free, takes minutes, and forces you to decide in advance what size of difference matters.
What if the required sample is too large for my budget?
Raise the minimum effect and test bolder changes, use a more frequent event, or extend the duration. If none of that works, decide without a formal test and treat the choice as a hypothesis.
How long should an ad A/B test run?
Until you reach the calculated sample, and at least one full week to cover day-of-week differences. The exact length depends on your daily traffic.
Are the built-in Meta and Google tests enough?
For most decisions, yes: they split audiences properly and show confidence. You still need to choose the metric and sample size thoughtfully, and check the result against actual sales.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.