The retention notebook · Signup forms

How to Measure Popup Tests Through Orders and Revenue

Evaluate popup tests beyond submissions with fair audience definitions, mature order cohorts, net revenue per eligible visitor, incentive costs, and clear decision rules.

How to Measure Popup Tests Through Orders and Revenue decision map in the Senditive brand palette.
Senditivesenditive.com
Connect the decision, its evidence, and the work your store can sustain.
On this page

A popup variation can collect more email addresses and still produce a less useful commercial result. It may attract low-intent signups, change who sees the offer, or offer a discount that absorbs the extra revenue. A submit-rate winner answers only part of the question.

Measure the test from the opportunity to see the invitation through the outcome you want to improve. This guide focuses on experiment design, denominators, order follow-up, and commercial interpretation. It does not promise that a simple comparison can prove causal lift or statistical certainty without an appropriate test design.

Write the commercial hypothesis before building variants

Start with the change and why it could affect the customer decision. A clearer benefit might attract more appropriate subscribers; a smaller incentive might preserve contribution while maintaining useful growth. Describe the mechanism so the test has a purpose beyond producing a different-looking form.

Select a primary outcome and supporting measures. If the business question concerns first orders, a submit-rate improvement alone is insufficient. If the objective is permissioned list growth for a longer purchase cycle, immediate revenue may be an incomplete measure. The outcome should fit the decision rather than the report that is easiest to open.

Klaviyo's form A/B guidance recommends testing one variable at a time. Keep the audience, follow-up, and other material conditions comparable so the result is interpretable. Record the current version, the proposed change, the owner, and the evidence needed to decide. That small brief prevents a design experiment from quietly becoming a simultaneous test of audience, offer, timing, and welcome strategy.

Reference: A/B testing sign-up forms.

Choose the question before the variationChoose the question before the variation: Hypothesis — Name the change and customer mechanism.; Primary outcome — Select the measure that answers the decision.; Guardrails — Protect permission and the browsing experience.THE RETENTION NOTEBOOKChoose the question before the variation01HypothesisName the change andcustomer mechanism.02Primary outcomeSelect the measure thatanswers the decision.03GuardrailsProtect permission and thebrowsing experience.senditive.com
Choose the question before the variationChoose the question before the variation: Hypothesis — Name the change and customer mechanism.; Primary outcome — Select the measure that answers the decision.; Guardrails — Protect permission and the browsing experience.THE RETENTION NOTEBOOKChoose the question beforethe variation01HypothesisName the change and customermechanism.02Primary outcomeSelect the measure that answers thedecision.03GuardrailsProtect permission and the browsingexperience.senditive.com
Choose the question before the variation. Protect permission and the browsing experience.

Define the unit and population being compared

Decide what is assigned to each variation: a person, browser, session, or another supported unit. Record how the platform handles returning visitors and how you will avoid counting repeated events as independent people. Your outcome denominator must fit the unit the test actually uses.

For a timing test, eligible visitors and actual viewers can differ. A delayed form reaches only those who remain long enough; comparing only people who saw it can select a different audience. Klaviyo documents eligible unique views for display-time tests. Read the relevant report rather than assuming every form experiment has the same denominator.

Keep a record of eligibility rules, devices, pages, countries, and existing-subscriber exclusions. Do not expand one variation's audience while treating the result as a pure creative comparison. If assignment is not suitably controlled, describe the result as an observational comparison and acknowledge the competing explanation. A careful label is more useful than a confident causal claim that the design cannot support.

Reference: A/B testing sign-up forms.

Keep assignment and outcomes compatibleKeep assignment and outcomes compatible: Unit — Visitor, browser, or session as documented.; Population — Record eligibility and audience exclusions.; Follow-up — Use the same opportunity to observe outcomes.THE RETENTION NOTEBOOKKeep assignment and outcomes compatible01UnitVisitor, browser, or sessionas documented.02PopulationRecord eligibility andaudience exclusions.03Follow-upUse the same opportunityto observe outcomes.senditive.com
Keep assignment and outcomes compatibleKeep assignment and outcomes compatible: Unit — Visitor, browser, or session as documented.; Population — Record eligibility and audience exclusions.; Follow-up — Use the same opportunity to observe outcomes.THE RETENTION NOTEBOOKKeep assignment andoutcomes compatible01UnitVisitor, browser, or session asdocumented.02PopulationRecord eligibility and audienceexclusions.03Follow-upUse the same opportunity to observeoutcomes.senditive.com
Keep assignment and outcomes compatible. Use the same opportunity to observe outcomes.

Preserve the full funnel beside the headline rate

Track the stages separately: eligible or assigned visitors, actual exposure, submit action, intended channel subscription, confirmation where required, and later purchase. This view explains whether a change improved reach, response, or the handoff into a usable audience.

Check the platform's conversion definition. Klaviyo Overview uses total events, while current form A/B reports use unique metrics. Multi-step forms can record the first collected channel as a submit, and list double opt-in adds a confirmation requirement. A higher first-step action rate does not establish more confirmed email subscribers or completed SMS opt-ins.

Reference: Sign-up form analytics; Double opt-in.

Keep the full popup funnel visibleKeep the full popup funnel visible: Opportunity — Eligible or assigned visitors.; Subscription — The intended action and channel eligibility.; Purchase — Unique buyers and consistently defined revenue.THE RETENTION NOTEBOOKKeep the full popup funnel visible01OpportunityEligible or assignedvisitors.02SubscriptionThe intended action andchannel eligibility.03PurchaseUnique buyers andconsistently definedrevenue.senditive.com
Keep the full popup funnel visibleKeep the full popup funnel visible: Opportunity — Eligible or assigned visitors.; Subscription — The intended action and channel eligibility.; Purchase — Unique buyers and consistently defined revenue.THE RETENTION NOTEBOOKKeep the full popup funnelvisible01OpportunityEligible or assigned visitors.02SubscriptionThe intended action and channeleligibility.03PurchaseUnique buyers and consistentlydefined revenue.senditive.com
Keep the full popup funnel visible. Unique buyers and consistently defined revenue.

Make the variant and its follow-up identifiable

Plan how a later order will be connected to the visitor's assigned variation. Keep a stable test name and version, and preserve the information needed for the agreed cohort analysis. Decide how repeat exposure, resubscription, existing customers, and multiple forms will be handled before collecting outcomes.

When testing different offers, the correct welcome promise must follow the correct variation. Klaviyo documents using distinct hidden properties to identify offer versions and connect their follow-up. Test that routing end to end; otherwise you may be comparing mismatched promises rather than the intended incentive.

Minimize unnecessary data collection and restrict access to the people doing the analysis. The goal is a reproducible connection between assignment and outcome, not an elaborate profile of every visitor. If identification is incomplete, show that limitation. Do not silently drop the missing group and assume the remaining matched profiles represent everyone assigned to the test.

Reference: A/B testing form offers.

Connect the variant to the later outcome: Assign — Record the intended variation and population.; Identify — Preserve the useful variant-to-profile connection.; Follow — Verify the matching promise and order window.
Connect the variant to the later outcome. Verify the matching promise and order window.

Calculate visitor outcomes and net economics

Report unique buyers per eligible or assigned visitor alongside buyers per subscriber. The subscriber-only rate describes a selected group created after the tested invitation. It can be useful diagnostically, but it should not replace the broader visitor outcome when the decision concerns the effect of offering the form.

Define net order revenue consistently. Decide how discounts, refunds, shipping, taxes, and currency are treated, and use the same rule for both variations. Revenue per eligible visitor divides that defined revenue by the comparable visitor count. It keeps people who did not submit in the analysis instead of rewarding only the post-signup subset.

For a commercial decision, add contribution assumptions where available. Do not subtract incentive cost twice if the revenue or contribution figure already accounts for it. The calculator shows observed rates and revenue per visitor from the inputs you provide. It does not calculate statistical significance or establish how much of the difference was caused by the form.

A free consultation can help you discuss the cohort definition and commercial measures for the next popup experiment.

Put it into practice

Build a small metric dictionary before calculating a winner. For each input, name its source, unit, dates, exclusions, and treatment of returns. State whether buyers are unique people and whether the visitor denominator represents assignment, eligibility, or actual exposure. If two exports use different windows, reconcile them before combining the counts. Store the dictionary with the test brief and keep the same definitions for both variations. Another reviewer should be able to reproduce the result from the inputs without relying on the original analyst to explain an undocumented exception.

Use visitor-level commercial measuresUse visitor-level commercial measures: Buyer rate — Unique buyers divided by comparable visitors.; Revenue per visitor — Consistent net revenue over that denominator.; Economics — Keep discounts, returns, and costs coherent.THE RETENTION NOTEBOOKUse visitor-level commercial measures01Buyer rateUnique buyers divided bycomparable visitors.02Revenue per visitorConsistent net revenueover that denominator.03EconomicsKeep discounts, returns,and costs coherent.senditive.com
Use visitor-level commercial measuresUse visitor-level commercial measures: Buyer rate — Unique buyers divided by comparable visitors.; Revenue per visitor — Consistent net revenue over that denominator.; Economics — Keep discounts, returns, and costs coherent.THE RETENTION NOTEBOOKUse visitor-level commercialmeasures01Buyer rateUnique buyers divided by comparablevisitors.02Revenue per visitorConsistent net revenue over thatdenominator.03EconomicsKeep discounts, returns, and costscoherent.senditive.com
Use visitor-level commercial measures. Keep discounts, returns, and costs coherent.

Give both groups the same opportunity to mature

Choose an observation window appropriate to the purchase decision and apply it consistently. A person assigned on the final day needs the same follow-up opportunity as someone assigned earlier. Separate recruitment dates from outcome follow-up dates so the latest group is not judged prematurely.

Account for refunds and cancellations if those can materially change the decision. An early order count can be a useful operating signal, while a later net-revenue view answers a different commercial question. Name the review points rather than quietly replacing the initial measure when another number looks better.

Keep seasonality and external changes in the test record. Inventory, pricing, advertising, shipping policy, and site changes can alter purchasing during the experiment. A suitably controlled comparison helps, but a broken product destination or uneven audience change still needs investigation. Do not extend the test indefinitely without checking that its original conditions and customer purpose remain valid.

Give both groups the same time to matureGive both groups the same time to mature: Recruitment — Record when each person enters the test.; Outcome window — Allow equal follow-up after entry.; Later adjustments — Define returns and cancellations consistently.THE RETENTION NOTEBOOKGive both groups the same time to mature01RecruitmentRecord when each personenters the test.02Outcome windowAllow equal follow-up afterentry.03Later adjustmentsDefine returns andcancellations consistently.senditive.com
Give both groups the same time to matureGive both groups the same time to mature: Recruitment — Record when each person enters the test.; Outcome window — Allow equal follow-up after entry.; Later adjustments — Define returns and cancellations consistently.THE RETENTION NOTEBOOKGive both groups the sametime to mature01RecruitmentRecord when each person enters thetest.02Outcome windowAllow equal follow-up after entry.03Later adjustmentsDefine returns and cancellationsconsistently.senditive.com
Give both groups the same time to mature. Define returns and cancellations consistently.

Understand what form revenue attribution actually means

Klaviyo links form revenue using a configurable lookback window and a choice about submitted or completed form events. The documentation also describes identification requirements. That linked revenue is a useful platform view, but its meaning depends on the selected settings.

Form revenue and message-attributed revenue are separate. The same purchase can be associated with a signup form and a later email under their respective rules. Do not add those figures together as though they were separate orders. Reconcile the order and cohort definitions before using several dashboards in a test report.

Reference: Revenue metrics for sign-up forms; Message attribution.

Do not add overlapping attribution creditDo not add overlapping attribution credit: Form view — A lookback links relevant signup events.; Message view — Different rules credit a later communication.; Order view — Reconcile the same underlying purchase.THE RETENTION NOTEBOOKDo not add overlapping attribution credit01Form viewA lookback links relevantsignup events.02Message viewDifferent rules credit a latercommunication.03Order viewReconcile the sameunderlying purchase.senditive.com
Do not add overlapping attribution creditDo not add overlapping attribution credit: Form view — A lookback links relevant signup events.; Message view — Different rules credit a later communication.; Order view — Reconcile the same underlying purchase.THE RETENTION NOTEBOOKDo not add overlappingattribution credit01Form viewA lookback links relevant signupevents.02Message viewDifferent rules credit a latercommunication.03Order viewReconcile the same underlyingpurchase.senditive.com
Do not add overlapping attribution credit. Reconcile the same underlying purchase.

Review uncertainty before calling a winner

Look at counts, allocation, maturity, and the size of the observed difference. A few orders can move revenue substantially in a small sample, especially with varied order values. Showing more decimal places does not make the difference dependable.

Use an appropriate experimentation method when the decision needs statistical confidence. Consider the unit of assignment, variance, planned sample, stopping rule, and the effect size worth acting on. A simple percentage comparison or a revenue-per-visitor calculator does not perform that work. Keep the confidence claim proportional to the analysis actually completed.

Examine conflicting outcomes. More signups with lower revenue per visitor may indicate a commercial tradeoff or an immature result. More revenue with materially higher complaints or an obstructive browsing experience may also be unsuitable. The decision triage tool helps identify what needs investigation next; it deliberately does not declare a statistically proven winner from incomplete evidence.

Review the evidence before rollout: Validate — Verify the population and metric definitions.; Mature — Allow the agreed outcome window.; Decide — Review uncertainty, economics, and guardrails.
Review the evidence before rollout. Review uncertainty, economics, and guardrails.

Keep experience and permission as guardrails

A test can increase a short-term response by making the invitation more intrusive or the promise less clear. Include practical guardrails for the experience: usable dismissal, mobile controls, understandable terms, and appropriate channel choice. Do not treat a forced or confusing interaction as desirable growth.

Review the resulting subscription path and early negative signals. A form may produce more initial actions while creating more unconfirmed signups, immediate unsubscribes, or complaints. These outcomes help explain whether the invitation attracted the audience you intended and whether the promise remained recognizable after signup.

Agree what requires stopping or repairing the test. A broken close control, wrong offer, missing confirmation, or incorrect follow-up is an operational failure, not merely a weak variation. Fix the issue, document the affected period, and decide whether its data can still answer the original question. Keep the customer's ability to choose intact throughout the experiment.

Reference: Collecting contact information.

An operational fault is not a weak variationAn operational fault is not a weak variation: Usability — The visitor can browse and dismiss normally.; Permission — Channel choices and terms remain clear.; Promise — The right offer reaches the right person.THE RETENTION NOTEBOOKAn operational fault is not a weak variation01UsabilityThe visitor can browse anddismiss normally.02PermissionChannel choices and termsremain clear.03PromiseThe right offer reaches theright person.senditive.com
An operational fault is not a weak variationAn operational fault is not a weak variation: Usability — The visitor can browse and dismiss normally.; Permission — Channel choices and terms remain clear.; Promise — The right offer reaches the right person.THE RETENTION NOTEBOOKAn operational fault is not aweak variation01UsabilityThe visitor can browse and dismissnormally.02PermissionChannel choices and terms remainclear.03PromiseThe right offer reaches the rightperson.senditive.com
An operational fault is not a weak variation. The right offer reaches the right person.

Record the decision and the next controlled change

End with a short decision record: hypothesis, audience, assignment, dates, metric definitions, observed outcomes, uncertainty, guardrails, and action. Keep the rejected or inconclusive version documented so the next team does not unknowingly repeat the same question.

Roll out the selected change with a check that the production form and welcome path match the tested version. Continue monitoring the key measures; a result can change when traffic or offers change. If the evidence is inconclusive, choose a narrower test, improve measurement, or retain the current version rather than inventing certainty.

A free consultation with Senditive can help you discuss the next signup experiment and a personalized roadmap for forms and follow-up. Good testing produces a better decision, even when the answer is to repair the setup or collect more comparable evidence. Treat orders and revenue as part of a well-defined customer experiment, not as decorations added after a submit-rate contest.

Finish with a reproducible decision recordFinish with a reproducible decision record: Design — Hypothesis, assignment, dates, and measures.; Evidence — Counts, outcomes, uncertainty, and guardrails.; Action — Roll out, repair, or plan a narrower question.THE RETENTION NOTEBOOKFinish with a reproducible decision record01DesignHypothesis, assignment,dates, and measures.02EvidenceCounts, outcomes,uncertainty, andguardrails.03ActionRoll out, repair, or plan anarrower question.senditive.com
Finish with a reproducible decision recordFinish with a reproducible decision record: Design — Hypothesis, assignment, dates, and measures.; Evidence — Counts, outcomes, uncertainty, and guardrails.; Action — Roll out, repair, or plan a narrower question.THE RETENTION NOTEBOOKFinish with a reproducibledecision record01DesignHypothesis, assignment, dates, andmeasures.02EvidenceCounts, outcomes, uncertainty, andguardrails.03ActionRoll out, repair, or plan a narrowerquestion.senditive.com
Finish with a reproducible decision record. Roll out, repair, or plan a narrower question.

Your questions, answered

Should I choose the variation with the highest submit rate?

Only if that measure answers the agreed question and relevant guardrails hold. For a commercial test, compare mature visitor-level order and net-revenue outcomes as well. More submissions can come with different audience quality or incentive economics.

Can I add form revenue and welcome email revenue together?

Those reporting views can associate the same order under different rules. Do not sum them as separate sales. Reconcile the underlying order and cohort definitions, and explain each attribution window.

Why use eligible visitors instead of only people who saw a delayed popup?

Actual viewers of a delayed popup are people who remained long enough to see it. An eligible-population denominator helps avoid comparing only that selected group with everyone who saw an immediate form. Check the test platform’s specific definition.

Does the calculator prove a statistically significant winner?

No. It calculates observed rates and revenue per visitor from compatible counts. Statistical confidence and causal interpretation need an appropriate assignment method, sample, stopping plan, and analysis. Use the output as a descriptive decision aid.

Choose your next improvement

Measure the invitation, the audience, and the commercial outcome with definitions that stay consistent. A useful result explains both what changed and what the evidence can support.

Sources and further reading
  1. Klaviyo: A/B testing sign-up forms
  2. Klaviyo: Sign-up form analytics
  3. Klaviyo: Double opt-in
  4. Klaviyo: A/B testing form offers
  5. Klaviyo: Revenue metrics for sign-up forms
  6. Klaviyo: Message attribution
  7. Shopify: Collecting contact information
Back to the top ↑
Book a free consultation