Analytics and Tracking

Segmentation of Results

Also called Segmented analysis, subgroup analysis

Splitting a test result by device, channel or audience to see whether the effect holds in every group.

Quick facts: Segmentation of Results

Category
Analytics and Tracking
Also called
Segmented analysis, subgroup analysis
Level
Intermediate
Affects
Test interpretation, roll-out scope, follow-up tests
Where to see it
GA4 explorations and comparisons, A/B testing platforms, Looker Studio
In this article4
  1. What segmentation of results measures
  2. Why segmentation of results matters
  3. Common mistakes with segmentation of results
  4. What to do about it

What segmentation of results measures

An overall result is an average of everyone who took part, and averages hide disagreement. Segmentation splits the same data by something you already know about the visitor — device, traffic source, country, new against returning, logged in against anonymous — and asks whether the effect was the same in each group.

Sometimes the answer is dull and reassuring: the change helped everyone by roughly the same amount, so the average tells the whole story. Sometimes it is not. A shorter form can lift mobile conversions while doing nothing on desktop, where the longer version was never the obstacle. A reworked page can win on search traffic and lose on paid traffic, because those visitors arrive with different expectations.

The important discipline is that segmentation reuses data collected for one comparison. Each split makes the groups smaller and the numbers noisier, so a segment result is always weaker evidence than the headline it came from.

Why segmentation of results matters

It stops a flat overall result being thrown away. A test that looks like nothing is quite often a clear win in one group cancelled by a clear loss in another, and the honest response is to serve the two groups differently rather than to conclude the idea failed.

In Nepal and similar markets the device split does most of the work, because traffic skews heavily towards mobile phones on variable connections. A desktop-flattering result that quietly fails on mobile is failing for most of the audience. Channel splits matter for the same reason: visitors from a conversion funnel that starts with an ad have already been sold something, and they read a page differently from someone arriving cold from search.

Common mistakes with segmentation of results

The big one is cutting the data until something looks impressive. Slice by enough dimensions and some segment will always appear to be a winner purely by chance, and reporting that slice as the finding is a way of manufacturing results rather than discovering them.

The second is acting on segments too small to mean anything. When a split leaves only a scattering of conversions on each side, the difference between the groups is mostly noise, and the range around it will be wide enough to include no effect at all.

The third is treating a segment discovery as proven. A split you did not plan is a hypothesis, not a conclusion. It earns a follow-up test aimed at that audience; it does not earn an immediate site-wide decision.

What to do about it

Decide before launch which two or three splits you will look at and why, and write them down. Device and traffic source are almost always worth it; long lists of demographic cuts rarely are. Naming them in advance is what separates analysis from fishing.

When you do find a difference, check the size of the groups and the range around each figure before you believe it, then ask whether the pattern makes sense as a story about behaviour. A mobile-only lift on a change that removes typing is plausible. A lift confined to one browser usually means a rendering bug worth investigating rather than an audience insight. Feed real differences back as the next test, and keep the site-wide decision anchored to the overall result.

Do and do not

Do

  • Name the splits you will check before launching
  • Always look at mobile separately from desktop
  • Confirm a segment finding with a follow-up test

Do not

  • Cut the data until something looks impressive
  • Act on segments with only a handful of conversions
  • Let a segment override the overall roll-out decision

Questions people ask about this

Which segments are worth checking on a test?

Device and traffic source cover most of what matters, because they change both the experience and the visitor's intent. New against returning is worth adding when the change is visually obvious. Beyond that, only split by something you have a genuine reason to expect will behave differently, and decide the list before the test launches.

Can I roll out a change for just one segment?

You can, and sometimes you should, but treat the segment finding as a hypothesis first. Run a follow-up test aimed at that audience to confirm the effect is real before committing to two versions. Maintaining separate experiences has an ongoing cost, so it needs stronger evidence than a split discovered while reading a single result.

Why do segment results look less reliable than the overall result?

Because splitting the data makes each group smaller. Fewer visitors and fewer conversions mean more random variation, so the plausible range around each segment figure is wider than the range around the headline. The more ways you cut the same data, the more likely it becomes that one slice looks striking by chance alone.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.