Analytics and Tracking

Bot Filtering

Also called Crawler exclusion, spider filtering

Removing traffic from known crawlers and automated scripts so reports describe people rather than machines.

Quick facts: Bot Filtering

Category
Analytics and Tracking
Also called
Crawler exclusion, spider filtering
Level
Intermediate
Affects
Session counts, conversion rate, remarketing audiences, test results
Where to see it
GA4 data filters, hostname conditions, server logs, CDN rules
In this article4
  1. How bot filtering works
  2. Why bot filtering matters
  3. Where bot filtering goes wrong
  4. What to do about it

How bot filtering works

A large share of requests to any website come from software rather than people: search engine crawlers, uptime monitors, SEO tools, screenshot services, security scanners and scrapers. Bot filtering is the process of keeping that activity out of the numbers you use to make decisions.

Google Analytics 4 does part of the job for you. Traffic from crawlers and spiders on a widely used industry list is excluded automatically, and there is no switch to turn it off. That list works by recognising agents that identify themselves honestly, which covers the well-behaved end of the spectrum. The badly behaved end — scrapers pretending to be a normal browser, headless browsers driven by scripts, click farms — is not on any list, so it arrives looking exactly like a visitor and has to be spotted by its behaviour instead.

Why bot filtering matters

Automated traffic distorts everything downstream. Sessions rise while conversion rate falls, so a page can appear to be failing when it is performing perfectly for the humans who reach it. Engagement metrics sink, landing page tests reach the wrong conclusions, and a sudden spike gets celebrated in a report as a marketing success it never was.

It matters for spend as well as for reporting. Remarketing audiences built from polluted sessions get filled with machines you can never sell to, so budget is spent showing ads to nobody. And when the noise arrives during a launch or a campaign, it is exactly the moment you most need the figures to be trustworthy.

Where bot filtering goes wrong

The first mistake is assuming the automatic list is complete. It removes known crawlers, not all non-human traffic, so a clean-looking property can still be carrying scrapers and script traffic.

The second is confusing bot traffic with ghost spam, which never touches your site at all and is injected directly into your measurement. Filtering by user agent does nothing against it; restricting reports to your real hostname does.

The third is over-correcting. Aggressive filters applied by a developer or a security plugin can block legitimate crawlers, including the ones search engines need, or exclude a country’s traffic wholesale because a few bad sessions came from it. Losing real customers to reduce noise in a report is a bad trade.

What to do about it

Learn the shape of the problem before changing anything. Look for sessions with no engagement, a single page, an unusual browser or operating system version, an unfamiliar hostname, or a sudden run of traffic from one city at odd hours. Compare against your server logs and your own sales records, which bots cannot fake.

Then act where the traffic actually is. Add a hostname condition so only your real domains report data, use internal traffic filters for your own team and suppliers, and handle scrapers and abusive scripts at the hosting or CDN layer rather than in analytics, where blocking them also saves bandwidth. Keep a note of every filter you apply and the date, because data removed by a filter does not come back, and an unexplained drop six months later is nearly always somebody’s forgotten rule.

Do and do not

Do

  • Judge suspicious traffic by behaviour, not volume
  • Restrict reporting to your own hostnames
  • Record every filter you add and the date

Do not

  • Assume automatic filtering catches every bot
  • Block search engine crawlers while chasing scrapers
  • Exclude a whole country because of a few sessions

Questions people ask about this

Does GA4 remove all bot traffic automatically?

It removes traffic from crawlers and spiders that appear on a recognised industry list, and that happens automatically with no setting to change. It does not remove automated traffic that disguises itself as an ordinary browser, or spam injected directly into your property without visiting the site. Those still need spotting by behaviour and handling separately.

How can I tell if a traffic spike is bots?

Look at how the sessions behave rather than how many there are. Machine traffic usually shows almost no engagement time, a single page per session, an unusual browser or operating system, and a burst pattern at odd hours from one location. Cross-check against your server logs and your actual enquiries or orders, which cannot be faked.

Should I block bots at the server instead of in analytics?

For abusive scrapers and scanners, yes, because blocking at the hosting or CDN layer stops the load as well as the noise. Keep it targeted: search engine crawlers must stay welcome, or your pages stop being indexed. Analytics filters are for cleaning reports, not for defending the site.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.