With AI crawlers and agents hitting websites constantly, how much of what analytics calls “direct traffic” is even human anymore?

If bots are getting counted as visits, conversion rates automatically look worse. And then decisions about landing pages, content, ad spend etc. are being made from numbers with a messed up denominator.

Blocking bots doesn't really solve it either. Some are obviously junk, but others are search crawlers, monitoring tools or AI agents that might actually send users your way.

Feels like analytics needs to get much better at telling why something is accessing a site, not just whether it looks like a bot.

Anyone seeing this noticeably mess with their analytics yet?

Been going back and forth on this for a while and I don't think I've landed anywhere good.

The situation is the usual one. App talks to a handful of external APIs. Some have sandbox environments, some don't. The ones that do have sandboxes that don't quite match prod, and the gap is never documented anywhere, you just find it eventually.

Options as far as I can tell:

Hit the real sandbox in tests. Slow, rate limited, and you can't run it in CI on every PR without burning through quota. The sandbox data is also usually three fake records, so pagination bugs and anything volume related never show up until a real customer hits them.

Record and replay, VCR style cassettes. Works great the day you record them. Six months later nobody remembers how to re-record, a chunk of them are for endpoints that have since changed, and the suite is green the entire time.

Hand-written mocks. This is what most places I've worked have done and I've slowly come around to thinking it's the worst of the three. You write the mock from your reading of the docs, so the mock encodes your misunderstanding of the API, and then it passes forever. The test isn't checking that your code works against the API, it's checking that your code works against your idea of the API, and those two drift apart silently.

Contract testing, pact and friends. Makes sense when both ends are yours. When the other end is Slack, nobody is publishing a contract for you.

So what do teams actually do in practice? Specifically curious about:

- whether anyone has anything that detects when the real API has drifted away from whatever you're testing against, or if you just find out from a bug report

- whether you bother making mock data realistic in volume, or accept that pagination and perf issues are things you'll only ever see in prod

- vendors that genuinely do sandboxes well, if any exist

Not looking for "just use msw" or a list of tools, I know they're out there. I'm asking what your team actually settled on and whether you'd do it that way again.