I’m pretty AI-pilled. Claude does most of my website updates now: headlines, landing pages, Woo order updates, bugfixes. Its so fast, but that created a new problem. Changing things got so easy that I was shipping update after update without knowing whether any of them helped or hurt conversions.

So I wrote two open-source agent skills that teach the AI when a change needs to be measured and what to measure.

prove-website-changes Stops AI from blindly publishing changes that might affect conversions. It proposes an A/B test instead.

website-optimisation-loop - Actively finds opportunities using reviews, analytics, best practice and heatmaps, runs experiments, and learns from the results.

Both are plain Markdown and MIT licensed. They work in Claude Code, Hermes, Codex, Grok Bot or anything that loads skills.

https://github.com/SiteSpot/human-web-feedback-loop

Disclosure: I make AB Split Test, and it’s the default testing tool in the skills. The free works fine for this. If you already use Optimizely, VWO, PostHog, GrowthBook or similar, connect its MCP server and the skills will use that instead.

Anyone else letting AI make changes to their live sites? How are you checking that it’s actually helping?