Measurement

How to prove a structured data change actually did something

In short

Take a baseline before you change anything, because you cannot reconstruct one afterwards. Then measure three things: whether the markup validates and resolves, whether AI crawlers are fetching the changed pages in your server logs, and whether you are cited across a fixed set of customer questions asked repeatedly. Ask each question at least ten times — model outputs vary, so a single before-and-after screenshot proves nothing.

The awkward thing about this discipline is that the measurement is harder than the work. Adding a connected graph to a template takes a day. Proving it changed anything is the part people get wrong, usually by accepting evidence that would not survive being questioned by a finance director.

Take the baseline first

This is the one irreversible mistake. You cannot construct a before position after you have shipped the change, and almost everyone tries.

Before touching anything, capture: the raw HTML of your key page types, your current server log sample showing AI crawler hits, and the answers you currently get to a fixed list of customer questions. Store them somewhere dated. It takes an hour and it is the difference between a claim and a demonstration.

Three things worth measuring

1. Does it validate and resolve

The cheapest check, and a genuine gate. Not just “is the JSON valid” but “does the graph resolve” — every @id referenced by one node should exist as another node on the page. Dangling references are the most common defect we find, and they are invisible to most validators because each block is individually legal.

This is worth automating in CI. A structured data error should fail a build the same way a type error does, because the failure mode otherwise is silent and nobody notices for a quarter.

2. Are the crawlers actually fetching

The most underrated signal, and it sits in logs you already have.

grep -iE 'GPTBot|ClaudeBot|PerplexityBot|Google-Extended' access.log \
  | awk '{print $9}' | sort | uniq -c

Status codes by crawler, before and after. If you removed a block and the 403s became 200s, that is not an inference — that is your own server recording the change. It is the most defensible number in the whole exercise, and it is the one people forget to collect.

3. Are you cited, across repeated asking

Here is where most reporting falls apart. Model outputs are not deterministic: ask the same question twice and you can get two different sets of sources. A screenshot of one good answer is not evidence, and neither is one bad one.

Fix a list of ten to twenty questions a real customer would ask. Ask each at least ten times, across the assistants you care about. Record whether your domain appears at all, and in what position among the sources. You are looking for a shift in a rate — cited in three runs out of ten, then six out of ten — not for a single answer changing.

That is a tedious thing to do by hand, which is why it tends to get skipped or replaced with an anecdote.

What you cannot claim

Some honesty about the limits, because overclaiming here is how this field loses credibility.

You cannot attribute a revenue change to a markup change. Too much moves at once, there is no controlled comparison available, and anyone offering you that attribution is guessing with confidence.

You cannot prove causation from a single site. What you can show is a chain: the crawler was blocked and now is not, the page had no resolvable entity and now has one, and citation rate across a fixed question set moved from x to y. That is a strong, checkable argument. It is not a randomised trial and should not be dressed as one.

You also cannot assume it holds. Answer engines change their retrieval and their crawler behaviour without announcement, so a measurement taken in March may not describe June. Re-run the question set on a schedule rather than treating the first result as settled.

The version that survives scrutiny

If you want one paragraph to put in front of a board, it looks like this: before, three of the four major AI crawlers received a 403 from our CDN and no page carried a resolvable entity. After, all four receive a 200, every page type emits one connected graph, and across twenty customer questions asked ten times each, our domain appears as a source in a materially higher share of answers than at baseline. Here are the logs and the raw outputs.

That is provable, it is checkable by someone hostile to it, and it does not require anyone to trust the person presenting it.