One AI check against twelve
Suppose you ask ChatGPT whether it recommends your company, once, and it does not name you. What have you learned? On this corpus, not much. In the prompt-months where ChatGPT named a brand at least once in 12 identical runs, a single run named it 45.9% of the time. This page is what twelve runs show that one cannot, on both engines, and how many runs a check needs before its number means anything.
One ChatGPT check can show that a brand is in play; a check that does not show it proves little. Where 12 runs ever named the brand, one run named it 45.9% of the time, all 12 agreed in 4.7% of cases, and a single check gave the minority answer 21.7% of the time. Perplexity is far steadier inside a month (87.1% and 75.1%), but its repeats are not independent draws. At 12 runs a per-prompt rate carries a margin of plus or minus 17.0 to 24.6 points.
What one run shows
ChatGPT (gpt-5-mini, web search on) named the brand at least once in 211 of its 517 prompt-months, 138 distinct prompts. In those, it named the brand in all 12 runs 10 times (4.7%, interval 1.8 to 8.2), a single run gave the minority answer 21.7% of the time on average (19.8 to 23.7; 23.9% leave-one-out, each run scored against the majority of the other 11), and in 104 prompt-months (49.3%, interval 42.0 to 56.4) it disagreed at least one time in four. Over all 517 ChatGPT prompt-months, never-named ones included, the mean minority share is 8.9%.
Single-check detection, the share of runs that named the brand where 12 ever did, is 45.9% on ChatGPT (40.8 to 51.2; September 46.7%, October 45.2%) and 87.1% on Perplexity (82.9 to 91.0). 54.5% of ChatGPT's ever-named prompt-months have a named rate under one half, and 41 of them are named in exactly 1 run of 12. On Perplexity a single run gave the minority answer 5.7% of the time (3.9 to 7.6; 6.0% leave-one-out).
Table view: named and cited, by engine
| Field | Engine | Ever (prompt-months) | Never | Always | Unstable, % of ever (n) | One check wrong at least 25% of the time, % of ever (n) | Wrong at least 25%, % of all 517 | Mean chance one check is wrong, % (ever) |
|---|---|---|---|---|---|---|---|---|
| Named | ChatGPT | 211 | 306 | 10 | 95.3% (201) | 49.3% (104) | 20.1% | 21.7% |
| Named | Perplexity | 169 | 348 | 127 | 24.9% (42) | 13.0% (22) | 4.3% | 5.7% |
| Cited | ChatGPT | 192 | 325 | 6 | 96.9% (186) | 50.5% (97) | 18.8% | 22.6% |
| Cited | Perplexity | 171 | 346 | 151 | 11.7% (20) | 8.8% (15) | 2.9% | 3.1% |
A cell is one prompt in one month on one engine, 12 runs. Named: a brand alias appears anywhere in the response, source titles and link markup included. Cited: a brand domain appears in the sources. Unstable: named or cited in some runs but not all. Minority: the share of the 12 runs that disagree with the majority.
The tracker's named flag counts source titles and link markup as well as prose. On answer prose only, the ChatGPT result holds: 9 of 188 ever-named prompt-months always named, a mean 22.5% chance that one check is wrong, and 53.2% of prompt-months wrong at least one time in four. Perplexity on prose only: 110 of 156 (70.5%) always named. Dropping the prompt-months where the brand is typed into the prompt leaves ChatGPT at 8 of 208.
Zero of twelve is not absence
Twelve repeats usually separate a brand that is named often from one that is rarely or never named, but 0 of 12 does not prove absence. Of the 104 matched prompts that scored 0 of 12 on ChatGPT in September, 13 named the brand in October, 10 of them in 1 run of 12; going the other way, 9 of 100 prompts at zero in October had been named in September. The arithmetic says the same: a brand named in one run of ten is missed by all 12 runs 28% of the time.
Instability is not just whether ChatGPT searched
32.7% of ChatGPT runs returned no sources, and only 161 of 517 ChatGPT prompt-months had sources on every run. Runs with no sources named the brand 0.5% of the time, against 27.6% when sources came back, so searching matters. It does not explain the variation. Among the 84 ever-named ChatGPT prompt-months where every one of the 12 runs returned sources, 76 are still unstable and 8 always named. Among prompt-months with at least 6 sourced runs (200, judged on all 12 runs), 190 are unstable, the mean minority is 22.1%, and 50.5% disagree at least one time in four. On the cited side, 186 of 192 ever-cited ChatGPT prompt-months are unstable, and 174 of those remain unstable when counted only over runs that returned sources.
Perplexity's twelve runs may be close to one observation
Perplexity (sonar) is far steadier inside one fill. In each month separately, 55 of 80 ever-named prompt-months were always named in September and 72 of 89 in October, against ChatGPT's 5 of 99 and 5 of 112. Its first 10 source hosts were identical in 51.3% of September repeat pairs (41.6% across all 20 sources, 38.2% on the last 10) and 83.6% of October's, when the API returned 10 sources instead of 20: mean pairwise Jaccard 0.840 over 18,084 September pairs and 0.951 over 16,038 October pairs. ChatGPT's source sets were identical in 1.95% of 19,598 pairs, with a median Jaccard of 0.250 (interquartile range 0.133 to 0.385, mean 0.280).
Perplexity's repeats are not independent draws, though. In its unstable prompt-months the runs that named the brand came earlier in the sequence (mean run order 6.09 against 6.88; within-cell permutation p = 0.003 over 42 cells, earlier in 66.7% of them), and 77.7% of October repeat pairs returned the same ordered URL list (27.8% in September). ChatGPT shows no such order effect (6.56 against 6.45, p = 0.258, 201 cells). So Perplexity's 12 repeats inside a window may be close to one observation, and its steadiness is reported here only as the per-month range above, not as a rate. We did not measure how either engine changes across weeks.
Competitor lists
| Engine | Prompt-months | Median distinct competitors over 12 runs (range) | Mean pairwise Jaccard, all pairs | Both runs named someone |
|---|---|---|---|---|
| ChatGPT | 410 | 6 (1 to 21) | 0.364 | 0.53 |
| Perplexity | 394 | 3 (1 to 15) | 0.801 | 0.87 |
Both-named pairs: 383 ChatGPT prompt-months and 386 Perplexity. Competitor lists are configured per site, 8 to 66 names, so set sizes are not comparable across sites. A run that named no competitor counts as an empty set in the all-pairs figure, so part of ChatGPT's gap is runs that named nobody, not runs that named someone else.
ChatGPT's competitor lists vary from run to run and Perplexity's mostly do not. Directional, because of the empty-set effect in the caption and the uneven list sizes.
How many repeats a check needs
If each repeat were an independent draw, a per-prompt 95% interval at 12 runs would be plus or minus 17.0 to 24.6 points, depending on the rate. Getting it under plus or minus 15 points takes 17 to 39 repeats for a true rate between 10% and 90%, fewer nearer 0 or 100%; at a rate of one half it is 43 by the normal approximation and 39 by Wilson. For Perplexity, whose repeats are not independent, that understates the count. Twelve repeats are enough to tell often from rarely; they are not enough to measure one prompt's rate.
What a 12-run test can detect month to month
Month to month, a 12-run test cannot read a change in one prompt's ChatGPT named rate as real unless it is at least 33 to 42 points (the smaller figure from a rate of zero, the larger from one half). 76 of 84 matched prompts whose ChatGPT named rate changed between the two fills moved by less than that. This measures the test's power, not stability: it says what the test can see, not that the rates held still. The model snapshot was the same in both months.
Method notes
0 prompt-months were dropped. The 12 answers in a prompt-month differ in text in 100% of ChatGPT prompt-months and 99.6% of Perplexity's, so these are not cached responses. Four of the 12 panels hold 150 of the 211 ever-named ChatGPT prompt-months, which is why intervals are clustered by prompt rather than by run. Fill windows, models and the rest of the instrument are on the method page. The figure that started this study, the share of ChatGPT answers naming nobody, is on belief against measurement.
Running your own repeats? Tell us the prompt and we will say whether it is in the November panel.
Quoting a figure? Link to this page as the source: https://www.bestseopodcast.com/research/one-ai-check-vs-twelve. Figures are dated 2026-10-25 and cover answers collected 12 to 26 September 2026 and 1 to 5 October 2026; the next refresh is after the November 2026 fill closes, about 15 November. The page is the citation: there is no download.
Who did this
Matthew Bertram runs EWR Digital, the SEO and AI-visibility agency whose monthly tracker produced this corpus, and he is the author of LLM Visibility: A Decision-Grade System for Winning AI-Mediated Discovery. The studies were built, re-derived from counts and reviewed before anything was written up; every figure on this page traces to a script in the project record. Questions about the method go to him through the contact page. More about Matthew Bertram.
Checked once and saw nothing?
The audit runs the question the way this study did: repeated, on both engines, with the sources kept.
Get a free AI visibility audit