Transparency first

Single-point visibility scores can mislead when AI answers flip run-to-run. We are running a controlled repeat-prompt study (Phase 2.2) before showing confidence bands in the product. Until then, treat scores as directional — not lab-grade point estimates.

What we measure#

Exact mention — your brand string appears in the answer text.
Semantic mention — embedding similarity catches paraphrases and descriptions (upgraded after each sweep via a reanalysis pass).
Citation — the model links to a URL we classify as owned, competitor, or third-party.
Used but not cited — for engines that expose retrieval metadata (Claude, Perplexity), URLs your site appeared in internal search results but were not linked in the final answer.

Prompt panel discipline#

Prompts are not silently rewritten in place. When you adopt a GSC-grounded replacement, we deactivate the old prompt and start a new row at panel_version + 1 with lineage preserved. Historical responses stay on the version they were measured under.

Engines & grounding#

Paid monitors run grounded(live web-search) calls where the provider supports it. Free monitors fall back to ungrounded completions — cheaper, but not comparable to grounded scores. Opt-in engines (Google AI Overviews, Google AI Mode via DataForSEO, Grok, DeepSeek) are toggled per monitor and respect your plan's engine cap.

Google AI Mode is English-only at our DataForSEO provider; we force en and record that in response metadata.

Visibility score composition#

Daily scores aggregate prompt-level responses into monitor-level percentages: overall visibility (any mention or citation), exact vs semantic mention rates, citation rate, share-of-voice vs competitors named in the answer, and average citation rank when cited.

Variance study & error bars (in progress)#

Our Phase 2.2 design repeats ~10 prompts × default engines × 10 runs over 48 hours on the dogfood monitor to measure flip rates and induced confidence intervals. Cost is estimated with the same provider-costs.ts table the engine uses — run the dry-run planner:

node --experimental-strip-types scripts/llm-visibility-variance-study.mts

After the study completes, we will publish data study #1and ship ± bands on the dashboard ("±X pts per weekly sweep"). The extraction script output will be committed as a frozen artifact.

Detection quality baseline#

We maintain a stratified hand-label set (~200 archived responses) under tests/fixtures/llm-response-detection-labels/ for mention-detection precision/recall regression tests. Labels are human-reviewed — not auto-generated ground truth.

What we do not claim#

Citation-outcome buckets and citation-likelihood scores are correlational until intervention data validates them.
Microsoft Copilot has no public answer-retrieval API — we do not scrape UI automation.
Third-party engine APIs change shape; fail-open paths may show empty results until we adapt parsers.

Questions? See also geo-tools comparison for engine coverage honesty notes.

AI visibility methodology