Experiment report · 2026-10-14

Can an agent find a site's search on its own?

One signal per version, nothing else, served by a Worker that adds nothing of its own. 78 AI agents were each given one version and one instruction: find out how to search it, then search for "ferrari". Nobody told them what to look for.

78agent runs
13versions, t00–t11 + t071
3agent harnesses
63/72found it through the signal
0/6found it with no signal (t00)
Agent discovery of site search · isolated signals on a Worker1 / 10

How the tests ran

The only instruction each agent receivedFind out how to search the website https://tNN.964433797.xyz/ and then use it to search for "ferrari". Tell me how you discovered the way to search, the exact request you used, and the top 3 results.

Isolation

  • One signal per version. Nothing else differs.
  • No JS search widget and no links to any other site
  • The search endpoint has a random path (e.g. /q-p6a1z), so it cannot be guessed
  • Served by one Worker on its own zone (tNN.964433797.xyz): no automatic redirects, no platform headers, no Markdown conversion
  • All 13 versions passed an automated check before the runs

Agents

  • A fresh OpenCode subagent (Claude) per run, ≤25 tool calls
  • Fetch: webfetch only (GET, no response headers)
  • curl: raw HTTP from the shell
  • Browser: headless Chrome, accessibility snapshots; can read the DOM, no JS eval

Evidence

  • 12 versions × 3 harnesses × 2 runs, in 2 waves of 36 (13:55–14:04 UTC); Browser on t07 repeated at 16:04–16:07, replacing its 2 runs; t071 added at 16:18–16:21
  • Every request logged server-side and attributed to its run
  • Found = a 200 on the version's own endpoint for "ferrari"
  • Through the signal = the report cites it; otherwise it counts as a guess
Agent discovery of site search · isolated signals on a Worker2 / 10

The 13 versions

VersionThe only signal on the siteVisible to
t00 ↗nothing–
t01 ↗<link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml">head file
t02 ↗<form role="search" action="/q-…" method="get"><input name="q">page
t03 ↗<link rel="service-desc" type="application/json" href="/openapi.json">head file
t04 ↗Link: </openapi.json>; rel="service-desc" (HTTP response header only)header file
t05 ↗<link rel="api-catalog" href="/.well-known/api-catalog"> → linkset → /openapi.jsonhead file
t06 ↗<a href="/developers">Developers / API</a> (footer)page
t07 ↗<meta name="agent:instructions" content="…"> <meta name="agent:search" content="…/q-…?q={query}">head
t071 ↗<meta name="agent:search" content="…/q-…?q={query}"> (no agent:instructions)head
t08 ↗<meta name="api-endpoint" content="…/q-…?q={query}">head
t09 ↗<script type="application/ld+json"> WebSite → SearchAction urlTemplatehead
t10 ↗/llms.txt only, not linked from any pagefile
t11 ↗<div style="display:none" class="agent-instructions">…</div>hidden

head HTML <head> · header HTTP response header · page visible on the page · hidden hidden element · file extra file to fetch. Every version: same 182 pages, endpoint returns JSON { query, results }. Click a version to open it.

Agent discovery of site search · isolated signals on a Worker3 / 10

Every run

FetchcurlBrowserFoundAvg req
run 1run 2run 1run 2run 1run 2
t00 —✗gave up · 20 req✗gave up · 22 req✗gave up · 53 req✗gave up · 64 req✗gave up · 14 req✗gave up · 14 req0/6–
t01 rel=search✓3 req✓3 req✓4 req✓4 req✗gave up · 37 req✗gave up · 18 req4/63.5
t02 <form>✓2 req✓2 req✓2 req✓2 req✓2 req✓2 req6/62.0
t03 rel=service-desc✓3 req✓3 req✓4 req✓4 req✓10 req✓*18 req6/67.0
t04 Link header✓*23 req✓*17 req✓3 req✓4 req✓*13 req✓*20 req6/613.3
t05 rel=api-catalog✓4 req✓4 req✓5 req✓5 req✓12 req✓*13 req6/67.2
t06 dev link✓3 req✓3 req✓4 req✓4 req✓3 req✓3 req6/63.3
t07 agent:search✓2 req✓2 req✓3 req✓3 req✓9 req✓12 req6/65.2
t071 agent:search only✓2 req✓2 req✓3 req✓3 req✓6 req✓7 req6/63.8
t08 meta api-endpoint✓2 req✓2 req✓2 req✓2 req✓13 req✓18 req6/66.5
t09 JSON-LD✓2 req✓2 req✓2 req✓2 req✓15 req✓6 req6/64.8
t10 llms.txt✓9 req✓5 req✓13 req✓11 req✓6 req✓5 req6/68.2
t11 hidden div✓4 req✓3 req✓4 req✓3 req✓18 req✗gave up · 31 req5/66.4
With a signal24/24 · avg 4.5 req24/24 · avg 4.0 req21/24 · avg 10.0 req69/726.0

✓ = got results from the version's endpoint · req = requests to the site up to and including the search (server logs) · ✓* = guessed /openapi.json or /.well-known/api-catalog without seeing the signal

Agent discovery of site search · isolated signals on a Worker4 / 10

How fast: requests until the search

VersionFetchcurlBrowser
t02 <form>
2.0
2.0
2.0
t09 JSON-LD
2.0
2.0
10.5
t08 meta api-endpoint
2.0
2.0
15.5
t071 agent:search only
2.0
3.0
6.5
t07 agent:search
2.0
3.0
10.5
t06 dev link
3.0
4.0
3.0
t03 rel=service-desc
3.0
4.0
14.0*
t11 hidden div
3.5
3.5
18.0
t01 rel=search
3.0
4.0
not found
t05 rel=api-catalog
4.0
5.0
12.5*
t10 llms.txt
7.0
12.0
5.5
t04 Link header
20.0*
3.5
16.5*

Average over the successful runs (2 per cell). The minimum possible is 2: the homepage and the search. Sorted by the fetch + curl average. * includes runs that guessed a conventional path instead of using the signal.

Agent discovery of site search · isolated signals on a Worker5 / 10

Does agent:search need its instructions?

FetchcurlBrowserFoundAvg req
run 1run 2run 1run 2run 1run 2
t07 agent:instructions + agent:search✓2 req✓2 req✓3 req✓3 req✓9 req✓12 req6/65.2
t071 agent:search only✓2 req✓2 req✓3 req✓3 req✓6 req✓7 req6/63.8

t071 markup

<meta name="agent:search" content="https://t071.964433797.xyz/q-n4t8v?q={query}">

No text says what the tag is for, what it returns or that no authentication is needed.

How the t071 agents read it

Fetch: “The `<head>` contains `<meta name="agent:search" content="https://t071.964433797.xyz/q-n4t8v?q={query}">`, which gives a URL template for searching. I put "ferrari" in place of `{query}` and requested that URL.”

Browser: “I then looked at the homepage's `<head>` HTML and found `<meta name="agent:search" content="https://t071.964433797.xyz/q-n4t8v?q={query}">`. This is a URL template for searching.”

The instructions did not change whether agents found the search, or how fast. The browser still has to choose to read the <head> in both versions.

Agent discovery of site search · isolated signals on a Worker6 / 10

Why runs failed, and what agents look for

The 9 failures and 6 guesses

  • t00 (no signal), 6/6 failed. Agents tried 31.2 URLs on average (up to 64) and all concluded the site has no search. None invented an endpoint.
  • t01, Browser 2/2 failed. Both saw <link rel="search">, but Chrome treats opensearch.xml as a download (ERR_ABORTED), so the template was never read.
  • t11, Browser 1/2 failed. The run never read the HTML, and a hidden div is not in the accessibility tree.
  • t04, Fetch 2/2 + Browser 2/2 guessed. Neither tool shows response headers, so only curl saw the Link header. The other 4 runs got there because /openapi.json is a conventional path.
  • t03 and t05, Browser 1/2 each guessed /openapi.json and /.well-known/api-catalog before reading the <head>.

URLs agents tried (runs, of 78)

/robots.txt
36
/llms.txt
32
/sitemap.xml
30
/opensearch.xml
33
/.well-known/api-catalog
25
/openapi.json
23
/search?q=
23
/api/search?q=
20
/.well-known/mcp.json
12

Includes requests to files that were the signal on some versions. /llms.txt is why t10 worked 6/6 without any link.

Agent discovery of site search · isolated signals on a Worker7 / 10

Pages vs Worker: same test, same prompt

Pages (previous round)Worker (this round)
HoststNN.dummy-website-80l.pages.devtNN.964433797.xyz
Automatic redirects (308)38 in 20 of 72 runs0
Headers added by the platformx-robots-tag: noindex, cache-control, CORSnone (only server, cf-ray)
No signal (t00) found0/60/6
Found through the signal57/6657/66
Found by guessing a path26
Failed73
Avg requests via the signal: Fetch / curl / Browser2.8 / 3.9 / 7.33.1 / 4.1 / 8.9

Removing the platform's own behaviour did not change what agents find: 57/66 through the signal in both rounds (t00–t11; t071 only exists on the Worker).

The 7 runs that changed

  • t04 Fetch run 1: failed → guess
  • t04 Browser run 1: failed → guess
  • t07 Browser run 1: failed → signal
  • t04 Fetch run 2: failed → guess
  • t03 Browser run 2: signal → guess
  • t07 Browser run 2: failed → signal
  • t11 Browser run 2: signal → failed

These are agent choices, not the platform: the browser read the <head> on t07 this time but not on one t11 run, and fetch and browser tried /openapi.json on t04.

Agent discovery of site search · isolated signals on a Worker8 / 10

Conclusions

Caveats: one model family (Claude via OpenCode), 2 runs per cell, three harnesses; between rounds, 7 of 72 runs changed outcome. Discouraged signals (t08, t09, t11) worked here, but the source recommends against them for other reasons (hidden content, non-standard markup).

Agent discovery of site search · isolated signals on a Worker9 / 10

References

Source and standards

Test site and tooling

Data: results/worker/ (this round) and results/clean/ (Pages round): prompts, 72 reports each, logs, conformance, summary.json

Agent discovery of site search · isolated signals on a Worker10 / 10
← → to navigate · F for fullscreen · P to print