CASE STUDY 2026-10-23 6 min read

Notlogin went from 48 to 85 in one deploy

We ran Prowl's benchmark on one of our own products, fixed what it found, and ran it again. Agent-readiness went from 48 to 85. Brand visibility in assistant answers stayed at 0. Here are both numbers and the reasons for each.

Disclosure. Notlogin is part of The Prowl Network, the set of services we build ourselves. We picked it because we could change its code the same day. The before and after runs used the same pipeline that scores every other service in the catalog.

The starting point: 48/100

Notlogin is "Sign in with Notlogin": OAuth-style identity that AI agents can use without a human login flow. That kind of product should be easy for an agent to find. Prowl's benchmark from 2026-05-31 said it wasn't.

The worker found no OpenAPI spec and no MCP manifest. Every discovery path returned 404. The ANALYZE phase, which only reads what a service publishes, saw a generic "platform" with no declared auth and no endpoints. It fell back to the platform_profile template, the one we use for marketing sites. An agent arriving cold would have reached the same conclusion: there is nothing here to call.

The API itself was fine. Agents just had no way to find it.

What we changed

Nothing in the product changed. We published three files and told Prowl where they are:

File	Path	What it tells an agent
OpenAPI 3.1 spec	`/openapi.json`	Every endpoint, its auth scheme and its error shape
MCP server manifest	`/server.json` and `/.well-known/mcp/server.json`	Where the MCP endpoint lives and which transport it speaks (streamable-http)
llms.txt	`/llms.txt`	A plain-text map pointing to the two files above

Those are the two locations the current MCP spec and registry use. Notlogin already served an older /.well-known/mcp.json, but nothing in the ecosystem reads it anymore. The whole change was 205 added lines across 8 files, all describing routes the app already served. It includes a test that fails if the spec documents a route that doesn't exist. It shipped as one deploy.

The re-run: 85/100

We re-benchmarked on 2026-10-23. This time the endpoint resolved from the declared manifest instead of a guess, and auto-detection picked mcp_compliance. The ONBOARD phase, which checks whether an agent can start using the service with no human help, succeeded as public_api. initialize answered 200 in 35 ms and tools/list answered 200 in 19 ms.

Dimension (0–10)	Before	After
doc_quality	2	8
response_parseability	3	9
first_try_success	4	9
error_clarity	3	7
auth_simplicity	5	10
token_efficiency	7	8
latency	10	10
consistency	8	8
Overall (0–100)	48	85

Latency and consistency were already good, because the servers were never the problem. The biggest gains came in the dimensions that measure whether an agent can tell what the service is: docs, parseability and first-try success.

We also ran ASO v2, which puts the same buyer question to real models and checks whether they pick the service out of Prowl's search results. Notlogin surfaced in 100% of runs and was picked in 80%. We have no "before" number for this test, because the old record had nothing for an agent to pick.

What the score still flags

Only the handshake was tested. The benchmark is read-only. It ran initialize and tools/list and never called a tool, because tool calls can have side effects. A vendor-submitted benchmark guide would let it go further.
One model, not two. Only one LLM provider responded during INTERPRET. Its raw grade was 78, and the stored score is 85. A two-model run could move the number a few points in either direction.
Unsigned manifest. Mycelio readiness is 30/100 because the MCP manifest isn't signed yet. That is the next fix.

The number that didn't move: brand visibility 0 → 0

We also ran our brand AI visibility checker before and after. It asks assistants ordinary buyer questions in the category ("best auth for AI agents", "alternatives to Auth0 for agents"). Neither run mentioned Notlogin in any of its 10 answers. The answers named Auth0, Privy, Turnkey, Dynamic, ZeroDev, Biconomy, Circle, Stripe and AWS instead.

We expected that. The checker measures what a model says from its own training data, with no web search. Discovery files help an agent that is looking: one that searches a directory, fetches a manifest or reads a spec. They don't change what a model already believes about a category. That only shifts when the brand turns up in enough text that later models train on, or when assistants answer with live search.

The takeaway. Agent-readiness and brand visibility are two different problems on two different timescales. You can fix the first in an afternoon by publishing files that tell agents what you are. The second takes months, and anyone who promises you a quick fix for it is guessing. Measure both, and don't let progress on one hide a zero on the other.

Do this for your own API

Check whether /openapi.json, /.well-known/mcp/server.json and /llms.txt exist on your domain. Missing files score as "nothing here".
If you run an MCP server, declare its URL and transport in server.json. Don't make agents guess /mcp.
Run a free benchmark, then claim your listing so the score is no longer capped at 75 and you can re-run it after each fix.

Benchmark your API free → Check brand AI visibility