CASE STUDY 2026-10-23 6 min read

RogerThat scored 79 with a bug two runs graded 15 points apart

RogerThat's listing went 71 → 64 → 79. The last number looks like a fix. It isn't: the bug that cost points in the second run was still there in the third, 20 minutes later. This is the case study about what a score can't tell you.

RogerThat agent-readiness case study illustration

RogerThatagent-readiness71 → 64 → 79

Disclosure. RogerThat is part of The Prowl Network, the set of services we build ourselves. The runs below used the same pipeline that scores every other service in the catalog, and every number comes from the stored benchmark records.

Three runs

Run	Template	What was tested	Score
2026-05-22	`platform_profile`	Website, `robots.txt`, `llms.txt`. No MCP calls.	71
2026-07-21 22:29	`mcp_compliance`	`initialize` 200, `tools/list` failed	64
2026-07-21 22:49	`mcp_compliance`	`initialize` 200, `tools/list` failed the same way	79

The first run never touched the MCP server. It graded the marketing site, so its 71 says nothing about the product. Once the record declared an MCP manifest, auto-detection switched to mcp_compliance and the benchmark spoke MCP to it.

The bug

In both MCP runs, initialize returned 200. The next request, tools/list, came back with JSON-RPC error -32600: "session not initialized; call initialize first". The client had just done exactly that. To an agent this is a dead end: the server accepts the handshake and then forgets it. It cannot list tools, so it cannot call any.

Dimension (0–10)	05-22	07-21 22:29	07-21 22:49
doc_quality	7	6	7
response_parseability	8	8	8
first_try_success	6	2	7
error_clarity	6	9	9
auth_simplicity	7	8	10
token_efficiency	7	7	8
latency	10	9	8
consistency	8	3	7
Overall (0–100)	71	64	79

Look at first_try_success and consistency. The 22:29 run gave them 2 and 3, which is right for a server that fails its second request. Twenty minutes later, with the same failure in the evidence, a grader gave them 7 and 7, and gave auth_simplicity a 10. The raw grade was 70, and the stored score came out 79. Only one LLM provider responded in each run (the second provider was rate-limited), so there was no second opinion to average against.

What we learned

A single-model grade on a broken handshake is noise. The same evidence produced 64 and 79. A 15-point swing is bigger than most of the improvements vendors pay to re-benchmark for.
Hard failures should cap the score, not just inform it. If tools/list fails, an agent can't use the server. That should limit the score no matter how lenient the grader is.
The bug is ours to fix. RogerThat is now Apuchat on the same backend, and the apuchat.com listing passed tools/list on 2026-08-30 (case study). The rogerthat.chat score has not been re-run since July, so 79 still stands on that listing, and we are not claiming it's fixed there until a run says so.

The takeaway. Don't read a score without the evidence behind it. A 79 with a failing tools/list is worse for an agent than a 64 that reports the failure honestly. When you look at any benchmark, ours included, check the raw call results first.

Do this for your own MCP server

Test the full handshake: initialize, then notifications/initialized, then tools/list, in separate HTTP requests. A server that keeps session state in memory can fail this behind a load balancer or with several workers.
If your server is stateless, don't require a session. If it isn't, issue an Mcp-Session-Id on initialize and accept it on the next request.
Run a free benchmark and read the per-call results, not only the overall number.

Benchmark your API free → Check brand AI visibility