How well does MCP server work?

[jzabroski]

Hi,

Just wondering if you've benchmarked basic harness prompts versus the MCP, and got results on turnaround time, tokens saved, and quality (pass/fail).

[ATrefzer]

No, I haven't benchmarked it. I added the MCP server as an experiment to learn how the protocol works in practice. It is one of the least-used features in the project. So I don't have any real-world data to share.
I made sure the MCP server gives reasonable answers, but I didn't go any further than that.

The "AI Advisor" was more useful for me (for generating ideas on how to resolve cycles)

If you have a benchmarking setup in mind, I'd be curious to hear about it.

[jzabroski]

a start would just be using generative Ai and asking each frontier lab SOTA model to generate a benchmark suite, two ways

1. blind exploration, perhaps human guided with a wayfinder skill
2. NDepend best practices like https://github.com/ardalis/NDependQueriesRules