Engineering MCP at Scale: Reducing Token Consumption by 100x

Enterprise Products requires enterprise scale. To build a truly immersive agentic chat experience for a large scale analytics data, we engineered a large scale MCP server for our B2C SAAS customer.

Building intelligent AI agents that interact with external APIs used to be a massive engineering challenge. The Model Context Protocol (MCP) changed that by giving developers an elegant, open standard to connect Large Language Models (LLMs) with databases, services, and internal workflows.

However, as engineering teams scale MCP integrations from proof-of-concept demos into production environments, they inevitably hit an expensive architectural wall: the hidden "token tax" of scale.

Without deliberate context optimization, MCP servers rapidly inflate LLM context windows, spiking API costs and introducing multi-second response latency. Here is an inside look at how naive MCP architectures break down under production loads — and the two architectural strategies that cut context token consumption by up to 100x.

The hidden "token tax" of MCP scale

Most of the MCP tools built are intuitive: wrap an existing REST API, expose each endpoint as an individual tool, and register all tool schemas with the LLM.

While this approach works smoothly in a short-term, it creates two major structural bottlenecks at scale:

• Payload over-fetching: Traditional REST APIs return broad JSON payloads filled with nested metadata. If an LLM needs a customer's primary email address, a naive getUser tool might return a payload containing profile details, system roles, feature flags, and historical telemetry. The model consumes tokens for every single field, most of them unrelated to the actual query.
• Tool proliferation: To prevent over-fetching, teams often split API wrappers into dozens of granular micro-tools. But every registered tool requires a full JSON schema definition injected directly into the LLM's system prompt before execution begins. Ingesting definitions for 50+ tools burns thousands of tokens on every user turn, regardless of whether those tools are actually invoked.

When multiplied across millions of daily agent steps, this token bloat transforms what should be a lightweight interaction into a massive cost and latency bottleneck.

To break this cost curve, our production-grade systems rely on two key optimization patterns.

Strategy #1: Just-in-time discovery via dynamic toolsets

Instead of feeding the LLM an exhaustive directory of every tool the system supports, dynamic toolsets adopt a just-in-time hydration strategy.

The core concept is to treat tool definitions as a searchable catalog rather than a permanent prompt fixture:

1. Category overview: The LLM is initialized with high-level descriptions of functional tool domains rather than granular schemas.
2. Natural language discovery: When given a task, the LLM uses a specialized meta-tool (search_tools) to query the catalog for relevant functions using natural language.
3. On-demand schema loading: Only the exact tool schemas required for the active step are dynamically loaded into the context window for execution.

By decoupling tool availability from prompt context size, this progressive discovery pattern supports massive tool ecosystems without context overhead. Production implementations demonstrate up to a 160x token reduction compared to static tool registration.

Engineering trade-offs

While dynamic toolsets unlock vast scalability, they introduce a distinct engineering choice: latency vs. context window size. Because tool discovery requires a pre-execution search step, single-turn interactions may require an additional round-trip turn. For complex multi-step workflows, however, the drastic reduction in prompt processing time significantly offsets the initial discovery overhead.

Strategy #2: Eliminating payload bloat with GraphQL MCP

For data-dense infrastructure, dynamic tool discovery solves schema bloat, but payload over-fetching remains a problem. That is where a GraphQL-powered MCP layer shines.

Instead of registering individual tools for every database entity or API resource, the entire data layer collapses into two universal MCP tools:

• get_graphql_schema: Allows the LLM to introspect available entities, relationships, and queries.
• execute_graphql_query: Enables the LLM to construct and run structured GraphQL queries.

This pattern shifts payload control directly to the model. Because GraphQL requires callers to explicitly specify requested fields, the LLM requests only what it needs to fulfill its task.

{ "user": { "id": "123", "name": "Jane", "email": "j@x.com",
    "roles": [...20 items...], "metadata": {...100 lines...} } }

{ "user": { "email": "j@x.com" } }

By requesting user { email } instead of receiving a full user object, context payloads shrink by 70% to 80%, while simplifying server architecture down to two universal interfaces.

Architectural decision matrix

Production optimization checklist

To optimize an existing MCP implementation today, these three architectural adjustments deliver fast returns:

1. Audit schema verbosity: Strip unnecessary field descriptions, non-essential optional parameters, and verbose metadata from existing tool JSON schemas.
2. Implement response caching: Wrap tool outputs in semantic caching layers to prevent identical tool calls from re-evaluating full execution logic.
3. Deploy a hybrid architecture: Combine dynamic toolsets to discover domain endpoints with a GraphQL layer underneath to execute surgical, field-level queries.

As AI agents transition from experimental chat interfaces to core operational infrastructure, context efficiency becomes the primary driver of system performance and cost control. Designing MCP architecture for context conservation today ensures your agentic systems scale reliably tomorrow.