How do I implement an MCP broker proxy between AI agents and downstream tool servers?

What an MCP broker proxy actually does

The Model Context Protocol (MCP) standardizes a JSON-RPC contract that AI agents use to discover, call, and stream from external tools. A broker proxy sits in the middle of that contract and takes on responsibilities that are awkward or unsafe to leave to either side: request routing, rate limiting, capability filtering, credential translation, and observability. Anthropic released the protocol in late 2024, and by September 2026 the ecosystem has shifted decisively toward mediated access rather than direct agent-to-server wiring.

Also worth reading: How do you implement a zero trust security architecture for Model Context Protocol servers in enterprise environments? · What is a policy decision point for AI agents and how do you implement one? · What are the standard MCP token delegation patterns for AI agents in 2026, and how do enterprise teams implement them securely?

The reason this matters is operational. Agents run in long-lived loops, generate dozens of tool calls per minute, and frequently connect to tools hosted by different teams with different identity systems. Without a proxy in the middle, each agent either has to hold raw credentials for every backend or every backend has to implement its own authentication. Neither pattern scales, and as VentureBeat reported in coverage of the Clawdbot incident, MCP itself shipped without authentication, which is precisely why most production teams now route traffic through a proxy that owns auth.

Practically, the broker is a stateless or near-stateless service that speaks MCP on both sides. On the upstream side it presents the tools the agent is allowed to see; on the downstream side it forwards the same calls to the real servers. The interception layer is where you add policy. The same pattern is now visible across major vendors: AWS announced general availability of the AWS MCP Server in 2025, Oracle shipped SQLcl MCP Server for AI-driven SQL, and AWS Bedrock AgentCore Runtime documented a pattern for running custom MCP proxies serverless.

Architectural choices for the proxy layer

There are three shapes a broker proxy can take, and the choice is driven by where your tools live and how many distinct trust boundaries you have. The first shape is a centralized gateway: one proxy in front of every downstream tool, owned by a platform team. This is the easiest to operate but creates a single point of failure and a bottleneck for every tool call. The second shape is a per-domain proxy: one proxy per business unit, each owning its own tools and policies. This is what most enterprises land on once they exceed roughly 20 to 30 distinct MCP servers. The third shape is a sidecar pattern, where the proxy runs next to each downstream tool and forwards to a shared policy engine. This minimizes blast radius but increases deployment complexity.

Each shape has a different cost profile. Centralized gateways benefit from consolidated logging and a single audit trail, but they require roughly 2x the throughput headroom of a sidecar because every call traverses an additional network hop. Per-domain proxies split the load naturally and let each team move at its own cadence. Sidecars are the safest under attack because compromise of one proxy does not expose adjacent tools, but they multiply the operational footprint and make version drift a constant headache. NVIDIA's practical security guidance for sandboxing agentic workflows explicitly recommends per-tool isolation over a single shared surface for any workflow handling sensitive operations.

The architectural decision should not be made in isolation from your identity model. If your agents authenticate with short-lived OAuth tokens issued by an identity provider, a centralized gateway is easier because token validation lives in one place. If your agents authenticate with long-lived API keys passed through environment variables, a per-domain proxy with explicit credential translation is safer because raw keys never leave the proxy boundary. Clawdbot's failure mode was essentially the latter case with no translation layer at all.

How to implement a minimal broker proxy

A working implementation has four components: an upstream MCP server endpoint, a downstream MCP client, a policy evaluator, and a credential vault. The upstream endpoint advertises a curated list of tools. The downstream client opens sessions to the real servers and forwards calls. The policy evaluator decides whether each call is allowed and rewrites it if needed. The credential vault swaps the agent's token for the backend's token.

The code skeleton is small. In Python with the official MCP SDK, you create a Server for the upstream side, register tool listings by reading from a config file or a policy service, and forward call_tool requests through an httpx client to the downstream URL. In Node, the same pattern uses @modelcontextprotocol/sdk with a Server and a Client instance connected over stdio or StreamableHTTPServerTransport. The proxy itself should be a thin layer that does not implement business logic; if you find yourself writing tool-specific code in the proxy, you have probably moved policy into the wrong place.

The credential vault does not need to be elaborate. AWS Secrets Manager, HashiCorp Vault, or even an environment variable scoped to the proxy process is enough for a first implementation. What matters is that the agent never sees the backend credential and the backend never sees the agent's identity token in raw form. The proxy should mint or fetch a backend-specific credential per request, attach it to the downstream call, and discard it. TTL should be measured in minutes, not hours.

The policy evaluator is where most teams underestimate the scope. A minimal policy is a list of allowed tools per agent identity. A realistic policy adds rate limits per tool, per agent, and per tenant; data loss prevention rules for tool arguments; and an allow-or-deny decision per call. NVIDIA's agentic sandboxing guidance recommends treating each tool call as untrusted input and evaluating the full request, including arguments, against policy before forwarding.

Comparison of common proxy patterns

The table is not a recommendation ladder; it is a decision matrix. Pick the row that matches your security posture, not your team size. A two-person team handling healthcare data should still choose sidecars. A 200-person platform team handling public marketing copy is fine with a centralized gateway.

Common mistakes when deploying MCP proxies

The first mistake is treating the proxy as a static config file. Tool catalogs drift constantly as backend teams add or deprecate tools, and a proxy that requires a redeploy to add a tool will become a bottleneck within weeks. The catalog should be dynamic, sourced either from a service registry or from the downstream servers themselves, and the proxy should re-discover on a short interval or in response to a webhook.

The second mistake is logging too much. MCP calls frequently contain sensitive data: database queries, customer identifiers, internal URLs. A proxy that logs full request bodies creates a compliance problem. Structured logging with explicit allowlists of fields, plus redaction of known-sensitive keys, is the minimum. AWS's published guidance on proxy configuration in Elastic Beanstalk emphasizes that the default proxy behavior of forwarding all request data should be paired with explicit allowlists at the application layer for sensitive workloads.

The third mistake is skipping rate limits because the underlying MCP servers already have them. Rate limits at the proxy serve a different purpose: they protect the proxy itself from being used as a denial-of-service amplification point. A reasonable starting limit is 60 calls per minute per agent identity and 600 calls per minute per tenant, with a burst allowance of 2x. These numbers come from observed usage patterns in agent-heavy workloads where a single misbehaving agent can generate thousands of calls in seconds.

The fourth mistake is ignoring the human in the loop. Agents that can execute real tool calls need a confirmation path for high-risk actions. The proxy is the natural place to inject this, either by holding a call pending human input or by rewriting it to a dry-run variant. The cost of skipping this is paid later, usually in incident reports.

When to act and what it costs

The right time to introduce a broker proxy is before the third MCP server, not after the thirtieth. Retrofitting a proxy onto a sprawling agent deployment is expensive because every existing call path has to be migrated and every downstream server has to accept proxied traffic. The marginal cost of standing up a proxy when you already have two servers is roughly a week of engineering time.

Running costs vary by deployment model. A proxy deployed on AWS Bedrock AgentCore Runtime scales to zero when idle and bills per invocation, which makes it cost-effective for low-traffic workloads. A proxy deployed on traditional ECS or Kubernetes has a baseline hourly cost but lower per-call cost at scale. As a rough rule, serverless proxies are economical below roughly 100,000 calls per month and conventional containers win above that threshold. AWS documentation on AgentCore Runtime pricing puts the per-invocation cost in the single-digit millicents as of mid-2026, though the exact number depends on compute configuration.

Build-versus-buy is a real decision. The MCP SDKs from Anthropic are open source and a basic proxy is a weekend project for an experienced engineer. Commercial gateways add policy engines, audit logs, and compliance reporting that would otherwise take months to build. For a team whose core product is not infrastructure, buying is usually cheaper within six months. For a team whose product is agent tooling, building is part of the product.

Alternatives and when to skip the proxy

The proxy pattern is not universal. Single-agent, single-tool deployments do not need a broker; the agent can call the tool directly. Embedding tools directly in the model runtime, the pattern used by OpenAI's function calling and Google's Gemini extensions, also bypasses the need for a separate proxy layer. The trade-off is portability: an agent wired to a specific provider's tool format is harder to move than an agent wired to MCP through a proxy.

Server-side MCP implementations from vendors like Oracle and AWS blur the line between tool server and proxy. The Oracle SQLcl MCP Server, for example, exposes database operations as MCP tools and handles authentication against the database directly. In that case, the proxy function is partially absorbed into the server. This works well for single-vendor stacks but creates the same vendor lock-in that MCP was designed to avoid.

Skip the proxy only when the workload is genuinely low-risk: read-only tools, no PII, no write actions, and a single tenant. The moment any of those conditions change, the proxy becomes the cheapest way to add the missing controls.

Operational checklist for the first 30 days

The first week should focus on the catalog and the credential vault. Wire the proxy to two non-production servers, populate the tool catalog from discovery, and confirm that credentials are never returned to the agent. The second week should add rate limits and structured logging with redaction. The third week should add the policy evaluator with at least one DLP rule and one human-in-the-loop rule for write actions. The fourth week should add observability: per-call latency histograms, error rates by downstream server, and a dashboard that shows policy decisions. By the end of the first month, a production proxy should be handling real traffic with at least one human-reviewed policy decision per write action. Anything less is a prototype, not a deployment.