The exhibit floor at Black Hat this August had one theme, and it was not subtle. Roughly 235 of the 450 or so vendors on the floor marketed AI or agentic capability in some form, and the program devoted 35 of its 121 briefings to AI security, red teaming, or LLM-assisted offense. Nearly every AI governance diagram on the floor had the same shape: a single hop between the user and the cloud.
That hop is an LLM proxy. It is a reverse proxy that sits between the AI clients an organization runs and the model providers those clients call, so that every model request traverses one service the organization controls. Vendor documentation usually calls it a gateway, the show floor tried several other names, and the component underneath the labels is the same one.
Most organizations arrive at the idea the same way, which is by looking at what is already calling inference endpoints from inside their network. The answer is rarely the one deployment that went through review. There is usually a coding assistant on a developer laptop running on a personal key, a SaaS product that quietly added an AI feature and is now calling a provider on your data, a CI job someone wired up to summarize pull requests, and a prototype or two that nobody decommissioned. Every one of those is software holding its own credential and deciding on its own what organizational data gets placed into a context window. An earlier article in this series called that software the harness, and the reason the name matters is that the harness, not the model, is what holds your keys.
A proxy answers that the way proxies have always answered it, by inserting one hop that every request has to take. Once that hop exists, the provider key stops living on developer laptops, usage becomes attributable to a person, spend becomes something that can be capped, and every request becomes something that can be audited. That is a good promise, and the component behind it turns out to be simpler than the booth diagrams suggest in most respects and harder in exactly one.
What an LLM proxy actually does
The easiest way to see both is to follow a single request through one. It leaves the client and arrives at the proxy, which terminates the protocol and reads enough of the request to know what was asked for. The proxy works out who is calling. It replaces whatever credential the client presented with the real provider key, held server-side. It selects an upstream, usually by looking at the model name. It relays the streamed response back to the client as the tokens arrive, and it writes a record of what happened.
Most of that is unremarkable engineering, and saying so plainly is useful to anyone sizing the work. Terminating an HTTP API, swapping a header, choosing a destination from a lookup table, and appending to a log are all well-trodden problems with well-known solutions. A competent team can stand up something that does all of it in a few days, which is roughly why so many organizations have.
The interesting step is the one that looks like the least work. When the proxy asks who is calling, what it actually has is a bearer token in a header, and a bearer token is a claim rather than a fact. The proxy can confirm that the token is one it issued, which tells it the request came from someone who possesses that token. It cannot tell whether the person holding it is the person it was issued to, whether the token is being used from a company laptop or a home desktop, or whether the process presenting it is the coding agent it was meant for or a script someone wrote on a Friday afternoon. That matters more than it first appears, because attribution, budgets, and audit are all the same question wearing different hats. Knowing which team to bill, which developer to ask about an unusual prompt, and whose access to revoke at offboarding are three phrasings of who is on the other end of this connection. The component that the entire value of a proxy rests on is the one a proxy cannot establish by itself.
The designs that handle this well do not try. Tailscale's Aperture is worth studying on exactly this point, because it declines to answer the identity question at the proxy and inherits the answer instead. Aperture sits inside a tailnet, which has already authenticated both the user and the device before any request exists, so attribution becomes a property of the connection rather than a header the client chose to send. It listens only on Tailscale interfaces, which means reachability is itself an access decision rather than a policy the proxy has to evaluate. Provider keys stay in server configuration and get injected per request, so they never reach a developer machine at all. None of that is a clever proxy feature. It is a proxy taking identity from a layer that was already doing the work.
The generalizable version is worth carrying into any product evaluation. If your organization already operates a device identity layer, a VPN, or an SSO-backed network, that is where your proxy should get its identity, and the ability to consume it is worth more than any row on a feature matrix. If you do not have such a layer yet, building one is the higher-leverage project, because a proxy deployed on top of it becomes substantially more capable on the day the identity layer arrives.
Connecting your agents
The configuration side is genuinely small, and it is the same idea in all three of the major coding agents: tell the client to send its requests somewhere else, and give it a credential the proxy will accept.
Claude Code takes two environment variables. ANTHROPIC_BASE_URL names the proxy, and a credential variable carries the token, with ANTHROPIC_AUTH_TOKEN sent as a bearer token and ANTHROPIC_API_KEY sent in the x-api-key header. Both can be exported in a shell profile or placed in the env block of a settings file, and the client speaks the Anthropic Messages format, so a proxy in this position serves /v1/messages. Nothing else in the client changes, which is the appeal:
{ "env": { "ANTHROPIC_BASE_URL": "https://llm-proxy.internal.example.com", "ANTHROPIC_AUTH_TOKEN": "your-proxy-credential" }}Environment JSON Config
Codex expresses the same thing as configuration rather than environment. In ~/.codex/config.toml you define a model provider entry carrying a base URL, the name of the environment variable holding the key, and the wire format to speak, then select that provider. Codex speaks the OpenAI protocol, and the wire format is either responses or chat depending on which surface your proxy exposes. Gemini CLI works through GOOGLE_GEMINI_BASE_URL when authenticating with a Gemini API key, or GOOGLE_VERTEX_BASE_URL when authenticating against Vertex, and both require HTTPS unless the destination is localhost. All three also honor the standard HTTPS_PROXY and NO_PROXY variables, which is a separate mechanism worth keeping distinct in your head: a network proxy moves the packets, while an LLM proxy participates in the conversation.
The one practical consequence of running more than one of these is that the clients do not share a wire protocol. Claude Code speaks the Anthropic format and Codex speaks the OpenAI format, so a proxy serving both is implementing two API surfaces rather than one. That is entirely doable, and it is also the main reason most teams end up buying a proxy rather than writing one. If you are evaluating products, the question to ask is not whether a proxy supports your providers but whether it exposes the client-facing formats your agents actually speak.
What changes when you turn it on
Two things are worth understanding before you commit, and both are easier to handle when you plan for them rather than discovering them in week three.
The first is where coverage comes from. Pointing one client at a proxy is a configuration change, and configuration lives on the developer's machine. Every vendor offers an administrative channel for distributing that configuration centrally — Anthropic through managed settings delivered from its console or from device management, OpenAI through managed requirements files, and Google through system-level settings that take precedence over user settings — and each of those channels assumes either enrolled devices or a business-tier subscription. The practical guidance follows directly and is worth adopting regardless of tier: treat client configuration as distribution and treat the network as enforcement. If AI traffic can only leave your environment through a path where the proxy is reachable and the providers are not, coverage stops depending on whether every client was configured correctly. This is the same reasoning behind Aperture listening only on tailnet interfaces, and it is available to any organization that already controls its egress.
The second is billing, and the details reward precision because the intuitive version is wrong in a useful way. When a proxy credential is active, it replaces the developer's subscription login for that session, the subscription's usage limits no longer apply, and the traffic bills per token to whoever owns the credential the proxy forwards. Setting the base URL without a proxy credential behaves differently: requests still route through your proxy, but the saved subscription login remains the active credential, and the session continues to bill against the subscription. That means visibility is compatible with subscription pricing. Taking custody of provider keys is not, because custody is precisely the act of putting your credential in the path instead of theirs.
That distinction turns out to be a useful planning tool rather than an obstacle. A team that wants attribution, logging, and a view of what its agents are doing can have all three while developers keep their existing subscriptions, which is a genuinely low-friction place to start. A team that needs provider keys off developer laptops is choosing metered billing as part of that decision, and it is much better to budget for it deliberately than to meet it in the first invoice. Knowing which of those two you are buying, and being able to say so in a planning meeting, is most of what separates a smooth rollout from a surprised one.
Where this is going
Every previous class of outbound traffic went through the same progression. It started unmanaged, became observed once someone put a collector in the path, and eventually became operated, with identity attached, budgets enforced, and a record kept. Web traffic took that route, and so did email, DNS, and code dependencies. AI traffic is moving along the same path unusually quickly, and the proxy is the step where it stops being something an organization discovers and starts being something an organization runs.
What makes the step worth taking carefully is that a proxy is not where identity gets created. It is where identity you already have gets applied. Teams that internalize that make better decisions all the way down the stack: they pick products that can consume an existing identity layer, they sequence the network work before the proxy work, and they end up with attribution that means something instead of a log full of bearer tokens. Authority is worth more when it arrives from somewhere the request cannot argue with.