Every token you log adds storage and privacy cost. The instinct when building an LLM proxy is to capture the prompt, completion, system message, tool arguments, and tool results. Full fidelity helps debugging: replay requests, reproduce issues, and trace conversations. That observability-maximalist view sounds sensible until compliance review or a log bill arrives three months later.
The alternative is not logging nothing. Be exact about what you need, then record only that.
The Case Against Logging All Content
A 4,000-token prompt is about 16KB of text. A 1,000-token completion is about 4KB. At 10,000 requests per day, content alone creates 200MB of logs daily, before metadata, trace headers, or anything else. At 100,000 requests per day, that reaches 2GB daily. Cloud log storage at typical rates can cost hundreds of dollars per month before producing value.
More importantly, you probably will not query prompt text in logs. When debugging unexpected output, a developer reproduces the request locally with a known input. They do not grep through 2GB of stored prompts. The content costs money to retain and rarely repays that cost.
The Compliance Case
Most real-world prompts include user-generated content, which often includes personal information: names, addresses, account details, health information, or whatever the user entered. Logging prompt content means logging PII.
GDPR Article 5 says personal data must be processed for specified, explicit, legitimate purposes and not retained longer than necessary. Defending a general-purpose log store with a 90-day retention policy is difficult when it holds PII from a user's prompt to an AI assistant. SOC 2 auditors will ask what you record, where it lives, who can access it, and how long you retain it. "We log every prompt" invites more questions.
Beyond GDPR and SOC 2, consider the attack surface. A successful prompt injection leaves its payload in your logs. Broad read access makes that log store a larger exfiltration target than metadata-only storage.
The Minimum Logging Surface
An LLM proxy log should tell you who made the request, which resource they reached, what it cost, whether it succeeded, and how long it took. Request metadata and token counts provide that without storing content.
Request Metadata
Always log: timestamp, request ID (for trace correlation), API key ID (hashed or the key prefix, not the full raw key), route and HTTP method, model requested, upstream provider selected, and wall-clock latency for the complete request.
Token Counts
Always record input token count, output token count, and estimated cost in USD using the provider's current pricing. Providers expose token counts in response headers or the response body. Record the counts, not the content.
Policy Events
Always record rate limit hits, including limit type, key, and window; auth failures, including key not found, key revoked, and quota exceeded; routing decisions, including fallback and secondary provider; plus policy rejections and their reason code.
Provider Response Status
Always record the upstream provider's HTTP status code, provider-specific error codes, and whether the primary or a fallback served the request.
Cost Attribution Without Content
This metadata supports complete cost attribution. Your key management store maps API key ID to an organization. Model and token counts show cost per request. Route identifies the product feature driving spend. Timestamp supports aggregation by day, billing period, or customer renewal cycle.
A platform engineer building an internal LLM integration put it this way: "I don't need to see what anyone asked the model. I need to know that key X consumed 2.3 million tokens in the past 30 days, 80% against gpt-4o on the /api/reason route. That's my cost report." Metadata provides that report without content risk.
Anomaly Detection from Metadata
Metadata logs cover the anomaly signals that matter most:
- Average input token count spikes: A consumer moved from short prompts to full-document inputs. Update the cost model and consider a per-request token cap.
- One key generates more 429 rate limit hits: A consumer may have a runaway loop or faulty retry behavior. The concurrency limit should catch it, while the 429 spike provides an early warning.
- Provider distribution shifts: More fallback activity than usual suggests stress at the primary provider. Investigate before consumers see failures.
- Latency rises without a token-count change: Average input tokens remain flat while p95 latency increases. That indicates provider slowdown rather than increased load on your side.
When Content Logs Are Necessary
Content logging can be justified in specific situations: building an AI coding assistant and reproducing exact sessions for bug reports, meeting legal hold requirements for full communications retention, or investigating a safety incident that requires knowing what was said.
For those cases, put content in a separate access-controlled store with an explicit retention policy and access audit trail. Keep it apart from operational logs. Specify who may access it and when. Do not combine compliance-sensitive content with operational observability in one log stream.
Ask not "should we log content?" but "what specific operational or compliance need requires content, and what minimal retention and access control meets it?" That starting point produces a defensible answer. "Log everything" does not.