The Ghost in the VPC: How a Missing HTTP Header Triggered a Deep Dive into Go's Proxy Quirks

When POST requests started failing inside Amazon VPC environments while working perfectly over public internet, Deliveroo engineers found themselves in a networking mystery worthy of a detective novel. The culprit? A single missing HTTP header exposing fundamental differences between public CDN behavior and private cloud networking.

The team was troubleshooting their Go-based edge service that proxies requests between restaurant tablets and backend systems. While GET requests functioned normally, POST requests consistently failed within their VPC with a cryptic Puma error: HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>.

Logs confirmed the request body existed when entering the proxy service but vanished before reaching downstream Rails applications. After eliminating typical suspects, engineers discovered a critical code nuance:

outgoingRequest, err := http.NewRequest(method, u, incomingRequest.Body)

Versus the working version:

bodyBytes, _ := ioutil.ReadAll(incomingRequest.Body)
outgoingRequest, err := http.NewRequest(method, u, bytes.NewBuffer(bodyBytes))

The root issue lay in how Go's http.NewRequest handles Content-Length. When passed an io.Reader without length information (like incomingRequest.Body's ioutil.nopCloser), Go omits Content-Length and defaults to chunked encoding. Explicitly setting the header solved it:

outgoingRequest.ContentLength = incomingRequest.ContentLength

But why did this only break inside the VPC? Public internet requests showed correct headers despite the same code flaw. Further investigation revealed their CDN was silently "fixing" requests:

"Our CDN is working as a man in the middle proxy... requests [are] batched in their entirety before they’re passed on... headers (including Content-Length) are updated... This gives efficiency benefits and prevents a type of DoS attack."

This buffering behavior—similar to NGINX's proxy buffers—masked the header omission over public internet but left VPC requests exposed. Key discoveries emerged:

1. Go's http.NewRequest only auto-sets Content-Length for types implementing .Len()
2. Missing Content-Length triggers Transfer-Encoding: chunked
3. CDNs often reconstruct headers during buffering
4. VPC traffic bypasses CDN normalization

The debugging journey underscores how cloud networking layers can transform application behavior. As the team concluded: "It’s a niche problem, but took quite a while to crack." The solution serves as a critical reminder for proxy implementations—never assume header inheritance when bridging network boundaries.

Source: Deliveroo Engineering Blog