upvote

points

by rkuska5 hours ago |
comments
upvoteby hxtk2 hours ago|
next
[-]
While it’s convenient, the Decoder isn’t recommended for those use cases precisely because of the edge cases around readers that don’t contain at most one JSON document with no suffixes.

The standard library maintainers recognized this as a problem that there was no “better,” more convenient way to do it and in encoding/json/v2 they have UnmarshalRead for that use case: https://pkg.go.dev/encoding/json/v2#UnmarshalRead

reply
upvoteby hnav3 hours ago|
prev|
next
[-]
It drains up to 256K unread bytes period, it doesn't just optimize the case of the body not being read at all.

https://cs.opensource.google/go/go/+/refs/tags/go1.27.2:src/...

What it won't do looking at the code is handle streamed responses over transfer-encoding:chunked which means that if the other end is doing

  json.NewEncoder(responseWriter).Encode(&someStruct)
this optimization won't do anything.
reply
upvoteby gandreani4 hours ago|
prev|
[-]
Oof, that is an edge case I wasn't aware of at all. Hard to know where it's documented since it's an integration of both those packages
reply
upvoteby cyphar37 minutes ago|
parent|
[-]
This can happen without HTTP, we hit this issue in runc a decade ago[1] -- some container invocations would mysteriously error out with ECONNRESET and if you added a dummy environment variable they would work again.

Turns out that json.Encoder always appends a newline but (as GP said) json.Decoder will not attempt to read it if the JSON is complete at the power-of-two byte boundary (IIRC the buffer doubles for the first few iterations). The extra byte in the pipe when closed then causes reads and writes to go boom. We switched to using json.Marshal because it doesn't output a newline (sadly the encoded form isn't streamed as a result).

I still get twitchy around encoding/json as a result of that experience.

[1]: https://github.com/docker/docker/issues/14203#issuecomment-1...

reply