Invalid strings in valid JSON

Have you ever seen JSON decoding to a generic "JSON value" type fail even though the input was actually valid JSON? I sure hadn't, before the start of last week when I got alerted because the processing of a webhook was causing errors in a piece of code that tried to convert the payload from one generic "JSON value" representation to another.

We never anticipated that this conversion could fail, but here I was, with the error message "unexpected end of hex escape" and an error location within the input... and without being able to see that input (obviously I can't just access customer data). Fortunately for me, somebody had previously opened an issue on the JSON library we use, referencing the exact same error message which quickly helped me to understand what was going on, no need to dig into the code.

Through that issue, I learned that a string within our input payload must have ended in a \uXXXX UTF-16 codepoint escape sequence where XXXX was a value between (hexadecimal) D800 and DBFF, a high UTF-16 surrogate. A high surrogate is meant to be always be followed up by a low surrogate to encode a unicode character outside of the Basic Multilingual Plane (U+0000 through U+FFFF) in UTF-16.

Before looking into this, I didn't even know JSON used UTF-16 for its escape sequences, much less did I imagine the possibility that the process of decoding a string from a JSON document could fail in ways not covered by JSON's grammar. So because we wanted to convert from a JSON representation that would not decode inner strings to one would try to fully decode them, our assumption that this conversion could never fail was wrong.

Fortunately again, it was no problem to update our code to gracefully handle this error and resume operation. We quickly deployed the fix and had the webhook retried and successfully delivered within a little over two hours.

Finally, here is a small demo of the issue for all the Rustaceans reading this: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=11add1ba542abee0b36eb07df10fa716

For more content like this, make sure to follow us on Twitter, Github, RSS, or our newsletter for the latest updates for the Svix webhook service, or join the discussion on our community Slack.