Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix(ish) for unicode encoding in packet #608

Merged
merged 5 commits into from
Aug 21, 2023

Conversation

grahamjenson
Copy link
Contributor

So this is a horrible problem and I am pretty sure this is not a 100% fix, but does fix some of the problems.

The packet encoding which includes the length of the packet data is incorrect because Javascript.length is not byte length and not Rune length, there is no direct analog in Golang but Javascript length is always less than either of those. Add to that, we must do this encoding/decoding in a stream makes it even harder.

I fixed this in

  1. Decode: increasing the limit of the limit reader while reading the packet based on the unicode header we just saw, this means the packet reader will keep reading further
  2. Encode: increasing the calculated length by scanning for unicode header bytes and adding to the length based on their byte value.

Both these solution are not 100% because the utf-8 header bytes are not a 1-1 value with the UCS-2 that JS uses. The proper solution would be to upgrade to socket.io version 4 (which fixes this using seperator bytes), but I need version 3 to at least kind-of work.

I have not performance tested either decoding or encoding, but I am pretty sure there wont be a massive overhead.

I added some tests, and fixed the old tests that were incorrect and errored when talking to a JS server with.

Without this fix you get some knarly errors where readers end half way though a message, or send to Javascript the incorrect amount of bytes which can cause server errors.

Reading more here:
https://mathiasbynens.be/notes/javascript-encoding
https://socket.io/blog/engine-io-4-release/#packet-encoding
https://socket.io/docs/v4/engine-io-protocol/#from-v3-to-v4

@grahamjenson
Copy link
Contributor Author

Not sure why these benchmarks are breaking, but this should be ready for merging and testing.

@erkie
Copy link
Collaborator

erkie commented Aug 21, 2023

This is a great fix. Nice that you were able to hunt it down and get to the bottom of it. Code looks great, and it doesn't look like a breaking change. So merging this 👍

@erkie erkie merged commit 59c0983 into googollee:master Aug 21, 2023
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants