Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client read deadlock on blocked write with driver mutex held #14

Open
SpComb opened this issue Jul 25, 2017 · 0 comments
Open

Client read deadlock on blocked write with driver mutex held #14

SpComb opened this issue Jul 25, 2017 · 0 comments
Labels

Comments

@SpComb
Copy link
Contributor

SpComb commented Jul 25, 2017

When benchmarking (#13) against an echo server, the client will saturate at a point where it is writing messages faster than it is able to read them back from the server. When the client's socket read buffer starts to fill up, the echo server's socket write buffer will also fill up, and the echo server will stop reading from the socket. Once the client's socket write buffer also fills up, the client will end up in a deadlock situation where it fails with a write timeout:

Kontena::Websocket::TimeoutError: write timeout after 5.0s
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:25:in `wait_socket_writable!'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:92:in `rescue in nonblocking_timeout'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:85:in `nonblocking_timeout'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:116:in `write'
  /home/kontena/kontena/kontena-websocket-client/vendor/bundle/ruby/2.3.0/gems/websocket-driver-kontena-0.6.5/lib/websocket/driver/hybi.rb:228:in `send_frame'
  /home/kontena/kontena/kontena-websocket-client/vendor/bundle/ruby/2.3.0/gems/websocket-driver-kontena-0.6.5/lib/websocket/driver/hybi.rb:191:in `frame'
  /home/kontena/kontena/kontena-websocket-client/vendor/bundle/ruby/2.3.0/gems/websocket-driver-kontena-0.6.5/lib/websocket/driver.rb:111:in `text'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:301:in `block in send'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:420:in `block in with_driver'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:417:in `synchronize'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:417:in `with_driver'
  /home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:300:in `send'
  benchmark/websocket-benchmark.rb:64:in `block in websocket_benchark_sender'
  benchmark/websocket-benchmark.rb:31:in `with_rate'
  benchmark/websocket-benchmark.rb:63:in `websocket_benchark_sender'
  benchmark/websocket-benchmark.rb:110:in `block (2 levels) in websocket_benchmark'

The reason for this happening is simple: the send thread calling Kontena::Websocket::Client#send will block on the Kontena::Websocket::Client::Connection#wait_socket_writable! with the @driver mutex held. This blocks the read thread calling Kontena::Websocket::Client#read -> #websocket_read from processing the data read from the socket (with_driver => driver.parse(...)), and the client deadlocks, unable to continue reading or writing:

I, [2017-07-25T15:12:48.638810 #13444]  INFO -- Kontena::Websocket::Client: read_yield -> dequeue 4
I, [2017-07-25T15:12:48.738506 #13444]  INFO -- Kontena::Websocket::Client: read_yield -> websocket_read...
I, [2017-07-25T15:12:48.838627 #13444]  INFO -- Kontena::Websocket::Client::Connection: read size=4096 timeout=nil...
I, [2017-07-25T15:12:48.916796 #13444]  INFO -- Kontena::Websocket::Client::Connection: wait write...
I, [2017-07-25T15:12:48.916900 #13444]  INFO -- Kontena::Websocket::Client::Connection: read size=4096: #buf=4096
I, [2017-07-25T15:12:48.916947 #13444]  INFO -- Kontena::Websocket::Client: read -> with_driver...

The client locking needs improvements to allow the socket.read -> driver.parse loop to continue if the driver.send -> socket.write would block, while also preventing any further driver.send -> socket.write calls from proceeding until the blocked write has proceeded in order to preserve write atomicity/ordering.

@SpComb SpComb added the bug label Jul 25, 2017
@SpComb SpComb changed the title Read deadlock on blocked write Client read deadlock on blocked write Jul 25, 2017
@SpComb SpComb changed the title Client read deadlock on blocked write Client read deadlock on blocked write with driver mutex held Jul 25, 2017
@SpComb SpComb mentioned this issue Jul 25, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant