Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry with backoff when requests are continously failing. #63

Open
nivemaham opened this issue Jul 20, 2020 · 0 comments
Open

Retry with backoff when requests are continously failing. #63

nivemaham opened this issue Jul 20, 2020 · 0 comments

Comments

@nivemaham
Copy link
Member

It is possible that the REST requests are continously failing for certain amount of time due to external causes. (such as server not reachable ).
In this case, the FitBit connector continuously retries until the request is successful. This produces large amount of logs and temporary overload.
To prevent this, we can implement the approach recommended by @blootsvoets

  • Build in an exponential backoff algorithm ( if the service is properly misbehaving).
    Each failed request that is not 401, assume that the server is misbehaving, then set a timeout. On each subsequent error, the timeout is doubled, plus some random number of milliseconds, up to a maximum and until the service is working as intended again.
  • To avoid a single connection error to cause the whole connector service to stall, we could set a threshold for number of requests that have failed until we think the server is disconnected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant