Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] frequent 504 behind NPM #1163

Open
3 of 7 tasks
working-name opened this issue Jun 26, 2024 · 1 comment
Open
3 of 7 tasks

[BUG] frequent 504 behind NPM #1163

working-name opened this issue Jun 26, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@working-name
Copy link

Describe the bug
For some reason I find myself doing docker restart the more I use search throughout the day. The container is just sitting pretty there, no load. Nothing in NPM's logs other than timeout connecting to whoogle container. Private instance, I see nothing in NPM logs to make me think someone anyone else is hitting it outside of me.

2024/06/26 13:24:21 [error] 878044#878044: *597529 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 1.1.1.1, server: whoogle.domain, request: "GET /search?q=Lumen HTTP/2.0", upstream: "http://172.17.0.1:18084/search?q=Lumen", host: "whoogle.domain"

Container sitting pretty, not choked, all the while ignoring incoming requests. Normally PIDS are about 8 when idle, so there's a few hanging for sure.

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT   MEM %     NET I/O        BLOCK I/O       PIDS
4be21cef5885   whoogle-search   0.15%     128.6MiB / 1GiB     12.56%    102MB / 75MB   108MB / 168kB   15

To Reproduce
Steps to reproduce the behavior:

  1. Run whoogle via docker, place it behind NPM
  2. Use as you would normally
  3. Bump into 504 errors

Deployment Method

  • Heroku (one-click deploy)
  • Docker
  • run executable
  • pip/pipx
  • Other: [describe setup]

Version of Whoogle Search

  • Latest build from [Docker Hub, benbusby/whoogle-search]
  • Version [Whoogle Search v0.8.4]

Desktop (please complete the following information):

  • OS: [Debian GNU/Linux 12 (bookworm)]
  • Browser [any]
  • Version [any]

Smartphone (please complete the following information): NOT applicable.

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
The container either recovers by itself after a while (is it single threaded and it waits for something to time out?), or it never does and I have to manually restart it.

It's not NPM, I checked logs, it passes the request along. I SSHed into the box, and ran a curl against whoogle container's locally exposed port, and it times out. It's 100% whoogle.

There's no useful errors logs in docker logs except this repeating issue with simple-theme.css:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1348, in handle_user_exception
    return self.ensure_sync(handler)(e)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/routes.py", line 624, in internal_error
    return render_template(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 151, in render_template
    return _render(app, template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 132, in _render
    rv = template.render(context)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/templates/error.html", line 8, in top-level template code
    <link rel="stylesheet" href="{{ cb_url(config.theme + '-theme.css') }}"/>
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/__init__.py", line 179, in <lambda>
    cb_url=lambda f: app.config['CACHE_BUSTING_MAP'][f])
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
KeyError: 'simple-theme.css'
@working-name working-name added the bug Something isn't working label Jun 26, 2024
@working-name
Copy link
Author

Found it dead again. This time the container says unhealthy. I just issued a killall -9 curl to kill all the health check processes that hung. Here's what's running on it:

$ docker compose exec whoogle-search ps aux
PID   USER     TIME  COMMAND
    1 whoogle   0:00 {run} /bin/sh ./run
    7 whoogle   0:00 {start-tor.sh} /bin/sh misc/tor/start-tor.sh
   11 whoogle   0:12 tor -f /etc/tor/torrc
   15 whoogle   0:59 python3 -um app --host 0.0.0.0 --port 5000
12776 whoogle   0:00 ps aux

Trying to curl from within the container results in the same behavior: timeout. Does this have to do with Tor? I don't remember enabling it.

$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received

Okay, let's rebuild the container.

$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2; docker compose up -d --force-recreate && sleep 10 && docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
[+] Running 1/1Container whoogle-search  Started                                                                                                                                            20.2s
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
### 10 seconds was not enough for the process to be ready...
containme@geek23:~/apps/whoogle-search$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=86400
< Content-Length: 12765
< Content-Type: text/html; charset=utf-8
< Date: Thu, 27 Jun 2024 03:21:07 GMT
< Server: waitress
< Set-Cookie: __Secure-session=.eJw1zMsKgkAUANBfibt2wEc-ElrmokDQRoU2cZ1H2ugU5kAl_nuzaXdWZ4HrU0wjaqFnSOfJCAfQzN3f7KFlf4N0WR1Q4mMBmxZSyN2waLMBc3_omrv6nvXwLrxOtqouK3rpyoDneKhPdcPCagxfNDhONMtURd092MuYntuGyZh7AhOCPjKy3blIEtm6JGERjzGIrD1Yf9UWNKw.Znzaow.JVGZZBm2hG4RsYpAPSdyjLnUzgU; Expires=Fri, 27 Jun 2025 03:21:07 GMT; Secure; HttpOnly; Path=/; SameSite=Lax
< Vary: Cookie
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
<
<html style="background: #000;">
<head>
... and so on. Normal response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant