Skip to content

How to force grant scraper to run

Christopher Keith edited this page Nov 25, 2022 · 4 revisions

Hit : http://localhost:3000/api/organizations/0/refresh/grants

Note that

  • Batch tasks are fired off, so you will have to wait minutes for a full 'refresh'.
  • There will be no output in the browser

Example (elided) output from docker compose logs app -f -t:

gost-app | 2022-10-28T21:28:45.063885800Z ::ffff:172.18.0.1 - - [28/Oct/2022:21:28:45 +0000] "GET /api/organizations/0/refresh/grants HTTP/1.1" 202 -
gost-app | 2022-10-28T21:28:46.307030800Z searching for @ 0
gost-app | 2022-10-28T21:29:11.383979400Z attempted to enrich grant but failed with RequestError: Client network socket disconnected before secure TLS connection was established
gost-app | 2022-10-28T21:29:40.271084300Z found 24 total results on grants.gov
gost-app | 2022-10-28T21:29:40.363038000Z created 344230 in grants
...
gost-app | 2022-10-28T21:29:40.500579900Z created 344233 in grants
gost-app | 2022-10-28T21:29:41.606799600Z searching for @ 10
gost-app | 2022-10-28T21:30:11.509703400Z attempted to enrich grant but failed with RequestError: Client network socket disconnected before secure TLS connection was established
gost-app | 2022-10-28T21:30:25.012761900Z found 24 total results on grants.gov
gost-app | 2022-10-28T21:30:26.133798900Z searching for @ 20
gost-app | 2022-10-28T21:31:05.413184900Z attempted to enrich grant but failed with RequestError: socket hang up
gost-app | 2022-10-28T21:31:09.340304900Z found 24 total results on grants.gov
gost-app | 2022-10-28T21:31:10.496642000Z searching for @ 30 gost-app | 2022-10-28T21:31:26.546007300Z attempted to enrich grant but failed with RequestError: Client network socket disconnected before secure TLS connection was established
gost-app | 2022-10-28T21:31:51.413026300Z attempted to enrich grant but failed with RequestError: Client network socket disconnected before secure TLS connection was established
gost-app | 2022-10-28T21:32:00.663738800Z found 23 total results on grants.gov
gost-app | 2022-10-28T21:32:01.804473200Z searching for @ 40
gost-app | 2022-10-28T21:32:49.992286700Z attempted to enrich grant but failed with TimeoutError: Timeout awaiting 'request' for 20000ms
gost-app | 2022-10-28T21:32:54.132631200Z found 23 total results on grants.gov
gost-app | 2022-10-28T21:32:55.267777300Z searching for @ 50
gost-app | 2022-10-28T21:33:26.464222400Z attempted to enrich grant but failed with RequestError: Client network socket disconnected before secure TLS connection was established
gost-app | 2022-10-28T21:33:26.464308400Z found 13 total results on grants.gov
gost-app | 2022-10-28T21:33:27.589712200Z searching for @ 60
gost-app | 2022-10-28T21:33:33.239615700Z found 4 total results on grants.gov
gost-app | 2022-10-28T21:33:34.289620100Z searching for @ 70
gost-app | 2022-10-28T21:33:39.414994000Z sync complete!