GRPC file handles not closing #86
Comments
|
Some relevant context yesodweb/wai#825 |
|
Asked to recreate this I simply setup a bash script to submit transactions and keep track of file handles: #! /usr/bin/bash
count=1000
for i in $(seq $count); do
echo "============"
echo "iteration $i"
echo "============"
ID=$(uuid -v4)
echo $ID
echo $(json-fs-store-cli insert-fact-statement --db $JS_STORE_DIR/json-store.db --fact_statement_id "$ID" --json '{"@context": "https://schema.org", "type": "PropertyValue", "name": "TEST1/TEST2", "value": 0.3142, "valueReference": [{"@type": "PropertyValue", "name": "validFrom", "value": 1705404151}, {"@type": "PropertyValue", "name": "validThrough", "value": 1705404451}], "ArklyID": "4343235324235235432453FAC710", "SecurityToken": "32R43243223432432"} ')
REQ=$(cat <<EOF
{
"fsInfos": [
{
"fsId": "$(echo -ne $ID | base64)",
"gcAfter": {
"extended": "NEG_INF"
}
}
],
"submitter": {
"base16": "$SUBMITTER_PKH"
}
}
EOF
)
COOP_PROTO=coop-proto
RESP=$(echo $REQ | grpcurl -insecure -import-path $COOP_PROTO \
-proto $COOP_PROTO/publisher-service.proto -d @ \
localhost:5080 coop.publisher.Publisher/createMintFsTx
)
echo "fs handles ($i): " $(lsof | grep "json-fs" | wc -l && date)
echo "publisher handles ($i)" $(lsof | grep "coop-publisher" | wc -l && date)
echo "coop-pab handles ($i)" $(lsof | grep "coop-pab" | wc -l && date)
echo $(sleep 120)
echo "done"
doneWe basically see the publisher grpc or the local fact store fail in different ways: Second failure: After setting We're seeing the Let me know if there's anything else I can test/look for. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've been running COOP as a long-running service for a decent while now. Every so often, we're seeing services stop with too many open file handles:
Network.Socket.accept: resource exhausted (Too many open files):Are there any known improvements in the underlying GRPC libraries that we can leverage here to prevent this? Or is there something else we can see in the code that might have been missed preventing file handles from being closed?
The text was updated successfully, but these errors were encountered: