Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Approved agreement not changing state and cannot be deleted #343

Closed
stan7123 opened this issue Feb 28, 2024 · 1 comment
Closed

Approved agreement not changing state and cannot be deleted #343

stan7123 opened this issue Feb 28, 2024 · 1 comment

Comments

@stan7123
Copy link

I have an agreement on yagna which was approved and did not change its state for a few days now, even when yagna was disconnected and restarted multiple times. I don't know if it should change its state in this situation.
But, when trying to terminate it I receive an error.

AgreementId: da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8
Yagna version: yagna 0.14.0 (9f9be57b 2023-12-07 build #365)

Yagna logs from creating the agreement (no other mentions about this agreement since then):

[2024-02-26T19:23:22.842+0000 INFO  ya_relay_client::session] ReverseConnection - got session with node: [0x21d9bfa212014d9bf5cfcef2345e40447a602748]
[2024-02-26T19:23:22.904+0000 INFO  ya_market::negotiation::common] Received counter Proposal [R-ba9ac55f2e4bafb791b79cbb1387bedbad27930eb659b35ec8bd13d21ac67d49] for Proposal [R-531467cae160
90b8d86dd639d0d40a71f70c838ce3c2df627ede5507e503239c] from [0x21d9bfa212014d9bf5cfcef2345e40447a602748].
[2024-02-26T19:23:22.945+0000 INFO  ya_market::negotiation::requestor] Requestor 'autoconfigured' [0x078de4ffddde3b129de4f3861c392ee96bc77197] created Agreement [R-da5eb50ede2744456bdc551358e
e7afd4b16ea50315ac7b6d91a5d343e00e1c8] from Proposal [R-ba9ac55f2e4bafb791b79cbb1387bedbad27930eb659b35ec8bd13d21ac67d49].
[2024-02-26T19:23:22.965+0000 INFO  ya_market::negotiation::requestor] Requestor 'autoconfigured' [0x078de4ffddde3b129de4f3861c392ee96bc77197] confirmed Agreement [R-da5eb50ede2744456bdc55135
8ee7afd4b16ea50315ac7b6d91a5d343e00e1c8] and sent to Provider.
[2024-02-26T19:23:22.965+0000 INFO  ya_market::negotiation::requestor] AppSession id [718def7baddf40219f2ea2570df687f8] set for Agreement [R-da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8].
[2024-02-26T19:23:22.969+0000 INFO  ya_market::negotiation::requestor] Agreement [R-da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8] approved by [0x21d9bfa212014d9bf5cfcef2345e40447a602748]. Committing...
[2024-02-26T19:23:22.977+0000 INFO  ya_market::negotiation::requestor] Agreement [R-da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8] committed (approved) by [0x21d9bfa212014d9bf5cfcef2345e40447a602748].
[2024-02-26T19:23:23.753+0000 INFO  ya_activity::requestor::control] Created Activity [0502b4ca1cd44abf85118b56cc23d6ae] for Agreement [da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8]
[2024-02-26T19:23:23.755+0000 INFO  ya_vpn::network] Creating network: e3aa6160085d4967b4685e524a6d3ef7 (10.1.195.145/31)
[2024-02-26T19:23:23.755+0000 INFO  ya_vpn::network] VPN e3aa6160085d4967b4685e524a6d3ef7 started
[2024-02-26T19:23:23.755+0000 INFO  ya_vpn::network] Network: e3aa6160085d4967b4685e524a6d3ef7 assigning new ip address: 10.1.195.144 for identity: 0x078de4ffddde3b129de4f3861c392ee96bc77197
[2024-02-26T19:23:23.756+0000 INFO  ya_vpn::network] Adding Node: 10.1.195.145 to network: e3aa6160085d4967b4685e524a6d3ef7

Agreement details:

root@db7296ed83c3:~/.local/share/yagna# yagna market agreements get --id da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8 --role Requestor
agreementId: da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8
appSessionId: 718def7baddf40219f2ea2570df687f8
approvedDate: 2024-02-26T19:23:22.964128218Z
approvedSignature: NoSignature
committedSignature: NoSignature
demand:
  constraints: "(&(golem.node.debug.subnet=gpu-test2)\n\t(golem.com.payment.platform.erc20next-goerli-tglm.address=*)\n\t(golem.runtime.name=vm-nvidia)\n\t(&(golem.runtime.capabilities=vpn)\n\t(golem.runtime.capabilities=inet)\n\t(golem.runtime.capabilities=manifest-support))\n\t(golem.inf.mem.gib>=16)\n\t(golem.inf.storage.gib>=20)\n\t(golem.inf.cpu.threads>=2))"
  demandId: 1db3f26e073345459934bfa3070ce5eb-41b1ab1e48daf06cd81ce6675a5ce358030299c39e0f8f4def7d4e8f3dbedebc
  properties:
    golem.com.payment.chosen-platform: erc20next-goerli-tglm
    golem.com.payment.debit-notes.accept-timeout?: 120
    golem.com.payment.platform.erc20next-goerli-tglm.address: 0x078de4ffddde3b129de4f3861c392ee96bc77197
    golem.node.debug.subnet: gpu-test2
    golem.srv.caps.multi-activity: true
    golem.srv.comp.expiration: 1709011402631
    golem.srv.comp.payload: ewogICJ2ZXJzaW9uIjogIjAuMS4wIiwKICAiY3JlYXRlZEF0IjogIjIwMjMtMTAtMjBUMTI6NTE6MDAuMDAwMDAwWiIsCiAgImV4cGlyZXNBdCI6ICIyMTAwLTAxLTAxVDAwOjAxOjAwLjAwMDAwMFoiLAogICJtZXRhZGF0YSI6IHsKICAgICJuYW1lIjogInNjYWxlcG9pbnRhaS9hdXRvbWF0aWMxMTExOjQiLAogICAgImRlc2NyaXB0aW9uIjogInNjYWxlcG9pbnRhaS9hdXRvbWF0aWMxMTExOjQiLAogICAgInZlcnNpb24iOiAiMC4xLjAiCiAgfSwKICAicGF5bG9hZCI6IFsKICAgIHsKICAgICAgInBsYXRmb3JtIjogewogICAgICAgICJhcmNoIjogIng4Nl82NCIsCiAgICAgICAgIm9zIjogImxpbnV4IgogICAgICB9LAogICAgICAidXJscyI6IFsKICAgICAgICAiaHR0cDovL3JlZ2lzdHJ5LmdvbGVtLm5ldHdvcmsvZG93bmxvYWQvNjg4ZDVlMzhjNmJmYWEyODRiYzVkNGM5YzQyMmJiNzk0MjNiNDI0NDQyZGQ3NzUwYWQ5YmYyYTkyZTBlODJkNyIKICAgICAgXSwKICAgICAgImhhc2giOiAic2hhMzpkOGQ0N2NjY2NmYzI0MmQ0NmI4NjA4MTllZmNkZDRkZDE3YjRmYjUxMDM0MjAwM2ZlOTlkNzQ1OCIKICAgIH0KICBdLAogICJjb21wTWFuaWZlc3QiOiB7CiAgICAidmVyc2lvbiI6ICIwLjEuMCIsCiAgICAic2NyaXB0IjogewogICAgICAiY29tbWFuZHMiOiBbCiAgICAgICAgInJ1biAuKiIsCiAgICAgICAgInRyYW5zZmVyIC4qIgogICAgICBdLAogICAgICAibWF0Y2giOiAicmVnZXgiCiAgICB9LAogICAgIm5ldCI6IHsKICAgICAgImluZXQiOiB7CiAgICAgICAgIm91dCI6IHsKICAgICAgICAgICJwcm90b2NvbHMiOiBbImh0dHBzIl0sCiAgICAgICAgICAidXJscyI6IFsiaHR0cHM6Ly9odWdnaW5nZmFjZS5jbyIsImh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jbyIsImh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvIiwiaHR0cHM6Ly9ncHUtcHJvdmlkZXIuZGV2LmdvbGVtLm5ldHdvcmsiXQogICAgICAgIH0KICAgICAgfQogICAgfQogIH0KfQo=
    golem.srv.comp.vm.package_format: gvmkit-squash
    maintenance_port: 8001
    service_port: 8000
  requestorId: 0x078de4ffddde3b129de4f3861c392ee96bc77197
  timestamp: 2024-02-26T19:23:22.944773310Z
offer:
  constraints: |-
    (&
      (golem.srv.comp.expiration>1708975336624)
      (golem.node.debug.subnet=gpu-test2)
    )
  offerId: ea393f1f8baf41df86b9a2bc642992d1-f3a22e9b90e4a77df99c9a53409fe911ee10a5f6a0ed7211f48f7f90d5135d17
  properties:
    golem.!exp.gap-35.v1.inf.gpu.clocks.graphics.mhz: 2115
    golem.!exp.gap-35.v1.inf.gpu.clocks.memory.mhz: 9751
    golem.!exp.gap-35.v1.inf.gpu.clocks.sm.mhz: 2115
    golem.!exp.gap-35.v1.inf.gpu.clocks.video.mhz: 1950
    golem.!exp.gap-35.v1.inf.gpu.cuda.cores: 10496
    golem.!exp.gap-35.v1.inf.gpu.cuda.enabled: true
    golem.!exp.gap-35.v1.inf.gpu.cuda.version: '12.0'
    golem.!exp.gap-35.v1.inf.gpu.memory.bandwidth.gib: 936
    golem.!exp.gap-35.v1.inf.gpu.memory.total.gib: 24.0
    golem.!exp.gap-35.v1.inf.gpu.model: NVIDIA GeForce RTX 3090
    golem.activity.caps.transfer.protocol:
    - https
    - http
    - gftp
    golem.com.payment.debit-notes.accept-timeout?: 120
    golem.com.payment.platform.erc20-goerli-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.payment.platform.erc20-holesky-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.payment.platform.erc20-mumbai-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.payment.platform.erc20next-goerli-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.payment.platform.erc20next-holesky-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.payment.platform.erc20next-mumbai-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.payment.platform.erc20next-rinkeby-tglm.address: 0x10e6c5ae67b8c12c30b4139ad7cebecae1917e7c
    golem.com.pricing.model: linear
    golem.com.pricing.model.linear.coeffs:
    - 0.0
    - 0.0002777777777777778
    - 0.0
    golem.com.scheme: payu
    golem.com.usage.vector:
    - golem.usage.cpu_sec
    - golem.usage.duration_sec
    golem.inf.cpu.architecture: x86_64
    golem.inf.cpu.brand: 13th Gen Intel(R) Core(TM) i5-13400F
    golem.inf.cpu.capabilities:
    - sse3
    - pclmulqdq
    - dtes64
    - monitor
    - dscpl
    - vmx
    - eist
    - tm2
    - ssse3
    - fma
    - cmpxchg16b
    - pdcm
    - pcid
    - sse41
    - sse42
    - x2apic
    - movbe
    - popcnt
    - tsc_deadline
    - aesni
    - xsave
    - osxsave
    - avx
    - f16c
    - rdrand
    - fpu
    - vme
    - de
    - pse
    - tsc
    - msr
    - pae
    - mce
    - cx8
    - apic
    - sep
    - mtrr
    - pge
    - mca
    - cmov
    - pat
    - pse36
    - clfsh
    - ds
    - acpi
    - mmx
    - fxsr
    - sse
    - sse2
    - ss
    - htt
    - tm
    - pbe
    - fsgsbase
    - adjust_msr
    - bmi1
    - avx2
    - fdp
    - smep
    - bmi2
    - rep_movsb_stosb
    - invpcid
    - deprecate_fpu_cs_ds
    - rdseed
    - adx
    - smap
    - clflushopt
    - processor_trace
    - sha
    - clwb
    - umip
    - pku
    - ospke
    - rdpid
    golem.inf.cpu.cores: 10
    golem.inf.cpu.model: Stepping 2 Family 6 Model 367
    golem.inf.cpu.threads: 15
    golem.inf.cpu.vendor: GenuineIntel
    golem.inf.mem.gib: 57.81822070479393
    golem.inf.storage.gib: 708.7353454589844
    golem.node.debug.subnet: gpu-test2
    golem.node.id.name: g4-3090
    golem.node.net.is-public: false
    golem.runtime.capabilities:
    - inet
    - vpn
    - manifest-support
    - start-entrypoint
    - '!exp:gpu'
    golem.runtime.name: vm-nvidia
    golem.runtime.version: 0.1.3-rc10
    golem.srv.caps.multi-activity: true
    golem.srv.caps.payload-manifest: true
  providerId: 0x21d9bfa212014d9bf5cfcef2345e40447a602748
  timestamp: 2024-02-26T19:23:22.944773310Z
proposedSignature: NoSignature
state: Approved
timestamp: 2024-02-26T19:23:22.944773310Z
validTo: 2024-02-26T19:24:22.944263Z

Error when trying to terminate it using golem-core-python:

golem_worker_api-1  | 2024-02-28 12:22:09,064 - uvicorn.error - ERROR - Exception in ASGI application
golem_worker_api-1  | Traceback (most recent call last):
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
golem_worker_api-1  |     result = await app(  # type: ignore[func-returns-value]
golem_worker_api-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
golem_worker_api-1  |     return await self.app(scope, receive, send)
golem_worker_api-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
golem_worker_api-1  |     await super().__call__(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
golem_worker_api-1  |     await self.middleware_stack(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
golem_worker_api-1  |     raise exc
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
golem_worker_api-1  |     await self.app(scope, receive, _send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
golem_worker_api-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
golem_worker_api-1  |     raise exc
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
golem_worker_api-1  |     await app(scope, receive, sender)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
golem_worker_api-1  |     await self.middleware_stack(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
golem_worker_api-1  |     await route.handle(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
golem_worker_api-1  |     await self.app(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
golem_worker_api-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
golem_worker_api-1  |     raise exc
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
golem_worker_api-1  |     await app(scope, receive, sender)
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
golem_worker_api-1  |     response = await func(request)
golem_worker_api-1  |                ^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app
golem_worker_api-1  |     raise e
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app
golem_worker_api-1  |     raw_response = await run_endpoint_function(
golem_worker_api-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
golem_worker_api-1  |     return await dependant.call(**values)
golem_worker_api-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/code/workers/routes.py", line 79, in delete_worker
golem_worker_api-1  |     output_dto = await DeleteWorker.execute(input_dto)
golem_worker_api-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/code/workers/use_cases/delete_worker.py", line 33, in execute
golem_worker_api-1  |     await agreement.terminate()
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/golem/resources/base.py", line 60, in wrapper
golem_worker_api-1  |     return await f(*args, **kwargs)
golem_worker_api-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/golem/resources/agreement/agreement.py", line 89, in terminate
golem_worker_api-1  |     await self.api.terminate_agreement(self.id, request_body={"message": reason})
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/ya_market/api_client.py", line 205, in __call_api
golem_worker_api-1  |     raise e
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/ya_market/api_client.py", line 193, in __call_api
golem_worker_api-1  |     response_data = await self.request(
golem_worker_api-1  |                     ^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/ya_market/rest.py", line 268, in POST
golem_worker_api-1  |     return await self.request(
golem_worker_api-1  |            ^^^^^^^^^^^^^^^^^^^
golem_worker_api-1  |   File "/usr/local/lib/python3.11/site-packages/ya_market/rest.py", line 180, in request
golem_worker_api-1  |     raise ApiException(http_resp=r)
golem_worker_api-1  | ya_market.exceptions.ApiException: (500)
golem_worker_api-1  | Reason: Internal Server Error
golem_worker_api-1  | HTTP response headers: <CIMultiDictProxy('Content-Length': '236', 'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'Content-Type': 'application/json', 'Date': 'Wed, 28 Feb 2024 12:22:08 GMT')>
golem_worker_api-1  | HTTP response body: {"message":"Protocol error while terminating: Terminate Agreement [R-da5eb50ede2744456bdc551358ee7afd4b16ea50315ac7b6d91a5d343e00e1c8] GSB error: GSB failure: Net: error forwarding message: Establishing Tcp connection: Socket closed.."}

In yagna logs I keep seeing:

[2024-02-28T09:06:51.715+0000 ERROR ya_market::protocol::discovery] Error broadcasting offers: GSB failure: Broadcast failed: Unable to query neighbors: Request failed: Request timed out after 3000 ms
[2024-02-28T09:06:52.547+0000 ERROR ya_market::protocol::discovery] Error broadcasting unsubscribed offers: GSB failure: Broadcast failed: Unable to query neighbors: Request failed: Request timed out after 3000 ms
@stan7123
Copy link
Author

Wrong repository

@stan7123 stan7123 closed this as not planned Won't fix, can't repro, duplicate, stale Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stan7123