failed to create a gRPC client for target, context deadline exceeded

[NinoFares]

Hi team,

We have some problems with our lab, when we try to do a gnmic get from a docker container outside of containerlab, we had the classic error:

target "172.20.20.5:57400" get request failed: failed to create a gRPC client for target "172.20.20.5:57400" : 172.20.20.5:57400: context deadline exceeded
target "172.20.20.5:57400" get request failed: failed to create a gRPC client for target "172.20.20.5:57400" : 172.20.20.5:57400: context deadline exceeded
Error: one or more requests failed

To debug we did a tcpdump, result:

172.17.0.4.40494 > 172.20.20.5.57400: Flags [S], cksum 0x6c5d (incorrect -> 0x4980), seq 1338585287, win 29200, options [mss 1460,sackOK,TS val 217125976 ecr 0,nop,wscale 7], length 0

So we followed https://containerlab.dev/cmd/tools/disable-tx-offload/#disable-tx-offload-command to disable tx offload. but same problem.

for information, we had this problem before and we fix it by doing:

ethtool --offload docker0 tx off

But since a little time, this doesn't work.

Thank you in advance for your help!

[hellt]

Hi,
this is a default docker security mechanism which doesn't allow communications between different docker networks.
Check this link (the Note element) for different solutions - https://learn.srlinux.dev/tutorials/programmability/json-rpc/ansible/#container

[NinoFares]

Yes, a SR Linux, with version 21.6.4

topology:
  nodes:
    srld3:
      kind: srl
      type: ixrd3
      image: ghcr.io/nokia/srlinux:21.6.4

It was working since 3/4 weeks now and it work from the containerlab/docker host.

[hellt]

Then try doing clab tools disable-tx-offload -c <gnmic container name>

[NinoFares]

Okey, so by doing clab tools disable-tx-offload -c <gnmic container name> on gnmic container works. I thought we had to do this command to srlinux container.

The problem now, we use a runner gitlab to deploy our ansible playbook on containerlab srlinux, how can we make this happen, if we have to do this command to each new container outside of our lab ? Any idea?

[hellt]

I don't remember how the tx offload propagates, the safest way would be to change your CI scripts to call the offload command for gnmic containers you spawn.
Maybe you can experiment and use this command on docker0 bridge and clab bridges to see if that can be done instead of changing the offload on the container interface.

And if you upgrade srlinux to 22.11 then you don't need to tune offload at all, as it has been done in the srlinux app itself

[NinoFares]

So I tried the solution of upgrading srlinux to 22.11, and it works! but I have to spawn my container to the same network as containerlab network. so I choose to create my lab on the default docker network.

Thank you for your help 😄!