-
Notifications
You must be signed in to change notification settings - Fork 10
Docker
For docker-based infrastructure it would be great if we could monitor connections to and from all docker instances from either the host system or one 'connbeat' docker container.
As docker provides some isolation we'd need some elevated permissions to collect connection information.
To find out which other peers are on the docker host, you'll need access to the docker daemon. A common way to achieve this (also employed by for example https://github.com/gliderlabs/registrator) is by mounting the docker hosts' /var/run/docker.sock as a volume.
Docker has 2 networking modes: host networking and bridge networking.
When using host networking, we can see all connections without any further privileges. We would, however, need some mechanism to find out to which docker image the connection belongs. This would require inode-walking like connbeat already does, but across all docker containers. This is probably possible through /var/run/docker.sock, but might be expensive.
I expect host networking to be rare, however, because it means no 2 docker containers on the same host can listen on the same address.
When using bridge networking, each container gets its own networking namespace.
There are 2 ways to collect connection information: via /proc or by entering each containers' namespace:
You can enter a docker instance's networking namespace and inspect the connections using nsenter (or a more low-level equivalent).
Doing this requires the CAP_SYS_ADMIN capability, which is granted to root and processes or docker containers which are given this capability.
To find the docker networking namespaces from within docker we might need to mount /var/run/docker/netns, and to associate them using /var/run/docker.sock again.
Alternative to switching to the networking namespace ourselves, we could also docker exec cat /proc/net/tcp for each container. This requires only /var/run/docker.sock, and only assumes cat is available in each docker image.
To share the connection information with StackState, we could use the 'regular' connbeat format and extend the 'beat' field with the docker instance id. This should give the StackState side of things sufficient context to assemble a unified topology view.
One caveat might be that several docker hosts might use overlapping internal IP addresses (e.g. 172.17.0.2), the StackState side of things would have to know how to disambiguate based on docker id.