ros2 docker multiple hosts

asked 2020-07-31 15:36:34 -0600

dawonn_haval
103 ●8 ●18 ●17

updated 2020-08-16 09:08:41 -0600

How can I get ROS2 to run in docker across multiple hosts reliably? I have two hosts, on an ethernet network:

Bridge networks do not support multicast
Overlay networks do not support multicast
macvlan networks cannot be accessed from the host machine (Workaround below)
Host networks partially work, but I only see a random subset of the nodes on the network. (Workaround below)
Weave network performance is very poor. (A simple topic that typically runs natively at 270Hz only manages to get up to 20Hz)

edit retag flag offensive close merge delete

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2020-08-12 16:55:53 -0600

dawonn_haval
103 ●8 ●18 ●17

updated 2020-08-16 09:08:34 -0600

There appear to be two potential workarounds:

HOST:

If you want to use host networking, you also much share the pid space so that each DDS node can generate a unique GUID. ( All thanks go to @ruffsl, more detail here )

docker run --rm -d --net=host --pid=host <IMAGE>

MACVLAN:

This workaround seems to work more often than not but sometimes the container needs to be restarted before it will see all nodes on the network.

docker network create -d macvlan \
    --subnet=192.168.1.0/24 \
    --gateway=192.168.1.1 \
    --ip-range=192.168.1.64/27 \
    --attachable \
    -o parent=enp8s0 \
    macnet

sudo ip link add macnet-shim link eth0 type macvlan  mode bridge
sudo ip addr add 192.168.1.64/27 dev macnet-shim
sudo ip link set macnet-shim up

docker run --rm -d --net=macnet <IMAGE>

edit flag offensive delete link

add a comment

answered 2020-08-12 16:33:06 -0600

ruffsl
1094 ●19 ●44 ●40

It looks like multicast support has been a long open issue for docker:

https://github.com/moby/libnetwork/is...
https://stackoverflow.com/questions/5...

As suggested in one of the stackoverflow answers, If you need multicast support, you could use the host network driver, as ROS2/DDS is capable of running multiple nodes/participants per host/container and could share the network interface. If the docker host's aren't on the same subnet then that would be a separate issue for DDS discovery and NAT translation over a LANs that still supports multicast. In any case, I'd also recommend enabling SROS2 if the hosts are operating over an unsecure network.

edit flag offensive delete link

Comments

The hosts are on the same subnet, and I initially started with 'host' networking but it seems to be very unreliable. Each node only sees a random subset of the nodes on the network, as if some of the multicast messages are not being routed correctly. Once a pub/sub connection is established, the connection is reliable however.

Also interesting, the ros2 multicast sent|receive tool almost always works with host networking.

As a brief status update; I am having some limited success with a macvlan network configuration. I was able to create a bridge between the docker macvlan subnet and the host machine. It's not super reliable yet though, I am often unable to establish a connection to some topics even though ros2 topic list & ros2 nodes list look good. Usually I am able to restart a container a couple of times to get things connected.

dawonn_haval ( 2020-08-12 16:45:23 -0600 )edit

It may not be the case here, but some DDS venders may fall back to shared memory transport if they think another participant is from the same host, or colliding GUID, thus complicating the use of the host network driver for such scenarios. See this question/answer for more details: https://answers.ros.org/question/2968...

ruffsl ( 2020-08-13 19:12:53 -0600 )edit

@dawonn_haval were you able to resolve this? I'm currently observing an identical problem in terms of random subsets of visible nodes with 'host networking'. Many thanks!

aaron98 ( 2020-12-02 16:12:00 -0600 )edit

Kinda sorta, yes: https://answers.ros.org/question/3584...

After assigning each docker container to use --net=host --pid=host I can now establish connections reliably. Both are necessary for the RMW to generate a unique GUID.

dawonn_haval ( 2020-12-03 08:49:21 -0600 )edit

add a comment

ros2 docker multiple hosts

2 Answers

Comments

Question Tools

Stats

Related questions

ros2 docker multiple hosts edit

2 Answers

Comments

Question Tools

Stats

Related questions

ros2 docker multiple hosts