ROS2 Multicast works but nodes can't communicate or see each other over multiple machines
I'm trying to get a ROS2 system working across a few machines using some Raspbery Pi 4's. I'm on Ubuntu 22.04 LTS and am using ROS2 Humble. The kernel version on the host machine is 5.15.0-46-generic and the kernel version on the raspberry pi is 5.15.0-1013-raspi.
I install Ubuntu 22 server onto the Raspberry PI then ssh into it via WiFi to configure a user and install ROS2. I go through all the steps of setting up an environment, I make sure the setup.bash file is correctly sourced and that the machines share the same ROS_DOMAIN_ID (default in this case). I make no changes to the DDS implementation.
I then follow advice in the troubleshooting section of the documentation and I update my firewall to allow multicast UDP messages. I tested this using multicast send and multicast receive and confirmed that ros2 instances on the same machine can see each other as well as receive multicast UDP messages across the WiFi network from the Raspberry Pi. So in theory the instances of ROS2 across the two machines should be able to see each other but running ros2 run demo_nodes_cpp talker & ros2 run demo_nodes_cpp listener only works locally. Only the talker and listener on the Pi can see each other and the same goes for my main machine.
What I've Tried:
- Setting Static IPs - The Pi has a static IP that it will always be at, it seems to have made no difference other than making logging in by ssh more convenient.
- Change DDS Implementation - I followed advice here and attempted to change the DDS implementation on both machines to Cyclone but that makes no difference either. The user on the Pi is technically a different user so I thought this may work but no change.
- I've Reflashed the Pi and reinstalled ROS2 several times over - It has made no difference.
- Check that multicast is enabled - I've used
ifconfigto double check and usedufwto provide a route for multicast traffic as per the documentations suggestion. It's resulted in themulticasttool working but that's it. - Try an ethernet connection instead - I've connected a network switch to my host machine and the pi and a similar issues occurs. I can force the use of the ethernet connection and
multicastworks but regular nodes can't communicate at all.
All packages are the latest versions and up to date. Unsure what to attempt next, I've been through every suggestion I can find and it all points to getting multicast working then it should just work, so the fact that multicast is working but still unable to communicate across machines has gotten me stuck. There aren't really any error logs or messages for me to share because it's not producing errors, it's simply not working as intended.
Are all your hosts using a service to keep an precise time-of-day? systemd-timesyncd is the one most commonly used. Your subnet will need access to the internet (unless you are running your own NTP server locally.)
All hosts are using
timedatectl(which i believe is built on top of systemd-timed) to handle timezone configuration while an NTP server on the main host machine is connected to each pi. The actual subnet is an ethernet LAN with no internet access but the host machine does have internet access so that it is in time sync.I have considered disabling the hotplugging behaviour on the Pi's so that they can maintain a wifi connection while the ethernet connection is active however I'm not sure what difference that would make if they're being kept in sync with a machine that has an internet connection anyway.