EGM dropping on long-running tasks
We're running a fairly standard MoveIt stack on ROS Noetic with abb_robot_driver and an ABB IRB 2600 robot. We can control the robot quite well using a velocity_controllers/JointTrajectoryController, both through rviz and using the MoveIt move_group stack.
However, we are attempting to do some long-running (manufacturing-related) tasks. During those, the EGM connection will almost always drop out at some point. When that happens, no errors are reported either on the robot side (looking at RobotStudio and the pendant) or in ROS (including all logs).
We have not been able to identify a pattern for when this happens, so it's been extremely hard to debug. It does not happen at a set time after either EGM activation or into the long-running tasks, even sometimes happening fairly early in them. It does not appear linked to joint configurations or location in the workspace. We have have explicitly run trajectories throughout the workspace and seen the same, semi-random timing of the EGM dropout.
Has anyone seen similar behavior? Any ideas for a fix?
2023_03_09 Update: We talked with ABB and upgraded the robot controller to a newer version, which had fixes for some EGM bugs. However, we are still getting seemingly random EGM dropouts. We're still working on it, so I'll hopefully report back with what we figure out as a fix.
2023_06_28 Update: We believe that we have found the problem and the fix. We were setting a variety of EGM settings via service calls. While the changes were reflected if we subsequently queried the settings via service, it appears that they were not actually getting set on the ABB robot controller. Once we changed the values via RobotStudio, our drops went away.
I would probably suggest you reach out to ABB directly.
They'll likely want to know whether the same problem happens with "just EGM" (so no ROS), but it's unlikely anyone here has an idea about what's going on exactly.
EGM is ABB's product, which they should be able to support. ROS isn't their product, so they likely won't support that, but anything interacting with EGM needs additional software, so that's not something you can avoid.
Any update here?
Yes, we think. I'll add details above in the original post.
ah.
So the services you mention only change settings for that particular session or 'run'. They don't persist.
For anything where the 'defaults' would need to be updated you'd indeed have to update the settings using RobotStudio (or directly on the pendant).
Btw: I would recommend you post your last update as an answer and then accept it as the answer.
+100 if you could add an / some example(s) of settings you had to change using RobotStudio. That would be very valuable for future readers.
Right. We were setting (and still do set) them via service call prior to every run. The cost of doing so seemed low relative to the length of time our trajectories take to execute.