Actionlib Delays when Using Multiple Clients/Servers

edit

Hi everyone,

I am observing some interesting behavior when using multiple actionlib clients and servers.

Here is my setup: I have one master process that maintains N SimpleActionClients. I have N separate ros nodes which each have an instance of the same SimpleActionServer (the same ros node is ran with different node names, and each action server is given a different name based on the node name). My use case involves running different scenarios and getting the results asynchronously, and trying to make sure each server is always working on something (as soon as I receive a completed goal, I send a new scenario to the server).

When N is 1, I find that a single execute callback on the server node takes some amount of time, let's call it X sec.

When N is 10, each server node takes roughly 1.2*X sec

When N is 20, each server node takes roughly 2*X sec

My intuition says that there should be some overhead for managing multiple actionlib clients like this, but I didn't expect that each individual execute callback is somehow affected by the number of active actionlib clients and/or servers. I find it also weird that I can run all of the servers in every test case, and still observe this behavior (Always running 20 servers, I still observe this behavior when I create 1, 10, and 20 clients and use them).

In order to know when a server has completed a goal, I am asking SimpleActionClient for its state and seeing if it is done. I cannot waitForResult since any of the currently active servers could return a result that I can process.

EDIT: I just want to add that I am running this on a machine with Ubuntu 14.04 that has 128GB of RAM and 32 cores, so computational resources aren't getting overtaxed, as far as I can measure. The system's cpu usage goes to ~60% when running 20 servers.

My main questions are:

Is this expected behavior that execute callbacks slow down with more active servers and clients?

Is there a way to remove these delays that are occurring with multiple clients?

I will provide any more clarifications as necessary. Thank you in advance for your insight and help with this issue!