Issues with apache2 and updating rosbridge_server from 0.11.3 to 0.11.4

edit

Hi All

Our setup: Ros melodic 1.14.5 , running in ubuntu bionic 18.04.1 LTS, apache2 2.4.29, rosbridge_server 0.11.3 (working), rosbridge_server 0.11.4-0.11.5 (not working)

We currently implement rosbridge_server to serve a couple of websockets that we use to connect the front end of our application to the ROS backend. We use apache as a reverse proxy to route the requests from an specific outer endpoint to our locally served websockets (e.g. ProxyPass /api ws://localhost:9090).

An example of our apache2 setup is seen below

<VirtualHost *:80>
  ProxyPreserveHost On
  ProxyRequests Off
  <Location /api>
    ProxyPass ws://localhost:9090 retry=0 keepalive=On
  </Location>
</VirtualHost>

We realized that upon updating rosbridge_server to 0.11.4, which changes the websocket backend in python to use the Autobahn library, our apache proxying stopped working and we could not get access to the web socket from the outer world. The interesting thing is that the websocket itself works correctly and we can have access to it if we connect directly, bypassing apache (e.g can connect to ws://<ip_of_machine>:9090).

Our launch scripts for the rosbridge websocket can be seen below

<launch>

  <include ns="customer" file="$(find rosbridge_server)/launch/rosbridge_websocket.launch">
    <!-- use default port -->
    <arg name="port" value="9090" />
    <arg name="topics_glob" default="[*]" />
    <arg name="services_glob" default="[*]" />
    <arg name="params_glob" default="[]" />
  </include>

</launch>

We have been throwing arrows in the dark for a while trying to see what has changed. One thing we have noticed is that with rosbridge_server 0.11.3, launching a rosbridge_websocket node binds a python process to IPV4 and IPV6 addresses, as seen by checking with the command below

user@webvenice: : sudo ss -lptn 'sport = :9090'

State             Recv-Q             Send-Q                           Local Address:Port                           Peer Address:Port                                                             
LISTEN            0                  128                                    0.0.0.0:9090                                0.0.0.0:*                 users:(("python",pid=24777,fd=14))             
LISTEN            0                  128                                       [::]:9090                                   [::]:*                 users:(("python",pid=24777,fd=15))

In contrast, when testing with 0.11.4 or 0.11.5 we see that only that there is a single python process binding IPV4 addresses

user@webvenice: sudo ss -lptn 'sport = :9090'

 State             Recv-Q        Send-Q                           Local Address:Port                  
Peer Address:Port                      LISTEN            0                  50                        
0.0.0.0:9090                                0.0.0.0:*                 users:(("python",pid=24777,fd=14))

We are sure that apache2 works and routes correctly the outer request but when routing to the local network it has trouble, as seen in the logs below

[Mon Apr 20 14:11:38.952259 2020] [proxy:error] [pid 601:tid 140657578448640] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:38.952292 2020] [proxy_wstunnel:error] [pid 601:tid 140657578448640] [client ::1:62928] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:39.960145 2020] [proxy:error] [pid 601:tid 140657477801728] (111)Connection refused: AH00957: WS: attempt to connect to ...