How to structure & link automated tests

asked 2018-01-24 07:54:19 -0600

felixd gravatar image

This question is about the structure of automated tests in a typical ROS/Catkin work space. The project has the following folder/file structure for the unit and node level tests:

image description

Only the following files are executable: node.py, test-correct-response.py

It is important to keep in mind that we use Python 3 in all the shown Python files and some generated message files need to be available at the runtime of the tests (both unit and node tests).

We were not able to get this configuration to run like one would expect that. And we tried a lot. -.- Out main problems were that the unit tests were not executed with Python 3, but rather with Python 2. They also did not have access to the generated message files from other nodes. Further, our folder structure seemed to be problematic. Further, the usage of unittest.TestSuite was not working. Reading this and the linked sites did not help with those difficulties.

My questions

  1. How does CMakeLists.txt need to be changed?
  2. How does package.xml need to be changed?
  3. Do other files need to be changed?
  4. Can we then execute all tests simply with catkin_make run_tests?

Appendix: File contents

node.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import rospy

if __name__ == '__main__':
    rospy.init_node('my_node')
    rospy.logerr("some awesome output")
    rospy.signal_shutdown()

test-correct-response.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import unittest

class TestBareBones(unittest.TestCase):
    def test_one_equals_one(self):
        self.assertEqual(1, 1, "1 != 1")

class MyTestSuite(unittest.TestSuite):
    def __init__(self):
        super().__init__()
        self.addTest(TestBareBones())

if __name__ == '__main__':
    import rostest
    rostest.rosrun('first_package', 'my_bare_bones_test', MyTestSuite)

test-correct-response.py

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<launch>

    <node name="my_awsome_node" pkg="first_package" type="node.py" output="screen" />

    <test test-name="nodetest_1" pkg="first_package" type="test-correct-response.py" />

</launch>

test-outputs.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import unittest

class CaseA(unittest.TestCase):
    def runTest(self):
        self.assertFalse(8 == 1)

class CaseB(unittest.TestCase):
    def runTest(self):
        self.assertEqual(1, 1)

class MyTestSuite(unittest.TestSuite):
    def __init__(self):
        super().__init__()
        self.addTest(CaseA())
        self.addTest(CaseB())

if __name__ == '__main__':
    import rostest
    rostest.rosrun('first_package', 'test 1', MyTestSuite)

CMakeLists.txt

Pretty much like generated by catkin_create_pkg.

package.xml

Pretty much like generated by catkin_create_pkg.

edit retag flag offensive close merge delete

Comments

2

Just an observation: ROS - even Kinetic & Lunar - officially supports Python 2 only, so mixing and matching with Python 3 is going to be messy at best.

This is going to change ( https://github.com/ros-infrastructure... ), but for now I believe that is the situation.

gvdhoorn gravatar image gvdhoorn  ( 2018-01-24 08:06:43 -0600 )edit

Packages not being available (or only in one of the two runtimes), your tests getting executed with Python 2 all points to issue with Python 2 vs 3 and mixing that. It can work, but you can run into issues like this.

gvdhoorn gravatar image gvdhoorn  ( 2018-01-24 08:07:31 -0600 )edit

Yes, Python 2/3 mixing can be problematic. But is there really no simple solution to this? I initially thought it would be possible simply by changing some line in the CMakeLists.txt file. Is there a macro for doing this?

felixd gravatar image felixd  ( 2018-01-24 09:04:38 -0600 )edit

But is there really no simple solution to this?

avoid using Python 3? You hard-code the python3 binary as your interpreter in all your Python scripts.

gvdhoorn gravatar image gvdhoorn  ( 2018-01-24 11:18:44 -0600 )edit

Unfortunately, Python3 support in ROS1 is still very much a work in progress. For current discussions on the subject see: https://github.com/ros-infrastructure...

joq gravatar image joq  ( 2018-01-24 11:19:20 -0600 )edit

Take a look at the discussion in the issue I linked in my first comment. Work towards being able to use Python 3 and Python 2 is ongoing, but certainly not "simple" at the moment.

gvdhoorn gravatar image gvdhoorn  ( 2018-01-24 11:20:40 -0600 )edit
1

I agree with @gvdhoorn that current best practice is to make your Python code "bilingual"; test it now using Python2; then hope for the best when practical Python3 testing tools become available.

joq gravatar image joq  ( 2018-01-24 11:31:26 -0600 )edit

Okay, thank you all for the linked Github PR. I will leave this open for now but we will probably use bilingual or Python 2 only code. It is sad to see ROS being that slow. :(

felixd gravatar image felixd  ( 2018-01-25 02:50:47 -0600 )edit