How to structure & link automated tests
This question is about the structure of automated tests in a typical ROS/Catkin work space. The project has the following folder/file structure for the unit and node level tests:
Only the following files are executable: node.py
, test-correct-response.py
It is important to keep in mind that we use Python 3 in all the shown Python files and some generated message files need to be available at the runtime of the tests (both unit and node tests).
We were not able to get this configuration to run like one would expect that. And we tried a lot. -.- Out main problems were that the unit tests were not executed with Python 3, but rather with Python 2. They also did not have access to the generated message files from other nodes. Further, our folder structure seemed to be problematic. Further, the usage of unittest.TestSuite
was not working. Reading this and the linked sites did not help with those difficulties.
My questions
- How does
CMakeLists.txt
need to be changed? - How does
package.xml
need to be changed? - Do other files need to be changed?
- Can we then execute all tests simply with
catkin_make run_tests
?
Appendix: File contents
node.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import rospy
if __name__ == '__main__':
rospy.init_node('my_node')
rospy.logerr("some awesome output")
rospy.signal_shutdown()
test-correct-response.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import unittest
class TestBareBones(unittest.TestCase):
def test_one_equals_one(self):
self.assertEqual(1, 1, "1 != 1")
class MyTestSuite(unittest.TestSuite):
def __init__(self):
super().__init__()
self.addTest(TestBareBones())
if __name__ == '__main__':
import rostest
rostest.rosrun('first_package', 'my_bare_bones_test', MyTestSuite)
test-correct-response.py
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<launch>
<node name="my_awsome_node" pkg="first_package" type="node.py" output="screen" />
<test test-name="nodetest_1" pkg="first_package" type="test-correct-response.py" />
</launch>
test-outputs.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import unittest
class CaseA(unittest.TestCase):
def runTest(self):
self.assertFalse(8 == 1)
class CaseB(unittest.TestCase):
def runTest(self):
self.assertEqual(1, 1)
class MyTestSuite(unittest.TestSuite):
def __init__(self):
super().__init__()
self.addTest(CaseA())
self.addTest(CaseB())
if __name__ == '__main__':
import rostest
rostest.rosrun('first_package', 'test 1', MyTestSuite)
CMakeLists.txt
Pretty much like generated by catkin_create_pkg
.
package.xml
Pretty much like generated by catkin_create_pkg
.
Just an observation: ROS - even Kinetic & Lunar - officially supports Python 2 only, so mixing and matching with Python 3 is going to be messy at best.
This is going to change ( https://github.com/ros-infrastructure... ), but for now I believe that is the situation.
Packages not being available (or only in one of the two runtimes), your tests getting executed with Python 2 all points to issue with Python 2 vs 3 and mixing that. It can work, but you can run into issues like this.
Yes, Python 2/3 mixing can be problematic. But is there really no simple solution to this? I initially thought it would be possible simply by changing some line in the
CMakeLists.txt
file. Is there a macro for doing this?avoid using Python 3? You hard-code the
python3
binary as your interpreter in all your Python scripts.Unfortunately, Python3 support in ROS1 is still very much a work in progress. For current discussions on the subject see: https://github.com/ros-infrastructure...
Take a look at the discussion in the issue I linked in my first comment. Work towards being able to use Python 3 and Python 2 is ongoing, but certainly not "simple" at the moment.
I agree with @gvdhoorn that current best practice is to make your Python code "bilingual"; test it now using Python2; then hope for the best when practical Python3 testing tools become available.
Okay, thank you all for the linked Github PR. I will leave this open for now but we will probably use bilingual or Python 2 only code. It is sad to see ROS being that slow. :(