The site is read-only. Please transition to use Robotics Stack Exchange
| ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange |
 | 1 | initial version |
What exactly I'm trying to achieve here ?
I have a git repo with me that takes in the camera images and estimates user pose, saves data and trains a classifier. I want to setup a build farm for this repo, perform continuous integration and post build steps. Oh ! and also, please let me know if this is possible with the ros buildfarm. Provide documentation that clearly explains what to do if someone like me wants to setup a buildfarm.
Depending on your end-goal, it may be easier to begin either without ros_buildfarm or using ros_buildfarm without a full buildfarm deployment:
A. Running your repositories tests.
If all you want is basic continuous testing, setting up an entire buildfarm might be overkill. There are examples of setting up ROS repositories in Travis. This repository provides a sample .travis.yml and information for doing exactly that. It might be a bit stale and it's not something I've worked with, but once set up, it would run your tests without the need for running your own buildfarm.
B. If you also want to build binary packages for your repository, it's still possible to do without setting up a buildfarm deployment, but you will need to create suitable rosdistro and ros_buildfarm_config configuration files so that the ros_buildfarm scripts can be run on your local machines.
C. Of course, you can also set up a complete buildfarm cluster to perform run ros_buildfarm operations. This is probably the best option if you're going to be adding more custom packages down the road or building for multiple rosdistros. It's also the course of action you're already on.
I thought of having a Jetson Tx2 as my agent and a Raspberry Pi 3 B as my repo but I'm willing to change this to two more VMs having same specs as my master.
I mentioned in your previous question that the buildfarm_deployment repository was originally written for x86 hosts only and that I know it doesn't support arm64 hosts out of the box and it probably won't work on a Raspberry Pi running Raspbian Stretch or even Ubuntu Xenial. I have deployed arm64 agents but I had to do the last 40% of the setup by hand which requires some experience and understanding of what the Puppet config management is doing for you so it won't be straightforward if you're new to Puppet and ros_buildfarm. You also won't be able to build for AMD64 (x86_64) targets with an arm64 agent, though you can build for arm64 with an AMD64 host using qemu, which the buildfarm_deployment sets up by default.
I would recommend using AMD64 virtual machines although they don't all need to be as powerful as your Jenkins master. This document has some recommended instances sizes using AWS instances sizes as a reference.
I understood that I have to replace the values for jenkins::slave::ui_user and jenkins::slave::ui_pass with my jenkins username & password and I changed it. If this wrong please advise me what exactly I should be doing here.
By default, these fields should match with the Jenkins admin user and password hash in the hiera config for the master. If you do change these values you need to make sure that a user with the credentials you chose exists _after_ running the reconfigure.bash script for the master host. If you want to use the same user for both admin and agent processes but don't want to keep the default password, you can use a tool like https://github.com/gavanderhoorn/jenkins_bcrypt_pw_util or copy the password hash from the configuration of a Jenkins instance that has your desired password.
2.3. Lines 17-23, I would like to know how to actually do this. First up, I generated a ssh keygen based on this link, which should generate two keys, a public and a private one. In line-21, should I copy paste the public key or the private key, I generated locally?
That location should hold the public key, the keys in common.yaml are going to be inserted into the ~/.ssh/authorized_keys files of the specified users. I've got an in-progress PR to update the documentation comments, you can read the updated version and let me know if it's helpful.
You'll want to paste the private key in the jenkins::ssh_private_key field of master.yaml (example). That private key will be added to specific jobs by jenkins to allow the running executor to publish artifacts via SSH.
| | 2 | No.2 Revision |
What exactly I'm trying to achieve here ?
I have a git repo with me that takes in the camera images and estimates user pose, saves data and trains a classifier. I want to setup a build farm for this repo, perform continuous integration and post build steps. Oh ! and also, please let me know if this is possible with the ros buildfarm. Provide documentation that clearly explains what to do if someone like me wants to setup a buildfarm.
Depending on your end-goal, it may be easier to begin either without ros_buildfarm or using ros_buildfarm without a full buildfarm deployment:
A. Running your repositories tests.
If all you want is basic continuous testing, setting up an entire buildfarm might be overkill. There are examples of setting up ROS repositories in Travis. This repository provides a sample .travis.yml and information for doing exactly that. It might be a bit stale and it's not something I've worked with, but once set up, it would run your tests without the need for running your own buildfarm.buildfarm. MoveIt! has a CI repository with an up-to-date Travis config as well. (Hat tip to @Geoff in the comments).
B. If you also want to build binary packages for your repository, it's still possible to do without setting up a buildfarm deployment, but you will need to create suitable rosdistro and ros_buildfarm_config configuration files so that the ros_buildfarm scripts can be run on your local machines.
C. Of course, you can also set up a complete buildfarm cluster to perform run ros_buildfarm operations. This is probably the best option if you're going to be adding more custom packages down the road or building for multiple rosdistros. It's also the course of action you're already on.
I thought of having a Jetson Tx2 as my agent and a Raspberry Pi 3 B as my repo but I'm willing to change this to two more VMs having same specs as my master.
I mentioned in your previous question that the buildfarm_deployment repository was originally written for x86 hosts only and that I know it doesn't support arm64 hosts out of the box and it probably won't work on a Raspberry Pi running Raspbian Stretch or even Ubuntu Xenial. I have deployed arm64 agents but I had to do the last 40% of the setup by hand which requires some experience and understanding of what the Puppet config management is doing for you so it won't be straightforward if you're new to Puppet and ros_buildfarm. You also won't be able to build for AMD64 (x86_64) targets with an arm64 agent, though you can build for arm64 with an AMD64 host using qemu, which the buildfarm_deployment sets up by default.
I would recommend using AMD64 virtual machines although they don't all need to be as powerful as your Jenkins master. This document has some recommended instances sizes using AWS instances sizes as a reference.
I understood that I have to replace the values for jenkins::slave::ui_user and jenkins::slave::ui_pass with my jenkins username & password and I changed it. If this wrong please advise me what exactly I should be doing here.
By default, these fields should match with the Jenkins admin user and password hash in the hiera config for the master. If you do change these values you need to make sure that a user with the credentials you chose exists _after_ running the reconfigure.bash script for the master host. If you want to use the same user for both admin and agent processes but don't want to keep the default password, you can use a tool like https://github.com/gavanderhoorn/jenkins_bcrypt_pw_util or copy the password hash from the configuration of a Jenkins instance that has your desired password.
2.3. Lines 17-23, I would like to know how to actually do this. First up, I generated a ssh keygen based on this link, which should generate two keys, a public and a private one. In line-21, should I copy paste the public key or the private key, I generated locally?
That location should hold the public key, the keys in common.yaml are going to be inserted into the ~/.ssh/authorized_keys files of the specified users. I've got an in-progress PR to update the documentation comments, you can read the updated version and let me know if it's helpful.
You'll want to paste the private key in the jenkins::ssh_private_key field of master.yaml (example). That private key will be added to specific jobs by jenkins to allow the running executor to publish artifacts via SSH.
| | 3 | No.3 Revision |
What exactly I'm trying to achieve here ?
I have a git repo with me that takes in the camera images and estimates user pose, saves data and trains a classifier. I want to setup a build farm for this repo, perform continuous integration and post build steps. Oh ! and also, please let me know if this is possible with the ros buildfarm. Provide documentation that clearly explains what to do if someone like me wants to setup a buildfarm.
Depending on your end-goal, it may be easier to begin either without ros_buildfarm or using ros_buildfarm without a full buildfarm deployment:
A. Running your repositories tests.
If all you want is basic continuous testing, setting up an entire buildfarm might be overkill. There are examples of setting up ROS repositories in Travis. This repository provides a sample .travis.yml and information for doing exactly that. It might be a bit stale and it's not something I've worked with, but once set up, it would run your tests without the need for running your own buildfarm. MoveIt! has a CI repository with an up-to-date Travis config as well. (Hat tip to @Geoff in the comments).
B. If you also want to build binary packages for your repository, it's still possible to do without setting up a buildfarm deployment, but you will need to create suitable rosdistro and ros_buildfarm_config configuration files so that the ros_buildfarm scripts can be run on your local machines.
C. Of course, you can also set up a complete buildfarm cluster to perform run ros_buildfarm operations. This is probably the best option if you're going to be adding more custom packages down the road or building for multiple rosdistros. It's also the course of action you're already on.
I thought of having a Jetson Tx2 as my agent and a Raspberry Pi 3 B as my repo but I'm willing to change this to two more VMs having same specs as my master.
I mentioned in your previous question that the buildfarm_deployment repository was originally written for x86 hosts only and that I know it doesn't support arm64 hosts out of the box and it probably won't work on a Raspberry Pi running Raspbian Stretch or even Ubuntu Xenial. I have deployed arm64 agents but I had to do the last 40% of the setup by hand which requires some experience and understanding of what the Puppet config management is doing for you so it won't be straightforward if you're new to Puppet and ros_buildfarm. You also won't be able to build for AMD64 (x86_64) targets with an arm64 agent, though you can build for arm64 with an AMD64 host using qemu, which the buildfarm_deployment sets up by default.
I would recommend using AMD64 virtual machines although they don't all need to be as powerful as your Jenkins master. This document has some recommended instances sizes using AWS instances sizes as a reference.
I understood that I have to replace the values for jenkins::slave::ui_user and jenkins::slave::ui_pass with my jenkins username & password and I changed it. If this wrong please advise me what exactly I should be doing here.
By default, these fields should match with the Jenkins admin user and password hash in the hiera config for the master. If you do change these values you need to make sure that a user with the credentials you chose exists _after_ running the reconfigure.bash script for the master host. If you want to use the same user for both admin and agent processes but don't want to keep the default password, you can use a tool like https://github.com/gavanderhoorn/jenkins_bcrypt_pw_util or copy the password hash from the configuration of a Jenkins instance that has your desired password.
2.3. Lines 17-23, I would like to know how to actually do this. First up, I generated a ssh keygen based on this link, which should generate two keys, a public and a private one. In line-21, should I copy paste the public key or the private key, I generated locally?
That location should hold the public key, the keys in common.yaml are going to be inserted into the ~/.ssh/authorized_keys files of the specified users. I've got an in-progress PR to update the documentation comments, you can read the updated version and let me know if it's helpful.
You'll want to paste the private key in the jenkins::ssh_private_key field of master.yaml (example). That private key will be added to specific jobs by jenkins to allow the running executor to publish artifacts via SSH.
1.1. According to the instructions here, I left lines 1-6, unchanged. But please advise me if I have to change anything here. An example other than the one that I'm already using will be very much appreciated.
Lines 1-5 should in general not be changed. Doing so would interfere with the desired behavior of the buildfarm cluster.
1.2. Next up would be lines 7-95. I would like to know whether I have create another ssh key pair or use the one from the key pair used with the common.yaml file. Also, please let me know if I have to use a different ssh key pair, should I create it in the same way I did for the common.yaml keys.
The Jenkins-agent key fields: jenkins-agent::gpg_key_id, jenkins-agent::gpg_private_key, and jenkins-agent::gpg_public_key are fields that you would want to change if you were handling a production buildfarm and wanted to provide secure packages for other users, but for testing it is easiest to leave both the gpg key pair and key ID as they are.
1.3. Should I just leave [lines 97-117] as it is ? If the answer is no, could you please explain it to me why are we having this and what should I do to make it fit for my package ? 1.4. The last instruction given basically tells me to change the reprepro_config, which I have no idea on how to edit. Do I have to change lines 120 - 131 ? Exhaustive answers are very much appreciated.
reprepro is the tool that the ROS buildfarm uses for managing its debian package repository. In order to facilitate common buildfarm actions we use a repository containing scripts for our common operations called reprepro-updater. The only changes that I would advise making would be to trim the platform architectures and operating system suites that you want to support.
The jenkins-agent::reprepro_updater_config and jenkins-agent::reprepro_config fields use two different formats and are used in different buildfarm situations. For straightforward deployments the two configurations should have the same information in different formats.
For example, if you only want to build for ubuntu xenial and the amd64, armhf, and arm64 architectures you would remove all other distros for each repository (ubuntu_building, ubuntu_testing, and ubuntu_main) and list only your desired platforms and source in the architectures section. An example of the ubuntu_building section only:
[ubuntu_building]
architectures: amd64 arm64 armhf source
distros: xenial
repository_path: /var/repos/ubuntu/building
signing_key: E5944336
upstream_config: /home/jenkins-agent/reprepro_config
The final reprepro::configwould then receive the same modifications:
'/home/jenkins-agent/reprepro_config/ros_bootstrap.yaml':
ensure: 'present'
content: |
name: ros_bootstrap
method: http://repos.ros.org/repos/ros_bootstrap
suites: [xenial]
component: main
architectures: [amd64, arm64, armhf, source]
# filter_formula: Package (% python* )
# needed until we deploy the gpg key
verify_release: blindtrust
ROS Answers is licensed under Creative Commons Attribution 3.0 Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.