How to setup bloom to generate a local deb, outside of ros.org buildfarm

asked 2014-09-10 13:29:35 -0600

Damien
203 ●10 ●16 ●18

Hi. This question is related to this one: http://answers.ros.org/question/17380...

Basically, what we need to do is to provide our ros packages to project partners and clients. But we do not want to distribute the source code for some of the nodes (some yes, and some no). And we don’t want either to appear yet in the official ROS buildfarm/jenkins/repo for project confidentiality required by some clients. And we will want in the future to add obfuscation of python code and license management for C++ code with LMX for example...

So we’ve been digging around the ROS release process. But things are not really clear to us yet.

Would you know how to do, or how would you advise us to do in order to generate such installers so that our clients can easily install it and that we can check release numbers, and send only compiled code. I think there are 2 ways:

The “easier” but not sure if it works well: generate a tar.gz of the catkin workspace after having cleaned it from source code…. Do you have experience on this? Do we need to remove the devel folder? if we remove the src folder this doesn't work, do we need to leave at least the config and lanch files? more?... Is this documented somewhere? Maybe we need to compile catkin with release flag?
The more complex but cleaner and more long-term: generating a debian and putting it on a server. I’ve seen that we can use Bloom to generate a debian automatically in GitHub. And then there is pull generated to populate the ROS buildfarm…
But then, where is it compiled? In local? By github? By ros.org?
How can we specify/include in the debian only compiled code and not source code? (or a mix)
Can we generate only one debian for a metapackage? or will we have to generate as many .debs as packages?
Do you know how to disable this pull to ros.org? and to github (or how to configure the deb to be generated on our git server?)
If we don’t publish it on ros.org , can we still get the track feature to manage release version and tags etc…
If/when we'll have generated the .deb, what kind of server would we need to implement to get a private server to be able to run apt-get? (or any similar method) Might PPA be the solution? Or can one call an apt-get on a private git server since the release debian will be there?
And finally, how/where does the obfuscation popsup in this process?

Thanks in advance for any advice and guidelines. You will have understood that we know more or less what we want, but have little idea of the available options and how to implement this... what a program :-)

For the question point 2, for a first package that in particular defines its own services, we ... (more)

edit retag flag offensive close merge delete

Comments

@Damien I haven't forgotten about this, I'm at ROSCon right now, so it might be a few days before I get to this. Sorry!

William ( 2014-09-12 16:44:58 -0600 )edit

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2014-09-14 10:53:29 -0600

William

17335 ●101 ●182 ●262 http://wjwwood.io/

I'll see if I can address your questions and then give a suggested approach:

The “easier” but not sure if it works well: generate a tar.gz of the catkin workspace after having cleaned it from source code…. Do you have experience on this? Do we need to remove the devel folder? if we remove the src folder this doesn't work, do we need to leave at least the config and lanch files? more?... Is this documented somewhere? Maybe we need to compile catkin with release flag?

When you build a catkin workspace, you can tell it to do make install. This can be done with catkin_make install or catkin_make_isolated --install, depending on the build tool you are using. The result will be an install folder along side your src, build, and devel folders. This install folder stands on its own, you can delete the src, build, and devel folders and it will still function. It is also relocatable (a feature that cross-compiler's frequently use), meaning you can move the folder and it should still work. Debian package generation does just this, it takes the result of bloom (source + debian configuration files) and builds then installs your packages into an install folder, then it packages up that install folder and that is what is distributed in the resulting .deb file.

The more complex but cleaner and more long-term: generating a debian and putting it on a server. I’ve seen that we can use Bloom to generate a debian automatically in GitHub. And then there is pull generated to populate the ROS buildfarm…

The bloom-release command will generate the required debian configuration files, and then open a pull request on github. The pull request is only required if you are going to have our build farm compile your packages into .deb files. This requires that your code is open source, so this will not be what you want. As @IsaacSaito suggested, this answer should help you manually build the .deb from the result of bloom without a pull request or our farm:

http://answers.ros.org/question/67345...

Then you can automate that process with a CI server like Jenkins or buildbot, see buildbot-ros:

https://github.com/mikeferguson/build...

But then, where is it compiled? In local? By github? By ros.org?

If you open the pull request it will be built on jenkins.ros.org and the result hosted on packages.ros.org (open source only).

How can we specify/include in the debian only compiled code and not source code? (or a mix)

If you are using our farm, the source code and binaries are both always provided in the result, there is no configuration, hence the requirement for open source.

Can we generate only one debian for a metapackage? or will we have to generate as many .debs as packages?

If you are using bloom, there will be one .deb per package and per metapackage (contains just the metapackage not the packages which belong ... (more)

edit flag offensive delete link

Comments

Thanks a lot William. We'll need a bit of time to digest all this and find our way to generate our packages. The buildbot-ros seems a good option... the checkinstall to generate a single deb also...

Damien ( 2014-09-16 04:12:55 -0600 )edit

William, I've came up to the conclusion that I first need to get catkin_make install produce the /install correctly. So, Can I/Should I/Where (adding a /install/src?) should I add the source code of a restricted number of my nodes to my future deb? ROS is only hosting it in github, right?

Damien ( 2014-09-18 10:36:32 -0600 )edit

I don't understand your comment. You don't need source code to ship the install space. If you want to include source code I would do it separately. ROS is not hosting anything on your behalf, if your code is on github it is because you put it there. I think I am just mis understanding the question.

William ( 2014-09-18 13:52:36 -0600 )edit

Goal is to ship a complete product to a client, with binary, docs and some source code. I dont want ros.org to host anything. But not sure about how to ship these src examples... So I though to config Cmakelist to generate in /install both binaries and copy source files; get deb with FPM and ship

Damien ( 2014-09-18 15:40:48 -0600 )edit

I referred to ROS just as a reference: in opt/ros there is no source code but the equivalent of \install. Apart, you provide source in github for us to compile. My option might be to make install, clean my catkin_ws folder from build devel and some src pckg, then get a FPM deb or a targz to ship.

Damien ( 2014-09-18 15:52:14 -0600 )edit

If your customers can use Debian packages, provide them with both source and binary packages.

joq ( 2014-09-18 16:10:43 -0600 )edit

@joq any hint on how to generate a source debian for ros? Bloom will only generate a binary package I think. no?

Damien ( 2014-09-18 16:49:05 -0600 )edit

The build farm uses the bloom git build package to generate source packages for each target, then uses them to generate the binaries. I don't know the details, but the information is all there.

joq ( 2014-09-18 17:08:36 -0600 )edit

see more comments

answered 2014-09-11 18:24:39 -0600

130s

10937 ●203 ●339 ●330 http://www.linkedin.co...

I'm sure some others have better, complete answer but here's what I know:

(1) To build ROS packages locally on Ubuntu, this answer by @William worked well for me.

(2) To automate process, as noted you could use Jenkins or any other CI tool. My colleague who has set up and is maintaining Jenkins recommends clound-based tool, like Travis CI. With its pro-plan you can upload private data as well with your private (paid) github organization.

edit flag offensive delete link

add a comment

How to setup bloom to generate a local deb, outside of ros.org buildfarm

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

How to setup bloom to generate a local deb, outside of ros.org buildfarm edit

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

How to setup bloom to generate a local deb, outside of ros.org buildfarm