Why does catkin build not consistently do the same thing?

edit

I'm only going to answer this part of your question:

Is there some way to build my packages that is determinstic/repeatable?

tl;dr: if you don't state your dependencies correctly, Catkin is free to start building pkgs, which may in fact depend on each other, in parallel, leading to compiler errors and ultimately failed and aborted builds.

As the exact point in time where builds get aborted is (almost) stochastic (due to scheduling, CPU and IO performance, etc), and dependents typically will not need everything from dependencies, partial builds of dependencies may result in successful builds of (some) dependents.

This leads to successive builds appearing to make progress, as more-and-more dependents will find their dependencies where they expect them, leading to those builds succeeding, leading to their dependents being able to start their build. Etc, etc.

With enough iterations, the whole graph of dependencies gets built, until finally all dependencies are built and there are no more failures.

As to the question: always running catkin_make -j1 could help make things fail at almost the same place, however it's no guarantee, as other factors may still cause slight ordering differences in operations.

Long version: the Catkin side to this is already largely deterministic I believe. It's the other parts which may not be.

Catkin performs two main tasks before it asks CMake to build everything:

1. discover all packages in your src space and construct a graph where the vertices are the packages and the edges encode dependencies (ie: if pkg A depends on B, there will be an edge between the vertices representing A and B expressing that dependency)
2. run a topological sort over that graph, to determine the order in which everything should be built which satisfies those dependencies

Your packages state their dependencies using build_depends in their manifests, which Catkin parses and then uses to build the graph. After the sort, it asks CMake to start the build by building the packages which:

1. have no dependencies (rare),
2. for which all dependencies are already built,
3. depend only on packages present in an underlay (or its underlay, ad infinitum)
4. do not depend on each other

for all of those, the order in which they are built does not really matter, as pkgs in category 1 don't depend on anything, in category 2 and 3 have all their dependencies ready and in category 4 cannot fail when built at the same time as they don't need anything from each other.

After the first layer of packages has been built, Catkin moves up to the next, which will be packages which depend on (some, or all) of the packages it just built. This process then repeats, until the "top" package(s) is (are) built.

Note: only the build_depend entries in your package.xml are taken into account, not the setup.py or CMakeLists.txt (actually, there's also build_export_depend et al., but I'll ignore those here, they don't significantly change the explanation).

Now ... (more)