feature: allow extending a python package in ament_python_install_package

[nadavelkabets]

Closes #514.

The most ament_cmake way to do this is to change up the way that ament_python_install_package works. For most things that are part of ament_cmake, the way that they work is that the call (to e.g. ament_export_libraries) doesn't do much work, and instead sets up a bunch of cmake variables, along with a hook. When ament_package is eventually called, it calls all of the hooks, and at that point the hook does the work based on the cmake variables. This is done this way so that ament_package has a full view of the environment, and avoids problems exactly like this.

For whatever reason, ament_python_install_package does not work like this at the moment. We should change ament_python_install_package to work in the more ament_cmake way, which will resolve this bug.

Following the suggestion from @clalancette,
calling ament_python_install_package now registers an environment hook and sets the parameters in environment variables, moving installation to the ament_package stage.

This allows to extend an existing package and thus allows to utilize rosidl_generate_interfaces and ament_python_install_package at the same time.

Related:

• Build error when using ament_python_install_package() and generated interfaces ros2/rosidl_python#141
• Add EXTEND_EXISTING argument to ament_python_install_package #517
• use ament_python_install_package when package is not installed ros2/rosidl_python#187

[rkent]

I've done an initial check of the code, and it generally looks good, except for one issue. As far as I can tell, you are not supporting AMENT_CMAKE_SYMLINK_INSTALL. Symlink install is probably incompatible with the concept of merging together multiple same-named python packages, but this need to merge is really rather rare, and symlink_install makes sense in the vast majority of packages that do not need this merger. It seems to me that the correct thing to do is to disable symlink install with a warning when multiple same-named packages are detected. Or did I miss something about how you supported this?

I have not actually tested yet, but will soon.

[nadavelkabets]

I've done an initial check of the code, and it generally looks good, except for one issue. As far as I can tell, you are not supporting AMENT_CMAKE_SYMLINK_INSTALL. Symlink install is probably incompatible with the concept of merging together multiple same-named python packages, but this need to merge is really rather rare, and symlink_install makes sense in the vast majority of packages that do not need this merger. It seems to me that the correct thing to do is to disable symlink install with a warning when multiple same-named packages are detected. Or did I miss something about how you supported this?

I kept the original package installation code without changes. To my knowledge, colcon changes the behavior of the install command depending on the AMENT_CMAKE_SYMLINK_INSTALL flag.
It appears like merging works correctly even with symlink install, due to the install command symlinking each file in the package recursively.

[rkent]

This is the code from the original that does not have an equivalent in your PR. (I matched every line of the old code with your new code, and this is the only difference that I found). I'll test it today though to confirm whether it works or not.

[rkent]

I've thoroughly checked the old code vs the new code, and confirmed that all of the old code has corresponding functionality in the new code, with one exception. I've also tested using a very simple combined msg/ directory with a python package of the same name, and everything works as I expect.

Concerning the exception code, I've compared results of the new code with old code for --symlink-install, and find that the new code works fine with the same result. As @nadavelkabets points out, the modifications to the install command are taking care of the symlinks. The only difference I could find is that in the old code, a symlink is created in the build/ directory, that is not present in the new code. But the end result in the /install directory is the same in both cases, so it is not clear to me why this symlink is needed. Trying to add that symlink would lead to a conflict between duplicate python packages and --symlink-install, so it is good that it is not needed.

But as @hidmic points out in PR #326 that created these symlinks, "There should be a Here be dragons sign in ament_cmake root README..." I'm afraid that fighting those dragons here is beyond my expertise. AFAICT that code is not needed. But perhaps @nadavelkabets you could give your own defense of why.

So with one nit, I've reviewed as thoroughly as I can and find no issues, so here's my gray checkmark.