[features] ImportError: DLL load failed with pybind11 and PCL

[adishavit]

I'm using pybind11 to create a Python wrapper for a small C++ class.
I'm getting the following error when importing the DLL (running python -v to show Traceback):

>>> from a_py import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing a_py: The specified module could not be found.

The class has just two members - types from the PCL library:

class A
{
   pcl::NormalEstimation< pcl::PointXYZ, pcl::Normal> normalEstimation_;
   pcl::PointCloud<pcl::Normal>::Ptr normals_;
};

If I remove the first member I can successfully import the module from the DLL.

This is the pybind11 code:

namespace py = pybind11;

PYBIND11_MODULE(a_py, m)
{
   py::class_<A>(m, "A");
}

PCL is found using CMake: find_package(PCL REQUIRED)

This happens on both Windows and Linux with the latest PCL (1.10) and older versions too.

(Adding #define PCL_NO_PRECOMPILE before the #includes does not help.)
[also posted on SO]

Your Environment (please complete the following information):

• OS: Windows 10 (but similar issues on Linux)
• Compiler: MSVC 2019
• PCL Version1.10.0

Possible Solution

Based on a SO suggestion I ran the Dependencies app on both DLLs (with and without the offending line).
The main difference was that the offending DLL depends on c:\Program Files\PCL 1.10.0\bin\pcl_common.dll. Strangely enough, it seems to only rely on one function: void __cdecl pcl::console::print(enum pcl::console::VERBOSITY_LEVEL,char const * __ptr64,...).
AFAICT, PCL is mostly header only, and somewhere within the guts of pcl::NormalEstimation there is a logging function which is not header-only and needs linking. Is there a way to prevent this call?

BTW, the DLL is in the DLL paths (and as can be seen in the screenshot - available) so it is still unclear why Python fails to load the module.

Additional context

Add any other context about the problem here.

[kunaltyagi]

If I remove the first member I can successfully import the module from the DLL.

What if you reorder them?

[adishavit]

Why would that change anything?
But in any case, I checked and it's still the same.

However, when I copy c:\Program Files\PCL 1.10.0\bin\pcl_common.dll to the output folder the class imports properly.
So, indeed, the problem is that although pcl::NormalEstimation<> is a template, using it introduces a runtime dependency on pcl_common.dll. If my investigation is correct then it is via some internal logger calls and, at the very least, is inconsistent with othe PCL template classes.

[kunaltyagi]

Can you present a MCVE, possibly without pybind11 to confirm this? The MCVE can be a simple CMakeLists.txt + the class definition and instantiation. If the issue was a PCL (CMake config or platform dependent behavior) bug, then merely depending on filters module of PCL for your tiny cpp file will fail at link time.

[adishavit]

Is there a CMake variable for specifying only one PCL component (after I required everything)?

I wrote a tiny C++ program:

#include <pcl/point_types.h>
#include <pcl/features/normal_3d.h>
#include <pcl/impl/point_types.hpp>

class A
{
   pcl::PointCloud<pcl::Normal>::Ptr normals_;
   pcl::NormalEstimation< pcl::PointXYZ, pcl::Normal> normalEstimation_;
};
int main()
{
   A a;
}

Using ProcExp I can see it depnds on a whole bunch of PCL DLLs (including pcl_common.dll)

[kunaltyagi]

Yes. find_package(PCL COMPONENTS features REQUIRED) will get only filters. Use only stuff from filters module, nothing else (since the error comes when filters is used)

If this doesn't give a link error, then PCL's CMake config is not at fault, and I wouldn't be able to help you anymore (since I don't use Windows, but don't worry, someone else will step in)

[adishavit]

A colleague reproduced the problem on Linux too.
It is resolved by copying libpcl_common.so.1.7 from /usr/lib/x86_64-linux-gnu/ to the build folder (but this is not a real solution).
I think it is some kind of problem with the installation of PCL on all platforms.
For some reason the PCL binaries path is not registered somewhere where Python can find them.

[kunaltyagi]

Again, if CMake finds the libraries, while not explicitly using common (which you haven't verified), then there's not anything more we can do from PCL side.

Perhaps Python doesn't look into transitive dependencies and thus fails to pick up dependencies of features. How to resolve (assuming you tried and CMake found the transitive dependencies):

1. If you know how Python looks up dependencies, there might be a possibility to provide that information. Perhaps your CMake can provide that information to Python somehow.
2. Another option would be to use static libraries instead of shared libraries to eliminate the issue of transitive dependencies.

[stale]

Marking this as stale due to 30 days of inactivity. It will be closed in 7 days if no further activity occurs.