Porting VTK from setuptools (setup.py) to scikit-build-core

[mathstuf]

VTK strains the Python packaging ecosystem in various ways, but this discussion is for focusing on the ways in which it creates its wheel can be improved using scikit-build-core. I'll go over how it currently works (at least as well as I can remember).

When configuring VTK, by enabling Python support, a VTK_WHEEL_BUILD option is made available. This variable causes VTK's CMake code to do some internal forcing of variables to structure its build tree to be used by a generated setup.py that uses setuptools to build a wheel. Part of this preparation is to divine the directory that Python packaging expects libraries to live in during wheel creation. distutils made it available (through obtuse access, but at least it existed). Now that Python 3.12+ removes it, we're pretty much "flying blind" as to any changes that happen (though CI where we make a venv, install the built wheel and make sure that import vtk works should catch catastrophic problems).

Once the configuration is complete, VTK inspects what is actually built (which is not immediately obvious from the input configuration settings) to "inform" the generated setup.py of what is actually available (e.g., matplotlib support, rendering backends, windowing system support, etc.). Once the build is run, the generated setup.py may be used to more-or-less just package up what the build puts in the right places for it to consume.

Things I believe would still need to continue to be true even with scikit-build-core (with explanations of why; if these bases are wrong, then the item may be resolved):

• CMake is the entry point to the build (i.e., no PEP517 support as some package metadata is not known until after cmake has completed; specifying arbitrary CMake options would be a must to have any other entry point)
• CMake needs to know where to place libraries (C++-only code needs to know how to locate resources relative to library locations; no pkg_resources access generally available)
• an SDK needs to be possible (ideally to build ParaView wheels on top of the VTK wheels)