new_from_memory causes "RuntimeError: cannot use string() on <cdata 'char *' NULL>" about pyvips HOT 5 CLOSED

Note, this is inside a conda env. I have found that outside the conda env it works, and that copying site-packages/_libvips.abi3.so from the other env into this conda env, it will work.

But, cleanly installing pyvips into that env will cause this failure again.

from pyvips.

Hello @lk-jeffpeck,

Yes, conda needs a special package, it won't work with the standard pyvips one. There have been a couple of tries at packaging libvips and pyvips for conda, but it hasn't worked out yet.

from pyvips.

Aha! Interesting... What is the issue?

Since it does technically work by installing it outside of the conda env and then copying the lib, I would think that there should be a more automated way to work around the issue?

from pyvips.

And is this documented anywhere?

from pyvips.

I expect you know much more about this than I do, but my understanding is that conda ships its own Python and that they don't include the development headers. So you need to install a matching version of Py, but with the headers, then build the extension with compiler flags that match the way that conda built their py, then copy the extension in to the conda py module area.

pyvips will work in ABI mode and run without a native module, but it's a bit slower.

There's a related problem with libvips itself: it depends on things like libjpeg, and conda also ship their own versions of libraries like these. What if libvips and conda have been built against external libraries which are not binary compatible? You can get a whole range of strange crashes as code calls a library it is not expecting.

The fix is to make a special conda package that builds libvips and also builds pyvips, then installs them together as a single binary which is compatible with your conda Python. This is the thing that's not been done yet (as far as I know).

conda is it's own special world and you can't really use external libraries or modules with it directly. This has advantages for simplicity, stability and consistency, but can be frustrating too.

I don't use conda myself and don't know a lot about it, but it seems it's quite popular, especially on Windows. There should probably be a note about this in the pyvips README.

from pyvips.