Running test suite on PyPy segfaults about pandas-feedstock HOT 10 OPEN

I think you should "advertise" the convenience of installation through conda-forge way harder

I just now rewrote the installation page. I posted a blog post in November. The download page starts with mentioning conda. The release notes don't mention conda, so I guess we could blog post again once the conda builds are available. Other than that can you think of where else we could mention conda?

from pandas-feedstock.

Pandas doesn't run well with PyPy. I think on the main repo we only basically test that it compiles too.

from pandas-feedstock.

Gotcha. Should we instead track via an upstream issue then? Maybe this one ( pandas-dev/pandas#42509 )?

from pandas-feedstock.

Over in #162, I re-enabled tests on pypy (using the new builds from conda-forge/pypy3.6-feedstock#103), and the picture is pretty bad. Here's a snippet from the test suite (on linux) before it eventually hangs indefinitely:

........F...................F....................F...................... [ 28%]
.F...................F....................F.....................F....... [ 28%]
................F.....................F.................F............... [ 28%]
........F...................F.......................F...................F [ 28%]
......................F.........................F....................... [ 28%]
...F....................F........................F..................F... [ 28%]
...............F.F.F.......F..........F...........F..............F...... [ 28%]
..............F....................F..........FxxFFxFFFFFFFFFxFxxFxFxFxx [ 28%]
xx......xxFxFxFFxFxFxxFxxFFxFx..F.....................F........F..F..... [ 28%]
..F...FFFF..........F.........FF..F......F...........F...............Fss [ 28%]

Windows and other linux flavours also segfault, though on OSX we do get a result at least:

= 10371 failed, 160211 passed, 23994 skipped, 909 xfailed, 6 xpassed, 454 warnings, 21 errors in 6942.61s (1:55:42) =

So around 6% failures, most of which (at a glance) seem datetime related.

It's very unfortunate that we only had the most basic import tests at the time when #106 resp. #133 was merged. Hindsight is of course 20/20, but IMO pypy support should not have been merged here before we could get down to ≪50 non-essential test failures. The problem is we cannot take this decision back, as the pypy migration has rolled on assuming that pandas-on-pypy is functional, when it appears to be quite far from that.

I'm not sure when pypy 3.10 will materialize, nor when we will do that migration in conda-forge, but I don't think we should merge the migrator here unless we can get rid of the segfaults and reduce the test failures very substantially. Even if it means that a lot of packages that have pypy support now will not be available until we have those fixes in place.

Raising this now so we have some time to discuss/plan before that decision actually has to be made.

@conda-forge/pandas

CC @conda-forge/core

from pandas-feedstock.

I think the datetime issues will trace back to what @mattip described in pandas-dev/pandas#50817 (comment)

from pandas-feedstock.

I think the datetime issues will trace back to what @mattip described in pandas-dev/pandas#50817 (comment)

The most recent run in #162 includes that fix already.

from pandas-feedstock.

Truthfully, I am not convinced the ROI on PyPy migrations in conda-forge have proved worthy. Maybe I am mistaken, but I have not seen a lot of end users reporting back about their experience using the pypy conda-forge packages, either with kudos, thanks, or even with issues and complaints. I personally will not be pushing conda-forge to do a pypy3.10 migration, but would support the effort if there is a clear consensus that it is worthwhile.

As for the pandas segfaults: I will try to look into what is going on in #162.

from pandas-feedstock.

Truthfully, I am not convinced the ROI on PyPy migrations in conda-forge have proved worthy.

I genuinely think that even aside from download numbers & volume of user feedback, it has been a huge boost to systematically chew through the package ecosystem and improve (or even introduce) PyPy compatibility either in conda-forge or directly upstream.

That said, to me it appears to be a messaging problem first and foremost. The fundamental question is: how do people interested in PyPy find out that they can install it through conda-forge - currently it's only a sidenote on the installation page, without instructions.

Given the effort you've invested in ensuring a large part of the ecosystem can be installed out of the box with conda-forge (and how challenging it is in turn to get started with PyPy on your own), I think you should "advertise" the convenience of installation through conda-forge way harder, to attract people who aren't already comfortable with dealing with all the build problems themselves. But obviously I'm biased and how to position yourselves is a choice for the PyPy project to make...

from pandas-feedstock.

Other than that can you think of where else we could mention conda?

I think the hands-on installation instructions (use these three lines if you have conda and you're off) is a great improvement. 👍

from pandas-feedstock.

Pandas + PyPy no longer segfaults for me when running tests locally (on ubuntu x86_64). The tests are slow, and made much slower by coverage. There are a number of test failures, I am tracking them in a PyPy issue and on #162

from pandas-feedstock.