conda-forge / arrow-cpp-feedstock Goto Github PK

View Code? Open in Web Editor NEW

10.0 10.0 61.0 2.7 MB

A conda-smithy repository for arrow-cpp.

License: BSD 3-Clause "New" or "Revised" License

Shell 63.98% Batchfile 36.02%

arrow-cpp-feedstock's Introduction

About arrow-cpp-feedstock

Feedstock license: BSD-3-Clause

About arrow-cpp

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow

About apache-arrow-proc

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: A meta-package to select Arrow build variant

About libarrow

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow core

About libarrow-acero

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow Acero

About libarrow-flight

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow Flight

About libarrow-gandiva

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow Gandiva

About libparquet

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Parquet

About libarrow-dataset

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow Dataset

About libarrow-flight-sql

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow Flight SQL

About libarrow-substrait

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Summary: C++ libraries for Apache Arrow Substrait

Current build status

Azure

Variant	Status
linux_64_c_compiler_version11c_stdlib_version2.17cuda_compiler_version11.8cxx_compiler_version11
linux_64_c_compiler_version12c_stdlib_version2.12cuda_compiler_versionNonecxx_compiler_version12
linux_aarch64_c_compiler_version10cuda_compiler_version11.8cxx_compiler_version10
linux_aarch64_c_compiler_version13cuda_compiler_versionNonecxx_compiler_version13
linux_ppc64le_c_compiler_version10cuda_compiler_version11.8cxx_compiler_version10
linux_ppc64le_c_compiler_version13cuda_compiler_versionNonecxx_compiler_version13
osx_64
osx_arm64
win_64_cuda_compilerNonecuda_compiler_versionNone
win_64_cuda_compilernvcccuda_compiler_version11.8

Current release info

Name	Downloads	Version	Platforms

Installing arrow-cpp

Installing arrow-cpp from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge
conda config --set channel_priority strict

Once the conda-forge channel has been enabled, apache-arrow-proc, libarrow, libarrow-acero, libarrow-all, libarrow-dataset, libarrow-flight, libarrow-flight-sql, libarrow-gandiva, libarrow-substrait, libparquet can be installed with conda:

conda install apache-arrow-proc libarrow libarrow-acero libarrow-all libarrow-dataset libarrow-flight libarrow-flight-sql libarrow-gandiva libarrow-substrait libparquet

or with mamba:

mamba install apache-arrow-proc libarrow libarrow-acero libarrow-all libarrow-dataset libarrow-flight libarrow-flight-sql libarrow-gandiva libarrow-substrait libparquet

It is possible to list all of the versions of apache-arrow-proc available on your platform with conda:

conda search apache-arrow-proc --channel conda-forge

or with mamba:

mamba search apache-arrow-proc --channel conda-forge

Alternatively, mamba repoquery may provide more information:

# Search all versions available on your platform:
mamba repoquery search apache-arrow-proc --channel conda-forge

# List packages depending on `apache-arrow-proc`:
mamba repoquery whoneeds apache-arrow-proc --channel conda-forge

# List dependencies of `apache-arrow-proc`:
mamba repoquery depends apache-arrow-proc --channel conda-forge

About conda-forge

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by Azure, GitHub, CircleCI, AppVeyor, Drone, and TravisCI it is possible to build and upload installable packages to the conda-forge anaconda.org channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating arrow-cpp-feedstock

If you would like to improve the arrow-cpp recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/arrow-cpp-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

If the version of a package is not being increased, please add or increase the build/number.
If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

arrow-cpp-feedstock's People

Contributors

Stargazers

Watchers

arrow-cpp-feedstock's Issues

Build 14 of 5.0.0 on Windows causes broken environment

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:
Build 13 allows aws-sdk-cpp 1.9.120, which causes the following issue when working with awswrangler, for example.

# reproduce_issue.py
import awswrangler

Fatal error condition occurred in D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
at 0x7FFFC2BC4380: aws_backtrace_print
at 0x7FFFC2BB3560: aws_fatal_assert
at 0x7FFFBD915F10: aws_event_loop_wait_for_stop_completion
at 0x7FFFC2BC1470: aws_ref_count_release
at 0x7FFFBD913D80: aws_server_bootstrap_set_alpn_callback
at 0x7FFFC2BC1470: aws_ref_count_release
at 0x7FFFBD913760: aws_client_bootstrap_release
at 0x7FFFBD9676F0: Aws::Crt::Io::ClientBootstrap::~ClientBootstrap
at 0x7FFFBDC6EB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FF836484180: execute_onexit_table
at 0x7FF836484180: execute_onexit_table
at 0x7FF836484180: execute_onexit_table
at 0x7FFFBDC6EB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFFBDC6EB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FF838B89900: RtlActivateActivationContextUnsafeFast
at 0x7FF838BCDA50: LdrShutdownProcess
at 0x7FF838BCD980: RtlExitUserProcess
at 0x7FF838A8E0A0: FatalExit
at 0x7FF8364903E0: exit
at 0x7FF8364903E0: exit
at 0x7FF6ED6E1160: OPENSSL_Applink
at 0x7FF838A87020: BaseThreadInitThunk
at 0x7FF838BC2630: RtlUserThreadStart

Build 14 and above have a dependency on 1.8.186, which resolves this issue. When installing the package, conda stops looking at builds larger than 13, because of the "downgrade" on aws-sdk-cpp. Would you say it makes sense to mark build 13 as broken on conda to circumvent this? I have not checked whether previous builds also allow aws-sdk-cpp 1.9.120, so this would apply to them as well.

Environment (conda list):

$ conda list
# packages in environment at C:\Users\rschmidtke\Miniconda3\envs\test-env:
#
# Name                    Version                   Build  Channel
abseil-cpp                20210324.2           h0e60522_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
arrow-cpp                 5.0.0           py39h47609a0_13_cpu    conda-forge
asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge
atomicwrites              1.4.0              pyh9f0ad1d_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
aws-c-auth                0.6.4                hd061354_3    conda-forge
aws-c-cal                 0.5.12               h395cb70_2    conda-forge
aws-c-common              0.6.11               h8ffe710_0    conda-forge
aws-c-compression         0.2.14               h82db3a7_2    conda-forge
aws-c-event-stream        0.2.7               he02faa2_24    conda-forge
aws-c-http                0.6.6                h77bbe4a_1    conda-forge
aws-c-io                  0.10.9               h3a8db39_3    conda-forge
aws-c-mqtt                0.7.8                h2f34d13_2    conda-forge
aws-c-s3                  0.1.27               habe44f2_1    conda-forge
aws-checksums             0.1.12               h82db3a7_1    conda-forge
aws-crt-cpp               0.17.1               haf3871f_3    conda-forge
aws-sam-translator        1.42.0             pyhd8ed1ab_0    conda-forge
aws-sdk-cpp               1.9.120              h37b9141_0    conda-forge
aws-xray-sdk              2.9.0              pyhd8ed1ab_0    conda-forge
awswrangler               2.11.0             pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.10.0             pyha770c72_0    conda-forge
boto3                     1.20.24            pyhd8ed1ab_0    conda-forge
botocore                  1.23.24            pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h8ffe710_6    conda-forge
brotli-bin                1.0.9                h8ffe710_6    conda-forge
brotlipy                  0.7.0           py39hb82d6ee_1003    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
c-ares                    1.18.1               h8ffe710_0    conda-forge
ca-certificates           2021.10.8            h5b45459_0    conda-forge
certifi                   2021.10.8        py39hcbf5309_1    conda-forge
cffi                      1.15.0           py39h0878f49_0    conda-forge
cfn-lint                  0.56.3             pyhd8ed1ab_0    conda-forge
charset-normalizer        2.0.9              pyhd8ed1ab_0    conda-forge
click                     8.0.3            py39hcbf5309_1    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
cryptography              36.0.0           py39h7bc7c5c_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
docker-py                 5.0.3            py39hcbf5309_2    conda-forge
docker-pycreds            0.4.0                      py_0    conda-forge
ecdsa                     0.17.0             pyhd8ed1ab_0    conda-forge
et_xmlfile                1.0.1                   py_1001    conda-forge
flask                     2.0.2              pyhd8ed1ab_0    conda-forge
flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
fonttools                 4.28.3           py39hb82d6ee_0    conda-forge
freetype                  2.10.4               h546665d_1    conda-forge
future                    0.18.2           py39hcbf5309_4    conda-forge
gflags                    2.2.2             ha925a31_1004    conda-forge
glog                      0.5.0                h4797de2_0    conda-forge
grpc-cpp                  1.41.1               ha2fd154_2    conda-forge
idna                      3.1                pyhd3deb0d_0    conda-forge
importlib-metadata        4.8.2            py39hcbf5309_0    conda-forge
importlib_metadata        4.8.2                hd8ed1ab_0    conda-forge
importlib_resources       3.3.1              pyhd8ed1ab_1    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
intel-openmp              2021.4.0          h57928b3_3556    conda-forge
itsdangerous              2.0.1              pyhd8ed1ab_0    conda-forge
jbig                      2.1               h8d14728_2003    conda-forge
jinja2                    3.0.3              pyhd8ed1ab_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
jpeg                      9d                   h8ffe710_0    conda-forge
jschema-to-python         1.2.3              pyhd8ed1ab_0    conda-forge
jsondiff                  1.3.0              pyhd8ed1ab_0    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpickle                2.0.0              pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
junit-xml                 1.9                pyh9f0ad1d_0    conda-forge
kiwisolver                1.3.2            py39h2e07f2f_1    conda-forge
krb5                      1.19.2               h20d022d_3    conda-forge
lcms2                     2.12                 h2a16943_0    conda-forge
lerc                      3.0                  h0e60522_0    conda-forge
libblas                   3.9.0              12_win64_mkl    conda-forge
libbrotlicommon           1.0.9                h8ffe710_6    conda-forge
libbrotlidec              1.0.9                h8ffe710_6    conda-forge
libbrotlienc              1.0.9                h8ffe710_6    conda-forge
libcblas                  3.9.0              12_win64_mkl    conda-forge
libcurl                   7.80.0               h789b8ee_1    conda-forge
libdeflate                1.8                  h8ffe710_0    conda-forge
libiconv                  1.16                 he774522_0    conda-forge
liblapack                 3.9.0              12_win64_mkl    conda-forge
libpng                    1.6.37               h1d00b33_2    conda-forge
libprotobuf               3.18.1               h7755175_0    conda-forge
libssh2                   1.10.0               h680486a_2    conda-forge
libthrift                 0.15.0               h636ae23_1    conda-forge
libtiff                   4.3.0                hd413186_2    conda-forge
libutf8proc               2.6.1                hcb41399_0    conda-forge
libxml2                   2.9.12               hf5bbc77_1    conda-forge
libxslt                   1.1.33               h65864e5_3    conda-forge
libzlib                   1.2.11            h8ffe710_1013    conda-forge
lxml                      4.7.1            py39h4fd7cdf_0    conda-forge
lz4-c                     1.9.3                h8ffe710_1    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
markupsafe                2.0.1            py39hb82d6ee_1    conda-forge
matplotlib-base           3.5.1            py39h581301d_0    conda-forge
mkl                       2021.4.0           h0e2418a_729    conda-forge
more-itertools            8.12.0             pyhd8ed1ab_0    conda-forge
moto                      2.2.19             pyhd8ed1ab_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
networkx                  2.6.3              pyhd8ed1ab_1    conda-forge
numpy                     1.21.4           py39h6635163_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb211442_1    conda-forge
openpyxl                  3.0.9              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1l               h8ffe710_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py39h2e25243_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
pbr                       5.8.0              pyhd8ed1ab_1    conda-forge
pg8000                    1.21.3             pyhd8ed1ab_0    conda-forge
pillow                    8.4.0            py39h916092e_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py39hcbf5309_2    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pyarrow                   5.0.0           py39ha05331a_13_cpu    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pymysql                   1.0.2              pyhd8ed1ab_0    conda-forge
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.6              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.0           py39hb82d6ee_0    conda-forge
pysocks                   1.7.1            py39hcbf5309_4    conda-forge
pytest                    6.2.5            py39hcbf5309_1    conda-forge
python                    3.9.7           h7840368_3_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-jose               3.3.0              pyh6c4a22f_1    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pywin32                   302              py39hb82d6ee_2    conda-forge
pyyaml                    6.0              py39hb82d6ee_3    conda-forge
re2                       2021.11.01           h0e60522_0    conda-forge
redshift_connector        2.0.902            pyhd8ed1ab_0    conda-forge
requests                  2.26.0             pyhd8ed1ab_1    conda-forge
responses                 0.16.0             pyhd8ed1ab_0    conda-forge
rsa                       4.8                pyhd8ed1ab_0    conda-forge
s3transfer                0.5.0              pyhd8ed1ab_0    conda-forge
sarif-om                  1.0.4              pyhd8ed1ab_0    conda-forge
scipy                     1.7.3            py39hc0c34ad_0    conda-forge
scramp                    1.4.1              pyhd8ed1ab_0    conda-forge
setuptools                59.6.0           py39hcbf5309_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.8                ha925a31_3    conda-forge
soupsieve                 2.3.1              pyhd8ed1ab_0    conda-forge
sqlite                    3.37.0               h8ffe710_0    conda-forge
sshpubkeys                3.1.0                      py_0    conda-forge
tbb                       2021.4.0             h2d74725_1    conda-forge
tk                        8.6.11               h8ffe710_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
vc                        14.2                 hb210afc_5    conda-forge
vs2015_runtime            14.29.30037          h902a5da_5    conda-forge
websocket-client          1.2.3              pyhd8ed1ab_0    conda-forge
werkzeug                  2.0.1              pyhd8ed1ab_0    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
win_inet_pton             1.1.0            py39hcbf5309_3    conda-forge
wrapt                     1.13.3           py39hb82d6ee_1    conda-forge
xmltodict                 0.12.0                     py_0    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge
yaml                      0.2.5                he774522_0    conda-forge
zipp                      3.6.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h8ffe710_1013    conda-forge
zstd                      1.5.0                h6255e5f_0    conda-forge

Details about conda and system ( conda info ):

$ conda info
active environment : test-env
active env location : C:\Users\rschmidtke\Miniconda3\envs\test-env
      shell level : 1
 user config file : C:\Users\rschmidtke\.condarc
populated config files : C:\Users\rschmidtke\.condarc
    conda version : 4.11.0
conda-build version : not installed
   python version : 3.9.7.final.0
 virtual packages : __cuda=11.4=0
                    __win=0=0
                    __archspec=1=x86_64
 base environment : C:\Users\rschmidtke\Miniconda3  (writable)
conda av data dir : C:\Users\rschmidtke\Miniconda3\etc\conda
conda av metadata url : None
     channel URLs : https://conda.anaconda.org/conda-forge/win-64
                    https://conda.anaconda.org/conda-forge/noarch
                    https://repo.anaconda.com/pkgs/main/win-64
                    https://repo.anaconda.com/pkgs/main/noarch
                    https://repo.anaconda.com/pkgs/r/win-64
                    https://repo.anaconda.com/pkgs/r/noarch
                    https://repo.anaconda.com/pkgs/msys2/win-64
                    https://repo.anaconda.com/pkgs/msys2/noarch
    package cache : C:\Users\rschmidtke\Miniconda3\pkgs
                    C:\Users\rschmidtke\.conda\pkgs
                    C:\Users\rschmidtke\AppData\Local\conda\conda\pkgs
 envs directories : C:\Users\rschmidtke\Miniconda3\envs
                    C:\Users\rschmidtke\.conda\envs
                    C:\Users\rschmidtke\AppData\Local\conda\conda\envs
         platform : win-64
       user-agent : conda/4.11.0 requests/2.26.0 CPython/3.9.7 Windows/10 Windows/10.0.19044
    administrator : False
       netrc file : None
     offline mode : False

pyarrow (as of py38 and py39h53c4050_6_cpu) seg-faults on import on osx-arm64

Issue:

As was referenced in #200 , while the current pyarrow feedstock successfully cross-compiles for osx-arm64, when you try to run tests on this platform you get an immediate segmentation fault when doing import pyarrow (specifically, when trying to load pyarrow.lib). Building natively on osx-arm64 fails with some strange sed error that I have not spent time investigating fully, so I cannot say whether this is a problem with the cross-compile or something else.

Environment (conda list):

$ conda list
# packages in environment at /Users/erykoff/miniforge3/envs/pyarrow-6:
#
# Name                    Version                   Build  Channel
abseil-cpp                20200225.2           hc88da5d_2    conda-forge
arrow-cpp                 2.0.0           py39hc12fbcd_6_cpu    https://conda-web.anaconda.org/conda-forge
aws-c-common              0.4.59               hb904e53_0    https://conda-web.anaconda.org/conda-forge
aws-c-event-stream        0.1.6                h0d44b10_6    https://conda-web.anaconda.org/conda-forge
aws-checksums             0.1.10               h2b027d1_0    https://conda-web.anaconda.org/conda-forge
aws-sdk-cpp               1.8.70               h00b3300_1    https://conda-web.anaconda.org/conda-forge
brotli                    1.0.9                hb904e53_3    https://conda-web.anaconda.org/conda-forge
bzip2                     1.0.8                h642e427_3    conda-forge
c-ares                    1.16.1               h642e427_3    conda-forge
ca-certificates           2020.11.8            h4653dfc_0    https://conda-web.anaconda.org/conda-forge
gflags                    2.2.2             hc88da5d_1004    conda-forge
glog                      0.4.0                h6361243_3    conda-forge
grpc-cpp                  1.33.2               h801cd35_1    https://conda-web.anaconda.org/conda-forge
krb5                      1.17.1               h58ff05e_3    conda-forge
libblas                   3.9.0                2_openblas    https://conda-web.anaconda.org/conda-forge
libcblas                  3.9.0                2_openblas    https://conda-web.anaconda.org/conda-forge
libcurl                   7.71.1               hd2aec06_8    https://conda-web.anaconda.org/conda-forge
libcxx                    11.0.0               hc89144c_0    https://conda-web.anaconda.org/conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libevent                  2.1.10               h93437e4_3    https://conda-web.anaconda.org/conda-forge
libffi                    3.2.1             hc88da5d_1007    conda-forge
libgfortran               5.0.0.dev0          h181927c_13    https://conda-web.anaconda.org/conda-forge
libgfortran5              11.0.0.dev0         h181927c_13    https://conda-web.anaconda.org/conda-forge
liblapack                 3.9.0                2_openblas    https://conda-web.anaconda.org/conda-forge
libnghttp2                1.41.0               h87e4072_2    conda-forge
libopenblas               0.3.12          openmp_h2ecc587_1    https://conda-web.anaconda.org/conda-forge
libprotobuf               3.13.0.1             h776f369_0    https://conda-web.anaconda.org/conda-forge
libssh2                   1.9.0                h1c49ba1_5    conda-forge
libthrift                 0.13.0               h854ee3b_6    https://conda-web.anaconda.org/conda-forge
libutf8proc               2.5.0                h642e427_2    conda-forge
llvm-openmp               11.0.0               hdb94862_1    https://conda-web.anaconda.org/conda-forge
lz4-c                     1.9.2                hc88da5d_3    conda-forge
ncurses                   6.2                  h9aa5885_4    https://conda-web.anaconda.org/conda-forge
numpy                     1.19.4           py39h69a04d8_1    https://conda-web.anaconda.org/conda-forge
openssl                   1.1.1h               h642e427_0    https://conda-web.anaconda.org/conda-forge
orc                       1.6.5                h0dbfa4a_0    https://conda-web.anaconda.org/conda-forge
parquet-cpp               1.5.1                         2    conda-forge
pyarrow                   2.0.0           py39h53c4050_6_cpu    https://conda-web.anaconda.org/conda-forge
python                    3.9.0           hf1b0709_4_cpython    https://conda-web.anaconda.org/conda-forge
python_abi                3.9                      1_cp39    https://conda-web.anaconda.org/conda-forge
re2                       2020.11.01           h9f76cd9_0    https://conda-web.anaconda.org/conda-forge
readline                  8.0                  hc8eb9b7_2    conda-forge
snappy                    1.1.8                hc88da5d_3    conda-forge
sqlite                    3.33.0               h7e9f57c_1    https://conda-web.anaconda.org/conda-forge
tk                        8.6.10               h99d78ee_1    https://conda-web.anaconda.org/conda-forge
tzdata                    2020d                h516909a_0    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
zlib                      1.2.11            h31e879b_1009    conda-forge
zstd                      1.4.5                hc019d7c_2    conda-forge

Details about conda and system ( conda info ):

$ conda info
     active environment : pyarrow-6
    active env location : /Users/erykoff/miniforge3/envs/pyarrow-6
            shell level : 2
       user config file : /Users/erykoff/.condarc
 populated config files : /Users/erykoff/miniforge3/.condarc
                          /Users/erykoff/.condarc
          conda version : 4.8.5
    conda-build version : not installed
         python version : 3.8.6.final.0
       virtual packages : __osx=11.0.1
       base environment : /Users/erykoff/miniforge3  (writable)
           channel URLs : https://conda-web.anaconda.org/conda-forge/osx-arm64
                          https://conda-web.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/erykoff/miniforge3/pkgs
                          /Users/erykoff/.conda/pkgs
       envs directories : /Users/erykoff/miniforge3/envs
                          /Users/erykoff/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.8.5 requests/2.24.0 CPython/3.8.6 Darwin/20.1.0 OSX/11.0.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

@conda-forge-admin please rerender

7.0.1 build 3 causes error on import

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

Build 3 over build 2 makes the following changes to google libraries (inspecting the meta.yaml files)

97c97
<     - google-cloud-cpp 2.1.0 h838d150_3
---
>     - google-cloud-cpp 2.2.0 h838d150_0

Which causes in ImportError

import pyarrow as pa
  File "/home/rschmidtke/miniconda3/conda-bld/tsawsdatahub_1663327389681/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.10/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: /home/rschmidtke/miniconda3/conda-bld/tsawsdatahub_1663327389681/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.10/site-packages/pyarrow/../../../libarrow.so.700: undefined symbol: _ZN6google5cloud6v2_2_06StatusD1Ev

Manually pinning google-cloud-cpp to 2.1.0 fixes the issue. It seems it may be necessary to restrict the library here as well?

Installed packages

_libgcc_mutex:            0.1-conda_forge             conda-forge
    _openmp_mutex:            4.5-2_gnu                   conda-forge
    aenum:                    2.2.6-pyhd3deb0d_0          conda-forge
    aiohttp:                  3.8.1-py310h5764c6d_1       conda-forge
    aiosignal:                1.2.0-pyhd8ed1ab_0          conda-forge
    appdirs:                  1.4.4-pyh9f0ad1d_0          conda-forge
    arrow-cpp:                7.0.1-py310h028ac91_3_cpu   conda-forge
    asn1crypto:               1.5.1-pyhd8ed1ab_0          conda-forge
    async-timeout:            4.0.2-pyhd8ed1ab_0          conda-forge
    attrs:                    22.1.0-pyh71513ae_1         conda-forge
    aws-c-cal:                0.5.11-h95a6274_0           conda-forge
    aws-c-common:             0.6.2-h7f98852_0            conda-forge
    aws-c-event-stream:       0.2.7-h3541f99_13           conda-forge
    aws-c-io:                 0.10.5-hfb6a706_0           conda-forge
    aws-checksums:            0.1.11-ha31a3da_7           conda-forge
    aws-sam-translator:       1.51.0-pyhd8ed1ab_0         conda-forge
    aws-sdk-cpp:              1.8.186-hb4091e7_3          conda-forge
    aws-xray-sdk:             2.10.0-pyhd8ed1ab_0         conda-forge
    awswrangler:              2.16.1-pyhd8ed1ab_0         conda-forge
    backoff:                  1.11.1-pyhd8ed1ab_0         conda-forge
    beautifulsoup4:           4.11.1-pyha770c72_0         conda-forge
    boto3:                    1.24.74-pyhd8ed1ab_0        conda-forge
    botocore:                 1.27.74-pyhd8ed1ab_0        conda-forge
    brotlipy:                 0.7.0-py310h5764c6d_1004    conda-forge
    bzip2:                    1.0.8-h7f98852_4            conda-forge
    c-ares:                   1.18.1-h7f98852_0           conda-forge
    ca-certificates:          2022.9.14-ha878542_0        conda-forge
    certifi:                  2022.9.14-pyhd8ed1ab_0      conda-forge
    cffi:                     1.15.1-py310h255011f_0      conda-forge
    cfn-lint:                 0.65.0-pyhd8ed1ab_0         conda-forge
    charset-normalizer:       2.0.12-pyhd8ed1ab_0         conda-forge
    click:                    8.1.3-py310hff52083_0       conda-forge
    cryptography:             37.0.4-py310h597c629_0      conda-forge
    decorator:                5.1.1-pyhd8ed1ab_0          conda-forge
    docker-py:                5.0.3-py310hff52083_2       conda-forge
    docker-pycreds:           0.4.0-py_0                  conda-forge
    ecdsa:                    0.18.0-pyhd8ed1ab_1         conda-forge
    et_xmlfile:               1.0.1-py_1001               conda-forge
    flask:                    2.1.3-pyhd8ed1ab_0          conda-forge
    flask_cors:               3.0.10-pyhd3deb0d_0         conda-forge
    frozenlist:               1.3.1-py310h5764c6d_0       conda-forge
    gflags:                   2.2.2-he1b5a44_1004         conda-forge
    glog:                     0.6.0-h6f12383_0            conda-forge
    gmp:                      6.2.1-h58526e2_0            conda-forge
    gmpy2:                    2.1.2-py310h92f7908_0       conda-forge
    graphql-core:             3.2.1-pyhd8ed1ab_0          conda-forge
    gremlinpython:            3.5.2-pyhd8ed1ab_0          conda-forge
    grpc-cpp:                 1.47.1-h05bd8bd_6           conda-forge
    icu:                      70.1-h27087fc_0             conda-forge
    idna:                     3.3-pyhd8ed1ab_0            conda-forge
    importlib-metadata:       4.11.4-py310hff52083_0      conda-forge
    importlib_metadata:       4.11.4-hd8ed1ab_0           conda-forge
    importlib_resources:      3.3.1-pyhd8ed1ab_1          conda-forge
    iniconfig:                1.1.1-pyh9f0ad1d_0          conda-forge
    isodate:                  0.6.1-pyhd8ed1ab_0          conda-forge
    itsdangerous:             2.1.2-pyhd8ed1ab_0          conda-forge
    jinja2:                   3.1.2-pyhd8ed1ab_1          conda-forge
    jmespath:                 1.0.1-pyhd8ed1ab_0          conda-forge
    jschema-to-python:        1.2.3-pyhd8ed1ab_0          conda-forge
    jsondiff:                 2.0.0-pyhd8ed1ab_0          conda-forge
    jsonpatch:                1.32-pyhd8ed1ab_0           conda-forge
    jsonpath-ng:              1.5.3-pyhd8ed1ab_0          conda-forge
    jsonpickle:               2.2.0-pyhd8ed1ab_0          conda-forge
    jsonpointer:              2.0-py_0                    conda-forge
    jsonschema:               3.2.0-pyhd8ed1ab_3          conda-forge
    junit-xml:                1.9-pyh9f0ad1d_0            conda-forge
    keyutils:                 1.6.1-h166bdaf_0            conda-forge
    krb5:                     1.19.3-h3790be6_0           conda-forge
    ld_impl_linux-64:         2.36.1-hea4e1c9_2           conda-forge
    libabseil:                20220623.0-cxx17_h48a1fff_4 conda-forge
    libblas:                  3.9.0-16_linux64_openblas   conda-forge
    libbrotlicommon:          1.0.9-h166bdaf_7            conda-forge
    libbrotlidec:             1.0.9-h166bdaf_7            conda-forge
    libbrotlienc:             1.0.9-h166bdaf_7            conda-forge
    libcblas:                 3.9.0-16_linux64_openblas   conda-forge
    libcrc32c:                1.1.2-h9c3ff4c_0            conda-forge
    libcurl:                  7.83.1-h7bff187_0           conda-forge
    libedit:                  3.1.20191231-he28a2e2_2     conda-forge
    libev:                    4.33-h516909a_1             conda-forge
    libevent:                 2.1.10-h9b69904_4           conda-forge
    libffi:                   3.4.2-h7f98852_5            conda-forge
    libgcc-ng:                12.1.0-h8d9b700_16          conda-forge
    libgfortran-ng:           12.1.0-h69a702a_16          conda-forge
    libgfortran5:             12.1.0-hdcd56e2_16          conda-forge
    libgomp:                  12.1.0-h8d9b700_16          conda-forge
    libgoogle-cloud:          2.2.1-h838d150_0            conda-forge
    libiconv:                 1.16-h516909a_0             conda-forge
    liblapack:                3.9.0-16_linux64_openblas   conda-forge
    libnghttp2:               1.47.0-hdcd2b5c_1           conda-forge
    libnsl:                   2.0.0-h7f98852_0            conda-forge
    libopenblas:              0.3.21-pthreads_h78a6416_3  conda-forge
    libprotobuf:              3.21.6-h6239696_0           conda-forge
    libsqlite:                3.39.3-h753d276_0           conda-forge
    libssh2:                  1.10.0-haa6b8db_3           conda-forge
    libstdcxx-ng:             12.1.0-ha89aaad_16          conda-forge
    libthrift:                0.16.0-h491838f_2           conda-forge
    libutf8proc:              2.7.0-h7f98852_0            conda-forge
    libuuid:                  2.32.1-h7f98852_1000        conda-forge
    libxml2:                  2.9.14-h22db469_4           conda-forge
    libxslt:                  1.1.35-h8affb1d_0           conda-forge
    libzlib:                  1.2.12-h166bdaf_3           conda-forge
    lxml:                     4.9.1-py310h5764c6d_0       conda-forge
    lz4-c:                    1.9.3-h9c3ff4c_1            conda-forge
    markupsafe:               2.1.1-py310h5764c6d_1       conda-forge
    moto:                     4.0.3-pyhd8ed1ab_0          conda-forge
    mpc:                      1.2.1-h9f54685_0            conda-forge
    mpfr:                     4.1.0-h9202a9a_1            conda-forge
    multidict:                6.0.2-py310h5764c6d_1       conda-forge
    ncurses:                  6.3-h27087fc_1              conda-forge
    nest-asyncio:             1.5.5-pyhd8ed1ab_0          conda-forge
    networkx:                 2.8.6-pyhd8ed1ab_0          conda-forge
    numpy:                    1.23.3-py310h53a5b5f_0      conda-forge
    openapi-schema-validator: 0.2.3-pyhd8ed1ab_0          conda-forge
    openapi-spec-validator:   0.4.0-pyhd8ed1ab_1          conda-forge
    openpyxl:                 3.0.10-py310h5764c6d_1      conda-forge
    opensearch-py:            1.1.0-pyhd8ed1ab_0          conda-forge
    openssl:                  1.1.1q-h166bdaf_0           conda-forge
    orc:                      1.7.6-h09e0d61_1            conda-forge
    packaging:                21.3-pyhd8ed1ab_0           conda-forge
    pandas:                   1.4.4-py310h769672d_0       conda-forge
    parquet-cpp:              1.5.1-2                     conda-forge
    pbr:                      5.10.0-pyhd8ed1ab_0         conda-forge
    pg8000:                   1.29.1-pyhd8ed1ab_0         conda-forge
    pip:                      22.2.2-pyhd8ed1ab_0         conda-forge
    pluggy:                   1.0.0-py310hff52083_3       conda-forge
    ply:                      3.11-py_1                   conda-forge
    progressbar2:             4.0.0-pyhd8ed1ab_0          conda-forge
    py:                       1.11.0-pyh6c4a22f_0         conda-forge
    pyarrow:                  7.0.1-py310h225c066_3_cpu   conda-forge
    pyasn1:                   0.4.8-py_0                  conda-forge
    pycparser:                2.21-pyhd8ed1ab_0           conda-forge
    pymysql:                  1.0.2-pyhd8ed1ab_0          conda-forge
    pyodbc:                   4.0.34-py310hd8f1fbe_0      conda-forge
    pyopenssl:                22.0.0-pyhd8ed1ab_0         conda-forge
    pyparsing:                3.0.9-pyhd8ed1ab_0          conda-forge
    pyrsistent:               0.18.1-py310h5764c6d_1      conda-forge
    pysocks:                  1.7.1-pyha2e5f31_6          conda-forge
    pytest:                   7.1.3-py310hff52083_0       conda-forge
    python:                   3.10.6-h582c2e5_0_cpython   conda-forge
    python-dateutil:          2.8.2-pyhd8ed1ab_0          conda-forge
    python-jose:              3.3.0-pyh6c4a22f_1          conda-forge
    python-utils:             3.3.3-pyhd8ed1ab_0          conda-forge
    python_abi:               3.10-2_cp310                conda-forge
    pytz:                     2022.1-pyhd8ed1ab_0         conda-forge
    pyyaml:                   6.0-py310h5764c6d_4         conda-forge
    re2:                      2022.06.01-h27087fc_0       conda-forge
    readline:                 8.1.2-h0f457ee_0            conda-forge
    redshift_connector:       2.0.908-pyhd8ed1ab_0        conda-forge
    requests:                 2.28.0-pyhd8ed1ab_1         conda-forge
    requests-aws4auth:        1.1.1-pyhd8ed1ab_0          conda-forge
    responses:                0.21.0-pyhd8ed1ab_0         conda-forge
    rsa:                      4.9-pyhd8ed1ab_0            conda-forge
    s2n:                      1.0.10-h9b69904_0           conda-forge
    s3transfer:               0.6.0-pyhd8ed1ab_0          conda-forge
    sarif-om:                 1.0.4-pyhd8ed1ab_0          conda-forge
    scramp:                   1.4.1-pyhd8ed1ab_0          conda-forge
    setuptools:               65.3.0-pyhd8ed1ab_1         conda-forge
    six:                      1.16.0-pyh6c4a22f_0         conda-forge
    snappy:                   1.1.9-hbd366e4_1            conda-forge
    soupsieve:                2.3.2.post1-pyhd8ed1ab_0    conda-forge
    sshpubkeys:               3.3.1-pyhd8ed1ab_0          conda-forge
    tk:                       8.6.12-h27826a3_0           conda-forge
    tomli:                    2.0.1-pyhd8ed1ab_0          conda-forge
    tsawsdatahub:             0.0.1-py_1                  local
    typing-extensions:        4.3.0-hd8ed1ab_0            conda-forge
    typing_extensions:        4.3.0-pyha770c72_0          conda-forge
    tzdata:                   2022c-h191b570_0            conda-forge
    unixodbc:                 2.3.10-h583eb01_0           conda-forge
    urllib3:                  1.26.11-pyhd8ed1ab_0        conda-forge
    websocket-client:         1.4.1-pyhd8ed1ab_0          conda-forge
    werkzeug:                 2.1.2-pyhd8ed1ab_1          conda-forge
    wheel:                    0.37.1-pyhd8ed1ab_0         conda-forge
    wrapt:                    1.14.1-py310h5764c6d_0      conda-forge
    xmltodict:                0.13.0-pyhd8ed1ab_0         conda-forge
    xz:                       5.2.6-h166bdaf_0            conda-forge
    yaml:                     0.2.5-h7f98852_2            conda-forge
    yarl:                     1.7.2-py310h5764c6d_2       conda-forge
    zipp:                     3.8.1-pyhd8ed1ab_0          conda-forge
    zlib:                     1.2.12-h166bdaf_3           conda-forge
    zstd:                     1.5.2-h6239696_4            conda-forge

Environment info

active environment : tsawsdatahub
    active env location : /home/rschmidtke/miniconda3/envs/tsawsdatahub
            shell level : 1
       user config file : /home/rschmidtke/.condarc
 populated config files : /home/rschmidtke/.condarc
          conda version : 4.14.0
    conda-build version : 3.22.0
         python version : 3.9.12.final.0
       virtual packages : __linux=5.10.102.1=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/rschmidtke/miniconda3  (writable)
      conda av data dir : /home/rschmidtke/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://<redacted>.jfrog.io/artifactory/api/conda/conda-local/linux-64
                          https://<redacted>.jfrog.io/artifactory/api/conda/conda-local/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/rschmidtke/miniconda3/pkgs
                          /home/rschmidtke/.conda/pkgs
       envs directories : /home/rschmidtke/miniconda3/envs
                          /home/rschmidtke/.conda/envs
               platform : linux-64
             user-agent : conda/4.14.0 requests/2.28.1 CPython/3.9.12 Linux/5.10.102.1-microsoft-standard-WSL2 ubuntu/20.04.5 glibc/2.31 solver/libmamba conda-libmamba-solver/22.8.1 libmambapy/0.25.0
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Is `python` a necessary run dependency for `arrow-cpp`?

One of my pure R environments ended up installingpython today. Tracing back the culprit r-* package, I found that the r-pins package depends on arrow-cpp, and this includes python as a run dependency. Why is that? Is this library truly dependent on having Python installed?

faileded conda install arrow-cpp

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

when i try to install arrow-app, error" conda.core.subdir_data.Response304ContentUnchanged" appears

(py36) stella-zhao@zsh-pc:~/research_space/LargeScaleClinicalEmbedding$ conda install arrow-cpp arrow-cpp-proc pyarrow pyarrow-tests
Collecting package metadata (current_repodata.json): failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 379, in _load
        repodata_fn=self.repodata_fn)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 815, in fetch_repodata_remote_request
        raise Response304ContentUnchanged()
    conda.core.subdir_data.Response304ContentUnchanged
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/cli/main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 83, in do_call
        return getattr(module, func_name)(args, parser)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/cli/main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/cli/install.py", line 265, in install
        should_retry_solve=(_should_retry_unfrozen or repodata_fn != repodata_fns[-1]),
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 117, in solve_for_transaction
        should_retry_solve)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 158, in solve_for_diff
        force_remove, should_retry_solve)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 262, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 425, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 1021, in _prepare
        self.subdirs, prepared_specs, self._repodata_fn)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/index.py", line 289, in get_reduced_index
        repodata_fn=repodata_fn)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 140, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "/home/stella-zhao/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
        yield fs.pop().result()
      File "/home/stella-zhao/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 428, in result
        return self.__get_result()
      File "/home/stella-zhao/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
        raise self._exception
      File "/home/stella-zhao/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 133, in <lambda>
        package_ref_or_match_spec))
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 145, in query
        self.load()
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 210, in load
        _internal_state = self._load()
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 394, in _load
        mod_etag_headers.get('_mod'))
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 442, in _read_local_repdata
        _internal_state = self._process_raw_repodata_str(raw_repodata_str)
      File "/home/stella-zhao/anaconda3/lib/python3.7/site-packages/conda/core/subdir_data.py", line 480, in _process_raw_repodata_str
        json_obj = json.loads(raw_repodata_str or '{}')
      File "/home/stella-zhao/anaconda3/lib/python3.7/json/__init__.py", line 348, in loads
        return _default_decoder.decode(s)
      File "/home/stella-zhao/anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/home/stella-zhao/anaconda3/lib/python3.7/json/decoder.py", line 353, in raw_decode
        obj, end = self.scan_once(s, idx)
    json.decoder.JSONDecodeError: Unterminated string starting at: line 226811 column 9 (char 6930251)

Environment (conda list):

$ conda list

# packages in environment at /home/stella-zhao/anaconda3/envs/py36:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
binutils_impl_linux-64    2.34                 h53a641e_7    conda-forge
binutils_linux-64         2.34                hc952b39_18    conda-forge
bzip2                     1.0.8                h7b6447c_0    defaults
ca-certificates           2021.7.5             h06a4308_1    defaults
certifi                   2021.5.30        py36h06a4308_0    defaults
cmake                     3.12.3               h011004d_0    conda-forge
curl                      7.61.0               h84994c4_0    defaults
expat                     2.4.1                h2531618_2    defaults
gcc_impl_linux-64         7.3.0                habb00fd_1    defaults
gcc_linux-64              7.3.0               h553295d_18    conda-forge
gfortran_impl_linux-64    7.3.0                hdf63c60_1    defaults
gfortran_linux-64         7.3.0               h553295d_18    conda-forge
gxx_impl_linux-64         7.3.0                hdf63c60_1    defaults
gxx_linux-64              7.3.0               h553295d_18    conda-forge
hypre                     2.18.2               hc98498a_1    conda-forge
ld_impl_linux-64          2.34                 h53a641e_7    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcurl                   7.61.0               h1ad7b7a_0    defaults
libgcc-ng                 7.3.0                hdf63c60_2    conda-forge
libgfortran-ng            7.5.0               ha8ba4b0_17    defaults
libgfortran4              7.5.0               ha8ba4b0_17    defaults
liblapack                 3.8.0               14_openblas    conda-forge
libopenblas               0.3.7                h5ec1e0e_6    conda-forge
libssh2                   1.8.0                h9cfc8f7_4    defaults
libstdcxx-ng              9.3.0               hd4cf53a_17    defaults
libuv                     1.40.0               h7b6447c_0    defaults
metis                     5.1.0                hf484d3e_4    defaults
mpi                       1.0                       mpich    defaults
mpich                     3.3.2                hc856adb_0    defaults
mumps-include             5.2.1                         8    conda-forge
mumps-mpi                 5.2.1                h1f24c91_8    conda-forge
ncurses                   6.2                  he6710b0_1    defaults
openssl                   1.0.2u               h7b6447c_0    defaults
parmetis                  4.0.3             h9f7b9cf_1005    conda-forge
petsc                     3.13.1               h7441afc_0    conda-forge
pip                       21.0.1           py36h06a4308_0    defaults
ptscotch                  6.0.9                h294ddb0_1    conda-forge
python                    3.6.0                         0    defaults
readline                  6.2                           2    defaults
rhash                     1.3.6                hb7f436b_0    defaults
scalapack                 2.0.2             hfacbc1e_1008    conda-forge
scotch                    6.0.9                h0eec0ba_1    conda-forge
setuptools                58.0.4           py36h06a4308_0    defaults
slepc                     3.13.2               ha8d41fe_0    conda-forge
sqlite                    3.13.0                        0    defaults
suitesparse               5.6.0                h717dc36_0    conda-forge
superlu                   5.2.1             hfe2efc7_1207    conda-forge
superlu_dist              6.2.0                h5e15a89_2    conda-forge
tbb                       2020.1               hc9558a2_0    conda-forge
tbb-devel                 2020.1               hc9558a2_0    conda-forge
tk                        8.5.18                        0    defaults
wheel                     0.37.0             pyhd3eb1b0_1    defaults
xz                        5.2.5                h7b6447c_0    defaults
zlib                      1.2.11               h7b6447c_3    defaults

Details about conda and system ( conda info ):

$ conda info

     active environment : py36
    active env location : /home/stella-zhao/anaconda3/envs/py36
            shell level : 2
       user config file : /home/stella-zhao/.condarc
 populated config files : /home/stella-zhao/.condarc
          conda version : 4.10.3
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __cuda=11.0=0
                          __linux=5.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/stella-zhao/anaconda3  (writable)
      conda av data dir : /home/stella-zhao/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/noarch
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/noarch
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/noarch
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/noarch
          package cache : /home/stella-zhao/anaconda3/pkgs
                          /home/stella-zhao/.conda/pkgs
       envs directories : /home/stella-zhao/anaconda3/envs
                          /home/stella-zhao/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.3 requests/2.22.0 CPython/3.7.6 Linux/5.4.0-89-generic ubuntu/20.04.3 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[ppc64le] Builds are failing with timeout errors on master

According to the builds the 10 minute no output timeout is reached during the environment creation.

https://github.com/conda-forge/arrow-cpp-feedstock/runs/686923031

Use dynamic linking with zlib

Statically linking to zlib is considered a security issue. apache/arrow#2483 adds the ability to dynamically link, and we should change the conda recipe to also do so

`pyarrow=1.0.1`, `boost-cpp=1.7.4`, `python=3.8` are currently in a broken state

To use boost-cpp=1.7.4 with python=3.8 you need arrow-cpp=1.0.1=py38hb4b7afc_3*. This breaks if using pyarrow as there's no pyarrow build associated with that arrow-cpp version and pyarrow pins to exact builds of arrow-cpp.

Relevant arrow-cpp info: https://anaconda.org/conda-forge/arrow-cpp/files/modal/info/5f62080f78d4370a3ca2bad5

Switch to dynamic linking for boost-cpp

There have been enough issues arise due to static linking of symbols that dynamic linking is likely to be safer, even if it increases the install sizes to include boost-cpp

Use lib instead of $PREFIX/lib for CMAKE_INSTALL_LIBDIR

My understanding (from cargo cult wisdom on Google searches) is that relative paths are preferred in this variable when also specifying CMAKE_INSTALL_PREFIX. This has better compatibility with pkgconfig / cpack

After importing pyarrow importing mysqldb throws exception

Somehow pyarrow modifies the conda environment in a way that mysqldb (mysqlclient) cannot be imported. Trying to import MySQLdb throws error libstdc++.so.6 error. cannot allocate memory in static TLS block.

aws-sdk-cpp not pinned in some cases

Earlier versions of arrow-cpp that depend on aws-sdk-cpp lack a pinning for it (like this one). These likely preceded PR ( conda-forge/aws-sdk-cpp-feedstock#126 ), which added run_exports to aws-sdk-cpp. With these older packages we may either want to mark them broken or hotfix them (assuming we know what the correct pinnings were).

Do we still want the subpackage pin as x.x.x?

Is arrow-cpp stable enough now to move this to x.x or even x

Undeclared dependency on very recent libcxx

Comment:

The linux build of arrow-cpp 7.0.0 on conda-forge (py310hf3556ec_6_cpu) dynamically loads a symbol containing GLIBCXX_3.4.30. Unfortunately mamba sometimes pulls in libstdcxx-ng 11.2.0 which only exports GLIBCXX_3.4.29. You can work around this by forcing the environment to include libstdcxx-ng 12.1.0 instead (which was only released 10 days ago) but my layman's understanding is that your arrow-cpp package should probably declare a dependency on this very recent libstdcxx-ng in order to avoid the older libstdcxx-ng being a valid solution.

Other versions of arrow-cpp are affected: the older Linux build of arrow-cpp 7.0.0 (py310h68f40c4_4_cpu) did not have this libcxx dependency, but the new 8.0.0 build py310hf3556ec_1_cpu does have it.

Unpin very old GCC on PPC

I've tried to unpin the very old GCC on PPC in #848, but this runs into a myriad of errors of the following kind when trying to assemble libgandiva.so:

[clone .localalias]: error: relocation overflow

@hmaarrfk seemed to run into similar errors, hence the comment in the cbc.yaml

Originally posted by @h-vetinari in #848 (comment)

PyArrow builds aren't correctly pinning to the locally produced arrow-cpp build

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

For example, see here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=538009&view=logs&j=3b077b59-19b3-5c3c-d9ea-4f2d359714ca&t=4427250c-9566-5227-2be0-fd328c273206

This produced 4 packages:

    /home/conda/feedstock_root/build_artifacts/linux-64/arrow-cpp-8.0.1-py39h2531139_0_cpu.tar.bz2
    /home/conda/feedstock_root/build_artifacts/linux-64/arrow-cpp-proc-3.0.0-cpu.tar.bz2
    /home/conda/feedstock_root/build_artifacts/linux-64/pyarrow-8.0.1-py39h42d110c_0_cpu.tar.bz2
    /home/conda/feedstock_root/build_artifacts/linux-64/pyarrow-tests-8.0.1-py39h42d110c_0_cpu.tar.bz2

If we look at the pyarrow-8.0.1-py39h42d110c_0_cpu build we can see that it pins to the incorrect arrow-cpp build:

pyarrow 8.0.1 py39h42d110c_0_cpu
--------------------------------
file name   : pyarrow-8.0.1-py39h42d110c_0_cpu.tar.bz2
name        : pyarrow
version     : 8.0.1
build       : py39h42d110c_0_cpu
build number: 0
size        : 3.3 MB
license     : Apache-2.0
subdir      : linux-64
url         : https://conda.anaconda.org/conda-forge/linux-64/pyarrow-8.0.1-py39h42d110c_0_cpu.tar.bz2
md5         : f8e42cd6b5bee024305502104efba504
timestamp   : 2022-07-29 21:55:46 UTC
constraints :
  - arrow-cpp-proc * cpu
dependencies:
  - arrow-cpp 8.0.1 py39h811ffd7_0_cpu
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - numpy >=1.16,<2.0a0
  - numpy >=1.19.5,<2.0a0
  - parquet-cpp 1.5.1.*
  - python >=3.9,<3.10.0a0
  - python_abi 3.9.* *_cp39

This is causing issues where for example we're building arrow-cpp with two abseil versions, but the pyarrow builds are only depending on arrow-cpp packages built against one of those abseil versions.

Installed packages

n/a

Environment info

n/a

re-render for np113

@jakirkham can I force a re-render to build for numpy 1.13 on this (and a couple downstream packages) ?

do we need to push a re-render PR (that is empty)?

eg. for recipes with numpy x.x

Yank older broken builds?

In short: Any Linux builds for pyarrow versions 6.0.1 and newer which do not have the dependency libstdcxx-ng >=12 should be yanked from Conda.

The following error occurs on Linux:

$ micromamba create -n test pyarrow 'libstdcxx-ng<11.4' -c conda-forge
.....
$ micromamba activate test
$ python
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: /opt/conda/envs/test/lib/python3.10/site-packages/pyarrow/../../../libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/envs/test/lib/python3.10/site-packages/pyarrow/../../../libarrow.so.800)

The problem is that there are some old builds of pyarrow which have an incorrect dependency on libstdcxx-ng. In this case, pyarrow=8.0.0=py310h225c066_1_cpu gets installed, which incorrectly depends on libstdcxx-ng >=10.3.0. Newer builds of the package correctly have libstdcxx-ng >=12.

The newest version of pyarrow in conda-forge which is actually compatible with libstdcxx-ng=11 appears to be 6.0.0 - insofar as the above error does not happen on import.

Original issue: JuliaPy/PythonCall.jl#247

newly released libthrift-0.14.2 renders pyarrow unusable

After reinstalling a previously working conda environment, I ran into this error using pandas.read_parquet

Traceback (most recent call last):
  File "/notebooks/xxxx/tests.py", line 35, in test_xxxx
[....]
  File "/notebooks/achim/.conda/envs/po-achim/lib/python3.9/site-packages/pandas/util/_decorators.py", line 199, in wrapper
    return func(*args, **kwargs)
  File "/notebooks/achim/.conda/envs/po-achim/lib/python3.9/site-packages/pandas/core/frame.py", line 2455, in to_parquet
    return to_parquet(
  File "/notebooks/achim/.conda/envs/po-achim/lib/python3.9/site-packages/pandas/io/parquet.py", line 386, in to_parquet
    impl = get_engine(engine)
  File "/notebooks/achim/.conda/envs/po-achim/lib/python3.9/site-packages/pandas/io/parquet.py", line 36, in get_engine
    raise ImportError(
ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - libthrift.so.0.14.1: cannot open shared object file: No such file or directory
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.

The environment contains pandas 1.2.4 and pyarrow 4.0.1. This failure is reproduced on MacOS and Linux.

The same error occurred when trying to use the pyarrow engine parquet IO with pandas directly.
Here for MacOS:

Python 3.9.5 | packaged by conda-forge | (default, Jun 19 2021, 00:27:35) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas.io.parquet
>>> pandas.io.parquet.PyArrowImpl()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Caskroom/miniconda/base/envs/xxxx/lib/python3.9/site-packages/pandas/io/parquet.py", line 129, in __init__
    import pyarrow.parquet
  File "/usr/local/Caskroom/miniconda/base/envs/xxxx/lib/python3.9/site-packages/pyarrow/parquet.py", line 33, in <module>
    import pyarrow._parquet as _parquet
ImportError: dlopen(/usr/local/Caskroom/miniconda/base/envs/xxx/lib/python3.9/site-packages/pyarrow/_parquet.cpython-39-darwin.so, 2): Library not loaded: @rpath/libthrift.0.14.1.dylib
  Referenced from: /usr/local/Caskroom/miniconda/base/envs/xxxlib/libparquet.400.0.0.dylib
  Reason: image not found
>>>

As a temporary solution, I added libthrift=0.14.1 to my conda environment file and pandas/pyarrow resume working fine.

How are arrow-cpp and pyarrow related?

It looks like arrow-cpp is building the Python pieces. However there is also pyarrow, which is doing the more typical Python build. Am trying to understand what Python pieces are included in arrow-cpp vs. what are included in pyarrow. Help understanding this would be most welcome 🙂

Unpin boost-cpp

Since #234, boost-cpp is not managed by the global pinning here, which looks a bit risky. boost-cpp does not have a run-export, so thisn't viral in terms of incompatibility at least, but depending on which parts of boost are used, this could still bite us.

Unpinning boost would mean having yet more migration PRs, so if we know we're using the header-only portions, keeping the lower bound might actually be better.

Thoughts @conda-forge/arrow-cpp?

Build with flatbuffers

It looks like we are not building with flatbuffers, but maybe we should be. Though please let me know if I'm missing something

Boost pinning issue

Users have reported dependency conflicts between the boost-cpp 1.67 and 1.66: see https://issues.apache.org/jira/browse/ARROW-2783

Disable(d) ppc64le support due to maintenance burden

I'm going to remove ppc64le builds from arrow-cpp as they cause a lot of headaches. Building on Travis fails in over 80% of cases. As of yesterday, I'm also no longer able to press restart in Travis, making it even harder for me. Building things in emulation takes way too long and thus we have to wait really long until we can push any change to the Apache Arrow stack.

As this is probably a bit of a precedent, pinging as an FYI: @conda-forge/help-ppc64le @conda-forge/core @jayfurmanek

@conda-forge-admin, please re-render

@conda-forge-admin, please re-render.

Please do only one PR/build at a time

With 70 jobs for each build, this feedstock has been holding up the queue multiple times in the last week and degrades the QoS of other feedstocks. For eg: there are currently 2 PRs and 1 main build which is 210 jobs.

cc @kkraus14, @xhochy

libstdcxx-ng>=9.3.0 requirement by arrow-cpp

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Hi,

I have a requirements conflict that is caused by arrow-cpp version restriction libstdcxx-ng >=9.3.0

I'll first note the versioning details --

arrow-cpp=4.0.1=py37h628fdca_0_cpu requires libstdcxx-ng >=9.3.0
libstdcxx-ng >=9.3.0 means libstdc++.so.6.0.28
Ubuntu:18.04 comes with libstdc++.so.6.0.25 which is the equivalent of libstdcxx-ng=8.2
pyarrow + arrow-cpp are tested also on Ubuntu:18.04 regardless of conda, relying on pre installed libstdc++
there are breaking api changes between libstdc++.so.6.0.25 and libstdc++.so.6.0.28

Unfortunately, my cpp package can not run with libstdcxx-ng=9.3 [ libstdc++.so.6.0.28 ] due to api differences, but still needs to be installed with arrow-cpp latest version [ 4.0.1 ]

But, as arrow is tested on ubuntu:18.04, it thus relies on OS installed libstd [ e.g. for none conda users ]

So, I basically have two questions:

why is this restriction necessary ?
where is this restriction specified ? it's a level one requirement according to info ( listed below )it's not mentioned in the recipe meta.yaml, and I could not find it in the globally pinned packages explained in conda-forge docs

>$conda search arrow-cpp=4.0.1=py37h628fdca_0_cpu --info
arrow-cpp 4.0.1 py37h628fdca_0_cpu
----------------------------------
file name   : arrow-cpp-4.0.1-py37h628fdca_0_cpu.tar.bz2
name        : arrow-cpp
version     : 4.0.1
build       : py37h628fdca_0_cpu
build number: 0
size        : 22.2 MB
license     : Apache-2.0
subdir      : linux-64
url         : https://artifactory.sddc.mobileye.com/artifactory/api/conda/conda-condaforge-remote/linux-64/arrow-cpp-4.0.1-py37h628fdca_0_cpu.tar.bz2
md5         : 384bea498466854edc3a43de493510fe
timestamp   : 2021-05-28 15:25:35 UTC
constraints : 
  - arrow-cpp-proc * cpu
dependencies: 
  - aws-sdk-cpp >=1.8.186,<1.8.187.0a0
  - brotli >=1.0.9,<2.0a0
  - bzip2 >=1.0.8,<2.0a0
  - c-ares >=1.17.1,<2.0a0
  - gflags >=2.2.2,<2.3.0a0
  - glog >=0.5.0,<0.6.0a0
  - grpc-cpp >=1.38.0,<1.39.0a0
  - libgcc-ng >=9.3.0
  - libprotobuf >=3.16.0,<3.17.0a0
  - libstdcxx-ng >=9.3.0
  - libthrift >=0.14.1,<0.15.0a0
  - libutf8proc >=2.6.1,<3.0a0
  - lz4-c >=1.9.3,<1.9.4.0a0
  - numpy >=1.16,<2.0a0
  - orc >=1.6.8,<1.6.9.0a0
  - python >=3.7,<3.8.0a0
  - python_abi 3.7.* *_cp37m
  - re2 >=2021.4.1,<2021.4.2.0a0
  - snappy >=1.1.8,<2.0a0
  - zlib >=1.2.11,<1.3.0a0
  - zstd >=1.5.0,<1.6.0a0

Thanks in advance,
Harel

Environment (conda list):

$ conda list

Details about conda and system ( conda info ):

$ conda info

@conda-forge-admin, please re-render

@conda-forge-admin, please re-render.

arrow-cpp 7.0.0 is incompatible with tensorflow-gpu (fails with requires libprotobuf >=3.19.4,<3.20.0a0)

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

arrow-cpp 7.0.0 is not compatible with tensorflow-gpu (arrow-cpp 6.0.1 works fine).

To reproduce:

FROM centos:centos7
ARG MAMBAFORGE=https://github.com/conda-forge/miniforge/releases/download/4.11.0-4/Mambaforge-Linux-x86_64.sh
RUN curl -sSL $MAMBAFORGE -o /tmp/mambaforge.sh && bash /tmp/mambaforge.sh -bfp /usr/local
RUN mamba install --yes arrow-cpp==7.0.0 tensorflow-gpu

The error:

Encountered problems while solving:
  - package arrow-cpp-7.0.0-py39h116028d_0_cpu requires libprotobuf >=3.19.4,<3.20.0a0, but none of the providers can be installed

I have another configuration (that I haven't simplified, so reproducer not attached) that complains similarly:

Encountered problems while solving:
  - package arrow-cpp-7.0.0-py37h1be195e_2_cuda requires libprotobuf >=3.19.4,<3.20.0a0, but none of the providers can be installed

Installed packages

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
brotlipy                  0.7.0           py39h3811e60_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
certifi                   2021.10.8        py39hf3d152e_1    conda-forge
cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda                     4.11.0           py39hf3d152e_0    conda-forge
conda-package-handling    1.7.3            py39h3811e60_1    conda-forge
cryptography              36.0.1           py39h95dcef6_0    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
krb5                      1.19.2               hcc1bbae_3    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libarchive                3.5.2                hccf745f_1    conda-forge                                                                            [54/101]libcurl                   7.81.0               h2574ce0_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_12    conda-forge
libgomp                   11.2.0              h1d223b6_12    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libmamba                  0.21.2               h3985d26_0    conda-forge
libmambapy                0.21.2           py39h8bfa403_0    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsolv                   0.7.19               h780b84a_5    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_12    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     0.21.2           py39hfa8f2c8_0    conda-forge
ncurses                   6.3                  h9c3ff4c_0    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.3           py39h3811e60_1009    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1            py39hf3d152e_4    conda-forge
python                    3.9.10          h85951f9_2_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
reproc                    14.2.3               h7f98852_0    conda-forge
reproc-cpp                14.2.3               h9c3ff4c_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
ruamel_yaml               0.15.80         py39h3811e60_1006    conda-forge
setuptools                60.9.3           py39hf3d152e_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
urllib3                   1.26.8             pyhd8ed1ab_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.6.3                he1b5a44_4    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge

Environment info

active environment : None
       user config file : /root/.condarc
 populated config files : /usr/local/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.10.final.0
       virtual packages : __linux=3.10.0=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /usr/local  (writable)
      conda av data dir : /usr/local/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /usr/local/pkgs
                          /root/.conda/pkgs
       envs directories : /usr/local/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.9.10 Linux/3.10.0-1160.41.1.el7.x86_64 centos/7.9.2009 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Unable to use the latest version of pyarrow with pandas

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

I can't use the latest version of pyarrow with pandas. There are a various moving parts (pyarrow and pandas, and their respective conda-forge recipe feedstocks). Please let me know if this issue could be better addressed elsewhere.

Here is a reproducible example:

mamba create -n pyarrow-latest -y -c conda-forge --override-channels pandas pyarrow
conda activate pyarrow-latest
python --version

# Able to import pyarrow
python -c "import pyarrow; print(pyarrow.__version__)"

# Unable to export to parquet
python -c "import pandas as pd; pd.DataFrame({'a': [1]}).to_parquet('/tmp/tmp.parquet')"

# Unable to import pyarrow after pandas
python -c "import pandas; import pyarrow; print(pyarrow.__version__)"

conda deactivate

Here's the full error message from pandas when failing to run to_parquet:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/redacted/mambaforge/envs/pyarrow-latest/lib/python3.9/site-packages/pandas/util/_decorators.py", line 199, in wrapper
    return func(*args, **kwargs)
  File "/redacted/mambaforge/envs/pyarrow-latest/lib/python3.9/site-packages/pandas/core/frame.py", line 2455, in to_parquet
    return to_parquet(
  File "/redacted/mambaforge/envs/pyarrow-latest/lib/python3.9/site-packages/pandas/io/parquet.py", line 386, in to_parquet
    impl = get_engine(engine)
  File "/redacted/mambaforge/envs/pyarrow-latest/lib/python3.9/site-packages/pandas/io/parquet.py", line 36, in get_engine
    raise ImportError(
ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Missing optional dependency 'pyarrow'. pyarrow is required for parquet support. Use pip or conda to install pyarrow.
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.

Here's the full error message from importing pyarrow after pandas:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/redacted/mambaforge/envs/pyarrow-latest/lib/python3.9/site-packages/pyarrow/__init__.py", line 63, in <module>
    import pyarrow.lib as _lib
ImportError: /redacted/GCCcore/6.3.0/lib64/libgcc_s.so.1: version `GCC_7.0.0' not found (required by /redacted/mambaforge/envs/pyarrow-latest/lib/python3.9/site-packages/pyarrow/../../../libarrow.so.400)

Environment (conda list -n pyarrow-latest):

$ conda list -n pyarrow-latest
# packages in environment at /redacted/mambaforge/envs/pyarrow-latest:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
arrow-cpp                 4.0.1           py39h96e570c_1_cpu    conda-forge
aws-c-cal                 0.5.10               h95a6274_1    conda-forge
aws-c-common              0.6.2                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7               h211b232_12    conda-forge
aws-c-io                  0.10.4               h72290dd_0    conda-forge
aws-checksums             0.1.11               ha31a3da_7    conda-forge
aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
brotli                    1.0.9                h9c3ff4c_4    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.5.30            ha878542_0    conda-forge
certifi                   2021.5.30        py39hf3d152e_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
grpc-cpp                  1.38.0               h36ce80c_2    conda-forge
krb5                      1.19.1               hcc1bbae_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libcurl                   7.77.0               h2574ce0_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgfortran-ng            9.3.0               hff62375_19    conda-forge
libgfortran5              9.3.0               hff62375_19    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
libprotobuf               3.16.0               h780b84a_0    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libthrift                 0.14.1               he6d91bd_1    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
numpy                     1.20.3           py39hdbf815f_1    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
orc                       1.6.8                h58a87f1_0    conda-forge
pandas                    1.2.4            py39hde0f152_0    conda-forge
parquet-cpp               1.5.1                         1    conda-forge
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
pyarrow                   4.0.1           py39h3ebc44c_1_cpu    conda-forge
python                    3.9.4           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
re2                       2021.06.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
s2n                       1.0.10               h9b69904_0    conda-forge
setuptools                49.6.0           py39hf3d152e_3    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sqlite                    3.35.5               h74cdb3f_0    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge

Note that the pyarrow integration with pandas works fine with pyarrow 0.17.1:

mamba create -n pyarrow-legacy -y -c conda-forge --override-channels pandas pyarrow=0.17
conda activate pyarrow-legacy
python -c "import pandas as pd; pd.DataFrame({'a': [1]}).to_parquet('/tmp/tmp.parquet')"
conda deactivate

Environment (conda list -n pyarrow-legacy):

$ conda list -n pyarrow-legacy
# packages in environment at /redacted/mambaforge/envs/pyarrow-legacy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
abseil-cpp                20210324.1           h9c3ff4c_0    conda-forge
arrow-cpp                 0.17.1          py39hc046a4d_33_cpu    conda-forge
aws-c-cal                 0.5.10               h95a6274_1    conda-forge
aws-c-common              0.6.2                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7               h211b232_12    conda-forge
aws-c-io                  0.10.4               h72290dd_0    conda-forge
aws-checksums             0.1.11               ha31a3da_7    conda-forge
aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_3    conda-forge
brotli                    1.0.9                h9c3ff4c_4    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.5.30            ha878542_0    conda-forge
certifi                   2021.5.30        py39hf3d152e_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
grpc-cpp                  1.37.1               h2519f57_2    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
krb5                      1.19.1               hcc1bbae_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libcurl                   7.77.0               h2574ce0_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgfortran-ng            9.3.0               hff62375_19    conda-forge
libgfortran5              9.3.0               hff62375_19    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
libprotobuf               3.16.0               h780b84a_0    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libthrift                 0.14.1               he6d91bd_1    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
numpy                     1.20.3           py39hdbf815f_1    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
orc                       1.6.7                h89a63ab_2    conda-forge
pandas                    1.2.4            py39hde0f152_0    conda-forge
parquet-cpp               1.5.1                         1    conda-forge
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
pyarrow                   0.17.1          py39hbbe4ea4_33_cpu    conda-forge
python                    3.9.4           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
re2                       2021.04.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
s2n                       1.0.10               h9b69904_0    conda-forge
setuptools                49.6.0           py39hf3d152e_3    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sqlite                    3.35.5               h74cdb3f_0    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge

Details about conda and system ( conda info ):

$ conda info
     active environment : base
    active env location : /redacted/mambaforge
            shell level : 1
       user config file : /redacted/.condarc
 populated config files : /redacted/mambaforge/.condarc
                          /redacted/.condarc
          conda version : 4.10.1
    conda-build version : 3.21.4
         python version : 3.9.4.final.0
       virtual packages : __linux=3.10.0=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /redacted/mambaforge  (writable)
      conda av data dir : /redacted/mambaforge/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
          package cache : /redacted/mambaforge/pkgs
                          /redacted/.conda/pkgs
       envs directories : /redacted/mambaforge/envs
                          /redacted/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.1 requests/2.25.1 CPython/3.9.4 Linux/3.10.0-1062.12.1.el7.x86_64 centos/7.7.1908 glibc/2.17
                UID:GID : 2008801:1950
             netrc file : None
           offline mode : False

Split package?

Should we split this package into a portion without python and one with the python bindings?

Feels a little weird that arrow-cpp requires a python to be installed at all times.

Thoughts?

Importing pyarrow on Summit fails with "Illegal instruction"

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

If I install pyarrow from conda-forge on the Summit cluster and try to import it I get an error message about an illegal instruction. I believe the source of this illegal instructions is the ppc64le version of libarrow.so distributed through conda-forge.

Here is a minimal way to reproduce:

cd /ccs/proj/gen119/mpapadak/
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-ppc64le.sh
bash ./Miniconda3-latest-Linux-ppc64le.sh -b -p ./conda
source ./conda/etc/profile.d/conda.sh
conda create --yes --name test -c conda-forge -c defaults python=3.8 pyarrow
conda activate test
bsub -W 10 -P gen119 -nnodes 1 -Is python -c 'import pyarrow'

The output of the last command is:

Job <1271336> is submitted to default queue <batch>.
<<Waiting for dispatch ...>>
<<Starting on batch3>>
/ccs/home/mpapadak/.lsbatch/1628210524.1271336: line 8: 67503 Illegal instruction     (core dumped) python -c "import pyarrow"

Environment (conda list):

$ conda list
# packages in environment at /ccs/proj/gen119/mpapadak/conda/envs/test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
abseil-cpp                20210324.2           h3b9df90_0    conda-forge
arrow-cpp                 5.0.0           py38h16d7865_1_cpu    conda-forge
aws-c-cal                 0.5.11               hb3fac3d_0    conda-forge
aws-c-common              0.6.2                h4e0d66e_0    conda-forge
aws-c-event-stream        0.2.7               h76da5f2_13    conda-forge
aws-c-io                  0.10.5               hf6a6c7c_0    conda-forge
aws-checksums             0.1.11               hfe76d68_7    conda-forge
aws-sdk-cpp               1.8.186              h90855e8_3    conda-forge
bzip2                     1.0.8                h4e0d66e_4    conda-forge
c-ares                    1.17.1               h4e0d66e_1    conda-forge
ca-certificates           2021.5.30            h1084571_0    conda-forge
certifi                   2021.5.30        py38hf8b3453_0    conda-forge
gflags                    2.2.2             hb209c28_1004    conda-forge
glog                      0.5.0                h4040248_0    conda-forge
grpc-cpp                  1.39.0               hd4cf575_1    conda-forge
krb5                      1.19.2               haf43566_0    conda-forge
ld_impl_linux-ppc64le     2.36.1               ha35d02b_2    conda-forge
libblas                   3.9.0               10_openblas    conda-forge
libbrotlicommon           1.0.9                h4e0d66e_5    conda-forge
libbrotlidec              1.0.9                h4e0d66e_5    conda-forge
libbrotlienc              1.0.9                h4e0d66e_5    conda-forge
libcblas                  3.9.0               10_openblas    conda-forge
libcurl                   7.78.0               he415e40_0    conda-forge
libedit                   3.1.20191231         h41a240f_2    conda-forge
libev                     4.33                 h6eb9509_1    conda-forge
libevent                  2.1.10               h7da059d_3    conda-forge
libffi                    3.3                  hea85c5d_2    conda-forge
libgcc-ng                 11.1.0               h16e2c27_8    conda-forge
libgfortran-ng            11.1.0               hfdc3801_8    conda-forge
libgfortran5              11.1.0               h24cf76c_8    conda-forge
libgomp                   11.1.0               h16e2c27_8    conda-forge
liblapack                 3.9.0               10_openblas    conda-forge
libnghttp2                1.43.0               h42039ad_0    conda-forge
libopenblas               0.3.17          pthreads_h486567c_1    conda-forge
libprotobuf               3.16.0               h690f14c_0    conda-forge
libssh2                   1.9.0                ha5a9321_6    conda-forge
libstdcxx-ng              11.1.0               h8186cfa_8    conda-forge
libthrift                 0.14.2               h54f692e_1    conda-forge
libutf8proc               2.6.1                h4e0d66e_0    conda-forge
lz4-c                     1.9.3                h3b9df90_1    conda-forge
ncurses                   6.2                  hea85c5d_4    conda-forge
numpy                     1.21.1           py38h81bd36e_0    conda-forge
openssl                   1.1.1k               h4e0d66e_0    conda-forge
orc                       1.6.9                h408c992_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
pip                       21.2.2             pyhd8ed1ab_0    conda-forge
pyarrow                   5.0.0           py38hec7d1ba_1_cpu    conda-forge
python                    3.8.10          h82ac395_1_cpython    conda-forge
python_abi                3.8                      2_cp38    conda-forge
re2                       2021.06.01           h3b9df90_0    conda-forge
readline                  8.1                  h5c45dff_0    conda-forge
s2n                       1.0.10               h97db324_0    conda-forge
setuptools                49.6.0           py38hf8b3453_3    conda-forge
snappy                    1.1.8                hb209c28_3    conda-forge
sqlite                    3.36.0               h4e2196e_0    conda-forge
tk                        8.6.10               h38e1d09_1    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                h6eb9509_1    conda-forge
zlib                      1.2.11            h6eb9509_1010    conda-forge
zstd                      1.5.0                h65c4b1a_0    conda-forge

Details about conda and system ( conda info ):

$ conda info

     active environment : test
    active env location : /ccs/proj/gen119/mpapadak/conda/envs/test
            shell level : 1
       user config file : /ccs/home/mpapadak/.condarc
 populated config files : /ccs/home/mpapadak/.condarc
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __cuda=10.1=0
                          __linux=4.14.0=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=ppc64le
       base environment : /ccs/proj/gen119/mpapadak/conda  (writable)
      conda av data dir : /ccs/proj/gen119/mpapadak/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-ppc64le
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-ppc64le
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /ccs/proj/gen119/mpapadak/conda/pkgs
                          /ccs/home/mpapadak/.conda/pkgs
       envs directories : /ccs/proj/gen119/mpapadak/conda/envs
                          /ccs/home/mpapadak/.conda/envs
               platform : linux-ppc64le
             user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.5 Linux/4.14.0-115.21.2.el7a.ppc64le rhel/7.6 glibc/2.17
                UID:GID : 13973:25182
             netrc file : None
           offline mode : False

pyarrow needs a run_constrained on s3fs<0.5

Due to https://issues.apache.org/jira/browse/ARROW-10433

Updating LLVM

Looks like LLVM is pinned to 7. Is this needed for some reason? If not, would it be possible to upgrade to a more recent version (like 9)?

Drop pandas dependency

We currently have a hard dependency on pandas in the pyarrow package whereas upstream only optionally depends on it. As people start to use pyarrow in some usecases without pandas, would it be ok to drop it? Any other opinion @conda-forge/arrow-cpp ?

Can the aws-sdk-cpp pin be loosened?

Comment:

I've been struggling with conda-forge dependency issues and now believe to have isolated it to the pinning on aws-sdk-cpp within this repo. Does this pin still need to exist with the current 6.0 branch version of 6.0.2, which should have ARROW-15141 fixed?

Thank you.

Remove accidentally committed pycache directory

Issue:

Environment (conda list):

$ conda list

Details about conda and system ( conda info ):

$ conda info

Did you really want so many abi branches

Comment:

Seems like we have quite a few ABI branches. I would like to help get the abseil migration complete but it seems like I would have to backport quite a few changes. Thoughts?

How to use CUDA enabled arrow-cpp from another recipe?

Consider conda-forge/pyarrow-feedstock#101 where the meta.yaml contains:

outputs:
  - name: pyarrow
<snip>
    requirements:
<snip>
      host:
        - arrow-cpp {{ version }}
<snip>
      run:
        - arrow-cpp {{ version }}
<snip>
      run_constrained:
        - arrow-cpp-proc * {{ build_ext }}

and I would expect that build_ext == cuda in the last line would force to pick up arrow-cpp packages (in host and run sections) with *_cuda build strings.
But as seen in https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=150318&view=logs&jobId=4f922444-fdfe-5dcf-b824-02f86439ef14&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=937c195f-508d-5135-dc9f-d4b5730df0f7 the arrow-cpp with *_cpu build string is used.

One possibility would be to specify

        - arrow-cpp {{ version }} *_{{ build_ext }}

but that would make the usage of arrow-cpp-proc unnecessary.

Hence the question in the title..

PyArrow cuda packages have `cudatoolkit 9.2|9.2.*` dependency

See _cuda packages here: https://anaconda.org/conda-forge/pyarrow/files?version=0.17.1

It looks like the ignore_run_export from arrow-cpp doesn't carry over to pyarrow in the split package?

Add run_exports

We may want to add run_exports here (as in AnacondaRecipes) so that ensuring abi compatibility is a bit simpler for downstream applications.

build:
  run_exports:
    - {{ pin_subpackage("arrow-cpp", max_pin="x.x") }}

Alternatively we may want to merge some of the changes from AnacondaRecipes

LLVM License violation

Please include the license of LLVM as arrow-cpp is statically linking to it and is a requirement of the MIT and UIUC licenses.

Note that conda-build can include multiple files as a yaml list in in license_file.

Gandiva module not working / wrong version of re2

A fresh environment with just pyarrow on Ubuntu installed gives:

In [1]: from pyarrow import gandiva                                                                                                                                                                                
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-94a5c86fabcc> in <module>
----> 1 from pyarrow import gandiva

ImportError: libre2.so.6: cannot open shared object file: No such file or directory

Quoting @pearu from conda-forge/pyarrow-feedstock#101 (comment):

The latest re2 package (v 2020.05) provides libre2.so.7 but arrow-cpp 0.16 was built against re2 package v 2020.04 that provides libre2.so.6.

And I can confirm that doing a conda install re2=2020.04 fixes the error.

Should the re2 package be pinned?

(arrow-017) $ conda info

     active environment : arrow-017
    active env location : /home/joris/miniconda3/envs/arrow-017
            shell level : 2
       user config file : /home/joris/.condarc
 populated config files : /home/joris/.condarc
          conda version : 4.8.3
    conda-build version : 3.18.9
         python version : 3.7.3.final.0
       virtual packages : __glibc=2.27
       base environment : /home/joris/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/joris/miniconda3/pkgs
                          /home/joris/.conda/pkgs
       envs directories : /home/joris/miniconda3/envs
                          /home/joris/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.3 Linux/4.15.0-1080-oem ubuntu/18.04.4 glibc/2.27
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

(arrow-017) $ conda list
# packages in environment at /home/joris/miniconda3/envs/arrow-017:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
abseil-cpp                20200225.2           he1b5a44_0    conda-forge
arrow-cpp                 0.17.0           py38h9d6855f_0    conda-forge
aws-sdk-cpp               1.7.164              hc831370_1    conda-forge
backcall                  0.1.0                      py_0    conda-forge
boost-cpp                 1.72.0               h8e57a91_0    conda-forge
brotli                    1.0.7             he1b5a44_1001    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
c-ares                    1.15.0            h516909a_1001    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py38h32f6830_0    conda-forge
curl                      7.69.1               h33f0ec9_0    conda-forge
decorator                 4.4.2                      py_0    conda-forge
gflags                    2.2.2             he1b5a44_1002    conda-forge
glog                      0.4.0                h49b9bf7_3    conda-forge
grpc-cpp                  1.28.1               h8e748ff_2    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
ipython                   7.14.0           py38h32f6830_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.17.0           py38h32f6830_0    conda-forge
krb5                      1.17.1               h2fd8d38_0    conda-forge
ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libcurl                   7.69.1               hf7181ac_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libevent                  2.1.10               h72c5cf5_0    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.5.0                hdf63c60_6    conda-forge
liblapack                 3.8.0               16_openblas    conda-forge
libopenblas               0.3.9                h5ec1e0e_0    conda-forge
libprotobuf               3.11.4               h8b12597_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
llvm-openmp               10.0.0               hc9558a2_0    conda-forge
lz4-c                     1.9.2                he1b5a44_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.18.4           py38h8854b6b_0    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
pandas                    1.0.3            py38hcb8c335_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.7.0              pyh9f0ad1d_0    conda-forge
pexpect                   4.8.0            py38h32f6830_1    conda-forge
pickleshare               0.7.5           py38h32f6830_1001    conda-forge
prompt-toolkit            3.0.5                      py_0    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pyarrow                   0.17.0           py38h56ab25f_0    conda-forge
pygments                  2.6.1                      py_0    conda-forge
python                    3.8.2           he5300dc_7_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2020.1             pyh9f0ad1d_0    conda-forge
re2                       2020.04.01           he1b5a44_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                46.3.1           py38h32f6830_0    conda-forge
six                       1.14.0                     py_1    conda-forge
snappy                    1.1.8                he1b5a44_1    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
thrift-cpp                0.13.0               h62aa4f2_2    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
traitlets                 4.3.3            py38h32f6830_1    conda-forge
wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
xz                        5.2.5                h516909a_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h6597ccf_3    conda-forge

[warning] failed package validation and/or copy for commit 989268e11365101c9d72a09cd284e303e99873fb

Hi @conda-forge/arrow-cpp! This is the friendly automated conda-forge-webservice!

It appears that one or more of your feedstock's outputs is either invalid or
did not copy from the staging channel (cf-staging) to the production channel (conda-forge). :(

This failure can happen for a lot of reasons, including an outdated feedstock
token, a feedstock output that is not allowed for that feedstock, or a feedstock output
with files that are not allowed for that output. Below we have put some information
about the failure to help you debug it.

Rerendering the feedstock will usually fix these problems.

If you have any issues or questions, you can find us on gitter in the
community chat room or you can bump us right here.

error messages:

invalid md5 sum for artifact https://conda.anaconda.org/cf-staging/linux-ppc64le/pyarrow-2.0.0-py37h66123b5_14_cpu.tar.bz2

Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io

Comment:

When I use Trainer module from transformers package, I find another problem: huggingface/datasets#3310 (comment)

I don't what happen.

The environment info by conda list:

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=main
_openmp_mutex=5.1=1_gnu
absl-py=1.0.0=pypi_0
accelerate=0.13.2=pypi_0
aiohttp=3.8.3=pypi_0
aiosignal=1.2.0=pypi_0
altair=4.2.0=pypi_0
async-timeout=4.0.2=pypi_0
attrs=22.1.0=pypi_0
backports-zoneinfo=0.2.1=pypi_0
blinker=1.5=pypi_0
ca-certificates=2022.4.26=h06a4308_0
cachetools=5.1.0=pypi_0
certifi=2022.5.18.1=py38h06a4308_0
charset-normalizer=2.0.12=pypi_0
click=8.1.3=pypi_0
commonmark=0.9.1=pypi_0
cudatoolkit=11.3.1=h2bc3f7f_2
cycler=0.11.0=pypi_0
cython=0.29.30=pypi_0
datasets=2.5.1=pypi_0
decorator=5.1.1=pypi_0
deepspeed=0.6.7=pypi_0
dill=0.3.5.1=pypi_0
docker-pycreds=0.4.0=pypi_0
docopt=0.6.2=pypi_0
entrypoints=0.4=pypi_0
et-xmlfile=1.1.0=pypi_0
filelock=3.7.0=pypi_0
fonttools=4.33.3=pypi_0
frozenlist=1.3.1=pypi_0
fsspec=2022.8.2=pypi_0
gensim=4.2.0=pypi_0
gitdb=4.0.9=pypi_0
gitpython=3.1.29=pypi_0
google-auth=2.6.6=pypi_0
google-auth-oauthlib=0.4.6=pypi_0
grpcio=1.46.3=pypi_0
hjson=3.1.0=pypi_0
huggingface-hub=0.9.1=pypi_0
idna=3.3=pypi_0
importlib-metadata=4.11.4=pypi_0
importlib-resources=5.10.0=pypi_0
jinja2=3.1.2=pypi_0
joblib=1.1.0=pypi_0
jsonschema=4.16.0=pypi_0
kiwisolver=1.4.2=pypi_0
ld_impl_linux-64=2.38=h1181459_1
libffi=3.3=he6710b0_2
libgcc-ng=11.2.0=h1234567_0
libgomp=11.2.0=h1234567_0
libstdcxx-ng=11.2.0=h1234567_0
markdown=3.3.7=pypi_0
markupsafe=2.1.1=pypi_0
matplotlib=3.5.2=pypi_0
multidict=6.0.2=pypi_0
multiprocess=0.70.13=pypi_0
ncurses=6.3=h7f8727e_2
ninja=1.10.2.4=pypi_0
nltk=3.7=pypi_0
numpy=1.22.4=pypi_0
oauthlib=3.2.0=pypi_0
openprompt=1.0.1=pypi_0
openpyxl=3.0.10=pypi_0
openssl=1.1.1o=h7f8727e_0
packaging=21.3=pypi_0
pandas=1.4.2=pypi_0
pathtools=0.1.2=pypi_0
pillow=9.1.1=pypi_0
pip=21.2.4=py38h06a4308_0
pipreqs=0.4.11=pypi_0
pkgutil-resolve-name=1.3.10=pypi_0
pkuseg=0.0.25=pypi_0
prettytable=3.3.0=pypi_0
promise=2.3=pypi_0
protobuf=3.20.1=pypi_0
psutil=5.9.3=pypi_0
py-cpuinfo=8.0.0=pypi_0
pyahocorasick=1.4.4=pypi_0
pyarrow=9.0.0=pypi_0
pyasn1=0.4.8=pypi_0
pyasn1-modules=0.2.8=pypi_0
pydantic=1.10.2=pypi_0
pydeck=0.8.0b4=pypi_0
pygments=2.13.0=pypi_0
pympler=1.0.1=pypi_0
pyparsing=3.0.9=pypi_0
pyrsistent=0.18.1=pypi_0
python=3.8.13=h12debd9_0
python-dateutil=2.8.2=pypi_0
pytorch-crf=0.7.2=pypi_0
pytz=2022.1=pypi_0
pytz-deprecation-shim=0.1.0.post0=pypi_0
pyyaml=6.0=pypi_0
readline=8.1.2=h7f8727e_1
regex=2022.4.24=pypi_0
requests=2.27.1=pypi_0
requests-oauthlib=1.3.1=pypi_0
responses=0.18.0=pypi_0
rich=12.6.0=pypi_0
rouge=1.0.0=pypi_0
rsa=4.8=pypi_0
sacremoses=0.0.53=pypi_0
scikit-learn=1.1.1=pypi_0
scipy=1.8.1=pypi_0
semver=2.13.0=pypi_0
sentence-transformers=2.2.2=pypi_0
sentencepiece=0.1.96=pypi_0
sentry-sdk=1.10.1=pypi_0
seqeval=1.2.2=pypi_0
setproctitle=1.3.2=pypi_0
setuptools=58.0.4=pypi_0
shortuuid=1.0.9=pypi_0
simpletransformers=0.63.9=pypi_0
six=1.16.0=pypi_0
smart-open=6.0.0=pypi_0
smmap=5.0.0=pypi_0
sqlite=3.38.3=hc218d9a_0
streamlit=1.13.0=pypi_0
tk=8.6.11=h1ccaba5_1
tokenizers=0.12.1=pypi_0
toml=0.10.2=pypi_0
toolz=0.12.0=pypi_0
torch=1.12.1=pypi_0
torch-summary=1.4.5=pypi_0
torch-tb-profiler=0.4.0=pypi_0
tornado=6.2=pypi_0
tqdm=4.64.0=pypi_0
transformers=4.22.1=pypi_0
typing-extensions=4.2.0=pypi_0
tzdata=2022.5=pypi_0
tzlocal=4.2=pypi_0
urllib3=1.26.12=pypi_0
validators=0.20.0=pypi_0
wandb=0.13.4=pypi_0
watchdog=2.1.9=pypi_0
wcwidth=0.2.5=pypi_0
werkzeug=2.1.2=pypi_0
wheel=0.37.1=pyhd3eb1b0_0
xxhash=3.0.0=pypi_0
xz=5.2.5=h7f8727e_1
yacs=0.1.8=pypi_0
yarg=0.1.9=pypi_0
yarl=1.8.1=pypi_0
zipp=3.8.0=pypi_0
zlib=1.2.12=h7f8727e_2

aws error after writing parquet file

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

After converting a dataframe to parquet, I get an aws-related error:

Fatal error condition occurred in D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
at 0x7FFFB9D94380: aws_backtrace_print
at 0x7FFFB9D83560: aws_fatal_assert
at 0x7FFFA8FB5F10: aws_event_loop_wait_for_stop_completion
at 0x7FFFB9D91470: aws_ref_count_release
at 0x7FFFA8FB3D80: aws_server_bootstrap_set_alpn_callback
at 0x7FFFB9D91470: aws_ref_count_release
at 0x7FFFA8FB3760: aws_client_bootstrap_release
at 0x7FFFC31976F0: Aws::Crt::Io::ClientBootstrap::~ClientBootstrap
at 0x7FFFC2F7EB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFFD9E71BA0: execute_onexit_table
at 0x7FFFD9E71BA0: execute_onexit_table
at 0x7FFFD9E71BA0: execute_onexit_table
at 0x7FFFC2F7EB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFFC2F7EB40: Aws::Utils::Stream::SimpleStreamBuf::xsputn
at 0x7FFFDCD55AA0: RtlAnsiStringToUnicodeString
at 0x7FFFDCD590F0: LdrShutdownProcess
at 0x7FFFDCD9CAB0: RtlExitUserProcess
at 0x7FFFDB4ED620: ExitProcess
at 0x7FFFD9E80C50: exit
at 0x7FFFD9E80C50: exit
at 0x7FF60DDA1160: OPENSSL_Applink
at 0x7FFFDB4E7960: BaseThreadInitThunk
at 0x7FFFDCD8A2D0: RtlUserThreadStart

D: does not exist on my system and D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74 looks a lot like a conda-build path. This may need to be narrowed down to the correct aws-* package but I am not familiar with those.

This code recreates the issue:

import numpy as np
import pandas as pd

def test_error():

    df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

    df.to_parquet('test.parquet')

if __name__ == '__main__':
    test_error()

Environment (conda list):

$conda list
# Name                    Version                   Build  Channel
abseil-cpp                20210324.2           h0e60522_0    
alabaster                 0.7.12                     py_0    
appdirs                   1.4.4              pyh9f0ad1d_0    
argon2-cffi               20.1.0           py39hb82d6ee_2    
arrow-cpp                 5.0.0           py39h22074fa_10_cpu    
astroid                   2.5              py39hcbf5309_1    
async_generator           1.10                       py_0    
atomicwrites              1.4.0              pyh9f0ad1d_0    
attrs                     20.3.0             pyhd3deb0d_0    
aws-c-auth                0.6.4                hd061354_3    
aws-c-cal                 0.5.12               h395cb70_2    
aws-c-common              0.6.11               h8ffe710_0    
aws-c-compression         0.2.14               h82db3a7_2    
aws-c-event-stream        0.2.7               he02faa2_24    
aws-c-http                0.6.6                h77bbe4a_1    
aws-c-io                  0.10.9               h3a8db39_3    
aws-c-mqtt                0.7.8                h2f34d13_2    
aws-c-s3                  0.1.27               habe44f2_1    
aws-checksums             0.1.12               h82db3a7_1    
aws-crt-cpp               0.17.1               haf3871f_3    
aws-sdk-cpp               1.9.120              h37b9141_0    
babel                     2.9.0              pyhd3deb0d_0    
backcall                  0.2.0              pyh9f0ad1d_0    
backports                 1.0                        py_2    
backports.functools_lru_cache 1.6.1                      py_0    
beautifulsoup4            4.9.3              pyhb0f4dca_0    
black                     20.8b1                     py_1    
bleach                    3.3.0              pyh44b312d_0    
blosc                     1.21.0               h0e60522_0    
brotli                    1.0.9                h0e60522_4    
brotlipy                  0.7.0           py39hb82d6ee_1001    
bzip2                     1.0.8                h8ffe710_4    
c-ares                    1.18.1               h8ffe710_0    
ca-certificates           2021.10.8            h5b45459_0    
cached-property           1.5.2                hd8ed1ab_1    
cached_property           1.5.2              pyha770c72_1    
cartopy                   0.19.0.post1     py39hfa2520c_0    
certifi                   2021.10.8        py39hcbf5309_1    
cffi                      1.14.5           py39h0878f49_0    
chardet                   4.0.0            py39hcbf5309_1    
click                     7.1.2              pyh9f0ad1d_0    
colorama                  0.4.4              pyh9f0ad1d_0    
commonmark                0.9.1                      py_0    
coverage                  5.4              py39hb82d6ee_0    
cryptography              3.4.6            py39hd8d06c1_0    
cycler                    0.10.0                     py_2    
dataclasses               0.7                pyhb2cacf7_7    
decorator                 4.4.2                      py_0    
defusedxml                0.6.0                      py_0    
docutils                  0.16             py39hcbf5309_3    
dpkt                      1.9.5              pyh44b312d_0    
entrypoints               0.3             pyhd8ed1ab_1003    
freetype                  2.10.4               h546665d_1    
future                    0.18.2           py39hcbf5309_3    
geos                      3.9.1                h39d44d4_2    
gflags                    2.2.2             ha925a31_1004    
glog                      0.5.0                h4797de2_0    
grpc-cpp                  1.40.0               h2431d41_2    
h5py                      3.4.0           nompi_py39hd4deaf1_101    
hdf5                      1.12.1          nompi_h2a0e4a3_100    
icu                       68.1                 h0e60522_0    
idna                      2.10               pyh9f0ad1d_0    
imagesize                 1.2.0                      py_0    
importlib-metadata        3.7.0            py39hcbf5309_0    
importlib_metadata        3.7.0                hd8ed1ab_0    
iniconfig                 1.1.1              pyh9f0ad1d_0    
intel-openmp              2020.3             h57928b3_311    
ipykernel                 5.5.0            py39h832f523_1    
ipython                   7.21.0           py39h832f523_0    
ipython_genutils          0.2.0                      py_1    
ipywidgets                7.6.3              pyhd3deb0d_0    
isort                     5.7.0              pyhd8ed1ab_0    
jbig                      2.1               h8d14728_2003    
jedi                      0.18.0           py39hcbf5309_2    
jinja2                    2.11.3             pyh44b312d_0    
jpeg                      9d                   he774522_0    
jsonschema                3.2.0                      py_2    
jupyter_client            6.1.11             pyhd8ed1ab_1    
jupyter_core              4.7.1            py39hcbf5309_0    
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    
jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    
kiwisolver                1.3.1            py39h2e07f2f_1    
krb5                      1.19.2               hbae68bd_0    
lazy-object-proxy         1.5.2            py39hb82d6ee_1    
lcms2                     2.12                 h2a16943_0    
lerc                      3.0                  h0e60522_0    
libblas                   3.9.0                     8_mkl    
libbrotlicommon           1.0.9                h8ffe710_6    
libbrotlidec              1.0.9                h8ffe710_6    
libbrotlienc              1.0.9                h8ffe710_6    
libcblas                  3.9.0                     8_mkl    
libclang                  11.1.0          default_h5c34c98_1    
libcurl                   7.79.1               h789b8ee_1    
libdeflate                1.8                  h8ffe710_0    
liblapack                 3.9.0                     8_mkl    
libpng                    1.6.37               ha81a0f5_2    
libprotobuf               3.18.1               h7755175_0    
libsodium                 1.0.18               h8d14728_1    
libssh2                   1.10.0               h680486a_0    
libthrift                 0.15.0               h636ae23_1    
libtiff                   4.3.0                hd413186_2    
libutf8proc               2.6.1                hcb41399_0    
libzlib                   1.2.11            h8ffe710_1013    
line_profiler             3.3.0            py39h54b28b2_0    
lz4-c                     1.9.3                h8ffe710_0    
m2w64-gcc-libgfortran     5.3.0                         6    
m2w64-gcc-libs            5.3.0                         7    
m2w64-gcc-libs-core       5.3.0                         7    
m2w64-gmp                 6.1.0                         2    
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    
markupsafe                1.1.1            py39hb82d6ee_3    
matplotlib                3.4.3            py39hcbf5309_0    
matplotlib-base           3.4.3            py39h581301d_0    
mccabe                    0.6.1                      py_1    
mistune                   0.8.4           py39hb82d6ee_1003    
mkl                       2020.4             hb70f87d_311    
mock                      4.0.3            py39hcbf5309_1    
more-itertools            8.7.0              pyhd8ed1ab_0    
msys2-conda-epoch         20160418                      1    
mypy_extensions           0.4.3            py39hcbf5309_3    
nbclient                  0.5.3              pyhd8ed1ab_0    
nbconvert                 6.0.7            py39hcbf5309_3    
nbformat                  5.1.2              pyhd8ed1ab_1    
nest-asyncio              1.4.3              pyhd8ed1ab_0    
notebook                  6.2.0            py39hcbf5309_0    
numexpr                   2.7.2            py39h2e25243_0    
numpy                     1.20.1           py39h6635163_0    
olefile                   0.46               pyh9f0ad1d_1    
openjpeg                  2.4.0                hb211442_1    
openssl                   1.1.1l               h8ffe710_0    
packaging                 20.9               pyh44b312d_0    
pandas                    1.3.1            py39h2e25243_0    
pandoc                    2.11.4               h8ffe710_0    
pandocfilters             1.4.2                      py_1    
parquet-cpp               1.5.1                         2    
parso                     0.8.1              pyhd8ed1ab_0    
pathspec                  0.8.1              pyhd3deb0d_0    
pickleshare               0.7.5                   py_1003    
pillow                    8.2.0            py39h1a9d4f7_1    
pip                       21.0.1             pyhd8ed1ab_0    
pluggy                    0.13.1           py39hcbf5309_4    
proj                      7.2.0                h1cfcee9_2    
prometheus_client         0.9.0              pyhd3deb0d_0    
prompt-toolkit            3.0.16             pyha770c72_0    
py                        1.10.0             pyhd3deb0d_0    
pyarrow                   5.0.0           py39ha05331a_10_cpu    
pycparser                 2.20               pyh9f0ad1d_2    
pydata-sphinx-theme       0.5.1              pyhd8ed1ab_0    
pygments                  2.8.0              pyhd8ed1ab_0    
pylint                    2.6.0            py39hcbf5309_1    
pynmea2                   1.16.0             pyh44b312d_0    
pyopenssl                 20.0.1             pyhd8ed1ab_0    
pyparsing                 2.4.7              pyh9f0ad1d_0    
pyqt                      5.12.3           py39hcbf5309_7    
pyqt-impl                 5.12.3           py39h415ef7b_7    
pyqt5-sip                 4.19.18          py39h415ef7b_7    
pyqtchart                 5.12             py39h415ef7b_7    
pyqtwebengine             5.12.1           py39h415ef7b_7    
pyrsistent                0.17.3           py39hb82d6ee_2    
pyshp                     2.1.3              pyh44b312d_0    
pysocks                   1.7.1            py39hcbf5309_3    
pytables                  3.6.1            py39hbcfe41f_4    
pytest                    6.2.2            py39hcbf5309_0    
pytest-cov                2.11.1             pyh44b312d_0    
pytest-repeat             0.8.0                      py_0    
python                    3.9.2           h7840368_0_cpython    
python-dateutil           2.8.1                      py_0    
python_abi                3.9                      1_cp39    
pytz                      2021.1             pyhd8ed1ab_0    
pywin32                   300              py39hb82d6ee_0    
pywinpty                  0.5.7            py39hde42818_1    
pyyaml                    5.4.1            py39hb82d6ee_0    
pyzmq                     22.0.3           py39he46f08e_1    
qt                        5.12.9               h5909a2a_4    
re2                       2021.09.01           h0e60522_0    
regex                     2020.11.13       py39hb82d6ee_1    
requests                  2.25.1             pyhd3deb0d_0    
scipy                     1.7.1            py39hc0c34ad_0    
send2trash                1.5.0                      py_0    
setuptools                49.6.0           py39hcbf5309_3    
shapely                   1.7.1            py39haadaec5_5    
six                       1.15.0             pyh9f0ad1d_0    
snappy                    1.1.8                ha925a31_3    
snowballstemmer           2.1.0              pyhd8ed1ab_0    
soupsieve                 2.0.1                      py_1    
sphinx                    3.5.3              pyhd8ed1ab_0    
sphinx-argparse           0.2.5              pyhd8ed1ab_1    
sphinxcontrib-applehelp   1.0.2                      py_0    
sphinxcontrib-devhelp     1.0.2                      py_0    
sphinxcontrib-htmlhelp    1.0.3                      py_0    
sphinxcontrib-jsmath      1.0.1                      py_0    
sphinxcontrib-qthelp      1.0.3                      py_0    
sphinxcontrib-serializinghtml 1.1.4                      py_0    
sqlite                    3.34.0               h8ffe710_0    
terminado                 0.9.2            py39hcbf5309_0    
testpath                  0.4.4                      py_0    
tk                        8.6.10               h8ffe710_1    
toml                      0.10.2             pyhd8ed1ab_0    
tornado                   6.1              py39hb82d6ee_1    
tqdm                      4.58.0             pyhd8ed1ab_0    
traitlets                 5.0.5                      py_0    
typed-ast                 1.4.2            py39hb82d6ee_0    
typing_extensions         3.7.4.3                    py_0    
tzdata                    2021a                he74cb21_0    
ucrt                      10.0.20348.0         h57928b3_0    
urllib3                   1.26.4             pyhd8ed1ab_0    
vc                        14.2                 hb210afc_3    
versioneer                0.20               pyhd8ed1ab_0    
vs2015_runtime            14.29.30037          h902a5da_5    
wcwidth                   0.2.5              pyh9f0ad1d_2    
webencodings              0.5.1                      py_1    
wheel                     0.36.2             pyhd3deb0d_0    
widgetsnbextension        3.5.1            py39hcbf5309_4    
win_inet_pton             1.1.0            py39hcbf5309_2    
wincertstore              0.2             py39hcbf5309_1006    
winpty                    0.4.3                         4    
wrapt                     1.12.1           py39hb82d6ee_3    
xz                        5.2.5                h62dcd97_1    
yaml                      0.2.5                he774522_0    
yamllint                  1.26.3             pyhd8ed1ab_0    
zeromq                    4.3.4                h0e60522_0    
zipp                      3.4.0                      py_0    
zlib                      1.2.11            h62dcd97_1010    
zstd                      1.5.0                h6255e5f_0

Details about conda and system ( conda info ):

$ conda info
     active environment : py106_39
    active env location : C:\apps\mambaforge\envs\py_39
            shell level : 2
       user config file : C:\Users\<username>\.condarc
 populated config files : C:\Users\<username>\.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.8.final.0
       virtual packages : __win=0=0
                          __archspec=1=x86_64
       base environment : C:\apps\mambaforge  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/
          package cache : C:\apps\mambaforge\pkgs
                          C:\Users\<username>\.conda\pkgs
                          C:\Users\<username>\AppData\Local\conda\conda\pkgs
       envs directories : C:\apps\mambaforge\envs
                          C:\Users\<username>\.conda\envs
                          C:\Users\<username>\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.8 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False

How to create conda env with CUDA enabled arrow-cpp?

With the current state, creating a conda environment using

conda create -c conda-forge -n test-arrow-cpp arrow-cpp "arrow-cpp-proc=*=cuda"

leads to

The following NEW packages will be INSTALLED:
  <snip>
  arrow-cpp          conda-forge/linux-64::arrow-cpp-0.17.0-py38h9d6855f_0
  arrow-cpp-proc     conda-forge/linux-64::arrow-cpp-proc-1.0.0-cuda
  <snip>

which is wrong as arrow-cpp-0.17.0-py38h9d6855f_0 does not have CUDA enabled.

Also

conda create -c conda-forge -n test-arrow-cpp "arrow-cpp=0.16" "arrow-cpp-proc=*=cuda"

will lead to wrong packages:

  arrow-cpp          conda-forge/linux-64::arrow-cpp-0.16.0-py38h9d6855f_4
  arrow-cpp-proc     conda-forge/linux-64::arrow-cpp-proc-1.0.0-cuda

However

conda create -c conda-forge -n test-arrow-cpp "arrow-cpp=*=*cuda"

gives correct result:

  arrow-cpp          conda-forge/linux-64::arrow-cpp-0.16.0-py38h25a4a40_5_cuda

The question of this issue is why the usage of arrow-cpp-proc=*=cuda is ineffective?

I guess the answer lies in the fact that arrow-cpp-0.16*_4 and arrow-cpp-0.17*_0 use old meta.yaml that is not arrow-cpp-proc aware.

As a fix, arrow-cpp-feedstock should start using the new arrow-cpp-proc-aware meta.yaml but it seems that the issue will continue to persist until arrow-cpp-0.18 will be packaged with the new meta.yaml only. What do you think?

CC: @jakirkham @xhochy @isuruf

Make arrow-cpp not depend on python anymore

This should now be possible, see comment upstream. This would also collapse the builds into one per aarch (meaning we have to either exempt the aarch/ppc builds from this, or figure out cross-compiling for CUDA).

conda-forge / arrow-cpp-feedstock Goto Github PK

arrow-cpp-feedstock's Introduction

About arrow-cpp-feedstock

About arrow-cpp

About apache-arrow-proc

About libarrow

About libarrow-acero

About libarrow-flight

About libarrow-gandiva

About libparquet

About libarrow-dataset

About libarrow-flight-sql

About libarrow-substrait

Current build status

Current release info

Installing arrow-cpp

About conda-forge

Terminology

Updating arrow-cpp-feedstock

Feedstock Maintainers

arrow-cpp-feedstock's People

Contributors

Stargazers

Watchers

Forkers

arrow-cpp-feedstock's Issues

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

Comment:

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

Comment:

Comment:

Comment:

Recommend Projects

Recommend Topics

Recommend Org