[Failing Test]: Beam ML unit test suite fails on Python 3.11 about beam HOT 5 CLOSED

Actually, "tensorflow_transform" alone is also not installable.

(py311c) :python$ pip install "tensorflow_transform"
Collecting tensorflow_transform
Using cached tensorflow_transform-1.15.0-py3-none-any.whl.metadata (13 kB)
Collecting absl-py<2.0.0,>=0.9 (from tensorflow_transform)
Using cached absl_py-1.4.0-py3-none-any.whl.metadata (2.3 kB)
Collecting numpy>=1.22.0 (from tensorflow_transform)
Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting pyarrow<11,>=10 (from tensorflow_transform)
Using cached pyarrow-10.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting pydot<2,>=1.2 (from tensorflow_transform)
Using cached pydot-1.4.2-py2.py3-none-any.whl.metadata (8.0 kB)
Collecting tensorflow<2.16,>=2.15 (from tensorflow_transform)
Using cached tensorflow-2.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting tensorflow-metadata<1.16.0,>=1.15.0 (from tensorflow_transform)
Using cached tensorflow_metadata-1.15.0-py3-none-any.whl.metadata (2.4 kB)
INFO: pip is looking at multiple versions of tensorflow-transform to determine which version is compatible with other requirements. This could take a while.
Collecting tensorflow_transform
Using cached tensorflow_transform-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting apache-beam<3,>=2.47 (from apache-beam[gcp]<3,>=2.47->tensorflow_transform)
Using cached apache_beam-2.56.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting protobuf<5,>=3.20.3 (from tensorflow_transform)
Using cached protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting tensorflow<3,>=2.13 (from tensorflow_transform)
Using cached tensorflow-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.3 kB)
Collecting tensorflow-metadata<1.15.0,>=1.14.0 (from tensorflow_transform)
Using cached tensorflow_metadata-1.14.0-py3-none-any.whl.metadata (2.1 kB)
Collecting tensorflow_transform
Using cached tensorflow_transform-1.13.0-py3-none-any.whl.metadata (12 kB)
Collecting numpy~=1.22.0 (from tensorflow_transform)
Using cached numpy-1.22.4.zip (11.5 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting pyarrow<7,>=6 (from tensorflow_transform)
Using cached pyarrow-6.0.1.tar.gz (770 kB)
Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Ignoring numpy: markers 'python_version < "3.8"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8"' don't match your environment
Ignoring numpy: markers 'python_version == "3.9"' don't match your environment
Collecting cython>=0.29
Using cached Cython-3.0.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement numpy==1.21.3 (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.13.3, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5, 1.20.0, 1.20.1, 1.20.2, 1.20.3, 1.21.0, 1.21.1, 1.22.0, 1.22.1, 1.22.2, 1.22.3, 1.22.4, 1.23.0rc1, 1.23.0rc2, 1.23.0rc3, 1.23.0, 1.23.1, 1.23.2, 1.23.3, 1.23.4, 1.23.5, 1.24.0rc1, 1.24.0rc2, 1.24.0, 1.24.1, 1.24.2, 1.24.3, 1.24.4, 1.25.0rc1, 1.25.0, 1.25.1, 1.25.2, 1.26.0b1, 1.26.0rc1, 1.26.0, 1.26.1, 1.26.2, 1.26.3, 1.26.4, 2.0.0b1, 2.0.0rc1, 2.0.0rc2)
ERROR: No matching distribution found for numpy==1.21.3
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

from beam.

According to https://github.com/tensorflow/transform/releases/tag/v1.15.0, Py3.11 support was just added in the release that happened 3 weeks ago.

from beam.

Opened tensorflow/transform#313

from beam.

new error:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

checkpoint_file = '/home/runner/.cache/huggingface/hub/models--sentence-transformers--all-mpnet-base-v2/snapshots/84f2bcc00d77236f9e89c8a360a00fb1139bf47d/model.safetensors'
is_quantized = False

    def load_state_dict(checkpoint_file: Union[str, os.PathLike], is_quantized: bool = False):
        """
        Reads a PyTorch checkpoint file, returning properly formatted errors if they arise.
        """
        if checkpoint_file.endswith(".safetensors") and is_safetensors_available():
            # Check format of the archive
>           with safe_open(checkpoint_file, framework="pt") as f:
E           FileNotFoundError: No such file or directory: "/home/runner/.cache/huggingface/hub/models--sentence-transformers--all-mpnet-base-v2/snapshots/84f2bcc00d77236f9e89c8a360a00fb1139bf47d/model.safetensors"

from beam.

possibly a race between test scenarios - looks like using no_xdist has helped.

from beam.