google / etils Goto Github PK

View Code? Open in Web Editor NEW

171.0 5.0 19.0 1.37 MB

Collection of eclectic utils for python.

Home Page: https://etils.readthedocs.io

License: Apache License 2.0

Python 55.75% JavaScript 0.78% CSS 0.43% Jupyter Notebook 43.05%

python numpy tensorflow jax utils

etils's People

Contributors

Stargazers

Watchers

Forkers

tanaysd isabella232 mbrukman daisylove666 ohadrubin ethanabrooks a-why-not-fork-repositories-good-luck davidjnevin zhuyijie slowy07 truncs daniel-character hmeyer chicham abaelhe thecodeofmontecristo lingjiekong ojh6404 sibgatulin

etils's Issues

[Enhancement] Add `optree` integration to `etils.etree`

optree is a standalone package (like dm-tree) aimed to high-performance PyTree manipulation (like jax.tree_util). It offers similar APIs to jax.tree_util but better.

Some initial benchmark results:

Average Time Cost (↓)	OpTree (v0.9.0)	JAX XLA (v0.4.6)	PyTorch (v2.0.0)	TensorFlow Nest (v2.12.0)	DM-Tree (v0.1.8)
Tree Flatten	x1.00	2.33	22.05	1.38	1.12
Tree UnFlatten	x1.00	2.69	4.28	13.69	16.23
Tree Flatten with Path	x1.00	16.16	Not Supported	21.10	27.59
Tree Copy	x1.00	2.56	9.97	9.62	11.02
Tree Map	x1.00	2.56	9.58	9.16	10.62
Tree Map (nargs)	x1.00	2.89	Not Supported	74.26	31.33
Tree Map with Path	x1.00	7.23	Not Supported	40.78	19.66
Tree Map with Path (nargs)	x1.00	6.56	Not Supported	69.63	29.61

We have already seen some etils folks get involved with optree and jax.tree_util discussions. I wonder if etils maintainers have interest to add optree to etils.etree.

Ref:

v1.5.2: Test fails for python3.11

I am on Arch and while installing the package, I see the following test failures

========================================================================================================================================================== FAILURES ==========================================================================================================================================================
__________________________________________________________________________________________________________________________________________________ test_interp_scalar[jnp] ___________________________________________________________________________________________________________________________________________________

xnp = <module 'jax.numpy' from '/usr/lib/python3.11/site-packages/jax/numpy/__init__.py'>

    @enp.testing.parametrize_xnp()
    def test_interp_scalar(xnp: enp.NpModule):
      vals = xnp.asarray(
          [
              [-1, -1],
              [-1, 0],
              [-1, 1],
              [0.5, 1],
              [1, 1],
          ]
      )
    
      #
    
      out = enp.interp(vals, from_=(-1, 1), to=(0, 256))
      assert enp.compat.is_array_xnp(out, xnp)
    
      np.testing.assert_allclose(
          out,
          xnp.asarray([
              [0, 0],
              [0, 128],
              [0, 256],
              [192, 256],
              [256, 256],
          ]),
      )
      np.testing.assert_allclose(
          enp.interp(vals, from_=(-1, 1), to=(0, 1)),
          xnp.asarray([
              [0, 0],
              [0, 0.5],
              [0, 1],
              [0.75, 1],
              [1, 1],
          ]),
      )
    
      vals = xnp.asarray(
          [
              [255, 255, 0],
              [255, 128, 0],
              [255, 0, 128],
          ]
      )
>     np.testing.assert_allclose(
          enp.interp(vals, from_=(0, 255), to=(0, 1)),
          xnp.asarray([
              [1, 1, 0],
              [1, 128 / 255, 0],
              [1, 0, 128 / 255],
          ]),
      )

etils/enp/interp_utils_test.py:70: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<function assert_allclose.<locals>.compare at 0x7fe9a8bc6840>, array([[0.99999994, 0.99999994, 0.        ],
       [0...     , 0.       ],
       [1.       , 0.5019608, 0.       ],
       [1.       , 0.       , 0.5019608]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Not equal to tolerance rtol=1e-07, atol=0
E           
E           Mismatched elements: 2 / 9 (22.2%)
E           Max absolute difference: 5.9604645e-08
E           Max relative difference: 1.1874362e-07
E            x: array([[1.      , 1.      , 0.      ],
E                  [1.      , 0.501961, 0.      ],
E                  [1.      , 0.      , 0.501961]], dtype=float32)
E            y: array([[1.      , 1.      , 0.      ],
E                  [1.      , 0.501961, 0.      ],
E                  [1.      , 0.      , 0.501961]], dtype=float32)

/usr/lib/python3.11/contextlib.py:81: AssertionError

Tests fail on Python 3.10

There are two test failures with Python 3.10. With Python 3.9 everything seems fine. Could you have a look?

================================================================================== FAILURES ==================================================================================
_________________________________________________________________________________ test_repr __________________________________________________________________________________

    def test_repr():
>     assert repr(R(123, R11(y='abc'))) == epy.dedent("""
      R(
          x=123,
          y=R11(
              x=None,
              y='abc',
              z=None,
          ),
      )
      """)
E     assert "R(x=123, y=R...bc', z=None))" == 'R(\n    x=12...e,\n    ),\n)'
E       + R(x=123, y=R11(x=None, y='abc', z=None))
E       - R(
E       -     x=123,
E       -     y=R11(
E       -         x=None,
E       -         y='abc',
E       -         z=None,...
E
E       ...Full output truncated (3 lines hidden), use '-vv' to show

etils/edc/dataclass_utils_test.py:108: AssertionError
_____________________________________________________________________________ test_resource_path _____________________________________________________________________________

    def test_resource_path():
      path = epath.resource_utils.ResourcePath(_make_zip_file())
      assert isinstance(path, os.PathLike)
      assert path.joinpath('b/c.txt').read_text() == 'content of c'
      sub_dirs = list(path.joinpath('b').iterdir())
      assert len(sub_dirs) == 3
      for p in sub_dirs:  # Childs should be `ResourcePath` instances
        assert isinstance(p, epath.resource_utils.ResourcePath)

      # Forwarded to `Path` keep the resource.
      path = epath.Path(path)
      assert isinstance(path, epath.resource_utils.ResourcePath)

>     assert path.joinpath() == path
E     AssertionError: assert ResourcePath('alpharep.zip', '') == ResourcePath('alpharep.zip', '')
E      +  where ResourcePath('alpharep.zip', '') = <bound method Path.joinpath of ResourcePath('alpharep.zip', '')>()
E      +    where <bound method Path.joinpath of ResourcePath('alpharep.zip', '')> = ResourcePath('alpharep.zip', '').joinpath

For test_repr, apparently custom __repr__ is not applied as __qualname__ is changed in Python 3.10.

For test_resource_path, joinpath() returns a new object for Python >= 3.10 as that function is not overridden.

I noticed those failures when I'm creating a unofficial package python-etils for Arch Linux as a new dependency for the latest python-tensorflow-datasets.

Environment: Arch Linux x86_64, Python 3.10.4

ecolab incompatible with current protobuf version on google colab

the google colab demo referenced in the readme of ecolab fails to run with errror:

AttributeError: module 'google.protobuf.internal.api_implementation' has no attribute '_c_module'

I tested installing other versions of protobuf with no success :
protobuf-5.26.1
4.25.3
4.24.4
3.20.3

Using etils's `epath` with AWS S3 links

Hey,

Apparently, one can't use S3 objects with etils:

from etils import epath
a = epath.Path( 's3://....' ) #link points to a folder of .tfrecords
sorted(epath.Path('s3://..').iterdir())

Error

Yields,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/etils/epath/gpath.py", line 126, in iterdir
    for f in self._backend.listdir(self._path_str):
  File "/usr/local/lib/python3.8/dist-packages/etils/epath/backend.py", line 191, in listdir
    return self.gfile.listdir(path)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 766, in list_directory_v2
    raise errors.NotFoundError(
tensorflow.python.framework.errors_impl.NotFoundError: Could not find directory s3://..

This utilized in TFDS simply to compute some meta-data for the datasets I'd be working in.

I can confirm the given link exists when copypasted to the aws s3 ls s3://.... command. It seems an issue with etils, or I'm using it wrongly.

Cheers!

Freezing unfrozen dataclasses requires unfreezing them beforehand.

import dataclasses
from etils import edc
@edc.dataclass(allow_unfrozen=True)
@dataclasses.dataclass(eq=True, kw_only=True)
class Conf:
    x: int = 3

a = Conf()
a.x = 2 # verify that a is not frozen
a.frozen() # complains that .frozen() can only be called after .unfrozen()

As the code above shows, allow_unfrozen seems to assume that the dataclass is frozen by default. I think that it should verify the assumption.

Package broken under Python 3.9

The epath sub-package is broken under Python 3.9, apparently due to the recent introduction of syntax only supported in Python 3.10 or later in binary_import.py:

import etils.epath

...

File ~/condahome/miniconda3/envs/py39t/lib/python3.9/site-packages/etils/epy/binary_import.py:55
     49       return True
     50   return False
     53 @contextlib.contextmanager
     54 def binary_adhoc(
---> 55     restrict: None | py_utils.StrOrStrList = None,
     56     verbose: bool = False,
     57     **kwargs: Any,
     58 ) -> Iterator[None]:
     59   yield

TypeError: unsupported operand type(s) for |: 'NoneType' and '_UnionGenericAlias'

`epath.Path("s3://...")` regression in python3.12?

Hi there. I am not at all sure if it is not a misunderstanding or a misconfiguration on my end, but I am seeing different behaviour of epath.Path with s3 (at least) on python 3.11 and 3.12. Please, consider the attached screenshot for details.

Both of the environments are minimal with the following package versions:

Package             Version
------------------- -----------
aiobotocore         2.13.0
aiohttp             3.9.5
aioitertools        0.11.0
aiosignal           1.3.1
attrs               23.2.0
boto3               1.34.106
botocore            1.34.106
etils               1.8.0
frozenlist          1.4.1
fsspec              2024.5.0
idna                3.7
importlib_resources 6.4.0
jmespath            1.0.1
multidict           6.0.5
pip                 24.0
python-dateutil     2.9.0.post0
s3fs                2024.5.0
s3transfer          0.10.1
six                 1.16.0
typing_extensions   4.11.0
urllib3             2.2.1
wrapt               1.16.0
yarl                1.9.4

(with the addition of setuptools==69.2.0 and wheel==0.43.0 in test311).

Infinite recursion on `is_relative_to` in `abstract_path.py`

Hi,

I'm running into infinite recursion using with etils as a part of tensorflow_datasets (in turn as a part of algoperf). It looks like calling is_relative_to on a PosixGPath calls relative_to in pathlib, which then (maybe) calls an overloaded version that sends it back to etils's is_relative_to?

It's likely that this is my fault, since it's so deep inside tensorflow_datasets, but I can't see what's concretely going wrong or how to fix it. I'm using etils version 1.8.0 with Python 3.12 on macOS 14.4.1 (tensorflow_datasets and algoperf are versions 4.9.4 and 0.1.5, respectively).

Here's the stack trace and some info from pdb:

RecursionError: maximum recursion depth exceeded
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pathlib.py(706)parts()
-> if self.drive or self.root:
(Pdb) up 9999
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pdb.py(1944)main()
-> pdb._run(target)
(Pdb) down
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pdb.py(1738)_run()
-> self.run(target.code)
(Pdb) 
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/bdb.py(600)run()
-> exec(cmd, globals, locals)
(Pdb) 
> <string>(1)<module>()->None
(Pdb) 
> /nix/store/lm2mxb7f1y9mhp85k6h8pw4r82k6mclm-h3xkz95yrwshm2hh99wzr1zqjhp7d089-source/submission_runner.py(714)<module>()->None
-> app.run(main)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/absl/app.py(308)run()
-> _run_main(main, args)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/absl/app.py(254)_run_main()
-> sys.exit(main(argv))
(Pdb) 
> /nix/store/lm2mxb7f1y9mhp85k6h8pw4r82k6mclm-h3xkz95yrwshm2hh99wzr1zqjhp7d089-source/submission_runner.py(682)main()
-> score = score_submission_on_workload(
(Pdb) 
> /nix/store/lm2mxb7f1y9mhp85k6h8pw4r82k6mclm-h3xkz95yrwshm2hh99wzr1zqjhp7d089-source/submission_runner.py(587)score_submission_on_workload()
-> timing, metrics = train_once(workload, workload_name,
(Pdb) 
> /nix/store/lm2mxb7f1y9mhp85k6h8pw4r82k6mclm-h3xkz95yrwshm2hh99wzr1zqjhp7d089-source/submission_runner.py(221)train_once()
-> input_queue = workload._build_input_queue(
(Pdb) 
> /nix/store/lm2mxb7f1y9mhp85k6h8pw4r82k6mclm-h3xkz95yrwshm2hh99wzr1zqjhp7d089-source/algorithmic_efficiency/workloads/mnist/workload.py(155)_build_input_queue()
-> ds = _build_mnist_dataset(
(Pdb) 
> /nix/store/lm2mxb7f1y9mhp85k6h8pw4r82k6mclm-h3xkz95yrwshm2hh99wzr1zqjhp7d089-source/algorithmic_efficiency/workloads/mnist/workload.py(44)_build_mnist_dataset()
-> ds = tfds.load(
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/logging/__init__.py(168)__call__()
-> return function(*args, **kwargs)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/load.py(649)load()
-> _download_and_prepare_builder(dbuilder, download, download_and_prepare_kwargs)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/load.py(508)_download_and_prepare_builder()
-> dbuilder.download_and_prepare(**download_and_prepare_kwargs)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/logging/__init__.py(168)__call__()
-> return function(*args, **kwargs)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/dataset_builder.py(691)download_and_prepare()
-> self._download_and_prepare(
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/dataset_builder.py(1547)_download_and_prepare()
-> split_generators = self._split_generators(  # pylint: disable=unexpected-keyword-arg
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/image_classification/mnist.py(119)_split_generators()
-> mnist_files = dl_manager.download_and_extract(
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/download/download_manager.py(688)download_and_extract()
-> return _map_promise(self._download_extract, url_or_urls)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/download/download_manager.py(831)_map_promise()
-> res = tree_utils.map_structure(
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tree/__init__.py(428)map_structure()
-> [func(*args) for args in zip(*map(flatten, structures))])
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/download/download_manager.py(832)<lambda>()
-> lambda p: p.get(), all_promises
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/promise/promise.py(512)get()
-> return self._target_settled_value(_raise=True)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/promise/promise.py(516)_target_settled_value()
-> return self._target()._settled_value(_raise)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/promise/promise.py(226)_settled_value()
-> reraise(type(raise_val), raise_val, self._traceback)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/six.py(719)reraise()
-> raise value
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/promise/promise.py(87)try_catch()
-> return (handler(*args, **kwargs), None)
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/download/download_manager.py(408)<lambda>()
-> lambda dl_result: self._register_or_validate_checksums(  # pylint: disable=g-long-lambda
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/download/download_manager.py(473)_register_or_validate_checksums()
-> return self._rename_and_get_final_dl_path(
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/tensorflow_datasets/core/download/download_manager.py(497)_rename_and_get_final_dl_path()
-> if self._manual_dir and path.is_relative_to(self._manual_dir):
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/etils/epath/abstract_path.py(78)is_relative_to()
-> self.relative_to(*other)
(Pdb) 
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pathlib.py(679)relative_to()
-> if self.is_relative_to(path):
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/etils/epath/abstract_path.py(78)is_relative_to()
-> self.relative_to(*other)
(Pdb) 
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pathlib.py(679)relative_to()
-> if self.is_relative_to(path):
(Pdb) 
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/etils/epath/abstract_path.py(78)is_relative_to()
-> self.relative_to(*other)
(Pdb) 
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pathlib.py(679)relative_to()
-> if self.is_relative_to(path):
(Pdb) pp self
PosixGPath('data/downloads/cvdf-datasets_mnist_t10k-images-idx3-ubytedDnaEPiC58ZczHNOp6ks9L4_JLids_rpvUj38kJNGMc.gz.tmp.10c0dd19afe942519ef7adcfbda3595e/t10k-images-idx3-ubyte.gz')
(Pdb) pp other
PosixGPath('data/downloads/manual')
(Pdb) down
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/etils/epath/abstract_path.py(78)is_relative_to()
-> self.relative_to(*other)
(Pdb) pp self
PosixGPath('data/downloads/cvdf-datasets_mnist_t10k-images-idx3-ubytedDnaEPiC58ZczHNOp6ks9L4_JLids_rpvUj38kJNGMc.gz.tmp.10c0dd19afe942519ef7adcfbda3595e/t10k-images-idx3-ubyte.gz')
(Pdb) pp other
(PosixGPath('data/downloads/manual'),)
(Pdb) down
> /nix/store/c7pshh939gfv6v1jjnbiai503385c8hl-python3-3.12.3/lib/python3.12/pathlib.py(679)relative_to()
-> if self.is_relative_to(path):
(Pdb) pp self
PosixGPath('data/downloads/cvdf-datasets_mnist_t10k-images-idx3-ubytedDnaEPiC58ZczHNOp6ks9L4_JLids_rpvUj38kJNGMc.gz.tmp.10c0dd19afe942519ef7adcfbda3595e/t10k-images-idx3-ubyte.gz')
(Pdb) pp path
PosixGPath('data/downloads/manual')
(Pdb) ll
663  	   def relative_to(self, other, /, *_deprecated, walk_up=False):
664  	       """Return the relative path to another path identified by the passed
665  	       arguments.  If the operation is not possible (because this is not
666  	       related to the other path), raise ValueError.
667  	
668  	       The *walk_up* parameter controls whether `..` may be used to resolve
669  	       the path.
670  	       """
671  	       if _deprecated:
672  	           msg = ("support for supplying more than one positional argument "
673  	                  "to pathlib.PurePath.relative_to() is deprecated and "
674  	                  "scheduled for removal in Python {remove}")
675  	           warnings._deprecated("pathlib.PurePath.relative_to(*args)", msg,
676  	                                remove=(3, 14))
677  	       other = self.with_segments(other, *_deprecated)
678  	       for step, path in enumerate([other] + list(other.parents)):
679  ->	           if self.is_relative_to(path):
680  	               break
681  	           elif not walk_up:
682  	               raise ValueError(f"{str(self)!r} is not in the subpath of {str(other)!r}")
683  	           elif path.name == '..':
684  	               raise ValueError(f"'..' segment in {str(other)!r} cannot be walked")
685  	       else:
686  	           raise ValueError(f"{str(self)!r} and {str(other)!r} have different anchors")
687  	       parts = ['..'] * step + self._tail[len(path._tail):]
688  	       return self.with_segments(*parts)
(Pdb) down
> /nix/store/jbjiw9c7hjr2zfnjc55y4rcg3pww01qa-python3-3.12.3-env/lib/python3.12/site-packages/etils/epath/abstract_path.py(78)is_relative_to()
-> self.relative_to(*other)
(Pdb) ll
 75  	 def is_relative_to(self, *other: PathLike) -> bool:
 76  	   """Return True if the path is relative to another path or False."""
 77  	   try:
 78  >>	     self.relative_to(*other)
 79  	     return True
 80  ->	   except ValueError:
 81  	     return False

I'd appreciate anything you could do to help!

pathlib.Path.rglob is recursive but etils.epath.Path.rglob isn't

Was trying to add gs://... support to a TFDS builder that was working with local files, and figured I'd try to simply replace pathlib with etils.epath since it was already in tfds.core.Path and would involve minimal changes.

Unfortunately got tripped up on

AssertionError: Not a single example present in the PCollection! [while running 'test_write/GetBoundaries']

which seems to be caused by rglob not actually being recursive, as per: https://github.com/google/etils/blob/main/etils/epath/abstract_path.py#L111

Mentioning this here just in case anyone else got confused by the same thing.

Example

In [1]: import pathlib

In [2]: import etils.epath

In [3]: list(etils.epath.Path('.').rglob("*.py"))
Out[3]: [PosixGPath('builders/__init__.py')]

In [4]: list(pathlib.Path('.').rglob("*.py"))
Out[4]: 
[PosixPath('builders/__init__.py'),
 PosixPath('builders/my_dataset/my_dataset_test.py'),
 PosixPath('builders/my_dataset/my_dataset.py'),
 PosixPath('builders/my_dataset/__init__.py')]

edc: unfrozen dataclasses are hashble

I think the general consensus in Python is that mutable types should not be hashable; however unfrozen dataclasses are hashable. See code below:

import dataclasses
from etils import edc
@edc.dataclass(allow_unfrozen=True)
@dataclasses.dataclass(eq=True, kw_only=True, frozen=True)
class Conf:
    x: int = 3
a = Conf()
a = a.unfrozen()
a.x = 2
hash(a) # should fail

ps. edc is really handy

broken link in ecolab/docs/demo.ipynb

In ecolab/docs/demo.ipynb replace

View all available imports in the code: https://github.com/google/etils/tree/main/etils/ecolab/lazy_imports.py;l=412

with

View all available imports in the code: https://github.com/google/etils/tree/main/etils/ecolab/lazy_imports.py#l=412

etils.epath.Path.glob() removes gs:// prefix

I am trying to iterate over a "directory" in a bucket using glob(). Unfortunately the returned object Path objects are missing the gs:// part:

dir = Path(
        "gs://gcp-public-data-arco-era5/ar/1959-2022-1h-240x121_equiangular_with_poles_conservative.zarr/lake_depth"
    )
files = list(dir.glob("*"))
files[0].read_bytes() # fails with FileNotFoundError

Is this intended behavior?

When using edc and frozen/unfrozen pylint and mypy complain

Hello,

Thanks for the cool library. When using the edc module and un particular frozen/unfrozen function both mypy and pylint complain about types.

This is fairly easy to reproduce, just by using the following snippet:

from dataclasses import dataclass
from etils import edc

@edc.dataclass(allow_unfrozen=True)
@dataclass(frozen=True)
class MyConf:
	my_var: str = "epic"

if __name__ == "__main__":
    qqq = MyConf(my_var="ff")
    
    # pylint and mypy complain!
    fff = qqq.unfrozen()
    ppp = fff.frozen()

Specifically pylint has the following output:

E1101: Instance of 'MyConf' has no 'unfrozen' member (no-member)

whereas mypy gives:

"MyConf" has no attribute "unfrozen"  [attr-defined]

I suspect both are for the same reason... is there a clean way (without ignoring these) to use this module while making both mypy and pylint happy?

For what is worth - I am using Ubuntu 22.04, python 3.10, pylint 2.14.5, and mypy 0.971.

pip installing etils spews warnings

I've tried doing a pip3 install etils[all] and then running a subsequent command that installs jax and by trans deps etils, and I get this error polluting the terminal:

WARNING: etils 0.2.0 does not provide the extra 'edc'
WARNING: etils 0.3.3 does not provide the extra 'edc'
WARNING: etils 0.3.2 does not provide the extra 'edc'
WARNING: etils 0.3.1 does not provide the extra 'edc'
WARNING: etils 0.3.0 does not provide the extra 'edc'
WARNING: etils 0.2.0 does not provide the extra 'edc'
WARNING: etils 0.3.3 does not provide the extra 'edc'
WARNING: etils 0.3.2 does not provide the extra 'edc'
WARNING: etils 0.3.1 does not provide the extra 'edc'
WARNING: etils 0.3.0 does not provide the extra 'edc'
WARNING: etils 0.2.0 does not provide the extra 'edc'
WARNING: etils 0.3.3 does not provide the extra 'edc'
WARNING: etils 0.3.2 does not provide the extra 'edc'
WARNING: etils 0.3.1 does not provide the extra 'edc'
WARNING: etils 0.3.0 does not provide the extra 'edc'
WARNING: etils 0.2.0 does not provide the extra 'edc'
WARNING: etils 0.3.3 does not provide the extra 'edc'

Can't you just integrate your code into jax or something? this lib has ELEVEN different sub-targets. you are creating a dependency game that really really nobody wants. at the very least please consider a way to silence this error, even if it means minimizing your package structure and getting your code into upstream.

epath behaves weird using fsspec on gcs

Via GCloud Console I clicked "CREATE FOLDER" to create empty_folder.
Then I did:

$ touch /tmp/test.txt
$ gsutil cp /tmp/test.txt gs://henning-test/folder/test.txt

$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from etils import epath
>>> epath.gpath._is_tf_installed()
False
>>> e = epath.Path('gs://henning-test/empty_folder')
>>> f'{e.exists()=} {e.is_dir()=}'
'e.exists()=True e.is_dir()=False'
>>> list(e.iterdir())
[PosixGPath('gs://henning-test/empty_folder')]

>>> f = epath.Path('gs://henning-test/folder')
>>> f'{f.exists()=} {f.is_dir()=}'
'f.exists()=True f.is_dir()=False'

>>> t = epath.Path('gs://henning-test/folder/test.txt')
>>> f'{t.exists()=} {t.is_dir()=}'
't.exists()=True t.is_dir()=False'
>>> list(t.iterdir())
[PosixGPath('gs://henning-test/folder/test.txt/test.txt')]

adlfs support

Hey y'all,

Currently you offer support for s3:// (using s3fs) and gs:// (using gcfs). If it's not too much work, could you please add support for az:// aka Azure blob using fsspec's adlfs.

cannot "pip install etils"

Hi,

I cannot install etils[path] through pip. This is the following error message:

ERROR: Could not find a version that satisfies the requirement etils[epath] (from versions: none)
ERROR: No matching distribution found for etils[epath]

Thank you in advance!

etils example fails to load on public colab due to missing mediapy dependency

running the example colab out of the box on the public instance of Google Colab (colab.research.google.com) immediately fails with:

[/usr/local/lib/python3.10/dist-packages/etils/ecolab/array_as_img.py](https://localhost:8080/#) in <module>
     37   import IPython
     38   import IPython.display
---> 39   import mediapy as media
     40   # pylint: enable=g-import-not-at-top
     41 

ModuleNotFoundError: No module named 'mediapy'

Each etils sub-modules require deps to be installed separately (e.g. `from etils import ecolab` -> `pip install etils[ecolab]`)

Support walk in epath?

hi there!

is there a plan to support walk in epath for different backends? i can contribute if this feels like a good idea.

AttributeError when calling etils.eapp.better_logging

Hello,

When I am trying to use eapp.better_logging, it seems the module epy is trying to call a non-existent method is_borg

Traceback (most recent call last):
  File "<project path>/test_etils.py", line 20, in <module>
    app.run(main, flags_parser=eapp.make_flags_parser(Args))
  File "<env path>/lib/python3.9/site-packages/absl/app.py", line 306, in run
    callback()
  File "<env path>/lib/python3.9/site-packages/etils/eapp/logging_utils.py", line 64, in _better_logging
    if epy.is_borg():
AttributeError: module 'etils.epy' has no attribute 'is_borg'

Minimal code for reproducing the issue with python3.9:

import dataclasses

from absl import app
from etils import eapp


@dataclasses.dataclass
class Args:  # Define `--user=some_user --verbose` CLI flags
    user: str
    verbose: bool = False


def main(args: Args):
    if args.verbose:
        print(args.user)


if __name__ == "__main__":
    eapp.better_logging()
    app.run(main, flags_parser=eapp.make_flags_parser(Args))

Best,
Hicham