Giter VIP home page Giter VIP logo

eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock's Introduction

pangeo-forge-recipes

PyPI version CI Codecov Documentation Status Code style: black NSF Award 2026932

pangeo-forge is an open-source tool designed to aid the extraction, transformation, and loading of datasets. The goal of pangeo-forge is to make it easy to extract datasets from traditional data repositories and deposit them into cloud object storage in analysis-ready, cloud-optimized format.

pangeo-forge is inspired by conda-forge, a community-led collection of recipes for building Conda packages. We hope that pangeo-forge can play the same role for datasets.

Documentation

More can be learned about pangeo-forge, its progress, and related subprojects in its official documentation.

Contributing

pangeo-forge is still early in development - there are several ways to contribute:

  1. Create a recipe for a dataset you are interested in
  2. Open an issue or pull request here or in any of the related subprojects (pangeo-smithy, staged-recipes)
  3. Check out the project roadmap

Get in touch

Discussions on Pangeo Forge are generally hosted biweekly on Mondays at 2pm ET. Calendar link here. We aim to announce cancellations on this discourse thread.

License

This project is licensed under the Apache License, Version 2.0.

eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock's People

Contributors

pangeo-forge[bot] avatar

Watchers

avatar avatar avatar avatar

eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock's Issues

`FSTimeoutError` during caching

The latest run of this feedstock failed with what appears to be an fsspec related error. @martindurant, does this ring any bells to you?

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
      response = task()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
      lambda: self.create_worker().do_instruction(request), request)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
      return getattr(self, request_type)(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
      bundle_processor.process_bundle(instruction_id))
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
      input_op_by_transform_id[element.transform_id].process_encoded(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
      self.output(decoded_value)
    File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1956, in <lambda>
    File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/executors/beam.py", line 40, in exec_stage
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 156, in cache_input
      config.storage_config.cache.cache_file(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 173, in cache_file
      _copy_btw_filesystems(input_opener, target_opener)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 43, in _copy_btw_filesystems
      data = source.read(BLOCK_SIZE)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/http.py", line 590, in read
      return super().read(length)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/spec.py", line 1643, in read
      out = self.cache._fetch(self.loc, self.loc + length)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/caching.py", line 377, in _fetch
      self.cache = self.fetcher(start, bend)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 111, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 94, in sync
      raise FSTimeoutError from return_result
  RuntimeError: fsspec.exceptions.FSTimeoutError [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/cache_input/Execute-ptransform-56']
  ,

    a6170692e70616e67656f2d66-10310835-wi8t-harness-drzf
        Root cause: Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
      result[0] = await coro
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/http.py", line 632, in async_fetch_range
      out = await r.read()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1037, in read
      self._body = await self.content.read()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiohttp/streams.py", line 375, in read
      block = await self.readany()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiohttp/streams.py", line 397, in readany
      await self._wait("readany")
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiohttp/streams.py", line 304, in _wait
      await waiter
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiohttp/helpers.py", line 720, in __exit__
      raise asyncio.TimeoutError from None
  asyncio.exceptions.TimeoutError

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1956, in <lambda>
    File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/executors/beam.py", line 40, in exec_stage
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 156, in cache_input
      config.storage_config.cache.cache_file(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 173, in cache_file
      _copy_btw_filesystems(input_opener, target_opener)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 43, in _copy_btw_filesystems
      data = source.read(BLOCK_SIZE)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/http.py", line 590, in read
      return super().read(length)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/spec.py", line 1643, in read
      out = self.cache._fetch(self.loc, self.loc + length)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/caching.py", line 377, in _fetch
      self.cache = self.fetcher(start, bend)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 111, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 94, in sync
      raise FSTimeoutError from return_result
  fsspec.exceptions.FSTimeoutError

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
      response = task()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
      lambda: self.create_worker().do_instruction(request), request)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
      return getattr(self, request_type)(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
      bundle_processor.process_bundle(instruction_id))
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
      input_op_by_transform_id[element.transform_id].process_encoded(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
      self.output(decoded_value)
    File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1956, in <lambda>
    File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/executors/beam.py", line 40, in exec_stage
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 156, in cache_input
      config.storage_config.cache.cache_file(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 173, in cache_file
      _copy_btw_filesystems(input_opener, target_opener)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 43, in _copy_btw_filesystems
      data = source.read(BLOCK_SIZE)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/http.py", line 590, in read
      return super().read(length)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/spec.py", line 1643, in read
      out = self.cache._fetch(self.loc, self.loc + length)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/caching.py", line 377, in _fetch
      self.cache = self.fetcher(start, bend)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 111, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 94, in sync
      raise FSTimeoutError from return_result
  RuntimeError: fsspec.exceptions.FSTimeoutError [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/cache_input/Execute-ptransform-56']
timestamp: '2022-10-31T16:07:53.806969626Z'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.