Comments (12)
Please refer to the entire code here: https://github.com/Timothy102/soniox/blob/main/main.py.
I have no idea what an EnhancedModule is. I'm running the code above and nothing else.
from ray.
Can you reformat the above description? It seemed that the repro steps are mis-formatted.
Can you also paste the error message?
from ray.
Here's the error I receieve.
TypeError: Could not serialize the function main.store_all_videos_and_captions_from_urls:
================================================================================
Checking Serializability of <function store_all_videos_and_captions_from_urls at 0x7f7e44fb8b80>
================================================================================
!!! FAIL serialization: cannot pickle 'EnhancedModule' object
Detected 3 global variables. Checking serializability...
Serializing 'Channels' <class '__main__.Channels'>...
Serializing 'Channel' <class '__main__.Channel'>...
Serializing 'preprocessing_video' <ray.remote_function.RemoteFunction object at 0x7f7e44faab80>...
!!! FAIL serialization: cannot pickle 'EnhancedModule' object
Serializing '_function' <function preprocessing_video at 0x7f7e49233dc0>...
!!! FAIL serialization: cannot pickle 'EnhancedModule' object
Detected 1 global variables. Checking serializability...
Serializing 'Video' <__main__.ActorClass(Video) object at 0x7f7e44faaa90>...
!!! FAIL serialization: cannot pickle 'EnhancedModule' object
Serializing '__generator_backpressure_num_objects' None...
================================================================================
Variable:
FailTuple(Video [obj=<__main__.ActorClass(Video) object at 0x7f7e44faaa90>, parent=<function preprocessing_video at 0x7f7e49233dc0>])
from ray.
What's the main.py
? Can you provide a repro script? Without it, it's very difficult for us to dig down further and do anything.
from ray.
FAIL serialization: cannot pickle 'EnhancedModule' object
What's this EnhancedModule
specifically?
from ray.
tag @hongchaodeng
from ray.
Thanks for providing it! Added it to our support backlogs. Will debug it.
from ray.
OK I looked at the code and figured out the root cause here.
The EnhancedModule comes form the yt-dlp library that your code is using:
This one is not serializable. There might be more though.
You have to do customized serialization as documented here: https://github.com/yt-dlp/yt-dlp/blob/ac817bc83efd939dca3e40c4b527d0ccfc77172b/yt_dlp/compat/compat_utils.py#L30
from ray.
Hey, thank you, but you pasted the same link twice and that still doesn't tell me how I can fix this issue.
Your prompt attention would be greatly appreciated.
from ray.
Sorry.
It is this doc: https://docs.ray.io/en/latest/ray-core/objects/serialization.html#customized-serialization
from ray.
So, how exactly should this class of mine look like?
class Channel: """Each Channel is a class of its own. """ def __init__(self, channel_id, url=None, args=None): self.url = url self.channel_id = channel_id self.output_folder = args.output_folder
from ray.
https://github.com/Timothy102/soniox -- please check do.py again. This code is not coming through. I'm dying here.
from ray.
Related Issues (20)
- C++ Python Cross-language invocation HOT 2
- [RLlib] Getting Started first example executed with TF2, is followed by an error
- [Core|Dataset] Ray job stuck with idle actors with no tasks HOT 3
- [Docs] Dark mode visibility of table in scalability and overhead benchmarks
- TorchTrainer distributed training is not running properly
- RLlib - The batch for module_id default_policy is empty! HOT 1
- Wheels for Windows on PyPI HOT 1
- Release test ml_user_tune_rllib_connect_test.aws failed HOT 1
- Release test rllib_learning_tests_pong_dreamerv3_tf2.aws failed
- Release test long_running_impala.aws failed HOT 1
- CI test windows://python/ray/serve/tests:test_metrics is flaky HOT 2
- [RLlib] Deprecation warnings won't turn off HOT 1
- [Core] Improve performance of actor task execution by amortizing cost of checking for `async` methods HOT 1
- [RLlib] Custom RLModule import fails using absolute imports (with non-zero `env_runners`) HOT 1
- [core][experimental] shared memory channel can't finish begin_read HOT 2
- [<Ray component: Core|RLlib|etc...>]
- Release test dataset_shuffle_sort_1tb.aws failed HOT 2
- [Dashboard] Ray dashboard should render structured logs in a nicer format
- Failed to start the dashboard , return code 1
- [docs] Column size for cards are not equal HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ray.