Comments (6)
I think we want to add here
https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/s3.py#L324-L341
Similar to:
https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/s3.py#L68-L69
from prefect-aws.
Actually, maybe it's not that straightforward since botocore.Config is not serializable as a pydantic field.
from prefect-aws.
No, serializing complex objects like botocore.Config or ssl.SSLContext is problematic - you could pickle and then use base64 to turn it all into a string, but how big is that string? I understand Blocks better now and my gut doesn't like this idea so much. Pydantic does support pickle, but to switch S3Bucket to using that is a bad idea.
But is it really necessary to serialize/store the Config object? How about this idea? It is a bit out-of-the-box, but it would work. Make S3Bucket.config a property and then there is Prefect's default implementation of the getter such that getting it actually creates it on the fly. Users could override this config getter with their own customizations. But their code would have to reassign the property every time it loads their special S3Bucket from the storage. Or the config property getter first looks for an optional create_custom() function attached to the object being manipulated. A small price to pay, but they are doing non-standard things.
Or just have S3Bucket.config be a function pointer such that internally, S3Bucket calls config() to get an actual Config. That can be overridden by user code. But that is basically the equiv of a property, just different syntax.
Pydantic is working on dealing with properties: pydantic/pydantic#935
If the creation function really needs to be stored, then the inspect module will return the actual source code and so it could be reconstructed upon loading from storage... and JSON will handle a text string. This is out of the box too, but I don't see any fundamental issues at the moment. S3Bucket would need a function to hook into Pydantic's json_encoder and a validator that understands how to reconstruct a string into a Config creation function, but that is not very complex.
For obsolete S3Buckets being read in, the validator could add the default create function etc. I am assuming any custom create function won't be very complex.
This can be solved, but what is the cost/benefit of adding this extra feature? I guess I'm the first to ask and others are just rolling their own solutions? I prefer my first idea.
from prefect-aws.
Would defining the attribute as
config: Optional[botocore.Config] = Field(None, exclude=True)
be acceptable? It gets documented and typed, but is prevented from being included in any export to JSON which it perhaps ought to be.
from prefect-aws.
If it's a Pydantic model/block, I think we want it to be portable; more info here https://orion-docs.prefect.io/ui/blocks/
from prefect-aws.
AwsCredentials now accepts client parameters.
from prefect-aws.
Related Issues (20)
- push_to_s3 and pull_from_s3 do not support additional AwsCredentials block parameters
- AWS S3 copy and move tasks and `S3Bucket` methods
- Prefect Network Configuration defined in work pool fails to get applied in some scenarios when using with ECS Fargate Task HOT 2
- AwsCredentials import raising PydanticUserError HOT 6
- boto3 client use_ssl wrong default
- put_directory creates a new s3_client for each file uploaded HOT 2
- v0.4.4 results in the prefect task to be given the masked PREFECT_API_KEY
- allow define only aws region name through AwsCredentials HOT 1
- Add a Lambda function block HOT 1
- Accept `None` as an argument to Launch Type for ECSTask with `publish_as_work_pool` HOT 1
- Task definition caching does not work if the task definitions come from separate deployments. HOT 2
- Error in version 0.4.8: TypeError: unhashable type: 'dict' HOT 3
- Credential use examples are in correct in documentation HOT 6
- ECS Works Pools Should Support Specifying Volumes for Flow Runs HOT 1
- `S3Bucket.copy_object`: Only resolve target path with self if `to_bucket` is not defined
- ECSTask block `publish_as_work_pool` does not set `network_configuration` HOT 2
- Improvement to ECS worker setup guide
- ECS worker: Updating family setting to template flow name/deployment
- Change logging prefix to avoid unnecessary task definition registrations
- Add Python 3.12 to test matrix
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prefect-aws.