Support AWS SDK for JavaScript v3 about dynamodb-toolbox HOT 49 CLOSED

Is there any update? I can see an open PR for a migration to the v3 SDK but it's been open for months now.

from dynamodb-toolbox.

v0.4 is finally live! Conversion to the v3 SDK is a high priority.

from dynamodb-toolbox.

This PR has been merged today, so v3 got DocumentClient.

aws/aws-sdk-js-v3#2097

from dynamodb-toolbox.

Yes, this is coming. We've been waiting on our migration to v0.4 that does automatic type inferencing. This should be ready soon.

from dynamodb-toolbox.

Here is the my example problem statement :

I have a service and I am using eventbridge and dynamodb. I want to use version 3 aws sdk to reduce the bundle size. I used eventbridge module. Now, I want to update to use dynamodb module. But I can't as I am dependent on dynamodb toolbox and it depends on aws sdk.

As we are not yet to version 1 and not to the stable release yet, to me it makes sense to use the optimised version of aws sdk.

from dynamodb-toolbox.

Hi all, what is the status of this?

This become more of an issue with release of nodejs18 runtime which contains only AWS SDK V3

aws.amazon.com/blogs/compute/node-js-18-x-runtime-now-available-in-aws-lambda

Hey :)

There's a discussion regarding this in #174.
If anybody wants to open a PR for this I'd be happy to review it,
otherwise I'll start working on it sometimes during next week probably.

from dynamodb-toolbox.

Just realized I didn't update here, my bad 🤦

I'm working on this in #471 🙏

from dynamodb-toolbox.

Hey everyone,
my pr is ready for review,
feel free to leave your review and hopefully we'll get it merged within 2~3 days 🙏
I want it to be reviewed carefully as possible because it's a big and breaking change.

from dynamodb-toolbox.

team any input or update regarding SDk3 support?

from dynamodb-toolbox.

Hey guys,
I've merged my PR,
this will be released in the next version which should be published in few hours (need to make some last adjustments) 🙏

from dynamodb-toolbox.

after the change, I keep getting SerializationException for some Entities which I think should be handled by dynamodb-toolbox?
I tried to configure unmarshall to wrapNumbers per https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_lib_dynamodb.html#configuration
but that does not help

SerializationException: STRING_VALUE cannot be converted to Integer
    at throwDefaultError (/opt/nodejs/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at de_ScanCommandError (/opt/nodejs/node_modules/@aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:2293:51)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /opt/nodejs/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/baseCommand/DynamoDBDocumentClientCommand.js:26:34
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async Table.scan (/opt/nodejs/node_modules/dynamodb-toolbox/dist/classes/Table/Table.js:522:28)

Are you using the limit parameter in your code by any chance? If so, I recommend to update ddb-toolbox 0.8.2, it contains a fix for that.

It fixed the problem, thank you

from dynamodb-toolbox.

I'm just trying this out today and sadly this doesn't seem to be the case.

DocumentClient doesn't seem to be part of https://github.com/aws/aws-sdk-js-v3 anymore and the API for DynamoDB in "@aws-sdk/client-dynamodb" seems to be at least slightly different to the one DocumentClient was providing.

from dynamodb-toolbox.

Yes, we discovered this right after v3 was released. Still looking for suggestions for handling this, but it would be really nice if AWS would reimplement the DocumentClient.

from dynamodb-toolbox.

Given the separate packages, could v2 and v3 coexist and the library auto-detect (or have a new mechanism for passing DDB for v3) which version is being used?

from dynamodb-toolbox.

The issue with v3 is that there isn’t a DocumentClient, which is what this library uses under the hood. I’ve already built some prototypes that use v3 and apply marshall/unmarshall in the library, but this would create a massive breaking change. I was hoping that v3 would have a DocumentClient before GA, but unfortunately, that didn’t happen.

from dynamodb-toolbox.

V3 has only a single concept and everything comes down to DynamoDBClient and Send Method. for the transaction we used the instance of a TransactWriteItemsCommand which accepts TransactWriteItemsCommandInput

from dynamodb-toolbox.

We're not planning to publish modular implementation of Document Client as of now

The modular implementation was added in c29b57d (#2031)

Having said that, this is still an experimental changes and have to be reviewed within the team.
Do comment on the PR if you have ideas on anything can be done better or differently.

from dynamodb-toolbox.

There is no update on this topic, even though many other libraries already implemented this. OneTable is already compatible with SDK3, and AWS released an update. Any ETA?

from dynamodb-toolbox.

@rati-dzidziguri there's a few issues as mentioned in this thread that definitely increase the complexity on getting this implemented in this library however, I'm sure Jeremy would be happy to review a PR solving this if submitted.

from dynamodb-toolbox.

Is it just bundle size that is the driver for updating to v3, or is there something else?

Does v2 have bugs that are fixed in v3? Etc.

I agree that it would be nice to use v3 and I agree that using un/marshall is my (totally uneducated and based upon the AWS media releases) preferred choice, however I just want to see this library get to v1 sooner rather than later.

from dynamodb-toolbox.

Any update on this?

from dynamodb-toolbox.

Hi all, what is the status of this?

This become more of an issue with release of nodejs18 runtime which contains only AWS SDK V3

https://aws.amazon.com/blogs/compute/node-js-18-x-runtime-now-available-in-aws-lambda/

from dynamodb-toolbox.

Hey guys,
I've merged my PR,
this will be released in the next version which should be published in few hours (need to make some last adjustments) 🙏

this is a breaking change and the documentation does not seem to be updated.
is there any example and/or instructions how to update existing code?

Hey! I've updated the docs with information regarding how to instantiate a DocumentClient with AWS SDK v3. Regarding the usage of the library, except for the DocumentClient everything stays the same pretty much :)

Let me know if you need anything else.

I found the updates in https://www.dynamodbtoolbox.com/docs - thank you!

Before I just had a quick look at the "Quick start" section in README.md which does not mention AWS SDK v3

from dynamodb-toolbox.

I was under the impression that v3 would have the same semantics as v2, just a different import method. If that's the case, I don't think any changes will be needed, especially since v0.2 of this library just requires the DocumentClient.

from dynamodb-toolbox.

Would the easiest option in the meantime be to create a proxy for DocumentClient?

I'll have some spare time over the next couple of weeks to see if I can add if you think it's a reasonable option.

from dynamodb-toolbox.

It’s either that, or we build the marshaling and unmarshalling into the platform.

from dynamodb-toolbox.

Marshaling is present in the @aws-sdk/util-dynamodb module, if that helps.

from dynamodb-toolbox.

Yup. The question is do we make that a dependency? We could do that and then require the user to pass in the v3 DynamoDB client like we do currently with the DocumentClient (so they could still instrument it). Of course, as soon as we build that, AWS will release an update with a DocumentClient. 🤦‍♂️

from dynamodb-toolbox.

Although this is one of the best libraries out there, the inability to use the latest SDK version that drastically reduces the bundle size for AWS lambda functions block people from installing this package. The upgrade process is not straightforward; however, it is very unpleasant to see such a great product that cannot catch up with the updates AWS even releases NodeJS 14 official support now, so it would be awesome to see this library as a community standard going forward.

from dynamodb-toolbox.

I can confirm in other projects; we have been using V3 without notable issues. The marshall/unmarshall indeed is the recommended way from AWS ATM. and is the one we have been referring to as well and can say it works without issues.

Would you please let us know what challenges the library is facing so that we as a community might try to help and support you in this migration journey?

from dynamodb-toolbox.

I can confirm in other projects; we have been using V3 without notable issues. The marshall/unmarshall indeed is the recommended way from AWS ATM. and is the one we have been referring to as well and can say it works without issues.

Would you please let us know what challenges the library is facing so that we as a community might try to help and support you in this migration journey?

How have you been doing transactions with v3 without the DocumentClient?

from dynamodb-toolbox.

It looks like they (@trivikr) are "prioritising" DocumentClient in v3.

Considering it's popularity in terms of reactions, and the comments on DocumentClient support - we should consider prioritizing it. I'll raise this request in our planning meeting.

Might be worth waiting to see what they do with this before investing any significant effort.

However, I don't know if this defeats the point raised above (taken from the experimental PR for DocumentClient put):

We're not planning to publish modular implementation of Document Client as of now, as that would not be straightforward.

V3 doesn't seem complete yet.

from dynamodb-toolbox.

I'll need to check the compatibility with the library!

from dynamodb-toolbox.

Do we have a plan for this? The implementation in aws/aws-sdk-js-v3#2097 is not compatible with this library.

I think we should not rely on documentClient and use marshal/unmarshal instead as it is the recommended path forward by AWS.

If we can agree on the path forward for this library, I am happy to implement it and create a PR.

from dynamodb-toolbox.

What benefits does v3 bring for us? Bundle size? Is there anything else? Anything we need from v3 that v2 doesn't already do?

Would the effort be better spent getting this library to v1 and then adding support for v3 after?

We haven't got a stable version of this library yet and there is a lot of feature spread which needs fixing prior to v1.

from dynamodb-toolbox.

@jeremydaly would love to here your thoughts on this. Would be nice to know if this is in roadmap or not.

from dynamodb-toolbox.

https://www.npmjs.com/package/@aws-sdk/lib-dynamodb

from dynamodb-toolbox.

Or: https://www.npmjs.com/package/dynamodb-onetable

from dynamodb-toolbox.

I like the way that typedorm have implemented this, with an abstraction that allows you to choose v2 or v3. It's quite neat.

from dynamodb-toolbox.

@jeremydaly How should I use the v3 client? I am getting type errors

from dynamodb-toolbox.

@jeremydaly How should I use the v3 client? I am getting type errors

You don't... yet.

from dynamodb-toolbox.

How much more time will it take for V4 to be released? And what are the changes we are looking for, I might try to send a PR

from dynamodb-toolbox.

How much more time will it take for V4 to be released? And what are the changes we are looking for, I might try to send a PR

It's been in the pipeline for a while, so hopefully we're close! I think the code was done but unmergeable. If you really need to use v3 have a look at TypedORM.

PS, it's v0.4 not v4. We haven't made it to v1 yet. Hopefully we're also close to that!

from dynamodb-toolbox.

Any updates on this?
As mentioned before nodejs18 runtime contains only AWS SDK V3 and node 16 support is ending.

from dynamodb-toolbox.

https://github.com/jeremydaly/dynamodb-toolbox/releases/tag/v0.8.0 here we go now. It would be good to see the updated documentation on how to use this version, and this is a significant milestone achieved, as this has been the case for a while now.

from dynamodb-toolbox.

Hey guys,
I've merged my PR,
this will be released in the next version which should be published in few hours (need to make some last adjustments) 🙏

this is a breaking change and the documentation does not seem to be updated.

is there any example and/or instructions how to update existing code?

from dynamodb-toolbox.

Hey guys,
I've merged my PR,
this will be released in the next version which should be published in few hours (need to make some last adjustments) 🙏

this is a breaking change and the documentation does not seem to be updated.

is there any example and/or instructions how to update existing code?

Hey!
I've updated the docs with information regarding how to instantiate a DocumentClient with AWS SDK v3.
Regarding the usage of the library, except for the DocumentClient everything stays the same pretty much :)

Let me know if you need anything else.

from dynamodb-toolbox.

after the change, I keep getting SerializationException for some Entities which I think should be handled by dynamodb-toolbox?

I tried to configure unmarshall to wrapNumbers per https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_lib_dynamodb.html#configuration
but that does not help

SerializationException: STRING_VALUE cannot be converted to Integer
    at throwDefaultError (/opt/nodejs/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at de_ScanCommandError (/opt/nodejs/node_modules/@aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:2293:51)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /opt/nodejs/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/baseCommand/DynamoDBDocumentClientCommand.js:26:34
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async Table.scan (/opt/nodejs/node_modules/dynamodb-toolbox/dist/classes/Table/Table.js:522:28)

from dynamodb-toolbox.

after the change, I keep getting SerializationException for some Entities which I think should be handled by dynamodb-toolbox?

I tried to configure unmarshall to wrapNumbers per https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_lib_dynamodb.html#configuration
but that does not help

SerializationException: STRING_VALUE cannot be converted to Integer
    at throwDefaultError (/opt/nodejs/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at de_ScanCommandError (/opt/nodejs/node_modules/@aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:2293:51)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /opt/nodejs/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/baseCommand/DynamoDBDocumentClientCommand.js:26:34
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /opt/nodejs/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async Table.scan (/opt/nodejs/node_modules/dynamodb-toolbox/dist/classes/Table/Table.js:522:28)

Are you using the limit parameter in your code by any chance?
If so, I recommend to update ddb-toolbox 0.8.2, it contains a fix for that.

from dynamodb-toolbox.