jeremydaly / dynamodb-toolbox Goto Github PK
View Code? Open in Web Editor NEWA simple set of tools for working with Amazon DynamoDB and the DocumentClient
Home Page: http://dynamodbtoolbox.com
License: MIT License
A simple set of tools for working with Amazon DynamoDB and the DocumentClient
Home Page: http://dynamodbtoolbox.com
License: MIT License
Another issue that has crept up with large tables using lots of entities/facets, is the 20 attribute projection limit per GSI. I've used mappings in the past to use generic attribute names, thought I know this can be difficult to manage. Perhaps if the library had a way to indicate which GSIs each field needed to be projected to, then the mapping could be optimized. This might need to be a CLI feature, but I could see it being useful.
The other possibility is to add an "attribute id" of some sort to the entity/facet definition and then use that to determine which attributes to map data to. I think this would probably allow for quite a bit of future flexibility, but I'd need to test that out more.
This is another placeholder for now.
Problem: when creating put parameters that attempt to set a field to zero (0), the Model throws an error that "(field) is a required field".
Expected Result: I expect to be able to set a required field to zero.
Steps to Reproduce: I have a code snippet that reproduces the issue, below.
const { Model } = require('dynamodb-toolbox');
const model = new Model('TestModel', {
table: 'TestTable',
partitionKey: 'pk',
timestamps: true,
schema: {
pk: { type: 'string', hidden: true, default: 'Test' },
count: { type: 'number', required: 'always' },
},
});
const result = model.put({ count: 0 }); // This throws
Note that I can work around the issue right now by changing the required value from 'always'
to false
.
The parsing utility is handy, but right now you need to pass returned items to the correct model to parse them. I anticipated this originally, so I added the model
option and defaulted it to true, which adds a __model
attribute that stores the model name. But since there is no query support (see #5), the library doesn’t currently have a way to automatically map to the correct model.
This also needs some more thinking. A model registry would make this easier, but that would require changing the instantiation method to create a new
instance of the DynamoDB Toolbox, and then use methods to build models. You could still expose just the Model
class, but that might be substandard to something like:
const Toolbox = require(‘dynamodb-toolbox’)
const DDB = new Toolbox({ ... some global configs ... })
const MyModel = new DDB.Model(‘MyModel’,{ .. config... })
Models could them be automatically registered.
Then when you get Items
back from DynamoDB, you could use a DDB.parse()
that could use the registered models to automatically parse each item correctly. Thoughts and ideas are welcome.
aws-sdk
is already included in Lambda NodeJS runtimes. If this lib is included, say, as part of a Lambda Layer, then ~50MB is added to the payload. Perhaps it can be added as a peer dependency instead?
/Question
Interested to know what you will be doing with the TODO regarding object schemas.
For example I am using data and pdata entity level map type attributes. The pdata will be an object that I want projected into the indexes and the data is for pretty much everything else.
Say I want to update the pdata object type with a single attribute and not overwrite the existing record then it would be nice to be able to do a:
const userDbObj = { pk: sub, sk: 'User', pdata: { $set: { 'address.town': 'timbuktoo' } } }
await User.update(userDbObj)
if the 'address' attr has on the pdata object has not already been created then an error is thrown.
Thanks for the lib.
Do you intend to add Typescript support at some point in the future. Currently looking at this as an alternative to the awslabs/datamapper package as this supports some features that it does not currently and the only major thing I would be loosing (as far as I can tell), would be the strict typing.
In otherwords, do you have an ETA for the PR out for adding it as of yet?
Method to export the CloudFormation for your Table configuration to JSON or YAML
Great job on the toolbox. I appreciate it.
It was a little unclear from the documentation what the difference between alias and map is. I figured it out by looking into the source, but a little clarification would probably help others.
I was trying to use the map property as a way to translate a string in the DB table to an object output. org:{id:1234}
into orgId:1234
in the db table and then back to org:{id:1234}
when read. On the input using the default
function solves it, but there seems to be no way to translate the output back when read. Is that correct?
Opening this in it's own ticket since it's sort of long and I wanted to allow it to be closed separately in the case I am completely insane.
I've been tinkering with this library for the last couple of days and trying to imagine how queries could work. A few things strike me as having potential here.
Model Introspection
First, the array definition for composite keys potentially gives the model some important information.For example, if only 1 of 2 parts of the composite exist, we may be able to infer that the query needs to have a begins_with() on the SK.
Potentially the GSI in use could be inferred by which data items are populated as well, though there may be a need to provide an explicit hint in some cases.
GSI Formatting
All GSI pk and sk schema formatting could probably follow the same schema formatting rules as the main pk and sk do. They could be dynamically constructed just like the main table SK constructed. The model would just need to flag those schema elements as belonging to a GSI. This could be done on the schema element itself, or in the main model definition.
As a side note, I think the gsi and table definition data could be lifted out of the model and defined separately for easier single table design.
DAT 403 Example
I'm a huge fan of Rick Houlihan's ReInvent presentations, as I know @jeremydaly is as well. I just spent a moment trying to envision how the employee model from his 2019 presentation might be constructed and queried. Here is what I came up with. There are likely errors
const Employee = new Model('Employee', {
// Specify table name
table: 'singletable',
// Define partition and sort keys
partitionKey: 'pk',
sortKey: 'sk',
// GSI here - type: gsi | local, defaults to gsi
indexes: [
{ name: 'gsi1', partitionKey: 'gsi1pk', sortKey: 'gsi1sk' },
{ name: 'gsi2', partitionKey: 'gsi2pk', sortKey: 'gsi2sk' },
{ name: 'gsi3', partitionKey: 'gsi3pk', sortKey: 'gsi3sk' }
],
// Define schema
schema: {
// primary keys
pk: { type: 'string', default: (data) => `${data.employeeid}` }, // Access Pattern 9
sk: { type: 'string', hidden: true },
sk0: ['sk', 0, { type: 'string', default: (data) => `${data.type}` }], // Access Pattern 9
sk1: ['sk', 1, { type: 'string', default: (data) => `${data.managerid}` }],
// GSI1
gsi1pk: { type: 'string', default: (data) => `${data.employeeemail}` }, // Access Pattern 10
gsi1sk: { type: 'string', default: (data) => `${data.sk}` }, // Access Pattern 10
// GSI2
gsi2pk: { type: 'string', default: (data) => `${data.manageremail}` }, // Access pattern 15
gsi2sk: { type: 'string', default: (data) => `${data.sk}` }, // Access pattern 15
// GSI3
gsi3pk: { type: 'string', default: (data) => `${data.city}` }, // Access pattern 14
gsi3sk: { type: 'string', hidden: true }, // Access pattern 14
gsi3sk0: ['gsi3sk', 0, { type: 'string', default: (data) => `${data.building}` }], // Access Pattern 14
gsi3sk1: ['gsi3sk', 1, { type: 'string', default: (data) => `${data.floor}` }], // Access Pattern 14
gsi3sk2: ['gsi3sk', 2, { type: 'string', default: (data) => `${data.aisle}` }], // Access Pattern 14
gsi3sk3: ['gsi3sk', 3, { type: 'string', default: (data) => `${data.desk}` }], // Access Pattern 14
// data
type: { type: 'string', default: 'E' },
employeeid: { type: 'string' },
employeeemail: { type: 'string' },
hiredate: { type: 'string' },
managerid: { type: 'string' },
manageremail: { type: 'string' },
city: { type: 'string' },
building: { type: 'string' },
floor: { type: 'string' },
aisle: { type: 'string' },
desk: { type: 'string' }
// ... other data attributes
}
})
Then you could query the access patterns thusly:
Access Pattern 9 - Employee By ID
Since type maps to the first part of the composite sk, we can infer begins_with(). The main table's pk and sk are populated but none of the GSI's are, so we can also infer that this is a query against the table and not a GSI.
let item = {
employeeid: 'employee22',
type: 'E'
}
let params = Employee.query(item)
Access Pattern 10 -Employee by Email
This is similar to the above example but the main table's pk will not be populated. Instead, GSI1's pk and sk will be populated, but no others, so we can infer that this query should be against GSI1.
let item = {
employeeemail: '[email protected]',
type: 'E'
}
let params = Employee.query(item)
Access Pattern 14 - employees by city, building, floor, aisle, desk
This one populates parts of GSI3, along with a partial gsi3sk, so we can infer that this query will be against gsi3, and will require a begins_with() on gsi3sk.
let item = {
city: 'Atlanta',
building: 'Davis West',
floor: '4'
}
let params = Employee.query(item)
Access Pattern 15 - Employees by Manager
This one is a little fuzzy because I am not entirely certain which elements in Rick's table contained the managerid - maybe it's the second element in the table's sk? I'm just guessing. But if that's true, this query will grab the employees that this person manages, again inferring gsi2 and begins_with() on gsi1sk.
let item = {
manageremail: '[email protected]',
type: 'E'
}
let params = Employee.query(item)
There it is, warts and all. Hopefully at least some of this will add to the conversation.
Would love to see TypeScript types
Reopening this in reference to #14 since v0.2 is out.
When I started building some of these helper utilities, I was only using them to generate specific parameters (like UpdateExpression
and ExpressionAttributeNames
). Now that additional DynamoDB parameters can be passed directly through each method, automatically calling the appropriate DocumentClient method would avoid the extra step. This could be optional, but the library already requires and uses the DocumentClient to work with set
types.
There are use case that would require you to pass in your own version of the DocumentClient. This could include wrapped versions for AWS X-Ray, or perhaps some other observability tool. The client
could be passed in on model creation (or as a global config, see #6). We could also support X-Ray out of the box by allowing you to pass in the aws-xray-sdk
as a configuration parameter. I need to see how conditionally applying X-Ray would affect tree-shaking algorithms.
The current test suite has over 100 tests, but they mostly verify expected parameters. It would be helpful (and safer) to test them against a DynamoDB instance. A cloud version would be best, but since we want TravisCI to run the tests for us, we should be able to use DynamoDB Local.
Typos are marked with bold:
put fails when field is configured as map: 'field'
and required: true
These would be like using functions with default
, except it would apply every time you input data to the field. This would be useful for enforcing case conversions or other types of formatting (like phone numbers or custom composite keys).
// Define schema
schema: {
pk: { type: 'string', alias: 'id' },
sk: { type: 'string', hidden: true },
someField: {
type: 'string',
alias: 'name' ,
convert: (val,data) => { // gets the submitted value and the rest of the values
return val.toLowerCase() // do something interesting here
}
},
}
The data
object here should probably NOT be a simple reference so that data in other fields isn't accidentally mutated. Though having a way to alter data in another input might be a useful feature.
As mentioned in #6, in order to achieve multi-model parsing from a single table design, the system needs to be able to understand what Model
each Item
represents. I was initially thinking about using a "model registry", but I'd rather the concepts translate better. My mental model is this:
A TABLE is directly mapped to a DynamoDB table. It has a KeySchema
, AttributeDefinitions
, LocalSecondaryIndexes
, and GlobalSecondaryIndexes
. A table is the raw repository that uses generic names (like pk
, sk
, and data
) to label attributes.
A MODEL is a "schema" definition that is mapped to a Table
's Item
. A model could represent something more relational, such as a denormalized copy of some data, or it could also represent a part of a larger "entity", such as in the attribute versioning case from Rick Houlihan's recent talk. Model
s are meant to be flexible in the sense that you are in control of how your data is stored in each Item
(or "record").
An ENTITY is any singular, identifiable and separate object. A single Model
may represent an Entity, but and Entity may also be made up of multiple Model
s (refer to the attribute version case linked above). I haven't fully thought through adding an Entity
concept to this library, but I could see cases where it might help.
An ACCESS PATTERN is a method used to retrieve Entities
and Model
s from the Table
. The NoSQL Workbench for DynamoDB has a concept of "facets", which sort of feels like this, but would require the developer to write the actually retrieval code.
An UPDATE PATTERN is a method used to update Entities
and Model
s in the Table
. This may be a simple putItem
or updateItem
, or it could be a more complex Transaction
with multiple updates, condition checks, etc. This would also be the responsibility of the developer to write the implementation.
The first step is adding the higher level wrapper. This would NOT be necessary, as each Model
could still be used independently. However, in order to support more sophisticated features, using the Table
class would be required.
Transactions using the library are possible by extracting the parameters and assembling the correct structure for the TransactWriteItems
and TransactGetItems
APIs, but it is a bit bulky. We could expose another class called Transactions
that would allow you to simply add an array of Model
s with the appropriate method call and parameters. The Transactions
class could parse the parameters and assemble them in the correct format.
const { Model, Transaction } = require('dynamodb-toolbox')
const MyModel = new Model( ‘MyModel’, { ... } )
const MyOtherModel = new Model( ‘MyOtherModel’, { ... } )
const AnotherModel = new Model( ‘AnotherModel’, { ... } )
let result = await Transaction(
[
MyModel.update(someItem1),
MyModel.update(someItem2),
MyOtherModel.update(someItem3),
MyOtherModel.delete(someItem4),
AnotherMode.put(someItem5)
],
{
ReturnConsumedCapacity: ‘TOTAL’
}
)
No need to specify the “write” or “get” transaction type since the library could do this automatically. It could also enforce the method types (delete
, update
, and put
for writes, and get
for get) and the maximum of 25 items per (or maybe it could auto split them 🤷♂️).
Also would need to allow for ConditionCheck
, so that might just need to be passed in the array.
Eg, 0 or "" or false
aren't set if just provided like
{ type: "whatever", default: <falsy-value> }
It looks like some of the tests are accidentally expecting this behavior, but I'll look more later.
Workaround for now is to just use a function:
{ type: "whatever", default: () => <falsy-value> }
Hello,
I recently started to use this and I noticed that currently using batchWrite I cannot do conditions on my put requests.
Example:
const DocumentClient = new DynamoDB.DocumentClient({
accessKeyId: 'hey',
secretAccessKey: 'hey',
endpoint: 'http://localhost:8000',
region: 'us-west-2'
})
const table = new Table({
name: 'testing123',
partitionKey: 'PK',
sortKey: 'SK',
DocumentClient
})
const TestEntity = new Entity({
name: 'Test',
attributes: {
PK: {
partitionKey: true,
type: 'string'
},
SK: {
sortKey: true,
type: 'string'
}
},
table
})
console.log(
JSON.stringify(
table.batchWriteParams([
TestEntity.putBatch(
{
PK: 'test',
SK: 'test'
},
{
conditions: [{ attr: 'PK', exists: false }]
}
)
])
)
)
Logs:
{
"RequestItems":{
"testing123":[
{
"PutRequest":{
"Item":{
"_ct":"2020-06-22T08:37:45.201Z",
"_md":"2020-06-22T08:37:45.201Z",
"entity":"Test",
"PK":"test",
"SK":"test"
}
}
}
]
}
}
Notice how there is no expression attached to this. I really need to have this feature as it is important I have it!
I'm happy to add automated versioned releases to GitHub and NPM via GitHub Actions + Semantic Release if you're interested. I've done this a few times on various projects. It's essentially Continuous Deployment for packages. 😄 You will need to start using https://www.conventionalcommits.org/ though.
Currently composite sortKey generation uses the #
symbol as a delimiter. This should be configurable.
Loving using the library so far Jeremy.
Can’t quite work out how to specify an IndexName
so I can query a GSI, what have I missed?
Firstly, thank you for this great library. I'm just getting into NoSQL database design and specifically trying to grasp the concept of a single table design. You probably have this on your roadmap, but I've noticed that you can't yet create an entity that references a partitionKey
with a custom GSI string name.
The following Entity setup throws an error Entity requires a partitionKey attribute
.
const MyTable = new Table({
name: "MyTable",
partitionKey: "pk",
sortKey: "sk",
indexes: {
GSI1: {partitionKey: "sk", sortKey: "data"},
},
DocumentClient,
});
const MyEntity = new Entity({
name: "MyEntity",
attributes: {
sk: {partitionKey: "GSI1"}, // This should work I think
data: {sortKey: "GSI1"},
},
table: MyTable,
});
It looks like the keys are correctly set on the Entity but there is a check which only checks the root track.keys.partitionKey
here:
https://github.com/jeremydaly/dynamodb-toolbox/blob/v0.2/lib/parseEntityAttributes.js#L29
The keys do exist in the child object under the custom "GSI1" key though track.keys.GSI1.partitionKey
.
As I said above, it may just be that this hasn't been implemented yet but just wanted to flag in case it was missed.
I tried to do
MyEntity.delete(
{
pk,
sk,
},
{ conditions: { attr: 'username', exists: true } },
)
but I'm getting ExpressionAttributeValues must not be empty
because the generated params are
{ TableName: '',
Key:
{ PK: '',
SK: '' },
ExpressionAttributeNames: { '#attr1': 'username' },
ExpressionAttributeValues: {},
ConditionExpression: 'attribute_exists(#attr1)' }
I'm not 100% sure my use case is valid DDB but this might be on the library ;)
Writing something like:
const params = MyEntity.putParams(network, {
conditions: [
{ attr: "PK", exists: false },
{ attr: "SK", exists: false }
]
});
Results in an ExpressionAttributeValues: {}
, which AWS rejects.
Workaround is to add:
delete params.ExpressionAttributeValues;
If you know they're empty.
If you try to query with a numeric sort key in a GSI, (like PK=x, { lte: 1234 }), you'll get <attr> must be a string
I've been doing a lot of background work on this library and I've found myself writing some scripts to deal with different conversions and tests. I'm thinking a CLI component might be handy to generate models, compile code, import models from NoSQL Workbench (or export them), etc.
Way down on the priority list, but adding it here so I don't forget.
First, thanks for the library, it makes it much easier to get started with DynamoDB.
There seem to be some problems with batch operations:
Batch get (https://github.com/jeremydaly/dynamodb-toolbox/blob/master/classes/Table.js#L786)
parseBatchGetResponse
call in the next
function is called with a nextResult
parameter which is a Promise, as it is not awaited on. This should probably be resolved before passing it to parseBatchGetResponse
.Batch write (https://github.com/jeremydaly/dynamodb-toolbox/blob/master/classes/Table.js#L959)
nextResult
should be awaited on.batchWrite
operation returns UnprocessedItems
instead of UnprocessedKeys
. (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#batchWrite-property)Expected Behavior: I can pass in an option to put()
for autoExecute
to override its behavior from the Entity.
Actual Behavior: I receive an error "Invalid put options: autoExecute".
Workaround: Turn off autoExecute
at the Entity property level before the put()
call, and set it back just afterward (see the example, below).
Sample code:
const updateNoAutoExecuteError = async () => {
const foo = { id: '123', name: 'Carla' };
const putParams = await Foos.put(foo, { autoExecute: false }); // ERROR - "Error: Invalid put options: autoExecute"
}
const updateNoAutoExecuteWorkaround = async () => {
const foo = { id: '123', name: 'Carla' };
Foos.autoExecute = false;
const putParams = await Foos.put(foo);
Foos.autoExecute = true;
console.log(`putParams: ${JSON.stringify(putParams)}`); // putParams: {"TableName":"test-foos", ..., "name":"Carla"}}
}
const FoosTable = new Table({
name: 'test-foos',
partitionKey: 'pk',
sortKey: 'sk',
DocumentClient,
});
const Foos = new Entity({
name: 'Foo',
table: FoosTable,
timestamps: true,
attributes: {
pk: { hidden: true, partitionKey: true, default: (data) => (`FOO#${data.id}`) },
sk: { hidden: true, sortKey: true, default: (data) => (`FOO#${data.id}`) },
id: { required: 'always' },
name: 'string',
},
});
DynamoDB doesn't have a date
type, but schemas could easily support it by requiring a native JavaScript Date
object as an input and converting it to the ISO 8601 format. When parsing raw Items, the library could convert the dates back to a JS Date object (given the fact that this is specified in the schema).
It could potentially support datetime
and date
since these are both valid ISO 8601 representations.
Wondering if we need other formats, like just time
?
Would a better name for the core classes be a Facet
instead of a Model
?
Model is very ORM'y , while Facet is more inline with NoSQL Workbench patterns.
Also, what's the status on the v0.2
branch (it looks like a great improvement).
I think this might just be a lack of my knowledge of DynamoDB again, but I thought that the following test shouldn't throw the error:
'pk' is required
I thought, that because the GSI partitionKey has been specified, that I can run a get query against it without having to include the main table's partitionKey. The error suggests that this is not the case. I'm a bit confused and I'm sure there is a simple answer to this which I'm struggling to find. Any pointer would be greatly appreciated.
const TestTable3 = new Table({
name: 'test-table',
partitionKey: 'pk',
sortKey: 'sk',
indexes: {
GSI1: { partitionKey: 'GSI1sk', sortKey: 'data' },
},
DocumentClient,
})
const TestEntity3 = new Entity({
name: 'TestEntity',
attributes: {
pk: { type: 'string', partitionKey: true },
sk: { type: 'string', partitionKey: 'GSI1', sortKey: true },
data: { type: 'string', sortKey: 'GSI1' },
},
table: TestTable3,
})
describe('get', () => {
it('gets the Key from GSI1 inputs (sync)', async () => {
const { TableName, Key } = TestEntity3.getParams({
sk: 'test-sk', // This is the GSI1 partitionKey I also tried the name `GSI1sk`
data: 'test-data',
})
expect(TableName).toBe('test-table')
expect(Key).toEqual({ sk: 'test-sk', data: 'test-data' })
})
})
Requesting support for querying by just partition key: #11. This is a valid use case that would be leveraged when you split up a given entity over multiple items in a given partition to alleviate concerns about the 400kb item size limit.
e.g. You have a class record, and are concerned about the combination of students/teacher/TAs/assignments/tests/labs relationship data exceeding the limit of 400kb for a single item.
You can split all this data out in to multiple items with the same partition key and different sort keys. Then querying by just the partition key you can get all information regarding a given class while alleviating any concern about exceeding the 400kb item size limit.
I'm implementing the northwind example in node using your library.
I'm trying to write the first query which is get employee by employee id (pk = employeeID
) with the following code:
EmployeeModel.get({
pk: 'employeeID#2'
})
but I get the following error:
'sk' is required`
Looking at the following test, it appears that querying by partition key only is not supported.
dynamodb-toolbox/__tests__/get.unit.test.js
Lines 45 to 47 in 92e3726
Is this correct? Or am I holding it wrong?
I've got an entity with the following attributes:
attributes: {
pk: {
type: 'string',
partitionKey: true,
},
sk: {
sortKey: true,
hidden: true,
},
username: ['sk', 1, { save: true }],
role: ['sk', 0],
},
According to the doc, I should get both username and role when querying this entity.
However, I only get the username (which is saved).
The quick fix is easy but I thought I'd report it so that you could fix it or update the doc.
Looks like this is in 0.1 and 0.2: https://github.com/jeremydaly/dynamodb-toolbox/blob/master/index.js#L531
Instead of throw err
, ought to be throw new Error(x)
so normal try/catch stacks can inspect it.
I'll shoot some PRs to fix in a bit if you're cool with it :)
This is just a note of something I found to be unexpected. For (good or bad) reasons I want the sk
to be the same as the pk
, but I cannot use the name of pk
in the input.
A simple table
const Drawer = new Table({
name: "Drawer",
partitionKey: "pk",
sortKey: "sk
})
and then we define an entity
const Pants = new Entity({
name: 'Pants',
attributes: {
id: { partitionKey: true},
sk: {hidden: true, sortKey: true, default: (data) => data.id};
},
table: Drawer
});
await Account.put({id: 5});
This is because data
does not contain id
, but only pk
:
data = {
sk: [Function: default],
_ct: [Function: default],
_md: [Function: default],
_et: 'Pants',
pk: '5',
const Pants = new Entity({
name: 'Pants',
attributes: {
id: { partitionKey: true},
sk: {hidden: true, sortKey: true, default: (data) => data.pk};
},
table: Drawer
});
await Account.put({id: 5});
I expect data
to contain the information as inputted i.e. in terms of the Entity and not the table 🤷♂️
Not sure whether more people would expect this, but I thought to submit an issue.
Unless I'm doing something wrong, Entity.get() returns something like
{
"Item": {
"uuid": "8f19ef46-c9c2-4c68-ad19-bd22e82a0eaf",
...
}
}
The keys under Item have been correctly parsed but this is not what I expected after having read the doc.
Should we change this behavior or the add it to the doc?
Only top level schemas are supported. By adding support for nested maps, we could enforce types and provide a cleaner way to update data maps. This should be optional, falling back to the current dot notation for unmapped structures.
const MyModel = new Model('MyModel',{
// Specify table name
table: 'my-dynamodb-table',
// Define partition and sort keys
partitionKey: 'pk',
sortKey: 'sk',
// Define schema
schema: {
pk: { type: 'string', alias: 'id' },
sk: { type: 'string', hidden: true },
someMap: {
type: 'map' ,
schema: { // $ shorthand
title: 'string', // support string-based type declarations
count: { type: 'number', default: 10 }, // support scalars with defaults and other options
someList: { type: 'list' },
someNestedMap: { // Supported nested maps with schemas
type: 'map',
schema: { ... }
}
}
}
}
})
This would enable support for schema validation as well as partial updates.
If an entity does not list an attribute that exists in the database, get
fails.
To make await entityName.get(item)
work, in the definition of the entity entityName
, it needs to list all attributes it may pull from the database for that particular item. Otherwise, it shows an alias of undefined
error.
This is counterproductive for a single-table NoSQL design as often times, there is data that is there in the database, may get pulled by DynamoDB get
, but the function doesn't need to be explicit about it.
If I understand correctly, the current method of specifying composite keys occurs with array attribute values. I saw this cause an issue in #32. I believe the solution is to migrate the api to something that looks like this:
Entity({
attributes: {
pk: {composite: ['STATIC_WORD', {attr: '_et'}, {attr: 'id'}], delimiter: '#'},
id: {default: () => nanoid()},
},
})
This would allow specifying multiple composites, and it would retain the space-saving features of the current implementation. Additionally, I think it's easier to parse. Does this break anything I'm not aware of?
it will be nice to switch to the modularized AWS SDK for JavaScript v3 (https://github.com/aws/aws-sdk-js-v3) since it reached gamma status. AWS Amplify is already using it.
dynamodb-toolbox doesn't require the full AWS SDK and will reduce the size of the bundle.
FWIW, it's a bit premature since AWS SDK for JavaScript v3 doesn't support DocumentClient yet.
So this is a big one that would likely require query support (see #5). However, defining methods like getUserById
, getOrdersByDate
, or setUser
seems to be a very common practice. These methods could be defined as higher level constructs that mapped to the underlying models. I would guess that a vast majority of get
access patterns are just mapped to a query with some conditions and filters, and set
s are a simple put
operation (maybe with a transaction).
Need to think about this more, but it could also potentially (at least for the gets
) help you keep track of your indexes and what they’re used for. Thoughts and ideas are welcome.
This works locally, but when I deploy to Lambda, the table.constructor.name
is empty, causing dynamodb-toolbox to throw https://github.com/jeremydaly/dynamodb-toolbox/blob/v0.2/classes/Entity.js#L46-L69.
I'm using serverless-webpack which may have something to do with it?
A few properties are auto-generated when saving data, e.g., created
, modified
, and entity
. However, while created
and modified
do not interfere with any subsequent update()
operation, the entity
property causes the update()
to fail with the message "Error: Field 'entity' does not have a mapping or alias".
A workaround is to explicitly delete myObj.entity
at some point in the chain.
Example Code:
const insertAndUpdate = async () => {
const foo = { id: 'abc', name: 'Alfred' };
await Foos.put(foo);
const { Item: fooFromDB } = await Foos.get({ id: foo.id });
fooFromDB.name = 'Bob'
await Foos.update(fooFromDB); // FAILS - "Error: Field 'entity' does not have a mapping or alias"
};
const FoosTable = new Table({
name: 'test-Foos',
partitionKey: 'pk',
sortKey: 'sk',
DocumentClient,
});
const Foos = new Entity({
name: 'Foo',
table: FoosTable,
timestamps: true,
attributes: {
pk: { hidden: true, partitionKey: true, default: (data) => (`FOO#${data.id}`) },
sk: { hidden: true, sortKey: true, default: (data) => (`FOO#${data.id}`) },
id: { required: 'always' },
name: 'string',
},
});
When using a function as a default, if another default relies on it, it stringifies the function instead of calling it. Kind of hard to explain, but here is an example.
attributes: {
id: {
partitionKey: true,
prefix: 'USER_ID#',
default: () => randomBytes(32).toString('hex'),
},
sk: {
sortKey: true,
prefix: 'USER_ID#',
default: (data) => data.pk,
}
}
The above schema results in the following entity.
{
pk: "USER_ID#f6591161cfba6267e0e59ca7405164e16733287870d01359ef089b696ae8d826"
sk: "USER_ID#() => crypto_1.randomBytes(32).toString('hex')"
}
I can easily generate the id before creation and avoid this problem. I just thought it'd be cool to use defaults.
making an issue in case you want to tie it to #40 :)
Someone mentioned this to me, but I’m not sure of the correct abstraction level. If the models and types can aid in building queries, then I think this would be useful, but I don’t want to cross into ORM territory.
There are a limited set of sortKey
operations, so some simple abstraction could work, but the library probably shouldn’t be parsing queries. There would likely need to be a different way to input it, and I don’t think chaining makes sense in this case (e.g. .field(‘somevalue‘).gt().value(someValue)
).
This needs more feedback.
So one of the interesting things I picked up in Rick Houlihan's talk at re:Invent (more details here), was the duplication of attributes in the SAME ITEM in order to make greater use of sparse indexes. Very, very, very cool stuff.
You could certainly use the default
mechanism to replicate data attributes, but this seems like it could be a handy shortcut.
// Define schema
schema: {
pk: { type: 'string', alias: 'id' },
sk: { type: 'string', hidden: true },
email: { type: 'string' },
someDupeField: { $ref: 'email' }
}
The $ref
(or something similar) would map the email
value to the someDupeField
attribute, always overwriting it whenever the data was updated in the main index.
Hi @jeremydaly it would be awesome for newcomers like me to have an example app that shows off how you should structure the code as a best practice. Nothing fancy but at least how you should load table/entities and do some basic operations.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.