Comments (10)
Hi Ulrik, thanks for the report!
On a scale of 1-10, how urgent is this? Unless it's very urgent, I can try take a look in the next day or two? Just juggling some high-priority tasks atm. If you need this ASAP I'll make sure to look into it today.
from faraday.
Actually, it's very urgent. I need to perform a migration in production and this is biting me. If you could have a look, that would be awesome.
from faraday.
Okay, could you get me some more details - incl. an example of the code you're running?
The unprocessed items returned after a throttled batch-write is in a mixed format
The :unprocessed
value isn't generally something you'd be touching when using Faraday - it's in the raw Java format documented here: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/model/BatchWriteItemResult.html#getUnprocessedItems()
The reason you wouldn't normally be using this value directly is because batch-write-item
already handles stitching together requests for you via the :span-reqs
option (this is described in the docstring / API docs).
So to clarify: are you using batch-write-item
's :span-reqs
option and it's malfunctioning somehow, or you're trying to do manual spanning? If you're doing manual spanning, could you explain why?
If you do need to do manual spanning, you can see the batch-write-item
and merge-more
source to see how it's done there.
The more clear info you can give me, the more likely I'll be able to help quickly.
Cheers!
from faraday.
I haven't used :span-reqs
. Let me try with that first.
I seem to have a hard time parsing the doc-strings into examples. :)
from faraday.
I haven't used :span-reqs. Let me try with that first. I seem to have a hard time parsing the doc-strings into examples. :)
Okay, great - that's probably what you want then.
If you previously had (batch-write-item my-creds my-req)
, now you'll want something like (batch-write-item my-creds my-req {:span-reqs {:max 100 :throttle-ms 10}})
.
This'll instruct the batch-write to stitch together up to 100 individual requests, pausing for 10msecs between each request. Note that an op like this can take quite some time to complete unless you have high throughput limits set.
from faraday.
I tried with a lower setting on :max
, 10, and then I still got some unprocessed items. When I increased it to 100, however, it all eventually went through even on a very small write provisioning.
So what do I do if I still get unprocessed items? It would be nice if there was a way to access lower-level utilities, like db-client
, if I briefly need to get under the hood and access the Java API. Or a way to convert the unprocessed map to something that batch-write
accepts. Or, better yet, unprocessed is automatically converted to the Clojure format.
from faraday.
It would be nice if there was a way to access lower-level utilities, like db-client, if I briefly need to get under the hood and access the Java API.
Nothing stops you from doing that: the entire, standard Java API is fully accessible and interoperable. db-client
, for example, just returns a standard AmazonDynamoDBClient object - you're free to bang on it with the standard Java API.
Likewise the :unprocessed-items
value is just a standard Java-API object, you could work with it directly (nothing stops you, it's not wrapped or anything) - but you will need to use the Java API which can be a real pita.
Or a way to convert the unprocessed map to something that batch-write accepts. Or, better yet, unprocessed is automatically converted to the Clojure format.
Sure, would be open to a PR for this. Haven't given it much thought, so can't recommend one approach over the other.
from faraday.
Sure, everything is available. I'm mainly trying to avoid that my code has to import all the Java classes and perform all the mapping as well. If it's possible to get a nice solution just by making some private functions public, I'll let you know.
from faraday.
I'm mainly trying to avoid that my code has to import all the Java classes and perform all the mapping as well.
There's two different things we're discussing here though:
- Ability to access the Java API.
- Mitigating the need to access the Java API by providing all the tools we might want in Clojure-space.
We already have 1. There's definitely still room for improvement on 2 (here for example, by modding either the :unprocessed-items
form and/or batch-write-item
to make it easier to do manual request spanning).
If it's possible to get a nice solution just by making some private functions public
Nothing obvious comes to mind in this case - I'd prefer to keep merge-more
private since it makes quite a few assumptions about the input/output form and those may change in future.
Your earlier suggestion sounds good to me: I would investigate tweaks to :unprocessed-items
and/or batch-write-item
to make them cooperate more easily in cases where folks don't want to use the automatic request spanning.
from faraday.
I just want to point out that I in fact did use request spanning and still got unprocessed items in some cases.
from faraday.
Related Issues (20)
- CI integration HOT 5
- Remove support for `:expected` HOT 3
- attribute_not_exists not respected consistently HOT 3
- Support for Streams HOT 8
- missing ScanIndexForward option with far/scan HOT 1
- Request: custom keywordization routine HOT 3
- Is the scan function safe for large tables? HOT 7
- Upgrade to 1.11.16+ of aws-sdk-java for task-level IAM role support in AWS ECS HOT 9
- Add support for DynamoDB Transactions HOT 6
- Upgrade to 1.11.459 of aws-sdk-java for management of On Demand tables
- Upload all doc versions to http://taoensso.github.io/faraday/{version}
- Porting to AWS SDK v2 HOT 16
- missing page-size/max-result-size option with query HOT 1
- Support empty strings HOT 4
- not listing tables HOT 14
- Setting `region` in client-opts causes ClassCastException
- String index names cause type error HOT 3
- Local docker example from readme only with `-p 8000:8000` arg HOT 1
- Possibly confusing dependencies found HOT 3
- FYI: have pushed some superficial changes HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faraday.