Comments (5)
csakoda
commented on August 31, 2024
Found that switching to the bulk API in the lambda script was enough to fix it.
Do you folks have any estimations on performance of this sample? Is it meant to work at scale? Or just connect the dots.. ?
from amazon-elasticsearch-lambda-samples.
srisub-amzn
commented on August 31, 2024
Thanks for trying out the sample!
The code is meant to demonstrate how Lambda could be used for ES data ingestion. It is simplified for the purposes of clarity and is not tuned to work at scale.
from amazon-elasticsearch-lambda-samples.
csakoda
commented on August 31, 2024
Cool, I figured as much.
It would be super helpful to see a sample that ingested aggregated KPL records. I see the records come through as base64 encoded JSON, delimited by..some hex values? I can't figure out what from casually reading the KPL code.
Is there anything like that coming?
Thanks again!
from amazon-elasticsearch-lambda-samples.
srisub-amzn
commented on August 31, 2024
I realized that this thread was left hanging. Could you elaborate what you mean by aggregated KPL records?
from amazon-elasticsearch-lambda-samples.
deanjez
commented on August 31, 2024
I appreciate these code samples. I'm interested in a version of this lambda function that utilises the ES bulk API also. I assume what is referred to above is that the existing function, iterates through the collection of Kinesis records and makes a separate HTTP request to ES to index each document. This would be very expensive at scale regarding I/O on the ES cluster. Iterating through the Kinesis record collection and building an ES bulk API request within the Lambda function would be an improved alternative.
from amazon-elasticsearch-lambda-samples.
Related Issues (17)
- for .gz log files
- Access Denied Error
- To push .log.gz (Application ELB) Logs to ElasticSearch HOT 1
- allow passing values from variables defined in lambda function
- CloudTrial to ElasticSearch HOT 1
- Elasticsearch 6.x compatibility issue HOT 6
- Error in ElasticSearch is not mapped to errorCallback of NodeHttpClient.handleRequest HOT 1
- js folder exposed to anonymous users
- Suggested IAM Policy for Lambda functions is too broad.
- Async Client? HOT 2
- Java Example HOT 6
- Getting "Error: write after end" on every other log file that is ingested from s3 HOT 1
- Sign error HOT 6
- context.succeed would be called multiple times HOT 1
- Elastic search configuration policy during lambda execution HOT 1
- Unable to import module es HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-elasticsearch-lambda-samples.