Comments (10)
Got it working with JestHttp. Forgot to change the dependency 🙈
from log4j2-elasticsearch.
I'm glad you got it working 👏
Watch out for the double $
for dynamic VirtualProperty
-ies. They need to use dynamic="true"
flag to resolve correctly:
<VirtualProperty name="client.ip" value="$${ctx:client.ip}" dynamic="true"/>
.. and ctx
doesn't work with AsyncLogger
- that's due to Log4j2 not copying over thread context vars in async mode.
Also, I highly recommend AHCHttp
. It's much more performant and has much lower footprint that JestHttp. It should work with netty-all
jar. AHC module has become the new "tip of the spear". It will be the main focus of further development and part of it's code will become the backbone of http layer in 2.0.
I hacked the HC example to use your configuration. After a few tweaks it worked like a charm (mappings are still missing, but I'm sure you'll figure it out)
Try
mvn clean install && java -jar -Delastic.apm.environment=apm-test -Delastic.apm.service_name=elasticsearch-ahc log4j2-elasticsearch-hc-springboot/target/log4j2-elasticsearch-hc-springboot-0.0.1-SNAPSHOT.jar
and then
curl -XPOST -H 'Content-Type: application/json' -d '{}' http://localhost:9200/log4j2-elasticsearch-ahc/_search | jq
on this branch
from log4j2-elasticsearch.
Hey Rafal,
ah, i thought the double $
is needed for escaping reasons. Got it from (https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#virtual-properties)
.. and ctx doesn't work with AsyncLogger - that's due to Log4j2 not copying over thread context vars in async mode.
If i understand the log4j2.x system property log4j2.isThreadContextMapInheritable
right, then the child thread inherit the Thread Context Map. This is explained here https://logging.apache.org/log4j/2.x/manual/thread-context.html#configuration. But maybe, i am wrong.
Also, I highly recommend AHCHttp. It's much more performant and has much lower footprint that JestHttp.
Thats why i'm tried AHCHttp first ;-).
from log4j2-elasticsearch.
It works as an escape char, yes. First $
is replaced somewhere around loading the xml file - Log4j2 goodness - only one remains and is VirtualProperty.dynamic=false
, it will be replaced while building the serialiser at this line in JacksonSerializer.
the child thread inherit the Thread Context Map
I never got ctx
working with StrSubstitutor. If values are copied, they're not copied where VirtualProperty
would expect it.
from log4j2-elasticsearch.
@thaarbach Is there anything else we can address regarding the original issue? Seems like the code - at least in this repository :) - works as advertised
from log4j2-elasticsearch.
@rfoltyns i'm using AsyncLoggers and set some informations e.g. the current user.id in the ThreadContext (in our case MDC because of using SLF4J) and they are indexed when log4j2.isThreadContextMapInheritable=true
. Mayby there is some magic done by SLF4J.
Is there anything else we can address regarding the original issue? Seems like the code - at least in this repository :) - works as advertised
Whould be nice, if ECSLayout also works with elasticsearch-ahc
from log4j2-elasticsearch.
Is the log4j2-elasticsearch-examples branch I mentioned above not working as you expect?
from log4j2-elasticsearch.
Is the log4j2-elasticsearch-examples branch I mentioned above not working as you expect?
Didn't try it yet. At the moment i'm fighting with ingest piplines, because they won't be executed on index requests.
Are ingest pipelines supported per request?
from log4j2-elasticsearch.
I never played around with these tbh. Elasticsearch docs mentions index.default_pipeline
setting. See what happens once you provide it in index-template/component-template-settings. With data stream and index template settings setup provided by this plugin, should work nicely..?
I'll look into Pipeline API and see if there's a possibility to properly support it in 1.7
from log4j2-elasticsearch.
Got it working. The trick is set the pipeline as index.final_pipeline
and force an reindex or rollover.
Old data can be updated with POST my_data_stream/_update_by_query?pipeline=my_pipeline
Now i'm be able to enrich the log entries with geo data and decode urls :-)
from log4j2-elasticsearch.
Related Issues (20)
- Supporting ecs-logging-java HOT 8
- Package naming conventions HOT 11
- How to pass the value from ThreadContext to ValueProperty? HOT 3
- Can not log to elastic using log4j2 <PatternLayout> HOT 11
- Support elasticsearch data_streams HOT 6
- Failed to Load StackTraceElementMixIn HOT 22
- InvalidTypeNameException on invalid mappingType in ES 5 HOT 8
- Adding custom properties, that change during processing HOT 4
- Log4j2 Zero-Day vulnerability (CVE-2021-44228) HOT 2
- Support OpenSearch 2.x and Elasticsearch 8.x HOT 9
- Unable to determine if index already exists HOT 1
- Is it possible to deseriailze JSON within the message into properties? HOT 2
- Run elastic search appender in specific environment HOT 2
- Logging-Cleaner ERROR Could not create plugin HOT 3
- How do I add mdc attributes? HOT 1
- Could not initialize ChronicleMapRetryFailoverPolicy HOT 2
- ServiceDiscovery ERROR HCServiceDiscovery: Unable to refresh addresses: Cannot invoke "org.appenders.log4j2.elasticsearch.hc.discovery.NodeInfo$PublishAddress.getPublishAddress()" HOT 1
- Using log4j2-elasticsearch-hc -> Does AsyncLogger still not support adding dynamic keyValue pairs in Jackson JSONLayout HOT 5
- IS JacksonModule is implemented in latest version? HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log4j2-elasticsearch.