Comments (6)
rfoltyns
commented on May 28, 2024
1
Your mappings are correct, but there are two issues in these stacktraces:
"reason":"Validation Failed: 1: index patterns are missing;"Lack of index pattern in log4jtemplate.json. You can fix it by addingtemplatefield to your JSON file:
{ "template": "log4j2*", mappings: ... }java.lang.ClassNotFoundException: com.lmax.disruptor.EventFactorycom.lmax:disruptor.jar was not provided on your classpath. You can fix it by adding this dependency to yourpom.xml(give it a go, it's worth it) or (not recommended in high throughput scenarios) switching to synchronous logger:
<Logger name="elasticsearch" level="info" additivity="false"> <AppenderRef ref="elasticsearchAsyncBatch" /> </Logger>
from log4j2-elasticsearch.
rfoltyns
commented on May 28, 2024
It looks like the index mappings are missing.
There's no timestamp field in serialized LogEvent - there's timeMillis instead. You can specify timeMillis mapping via IndexTemplate tag.
It seems that the usage example from the main page is a bit too simple. I'll update the documentation to try to prevent these kind of issues.
UPDATE: Since 2.11.0, log4j-core:org.apache.logging.log4j.core.jackson.LogEventJsonMixIn ignores LogEvent.getTimeMillis. Given that, you have a couple of options here:
- switch to log4j-core <= 2.10.0
- provide your own JsonLayout that doesn't ignore this getter
- specify
messageOnly="true"at appender level and start logging your own objects that you fully control (see user-provided format example)
In the meantime, I'll start work on a workaround for this.
from log4j2-elasticsearch.
kajahno
commented on May 28, 2024
In my case something I can notice is that Kibana won't recognize the field timeMillis as a date field, because in the default template this field is coming as a long (see this one):
{
"mapping": {
"index": {
"properties": {
[...]
},
"timeMillis": {
"type": "long"
}
[...]
My very basic configuration in log4j2.xml is the following:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xml>
<configuration>
<appenders>
<Elasticsearch name="elasticsearchAsyncBatch">
<RollingIndexName indexName="log4j2" pattern="yyyy-MM-dd" />
<AsyncBatchDelivery>
<JestHttp serverUris="http://[HIDING MY SERVER IP]:9200" />
</AsyncBatchDelivery>
</Elasticsearch>
</appenders>
<loggers>
<root level="debug">
<appender-ref ref="elasticsearchAsyncBatch" />
</root>
</loggers>
</configuration>
And I'm testing it with the following very basic Java code (I don't really know almost anything about coding in Java):
package org.elasticsearch.elastictest;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
/**
* Hello world!
*
*/
public class App
{
private static Logger logger = LogManager.getLogger(App.class);
public static void main( String[] args ) throws InterruptedException
{
System.out.println("Before");
logger.info("Some informative log line. ZzZzZ");
logger.warn("Now adding some warnings");
logger.error("Exception",new Exception("Unexpected DB exception. Some data missing in the database. Tell somebody and go to the pub in the meantime"));
System.out.println("After");
Thread.sleep(2000);
System.exit(0);
}
}
Temporary Solution
So, one way I managed to fix this is to create the index in elasticsearch before sending the log data making sure the field timeMillis is a date (with the command curl -H 'Content-Type: application/json' -XPUT http://localhost:9200/log4j2-2018-08-14?pretty -d @mymapping.json ). The content of "mymapping.json" is the following:
log4jtemplate.json.txt
Something I'd love is if by default the library would send this field as a date rather than a long, so Kibana is able to interpret it as a timestamp (without doing much voodoo).
Saying that, I tried to change my log4j2 configuration so it will load the template file in the appender itself, which looks very very similar to the example provided here:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xml>
<configuration>
<appenders>
<Elasticsearch name="elasticsearchAsyncBatch">
<RollingIndexName indexName="log4j2" pattern="yyyy-MM-dd" />
<AsyncBatchDelivery>
<IndexTemplate name="log4j2_template" path="classpath:log4jtemplate.json" />
<JestHttp serverUris="http://[HIDING MY SERVER IP]:9200" />
</AsyncBatchDelivery>
</Elasticsearch>
<Async name="asyncLogger">
<AppenderRef ref="elasticsearchAsyncBatch" />
</Async>
</appenders>
<loggers>
<AsyncLogger name="elasticsearch" level="info" additivity="false">
<AppenderRef ref="asyncLogger" />
</AsyncLogger>
<Root level="info" />
</loggers>
</configuration>
But the only thing I get back is:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2018-08-14 17:52:42,349 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.AsyncBatchDelivery for element AsyncBatchDelivery org.apache.logging.log4j.core.config.ConfigurationException: IndexTemplate not added: {"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: index patterns are missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: index patterns are missing;"}
at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory.execute(JestHttpObjectFactory.java:154)
at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery.<init>(AsyncBatchDelivery.java:62)
at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery$Builder.build(AsyncBatchDelivery.java:147)
at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery$Builder.build(AsyncBatchDelivery.java:100)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:890)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:890)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:513)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:237)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:249)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
at org.elasticsearch.elastictest.App.<clinit>(App.java:15)
2018-08-14 17:52:42,356 main ERROR No BatchDelivery method provided for ElasticSearch appender: batchDelivery
2018-08-14 17:52:42,356 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.ElasticsearchAppender for element Elasticsearch org.apache.logging.log4j.core.config.ConfigurationException: Arguments given for element Elasticsearch are invalid
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:203)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:890)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:513)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:237)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:249)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
at org.elasticsearch.elastictest.App.<clinit>(App.java:15)
2018-08-14 17:52:42,370 main ERROR Null object returned for Elasticsearch in appenders.
2018-08-14 17:52:42,378 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.async.AsyncLoggerConfig for element AsyncLogger: java.lang.NoClassDefFoundError: com/lmax/disruptor/EventFactory java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:890)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:513)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:237)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:249)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
at org.elasticsearch.elastictest.App.<clinit>(App.java:15)
Caused by: java.lang.NoClassDefFoundError: com/lmax/disruptor/EventFactory
at org.apache.logging.log4j.core.config.AbstractConfiguration.getAsyncLoggerConfigDelegate(AbstractConfiguration.java:202)
at org.apache.logging.log4j.core.async.AsyncLoggerConfig.<init>(AsyncLoggerConfig.java:82)
at org.apache.logging.log4j.core.async.AsyncLoggerConfig.createLogger(AsyncLoggerConfig.java:189)
... 20 more
Caused by: java.lang.ClassNotFoundException: com.lmax.disruptor.EventFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 23 more
2018-08-14 17:52:42,387 main ERROR Null object returned for AsyncLogger in loggers.
2018-08-14 17:52:42,388 main ERROR No appender named elasticsearchAsyncBatch was configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No appenders are available for AsyncAppender asyncLogger
at org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:120)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:265)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
at org.elasticsearch.elastictest.App.<clinit>(App.java:15)
Again, maybe I'm doing something bad (my knowledge of Java is pretty bad), so it'll be really nice if you could point me in the right direction for this one.
In any case I'd appreciate any sort of feedback.
Cheers!
from log4j2-elasticsearch.
kajahno
commented on May 28, 2024
Cheers @rfoltyns , works fine now!
from log4j2-elasticsearch.
rfoltyns
commented on May 28, 2024
@ZulfuqarAliyev @kajahno Log4j2 version compatibility issues were addressed in release 1.3.0. JacksonJsonLayout uses a set of patched MixIns, so timeMillis should now be serialized OOTB.
Could you retest, please?
from log4j2-elasticsearch.
rfoltyns
commented on May 28, 2024
Fixed in 1.3.0
from log4j2-elasticsearch.
Related Issues (20)
- Supporting ecs-logging-java HOT 8
- Package naming conventions HOT 11
- How to pass the value from ThreadContext to ValueProperty? HOT 3
- Can not log to elastic using log4j2 <PatternLayout> HOT 11
- Support elasticsearch data_streams HOT 6
- Failed to Load StackTraceElementMixIn HOT 22
- InvalidTypeNameException on invalid mappingType in ES 5 HOT 8
- Adding custom properties, that change during processing HOT 4
- Log4j2 Zero-Day vulnerability (CVE-2021-44228) HOT 2
- Support OpenSearch 2.x and Elasticsearch 8.x HOT 9
- Unable to determine if index already exists HOT 1
- Support for ECSLayout for elasticsearch-ahc and / or elaticsearch-jest in combination with data streams HOT 10
- Is it possible to deseriailze JSON within the message into properties? HOT 2
- Run elastic search appender in specific environment HOT 2
- Logging-Cleaner ERROR Could not create plugin HOT 3
- How do I add mdc attributes? HOT 1
- Could not initialize ChronicleMapRetryFailoverPolicy HOT 2
- ServiceDiscovery ERROR HCServiceDiscovery: Unable to refresh addresses: Cannot invoke "org.appenders.log4j2.elasticsearch.hc.discovery.NodeInfo$PublishAddress.getPublishAddress()" HOT 1
- Using log4j2-elasticsearch-hc -> Does AsyncLogger still not support adding dynamic keyValue pairs in Jackson JSONLayout HOT 5
- IS JacksonModule is implemented in latest version? HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log4j2-elasticsearch.