Comments (16)
Since 1.3, JacksonJsonLayout
uses LogEventJacksonJsonMixIn (since 1.5 it's ExtendedLogEventJacksonJsonMixIn) to control the shape and contents of serialized org.apache.logging.log4j.core.impl.Log4jLogEvent
.
By default, date is serialized to timeMillis
property which is of type long, but it can be changed if you create your own, custom mixin class:
package foo.bar;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import org.appenders.log4j2.elasticsearch.json.jackson.ExtendedLogEventJacksonJsonMixIn;
public abstract class CustomLogEventMixIn extends ExtendedLogEventJacksonJsonMixIn {
private static final long serialVersionUID = 1L;
@JsonProperty
@JsonSerialize(using = CustomDateSerializer.class)
@Override
public abstract long getTimeMillis();
}
and configure JacksonJsonLayout
to replace the default JacksonMixIn
config for org.apache.logging.log4j.core.LogEvent
in log4j2.xml
:
<Elasticsearch name="APPENDER_NAME">
(...)
<JacksonJsonLayout>
<!-- let's override LogEvent mixin -->
<JacksonMixIn mixInClass="foo.bar.CustomLogEventMixIn"
targetClass="org.apache.logging.log4j.core.LogEvent"/>
</JacksonJsonLayout>
(...)
</Elasticsearch>
Obviously, if you change millis to formatted date, timeMillis
property doesn't make much sense any more, so you can also rename it with @JsonProperty("myNewFormattedDate")
..
from log4j2-elasticsearch.
it worked with
package com.example.log4j2demo;
import java.io.IOException;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.SerializerProvider;
public class CustomDateSerializer extends JsonSerializer {
@Override
public void serialize(Object value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
gen.writeString(Instant.ofEpochMilli((Long) value).atOffset(ZoneOffset.UTC).format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
}
}
thanks
from log4j2-elasticsearch.
@yunleiz Did the solution described above work for you?
from log4j2-elasticsearch.
can you give an example of using time stamp yyyy-mm-dd and time
from log4j2-elasticsearch.
@yassermog you can easily produce it using config above, @JsonProperty(name="date")
, following serializer:
public class CustomDateSerializer extends JsonSerializer {
@Override
public void serialize(Object value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
gen.writeString(Instant.ofEpochMilli((Long) value).atOffset(ZoneOffset.UTC).format(DateTimeFormatter.ISO_LOCAL_DATE));
}
}
and mappings if needed:
"mappings": {
"@date": {
"type": "date",
"format": "strict_date"
}
}
}
from log4j2-elasticsearch.
package com.example.log4j2demo;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.jackson.LogEventJacksonJsonMixIn;
import org.apache.logging.log4j.core.time.Instant;
@JsonSerialize(as = LogEvent.class)
public abstract class CustomLogEventMixIn extends LogEventJacksonJsonMixIn implements LogEvent {
private static final long serialVersionUID = 1L;
@JsonProperty("@timestamp")
@JsonSerialize(using = CustomDateSerializer.class)
@JsonIgnore(false)
@Override
public abstract Instant getInstant();
}
package com.example.log4j2demo;
import java.io.IOException;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.SerializerProvider;
public class CustomDateSerializer extends JsonSerializer {
@Override
public void serialize(Object value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
gen.writeString(Instant.ofEpochMilli((Long) value).atOffset(ZoneOffset.UTC).format(DateTimeFormatter.ISO_LOCAL_DATE));
}
}
I got this error
I got this error
2019-08-29 15:15:41,983 main ERROR Cannot write item source: class org.apache.logging.log4j.core.time.MutableInstant cannot be cast to class java.lang.Long (org.apache.logging.log4j.core.time.MutableInstant is in unnamed module of loader 'app'; java.lang.Long is in module java.base of loader 'bootstrap') (through reference chain: org.apache.logging.log4j.core.impl.Log4jLogEvent["@timestamp"]) 2019-08-29 15:15:41,983 Log4j2-TF-7-AsyncLoggerConfig-4 ERROR Cannot write item source: class org.apache.logging.log4j.core.time.MutableInstant cannot be cast to class java.lang.Long (org.apache.logging.log4j.core.time.MutableInstant is in unnamed module of loader 'app'; java.lang.Long is in module java.base of loader 'bootstrap') (through reference chain: org.apache.logging.log4j.core.impl.Log4jLogEvent["@timestamp"]) 2019-08-29 15:15:41,985 Log4j2-TF-7-AsyncLoggerConfig-4 ERROR An exception occurred processing Appender elasticsearchAsyncBatch java.lang.NullPointerException at org.appenders.log4j2.elasticsearch.jest.JestBulkOperations.createBatchItem(JestBulkOperations.java:59) at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery.add(AsyncBatchDelivery.java:76)
from log4j2-elasticsearch.
if I change to getTimeMillis
@JsonProperty("@timestamp")
@JsonSerialize(using = CustomDateSerializer.class)
@JsonIgnore(false)
@Override
public abstract long getTimeMillis();
I got this error
2019-08-29 15:34:43,924 I/O dispatcher 1 WARN One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-08-29 15:34:43,925 I/O dispatcher 1 WARN Batch of 33 items failed. Redirecting to org.appenders.log4j2.elasticsearch.NoopFailoverPolicy
2019-08-29 15:34:44,761 I/O dispatcher 1 WARN One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-08-29 15:34:44,762 I/O dispatcher 1 WARN Batch of 5 items failed. Redirecting to org.appenders.log4j2.elasticsearch.NoopFailoverPolicy
and the log is not sent
from log4j2-elasticsearch.
@yassermog In case of getTimeMillis
(as suggested) - did you check the logs from your ES cluster?
Make sure that you got your mappings right. ES may not be able to parse the document without them.
In case of MutableInstant, class cast issues come from the custom serializer I've provided - notice Instant.ofEpochMilli((Long) value)
in it's code. It's meant to work with long (and timeMillis
value). Feel free to change it to work with MutableInstant as you expect.
from log4j2-elasticsearch.
@yassermog Glad to help 👍
I could consider changing the timeMillis
to @timestamp
by default in future releases, but I'd rather discuss it more broadly as it'd could be a breaking change for current users.
Current implementation already allows full flexibility in this area (Instant vs long, local date vs date time, etc.), so I would rather not rush it and explore the options in 1.5/2.0 release (~next year).
from log4j2-elasticsearch.
@rfoltyns I looked around a little bit, but couldn't find how to use @timestamp
instead of timeMillis
. Could you pls help ?
Also I want to prefix all my batches with { "create" : { "_index" : "some-index-name" } }
is that possible ?
Thanks in advance.
from log4j2-elasticsearch.
@gvrayden Overriding timeMillis
is described in the comment above. Just use @JsonProperty("@timestamp")
to get the desired field name.
As for prefixing batches with { "create" : { "_index" : "some-index-name" } }
, there's no support for that. Index is created automatically if it doesn't exist with name provided by IndexName
or RollingIndexName
. I'm not planning to add anything to _bulk
request payload it that's what you're asking.
But..
It will be possible to define custom requests on startup since 1.5 - you'll be able to do literally any administrative/maintenance work you want in the setup phase. I'll push this feature in the next few weeks, but I don't know when it will be released (possibly end of the year).
from log4j2-elasticsearch.
@rfoltyns Thanks, I will give it a try to change the timestamp field.
from log4j2-elasticsearch.
Hi, I used "JacksonMixIn" tag, it works for timestamp, but when defined some VirtualProperty(s) alongside it, these properties do not appear in the output. Does anyone have any idea? we use version 1.5.2
from log4j2-elasticsearch.
Hi @EhsanMo. By design, JacksonMixIn
applies only to org.apache.logging.log4j.core.LogEvent
.
VirtualProperty
(-ies) is just an overlay on top of Log4j2 LogEvent class and a it's a Jackson virtualProperties
property itself (see virtualProperties definition
). This overlay allows you to define properties on your own, outside of JacksonAnnotationIntrospector
scope. By default, in this project, "power" is delegated to Log4j2 Lookups, so customisations toVirtualProperty
are available only with :
- custom Log4j2 lookup like this
- custom
ValueResolver
implementation to replace Log4j2Lookups - custom
VirtualPropertyFilter
implementations (not relevant here)
I'm not planning any work in this scope, but I'm open for PRs.
from log4j2-elasticsearch.
Thanks, maybe I could not explain the problem properly. These two does not work correctly when using in the same context, for example:
<JacksonJsonLayout>
<VirtualProperty name="hostname" value="$${sys:hostname:-undefined}" />
<JacksonMixIn mixInClass="xyz.logging.CustomLogEventMixIn" targetClass="org.apache.logging.log4j.core.LogEvent"/>
</JacksonJsonLayout>
Only CustomLogEventMixIn works and there's no virtual property in logs. Am I using it correctly?
from log4j2-elasticsearch.
@EhsanMo I just tested it with log4j2-elasticsearch-examples. Works perfectly fine.
Are you extending the org.appenders.log4j2.elasticsearch.json.jackson.ExtendedLogEventJacksonJsonMixIn
class? virtualProperties
definition was moved there from LogEventJacksonJsonMixIn
in 1.5.
from log4j2-elasticsearch.
Related Issues (20)
- Supporting ecs-logging-java HOT 8
- Package naming conventions HOT 11
- How to pass the value from ThreadContext to ValueProperty? HOT 3
- Can not log to elastic using log4j2 <PatternLayout> HOT 11
- Support elasticsearch data_streams HOT 6
- Failed to Load StackTraceElementMixIn HOT 22
- InvalidTypeNameException on invalid mappingType in ES 5 HOT 8
- Adding custom properties, that change during processing HOT 4
- Log4j2 Zero-Day vulnerability (CVE-2021-44228) HOT 2
- Support OpenSearch 2.x and Elasticsearch 8.x HOT 9
- Unable to determine if index already exists HOT 1
- Support for ECSLayout for elasticsearch-ahc and / or elaticsearch-jest in combination with data streams HOT 10
- Is it possible to deseriailze JSON within the message into properties? HOT 2
- Run elastic search appender in specific environment HOT 2
- Logging-Cleaner ERROR Could not create plugin HOT 3
- How do I add mdc attributes? HOT 1
- Could not initialize ChronicleMapRetryFailoverPolicy HOT 2
- ServiceDiscovery ERROR HCServiceDiscovery: Unable to refresh addresses: Cannot invoke "org.appenders.log4j2.elasticsearch.hc.discovery.NodeInfo$PublishAddress.getPublishAddress()" HOT 1
- Using log4j2-elasticsearch-hc -> Does AsyncLogger still not support adding dynamic keyValue pairs in Jackson JSONLayout HOT 5
- IS JacksonModule is implemented in latest version? HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log4j2-elasticsearch.