boxfuse / cloudwatchlogs-java-appender Goto Github PK
View Code? Open in Web Editor NEWJava log appender for AWS CloudWatch Logs when running within Boxfuse instances
License: Apache License 2.0
Java log appender for AWS CloudWatch Logs when running within Boxfuse instances
License: Apache License 2.0
To auto-recover from errors such as
com.amazonaws.services.logs.model.DataAlreadyAcceptedException: The given batch of log events has already been accepted. The next batch can be sent with sequenceToken: 49576308050167475836136287314051956388877295249203948850 (Service: AWSLogs; Status Code: 400; Error Code: DataAlreadyAcceptedException; Request ID: 0394f028-9c6d-11e7-818f-cdfcfc027d79)
Right now, CloudwatchLogsLog4J2Appender
doesn't accept AWS region from configuration, even though CloudwatchLogsConfig
supports it, and it is referenced correctly in CloudwatchLogsLogEventPutter
.
Hi boxfuse team -
I tried creating a generic version of cloudwatch logs appender using boxfuse's appender as a starting point. I faced several issues during the process - one of them being that - some of the logs are dropped off and are not fed to CloudWatch. This is because the while loop in the run() method of the CloudwatchLogsLogEventPutter.java is polling for one event while there could be 100's of events in the BlockingQueue and it is very much possible that the program execution stops before the while loop can run as many times as there are log events in the queue. To fix this, I had modified the design to use BlockingQueue.drainTo(collection)
approach and push all logs to cloudwatch at one go. Please see https://stackoverflow.com/a/48230308/829542 for implementation info. Hope this helps in enhancing the appender on your end for boxfuse environments. Thanks.
This appender looks great but it's unfortunate that the log group appears to be mostly hard-coded.
We had a hard time last week to understand why our system went down regularly. But finally we managed by some black box testing to find at least which component seems to be responsible for our frequent downtimes.
It turns out that the log appender seems not to handle our amount of logs (>=20logs/sec) we are sending to it. When we send the logs, the app is not responding anymore and thus boxfuse eventually redeploys it.
We still wonder how this can be the case as the appender should be able to handle much more throughput. We are using the latest version 1.1.3.33, Unfortunately, we can't tell more about the exact error as we won't see anything in the logs nor can't we access the application. What we can tell from our debugging with New Relic and jxm though, is that there seems every thing alright with the memory, threads and open connections on our server.
We found that the logappender is to blame for our downtime when we reduced the functionality of some of our components to only perform some logging and nothing else. When we finally also disabled that, our system became stable again.
To be precisely, I can change the behavior of our backend from a mostly stable to an unstable system by changing:
<!-- boxfuse aws -->
<appender name="Boxfuse-CloudwatchLogs" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender"/>
<root level="INFO">
<appender-ref ref="Boxfuse-CloudwatchLogs" />
</root>
to
<!-- boxfuse aws -->
<appender name="Boxfuse-CloudwatchLogs" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender"/>
<root level="DEBUG">
<appender-ref ref="Boxfuse-CloudwatchLogs" />
</root>
For example, with Log4j2 ver. 2.6.2 (which we were using in production), appender failed on first logging statement with Exception in thread "main" java.lang.NoSuchMethodError: org.apache.logging.log4j.core.LogEvent.getContextData()Lorg/apache/logging/log4j/util/ReadOnlyStringMap;
- which is understandable, as it was only introduced in ver. 2.7, as far as I can see.
I upgrade to 1.1.8.60 to get a hold of the custom MDC fields (thank you by the way)
However since, my messages in cloud watch are hard to read because they now contain all the meta data + the actual log message.
Before (version 1.1.5.46):
{
"image": "[imagename]",
"instance": "i-xxxxxxxx",
"level": "WARN",
"logger": "[logger]",
"message": "[actual log message",
"thread": "[threadid]"
}
Nice and clean...
After:
{
"instance": "i-xxxxx",
"image": "[image]",
"level": "INFO",
"message": "2018-02-01 18:07:27.579 boxfuse/xxx [application] {"image":"[boxfuse image]","instance":"i-xxxxxx","level":"INFO","logger":"[logger]","message":"[THE LOG MESSAGE"}"
}
Logback configuration is as follows:
<appender name="cloud-watch" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender">
<config>
<stdoutFallback>true</stdoutFallback>
<customMdcKey>my-custom-mdc-key</customMdcKey>
</config>
</appender>
Please publish these appenders to Maven Central, so that the community can benefit from them without depending on your Maven repository.
@axelfontaine
Would be it possible to configure the appender in logback.xml so that it will include a named MDC attribute (that is outside of the list of pre-determined MDC attributes)
<appender name="cloud-watch" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender">
<mdc key="my-mdc-key"/>
</appender>
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.