fundingcircle / clj-loga Goto Github PK
View Code? Open in Web Editor NEWCustom log formatting
License: BSD 3-Clause "New" or "Revised" License
Custom log formatting
License: BSD 3-Clause "New" or "Revised" License
Exception:
com.fasterxml.jackson.core.JsonGenerationException: Cannot JSON encode object of class: .....
Some objects can't be serialised to JSON and the library should have fallback functionality
cheshire.generate$generate.invokeStatic(generate.clj:147)
cheshire.generate$generate.invoke(generate.clj:111)
cheshire.generate$generate.invokeStatic(generate.clj:119)
cheshire.generate$generate.invoke(generate.clj:111)
cheshire.generate$generate.invokeStatic(generate.clj:119)
cheshire.generate$generate.invoke(generate.clj:111)
cheshire.core$generate_string.invokeStatic(core.clj:29)
cheshire.core$generate_string.invoke(core.clj:13)
cheshire.core$generate_string.invokeStatic(core.clj:19)
cheshire.core$generate_string.invoke(core.clj:13)
clj_loga.core$output_fn.invokeStatic(core.clj:105)
clj_loga.core$output_fn.invoke(core.clj:102)
clj_loga.core$output_fn.invokeStatic(core.clj:103)
clj_loga.core$output_fn.invoke(core.clj:102)
taoensso.timbre.appenders.core$println_appender$fn__1670.invoke(core.clj:101)
taoensso.timbre$log1_fn$fn__1814.invoke(timbre.clj:398)
Exception is coming from cheshire
Sometimes logs are a source of sensitive information leaks. In an event-driven architecture, it's common to log processed events messages that usually are represented as maps in Clojure.
Maybe we could add an option to whitelist which fields contain sensitive data and obfuscate them in production logs. So the production log output for a message with a whitelisted :c field would be:
;; {"timestamp":"2015-12-31T11:19:28.146Z","level":"INFO","message":"started: processing message {:a 1 :b 2 :c XXXX :d 4}","namespace":"clj-loga.core","tag":"some-tag"}
๐ This project simultaneously claims to use both an Eclipse and BSD 3-clause license. It's unclear which license applies to the project. We need to remove the Eclipse license immediately and bump the project version.
The readme says:
#export env variable to apply formatting
ENABLE_LOGA=true
But that environment variable is never used. What's up?
Might be worth looking at the RobertHooke library for adding log-wrapper
style aspect oriented logging to functions. It would make it possible to define all the logging in a separate namespace (logging.clj
?), without cluttering up the business logic.
So it could look something like this:
;;; handlers.clj
(defn my-handler-fn
[& args]
"Do all the things...")
;;; logging.clj
(add-hook #'loga/log-wrapper #'handlers/my-handler-fn)
If you wanted to get really fancy, you could add metadata to the handler functions that would allow you to specify things like tags, custom messages, etc.
(defn my-handler-fn
^{:loga/tags [:foo :bar :baz]}
[& args]
"Do all the things...")
This would generally help when scanning the logs for problems.
๐๐๐ Hello from the Janus "non-functional" team! ๐๐๐
We're a new team set up to tackle cross-cutting concerns such as logging, monitoring and alerting, and end-to-end testing. We'll be building tools to help make these things easier, but it's not just about code: we'll also be helping teams adopt consistent approaches across the board.
And we'll sometimes create issues on your repos, like this one. We're not just dropping work in your lap, though. Give any one of us a shout and we'll pair with you on implementing whatever changes are required.
First up is logging. We now have a cool new library, clj-loga, written by @apmaros and @timstott, that makes it easy for you to use JSON logging in your apps. This means that logs in Kibana will start to become a bit more meaningful - and searchable - like this:
This is how @apmaros describes clj-loga
:
We have worked on improving quality and usability of searching and filtering logs. Its result is library
clj-loga
which help us to log easier and filter faster by:
- JSON format - allows easily parse and add new tags
- New tag - namespace and more to come
- User can set a 'tag' - to filter it specifically instead of regexing it from message
- Wrap number of functions (with a common purpose) with loggs - This is useful for operations such as 'Processing xyz kafka message'. Cleaner functions.
- Removing all noise from the log @message
- Support for stacktrace - not encouraged, but stacktrace is logged as a single log event.
To use clj-loga
, you will need to be using timbre
>= 4.1.1. There is also a small amount of setup code you'll need in your app, described in the clj-loga
readme. If you'd like to pair on setting it up in one of your apps, grab either @apmaros or @rsslldnphy, who will be happy to work with you on it. Any questions let us know!
I think the common case for an app that includes clj-loga
in it's dependencies will be to want it to be enabled ;-). To make the common case the easiest, should we default ENABLE_LOGA
to true
, or swap it out for an env var called LOGA_DISABLED
?
We are lacking the ability to set the log level via env when running the application.
Timbre only makes use of the TIMBRE_LEVEL
env at compile time.
Perhaps we could introduce the following env:
LOGA_LEVEL=:debug
๐ก should levels be specified as strings debug
or keyword :debug
?
With the newest version of clj-loga (0.5.0) we seem to lose the exception's original message, example:
(timbre/error (Exception. "BOOM") "Something bad happened")
;=>
{"timestamp":"2016-04-22T13:21:20.580Z","level":"ERROR","message":"Something bad happened","namespace":"gocardless-reconciler.core","stacktrace":"gocardless_reconciler.core$eval25772$fn__25773.invoke(core.clj:29)\n...
(Result clipped)
In addition if the type of the exception is a clojure.lang.ExceptionInfo
we lose the map that is stored in the object, example:
(timbre/error (ex-info "BOOM" {:foo :bar}) "Something bad happened")
;=>
{"timestamp":"2016-04-22T13:23:55.040Z","level":"ERROR","message":"Something bad happened","namespace":"gocardless-reconciler.core","stacktrace":"clojure.core$ex_info.invokeStatic(c...
(Result clipped)
The use of environ's env
here: https://github.com/FundingCircle/clj-loga/blob/master/src/clj_loga/core.clj#L86 will break in some circumstances (if I remember correctly, either when the code is compiled into a jar, or when AOT compilation is used).
(def ^:private loga-enabled?
(= (env :enable-loga) "true"))
This is because the def
is calculated at compile-time rather than run-time. And at compile time, the environment variables won't be set.
When running with leiningen (i.e. in development), the environment variables will be set at compile time, so the problem won't manifest until we push it to an environment.
Easy fix is to make this a function rather than a def
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.