fundingcircle / clj-loga Goto Github PK

Might be worth looking at the RobertHooke library for adding log-wrapper style aspect oriented logging to functions. It would make it possible to define all the logging in a separate namespace (logging.clj?), without cluttering up the business logic.

So it could look something like this:

;;; handlers.clj
(defn my-handler-fn
  [& args]
  "Do all the things...")

;;; logging.clj
(add-hook #'loga/log-wrapper #'handlers/my-handler-fn)

If you wanted to get really fancy, you could add metadata to the handler functions that would allow you to specify things like tags, custom messages, etc.

(defn my-handler-fn
  ^{:loga/tags [:foo :bar :baz]}
  [& args]
  "Do all the things...")

Sometimes logs are a source of sensitive information leaks. In an event-driven architecture, it's common to log processed events messages that usually are represented as maps in Clojure.

Maybe we could add an option to whitelist which fields contain sensitive data and obfuscate them in production logs. So the production log output for a message with a whitelisted :c field would be:

;; {"timestamp":"2015-12-31T11:19:28.146Z","level":"INFO","message":"started: processing message {:a 1 :b 2 :c XXXX :d 4}","namespace":"clj-loga.core","tag":"some-tag"}

💃💃💃 Hello from the Janus "non-functional" team! 💃💃💃

We're a new team set up to tackle cross-cutting concerns such as logging, monitoring and alerting, and end-to-end testing. We'll be building tools to help make these things easier, but it's not just about code: we'll also be helping teams adopt consistent approaches across the board.

And we'll sometimes create issues on your repos, like this one. We're not just dropping work in your lap, though. Give any one of us a shout and we'll pair with you on implementing whatever changes are required.

First up is logging. We now have a cool new library, clj-loga, written by @apmaros and @timstott, that makes it easy for you to use JSON logging in your apps. This means that logs in Kibana will start to become a bit more meaningful - and searchable - like this:

This is how @apmaros describes clj-loga:

We have worked on improving quality and usability of searching and filtering logs. Its result is library clj-loga which help us to log easier and filter faster by:

• JSON format - allows easily parse and add new tags
• New tag - namespace and more to come
• User can set a 'tag' - to filter it specifically instead of regexing it from message
• Wrap number of functions (with a common purpose) with loggs - This is useful for operations such as 'Processing xyz kafka message'. Cleaner functions.
• Removing all noise from the log @message
• Support for stacktrace - not encouraged, but stacktrace is logged as a single log event.

What you need to do

To use clj-loga, you will need to be using timbre >= 4.1.1. There is also a small amount of setup code you'll need in your app, described in the clj-loga readme. If you'd like to pair on setting it up in one of your apps, grab either @apmaros or @rsslldnphy, who will be happy to work with you on it. Any questions let us know!

With the newest version of clj-loga (0.5.0) we seem to lose the exception's original message, example:

(timbre/error (Exception. "BOOM") "Something bad happened")
;=>
{"timestamp":"2016-04-22T13:21:20.580Z","level":"ERROR","message":"Something bad happened","namespace":"gocardless-reconciler.core","stacktrace":"gocardless_reconciler.core$eval25772$fn__25773.invoke(core.clj:29)\n...

(Result clipped)

In addition if the type of the exception is a clojure.lang.ExceptionInfo we lose the map that is stored in the object, example:

(timbre/error (ex-info "BOOM" {:foo :bar}) "Something bad happened")
;=>
{"timestamp":"2016-04-22T13:23:55.040Z","level":"ERROR","message":"Something bad happened","namespace":"gocardless-reconciler.core","stacktrace":"clojure.core$ex_info.invokeStatic(c...

(Result clipped)

We are lacking the ability to set the log level via env when running the application.
Timbre only makes use of the TIMBRE_LEVEL env at compile time.

Perhaps we could introduce the following env:

LOGA_LEVEL=:debug

💡 should levels be specified as strings debug or keyword :debug?

The use of environ's env here: https://github.com/FundingCircle/clj-loga/blob/master/src/clj_loga/core.clj#L86 will break in some circumstances (if I remember correctly, either when the code is compiled into a jar, or when AOT compilation is used).

(def ^:private loga-enabled?
  (= (env :enable-loga) "true"))

This is because the def is calculated at compile-time rather than run-time. And at compile time, the environment variables won't be set.

When running with leiningen (i.e. in development), the environment variables will be set at compile time, so the problem won't manifest until we push it to an environment.

Easy fix is to make this a function rather than a def.

The readme says:

#export env variable to apply formatting
ENABLE_LOGA=true

But that environment variable is never used. What's up?

I think the common case for an app that includes clj-loga in it's dependencies will be to want it to be enabled ;-). To make the common case the easiest, should we default ENABLE_LOGA to true, or swap it out for an env var called LOGA_DISABLED?

Exception:
com.fasterxml.jackson.core.JsonGenerationException: Cannot JSON encode object of class: .....

Some objects can't be serialised to JSON and the library should have fallback functionality

cheshire.generate$generate.invokeStatic(generate.clj:147)
 cheshire.generate$generate.invoke(generate.clj:111)
 cheshire.generate$generate.invokeStatic(generate.clj:119)
 cheshire.generate$generate.invoke(generate.clj:111)
 cheshire.generate$generate.invokeStatic(generate.clj:119)
 cheshire.generate$generate.invoke(generate.clj:111)
 cheshire.core$generate_string.invokeStatic(core.clj:29)
 cheshire.core$generate_string.invoke(core.clj:13)
 cheshire.core$generate_string.invokeStatic(core.clj:19)
 cheshire.core$generate_string.invoke(core.clj:13)
 clj_loga.core$output_fn.invokeStatic(core.clj:105)
 clj_loga.core$output_fn.invoke(core.clj:102)
 clj_loga.core$output_fn.invokeStatic(core.clj:103)
 clj_loga.core$output_fn.invoke(core.clj:102)
 taoensso.timbre.appenders.core$println_appender$fn__1670.invoke(core.clj:101)
 taoensso.timbre$log1_fn$fn__1814.invoke(timbre.clj:398)

Exception is coming from cheshire

🙀 This project simultaneously claims to use both an Eclipse and BSD 3-clause license. It's unclear which license applies to the project. We need to remove the Eclipse license immediately and bump the project version.

This would generally help when scanning the logs for problems.