server side workflow log filtering about digdag HOT 5 CLOSED

👍

from digdag.

it sounds good idea to use a structured log format. what we need to take care is:

• BufferedRemoteTaskLogger needs to use a regular ReentrantLock or synchronized-block instead of ReentrantReadWriteLock.
• timestamp should include at least milliseconds. Preferably, microseconds. I think it's ok to use this proposal: msgpack/msgpack#209
• MessagePack weakly distinguishes strings from binary. I think "message" field should use string if stream is LOG, and binary if stream is OUT or ERR.

from digdag.

About REST API, from past experiences, users (and Hadoop!) occasionally produce large amount of logs (GBs of logs) intentionally or accidentally. Current implementation lets agents upload files directly to S3, and lets clients download directly from S3. That's because I don't want to fill all of server threads for log processing and make them unavailable for the other requests.
So, server-side filtering is ok for browsers/GUI if they use appropriate pagination. But not good especially for CLI when they download the entire logs.

from digdag.

Another note:
In digdag's design, agents may be hosted on an user's servers (although we distribute the code). REST API for agents to upload logs are receiving log files. In this case, contents of those files are user-data are not 100% trusted. If we have REST API on server that processes those data on server, we should make sure that unzip of gzip or unpack of msgpack might cause errors because of broken user-data.

from digdag.

Close the old issue. Please reopen or file new one if you need.

from digdag.