Bug Report

• Logs in ALA are getting missed on a regular basis without any pattern on a daily basis
  example: About more than 10% of logs are missed per day

Description

• Because of missing logs we are not able to debug production application issues, as we cannot rely on this data.
  We are facing this challenge in West Europe region, because our application runs on that region.
• Our application workloads are running in a managed AKS cluster
• Current design to push logs from container to ALA using fluentbit agents and we have found that there is no lag between pushing logs from fluentbit to ALA
• Here is our fluentbit config.yaml configuration:
  • Log_Level=ERROR
  • Flush=5
• Just for comparison we have Datadog setup for the same application configuration, and here we have Datadog agents that keeps pushing logs to Datadog and we have not seen this kind of issues in Datadog.

Expected Behavior

• There should not be any loss of logs in ALA