Duplicate log lines on CeleryExecutor not fixed with AIRFLOW__COSMOS__PROPAGATE_LOGS about astronomer-cosmos HOT 5 CLOSED

🤖

Hello again @adammarples! It's good to see you back in our issue threads. I hope you're doing well.

I understand that you're experiencing an issue with duplicate log lines when running a CeleryExecutor on Astronomer hosted, using astronomer-cosmos==1.2.5, even after setting AIRFLOW__COSMOS__PROPAGATE_LOGS = "False" or AIRFLOW__COSMOS__PROPAGATE_LOGS=false.

This issue might be due to the logger not correctly picking up the propagate_logs configuration from the Airflow settings. The get_logger function in the Cosmos codebase is trying to read the propagate_logs configuration from the cosmos section of the Airflow configuration. If the configuration is not found, it defaults to True, which means the logs will be propagated to the parent loggers, potentially causing duplicate log lines. You can find this in the cosmos/log.py file.

To prevent duplicate log lines, you need to ensure that the propagate_logs configuration is correctly set in the Airflow configuration. However, you mentioned that you have already tried setting AIRFLOW__COSMOS__PROPAGATE_LOGS = "False" or AIRFLOW__COSMOS__PROPAGATE_LOGS=false, but it didn't work.

This might be due to a mismatch between the environment variable and the configuration option that the get_logger function is trying to read. The get_logger function is looking for propagate_logs in the cosmos section, but the environment variable you're setting is AIRFLOW__COSMOS__PROPAGATE_LOGS.

To resolve this issue, you can try setting the propagate_logs configuration directly in the airflow.cfg file under the cosmos section:

[cosmos]
propagate_logs = False

If this doesn't work, there might be an issue with the way the get_logger function is reading the configuration. You can try modifying the get_logger function to print out the value of propagateLogs after it's read from the configuration to see if it's correctly picking up the setting:

def get_logger(name: str | None = None) -> logging.Logger:
    propagateLogs: bool = True
    if conf.has_option("cosmos", "propagate_logs"):
        propagateLogs = conf.getboolean("cosmos", "propagate_logs")
    print(f"propagateLogs: {propagateLogs}")  # Add this line
    logger = logging.getLogger(name)
    formatter: logging.Formatter = CustomTTYColoredFormatter(fmt=LOG_FORMAT)  # type: ignore
    handler = logging.StreamHandler()
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    logger.propagate = propagateLogs
    return logger

This will help you debug whether the propagate_logs configuration is being correctly read from the Airflow configuration.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• Annoying duplicate celery logging in Airflow task logs when running in celery worker
• cosmos/log.py
• docs/configuration/logging.rst

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

from astronomer-cosmos.

Hi @tatiana, I am just investigating it. It's my bad, it's not a bug, these log propagation changes work fine in main branch but are not included in version 1.2.5, so I'll wait for the update.

from astronomer-cosmos.

@adammarples Sorry - we are really lacking versioned docs, I logged #750 so we don't lose track of this issue.

from astronomer-cosmos.

@adammarples, thanks for reporting the issue!

This may be a bug, and the robot suggested using the configuration file as a workaround. Could you confirm this works?

A possible fix could be done along these lines:

And we could add a test to make sure things work with the env vars:

Would you like to make this contribution?

from astronomer-cosmos.

@adammarples this change is available in the release 1.3.0a2, in case you'd like to try it out:
https://pypi.org/project/astronomer-cosmos/1.3.0a2/

Since it is a new feature, we didn't release as part of a 1.2 micro release. For some reason we missed to include it in our release notes, I'll fix this:
https://github.com/astronomer/astronomer-cosmos/releases/tag/astronomer-cosmos-v1.3.0a2

from astronomer-cosmos.