raybeam / rb_status_plugin Goto Github PK

I was messing around with different types of quality checks and I noticed that the Lumen report DAGs actually have 2 different types of failures.

1. Fail if any of the list of tests fail
2. Fail if it's set up incorrectly, for example, the send email task fails or a test no longer exists.

So, the second option is basically a status of the status tests ... kinda meta, but important. We should think of a different way for the Lumen report DAGs to report success or failure.

Adding tests for the new attributes added to Report class

• adding metadata for tasks (schedule, description, last run)
• differentiating between tasks that are part of active/disabled dag when creating report
• do we alert owner/user if underlying dag changes schedule?
• how does a task's last run date (vs. report's last run date) affect status

We probably want the value displayed to be the name of the dag and not the number of the run. The number doesn't have much semantic value alone. Example:

1. Report is called example_dag
2. The id passed to the UI is an integer like '9' instead of example_Dag

We need to populate the existing report configurations into the UI when a user edit's an existing report.

Test configs will need to create DAGs

Showing report's status in the status page

1. Implement email callback function using email operator with templated html from Andrei
2. Ensure one email is sent at the end of dag reporting all failures and success
3. Test in Astro

cron schedule, day of week, and time appear as options in UI, regardless of which schedule is chosen.

We need to upload the list of all available tests for the user, when they are building a report.

Currently the UI is populating sample tests, but we need to extract the list of tests as dag_name.task_name from airflow's postgres db.

If a report doesn't exist, visiting the edit page for that report should raise an error and redirect to /reports.

Need to add tests for saving an edited report.

Change LumenBaseBuilderView -> LumenStatusView to build email link

Node: weightless-sun-9695-webserver-b67c544b6-cr5h2

Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
return f(self, *args, **kwargs)
File "/usr/local/airflow/plugins/lumen_plugin/views.py", line 176, in form_post
extract_report_data_into_airflow(form)
File "/usr/local/airflow/plugins/lumen_plugin/helpers/report_save_helpers.py", line 29, in extract_report_data_into_airflow
convert_schedule_to_cron_expression(report_dict, form)
File "/usr/local/airflow/plugins/lumen_plugin/helpers/report_save_helpers.py", line 79, in convert_schedule_to_cron_expression
time_of_day = form.schedule_time.data.strftime("%H:%M")
AttributeError: 'NoneType' object has no attribute 'strftime'

Add logic to handle manual schedule (aka setting schedule_interval=None)

Most likely a front-end change

The UI is already in place, but we need to create a hook that will call the underlying dag to run in real-time when a user clicks the trigger report button.

The UI is already in place, but we need to create a hook that will change the state of the report's underlying dag to ON or OFF.

The UI is already in place, but we need to create a hook that will first delete the airflow variable for the report and then the underlying dag for the report, when the user clicks delete report button.

No error thrown currently I believe...(although I can't tell with string format error occuring)

Lumen DAGs should have catchup set to False so they don't try to backfill.

Create form setup:

I think this is because form.tests.data used to set the variable only returns the position of the test in the list and not the test value.

Configs looks like this:

{
  "tests": [
    "dag_name.operator_1", 
    "dag_name.operator_3"
  ],
  "emails": ["[email protected]", "[email protected]"],
  "schedule": "0 7 * * *"
}

Lumen needs to be able to read all variables and parse the report configs.

Lumen test config variables are prefixed with lumen_test_

Right now we're passing a link to the error log. We want to pass a link to the status page

We need to move the DAG part of AIrflow into it's own repo and just have the plugin stand alone. The aim is to have the plugin in it's own repo but still deploy into an AIrflow instance for CI and demo.

Here's an example of an Airflow plugin repository: https://github.com/airflow-plugins/salesforce_plugin

It should not assume Astronomer but should accommodate it.

When editing an existing report, the schedule should show up in the same format as it was during last save.

Ie. We don't want to display a cron expression when a user goes back to edit a report, if the user originally saved it using the non-cron options.

Create Lumen page for reports management.
Spec https://docs.google.com/document/d/1mUuVnksqLfKGnVXMUHh8UA7vWynSLgf1WD7JLyI8j94/edit#heading=h.3131wg1wryom

We need to embed into the report management workflow a pattern such that:

• the initial report title is tied to the variable name
• if a user changes the report title, only the report title attribute is changed in the variable and not the actual variable's name
• the Lumen UI can still find the report and populate the proper report title in each of the pages

Right now, if you delete a Lumen report DAG in the Airflow UI, it just comes back later. You can only truly remove it if you remove the Variable that creates it. When we change to a DB backed management system, it will only be removed when you remove it from the Lumen database tables.

Should removing that DAG in the UI remove it in Lumen?

Here we passing emails into owner_email that results in this bug

<a href="mailto:['[email protected]', '[email protected]']">['[email protected]', '[email protected]']</a>

Is it our intention to have multiple owners?

To my understanding:

• emails - a list of subscribers
• owner_name and owner_email - creator/owner of the report, one person in the current design

Test choices population using the task instance object.

Do we want to populate tasks that haven't run? Is this part of the MVP?
We need to find a new model to use that isn't taskinstance. I'm not sure where to find operator data though.

How to handle errors when user submits form (ie. bad email).

Currently, it will throw an airflow error on a new page.

To fix this issue, we should show error(s) at top of the same page when user clicks Save Button.

The reports_data method (and other methods) on Lumen views should handle errors gracefully. Probably logging them and maybe showing an error in the page. It should not throw a 500 and fail like it does now.

Example

When the ReportInstance tries to get_latest if there is no available run, it throws an exception. This exception results in a 500 in the report view.

In order to easily run Lumen, we'll need a script to set up the environment.

Lumen Back End should be able to extract the data from the Report Management UI into an airflow variable when the user clicks the "Save" button on a new report.

We need unit tests for email.

1. Test each of the three statuses: [Success, Failed, Unknown]

Adding all attributes about a report, so that a Report object has all required information about itself.

I think this needs to be changed to only take into consideration test tasks. Example:

1. The email send operator fails but all the test tasks succeed
2. The report will show as failed even though all the tests tasks passed because the email operator failure set the dag status to failed

I have helper functions in email_helpers.py that determine test tasks and I think should be moved to the reportinstance class

We need to have the following attributes in the value of the airflow variable:
report_title
description
owner_name
owner_email
subscribers
tests
schedule

Note that emails has been renamed to subscribers and also that subscribers is a union of owner_email and subscribers.

Queries postgres for status of task_instance (aka a test) and then fails or succeeds based on task_instance's status.

What fields are required in the create form field?

The only field that prevents submission from what I can tell is emails not being formatted. I was able to have "report submitted" appear with this report with no title and no tests and no schedule

Moving helper functions into classes (based on their functionality) in order to give a more holistic view of how each function ties together.

Create email template using Foundation and test it in litmus

Currently, /lumen/reports/ is being populated by dummy_reports.

The page should be populated with real report data.