As seen in @jbusecke's https://github.com/leap-stc/data-management/actions/runs/4953296291/jobs/8860631786, the current logic does not pick up existing labels on merged PRs, if the workflow is triggered by a push event.

This leads pangeo-forge-runner to try to run everything, which in turn results in:

1. Not using the custom job_names defined in the action
2. Due to the bug referenced in the inline comments in the action, naming collisions occur

pangeo-forge/pangeo-forge-runner#90 drops use of the pangeo/forge image dependency in pangeo-forge-runner so we should probably do the same. This will allow us to keep pace with current beam releases, and should also speed up builds, which @jbusecke has reported are (unsurprisingly) slow. This should be pretty simple to experiment with:

1. Choose a lighter image to update this line with

   deploy-recipe-action/Dockerfile

   Line 1 in 256da29

   FROM quay.io/pangeo/forge:5e51a29

   Maybe read this or similar blog posts. Probably end up just trying one of these?

2. Replace quay.io/pangeo/forge:5e51a29 on that line with the URI for the chosen image tag

3. Now we don't have conda, so drop the conda run -n notebook from this line:

   deploy-recipe-action/Dockerfile

   Line 3 in 256da29

   RUN conda run -n notebook pip install git+https://github.com/pangeo-forge/pangeo-forge-runner@main

4. That might be all it takes?

5. Now make a PR with these changes, try running the action from @pr-branch and see if it works (and is faster).

6. We don't actually test the docker build in our testing here. Maybe do that before merge, I can help with that when we get to that point.

@jbusecke wanna give it a shot?

In this action, we have an actions/checkout step, so the action is running in a locally-checked-out copy of the feedstock repo. This makes fetching the repo over https redundant (and unnecessarily costly). So instead of:

I think we can save networking time and just do:

"--repo=."

This (which I'm responsible for) seems like it's asking for trouble:

...especially considering (IIUC) that the Dockerfile is built on every workflow invocation.

Pinning to a specific pangeo-forge-runner release would be better.

I just discovered that when you specify recipe labels with leading spaces (run: some_recipe_id), the recipe will get picked up by the action (it tries to run it), but will fail due to the constucted jobname still containing spaces (which dataflow hates).

We should make this behavior consistent by doing one of the below:

• Error out earlier in the action (with helpful error message) if the label based id contains spaces
• Or remove spaces automatically as part of the name generation.

I guess if we are implementing one or the other we want to take care of trailing spaces too?

#4 highlights the need for tests (in particular integration tests). This is a top priority.

I seem to often get confused which dataflow jobs are associated with which run of the GitHub deploy workflow. I think we have all the necessary info to create a list of clickable links.

Ideally this would be shown in some sort of expandable field in the 'checks' section under a PR and could link the recipe name to a clickable link to the dataflow console.

I just ran into an issue when I pushed to main on cmip6-leap-feedstock.

Seems that for pushing to main the following is not true?

assert len(pulls) == 1  # pretty sure this is always true, but just making sure

Just wanted to flag this.

I just noticed a quirk with the deploy action for a brand new repo where the action was entirely triggered by pushes to main.

I was trying to separate a recipe example from a more mature and complex bakery repo and ran into this error.

assert len(pulls) == 1  # pretty sure this is always true, but just making sure

I think this line here basically fails because there is not a single PR in the repo yet (will test this in a minute). But just wanted to document this here.

There is not yet any documentation regarding how to use this action.

Here are some examples used during development:

• https://github.com/pforgetest/gpcp-from-gcs-feedstock/pull/8/files#diff-4f9e38227ed64fefb17f4668a7ac4ab55b6149994d5ac2fd96182d5958479b54
• https://github.com/leap-stc/data-management/pull/19/files#diff-4f9e38227ed64fefb17f4668a7ac4ab55b6149994d5ac2fd96182d5958479b54

(These may be out of date, and should only serve as reference.)

Filling out the README here with usage examples/instructions is a top priority.

This was mentioned by @cisaacstern in the weekly call today.

Even though I had the action version pinned in my CMIP bakery the recipe deployment 'just broke', likely since the runner version is not explicitly pinned. Wondering if we should be more explicit here?

I am getting a lot of errors like this

Submitting job...
[54](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:55)
recipe_ids = ['METAFLUX_GPP_RECO_monthly']
[55](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:56)
Submission jobname = 'METAFLUX_GPP_RECO_monthlybc9fe248'
[56](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:57)
Running PGF runner with extra_cmd = ['--Bake.recipe_id=METAFLUX_GPP_RECO_monthly', '--Bake.job_name=METAFLUX_GPP_RECO_monthlybc9fe248']
[57](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:58)
  File "/deploy_recipe.py", line 100, in <module>
[58](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:59)
    deploy_recipe_cmd(cmd + extra_cmd)
[59](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:60)
  File "/deploy_recipe.py", line 17, in deploy_recipe_cmd
[60](https://github.com/leap-stc/data-management/actions/runs/4942452173/jobs/8836187762#step:5:61)
    raise ValueError("Job submission failed.")

but there is no information provided about why it failed. Could we capture the stderr just like here and print it in case of failure?

See leap-stc/data-management#21 (comment)

cc @jbusecke

Just wanted to write down some thoughts about the label based triggering of deployments.

I am still finding myself in a lot of pretty messy situations for two reasons:

• Mixing the running of recipes with code changes. I often want to just rerun a recipe and currently have to open a dummy PR or mix the running of recipes into other PRs. This gets messy over time and is not very easy to grok after a while (or other contributors).
• I have to add/remove labels one by one. I often find myself in situations where I want to run 2+ recipes. But each setting of a label 'fires' off a github action. This means if I add run:a, run:b as labels in one go I will fire of two github workflows (one deploying recipe a, the other deploying recipe a and b ). This is very hard to keep track of after a while or with many recipes. The only solution I have found so far is to add run:a, then remove run:a, then add run:b and remove run:b to fire of two jobs (the first one runs a, the second one runs b).

I am not quite sure how to improve this behavior but wanted to note this down somewhere for future discussion.

Failed parsing of requirements.txt in leap-stc/data-management@d30240a:

https://github.com/leap-stc/data-management/actions/runs/5685911763/job/15411769228?pr=33#step:6:58

reveals that this logic for parsing requirements.txt

is too naive.

We should refine that logic to parse requirements.txt's that have comments.