project structure and workflow suffix changes about digdag HOT 8 CLOSED

Implementation idea:

• Change NoopWorkspaceManager:
  • rename to LocalDirectoryWorkspaceManager
  • take "_workspace" from TaskRequest.getConfig. if it's set, use the directory as workspace path. Otherwise, use current directory (that is same with current behavior).
• (Rename LocalWorkspaceManager to ExtractArchiveWorkspaceManager because the original name becomes confusing)
• Change ProjectArchiveLoader:
  • loadProject takes path to a directory that has .digdag/config instead of path to a file named digdag.yml.
  • it finds *.dig files recursively and load all of them
  • it calls resourceNameToWorkflowName(realPathToResourcename(Path)) methods to convert path of *.dig file to workflow name.
• Change ProjectArchive:
  • add private final Path projectDirectoryPath method.
  • add realPathToResourceName(Path) method that converts Path to a resource name that is relative from projectDirectoryPath.
  • change listFiles((path) -> { ... }) to listFiles((resourceName, realPath) -> { ... })
  • (add Properties getProjectConfig() that returns parsed contents of .digdag/config).
• Change cli.Run:
  1. requires at least 1 argument.
  2. finds .digdag/config file from current directory to parent directories until root (/) or at most N depth.
  3. if it's found, call ProjectArchiveLoader.loadProject with that path. Otherwise, throws an exception.
  4. if given name doesn't end with ".dig", add it. Then, call resourceNameToWorkflowName(projectArchive.realPathToResourceName(path)) to get the workflow name using the same way with ProjectArchiveLoader.
  5. lookup that name from projectArchive.getArchiveMetadata().getWorkflowList().
  6. if it's found, store it to ProjectStoreManager and run it. otherwise, throws an exception.
  7. when it stores a workflow, set _workspace = projectArchive.getProjectDirectoryPath() to Revision.defaultParams so that LocalDirectoryWorkspaceManager uses it as the workspace path. this step can be omitted if projectArchive.getProjectDirectoryPath() is same with current directory.
• Change cli.Push:
  • use same behavior with cli.Run for step ii. and iii.

from digdag.

I believe we said that digdag should not be recursively searching for .dig files in subdirectories. Instead, if a user wants to use subdirectories, they have to list either the subdirectories or the workflows explicitly in a project file.

from digdag.

no, we said that digdag recursively searches for .dig files in subdirectories. otherwise this behavior doesn't make sense:

digdag push pushes entire project (as identified by .digdag) in some parent dir, even if executed in a subdir.

from digdag.

That still makes sense without recursively searching for .dig in subdirectories. The digdag cli recurses to parent directories to find the .digdag directory. digdag push pushes all files and subdirectories in the project directory, regardless of file extension.

from digdag.

Note the difference between recursing upwards into parent directories to find .digdag and recursing downwards into subdirectories.

from digdag.

then why does digdag push look for .digdag in some parent dir when it's executed in a subdir?

from digdag.

So that the digdag command can behave like git. I.e. a digdag project is defined by the presence of .digdag in some parent dir and doing digdag push in a subdir uploads the entire digdag project.

Given that digdag push uploads the entire directory tree rooted in the directory containing .digdag I'm not sure why it would be necessary to search specifically for .dig files in subdirectories. It seems to be that all those subdirectories along with the .dig files will be uploaded anyway. Am I missing/misunderstanding something?

from digdag.

Close the old issue. Please reopen or file new one if you need.

from digdag.