for_each parameterization capability is limited about digdag HOT 4 CLOSED

This was inspired by seeing the programmatic looping here in the digdag-analytics poc: https://github.com/treasure-data/digdag-analytics/blob/master/se_poc_emails/tasks/poc_workflow.rb#L15

It seems desirable to be able to use digdag to perform these queries and leverage the digdag parallelism, retry and dependency tracking etc facilities instead.

from digdag.

1. Use YAML reference (& and *). This works now as you mentioned.
2. Introduce a new syntax like key: @foo, key: @{foo}, key: @${foo}, key: $@foo, key=: foo, key: !param foo etc.
3. Preserve type of foo if a value is ${foo} without any characters around it.
   If foo is an integer, prefix_${foo} is string, ${foo}_suffix is string, but ${foo} is integer.
   I think this is problematic when users want to convert integer to string. In YAML syntax, you can use "${foo}" but there're no ways to distinguish it from ${foo}. So, users need to use ${foo.toString()}. I think this is not intuitive.
4. Parse value as a JSON string if an operator gets it with array or map type.
   If operator is written in Java, it uses int foo = conf.get("foo", int.class) or List<String> foo = conf.getList("foo", String.class) API to get a config value. This get or getList methods have a chance to convert value type.
   Currently, conf.getList("foo", String.class) fails if actual value of foo is not a list. But we can change the behavior so that it tries to parse the string as JSON if the actual value is string.
   This works because ${foo} converts foo to a JSON string if foo is a list or map.
   But this doesn't work if operator is Ruby or Python (dynamically-typed). params.get("foo") is string even if the operator expects list.
5. Similar to 3, but use string by default. But raw(...) to preserve type. So, ${foo} is string, but ${raw(foo)} is list. prefix_${raw(foo)} is string, though.

In a very old version, syntax was key=: foo (so, idea 2).

from digdag.

I like the key: @foos syntax but @ is reserved and cannot be used to start a token in yaml =/

http://yaml.org/spec/current.html#c-directive

On the other hand, less ways to parameterize things means less surprise and mistakes for our users, so allowing key: ${foos} (idea 4) might be more desirable.

from digdag.

It would also be useful to be able to do things like:

run: +main
+main:
  for_each>:
    foo:
      - bar: a
        baz: b
      - bar: c
        baz: d
  _do:
    sh>: "echo ${foo.bar}: ${foo.baz}"

Where the tasks executed for the above would be:

echo a b
echo c d

Currently the above produces:

error:
  * +main:
    io.digdag.core.repository.ModelValidationException: Validating workflow task failed
name can't contain   character "+for-foo={bar=a, baz=b}"

from digdag.