Comments (5)
Why is that confusing? What do you think better?
"session_time" has concept of years, days, hours, etc. as numbers. They're important because workflow often uses timestamp formatted in text. Therefore digdag CLI shouldn't convert it to local time.
Next run time is different. Formatting means nothing for machines. It's OK to use local time for humans.
from digdag.
Maybe it's just that I'm confused about session vs run time.
from digdag.
Are there any docs/material where I can read up on the significance of session vs run time?
from digdag.
That is extremely very important concept but no documents!!
Well, for example, you have a workflow that runs every day. It puts results to a table on TD with date's suffix like result_20160412
. You'll use create_table: [result_${last_session_date_compact}]
option with td>
operator.
Query will be like this:
select count(*) from data
where TD_TIME_RANGE(time, '2016-04-12 00:00:00 -0700', '2016-04-13 00:00:00 -0700')
which will be written in workflow definition as following:
select count(*) from data
where TD_TIME_RANGE(time, '${last_session_time}', '${session_time}')
But you noticed that you can't run the workflow at 00:00:00 because data is not ready at 00:00:00. You need to delay the start time for 3 hours. In this case, session_time should be 2016-04-13 00:00:00 -0700, but run time should be 2016-04-13 03:00:00 -0700.
This also happens when you retry the workflow. If you retry it 1 week later, run time is 1 week later but session_time should be consistently 2016-04-13 00:00:00 -0700.
from digdag.
Ok, that makes sense, thanks for explaining! =)
from digdag.
Related Issues (20)
- InvalidParameterException occurred only when executing sh operator on ECSCommandExecutor. HOT 9
- ECS Command Executor fails if task continues for 10 minutes or more. HOT 1
- How do I point to the parent path? HOT 1
- [feature-request] Oracle operator HOT 1
- Unable to kill node.js task HOT 3
- How to create a global config for multiple projects in the workspace? HOT 2
- Unable to exit as success status when using sys.exit(0) from py:> operator HOT 2
- broken dependency in digdag-client
- Please add route /workflows/{workflow_id}/edit
- [Regression] Commit breaks --enable-swagger HOT 3
- Concurrent update in table "QUEUED_TASK_LOCKS": another transaction has updated or deleted the same row [90131-192] HOT 5
- Please release v0.10.5 HOT 2
- Swagger is broken in v0.10.5 and v0.10.5.1 HOT 2
- Docker build environment fails to build HOT 1
- How to exit or continue while using if or for_each or range or loop in digdag?
- How to add two conditions in the if statement in digdag? HOT 2
- Workflow task is showing running status
- Error with http call HOT 5
- Incorrect description of the `last_id` query param on GET /api/attempts endpoint. HOT 1
- Directory extraction task in workflow extracts unexpected items HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from digdag.