Comments (9)
Thanks for bearing with us on this. I can confirm we are also seeing this issue on our jobs (of all lengths). We're looking into it and will hopefully have a resolution soon.
from tibanna.
What Python version are you using? Note in the output you provided:
2.0.0 Requires-Python >=3.7,<3.9
I assume you are still on 3.6, which is past EOL. Try using a newer Python version? Also, the job log may be helpful here in deciphering the issue.
from tibanna.
In my case, the issue was my python version was too new.
Upgrading didn't resolve the plot_metrics issue. Here is a new plot_metrics output...
https://salk-tm-logs.s3.amazonaws.com/aA6bccyAeLQt.metrics/metrics.html
from tibanna.
Seems to me like the issue is that your job is too short to extract the metrics for those plots from CloudWatch, and if it were longer you would see the plots. Odds are since it is very close to the CW reporting delta, you may not see any data points over that time frame (or the job finishes and makes the API call before CW has processed the data point and made it available). See these functions that pull that data. My guess is those API calls are returning no data points due to a very short job. You could write a small script that makes such calls and run them simultaneously while your job executes to verify.
With that said, if you go into CloudWatch and do in fact see metrics over the job time frame, we will look into it further. But it could also be the case that the data point (probably singular) just comes in too late for Tibanna to catch it. If you add a 15 minute sleep at the end of your job, I suspect the metrics will report as expected.
from tibanna.
Ok, this job ran for an hour:
https://salk-tm-logs.s3.amazonaws.com/eTcBZv8sWlB8.metrics/metrics.html
...Here are all the "log_files" that were uploaed to s3 and are associated with the job...
...Cloudwatch logs associated with the job below...
log-events-viewer-result.eTcBZv8sWlB8.csv
from tibanna.
from tibanna.
Not 100% sure what you are referring to but I don't think I use GCAP/4DN.
I assume tibanna uses some specific image when spinning up ec2 instances that includes tibanna and dependencies. If the issue is with perl, it seems like the sollution is updating that image?
from tibanna.
We are developing Tibanna for the CGAP/4DN projects.
This is a strange issue. I checked a few recent 4DN runs and the metrics reports look ok. However, I found one where it is incomplete as well:
https://tibanna-output.s3.amazonaws.com/BuYiLLidEFPL.metrics/metrics.html (complete)
https://tibanna-output.s3.amazonaws.com/lDZmmG1vMVnT.metrics/metrics.html (incomplete)
Same workflow, same machine, both ran yesterday with the latest version of Tibanna for more than 5 hours, but the second one has an empty metrics.tsv
and metrics_report.tsv
.
from tibanna.
It is possible there are multiple issues happening here. I haven't seen complete metrics on any of my jobs that I've checked. I'm not sure how many I've checked, but at least a dozen. I'm about to kick off a bunch of assemblies again so I'll check more jobs soonish.
from tibanna.
Related Issues (20)
- S3 Upload Encryption Argument HOT 11
- Minor bug when downloading files containing spaces to EC2 instance
- Tibanna1.0 errors out with Snakemake HOT 1
- Forked repo isn't used on EC2 instances even though it is declared at deployment HOT 3
- MissingInputException with Snakemake and Tibanna HOT 2
- Log Differences between 0.18.3 and 1.0+ HOT 3
- Specifying ECR AWSF_IMAGE in snakemake HOT 1
- Step functions fail for snakemake rules HOT 1
- Tibanna/Snakemake compatibility issue? HOT 4
- Docker image for 1.9.2 doesn't exist HOT 1
- AWS ending python 3.6 support for lambda functions HOT 1
- botocore client error Rate Exceeded HOT 1
- Tibanna instance type error with snakemake HOT 29
- runtask failure : InvalidFleetConfig HOT 6
- Transfer ownership of Snakemake tibanna plugin HOT 8
- Large amounts of NAT gateway costs HOT 4
- Turn off cloudwatch and costexplorer HOT 6
- KMS Authentication Error in Lambda When Running Tibanna HOT 1
- deploy_unicorn: Pip Install Conflict Error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tibanna.