Comments (3)
I'm afraid without any manifests loaded, there's nothing to query. I'm curious, how many models do you have?
We've got around 1.2k models and this just about still works (although it is getting very slow).
I'm considering doing more of the flattening on upload - it changes the paradigm of this package quite a lot but I think is more useful for larger projects like yours. How does that sound?
from dbt_artifacts.
Thanks for taking a look! I'd like to challenge the notion that there's nothing to query without the manifests. We've worked around this by writing our own version of the plugin and still get all of the things I noted above (last model run time, frequency of model/test failure, longest model run times, etc) from just the run_results.
Our biggest project has 700 models, but 2k sources. I think flattening on upload probably makes the most sense for being able to load large manifests, but does certainly complicate things.
from dbt_artifacts.
Hi @bmedx - this still requires the manifest to load however the latest version of the code will now flatten manifests on load which should overcome your large manifest issue. Going to close this out.
from dbt_artifacts.
Related Issues (20)
- [Bug]: test "failures" has a misleading definition HOT 1
- [Bug]: The latest dbt_artifacts version is not compatible with dbt 1.7.0 HOT 1
- [Bug]: dbt-project.yml file of the dbt-artifacts 2.6.1 version excludes dbt versions 1.7.0 and above HOT 1
- [Bug]: ROWS_AFFECTED column in table FCT_DBT__MODEL_EXECUTIONS shows always 1 HOT 1
- [Bug]: Error creating sql incremental model dbt_artifacts.invocations
- [Feature]: Snapshots of "source" tables? HOT 1
- [Feature]: support for duckdb
- [Bug]: 'dict object' has no attribute 'raw_code' - incompatibility with other packages HOT 2
- [Bug]: Object of type date is not JSON serializable HOT 1
- [Bug]: Parsing Error in dbt Artifacts JSON Conversion of Source with Case-Sensitive Column Names in Filters for dbt freshness
- [Bug]: Syntax error while creating all staging models in dbt artifacts HOT 1
- Support dbt v1.8 HOT 2
- [Bug]: "compile_started_at" and "query_completed_at" times for test get extracted from wrong object resulting in nulls HOT 2
- [Bug]: Concurrency issue in Snowflake when running upload_results macro
- [Feature]: Include relation_name in test_executions for table reference after store-failures HOT 1
- [Bug]: Unable to write dbt_artifacts results to different dataset in Bigquery HOT 5
- Dbt Databricks
- [Feature]: Add Athena + Iceberg support HOT 1
- [Bug]: Object of type date is not JSON serializable (Databricks) HOT 1
- Add tox-based Integration Testing support for this package
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt_artifacts.