Comments (4)
My original idea was that for the posterior 8_schools|centered
we would store a bayesbench result object in gold_standards/8_schools|centered.zip
(or json
instead of zip
)
That makes the posterior database coupled to bayesbench though, so if bayesbench changes the output format then every gold standard would need to be updated. Maybe there is some better way that doesn't couple the posterior database to bayesbench.
In any case, the gold standard needs to include
- Posterior samples
- Which posterior produced the samples (it doesn't need to be the same one, in this example it could also be
8_schools|noncentered
) - Diagnostic values that were used to determine that the posterior is good enough
- Which inference engine was used (I'm not sure yet if this is 100% mandatory)
from posteriordb.
Yes. We want to have a bayesbenchr object as gold standard. But we should not call it bayesbenchr object and it should not be coupled. But the structure should be the same. I think our structure was good.
from posteriordb.
What would be the workflow for adding gold standards to posterior database, something like this?
Lets say we want to add gold standard for 8_schools|noncentered
- Run bayesbench on
8_schools|noncentered
using some inference method we believe to produce correct results - Check that the result is good (using something like checking diagnostic values)
- Copy the output file to
posterior_database/gold_standards
and rename it to be called8_schools|noncentered.zip
- Add
gold_standards/8_schools|noncentered.zip
to the posterior info file under the keygold_standard
This workflow means that posterior database is coupled to bayesbench (as we rely bayesbench to return the exact format that posterior database uses). If we want to avoid it here's a few possible solutions:
- Add step 5: run
./bayesbench_to_posterior_db_gold_standard.R 8_schools|noncentered.zip
that modifies the file to a format that is independent of bayesbench. Then if bayesbench changes output format we would have another script like/bayesbench_v2_to_posterior_db_gold_standard.R
- Add
version
field to bayesbench output. The gold standards directory is allowed to have gold standards in different versions and when the user requests a gold standard we have code to change every version to a common format - Just store the bayesbench output as is. When bayesbench changes the output format write a script to convert each stored gold standard to the new format. If this conversion is impossible (the new format includes something that is not present in the old format) then just run inference again to obtain a result in the new format (and check that it satisfies the diagnostics)
- Never change the output format of bayesbench (probably impossible)
from posteriordb.
Now added and fixed
from posteriordb.
Related Issues (20)
- Example problems from astrophysics HOT 5
- Transferring to stan-dev HOT 1
- Updating Stan Models to be more performant HOT 3
- Convert golden samples to arviz IData HOT 3
- Add model code dependency structure
- Document data variables separately
- Handling of posterior licenses HOT 11
- Include explicit stan version in reference posterior computations
- install instructions for R package HOT 4
- Add eight schools with flat prior
- Add the occupancy model (population biology)
- Better examples needed HOT 5
- How to get the log probabilities of MCMC samples
- dogs-dogs not constrained properly in Stan HOT 1
- PyMC3 eight school example HOT 5
- Missing reference posteriors HOT 2
- Make a new release? HOT 3
- Change Stan syntax to new syntax HOT 6
- Proposal: Add `data-used` to posterior `.json` files where relevant
- Discussion: Correctness Checking Between PPLs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from posteriordb.