The two scripts prevalence and risk are pretty messy functions and should be cleaned up.

Because the hist_kwargs are updated in the wrong order, histogram settings such as the alpha value cannot globally be changed from their hardcoded values.

When we receive raw data from our collaborators, it needs to be preprocessed before we can upload it to LyProX. For that a dedicated general script would be useful that only parses some column mapping instructions for any given raw data.

Both the enhance script and the sample script (and maybe some more) access the key modalities in the params.yaml file. However, they use it for different purposes: The former combines all defined modalities into "consensus" diagnoses, while the sampling program uses all defined modalities for inference.

This clash needs to be resolved. An idea would be to use different lists of modalities for the different scripts or have it provided to the scripts via an optional argument.

I would like to refactor the scripts as they are into libraries of atomic, reusable functions that I can then again combine into versatile and declarative scripts.

Make this package compatible with version 1.0 of lymph-model.

This will represent a major version change!

For speedier computation of risks and prevalences, it can make sense to precompute prior and posterior state distributions. The new lymph API allows that via the methods state_dist() and posterior_state_dist(). Thus, this package should take advantage of that.

The output of the lyscripts data enhance command is not fully deterministic: The order of the columns varies from run to run. As far as I can tell, the content remains the same. Nonetheless, this is annoying and should be fixed.

When a function that is wrapped in the report_state decorator gets called inside a report.status context, a rich.errors.LiveError gets raised. It complains about two "live displays" running at the same time.

So, I should make sure that all such wrapped functions stand on their own and not inside rich contexts.

When running the prevalence prediction with the Unilateral model, an error is raised that says the information about the midline extension is missing.

Comparing the results of the logical OR consensus computed by lyscripts to what LyProX does, I see quite some differences.

Right now, the documentation exists always only for the latest version. But it would be helpful too look at earlier version's docs. Maybe I can adapt the respective GitHub action to enable that.

See here for some ideas.

It would be great to have the ability to log progress and intermediate results in a file. I think this should be quite straightforward with the use of rich, but extending this would also be nice.

Use numpy's seed function before starting the sampling rounds so that by providing the same seed value can reproduce the same sampling round.

I am not sure this will work, as I think I have tried this before.

The utility function generate_markdown_docs in the lyproxify.py file uses three spaces as depth of indentation for nested lists. This is wrong, it should be four spaces.

Since I have started working on this, the package rich_argparse has been published on PyPI. I should use this and maybe add some custom syntax highlighting.

It could be useful to have a command that filters datasets based on some common features. E.g., filter based on tumor location, subsite, T-category, ...

Right now lyscripts contains utilities and commands to compute predicted and observed distributions over prevalences and risks. It also defines functions to plot these computed values, e.g. when the computations have been stored as HDF5 files. What it is missing is methods to export these plots to re-read them later. This is sort of the missing link to efficiently use lyscripts as a library for computing and plotting histograms over risks and prevalences.

When an input file contains e.g. the columns max_llh and one wants to (re)compute the max_llh, it raises an exception when concatenating.

When sampling until convergence, the script does not thin out the chain by e.g. keeping only every fifth sample. This leads to repeated values when a new proposal for a walker gets rejected multiple times and hence decreases the statistical power of subsequently computed values.

Ideally, the thin_by parameter that is read from the parameter YAML file and used in the TI procedure should also apply to the convergence sampling procedure.

There are several issues with the information provided by the sampling script:

1. During burnin, it implies to know how long the sampling takes, but doesn't. It'd be better if it conveyed to the user that it samples until convergence.
2. There's always a warning about not finding the attribute _random in the ConvenienceSampler. I think this is due to a recent update of the emcee package.
3. When performing a thermodynamic integration, it only provides the value of the beta parameter. But also providing acceptance rates or maybe even mean and standard deviation of the sampled parameters would also be super helpful.

I am still using the old syntax for type hints. I can replace the typing type hints with the built-in version of Python 3.10 now.

The argparse package provides functionality for sub-commands, which I would like to implement, so that I could do stuff like

python -m lyscripts thermoint --args

and stuff like that. Essentially similar to how git has many subcommands.

Instead of complicated custom enumerators inside functions that compute likelihoods, prevalences and risks, I could simply implement them as generators. In this way, I could set up progress bars and what not outside the function generating these values.

Add an argument to the sample script that allows one to choose the number of cores to use in parallel and - if possible - disable using the multiprocessing library altogether.

When the optional argument pools is not given, the sampling crashes. This is because I forgot to handle the case where the variable npools inside the run_mcmc_with_burnin function is set to None.

While pydoc is super nice for small and straightforward projects, I think the flexibility of sphinx is needed here. E.g., there is this nice extension for automatically inserting commands and their output.

Right now, I have the subcommands generate, clean, join, enhance and split that all deal with the data before sampling. It would make sense to group them under one intermediate command data.

The enhance script crashes when two modalities have different numbers of LNLs.

While writing my thesis I implemented some nice plotting utilities using simple dataclasses and a functional approach. Copy that over to the plot commands in lyscripts.

In case two sublevels, e.g. "IIa" and "IIb", are both reported/inferred to be healthy, they override the information in the superlevel, even if that does report involvement.

After sampling and the prediction of the prevalences, the histograms.py file does not output the plotted histograms but shows following error:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/lynforigin/bin/lyscripts", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/lyscripts/__init__.py", line 123, in main
    args.run_main(args)
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/lyscripts/plot/histograms.py", line 124, in main
    draw(
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/lyscripts/plot/utils.py", line 270, in draw
    axes.hist(content.values, **tmp_hist_kwargs)
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/matplotlib/__init__.py", line 1446, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/matplotlib/axes/_axes.py", line 6944, in hist
    p._internal_update(kwargs)
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/matplotlib/artist.py", line 1223, in _internal_update
    return self._update_props(
  File "/opt/anaconda3/envs/lynforigin/lib/python3.8/site-packages/matplotlib/artist.py", line 1197, in _update_props
    raise AttributeError(
AttributeError: Polygon.set() got an unexpected keyword argument 'kwargs'

Used command: lyscripts plot histograms models/prevalences.hdf5 plots/hist_prev_ipsiI.png --names ipsiI/early ipsiI/late

Lyscripts version: 0.7.2
Lymph version: 0.4.3

Used files: used_files.zip

Workaround: Use lyscripts version 0.5.11 to plot the histograms.