mrc-ide / naomi Goto Github PK

Example: troubleshooting issue 49 for Haiti:

• User selected level 3 for model fitting
• Population data only available for levels 0-2.

This only threw an error at the model fitting stage and it was not user friendly. This should throw an error in validate_model_options().

See naomi debug 95 for Mozambique:

• Peads ART data only avalible for 2020, naomi_prepare_data() crashes for the approx() function in artnum_mf()
• TO DO: create a fix to bypass interpolation when exact quarter is requested (reference: #227)

New age group coding scheme agreed with UNAIDS and Fjelltop:

• YXXX_XXX, e.g. Y000-004 for age 0-4.
• For open ended age ranges, use YXXX_999, e.g. Y015_999 for age 15+ years.
• MXXX_XXX for age ranges, e.g. M000_017 for age 0-17 months, M018_059 for age 18 to 59 months.

Steps

• Update age group metadata table in naomi::get_age_groups().
• Update example datasets in data-raw/.
• Update model code.
• Test in hint and hintr (don't expect this to require changes, except maybe in some hintr data validation functions).
• Update country datasets

Aggregate with NAs and have some indicator (grey dot) that aggregate value is

Code changes required in naomi:

• Refactor data aggregation scripts to aggregate with NAS
• Add column to flag aggregates containing NAs

Front end changes:

• Colour dots by missing/ non missing
• Include tooltip for which area_ids are missing

Add tests to check for errors in validating and formatting calibration options

Wilford Kirungi (and others) report challenges where district-level post calibrated model results are implausible, for example ART coverage >100% for some age groups for some districts. The reason for this is the current model calibration applies a simple proportional scaling across all districts to match the target aggregate. This makes it pops

The proposed solution is to provide an option for logistic scaling of an outcome rather than proportional scaling. For example, scale the district ART coverage on a logit scale rather than the total number on ART proportionally. This will require an optimisation step because the logistic calibration is non-linear, but this should be pretty fast.

Things that need more thinking:

• What are invalid calibration options? E.g. calibrating ART number but not PLHIV could result in >100% aggregate coverage which would cause the logistic scaling to fail.
• How to handle uncertainty ranges.
• What this means for ART attendees outputs. This should be minimal impact so long as ART number input to Spectrum and Naomi are consistent.

Steps

• Split model calibration function into multiple steps. During the the model model frame preparation, a static 'calibration' table should be extracted from the PJNZ.
• Implement logistic scaling option for prevalence and ART coverage; maybe infections.
• Check if this works well in Uganda example and others.
• Think about the other details above.

A test model options list is saved here and used in the model fitting tests: https://github.com/mrc-ide/naomi/blob/master/tests/testthat/setup-run-model.R#L14

We are getting a lot of errors trickling through to staging deployments because this test options list is not consistent with the model options meta data schema in the JSON here: https://github.com/mrc-ide/naomi/tree/master/inst/metadata

We need to add a test to Naomi to check that the model options list in our tests is fully consistent with the current JSON schema specifications.

• Names of options
• Variable type of options (e.g. strings passed as strings)
• 'Required' variables only are recorded

This isn't straightforward to do because of the glue tags in the JSON schema, so the JSON can't be directly read. They need some processing to fill in the glue tags.

Adding to this issue for JSON schema validation. @Nicolas-Dolan notes that front end expect all labels to be unique. This should be checked.

• options are consistent with tests
• JSON is well formed
• Labels are unique
• Any other UI expectations for the schemas

The new ANC testing projection may be numerically unstable if no ANC programme data are included. Test what happens across many countries with different amounts of ANC testing data included.

This issue is motivated by ZWE test fit:

data <- list(
  pjnz = "depends/zwe_pjnz.zip",
  population = "depends/zwe_population_nso.csv",
  shape = "depends/zwe_areas.geojson",
  survey = "depends/zwe_survey_hiv_indicators.csv",
  art_number = "depends/zwe_art_number.csv",
  anc_testing = "depends/zwe_anc_testing.csv"
)



options <- list(
  area_scope = "ZWE",
  area_level = "2",
  calendar_quarter_t1 = "CY2016Q3",
  calendar_quarter_t2 = "CY2019Q4",
  calendar_quarter_t3 = "CY2020Q3",
  survey_prevalence = "ZWE2016PHIA",
  survey_art_coverage = "ZWE2016PHIA",
  survey_recently_infected = "ZWE2016PHIA",
  include_art_t1 = "true",
  include_art_t2 = "true",
  artattend = "true",
  artattend_log_gamma_offset = -4,
  ## anc_prevalence_year1 = 2016,
  ## anc_prevalence_year2 = 2019,
  ## anc_art_coverage_year1 = 2016,
  ## anc_art_coverage_year2 = 2019,
  no_of_samples = 1000,
  spectrum_population_calibration = "national",
  spectrum_plhiv_calibration_level = "none",
  spectrum_plhiv_calibration_strat = "sex_age_coarse",
  spectrum_artnum_calibration_level = "none",
  spectrum_artnum_calibration_strat = "sex_age_coarse",
  spectrum_infections_calibration_level = "none",
  spectrum_infections_calibration_strat = "sex_age_coarse"
)

The model converged by threw an error when simulating outputs:

converged: relative convergence (4)
Error in quantile.default(newX[, i], ...) : 
  missing values and NaN's not allowed if 'na.rm' is FALSE
> traceback()
11: stop("missing values and NaN's not allowed if 'na.rm' is FALSE")
10: quantile.default(newX[, i], ...)
9: FUN(newX[, i], ...)
8: apply(sample, 1, stats::quantile, c(0.5, 0.025, 0.975)) at outputs.R#34
7: add_stats(v, naomi_fit$mode[[varname]], naomi_fit$sample[[varname]]) at outputs.R#60
6: (function (varname, indicator, calendar_quarter, mf = naomi_mf$mf_out) 
   {
       v <- dplyr::mutate(mf, calendar_quarter = calendar_quarter, 
           indicator = indicator)
       if (!is.null(naomi_fit$sample)) {
           v <- add_stats(v, naomi_fit$mode[[varname]], naomi_fit$sample[[varname]])
       }
       else {
           v <- add_stats(v, naomi_fit$mode[[varname]])
       }
       v
   })(dots[[1L]][[2L]], dots[[2L]][[2L]], dots[[3L]][[1L]], dots[[4L]][[1L]])
5: mapply(FUN = f, ..., SIMPLIFY = FALSE)
4: Map(get_est, names(indicators_anc_t3), indicators_anc_t3, naomi_mf$calendar_quarter3, 
       list(naomi_mf$mf_anc_out)) at outputs.R#147
3: extract_indicators(naomi_fit, naomi_mf) at outputs.R#278
2: output_package(fit, naomi_data) at run-model.R#136
1: hintr_run_model(data, options)

This error occurs when the model simulates and NA value for an indicator in producing the outputs package.

converged: false convergence (8)
Error in quantile.default(newX[, i], ...) : 
  missing values and NaN's not allowed if 'na.rm' is FALSE

This error should be caught and return which indicator threw the error.

Now these are split over model options and calibration options we should update the help text

The Naomi output package has a function get_meta_indicator() that to produce an indicator metadata table used to construct the Naomi output package.

There is another function get_metadata() that retrieves indicator metadata from inst/metadata/metadata.csv, which is used to provide indicator display information to HINT.

These should be consolidated into one resource. They should also be consistent with the metadata agreed with Fjelltop for the ADR: https://github.com/fjelltopp/unaids_data_specifications/tree/development/meta_indicators. However, I don't think they are using those metadata at the moment so we can dictate changes to those tables.

Spectrum-DataPack-indicator_code-Map_20201027.xlsx

Revise the handling of survey data to be constructed as aggregation of age/sex/areas rather than picking indices, similar to ART and ANC testing data.

Changes required:

• Update survey data likelihood specification to use aggregation matrices
  • This probably requires a bit of generalisation of the code because multiple surveys can be specified per time point.
• Handle selection of survey data -- need to intelligently choose most granular data available for a specific survey.
• Constrain age/sex random effects intelligently if no age/sex stratified data are available. Probably this involves removing the age/sex random effects and using only Spectrum offsets.

Update spectrum_calibration.csv produced in naomi outputs package. The following updates to the current output are required for the front end calibration results visualization endpoint:

• add spectrum region codes and spectrum region names to be displayed on visualization x-axis
• add rates indicators: HIV prevalence, art coverage and HIV incidence
• add calibrated results for all indicators: currently only present for population
• add coarse disaggregates for all indicators: currently disaggregated by 5-year age bands

Write a check for whether any area/sex/age group combinations have missing programme data. The check is slightly complex because the areas are represented by ad hoc aggregations

Checks:

1. Require that every stratification is represented by a row in the data.
2. Provide an option whether to allow NA values or not. Probably a on model option page for this.

Steps:

1. Filter to the finest area / sex / age group stratification.
2. Map the observed aggregations to the fine stratifications.
3. anti_join() to find any that are missing

Report back as (1) entire area missing data, (2) area / sex / age stratifications missing data.

https://s3.us-west-2.amazonaws.com/secure.notion-static.com/86b2146f-e1db-482d-ad65-1a1fbb6d07d6/hyperpar_plots.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAT73L2G45EIPT3X45%2F20220825%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220825T122135Z&X-Amz-Expires=86400&X-Amz-Signature=39bd136952516700e963e020290cf60b8c3d8a0ed1e208523aaccacb5fc8bdd3&X-Amz-SignedHeaders=host&response-content-disposition=filename%20%3D%22hyperpar_plots.pdf%22&x-id=GetObject

This error message would be more helpful if it stated which region code was missing from the PJNZ.

This will be in the validate_model_options() function.

@rtesra -- do you want to take a crack at this?

The model is still trying to estimate all ART coverage parameters If no data about ART coverage are available -- either number on ART or survey ART coverage. This causes the model to explode.

Need some combination of:

• Constrain model parameters if no ART data are included.
• Include national number on ART from Spectrum if no district ART data are included.

See troubleshooting-21 for an example of issue.

Currently internally the model interpolates ART data even if the exact selected quarter is included in the ART dataset.

This has no effect on the data selected -- the interpolation simply returns the input values. However the interpolation function requires data at two time points, and so throws an error if only one time point is uploaded, which is unnecessary..

Solution: update select data function to not interpolate if the exact time point is found.

Example: Chad, naomi troubleshooting 54

Making notes here as reviewing report draft:

Small fixes:

• Swap order of UNAIDS and Naomi logos in header? What do you think?
• Change title 'District HIV estimates' to 'Subnational HIV estimates'
• Add commas if multiple survey IDs (paste(..., collapse = ", "))
• Convert calibration option labels to human readable. Can use the same labels as specified for the drop downs in Naomi web tool.
• Indicate whether ANC testing and ART data were included and which years if so.
• Remove line header anchors. (Default changed in recent update of knitr. There's an option to specify in the YAML header to revert to old behaviour)
• Adjust number formatting in table to be consistent with inst/metadata/metadata.csv specifications
  - One decimal digit for prevalence, ART coverage, and incidence per 1000
  - PLHIV and number on ART rounded to nearest 100
  - Infections rounded to nearest 10
• Sort table by area_sort_order

Medium effort:

• Update date in header to reflect the reference data for estimates and date produced. E.g.: September 2018 (report 27 November 2020).
  I think that doing this will involve restructuring the report so most of the data inputs are done outside the report and
  passed as parameters, so it's a bit of effort.

Currently spectrum calibration level gives options "None", "National", "Subnational" for all cases. Ideally this should only show "None" or "National" if it's a national Spectrum file. This endpoint can be delivered based on if the all(spectrum_region_code == 0) in the area hierarchy.

This applies to the Population calibration on the Model options page, and all four calibration options on the Model calibration page.

• Change to warning -> currently reads as error
• Add some more information on what false convergence means (model not fit well, results should be interpreted with caution)

PR:
#402

• UNAIDS request for 1 decimal digit on prevalence and ART coverage esima

Notes below from @Nicolas-Dolan:

Below are some notes/suggested changes to the metadata that I mentioned. Let me know if you want to discuss.

Note: the frontend currently ignores scale and accuracy values if format is set to '%'; let me know if scaling/accuracy still needs to be taken account in these cases

I take it the reason for some colour scales being inverted is dark is bad and light is good. Suggest population use a different colour scheme as it is neutral.

ART number (attending): max only 1 - needs to be much higher (highest value in dataset 65761)

Population, PLHIV, and New infections: max seems to be too high (at least for Malawi dataset; although perhaps appropriate at the country level)

HIV incidence and Proportion recently infected: format should be '0.00%' not '0'; these also have scale and accuracy values which could be set to 1 and null respectively (they will be ignored by the frontend anyway)

HIV incidence will need its min/max values changed to 0/1 if it is a percentage (max could be 0.5 if you want to cap at 50%)

• Update in inst/metadata/metadata.csv for changes in Naomi webapp
• Update in inst/report/summary_report.Rmd for changes in @rtesra outputs report
• Send examples to UNAIDS for review and confirm.

The following crude translations were added with Google translate in PR #157.

    "ANC_CLIENTS": "Clients de la CPN",
    "ANC_PLHIV": "Participants séropositifs aux CPN",
    "ANC_ALREADY_ART": "Les participants à l'CPN déjà sur ART",
    "ANC_ART_NEW": "Initiatives ART à l'CPN",
    "ANC_KNOWN_POS": "CPN connu positif",
    "ANC_TESTED_POS": "CPN testé positif",
    "ANC_TESTED_NEG": "CPN testé négatif",
    "INDICATOR_LABEL_ANC_CLIENTS": "Nombre total de participants au premier CPN",
    "INDICATOR_LABEL_ANC_PLHIV": "Nombre de femmes séropositives fréquentant les CPN",
    "INDICATOR_LABEL_ANC_ALREADY_ART": "Nombre déjà sous TAR avant la première visite prénatale",
    "INDICATOR_LABEL_ANC_ART_NEW": "Nombre de femmes enceintes initiées sous TAR au CPN",
    "INDICATOR_LABEL_ANC_KNOWN_POS": "Nombre de participants prénatals séropositifs au courant de leur statut avant la première visite prénatale",
    "INDICATOR_LABEL_ANC_TESTED_POS": "Nombre de participants séropositifs aux CPN testés aux CPN",
    "INDICATOR_LABEL_ANC_TESTED_NEG": "Nombre de participants aux CPN séronégatifs testés aux CPN"

    "UNTREATED_PLHIV_NUM": "PVVIH non sous TAR",
    "AWARE_PLHIV_PROP": "Proportion de PVVIH conscients",
    "UNAWARE_PLHIV_NUM": "Nombre de PVVIH non informés",
    "INDICATOR_LABEL_UNTREATED_PLHIV_NUM": "Nombre de PVVIH non sous TAR",
    "INDICATOR_LABEL_AWARE_PLHIV_PROP": "Proportion de PVVIH conscients de leur statut VIH positif",
    "INDICATOR_LABEL_UNAWARE_PLHIV_NUM": "Nombre de PVVIH non informés de leur statut VIH positif"    
    "OPTIONS_CALIBRATION_ADJUST_TO_SPECTRUM_AWARE_LABEL": "Ajustement au nombre conscients de leur statut VIH de Spectrum",
    "OPTIONS_CALIBRATION_ADJUST_TO_SPECTRUM_AWARE_HELP": "Étalonner sur la moyenne totale de Spectrum conscients de leur statut VIH",

   "OPTIONS_USE_SURVEY_AGGREGATE_LABEL": "Utiliser un ensemble de données d'enquête agrégées",
   "OPTIONS_USE_SURVEY_AGGREGATE_HELP": "L'ensemble de données d'enquête téléchargé doit être sous-défini exactement aux données à utiliser dans le calibrage du modèle."

These should be verified. Some might be available in the French version of ADR shell tables as they correspond to existing ANC testing data input columns.

Steps

• Revise ANC bias model to add bias before aggregating to all ages.
• Revise ANC likelihood specification to use same aggregation matrix approach as ART number outputs
• Implement HIV subfertility by age as part of ANC cascade model in Naomi.
• Consider how to calibrate to total counts. Revise the likelihood to Poisson multinomial?
• Review age specific ANC cascade against PEPFAR fine age reporting data.
• Project ANC testing cascade outputs to 'time 3'.
• Add ANC cascade outputs to Data Pack export.

To be used in orderly tasks and debugging (as opposed to calling with :::)

• Change default 'current estimate period' to Dec 2020
  • Confirm with UNAIDS December 2020 or September 2020 for first workshop
• Move 'future projection' calendar quarter up to first block.
• Add option for number of quarters reflected in ANC programme data input
  • Confirm with UNAIDS if we want "number of months" or "number of quarters"
• Add select boxes for using Kish design effect
• Pre-populate most recent survey calendar quarter midpoint (@r-ash please help)

• Calculate 15-49 offset between ANC and population age\sex matched prevalence for ANC matched to population prevalence and art coverage plots
• Reconfigure comparison report for edge cases where different surveys are selected for ART coverage and HIV prevalence (Cameroon)

We need a user-friendly tool for reviewing ANC and ART programme data inputs.

Minimal functionality:

• User inputs URL for the ADR data package and the ADR API key.
• Tool downloads the relevant datasets.
• Displays the Glaubius data validation plots, for example on several tabs.

It would be nice to be able to quickly and easily extend for other checks in addition to the core Glaubius plots. (For example, checks I often looked at during workshop debugging scripts.)

This tool could be done in an Rmarkdown document or a shiny app. Shiny app is the preferred format from UNAIDS. In either format, basic interactivity, e.g. mouse over tooltips, should be easy to include.

More advanced features to be considered later based on user feedback.

• Visually flagging suspicious data.
• Filtering data, especially filtering by the area hierarchy
• Aggregating and disaggregating data for inspection.
• Linked plot and table viewing.
• Data editing and re-saving to ADR.

This tool should probably be in a separate package and repo from the Naomi model.

Harmonise the indicator names used in the input datasets and the model outputs when they report on the same thing.

survey_hiv_indicators dataset

• prev -> `prevalence
• artcov -> art_coverage

art_number dataset

• current_art -> art_current or art_number (check w/ Fjelltop repo what we agreed)
• Add the art_new column

anc_testing dataset

• Change the ancrt_* prefixes to anc_*.

Steps

• Update dataset parser functions (read_*())
• Update testing datasets in Naomi package
• Update model frame setup functions
• Test in hintr and hint
• Update country datasets

The model calibration options are recorded in the HTML summary report as the same values every time, irrespective of selection on the calibration page:

Reported by @osymandius

I think the option IDs in this test might be scrambled.

The _level options should be "none", "national", or "subnational", and _strat should be the sex/age group options.

          @rtesra -- One more request on this:

• Make district borders thinner on the chloropleth plot.

This might make the differences between districts stand out m ore.

See Naomi debug 93 for Mali:

• Model gave a gradient function error during fit_tmb() when ART data was included with data missing (NAs) for two regions.
• Model fit after filtering ART dataset to exclude regions with missing data

Update PLHIV outputs to include following age bands:

• <01
• 01-04
• 05-09
• 10-14
• 15-17
• 18-20

@r-ash: Now that the 'validate' button on the model options page is active, can we remove validate_model() from hintr_run_model()?

Example 2023 debug # 50, TZA:

• (mf_model[["population_t1"]]) → Population data not available for selected area level

• Update calc_survey_hiv_indicators() to include column for Kish DEFF
• Update demo datasets
• Update likelihood and model options to provide option for using Kish DEFF
• Update country survey datasets
• Compare results across countries with Kish DEFF

Adding a bit more to the desired spec on this:

• Select most recent household survey for prevalence based on survey_mid_calendar_quarter
• Select most recent household survey for art_coverage if data exist
• If ANC data exist in the most recent survey year [defaulted for prevalence], default select that year for Time 1 (both prevalence and ART coverage). If no data in that year, do not select default
• If ANC data exist in the year 2020, preselect that for 'Time 2' in ANC clients, ANC prevalence, and ANC coverage.

This requires:

• Adding a glue tag in inst/metadata/survey_run_options.json
• Add calculation of the endpoint in hintr.
• Add a test in hintr
• Check that it works in staging server.

This code checks to see if ART data has been multiply entered for an area_id, for example entered at the district level and also at the provincial level:

It should also check if a sex or age group has been duplicated. For example, data should not be entered for both male/female and both sexes in a particular age group, or for age 15+ and all ages.

• Expand check for sex and age dimensions
• This check should also be applied to ANC testing data now
• This check should be performed as part of input data validation, not at model fitting step.

I recall writing some code that does that check...but I can't remember where and I can't think of where it would be if not in this ART input data validation.

Reproducible example: this error of both male/female and both reporting in the Zimbabwe datasets should be identified:

setwd("~/Imperial College London/Naomi Support - WP - General/7/")

source("../debug-functions.R")
jobid <- "b64d9f9f0133fb63e4f6049a76cb1090"

devtools::load_all("~/Downloads/naomi")

inp <- hintr_inputs_ready(jobid)

data <- format_data_input(inp$data)
options <- format_options(inp$options)
validate_model_options(data, options)

naomi_data <- naomi_prepare_data(data, options)

This throws a different error right now. But it should error for the inputs issue. And ideally error on validate_model_options().

In the test data constructed, what is a_output_full$indicators and why does it not have national level outputs?

> a_output_full$indicators %>% count(area_id)
# A tibble: 15 × 2
   area_id          n
   <chr>        <int>
 1 MWI_1_1_demo  3888
 2 MWI_2_1_demo  3888
 3 MWI_3_1_demo  3888
 4 MWI_3_2_demo  3888
 5 MWI_3_3_demo  3888
 6 MWI_3_4_demo  3888
 7 MWI_3_5_demo  3888
 8 MWI_3_6_demo  3888
 9 MWI_4_1_demo  3888
10 MWI_4_2_demo  3888
11 MWI_4_3_demo  3888
12 MWI_4_4_demo  3888
13 MWI_4_5_demo  3888
14 MWI_4_6_demo  3888
15 MWI_4_7_demo  3888
> 

Reproduce by:

devtools::load_all("naomi")
source("naomi/tests/testthat/setup-model-frame.R")
a_output_full$indicators 

For example: Ngabwe District in Zambia -- 23 observations, 8% prevalence --> 2 cases, but only 1 survey cluster so the SE is degenerate.

Approximate awareness of HIV status ('first 90') based on the applying the % aware among the untreated population from the national Shiny90 estimates to district level.

Steps

• Add functions to parse .shiny90 digest file to create a table of age/sex outputs during model frame preparation step.
• Add awareness of status to model indicators metadata table.
• Add function to calculate % aware of status by district. Probably this gets run routinely after model calibration.
• Add awareness of status calibration options.
• Test in hint and hintr.
• Test with PJNZ slim export including .shiny90 digest.

Remove the following redundant columns from the outputs package:

• age_group_id

• indicator_id

• quarter_id

• Remove from naomi outputs

• Update hintr to be consistent

• Confirm format to be okay for avenir