modeloriented / modeldown Goto Github PK

modelDown generates a website with HTML summaries for predictive models

Home Page: https://ModelOriented.github.io/modelDown/

R 87.78% TeX 12.22%

modeldown's Introduction

modelDown

modelDown generates a website with HTML summaries for predictive models. Is uses DALEX explainers to compute and plot summaries of how given models behave. We can see how well models behave (Model Performance, Auditor), how much each variable contributes to predictions (Variable Response) and which variables are the most important for a given model (Variable Importance). We can also compare Concept Drift for pairs of models (Drifter). Additionally, data available on the website can be easily recreated in current R session (using the archivist package).

pkgdown documentation: https://ModelOriented.github.io/modelDown/

An example website for regression models: https://mi2datalab.github.io/modelDown_example/

Getting started

Do you want to start right now ? Check out our getting started guide.

Or just simply install it like below:

Stable version: devtools::install_github("ModelOriented/modelDown")

And if you want to get the latest changes:

Development version: devtools::install_github("ModelOriented/modelDown@dev")

Contributing

If you spot a bug or you have a feature proposal feel free to create an issue in this repository. We are also open to contributions in a form of pull requests. Just follow steps below:

Open a new issue (specify an issue type as a label - a bug or an enhancement).

Additionally you can:

Start a new branch from the dev branch. It should be named bugfix/XX-short-description or feature/XX-short-description where XX is an issue number.
Create commits with descriptive messages starting with #XX.
Create a pull request of the created branch to the dev branch.
Wait for a review from one of the modelDown maintainers.

Help us build better software!

Index page

Index page presents basic information about data provided in explainers. You can also see types of all explainers given as parameters. Additionally, summary statistics are available for numerical variables. For categorical variables, tables with frequencies of factor levels are presented.

Auditor

Module shows plots generated by auditor package.

Drifter

Results of drifter package are displayed in this tab. In order to see the comparison charts, you have to provide pair of explainers as parameters (for example: list(explainer_glm_old, explainer_glm_new)).

Model Performance

Module shows result of function model_performance.

Variable Importance

Output of function variable_importance is presented in form of a plot as well as a table.

Variable Response

For each variable, plot is created by using function variable_response. Plots can be easily navigated using links on the left side. One can provide names of variables to include in the module with argument vr.vars (if argument is not used, plots for all variables of first explainer are generated).

Loading data in R

In each tab you can find links with R commands. If you execute them, you can load relevant objects into current R session (archivist package is necessary). By default data is stored and loaded from local repository. If you wish to store data on GitHub repository, please provide argument remote_repository_path. After generating modelDown website, repository folder must be placed under this path.

Acknowledgments

Work on this package is financially supported by Warsaw University of Technology, Faculty of Mathematics and Information Science.

modeldown's People

Stargazers

Watchers

Forkers

spicer23 vishalbelsare gridl kromash imarcello kant abenavs a3digit queleanalytics pragyanaischool subodhk26 michkam89 datastrategist stjordanis

modeldown's Issues

Update link to technical documentation

ModelOriented instead of pbiecek

Add plot description

Each plot (or plot type?) should also contain links to provide information on how to read the plot, what it presents, how it was generated:

one link with business description
one link with technical description (from DALEX package)

Broken donttest example

We need to update modelDown donttest examples.

Full logs from command check --as-cran --run-donttest are here: https://www.stats.ox.ac.uk/pub/bdr/donttest/modelDown.out

Save plots as svg images

Possibly with parameter to enable plots generation as png images (what's the default value?)

Check efficiency - initial implementation slow on Chrome

Session info

Use archivist to save session state

In case of problem with archivist intergration, add link to R's sessionInfo (as text file)

Description of package (as an article)

Write description of how to use package with new features as blog post.

devtools::check() problems

A.
Consider adding importFrom("utils", "capture.output", "tail") - roxygen
B.
Undefined global functions or variables: DEFAULT_DEVICE DEFAULT_FONT_SIZE
(from config.R file - should it be removed and variables moved to R script?)

Typo in 'Mateusz Urba?ski'

in pkgdown and DESCRIPTION

The Variable importance section

Please add the table produced by 'print()' function for variable importance

Archivist link generation

output_folder should be ignored, for example:
for parameters
remote_repository_path = "MI2DataLab/modelDown_example/docs",
output_folder = "../modelDown_Titanic_example"

correct link: archivist::aread("MI2DataLab/modelDown_example/docs/repository/e617d10e1606257c56ba7d192c5b8fe0")

current link:
archivist::aread("MI2DataLab/modelDown_example/docs/../modelDown_Titanic_example/repository/e617d10e1606257c56ba7d192c5b8fe0")

Add date and time to index.html

"Generating variable_response..." fails with Error in Error in seq.default(from = min(y, na.rm = TRUE), to = max(y, na.rm = TRUE), : 'from' must be a finite number

Hello,
Running the modelDown_example script raise the following error:

[1] "Generating variable_response..."
Variable district  is of the class factor. Type of explainer changed to 'factor'.
Variable district  is of the class factor. Type of explain
er changed to 'factor'.
Variable district  is of the class factor. Type of explainer changed to 'factor'.
Error in pairs[[whichMax]] : 
  attempt to select less than one element in get1index
Called from: mergePairLRT(fm, successive, fmList$factor, fmList$model)

The issue is maybe related to DALEX::variable_response() but is easily reproducible here.

Best regards,
Christophe
Stack-trace

Error in pairs[[whichMax]] : 
  attempt to select less than one element in get1index 
21.
mergePairLRT(fm, successive, fmList$factor, fmList$model) 
20.
mergeLRT(fm, successive) 
19.
mergeFactors.default(preds_combined$scores, preds_combined$level, 
    abbreviate = FALSE) 
18.
mergeFactors(preds_combined$scores, preds_combined$level, abbreviate = FALSE) 
17.
variable_response(explainer, variable_name, type = type) at generator.R#17
16.
FUN(X[[i]], ...) 
15.
lapply(explainers, function(explainer) {
    variable_response(explainer, variable_name, type = type)
}) at generator.R#17
14.
FUN(X[[i]], ...) 
13.
lapply(types, function(type) {
    lapply(explainers, function(explainer) {
        variable_response(explainer, variable_name, type = type)
    }) ... at generator.R#16
12.
make_variable_plot(variable_name, types, explainers, img_folder, 
    options) at generator.R#35
11.
FUN(X[[i]], ...) 
10.
lapply(variables, make_variable_plot_model, explainers, img_folder, 
    options) at generator.R#51
9.
generator_env$generator(explainers, options, file.path(output_folder, 
    "img")) 
8.
FUN(X[[i]], ...) 
7.
lapply(modules_names, function(module_name) {
    print(paste("Generating ", module_name, "...", sep = ""))
    generator_path <- system.file("extdata", "modules", module_name, 
        "generator.R", package = "modelDown") ... 
6.
generateModules(modules, output_folder, explainers, options) 
5.
modelDown(explainer_lm, explainer_rf, explainer_gbm, explainer_svm, 
    modules = c("model_performance", "variable_importance", "variable_response", 
        "prediction_breakdown"), output_folder = "modelDown_example", 
    pb.observations = c(161, 731, 2741, 4454), vr.type = "ale",  ... at modelDown_example.R#39
4.
eval(ei, envir) 
3.
eval(ei, envir) 
2.
withVisible(eval(ei, envir)) 
1.
source("~/R/modelDown_example.R", echo = TRUE)

sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=fr_FR.UTF-8       LC_NUMERIC=C               LC_TIME=fr_FR.UTF-8        LC_COLLATE=fr_FR.UTF-8     LC_MONETARY=fr_FR.UTF-8    LC_MESSAGES=fr_FR.UTF-8   
 [7] LC_PAPER=fr_FR.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] bindrcpp_0.2.2      kableExtra_0.9.0    ggplot2_3.1.0       modelDown_0.1.1     e1071_1.7-0         gbm_2.1.4           DALEX_0.2.4         breakDown_0.1.6    
[9] randomForest_4.6-14

loaded via a namespace (and not attached):
 [1] httr_1.3.1         viridisLite_0.3.0  splines_3.5.1      gtools_3.8.1       shiny_1.2.0        assertthat_0.2.0   expm_0.999-3       sp_1.3-1          
 [9] highr_0.7          pdp_0.7.0          LearnBayes_2.15.1  pillar_1.3.0       lattice_0.20-38    glue_1.3.0         digest_0.6.18      RColorBrewer_1.1-2
[17] promises_1.0.1     rvest_0.3.2        colorspace_1.3-2   htmltools_0.3.6    httpuv_1.4.5       Matrix_1.2-15      plyr_1.8.4         klaR_0.6-14       
[25] pkgconfig_2.0.2    questionr_0.7.0    gmodels_2.18.1     purrr_0.2.5        xtable_1.8-3       mvtnorm_1.0-8      scales_1.0.0       gdata_2.18.0      
[33] whisker_0.3-2      later_0.7.5        tibble_1.4.2       proxy_0.4-22       combinat_0.0-8     ggpubr_0.2         withr_2.1.2        ALEPlot_1.1       
[41] agricolae_1.2-8    lazyeval_0.2.1     survival_2.43-3    magrittr_1.5       crayon_1.3.4       mime_0.6           deldir_0.1-15      evaluate_0.12     
[49] nlme_3.1-137       MASS_7.3-51.1      xml2_1.2.0         class_7.3-14       tools_3.5.1        hms_0.4.2          stringr_1.3.1      munsell_0.5.0     
[57] cluster_2.0.7-1    compiler_3.5.1     rlang_0.3.0.1      grid_3.5.1         rstudioapi_0.8     miniUI_0.1.1.1     labeling_0.3       rmarkdown_1.11    
[65] boot_1.3-20        gtable_0.2.0       reshape2_1.4.3     AlgDesign_1.1-7.3  R6_2.3.0           yaImpute_1.0-30    gridExtra_2.3      knitr_1.20        
[73] dplyr_0.7.8        bindr_0.1.1        factorMerger_0.3.6 spdep_0.8-1        readr_1.2.1        stringi_1.2.4      Rcpp_1.0.0         spData_0.2.9.6    
[81] tidyselect_0.2.5   coda_0.19-2 ```

First tab design

First tab should present information on model correctness. Looks should be improved. Plots from model performance and from auditor should be ordered so that it's easy to analyze model.

Box plot from model performance should be placed before plot with distribution of residuals

Add auditor features

Add results from auditor package to first tab

Model Ranking Plot as first plot.

Autocorrelation, LIFT chart, model correlation, ROC, RROC.

plotScaleLocation?

Transfer moedDown from MI2DataLab to ModelOriented

Let's move the MI2DataLab/modelDown package to ModelOriented/modelDown. All DrWhy projects will be in one organization.
Suggested date for the movement: August 21st

Downloading generated data

Each plot should contain link that allows downloading data frames (DALEX results) used to generate the plot.

No need to save the plot itself.

Do we want to save the data only as R object or as text as well?

Example in doc could work on DALEX dataset and site needs to be updated

Storing results

Are the results of computations stored in any way? For example, PDP takes some time to compute and it would be nice to have access to the results

Update README.md

New plots in https://mi2datalab.github.io/modelDown_example/index.html

Add info about grant

Please add example screenshots to the README.md

Screenshots for each view with short (1 sentence) comments

Apply MI2DataLab css styles

https://github.com/mi2-warsaw/MI2template/tree/master/inst/pkgdown

Please add Roxygen documentation

there is only one function, but it should be described along with all its parameters

packages can't install

To be honest the result through the web visualization, this idea is very good, but your package loading will take a long time, there are a lot of related packages at the same time, I found that after carefully explore the package you just use the DALEX function through the R - markdown displayed in package, I can understand for plagiarism? Also, your package is hard to download on AZURE or on your local computer, it takes a long time, and it's basically unusable.

Add pkgdown website

For R packages you can automatically add pkgdown website with pkgdown package
Find some examples here: https://github.com/mi2-warsaw/MI2DataLab_Seminarium/tree/master/2017_11_28_pkgdown

css styles

Please not that the footer is still grey.
(it's in the class="footer")

New theme for DALEX plots is on GitHub here: https://github.com/pbiecek/DALEX/
So you can also use color codes from
https://github.com/pbiecek/DALEX/blob/master/R/theme_drwhy.R

Template for pkgdown generation

https://github.com/mi2-warsaw/MI2template

Custom parameters for explainer functions

It would be great to have an option to pass arguments to explainers.

For example, I would like to generate modelDown with my own loss function for a variable_importance() function.
An analogy for doing something like this, but with modelDown:

library("breakDown")
library("randomForest")
HR_rf_model <- randomForest(factor(status == "fired")~., data = HR, ntree = 100)
explainer_rf  <- explain(HR_rf_model, data = HR, y = HR$status == "fired")

custom_loss_auc <- function(y, yhat) {
  1 - mltools::auc_roc(yhat, y)
}

vd_rf <- variable_importance(explainer_rf, loss = custom_loss_auc)
vd_rf

How about adding parameter params that takes a list with names corresponding to explainer's parameters? For the example above it would be something like this:

modelDown(explainer_rf, params = list("variable_importance" = list(loss = custom_loss))

Empty Explainers section

In the index.html page on the right side, there is an Explainers section, but it's empty

Add package to CRAN

Variable response order

Variables in 'Variable Response' tab should be order from most to least important - as in output of variable importance.

What if we have many explainers? Which order should be used then?

Upgrade package version to 0.1 or higher (in the DESCRIPTION)

Archivist link placement

Archivist link is presented differently across modules:
In auditor - before the plot, with label: 'Load:'
In Model Performance and other - under the plot, with label: 'Get this object:'

@pbiecek which one is better?

Fill up DESCRIPTION

Lots of missing information in DESCRIPTION
please update
Do you really need R in 3.5?
You can start with R 3.0 in the requirements

Wrong link at the bottom

Currently it links to https://gitlab.com/Romaszko/modeldown

The footer 'Site built with modelDown.' should be visible in all pages

Error 'factor_columns[[i]]': index is out of range

example code

library(DALEX)
library(live)
library(modelDown)
proba <- e1071::svm(quality~.,data=wine)
modelDown(DALEX::explain(proba, data = wine, y = wine$quality))

output

[1] "Generating model_performance..."
[1] "Generating variable_importance..."
[1] "Generating variable_response..."
[1] "Generating prediction_breakdown..."
Błąd w poleceniu 'factor_columns[[i]]':indeks jest poza granicami

DALEX on custom stacked models

Pseudo-reprex below to illustrate workflow.

There are 2 stages of stacking, and below is abbreviated to final stage.

# input data for prediction; this data are themselves a result of stacked model
df <- tibble::tribble(
  ~x1, ~x2, ~true,           ~pred1,           ~pred2,           ~pred3,           ~pred4,           ~pred5,
  "0016",   1, 11255, 9782.06546666667, 8226.73783726366, 8423.53411898339, 7663.85714285714, 7778.32234611454,
  "0016",   2, 10155, 9917.16225000001,  7390.2726470072, 7548.50621212894, 6011.57142857143,  7020.0197927677,
  "0016",   3,  9905, 8365.66048333333, 4748.35733132711, 4897.40398331136, 5625.14285714286, 5197.59820269678,
  "0026",   1,  9569, 10542.7790333333, 12448.8281473898, 12982.2853847065, 9529.42857142857, 9913.60100542533,
  "0026",   2, 15004,      12332.88455, 13118.3179554928, 13490.4519001908, 9449.14285714286, 9782.48187764126,
  "0027",   1,  4623, 6228.92556666668, 7901.02224985066,  8072.3059097473, 7663.85714285714,  7564.7019858157,
  "0027",   2,  3666, 3902.33416666666, 5351.58779239503, 5501.55032427708, 5757.85714285714, 5791.90612060224,
  "0027",   3,  2046, 3730.91108333333, 5405.90164588071, 5431.22100425988,             5700, 5574.85787520228,
  "0345",   1,  7848, 7911.66811666667, 7332.14726332333, 7535.03388134704, 8428.85714285714, 7504.20919309283,
  "0345",   2,  5594,        6249.8431, 5302.09068924222, 5602.24650648537,             6253, 5936.17306199591,
  "0348",   1,  6118,        5888.9112,  6782.1549012783, 6983.85792156352, 7145.28571428571, 6996.64665890851,
  "0348",   2,  4115,        4655.3621, 4061.92478416692,  4339.3944039624, 5379.71428571429, 5201.36079952954,
  "0348",   3,  3792, 4703.56786666666, 4862.77758785772, 4886.36623749198, 5413.85714285714,  5316.2047603152,
  "1000",   1,  9982,        8894.2428, 8950.05680053561, 8724.27457157357, 7643.14285714286, 8427.52273508174,
  "1000",   2,  4218, 5103.73553333333, 6755.30317981863, 6492.15505744351,             7836, 6900.52725335413,
  "1022",   1,  9021, 8966.84941666667, 8921.14926298024, 8514.45660876879, 8590.57142857143, 8566.07119574923,
  "1022",   2, 11692, 10205.8180333333, 8895.88440879051, 8417.59814231434, 8185.85714285714, 8225.60579235643,
  "1022",   3,  9420, 9664.82173333334, 9422.99681882565, 8835.71873031759, 7853.57142857143, 8126.76078652109,
  "1022",   4,  6850, 7419.07043333333, 8995.48869657391, 8194.63910112673, 7604.14285714286, 7815.14405713875,
  "1022",   5,  6850, 7419.07043333333,  8817.8438463534, 7883.22080414475, 6846.14285714286, 7515.84608489043
)

# model list for stacking
md <- list(rf,
           pca,
           svm,
           enet)

model_pred_stack <- function(df, md) {
  
  # iterate over list of models in md, and average prediction
  temp <- 0
  for (i in 1:length(md)) {
    temp <- temp + predict(md[[i]], df)
  }
  temp <- temp / length(md)
  return(temp)
}

model_pred <- model_pred_stack(df = df, md)

# with DALEX, have to loop over list of models one by one, which doesn't reflect intention of stacking; otherwise, modelDown will complain
explain_stacked <- explain(
  md[[1]],
  data = df,
  y = df$true,
  label = "stacked"
)

modelDown(explain_stacked,
          device = "svg",
          output_folder = "output_data/modelDown_stacked")

# passing list of models; modelDown fails to generate diagnostics, aside from data description
explain_stacked <- explain(
  md,
  data = df,
  y = df$true,
  label = "stacked"
)

modelDown(explain_stacked,
          device = "svg",
          output_folder = "output_data/modelDown_stacked")

Transform sample.R into examples for modelDown function

Add examples to modelDown

Add travis support

To maintain the code please add travis support
You can find some hints here: https://github.com/mi2-warsaw/MI2DataLab_Seminarium/tree/master/2017_12_19_travis

Table presented in the index.html is not working for factors

sales and salary are factor variables, instead of averages it would be better to show counts

404 - Do you want to start right now ? Check out our getting started guide.

The link to the getting started guide 404's. Is there another location for this or has this repository been sunsetted?

Error in file.copy

I was executing your examples: https://htmlpreview.github.io/?https://raw.githubusercontent.com/kromash/modelDown/master/docs/reference/modelDown.html

and got following error:

> modelDown::modelDown(explainer_ranger, explainer_glm) #all defaults
Error in file.copy(css_files_paths, to, recursive = TRUE, overwrite = TRUE) : 
  invalid 'from' argument

Here is the traceback:

> traceback()
3: file.copy(css_files_paths, to, recursive = TRUE, overwrite = TRUE)
2: copyAssets(system.file("extdata", "template", package = "modelDown"), 
       output_folder)
1: modelDown::modelDown(explainer_ranger, explainer_glm)

and here is my session info

> devtools::session_info()
Session info ---------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.4.4 (2018-03-15)
 system   x86_64, darwin15.6.0        
 ui       RStudio (1.1.442)           
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       Europe/Warsaw               
 date     2018-05-31                  

Packages -------------------------------------------------------------------------------------
 package       * version    date       source                                 
 agricolae       1.2-8      2017-09-12 cran (@1.2-8)                          
 ALEPlot         1.0        2017-11-13 CRAN (R 3.4.2)                         
 AlgDesign       1.1-7.3    2014-10-15 CRAN (R 3.2.0)                         
 assertthat      0.2.0      2017-04-11 CRAN (R 3.4.0)                         
 backports       1.1.2      2017-12-13 cran (@1.1.2)                          
 base          * 3.4.4      2018-03-15 local                                  
 base64enc       0.1-3      2015-07-28 CRAN (R 3.2.0)                         
 bindr           0.1.1      2018-03-13 CRAN (R 3.4.4)                         
 bindrcpp        0.2.2      2018-03-29 CRAN (R 3.4.4)                         
 BiocInstaller   1.16.5     2015-05-20 Bioconductor                           
 boot            1.3-20     2017-08-06 CRAN (R 3.4.4)                         
 breakDown     * 0.1.6      2018-05-17 local (pbiecek/breakDown@NA)           
 callr           2.0.3      2018-04-11 CRAN (R 3.4.4)                         
 cluster         2.0.7-1    2018-04-09 CRAN (R 3.4.4)                         
 coda            0.19-1     2016-12-08 cran (@0.19-1)                         
 colorspace      1.3-2      2016-12-14 CRAN (R 3.4.0)                         
 combinat        0.0-8      2012-10-29 CRAN (R 3.1.0)                         
 commonmark      1.4        2017-09-01 cran (@1.4)                            
 compiler        3.4.4      2018-03-15 local                                  
 crayon          1.3.4      2017-09-16 CRAN (R 3.4.1)                         
 curl            3.2        2018-03-28 CRAN (R 3.4.4)                         
 DALEX         * 0.2.2      2018-05-22 CRAN (R 3.4.4)                         
 datasets      * 3.4.4      2018-03-15 local                                  
 debugme         1.1.0      2017-10-22 CRAN (R 3.4.2)                         
 deldir          0.1-15     2018-04-01 CRAN (R 3.4.4)                         
 desc            1.1.1      2017-08-03 cran (@1.1.1)                          
 devtools        1.13.5     2018-02-18 CRAN (R 3.4.3)                         
 digest          0.6.15     2018-01-28 cran (@0.6.15)                         
 dplyr           0.7.4      2017-09-28 CRAN (R 3.4.2)                         
 evaluate        0.10.1     2017-06-24 CRAN (R 3.4.1)                         
 expm            0.999-2    2017-03-29 cran (@0.999-2)                        
 factorMerger    0.3.6      2018-04-04 CRAN (R 3.4.4)                         
 gdata           2.18.0     2017-06-06 CRAN (R 3.4.0)                         
 gdtools       * 0.1.7      2018-02-27 CRAN (R 3.4.3)                         
 ggiraph       * 0.4.2      2017-12-19 CRAN (R 3.4.3)                         
 ggplot2       * 2.2.1.9000 2018-05-31 Github (thomasp85/ggplot2@dfa0bc3)     
 ggpubr          0.1.6      2017-11-14 cran (@0.1.6)                          
 git2r           0.21.0     2018-01-04 CRAN (R 3.4.3)                         
 glue            1.2.0      2017-10-29 cran (@1.2.0)                          
 gmodels         2.16.2     2015-07-22 CRAN (R 3.4.0)                         
 graphics      * 3.4.4      2018-03-15 local                                  
 grDevices     * 3.4.4      2018-03-15 local                                  
 grid            3.4.4      2018-03-15 local                                  
 gridExtra       2.3        2017-09-09 CRAN (R 3.4.1)                         
 gtable          0.2.0      2016-02-26 CRAN (R 3.2.3)                         
 gtools          3.5.0      2015-05-29 CRAN (R 3.2.0)                         
 highlight       0.4.7.2    2017-10-04 cran (@0.4.7.2)                        
 highr           0.6        2016-05-09 CRAN (R 3.4.0)                         
 hms             0.4.2      2018-03-10 CRAN (R 3.4.4)                         
 htmltools       0.3.6      2017-04-28 CRAN (R 3.4.0)                         
 htmlwidgets     1.0        2018-01-20 cran (@1.0)                            
 httpuv          1.3.6.2    2018-03-02 CRAN (R 3.4.3)                         
 httr            1.3.1      2017-08-20 CRAN (R 3.4.1)                         
 jsonlite        1.5        2017-06-01 CRAN (R 3.4.0)                         
 kableExtra      0.9.0      2018-05-21 CRAN (R 3.4.4)                         
 klaR            0.6-14     2018-03-19 CRAN (R 3.4.4)                         
 knitr           1.20       2018-02-20 cran (@1.20)                           
 labeling        0.3        2014-08-23 CRAN (R 3.2.2)                         
 lattice         0.20-35    2017-03-25 CRAN (R 3.4.4)                         
 lazyeval        0.2.1      2017-10-29 CRAN (R 3.4.2)                         
 LearnBayes      2.15.1     2018-03-18 CRAN (R 3.4.4)                         
 magrittr        1.5        2014-11-22 CRAN (R 3.2.2)                         
 MASS            7.3-49     2018-02-23 CRAN (R 3.4.4)                         
 Matrix          1.2-14     2018-04-09 CRAN (R 3.4.4)                         
 memoise         1.1.0      2017-04-21 CRAN (R 3.4.0)                         
 methods       * 3.4.4      2018-03-15 local                                  
 MI2template     0.1.0.0000 2017-12-02 Github (mi2-warsaw/MI2template@a2e7f45)
 mime            0.5        2016-07-07 CRAN (R 3.4.0)                         
 miniUI          0.1.1      2016-01-15 CRAN (R 3.2.3)                         
 modelDown       0.0.0.9000 2018-05-31 Github (kromash/modelDown@8ff72a4)     
 munsell         0.4.3      2016-02-13 CRAN (R 3.2.3)                         
 mvtnorm         1.0-7      2018-01-25 cran (@1.0-7)                          
 nlme            3.1-137    2018-04-07 CRAN (R 3.4.4)                         
 officer         0.2.2      2018-03-14 CRAN (R 3.4.4)                         
 pdp             0.6.0      2017-07-20 CRAN (R 3.4.1)                         
 pillar          1.2.1      2018-02-27 CRAN (R 3.4.3)                         
 pkgbuild        0.0.0.9000 2017-10-25 Github (r-lib/pkgbuild@a70858f)        
 pkgconfig       2.0.1      2017-03-21 CRAN (R 3.4.0)                         
 pkgdown         0.1.0.9000 2017-11-21 Github (hadley/pkgdown@33673a9)        
 pkgload         0.0.0.9000 2017-11-21 Github (r-lib/pkgload@70eaef8)         
 plyr            1.8.4      2016-06-08 CRAN (R 3.4.0)                         
 proxy           0.4-22     2018-04-08 CRAN (R 3.4.4)                         
 purrr           0.2.4      2017-10-18 cran (@0.2.4)                          
 questionr       0.6.2      2017-11-01 CRAN (R 3.4.2)                         
 R.methodsS3     1.7.1      2016-02-16 CRAN (R 3.2.3)                         
 R.oo            1.21.0     2016-11-01 CRAN (R 3.4.0)                         
 R.utils         2.6.0      2017-11-05 CRAN (R 3.4.2)                         
 R6              2.2.2      2017-06-17 CRAN (R 3.4.0)                         
 randomForest  * 4.6-14     2018-03-25 CRAN (R 3.4.4)                         
 ranger        * 0.9.0      2018-01-09 cran (@0.9.0)                          
 Rcpp            0.12.16    2018-03-13 cran (@0.12.16)                        
 readr           1.1.1      2017-05-16 CRAN (R 3.4.0)                         
 reshape2        1.4.3      2017-12-11 cran (@1.4.3)                          
 rlang           0.2.0.9001 2018-05-31 Github (r-lib/rlang@4e7e8f7)           
 rmarkdown       1.9        2018-03-01 CRAN (R 3.4.3)                         
 roxygen2        6.0.1.9000 2017-10-25 Github (klutometis/roxygen@bbf259d)    
 rprojroot       1.3-2      2018-01-03 CRAN (R 3.4.3)                         
 rstudioapi      0.7        2017-09-07 CRAN (R 3.4.1)                         
 rvest           0.3.2      2016-06-17 CRAN (R 3.4.0)                         
 rvg             0.1.8      2018-02-13 CRAN (R 3.4.3)                         
 scales          0.5.0.9000 2018-05-31 Github (hadley/scales@d767915)         
 shiny           1.0.5      2017-08-23 CRAN (R 3.4.1)                         
 sp              1.2-7      2018-01-19 cran (@1.2-7)                          
 spData          0.2.8.3    2018-03-25 CRAN (R 3.4.4)                         
 spdep           0.7-7      2018-04-03 CRAN (R 3.4.4)                         
 splines         3.4.4      2018-03-15 local                                  
 stats         * 3.4.4      2018-03-15 local                                  
 stringi         1.1.7      2018-03-12 cran (@1.1.7)                          
 stringr         1.3.0      2018-02-19 cran (@1.3.0)                          
 survival        2.41-3     2017-04-04 CRAN (R 3.4.4)                         
 testthat        2.0.0      2017-12-13 CRAN (R 3.4.3)                         
 tibble          1.4.2      2018-01-22 cran (@1.4.2)                          
 tools           3.4.4      2018-03-15 local                                  
 usethis         1.3.0      2018-02-24 CRAN (R 3.4.3)                         
 utils         * 3.4.4      2018-03-15 local                                  
 uuid            0.1-2      2015-07-28 CRAN (R 3.4.0)                         
 viridisLite     0.3.0      2018-02-01 cran (@0.3.0)                          
 WhatIfPlots   * 0.1        <NA>       local                                  
 whisker         0.3-2      2013-04-28 CRAN (R 3.1.0)                         
 withr           2.1.2      2018-05-31 Github (jimhester/withr@70d6321)       
 xml2            1.2.0      2018-01-24 cran (@1.2.0)                          
 xtable          1.8-2      2016-02-05 CRAN (R 3.2.3)                         
 yaImpute        1.0-29     2017-12-10 CRAN (R 3.4.3)                         
 yaml            2.1.18     2018-03-08 cran (@2.1.18)                         
 zip             1.0.0      2017-04-25 CRAN (R 3.4.0)

Enable passing additional model (models?) (as DALEX explainers or just models with data and y?) to modelDown function so that check_drift function can be used