stitchfix / diamond Goto Github PK

The model object includes lots of bloat, e.g the full training data. It would be nice to serialize a minimal version containing only that which is necessary to make predictions.

Here's a tentative list of attributes needed:

• num_main
• effects
• groupings
• results_dict (nice to have)
• main_map
• main_effects
• inter_maps
• level_maps

..../Users/timothysweetser/git/diamond/diamond/solvers/diamond_cumulative_logistic.py:75: RuntimeWarning: invalid value encountered in double_scalars
  np.linalg.norm(effects[grouping])
...../Users/timothysweetser/git/diamond/diamond/solvers/diamond_logistic.py:73: RuntimeWarning: invalid value encountered in double_scalars
  np.linalg.norm(effects[grouping])

This is caused by dividing by the norm of the coefficients, which are initialized to zero.

Stacktrace:
Traceback (most recent call last):
File "hhcf.py", line 131, in
main()
File "/home/ubuntu/work/lib/python3.5/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/ubuntu/work/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/ubuntu/work/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ubuntu/work/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "hhcf.py", line 79, in main
diamond_model = fit_diamond_model(df_train)
File "/home/ubuntu/work/lib/python3.5/site-packages/joblib/memory.py", line 562, in call
return self._cached_call(args, kwargs)[0]
File "/home/ubuntu/work/lib/python3.5/site-packages/joblib/memory.py", line 510, in _cached_call
out, metadata = self.call(*args, **kwargs)
File "/home/ubuntu/work/lib/python3.5/site-packages/joblib/memory.py", line 744, in call
output = self.func(*args, **kwargs)
File "/home/ubuntu/git/kaggle/kkbox/utils.py", line 157, in fit_diamond_model
diamond_model.fit(formula, tol=1e-5, verbose=False, max_its=200)
File "/home/ubuntu/diamond/diamond/glms/logistic.py", line 84, in fit
self.initialize(formula, **kwargs)
File "/home/ubuntu/diamond/diamond/glms/glm.py", line 96, in initialize
self._create_hessians()
File "/home/ubuntu/diamond/diamond/glms/logistic.py", line 62, in _create_hessians
(k * block_length):((k + 1) * block_length)]
File "/home/ubuntu/work/lib/python3.5/site-packages/scipy/sparse/csc.py", line 167, in getitem
return self.T[col, row].T
File "/home/ubuntu/work/lib/python3.5/site-packages/scipy/sparse/csr.py", line 304, in getitem
return self._get_submatrix(row, col)
File "/home/ubuntu/work/lib/python3.5/site-packages/scipy/sparse/csr.py", line 455, in _get_submatrix
i0, i1 = process_slice(row_slice, M)
File "/home/ubuntu/work/lib/python3.5/site-packages/scipy/sparse/csr.py", line 438, in process_slice
i0, i1, stride = sl.indices(num)
TypeError: slice indices must be integers or None or have an index method

They are:

• dplyr
• ordinal
• reshape2
• lme4
• logging
• MCMCpack
• gtools

Hi,

Is this in the pipeline for this project?

Currently, fitting the model modifies the training data frame in place when copy=False. I think we can improve the manners here.

1. Remove the columns created: row_index and intercept
2. Before creating new columns, see if they're already present. If so, raise an error.

Per https://docs.python-guide.org/writing/logging/, best practice is to add a null handler.

I think we use the full variety of Python 2+3 compatible string formatting methods. It might be cleaner to pick one and stick with it.

I've been using diamond with reticulate especially because it's easy to create the priors in R and then also fit the model in R. This would be a prime package to create an R version of the package with standard methods like predict, ranef, fixef, etc. I'll gladly do it, just wasn't sure if this is being actively developed.