pathologydatascience / survivalnet Goto Github PK

@safooray

Hi,

Thank you for sharing your code. I'm having a problem with running the Model_Analysis script. For certain dataset (subgroups of variables), I can run Model_Analysis with no issue. However, for other subset of variables, I encountered the following error:

Generting risk gradient profiles...
Traceback (most recent call last):
File "ModelAnalysis.py", line 43, in
Tau=5e-2, Path=Path)
File "/home/an/python2/SurvivalNet/survivalnet/analysis/FeatureAnalysis.py", line 67, in FeatureAnalysis
Gradients = RiskCohort(Model, Normalized)
File "/home/an/python2/SurvivalNet/survivalnet/analysis/RiskCohort.py", line 44, in RiskCohort
Gradients[i, :] = _RiskBackpropagate(Model, Matrix[i, :])
File "/home/an/python2/SurvivalNet/survivalnet/analysis/RiskCohort.py", line 84, in _RiskBackpropagate
name='partial_derivative')
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1839, in orig_function
name=name)
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1487, in init
accept_inplace)
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 181, in std_fgraph
update_mapping=update_mapping)
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/gof/fg.py", line 175, in init
self.import_r(output, reason="init")
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/gof/fg.py", line 346, in import_r
self.import(variable.owner, reason=reason)
File "/home/an/python2/env/local/lib/python2.7/site-packages/theano/gof/fg.py", line 391, in import
raise MissingInputError(error_msg, variable=r)
theano.gof.fg.MissingInputError: Input 0 of the graph (indices start from 0), used to compute Elemwise{Cast{float64}}(mask_1), was not provided and not given a value. Use the Theano flag exception_verbosity='high', for more information on this error.

Is there any way to resolve this issue? Thank you.

Hello,

I've mistake below when I'm trying to use ModelAnalysis.py:
Cannot convert Type TensorType(float64, matrix) (of Variable X) into Type TensorType(float32, matrix). You can try to manually convert X into a TensorType(float32, matrix)
Found solution in closed issues(change version of Theano) and changed version of Theano inside docker from version 0.10.0 to version 0.8.2. But it leaded to new error:

Model = pickle.load(f)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.new(cls, *args)
AttributeError: class scratchpad has no attribute 'new'

Before I didn't have this issue.

I'll be very glad of any ideas how to solve it

I read your preprint and have a few questions about running SurvivalNet on TCGA data or my own data.

The paper mentioned that a Docker container is available. Can you please provide a link?

Do you have a full list of dependencies for running SurvivalNet outside Docker? requirements.txt is empty.

What license is applicable? The LICENSE file is also empty.

Do you have an example of the input data format? I saw in examples/Run.py that the -ip argument takes a .mat file as input, but I'm not sure how to prepare that file. Could you add the default ./data/Brain_Integ.mat dataset to the repository to provide a runnable example?

Thanks for the help. I'm looking forward to using your method.

Trying to install on ubuntu 16.04 and getting

$ pip install bayesopt
Collecting bayesopt
Could not find a version that satisfies the requirement bayesopt (from versions: )
No matching distribution found for bayesopt

Hi,

Could you please include your code on how to convert a typical CSV dataset to the .mat format required to use SurvivalNet? Can you explain the difference in Integ_X and Integ_X_raw?

Thank you!

Have the BayesOpt cost function read from a config file, instead of setting constants.

Hello. Thank you for providing your code and detailed descriptions of how to run. This is helpful as I begin to explore various Deep Learning methods. I was trying to run the example code through the Docker container (which defaults to the Brain_Integ.mat sample data) but ran into a few issues, which worked after some debugging. It appears that the example code on the GitHub page is different from the code in Docker. Quite a few errors that resulted were solved after realizing this and copying the GitHub codes into Docker and running those instead. Below is what I did to attempt to resolve the issues.

SurvivalNet Process/Debug:

First, enter Docker.

docker pull cancerdatascience/snet:version1
docker run -it cancerdatascience/snet:version1 /bin/bash
cd SurvivalNet/examples
python Run.py (obtain an Error about './data/Brain_Integ.mat')

Edit the following python scripts: Run.py: change line 133 to "../data/Brain_Integ.mat":
parser.add_argument('-ip', '--input_path', dest='input_path',
default='../data/Brain_Integ.mat',
help='Path specifying location of dataset.')

Re-run:
python Run.py
python ModelAnalysis.py

Modify ModelAnalysis.py to fix the following errors (fix one then re-run to obtain the next):

Error about 'NameError: name 'scipy' is not defined'': change line 15 from "X = scipy.io.loadmat(Data[i])" to:
# Loads normalized data.
X = sio.loadmat(Data[i])

Error about 'No such file or directory: 'i../data/Brain_Integ.mat'': change line 9 to:
Data = ['../data/Brain_Integ.mat']

Error about 'No such file or directory: '../models/nl1-hs10-dor0.62-id0final_model'': change line 7-8 to:
ModelPaths = ['results/']
Models = ['final_model']

We compared the script within the Docker to the GitHub repo script and found that this error was not made in the GitHub -- only the Docker container!

Re-run.

We obtain an error when running ModelAnalysis for no "results/final_model" path -- We compared the script within the Docker to the GitHub repo script and found that the GitHub repo script was different than within the Docker image.
We re-wrote Run2.py which contained the GitHub version of Run.py (BUT STILL making the same correction regarding ../data/Brain_Integ.mat) and the code ran.

python Run2.py

We re-run ModelAnalysis.py but this time we note there are also discrepancies between GitHub Repo and Docker container file (since we get an error saying "Nbox" and there is no Nbox for GitHub even though there is for Docker image file). We create ModelAnalysis2.py by copying the GitHub repo version (maintaining the same changes we made previously for ModelAnalysis.py within Docker since those errors are the same).

python ModelAnalysis2.py

Obtain error: "TypeError: Cannot convert Type TensorType(float64, matrix) (of Variable X) into Type TensorType(float32, matrix). You can try to manually convert X into a TensorType(float32, matrix)."

NOTE: IN ORDER TO DEBUG AND EDIT SCRIPTS, we need to edit the scripts here to edit the survivalnet package (NOT in the SurvivalNet/analysis folder!): "/usr/local/lib/python2.7/dist-packages/survivalnet-0.1.0-py2.7.egg/survivalnet/analysis/"

Edit RiskCohort.py so that line 28 X is float32 (such that theano.function will run):
# define partial derivative
X = T.matrix('X', dtype='float32')

To avoid error of 'Unique' not being defined before used, edit Visualization.py (go to KMPlots definition) to move 'Unique' outside of (i.e., before) "if Types[i] == ''" statements:
# get unique values to determine if binary or continuous
Unique = np.unique(Raw[:, i])

-- done with debugging. Now example code will run (Run.py and ModelAnalysis.py)

Command line warnings (see screenshot):
root@e816d5997cd4:/SurvivalNet/examples# python ModelAnalysis2.py

Generting risk gradient profiles...
Generating risk gradient boxplot...
/usr/local/lib/python2.7/dist-packages/survivalnet-0.1.0-py2.7.egg/survivalnet/analysis/Visualization.py:157: MatplotlibDeprecationWarning: The set_axis_bgcolor function was deprecated in version 2.0. Use set_facecolor instead.
Axes.set_axis_bgcolor('white')
Generating paired scatter gradient plots...
Generating paired scatter feature plots...
/usr/local/lib/python2.7/dist-packages/numpy/lib/function_base.py:4011: RuntimeWarning: Invalid value encountered in median
r = func(a, **kwargs)
Generating cluster analysis...
/usr/local/lib/python2.7/dist-packages/survivalnet-0.1.0-py2.7.egg/survivalnet/analysis/RiskCluster.py:88: ClusterWarning: scipy.cluster: The symmetric non-negative hollow observation matrix looks suspiciously like an uncondensed distance matrix
metric='correlation')
/usr/local/lib/python2.7/dist-packages/survivalnet-0.1.0-py2.7.egg/survivalnet/analysis/RiskCluster.py:96: ClusterWarning: scipy.cluster: The symmetric non-negative hollow observation matrix looks suspiciously like an uncondensed distance matrix
metric='correlation')
Generating Kaplan-Meier plots...
Saving figures...

I wanted to check, are these Warnings expected for a default run of the example code with the sample data provided?

Thank you for your time.

Hello,

I can successfully generate model with and without bayes optimization.
I run into the following error during the partial_derivative calculation step, line num 76 in _RiskBackpropagate while doing the Model Analysis. A related error has been reported before, but was due to changes in the input and test dimensions. But, I get the error even by using the provided data "Brain_Integ.mat", when used both for Model Generation and Model Analysis.

The error only occurs if the n_layers is manully set to more than 1 (line 99 of Run.py) or when is changed during Bayes Optimization.

The error (for n_layers = 2)
Traceback (most recent call last): File "testAnalysis.py", line 114, in <module> Gradients[i, :] = _RiskBackpropagate(Model, Matrix[i, :]) File "testAnalysis.py", line 70, in _RiskBackpropagate name='partial_derivative') File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/compile/function.py", line 320, in function output_keys=output_keys) File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 479, in pfunc output_keys=output_keys) File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1776, in orig_function output_keys=output_keys).create( File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1428, in __init__ accept_inplace) File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 177, in std_fgraph update_mapping=update_mapping) File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/gof/fg.py", line 171, in __init__ self.__import_r__(output, reason="init") File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/gof/fg.py", line 360, in __import_r__ self.__import__(variable.owner, reason=reason) File "/home/satyan/py27_venvs/survNet/local/lib/python2.7/site-packages/theano/gof/fg.py", line 474, in __import__ r) theano.gof.fg.MissingInputError: ("An input of the graph, used to compute Elemwise{Cast{float64}}(mask_1), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.", mask_1)

My minimal environment installed by compiling python 2.7.13
`
Package Version

appdirs 1.4.3
cycler 0.10.0
funcsigs 1.0.2
functools32 3.2.3.post2
lifelines 0.8.0.0
matplotlib 1.4.3
mock 3.0.5
nose 1.3.7
numpy 1.16.5
packaging 19.2
pandas 0.24.2
patsy 0.5.1
pip 19.3.1
pycairo 1.18.2
PyGObject 3.28.1
pyparsing 2.4.4
python-dateutil 2.8.1
pytz 2019.3
scipy 0.18.0
setuptools 34.0.2
six 1.13.0
statsmodels 0.8.0
subprocess32 3.5.4
survivalnet 0.1.0
Theano 0.9.0
wheel 0.33.6
`
Any suggestion would be helpful.

Hi, I was trying to run the Model Analysis file after generating the final_model, but I am getting this error:
Cannot convert Type TensorType(float64, matrix) (of Variable X) into Type TensorType(float32, matrix). You can try to manually convert X into a TensorType(float32, matrix)

@safooray

Used to check for consistency between models and data in analysis modules.

Is it possible to port this package to python3.6?

how do I use the saved model, to predict the outcomes on new dataset

Hi dear authors, thanks a lot for your sharing code! I have a question, in the design of RiskLayer, the following line:
https://github.com/CancerDataScience/SurvivalNet/blob/83b99fffea15bca5f0017fa08bef61de9f165eea/survivalnet/model/RiskLayer.py#L46

Do we need to add a negative sign? Because the negative log partial likelihood (Eqn. 1 in your paper) has this sign. Thanks.

Could you please share the source code of your published paper titled Predicting cancer outcomes from histology and genomics using convolutional networks on PNAS journal? Thank your so much!!!

https://github.com/CancerDataScience/SurvivalNet/blob/83b99fffea15bca5f0017fa08bef61de9f165eea/survivalnet/model/Model.py#L197

Whether the regularization of the objective function is incorrect？

Whether the regularization item needs to be added instead of minus