jamesritchie / scikit-rvm Goto Github PK

Relevance Vector Machine implementation using the scikit-learn API.

Python 100.00%

scikit-rvm's Introduction

scikit-rvm

scikit-rvm is a Python module implementing the Relevance Vector Machine (RVM) machine learning technique using the scikit-learn API.

Quickstart

With NumPy, SciPy and scikit-learn available in your environment, install with:

pip install https://github.com/JamesRitchie/scikit-rvm/archive/master.zip

Regression is done with the RVR class:

>>> from skrvm import RVR
>>> X = [[0, 0], [2, 2]]
>>> y = [0.5, 2.5 ]
>>> clf = RVR(kernel='linear')
>>> clf.fit(X, y)
RVR(alpha=1e-06, beta=1e-06, beta_fixed=False, bias_used=True, coef0=0.0,
coef1=None, degree=3, kernel='linear', n_iter=3000,
threshold_alpha=1000000000.0, tol=0.001, verbose=False)
>>> clf.predict([[1, 1]])
array([ 1.49995187])

Classification is done with the RVC class:

>>> from skrvm import RVC
>>> from sklearn.datasets import load_iris
>>> clf = RVC()
>>> clf.fit(iris.data, iris.target)
RVC(alpha=1e-06, beta=1e-06, beta_fixed=False, bias_used=True, coef0=0.0,
coef1=None, degree=3, kernel='rbf', n_iter=3000, n_iter_posterior=50,
threshold_alpha=1000000000.0, tol=0.001, verbose=False)
>>> clf.score(iris.data, iris.target)
0.97999999999999998

Theory

The RVM is a sparse Bayesian analogue to the Support Vector Machine, with a number of advantages:

It provides probabilistic estimates, as opposed to the SVM's point estimates.
Typically provides a sparser solution than the SVM, which tends to have the number of support vectors grow linearly with the size of the training set.
Does not need a complexity parameter to be selected in order to avoid overfitting.

However it is more expensive to train than the SVM, although prediction is faster and no cross-validation runs are required.

The RVM's original creator Mike Tipping provides a selection of papers offering detailed insight into the formulation of the RVM (and sparse Bayesian learning in general) on a dedicated page, along with a Matlab implementation.

Most of this implementation was written working from Section 7.2 of Christopher M. Bishops's Pattern Recognition and Machine Learning.

Contributors

Future Improvements

Implement the fast Sequential Sparse Bayesian Learning Algorithm outlined in Section 7.2.3 of Pattern Recognition and Machine Learning
Handle ill-conditioning errors more gracefully.
Implement more kernel choices.
Create more detailed examples with IPython notebooks.

scikit-rvm's People

Contributors

Stargazers

Watchers

scikit-rvm's Issues

No module named 'skrvm'

Hi there,

Thanks so much for the RVM code. However, I met a difficulty at the very beginning. I installed scikit-svm and it said：

I tried to employ it in python though anaconda then but failed:

I don't understand why. Any help will be appreciated.

Thanks

epistemic or aleatoric uncertainty?

Hello,

I'm wondering whether the uncertainty captured here is epistemic or aleatoric uncertainty? Or both?

self.m_ is approximately equal to zero.

Hi, when I trained a model using the codes:

"clf = RVR(kernel = 'rbf',n_iter=300,alpha=1e-3,beta=1.e-3,threshold_alpha=1e3)
clf.fit(input_train, output_train)
output_2_rvr = clf.predict(input_test)"

input_train is a 40003 matrix, output_train is a 40001 matrix.

error:
"
D:\ProgramData\Anaconda3\lib\site-packages\skrvm\rvm.py:153: RuntimeWarning: divide by zero encountered in true_divide
self.alpha_ = self.gamma/(self.m_ ** 2)
D:\ProgramData\Anaconda3\lib\site-packages\skrvm\rvm.py:152: RuntimeWarning: invalid value encountered in multiply
self.gamma = 1 - self.alpha_*np.diag(self.sigma_)
D:\ProgramData\Anaconda3\lib\site-packages\skrvm\rvm.py:153: RuntimeWarning: invalid value encountered in true_divide
self.alpha_ = self.gamma/(self.m_ ** 2)
D:\ProgramData\Anaconda3\lib\site-packages\skrvm\rvm.py:108: RuntimeWarning: invalid value encountered in less
keep_alpha = self.alpha_ < self.threshold_alpha
1
2
Traceback (most recent call last):
File "E:\python_work\pitch_system\src\relevance_vector_machine\rvr_train_test1.py", line 50, in
mse_rvr = mean_squared_error(output_2_rvr, output_test)
File "D:\ProgramData\Anaconda3\lib\site-packages\sklearn\metrics\regression.py", line 238, in mean_squared_error
y_true, y_pred, multioutput)
File "D:\ProgramData\Anaconda3\lib\site-packages\sklearn\metrics\regression.py", line 76, in _check_reg_targets
y_true = check_array(y_true, ensure_2d=False)
File "D:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 453, in check_array
_assert_all_finite(array)
File "D:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 44, in _assert_all_finite
" or a value too large for %r." % X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
"

How can I deal with it ? Thank you very much!

up to 3000 epoch, delta is still too high

In my datasets, up to 3000 epoch , delta is to still too high. although i have 30000 epoch, delta can't decrease. And the time cost is large. it just fits small samples?

"Fit" function for (binary) classification

I found that the training data of both classes are required to have the same size, otherwise some bugs will be reported (e.g. dataset1 -- 49 training samples, dataset2 -- 50 training samples).

efficient implementation algorithm

Hello,
An efficient implementation technique is describbed in there:
"Tipping, M., & Faul, A. (2003). Fast marginal likelihood maximization for sparse Bayesian models, paper presented at Ninth International Workshop on Artificial Intelligence and Statistics, Soc. for Artif. Intel. and Stat., Key West, Florida."
http://www.miketipping.com/papers/met-fastsbl.pdf
Regards,
F.

Exceptions when the training data is too big using RVC

for my example, it will cause an error when I am using RVC.fit function

the size of my training data is only 200*21 (200 samples and 21 features)

Getting the confidence of the prediction?

Hello James,

when using Relevance Vector Regression, is it possible to somehow get the confidence of the predictions? For example lets say that my regression model outputs some predictions, what I want to know is how sure my regression model is that those predictions are true.

Kind regards,
ac1dxtrem

can't use predict_proba() no relevance_ attribute

clf = RVC()
clf.fit(iris.data, iris.target)
clf.predict_proba(iris.data[0])

AttributeError                            Traceback (most recent call last)
<ipython-input-100-dc59cdb5fb6b> in <module>()
----> 1 clf.predict_proba(iris.data[0])

//anaconda/lib/python2.7/site-packages/skrvm/rvm.pyc in predict_proba(self, X)
    286     def predict_proba(self, X):
    287         """Return an array of class probabilities."""
--> 288         phi = self._apply_kernel(X, self.relevance_)
    289         y = self._classify(self.m_, phi)
    290         return np.column_stack((1-y, y))

AttributeError: 'RVC' object has no attribute 'relevance_'

Feature importance

Hi,

Is there any way to get something similar to the feature importance of the model (for example, in SVM we can check the feature weights with .coef_ and use the abs(svm.coef_) as the importance)?

Multiclass Classification not working.

It's been reported to me that multi-class classification isn't working with the MNIST digits example. I'll look into this when I have some spare time.

Singular matrix error

Fitting a relatively small model where X is 9249 examples and 50 features and I'm fitting the RVR to 9249 float predictions, getting a problem with matrix inversion:

/imaging/av02/python/lib/python2.7/site-packages/numpy-1.10.1-py2.7-linux-x86_64.egg/numpy/linalg/linalg.py(90)_raise_linalgerror_singular()
-> raise LinAlgError("Singular matrix")

What could be the issue?

Thanks!
Sasha

Kernel died and restarted

Hi there, thanks for the code.

But I met a difficulty at the very beginning and wish can get some help.

My dataset has 250000 samples with 50 dimensions after One Hot Encoding. Every time my Jupyter stops working at the step "clf.fit" and says " kernel appears to have died. it will restart automatically“. I checked Console (My laptop is Macbook ) and it told me "low swap: killing largest compressed process with pid 34341 (python3.7) and size 325 MB" and "memorysatus: killing_idel_process pid 508".

Is there any possible way to fix it? Any suggestion will be appreciated!

NameError: free variable 'gfk' referenced before assignment in enclosing scope

error with unequal sample sizes for classes

It seems the RVC function only works when number of both classes are of the same sample size.
When train with a sample with 72 [0] and 78 [1], I receive the following error:
Error in python.exec("clf.fit(H,label)") :
operands could not be broadcast together with shapes (72,) (78,)

It worked when I set both classes with equal size 75 for each.

没有参数介绍

free variable 'gfk' referenced before assignment in enclosing scope

NameError Traceback (most recent call last)
in ()
----> 1 calAcc()

in calAcc()
24
25 rvcObj = RVC()
---> 26 rvcFit = rvcObj.fit(featuresTrain, y_real[trainI])

/home/gnl/TF/local/lib/python2.7/site-packages/skrvm/rvm.pyc in fit(self, X, y)
281 self.multi_ = None
282 self.multi_ = OneVsOneClassifier(self)
--> 283 self.multi_.fit(X, y)
284 return self
285

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/multiclass.pyc in fit(self, X, y)
502 delayed(fit_ovo_binary)
503 (self.estimator, X, y, self.classes[i], self.classes_[j])
--> 504 for i in range(n_classes) for j in range(i + 1, n_classes)))))
505
506 self.estimators_ = estimators_indices[0]

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in call(self, iterable)
777 # was dispatched. In particular this covers the edge
778 # case of Parallel used with an exhausted iterator.
--> 779 while self.dispatch_one_batch(iterator):
780 self._iterating = True
781 else:

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch_one_batch(self, iterator)
623 return False
624 else:
--> 625 self._dispatch(tasks)
626 return True
627

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in _dispatch(self, batch)
586 dispatch_timestamp = time.time()
587 cb = BatchCompletionCallBack(dispatch_timestamp, len(batch), self)
--> 588 job = self._backend.apply_async(batch, callback=cb)
589 self._jobs.append(job)
590

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in apply_async(self, func, callback)
109 def apply_async(self, func, callback=None):
110 """Schedule a func to be run"""
--> 111 result = ImmediateResult(func)
112 if callback:
113 callback(result)

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in init(self, batch)
330 # Don't delay the application, to avoid keeping the input
331 # arguments in memory
--> 332 self.results = batch()
333
334 def get(self):

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in call(self)
129
130 def call(self):
--> 131 return [func(*args, **kwargs) for func, args, kwargs in self.items]
132
133 def len(self):

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/multiclass.pyc in _fit_ovo_binary(estimator, X, y, i, j)
421 return _fit_binary(estimator,
422 _safe_split(estimator, X, None, indices=indcond)[0],
--> 423 y_binary, classes=[i, j]), indcond
424
425

/home/gnl/TF/local/lib/python2.7/site-packages/sklearn/multiclass.pyc in _fit_binary(estimator, X, y, classes)
78 else:
79 estimator = clone(estimator)
---> 80 estimator.fit(X, y)
81 return estimator
82

/home/gnl/TF/local/lib/python2.7/site-packages/skrvm/rvm.pyc in fit(self, X, y)
277 self.t = np.zeros(y.shape)
278 self.t[y == self.classes_[1]] = 1
--> 279 return super(RVC, self).fit(X, self.t)
280 else:
281 self.multi_ = None

/home/gnl/TF/local/lib/python2.7/site-packages/skrvm/rvm.pyc in fit(self, X, y)
148
149 for i in range(self.n_iter):
--> 150 self.posterior()
151
152 self.gamma = 1 - self.alpha*np.diag(self.sigma_)

/home/gnl/TF/local/lib/python2.7/site-packages/skrvm/rvm.pyc in _posterior(self)
258 jac=True,
259 options={
--> 260 'maxiter': self.n_iter_posterior
261 }
262 )

/home/gnl/TF/local/lib/python2.7/site-packages/scipy/optimize/_minimize.pyc in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
482 elif meth == 'newton-cg':
483 return _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,
--> 484 **options)
485 elif meth == 'l-bfgs-b':
486 return _minimize_lbfgsb(fun, x0, args, jac, bounds,

/home/gnl/TF/local/lib/python2.7/site-packages/scipy/optimize/optimize.pyc in _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback, xtol, eps, maxiter, disp, return_all, **unknown_options)
1597 msg = ("Warning: CG iterations didn't converge. The Hessian is not "
1598 "positive definite.")
-> 1599 return terminate(3, msg)
1600
1601 pk = xsupi # search direction is solution to system.

/home/gnl/TF/local/lib/python2.7/site-packages/scipy/optimize/optimize.pyc in terminate(warnflag, msg)
1517 print(" Hessian evaluations: %d" % hcalls)
1518 fval = old_fval
-> 1519 result = OptimizeResult(fun=fval, jac=gfk, nfev=fcalls[0],
1520 njev=gcalls[0], nhev=hcalls, status=warnflag,
1521 success=(warnflag == 0), message=msg, x=xk,

NameError: free variable 'gfk' referenced before assignment in enclosing scope.

Please fix this issue.

Found array with 0 sample

Hi. Why am I getting this error:

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    print(clf.predict([[1, 1]]))
  File "/usr/lib/python3.5/site-packages/skrvm/rvm.py", line 201, in predict
    phi = self._apply_kernel(X, self.relevance_)
  File "/usr/lib/python3.5/site-packages/skrvm/rvm.py", line 82, in _apply_kernel
    phi = linear_kernel(x, y)
  File "/usr/lib/python3.5/site-packages/sklearn/metrics/pairwise.py", line 729, in linear_kernel
    X, Y = check_pairwise_arrays(X, Y)
  File "/usr/lib/python3.5/site-packages/sklearn/metrics/pairwise.py", line 111, in check_pairwise_arrays
    warn_on_dtype=warn_on_dtype, estimator=estimator)
  File "/usr/lib/python3.5/site-packages/sklearn/utils/validation.py", line 416, in check_array
    context))
ValueError: Found array with 0 sample(s) (shape=(0, 2)) while a minimum of 1 is required by check_pairwise_arrays.

with following code:

from skrvm import RVR

X = [[2.1, 4], [2, 2]]
y = [0.5, 2.5 ]

clf = RVR(kernel='linear')

clf.fit(X, y)

print(clf.predict([[1, 1]]))

Am I using it correctly?

jamesritchie / scikit-rvm Goto Github PK

scikit-rvm's Introduction

scikit-rvm

Quickstart

Theory

Contributors

Future Improvements

scikit-rvm's People

Contributors

Stargazers

Watchers

Forkers

scikit-rvm's Issues

Recommend Projects

Recommend Topics

Recommend Org