sfalkner / pysmac Goto Github PK

View Code? Open in Web Editor NEW

This project forked from tdomhan/pysmac

10.0 10.0 10.0 22.13 MB

simple python interface to SMAC.

License: GNU Affero General Public License v3.0

Python 58.15% Shell 36.77% Batchfile 5.08%

pysmac's People

Contributors

Stargazers

Watchers

Forkers

mfeurer gothma pfhayes ernestum newmanne codeaudit wutenghu niilante jayden11 kumarjitpathakbangalore

pysmac's Issues

Unable to run simple example

I have installed pysmac and try to run the simple rosenbrock problem. When I do so, i get the following error:

python rosenbrock_example.py
Traceback (most recent call last):
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 284, in remote_smac_function
    config_dict = smac.next_configuration()
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 193, in next_configuration
    self.__conn, addr = self.__sock.accept()
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 206, in accept
    sock, addr = self._sock.accept()
timeout: timed out
Traceback (most recent call last):
  File "rosenbrock_example.py", line 58, in <module>
    parameters)    # the parameter dictionary
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/optimizer.py", line 303, in minimize
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/utils/smac_output_readers.py", line 151, in read_trajectory_file
IOError: [Errno 2] No such file or directory: '/var/folders/9_/mztwlxpx5bv0_ljvn3614wb80000gp/T/tmpoknTRc/out/scenario/traj-run-0.txt'

I thought that it might be a problem with the working directory, so I have changed as:

 opt = opt = pysmac.SMAC_optimizer(
                             working_directory = '/Users/jesus/pysmac/')

But I get the same exactly error:

python rosenbrock_example.py
Traceback (most recent call last):
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 284, in remote_smac_function
    config_dict = smac.next_configuration()
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 193, in next_configuration
    self.__conn, addr = self.__sock.accept()
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 206, in accept
    sock, addr = self._sock.accept()
timeout: timed out
Traceback (most recent call last):
  File "rosenbrock_example.py", line 59, in <module>
    parameters)    # the parameter dictionary
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/optimizer.py", line 303, in minimize
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/utils/smac_output_readers.py", line 151, in read_trajectory_file
IOError: [Errno 2] No such file or directory: '/Users/jesus/pysmac/out/scenario/traj-run-0.txt'

~/projects/scratch/pySMAC/examples   master  python branin_example.py Traceback (most recent call last): File "build/bdist.macosx-10.6-x86_64/egg/pysmac/remote_smac.py", line 274, in remote_smac_function config_dict = smac.next_configuration() File "build/bdist.macosx-10.6-x86_64/egg/pysmac/remote_smac.py", line 195, in next_configuration config_str = fconn.readline() File "/Users/jaanaltosaar/anaconda2/lib/python2.7/socket.py", line 451, in readline data = self._sock.recv(self._rbufsize) error: [Errno 35] Resource temporarily unavailable Lowest function value found: 179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368.000000 Parameter setting {'x2': -1.0, 'x3': 1, 'x1': 1.0} ~/projects/scratch/pySMAC/examples   master 

Could this be related? https://bugs.python.org/msg64566

Would love to try pySMAC - thank you!

add a (maybe hidden) option to specify the used java binary

see sfalkner/SpySMAC#2

feature request: resumable optimization

It would be useful to me to be able to run SMAC for say 30 minutes, consider the results, and then run it again for another 30 minutes, building on the previous results without starting over. If this is already a feature, a demo would be useful, as I've not been able to figure it out.

TIMEOUT resulting in quality of 0.0

Hi,

I'm trying to tune a target algorithm for quality with a time limit for each function call. This results in SMAC return an error of 0.0 in case one function call hits the time limit. The following adaption of the example demonstrates the problem:

from __future__ import print_function, division

import pysmac

import sklearn.ensemble
import sklearn.datasets
import sklearn.cross_validation

X,Y = sklearn.datasets.make_classification(100000, 20, random_state=2)
X_train, X_test, Y_train, Y_test = sklearn.cross_validation.train_test_split(X,Y, test_size=0.33, random_state=1)

def random_forest(n_estimators,criterion, max_features, max_depth):

    predictor = sklearn.ensemble.RandomForestClassifier(n_estimators, criterion, max_features, max_depth)
    predictor.fit(X_train, Y_train)

    return -predictor.score(X_test, Y_test)

parameter_definition=dict(\
    max_depth   =("integer", [1,10],    4),
    max_features=("integer", [1,20],   10),
    n_estimators=("integer", [10,100], 10, 'log'),
    criterion   =("categorical", ['gini', 'entropy'], 'entropy'),
    )

opt = pysmac.SMAC_optimizer(working_directory = '/tmp/pysmac_test/',
                            persistent_files=False, debug = False)

predictor = sklearn.ensemble.RandomForestClassifier()
predictor.fit(X_train, Y_train)
print(('The default accuracy is %f'%predictor.score(X_test, Y_test)))


# The minimize method also has optional arguments
value, parameters = opt.minimize(random_forest,
                    100 , parameter_definition,
                    num_runs = 2,
                    seed = 2,
                    num_procs = 2,
                    mem_limit_function_mb=1000,
                    t_limit_function_s = 2)


print(('The highest accuracy found: %f'%(-value)))
print(('Parameter setting %s'%parameters))

I think for optimizing quality, a TIMEOUT not result in a good evaluation of a configuration but should have the same quality as a CRASH.

bug in state_merge

File "build/bdist.linux-x86_64/egg/pysmac/utils/state_merge.py", line 125, in state_merge
TypeError: unhashable type: 'list'

Line 125:

       ff_header.add(header_feats)

you cannot use "add" for lists. you could use "update". But looking at Line 127, I don't think you want a "update".

Please note that the problem exists for Python2 and Python3

Parameters File

paramstrings*.txt
1: cat_A='con', cat_B='plant', int_A='1', int_B='17', obj='A', ord_A='cold', ord_B='low'

This is the first utilized parameter set from the following parameters definition
parameter_definition=dict(
int_A = ("integer", [1, 3], 1),
cat_A = ("categorical", ["con","te","nance"], "con"),
ord_A = ("ordinal", ["cold", "hot"], "cold"),
int_B = ("integer", [17, 20], 17),
cat_B = ("categorical", ["plant", "human", "animal"], "plant"),
ord_B = ("ordinal", ["low", "high"], "low"),
obj = ("categorical", ["A", "B"], "A")
)

applied conditionals
    int_A    | obj in {A}
   cat_A    | obj in {A}
   ord_A    | obj in {A}
   int_B    | obj in {B}
   cat_B    | obj in {B}
   ord_B    | obj in {B}

During the runs the excluded parameters, according to the conditionals, are set to None.
active class: A:
int_A: 1, cat_A: con, ord_A: cold
int_B: None, cat_B: None, ord_B: None

However, in the paramstrings file the None values are replaced by random possible values from the input dictionary (see line 2). This is confusing and makes later evaluations more difficult. For example if you want to figure out how pysmac tries to optimize the parameters, by taking a look at the execution order.

Argument of SMAC_optimizer.minimize() must be a top-level function

Because of the use of multiprocessing library of Python, the first argument of minimize() must be a top-level function to become pickle-able. Hence, Smac library cannot be used in a class.

problem with setup.py

Problem 1:

byte-compiling build/bdist.linux-x86_64/egg/pysmac/utils/state-merge.py to state-merge.pyc
  File "build/bdist.linux-x86_64/egg/pysmac/utils/state-merge.py", line 41
    if conf is not in configurations:
                    ^
SyntaxError: invalid syntax

Problem 2:
Searching for pynisher>=0.1.1 Reading https://pypi.python.org/simple/pynisher/ Couldn't retrieve index page for 'pynisher' Scanning index of all packages (this may take a while) Reading https://pypi.python.org/simple/ No local packages or download links found for pynisher>=0.1.1 error: Could not find suitable distribution for Requirement.parse('pynisher>=0.1.1')

Socket timeout on Python 2.7 and 3.4

When running pySmac with python 2.7 on Ubuntu 14.04 with openJdk/Jre, a socket always times out after all rounds have been completed. It happens with the provided exaple files also.

Minimal example:

i = 1

def testfunction(x,y):
  global i
  print('Testing %i'%i)
  i = i+1
  return x+y

def testsmac():
  parameters = dict(x = ('real', [-1.,1.], 1),
                    y = ('real', [-1.,1.], 1),)
  opt = pysmac.SMAC_optimizer(debug=False)
  value, params = opt.minimize(testfunction, 5, parameters, num_procs = 1)
  print('Lowest: %f with %s'%(value, params))


if __name__ == '__main__':
  print('Starting optimization')
  start = time.time()
  testsmac()
  print('Needed %s seconds'%(time.time() - start))

to which the output is:


Starting optimization
Testing 1
Testing 1
Testing 1
Testing 1
Testing 1
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pysmac/remote_smac.py", line 284, in remote_smac_function
    config_dict = smac.next_configuration()
  File "/usr/local/lib/python2.7/dist-packages/pysmac/remote_smac.py", line 193, in next_configuration
    self.__conn, addr = self.__sock.accept()
  File "/usr/lib/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
timeout: timed out
Lowest: -1.312829 with {'y': -0.7396858875859076, 'x': -0.5731433730945221}
Needed 6.80100393295 seconds

So in the end the result is still computed and returned, but the socket times out.
The timeout depends on the parameter "udp_timeout" defined in pysmac.remote_smac.
If that is increased, it waits for that time until the timout happens.
The output with debug=True enabled is attached.
I tried to find where that error originates and found that the timeoout occurs during the call
of

pool.map(pysmac.remote_smac.remote_smac_function, argument_lists)

in optimizer.py, but after all calls to pysmac.remote_smac.remote_smac_function have been executed. The subsequent call to pool.close() is executed after the timeout.

When trying to run the example above with python3.4, I get the following error:

Edit: That error was a problem with the python3 virtualenv. The main issue (socket timeout) is also happening in python 3.4.

If more info is needed, I'm happy to provide it.
Thanks!
testsmacDebug.txt

TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

This happens to me when the target algorithm crashes and no time limit is specified. The full error message is:

Traceback (most recent call last):
  File "/home/feurerm/Software/SMAC/pysmac/pysmac/remote_smac.py", line 368, in remote_smac_function
Process Call to your function:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
  File "/home/feurerm/virtualenvs/autosklearn/local/lib/python2.7/site-packages/pynisher/limit_function_call.py", line 76, in subprocess_func
if (res is None) and ((cpu_time > current_t_limit - 2e-2) or
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

No module named 'resource'

Windows. Installation using pip install git+https...
http://prntscr.com/dj9vee

This may help
prometheus/client_python#27
prometheus/client_python@50605d3

AttributeError: 'module' object has no attribute 'SMAC_optimizer'

I try to run an example from Quickstart Guide, but it raises the exception:
AttributeError: 'module' object has no attribute 'SMAC_optimizer'
In import section is:
import pysmac
Exception is in line:
opt = pysmac.SMAC_optimizer()
How can I remove this exception?

import problem

>>> import pysmac
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/pysmac/__init__.py", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/pysmac/optimizer.py", line 16, in <module>
ImportError: No module named java_helper
>>> import pysmac
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/pysmac/__init__.py", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/pysmac/optimizer.py", line 14, in <module>
ImportError: cannot import name remote_smac

read_instance_features_file bug

File "build/bdist.linux-x86_64/egg/pysmac/utils/smac_output_readers.py", line 163, in read_instance_features_file
NameError: global name 'join' is not defined