Hi, I am try to install pypuf but I have the next error

`======================================================================
ERROR: test.test_example (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_example
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_example.py", line 6, in
import example
File "/home/aerobles/Descargas/pypuf/pypuf-master/example.py", line 5, in
from pypuf.learner.regression.logistic_regression import LogisticRegression
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/learner/regression/logistic_regression.py", line 11, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_experiment (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_experiment
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_experiment.py", line 4, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray, NoisyLTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_experimenter (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_experimenter
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_experimenter.py", line 5, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray, NoisyLTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_logistic_regression (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_logistic_regression
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_logistic_regression.py", line 4, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_low_degree (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_low_degree
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_low_degree.py", line 6, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_mv_num_of_votes (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_mv_num_of_votes
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_mv_num_of_votes.py", line 7, in
import mv_num_of_votes
File "/home/aerobles/Descargas/pypuf/pypuf-master/mv_num_of_votes.py", line 36, in
from pypuf.experiments.experiment.majority_vote import ExperimentMajorityVoteFindVotes
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/experiments/experiment/majority_vote.py", line 10, in
from pypuf.simulation.arbiter_based.ltfarray import SimulationMajorityLTFArray, LTFArray, NoisyLTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_property (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_property
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_property.py", line 6, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray, NoisyLTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_sim_learn (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_sim_learn
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_sim_learn.py", line 4, in
import sim_learn
File "/home/aerobles/Descargas/pypuf/pypuf-master/sim_learn.py", line 7, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_simulation (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_simulation
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_simulation.py", line 7, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray, NoisyLTFArray, SimulationMajorityLTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

======================================================================
ERROR: test.test_tools (unittest.loader._FailedTest)

ImportError: Failed to import test module: test.test_tools
Traceback (most recent call last):
File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
import(name)
File "/home/aerobles/Descargas/pypuf/pypuf-master/test/test_tools.py", line 6, in
from pypuf.simulation.arbiter_based.ltfarray import LTFArray
File "/home/aerobles/Descargas/pypuf/pypuf-master/pypuf/simulation/arbiter_based/ltfarray.py", line 10, in
import pypuf_helper as ph
SystemError: initialization of pypuf_helper raised unreported exception

Ran 10 tests in 0.006s

FAILED (errors=10)
`

Rework the logistic_regression learning module in order to satisfy our code conventions.

• rework the docstrings

I lost simulation results by running unit tests locally.

I propose to change the unit tests to save log files into a separate directory, or with a certain prefix, to preserve other log files when running the tests locally.

The experiments currently rely on the wall time as one metric of speed. This should be replaced by (a) different metric(s) to make running jobs on shared machines easier, and to improve comparison across different platforms.

As a note, for many ML algorithms one iteration typically uses the same amount of time as other iterations on the same training set, so the number of iterations should be linear in the wall time (on non-shared machines).

As discussed we should rework the convergence criteria for the LR learner https://github.com/nils-wisiol/pypuf/blob/master/pypuf/learner/regression/logistic_regression.py#L278-L301 . For this purpose we could:

• delete/alter the distance threshold
• use normal distributed challenges for the calculation of the distance instead of the Trainingset.responses
• norm the weight_array

In pr #63 I integrated code analysis to the project. I recognized that the some functions have inconsistent doc strings regarding to the whole project. Good doc string should look like this:

"""
Function description. Explain the benefit of this function.
:param parameter_name: type
                                       Parameter description
:return: type
            Description of the return value     
"""

For some functions I am not able to create good doc string. Like in #66 please give a meaningful name for sss. I Marked every doc string with TODO which should be improved.

Feature

The framework should provide the possibility to execute the learning algorithm from Georg T. Becker on an arbitrary simulation of a puf.

In the past, sim_learn was 'broken' a couple of times and did either not start at all or did not behave as described in README or help output. Add tests to make sure it is working properly.

Feature

The framework should be able to execute the pac learning algorithm on an arbitrary puf simulation.

Feature

It should be possible to compute a multiple experiments in parallel over all available cpu cores. The results should be written to one log file per experiment. It should be possible to kill specific processes.

Implementation

Construct a abstract experiment class which deals as interface between processes pool and experiments.
The processes pool should be used to schedule the experiments based on the interface.

wip with @maoswald

• environment setup (virtualenv, polymath)
• more examples
• execute tests locally
• easier access

The pypuf.tools.Trainingset chooses challenges random without a seed.
In order to ensure replicability of experiments we should introduce a random_instance to the Trainingset.

simulation.eval() can return zero (if val() returns zero). But responses should not be zero.

def eval(self, inputs):
    """
    evaluates a given list of challenges regarding bias
    :param x: list of challenges
    :return: list of responses
    """
    if self.bias:
        inputs = tools.iter_append_last(inputs, 1)
    return sign(self.val(inputs)) # <-----

def val(self, inputs):
    return self.combiner(self.ltf_eval(self.transform(inputs, self.k)))

tools.sample_inputs() returns an iterator over arrays of challenges, while the common usage of those is to unpack them into a 2-d-array. Maybe we should maintain the iterator object instead of unpacking it for the purpose of runtime advantages.

class TrainingSet():
    """
    Basic data structure to hold a collection of challenge response pairs.
    Note that this is, strictly speaking, not a set.
    """

    def __init__(self, instance, N):
        self.instance = instance
        self.challenges = array(list(sample_inputs(instance.n, N))) # <-----
        self.responses = instance.eval(self.challenges)
        self.N = N

I am not entirely sure if this is a bug or just my personal misunderstanding...
The following code (branch cma_hansen, pypuf/test/test_reliability_based_cmaes.py) leads to an error when executed within unittest, while working fine in other cases:

[...]
n = 16
k = 2
mu_weight = 0
sigma_weight = 1
transform = LTFArray.transform_atf
combiner = LTFArray.combiner_xor
seed_instance = 1234
prng_i = np.random.RandomState(seed_instance)

weight_array = LTFArray.normal_weights(n, k, mu_weight, sigma_weight, prng_i)
sigma_noise = NoisyLTFArray.sigma_noise_from_random_weights(n, sigma_weight, noisiness=0.05)
instance = NoisyLTFArray(weight_array, transform, combiner, sigma_noise, prng_i)

def test_create_abortion_function(self):
    is_same_solution = Learner.create_abortion_function(
        chains_learned=self.instance.weight_array,
        num_learned=1,
        transform=self.transform,
        combiner=self.combiner,
        threshold=0.25,
    )
    assert is_same_solution(self.instance.weight_array[0, :])

The corresponding error description is the following:
Error
Traceback (most recent call last):
File "/usr/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.6/unittest/case.py", line 605, in run
testMethod()
File "/home/dude/workspace/pypuf/test/test_reliability_based_cmaes.py", line 54, in test_create_abortion_function
assert is_same_solution(self.instance.weight_array[0, :])
File "/home/dude/workspace/pypuf/pypuf/learner/evolution_strategies/reliability_based_cmaes.py", line 195, in is_same_solution
dist = tools.approx_dist(current_ltf_array, new_ltf_array, this.APPROX_CHALLENGE_NUM)
File "/home/dude/workspace/pypuf/pypuf/tools.py", line 125, in approx_dist
return (num - count_nonzero(instance1.eval(inputs) == instance2.eval(inputs))) / num
File "/home/dude/workspace/pypuf/pypuf/simulation/arbiter_based/ltfarray.py", line 657, in eval
return sign(self.val(inputs))
File "/home/dude/workspace/pypuf/pypuf/simulation/arbiter_based/ltfarray.py", line 669, in val
return self.combiner(self.ltf_eval(self.transform(inputs, self.k)))

TypeError: combiner_xor() takes 1 positional argument but 2 were given

We should provide useful docstrings for all code of this project. Useful docstrings should look like

def val(self, inputs):
        """
        This function a calculates the output of the LTFArray based on weights with majority vote. 
        :param inputs: array of int shape(N,k,n)
                       Array of challenges which should be evaluated by the simulation.
        :return: array of int shape(N)
                 Array of responses for the N different challenges.
        """
return self.combiner(self.majority_vote(self.transform(inputs, self.k)))

rather than

def ltf_eval(self, inputs):
        """
        :return: array
        """
        return transpose(
            array([
                dot(
                    inputs[:,l],
                    self.weight_array[l]
                )
                for l in range(self.k)
            ])
)

When an exception occurs in an experiment the traceback should be printed. However, this line seems not to be working:

Maybe it's because it's not directly in an except block.

The code written in this repository should run on all Python 3 versions.
Furthermore all test have to pass in order to be satisfied.

Design a modular architecture for this project. In terms of modular it must be easy to add new puf simulations and learning algorithms in order to run experiments.
There some components which should be considered:

• PUF simulation
• Learning algorithms
• Experiments
• Results
• Process pool
• Command line interface

It should not be hard to reuse a lot of existing code for that.

When running sim_learn with a large number of experiments (e.g., sim_learn.py 32 4 lightweight_secure_original xor 12000 100 1 0xabc 0x1), no output is generated until all experiments are finished.

In order to increase credibility of this project it is important to enhance the traceability.
I suggest to write:

• A better README.md which shows the purpose about the project.
• a user guide (Wiki)
• a developer guide (Wiki)
• contributing guidelines (CONTRIBUTING.md)
• a summary for each learning algorithm (Wiki)
• a summary for each puf simulation model (Wiki)
• comments in code

Rework the fourier_expansion.py module in order to satisfy our code conventions.

• rework the docstrings
• change the name of FourierCoefficient.s to something meaningful

It would be a convenient feature to be able to suspend, restart and pause experiments at run time through interaction with the experimenter. This is particularly useful on shared machines, to give way to other, more important computations.

All methods should use Type Hints. This is really helpful for new contributors and readability of the code. Lets see whether it is possible to be consistent to #11 .

Example:

def greeting(name: str) -> str:
    return 'Hello ' + name

The test case test_init_normal_empirical in test module test.test_simulation.TestNoisyLTFArray is not deterministic and sometimes generates the following output:

======================================================================
FAIL: test_init_normal_empirical (test.test_simulation.TestNoisyLTFArray)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/arbeit/pypuf/test/test_simulation.py", line 756, in test_init_normal_empirical
    self.assertTrue(abs(tools.approx_dist(nla, nla, 10000) - intra_dist) < .02)
AssertionError: False is not true

I this phänomenon occured under python 3.4.5 and packages:

astroid (1.5.3)
isort (4.2.15)
lazy-object-proxy (1.3.1)
mccabe (0.6.1)
numpy (1.13.3)
pep8 (1.7.1)
pip (9.0.1)
polymath (0.1.17)
pylint (1.7.5)
pypuf-helper (0.1.4)
scipy (0.19.1)
setuptools (27.2.0)
six (1.11.0)
wheel (0.29.0)
wrapt (1.10.11)

I recognized that LogisticRegression.gradient(self, model): does not use model.eval(self, inputs). This was not problematic for a some time but now the pull request #52 introduces a majority vote ltf array simulation which overwrites the val function but does not touch the ltf_eval function. I think it is important to use the logistic regression learner in combination with the majority vote simulation. I will think about a solutions for this issue and post it. What do you think?

Code style is sometimes not cohesive or is forgotten. Add linter to the tests.

Close the log file in

The name of pep8 changed. The usage of the old name generates output.

pep8 has been renamed to pycodestyle (GitHub issue #466)
Use of the pep8 tool will be removed in a future release.
Please install and use `pycodestyle` instead.

$ pip install pycodestyle
$ pycodestyle ...

  '\n\n'

We should adapt the linter name in order to remove the output.

We should have a look at the memory consumption of pypuf, especially regarding the challenges. Right now the challenges are consisting of just the elements -1 and 1. The corresponding numpy arrays are created without a detailed specification of the data type of the array. According to the numpy reference [1] in that case the array is created using "the minimum type required to hold the objects in the sequence". I can only assume that this refers just to types like int and float and does not include the size of the type.
If I do not specify a type, I get on my computer only arrays with dtypes of size 64. I think we could reduce the memory consumption for the challenges to one eighth of the memory we need right now by stating the type int8 within the array creation. That would require a more strict definition and checking of the input/output types of the functions.
Regarding the speed there should be no big difference between different integer types in our case. But that clearly depends also on the used hardware.

[1] https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.array.html

numpy (at least in some configurations/compliations) automatically multithreads some functions like dot. On big machines, this yields a huge overload of work, as each of the, let's say, 64 worker processes spawns another 64 worker threads. Resulting load is 64 times as high as intended.

Ways to reproduce: with 6e1f120, run input_trans_comparison.py -n 64 -k 5 -s 1 -m 0xbad -t lightweight_secure_original -N 250000 -i 1000. Use top to monitor CPU usage, use pstree -c $(whoami) to monitor processes and threads (threads are shown in curly braces). There should only be two processes, one for logging, one for the single experiment we are running (-s 1).

Workaround: set the following environment variables: OMP_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 MKL_NUM_THREADS=1

I am assuming some users will benefit from automatic multithreading; in other cases (like the one above) it interferes with our experiment schedule.

@taudor @MrM0nkey What is a good course of action? Disable multithreading entirely? Disable only when using the Experimenter class?

When I ran an experiment, I encountered OSError: [Errno 24] Too many open files

  File "input_trans_comparison.py", line 54, in <module>
	main(argv)
  File "input_trans_comparison.py", line 51, in main
	Experimenter(log_file_prefix + '.log', experiments).run()
  File "/home/nils/git/pypuf/pypuf/experiments/experimenter.py", line 52, in run
	job.start()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
	self._popen = self._Popen(self)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
	return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
	return Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
	self._launch(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 66, in _launch
	parent_r, child_w = os.pipe()

The command line script mv_num_of_votes.py hard codes some experiment parameters.
This file should be reworked in order to be able to pass all experiment parameters.
I started to rework this file in c903156 but this did not satisfy tests at the moment and need some additional work.

README.md line 105
example.py line 16
I think "4-XOR" should be "2-XOR".
Typos or I am thinking in the wrong way?

When using an LTFArray with bias, it will be inserted into the calculation in the eval method. However, if using the val method, it will not be used at all, as can bee seen from the code.

When running the tests using python3 -m unittest, a lot of messy, unimportant output is produced. I'd be nicer if we could just follow the testing progress and see errors, if any.

Readme states

python3 sim_learn.py 64 2 atf xor 12000 1 0xdead 0xbeef

as a CLI example, but the number of instances is missing and thus it yields an error.

Feature

The framework should be able to execute the optimization based learning algorithm on an arbitrary puf simulation.

Rework the low_degree,py module in order to satisfy our code conventions.

• rework the docstrings search for TODOs

I cannot run the example directly. I think the import part has something wrong.
Also, the training_set of LTFArray should be t_set

In order to fix #31 I found a Bug in TestExperimentMajorityVoteFindVotes.test_run_and_analyze.

In line https://github.com/nils-wisiol/pypuf/blob/master/test/test_experiment.py#L94 it checks for the self.result_overall_stab which is 0.0 if lines https://github.com/nils-wisiol/pypuf/blob/master/pypuf/experiments/experiment/majority_vote.py#L114-L121 found the best vote count .

Most (all?) functions in tools.py are untested.

When there are no new jobs in the queue, experimenter won't output a status line whenever a job as ended. So the last line always looks like:

Sat Jan 26 02:39:28 2019 90 jobs total, 69 finished, 21 running, 0 queued, progress 0.77, remaining time: 3:12:00

There will be no update when there are 20, 19, ... jobs running.

For better consistency, it would be better to change pypuf-helper==0.1.5 in requirenments.txt to pypuf-helper==0.2. See https://github.com/taudor/pypuf_helper/releases for more information.

python3 sim_learn.py 32 4 lightweight_secure_original xor 12000 100 1 0xabc 0x1 runs only one experiment, but 100 were expected.

By default, pep8 ignores missing whitespace around an operator. For example "testbla"+LOG_PATH will not be recognized as bad. It is possible to enforce pep8 to check E226 in order to enable a warning. See http://pycodestyle.pycqa.org/en/latest/intro.html for warning numbering.

Feature

The framework should be able to create a model with a eligible distribution.
It should be possible to apply input and output transformations on it.

Implementation

See simulation.py for a good a approach. The structure misses the possibility to add a new simulation model in order to be compatible with the learner.py.

I just found out (with help from @nils-wisiol ) that the unit-tests on a local master branch are failing. The cause is the absence of 2 directories which are not being created when pulling the master branch.
The directories are .../pypuf/test/test and .../pypuf/test/test/logs.

Feature

The framework should be able to execute Sölters logistic regression learning algorithm on an arbitrary simulation of a puf.