htm-community / nab Goto Github PK

Would be useful for #6

• we already use a JSON param structure
• a "runner" class (which one is it?) in NAB needs to return score for a single detector on all datasets
• the runner needs be callable as "main"

Would this be feasible?

I have noticed, that htmcore_detector.py use random seed (=0).
In fact, it is just about the RDSE encoder, there is the seed param not supplied. I can add this param with PR.

I am not sure how optimalization exactly works here in NAB (the one with docker?), but in my opinion for optimalization process, the detector should be deterministic. (i am imagining itself as optimizer and i am trying to find better params, but each call the result is different with no dependency of what params i have changed.) This potentially could help @steinroe.

For example, for art_daily_flatmiddle i get these scores each call: -0.787,-0.567,-0.677....

What you think about this @breznak ?

https://github.com/htm-community/htm.core/runs/247778963

compute err metric from predicted values, such as RMSE, SSE, R2,..

• this is a common comparison for time-series algorithms in papers
• Require each model to predict a value? (then we can compute err metric easily)

numenta#348

as the numenta/NAB does.

• naive: #4
• spatial anomaly: "a hack technique", let's see how significant it is, #15

numenta#212

numenta#350

A possible bug, the self.inputMin, inputMax is provided by the NAB API.

• one some dataset (I don't know which) the values are all the same (ie the dataset is just a flat line)

57: Results have been written to /mnt/store/devel/HTM/NAB/results/htmcore/realTweets/htmcore_Twitter_volume_UPS.csv
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/mnt/store/devel/HTM/NAB/nab/detectors/base.py", line 147, in detectDataSet
detectorInstance.initialize()
File "/mnt/store/devel/HTM/NAB/nab/detectors/htmcore/htmcore_detector.py", line 284, in initialize
assert self.inputMin < self.inputMax, "FAILED "+str(self.inputMin)+" "+str(self.inputMax)
AssertionError: FAILED 45.0 45.0
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "run.py", line 247, in
main(args)
File "run.py", line 96, in main
runner.detect(detectorConstructors)
File "/mnt/store/devel/HTM/NAB/nab/runner.py", line 134, in detect
self.pool.map_async(detectDataSet, args).get(99999999)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
AssertionError: FAILED 45.0 45.0

• our c++ ScalarEncoder has a check that min < max

  • which is reasonable, for otherwise the encoder would use 0 for some internal sizes.

• I don't know if the problem is a bug in how this NAB implementation computes/deals with setting the min.max,

• or if we have to workaround that in c++ ScalarEncoder?

I was installing NAB in Win7 environment, the error came after running python setup.py install

Using e:\python37\lib\site-packages
Finished processing dependencies for nab==1.0
The system cannot find the path specified.
python: can't open file 'nab/detectors/htmcore/setup.py': [Errno 2] No such file or directory

I had nab-1.0 in site-package folder and pip list can detect nab. How would it affect usage? (My guess is that I couldn't use htmcore as a detector)

By the way, I commented out long_description=read("README.md") in setup.py due to encoding issue(I got unicode value error and couldn't install it even I particularly chcp 65001). I don't know if it was related.

If it's not already provided(?)

• add "hotgym" data from htm.core (this is the "Hellow world" time-series dataset used in HTM commonly)
• install NAB as a part for QA in htm.core and run htmcore detector results on (some) datasets

• create the model
• see if I can publish the data under CC

The current scores are not really good!

Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 50.83
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 49.95
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 52.64
Final scores have been written to /mnt/store/devel/HTM/NAB/results/final_results.json.

LSTM is currently state of the art for time-series, see how it compares with HTM

numenta#339

I am wondering how/if we would merge with upstream NAB from Numenta, which is now Python3 too.

This codebase has been converted a long time before that by @ctrl-z-9000-times , and a bunch of other work has been done atop of that.

What do you think? Attempt merging, start anew on theirs master and add our changes, or keep this as diverged fork?