hastic-zzz / hastic-server Goto Github PK
View Code? Open in Web Editor NEWHastic data management server for analyzing patterns and anomalies from Grafana
License: GNU General Public License v3.0
Hastic data management server for analyzing patterns and anomalies from Grafana
License: GNU General Public License v3.0
We want to send data source where we will find correlated metrics automatically. When just reports than metrics which analytic found.
Sounds futuristic , but lets make subtasks.
For example, we need to detect patterns in for few metrics together like in merge sort.
Another example is: https://github.com/hastic/hastic-server/issues/44 where apply model to predict class.
So we need to keep in memory model for few pattern predictors.
We want to support old versions of Node.js
I think we better setup CI for this and update it in docker
I am sorry to say that, but I need to use google to find instructions about how should I install python and pip.
I just want to start to use software. Is there a nice way to install Python/pip on my machine without googling ? :)
For example, with node.js you only need to link to: https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions and that's it. Just one command.
@rozetko what do you think?
All of your models: drops, peaks, ... don`t give a shit about deletion.
We want to make node.js part the service which responsible for data processing.
So python doesn't write to anywhere.
We have entities like:
Current release-archive size is ~55MB
We should reduce it somehow
Steps to reproduce:
data/anomalies/all_anomalies.json
Now there is only new anomaly in all_anomalies.json
It happens because loadAnomaliesMap()
method doesn't get executed on anomaly insert
It would be nice to have Kalman filter for example before we do analytics
We want to use zeromq for messaging between node.js / python
It will allow to separate processes and debug easier
We update alerts.json
to send "notifications" about new detected alerts. It is crazy.
I think it is not the only place.
We need to add documentation on how to build release version
We can't scale with files. Obviously.
We can face a problem where user has an older version of hastic-panel and we need to return version of hastic-server. Also it is necessary for bug reports.
So we need to
The first time we get this problem: https://github.com/hastic/hastic-server/pull/65
If you make a new pattern with name "drop" and you already have "drop" in database, learning state would start
Currently only InfluxDB datasource is supported
We should find a way to support other Grafana's datasources
has both transport & logic in one code. Lets separate it.
https://nodejs.org/api/child_process.html#child_process_options_detached
I am not sure that we close it properly @rozetko
let's check it
Guess we don't need 2 types of build because node 6.14 build would work in any 6.14+ version.
Also, documentation with different types of build looks confusing.
@jonyrock what do you think?
Currently, dataset doesn't get updated after first downloading
We should update it on every learning
We now want to have pattern for multiple series.
We need to make
@rozetko
I think our docker file is not actual. We use requirements.txt
and least
We should send stacktrace and some additional data somewhere on errors for easier debugging
Imagine log which looks like this:
[ANALYTICS] something
[ANALYTICS] something more
[SERVER] oh oh
[SERVER] more logs
[ANALYTICS] learning
Steps to reproduce:
analytics/
pip3 install -r requirements.txt
python3 server.py
You would see error:
Traceback (most recent call last):
File "server.py", line 7, in <module>
from worker import Worker
File "/mnt/c/Users/rozetko/git/hastic-server/analytics/worker.py", line 2, in <module>
from anomaly_model import AnomalyModel
File "/mnt/c/Users/rozetko/git/hastic-server/analytics/anomaly_model.py", line 2, in <module>
from data_provider import DataProvider
File "/mnt/c/Users/rozetko/git/hastic-server/analytics/data_provider.py", line 1, in <module>
import pandas as pd
File "/home/rozetko/.local/lib/python3.5/site-packages/pandas/__init__.py", line 42, in <module>
from pandas.core.api import *
File "/home/rozetko/.local/lib/python3.5/site-packages/pandas/core/api.py", line 10, in <module>
from pandas.core.groupby import Grouper
File "/home/rozetko/.local/lib/python3.5/site-packages/pandas/core/groupby/__init__.py", line 2, in <module>
from pandas.core.groupby.groupby import (
File "/home/rozetko/.local/lib/python3.5/site-packages/pandas/core/groupby/groupby.py", line 42, in <module>
from pandas.core.dtypes.missing import isna, isnull, notna, _maybe_fill
ImportError: cannot import name 'isna'
I believe that analytics should know about how extract data. It is server responsibility.
We need to send data for initial learning and then data about new data from datasource.
There are many benefits from this.
@rozetko lets stop querying grafana from python
I like how Grafana does it:
https://grafana.com/grafana/download?platform=linux
just installation from source and everything works
I think we should rethink this part: https://github.com/hastic/hastic-server#known-bugs--issues
Let's check what is actual and make issues here on github if we have bugs.
We still use anomaly names in some cases instead of IDs
It is really confusing
We should use only IDs
we use any
- like definitions for task, but we can do it more accurate
let task = {
type: 'learn',
anomaly_id: anomalyId,
pattern,
segments: segments
};
When server already started learning / prediction task and you delete anomaly - it doesn't know about this fact and continue doing the task => you have to wait for it to finish before it starts analysing another anomaly
Now anomaly name is case-sensitive
If you create anomaly with capital letter(s) in name - you get "Not found" or "Internal error" alert
We should change all names to lower-case
We should get HASTIC_API_KEY from config if it exists. Otherwise, use environment variable.
We need to wrap task result payload in res
with metadata with taskId
and result of execution.
so it's like:
{
"taskId": 123123,
"status": "ok",
"data": {
... payload data ...
}
}
SImillar to drops. Just want to see it working
It would be nice to see a proper error message in panel alert instead of just "Internal error"
We haven't defined how system should behave when we have no data. We need to understand requirements to the system first.
It is simpler version of https://github.com/hastic/hastic-server/issues/44
where we don't make proper classification, we just select the pattern with best fit.
It would be also nice to have a way to debug python code
Any behavior even if is defined should be rethinked . I think it is the case where user should define which strategy to choose.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.