Giter VIP home page Giter VIP logo

kapacitor_prophet-udf's Introduction

a udf (batch) for kapacitor using prophet

facebook prophet doc
main thread on influxdata community

it's a unix socket based udf.
by default it use /tmp/udf_prophet.sock , make sure you give enough read/write right for kapacitor daemon/process to interact with the file.
you can override the file location as the first argument running this script.
(i was able to use run it on docker and/or kubernetes)

Inspired by this talk, by the creator of kapacitor.

Nathaniel Cook - Forecasting Time Series Data at scale with the TICK stack

InfluxDb support by default holt_winters for forecasting timeseries.
Note that you need to specify the seasonality pattern to forecast with holt_winters, without the right seasonality pattern your forcasted value may endup being not that usefull.
HOLT_WINTERS()
Finding the seasonality pattern of a timeseries is not that easy ! :).
Prophet on the other hand try to find the seasonal pattern it self, and is pretty good a it.

Using it with a system like kapacitor make sense, since you timeseries and seasonality pattern may change in the future.
And unless you have enough time/people to track those change, you had rather have something else doing it for you !

another nice article about forecasting jobs, if you are curious!

kapacitor_prophet-udf's People

Contributors

atroncy avatar

Stargazers

avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

voiprodrigo

kapacitor_prophet-udf's Issues

ERROR:root:error processing request of type end: signed integer is greater than maximum

Thanks a lot of the creation of the script and advise on the last issue!
I have made an immense progress, kapacitor can now access udf_prophet.sock and i can access @prophet in kapacitor tickscript.

But now i am getting an error after i enable the task.
Please advise.

Call stack-

INFO:root:Agent finished connection 2
INFO:root:Starting Agent for connection 3
INFO:root:init 
INFO:root:begin batch
INFO:root:fit model

Traceback (most recent call last):
  File "/home/linuxuser/.local/lib/python2.7/site-packages/kapacitor/udf/agent.py", line 124, in _read_loop
    self.handler.end_batch(request.end)
  File "/etc/kapacitor/prophet.py", line 145, in end_batch
    m.fit(df)
  File "/home/linuxuser/.local/lib/python2.7/site-packages/fbprophet/forecaster.py", line 1057, in fit
    history = self.setup_dataframe(history, initialize_scales=True)
  File "/home/linuxuser/.local/lib/python2.7/site-packages/fbprophet/forecaster.py", line 241, in setup_dataframe
    df['ds'] = pd.to_datetime(df['ds'])
  File "/home/linuxuser/.local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 592, in to_datetime
    values = convert_listlike(arg._values, True, format)
  File "/home/linuxuser/.local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 302, in _convert_listlike_datetimes
    allow_object=True)
  File "/home/linuxuser/.local/lib/python2.7/site-packages/pandas/core/arrays/datetimes.py", line 1857, in objects_to_datetime64ns
    require_iso8601=require_iso8601
  File "pandas/_libs/tslib.pyx", line 460, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 685, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 809, in pandas._libs.tslib.array_to_datetime_object
  File "pandas/_libs/tslib.pyx", line 803, in pandas._libs.tslib.array_to_datetime_object
  File "pandas/_libs/tslibs/parsing.pyx", line 99, in pandas._libs.tslibs.parsing.parse_datetime_string
  File "/home/linuxuser/.local/lib/python2.7/site-packages/dateutil/parser/_parser.py", line 1374, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/home/linuxuser/.local/lib/python2.7/site-packages/dateutil/parser/_parser.py", line 655, in parse
    ret = self._build_naive(res, default)
  File "/home/linuxuser/.local/lib/python2.7/site-packages/dateutil/parser/_parser.py", line 1238, in _build_naive
    if cday > monthrange(cyear, cmonth)[1]:
  File "/usr/lib/python2.7/calendar.py", line 121, in monthrange
    day1 = weekday(year, month, 1)
  File "/usr/lib/python2.7/calendar.py", line 113, in weekday
    return datetime.date(year, month, day).weekday()
OverflowError: signed integer is greater than maximum
ERROR:root:error processing request of type end: signed integer is greater than maximum

Thanks

growth=logistic

Hi @atroncy,

If i need to use growth='logistic' then i need to provide cap and floor, how do i achieve that in prophet.py?

Example CPU usage where cap is 100% and floor is 0%

Thanks

AttributeError: 'module' object has no attribute 'AF_UNIX'

Hi,

Thanks for the script!

Currently getting the following error in kapacitor Log file when I start kapacitor from command line:
kapacitord.exe -config kapacitor.conf
ts=2019-11-25T07:59:18.297-05:00 lvl=info msg="UDF log" service=udf text=" server = Server(path, accepter())"
ts=2019-11-25T07:59:18.297-05:00 lvl=info msg="UDF log" service=udf text=" File "C:\Monitoring\kapacitor_udf\kapacitor\udf\agent\py\kapacitor\udf\agent.py", line 180, in init"
ts=2019-11-25T07:59:18.298-05:00 lvl=info msg="UDF log" service=udf text=" self._listener = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM )"
ts=2019-11-25T07:59:18.298-05:00 lvl=info msg="UDF log" service=udf text="AttributeError: 'module' object has no attribute 'AF_UNIX'"
Do i need to use Socket or something similar to this "#socket = "/path/to/socket"" ? If yes then please advise.

Following is the UDF section form kapactor.conf

[udf.functions.prophet]
prog = "C:/ProgramData/Anaconda2/python.exe"
args = ["-u", "C:/Monitoring/kapacitor_udf/kapacitor/udf/agent/examples/prophet/prophet.py"]
timeout = "60s"
[udf.functions.prophet.env]
PYTHONPATH = "C:/Monitoring/kapacitor_udf/kapacitor/udf/agent/py"

I can use Prophet package outside kapacitor (Anaconda jupyter notebook), therefore I understand that I need to configure UDF in a very specific way to be able to use Prophet.py as UDF with Kapacitor.

I know there has been issues using UDF with python3 therefore i have been trying with python2 conda environment.
Kapacitor Version: 1.5.3
OS: Windows

Other references:
https://community.influxdata.com/t/prophet-forecasting-udf-unable-to-write-response-to-influxdb/6348/20

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.