Giter VIP home page Giter VIP logo

anomalydetection's Introduction

AnomalyDetection

LSTM对时间序列的数据做预测,通过偏离值的大小达到异常检测的目的

1.思路

  用一个网格7x24个小时的数据预测该网格第7x24+1个小时的in out,如果这个预测in out与真实值的差距大于一个阈值(该阈值最终取3,见Fig3),就把这小时对应的网格当做一个异常的网格,可以达到实时检测的效果

Fig1.某网格7*24小时(一周)的时序图

Fig2.某网格相邻四周的时序图

Fig3_1.训练loss与epoch的箱型图

Fig3_2.最后一个epoch的loss箱型图

(异常的阈值取3因为最后一个epoch的均方差loss基本小于9)

Fig4.训练loss与每个step的曲线图

Fig5.某网格第五周的预测图

Fig5中红线是真实值±3,代表异常的阈值曲线,绿线是预测值曲线,蓝线是真实值的曲线,绿线超出红线即视为异常

2.流程

1)过滤掉稀疏的网格,一个网格用一个向量存储2088个小时的in/out,如果2088个值里边有超过三分之一的值为0,则过滤掉,不用该网格的数据进行模型训练。原因:一天24小时,可以有三分之一的时间比如0a.m.-8a.m. 没有in/out,其他时间段有in/out,比较符合该数据合理的直觉,共1317个符合条件的向量,原始数据有104000*2个向量

2)最终训练数据生成:将符合条件的向量用一个24*7的窗口划分

A.为了减少模型过拟合,窗口之间不重合

B.为了增加模型的适用性,划分窗口时加一个offset,使得模型从一周的随便一个小时起开始预测都可以, 最终有大约1317*(2088/(24*7))=16368个训练样本

in out and some temp images zip file

anomalydetection's People

Contributors

deriq-qian-dong avatar lihaolixuewei112612 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.