Giter VIP home page Giter VIP logo

research-iobserve / iobserve-analysis Goto Github PK

View Code? Open in Web Editor NEW
6.0 9.0 1.0 212.91 MB

Tools, probes and libraries used in iObserve to monitor and analyze software, as well as to plan and execute its modification (MAPE-K loop)

Home Page: https://www.iobserve-devops.net

License: Apache License 2.0

Java 99.78% AspectJ 0.02% Shell 0.18% sed 0.01%
analysis monitoring monitoring-data iobserve-analysis extraction linux mape-k

iobserve-analysis's Introduction

iObserve Monitoring and Analysis

Build Status

This repository contains the sources for the iobserve analysis including

  • common for the monitoring records (event types)
  • monitoring for the monitoring probes
  • analysis for the analysis plugins on basis of teetime
  • analysis.cli command line version of the analysis
  • analysis.service the service version of the analysis
  • splitter separates monitoring data streams by host
  • rac.creator this tool allows to create a RAC mapping based on monitoring data, a PCM model and a supplementary mapping file.
  • test.setup test tooling

The code is in a prototype stage and not fully functional, especially, as some parts are still located in other SVN repositories.

Setting up iObserve gradle build

Prerequisites

  • We use Java 8. you need an Java 8 JDK installed on your machine.
  • You need additional packages which are compiled in iobserve-repository. Checkout the iobserve-repository git repository alongside iobserve-analysis git clone [email protected]:research-iobserve/iobserve-repository.git
  • You need Kieker example data, a RAC and a corresponding palladio model

'Note: This is outdated, we switched to gradle'

Setup Build Environment

In the iobserve-analysis directory, create a file gradle.properties. Add the following variable declaration to that file and save it.

api.baseline=PATH_TO_IOBSERVE_REPOSITORY/mvn-repo/

Substitute PATH_TO_IOBSERVE_REPOSITORY with the actual absolute path on your computer to the iobserve-repository directory.

Compile

Depending on your local setup you may use:

  • ./gradlew build (linux, mac, etc.)
  • gradlew.bat build (windows)
  • gradle build (in case you have gradle 3.2.1 installed on your system)

Execution

After compilation, you may find in various folders packages containing executables, including the analysis, the replayer, the session reconstructor, etc. The process to use them is quite similar. In the following we introduce the most common tools.

Alongside all executables we have example configurations placed in the respective sub project folder with the suffix config. All executables support an environment variable project_OPTS where project must be substituted by the executable name. for analysis it is ANALYSIS_OPTS. Through this mechanisms further parameter can be specified which is quite helpful.

To activate logging use: export ANALYSIS_OPTS="-Dlog4j.configuration=file:///full/qualified/pathname/log4j.cfg" Please note that you have to adjust path and file name to your setup.

Analysis

The analysis is located in analysis/build/distributions/ You may find two archives containing executables.

  • analysis-0.0.3-SNAPSHOT.tar
  • analysis-0.0.3-SNAPSHOT.zip Choose your preferred kind of archive and extract it in our analysis directory (preferably outside of the directory of the git repository). After extraction you can start the analysis with:
  • analysis.cli-0.0.3-SNAPSHOT/bin/analysis (linux, mac, etc.)
  • analysis.cli-0.0.3-SNAPSHOT/bin/analysis.bat (windows)

usage: iobserve-analysis -h,--help show usage information -c configuration file

For a minimal analysis, type: analysis-0.0.3-SNAPSHOT/bin/analysis -c analysis.config

Collector

Usage:

[options] Options:

  • -c, --configuration Configuration file.

Log-Replayer

Session Reconstructor

Service Privacy Violation

iobserve-analysis's People

Contributors

alessandrogiusa avatar davemuc avatar dietmarkracht avatar janniskuckei avatar jwegert avatar larsjur avatar lbl-dev avatar madcau avatar nicolas-boltz avatar robertheinrich avatar slenga avatar stu78250 avatar tpoeppke avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

nicolas-boltz

iobserve-analysis's Issues

Fix issue with DynamicEventDispatcher

The current event dispatcher implies an internal sequence on the events sent out to a specific port. This can be quite nontransparent, as you need to know all previous port declaration to understand the output of a specific port.

Merging two different CorrespondenceModel concepts

An older bachelor thesis defines another kind of correspondence model, than the one currently used and extended ProtoComin a way that a valid correspondence model is automatically being built by ProtoCom.

The current and new correspondence models need to be merged in a way to keep the ProtoCom functionality of the new and the more intuitive way of accessing the model data of the current model.

UDE: Implementation privacy warner

An empty privacy warner already exists in /service.privacy.violation/src/main/java/org/iobserve/service/privacy/violation/filter/PrivacyWarner.java

Fix logging.

Logging does nor work everywhere as expected. This might have to do with additional log configurations.

Feature Toogles

At the moment all features of iObserve are bundled in the same configuration.
This causes some issues if single features should be tested as unrelated ones also have to be provided.
For example, all visualizations and analysis steps are enabled as default.
This creates dependencies to other services that may not be used in all executions.

Therefore, it will be useful to create feature toggles, that also define the expected parameters or split the larger configuration into smaller and more specialized ones.

Organize libraries for planning, adaptation, and execution

So far, AdaptationData records and a so called ModelGraph were used to pass data within iObserve's control loop. With the restructuring of the phases into separate gradle-projects the adaptationdata library was created to share these classes between the projects. However, it might make sense to restructure/split up this library in the future.

Fix database initialization

Presently, we have two kind of model providers:

  • File based model providers
  • Database based model providers
  • Generic database model provider

The file based providers could be used to initialize the NEO4J database. Beyond that they should be removed.

UDE: Create alarm analysis

An empty filter called AlarmAnalysis exists in /service.privacy.violation/src/main/java/org/iobserve/service/privacy/violation/filter/AlarmAnalysis.java

This has to be filled with the code necessary for the AlarmAnalysis. In advance it must be defined which information is necessary to detect data flow and classify it as an alarm.

You may document everything in the iobserve-documents project.

Gradle build system design broken

I have peeked into the Gradle build set up for the iObserve analysis, which I guess is a intended to be the replacement of the original Maven build. While I recently improved the Maven build to mitigate issues with Eclipse packages, the Gradle build does not support these features. However, it should. In research-iobserve/iobserve-analysis branch merged-observe, you may find all the necessary information in the README.md of the org.observe.releng directory.

KIT: Create RAC mapper for dataflow events.

We need an RAC mapper for events which are used to determine data flow. You have to clarify which events we can receive, e.g., EntryLevelEvents, to indicate data flow. Then these events must be transformed into their PCM equivalent.

Configuration File for iobserve

the iObserve analysis is a rather complex piece of software utilizing a vast set of simple filters and composite stages. The configuration over command line parameters for all potential combinations is unsatisfying. Therefore, we should switch to a configuration file.

In general the analysis supports X distinct analyzes:

  • Behavior analysis
  • Deployment and Allocation
  • Dataflow and Geolocation
  • Separate Geolocation
  • Call and Messages Traces

Not all these features are sufficiently implemented. However, they are intended features which should be activated separately. Also for some analyses, there are different strategies and dependencies. for example, the behavior analysis uses a set of different clustering and classification mechanisms.

Tests restructure and test coverage

Tests need to be (partly) restructured as it is hard r.n. to find a test to a given class. Also test coverage, as specified by clover, needs to be increased with meaningfull tests.

NeoEMF to improve the ModelProvider

The current ModelProvider uses our own EMF to Neo4J mapping. It was developed as the Neo4EMF-project seemed to be dead and incomplete. However, they renamed their project in NeoEMF and did not remove the old github projects. Anyway, with continuous support for NeoEMF, we could try to use it instead of our own EMFtoNeo4J mapping. Therefore, we have to evaluate the fitness of NeoEMF with Neo4J and other DB-systems.

http://www.neoemf.com/

And in case it works out, port the ModelProvider to this new library.

NEO4J Proxy Node Resolving

In our current implementation f the model database, models that are referenced by the saved node (and are saved elsewhere) are as 'proxy nodes'.
These nodes contain only the reference to the represented model, but can't be automatically resolved.
For future work, it can be useful to enable Proxy Node Resolving and add further information like model version / variant.

Collect configurations in properties and ConfigurationKeys

Collect all configurations that are used for filtern in the properties file (for documentation) and ConfigurationKeys, to make sure they can be referenced. Also emigrate configurations out of filters, where the configuration is parsed internally (should happen externally).

Support for SE software project

We want to log user behavior during the the SE software project suitable for the iObserve behavior analysis. We already have a property SessionAndTraceRegistrationPayloadFilter which creates EntryLevelBeforeOperationEvents, AfterOperationFailedEvents, and AfterOperationEvents. However, there exists already another filter to log behavior, which is used by Henning. He has an analysis tailored for this needs. We need to port his analysis to iObserve or adapt his analysis to support the new record types.

Missing source code

When I try to build the analysis the following packages and all classes in them are missing. In detail they are:

  • org.gag.* BTW the name is, well, misleading ;-) Is this a funny organization?
  • protocom.extension.*
  • giusa.parser.*

These packages should either be integrated or, in case they are general purpose and used in different projects, should be at least provided in their own repository with their own Maven or Gradle build.

KIT: Geolocation Updater for PCM Model

Fix the geolocation updater for the PCM model.
The present implementation /analysis/src/main/java/org/iobserve/analysis/privacy/GeoLocation.java
Might not do what we want.

UDE: implement Model Probe Controller

The model probe controller gets a set of warnings and computes which probes have to be deactivated and activated to accommodate these warnings. As it always gets a full set of warnings, it only needs to remember which probes it has activated in the past so that it can deactivate them if they are no longer needed for the incoming set of warnings.

Create separate sub-projects for planning, adaptation and execution

Currently the analysis sub project contains all sorts of code which does not belong to the analysis service. Unfortunately, they are packed in the same org.iobserve.analysis. To fix this issue, I propose to move them to separate projects which would also allow to really create executables for planning and adaptation.

What to do:

  • Create a new directory for planning, adaptation and execution
  • Copy the build.gradle from analysis.cli to all these directories and adjust the parameters
  • Add the directories to the settings.gradle file (after all the library components)
  • Re-import the gradle main project in Eclipse this will automatically import the new directories as empty projects
  • Create a src/main/java and src/test/java source folder in each project
  • Move packages within Eclipse from analysis to the respective projects
  • Fix the dependencies in the build.gradle file if necessary

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.