Monitor RAM/CPU usage of OWTF (and its child processes) as well as the total consumption for the testing machine (where OWTF is running). Use SSH to login to that machine and determine the stats.

In a multi module web application or a very big web application, there can be changes as time progress.
You can assume it as a continues development. So in this kind of scenarios, teams are only interested to run security scans/checks only on those webpages or modules or series of steps (ZEST recordings) which are developed or modified recently.

Having this feature will make OWTF more useful on a daily basis and more acceptable for Dev community.
Particularly, more useful for Agile based development environment.

The logging has to be implemented in the webui module.

Implement a function to save necessary config on receiving SIGINT signal.

Use the OWTF API to get targets for the active session and determine if they are up/down. If down, then pause the worker for that target and all other work in the worklist (for that target).

Right now, the health monitor is stable and working. But as OWTF's internal architecture changes (and will soon), the code will have to modified accordingly.

We needs tests (functional/unittests) to ensure the tool does not break on changes.

Work remaining:

• Clean cache, apt cache, temporary files, etc (disk)
• Pause/resume OWTF targets based on network status (network)

Implement a network connectivity monitor

Every 2-3 minutes, check if connectivity is OK
If connectivity fails, check again, after 5-10 seconds (things might fail under heavy load)
if it still fails, then pause ALL workers
Keep checking connectivity
If connectivity works again, check again, after 5-10 seconds
If it still works, then restart ALL workers, 1 at a time, while connectivity continues to be checked
Before launching ANY plugin, check connectivity
What we achieve: If an ISP takes down the traffic, there is an outage, etc. We lose as little results as possible, because we will auto-pause and auto-restart the whole scan.

notify module is currently not available on settings page to control its parameters.

Add functionality of the actions taken by module to UI. Like add command in CLI to clean disk or resume/pause OWTF.

Cross-posted from OWTF issue tracker owtf/owtf#424

@7a: I ran into a situation where there was enough disk space but the number of free inodes became 0, we need to:

Safely pause everything + abort if the number of free inodes is getting “too close” to zero
Try to reduce the number of files we create, which has something to do with “running out of inodes” problem.
How to check if there are enough free inodes, running this command:
df -i

UPDATE: After rebooting, 1 million (!) inodes became available, suggesting that our temporary files (and/or the ones created by the tools we run) on /tmp consumed these many inodes. Because nothing else was changed.
Clean-up script:

NOTE: to be run from tmp/owtf/proxy-cache, with all workers paused
clean_up.sh

#!/bin/bash

for p_dir in $(ls); do
    if [ -d "$p_dir" ]; then
        (
            cd $p_dir
            echo "Cleaning up: $(pwd) - $(ls -l | wc -l) files"
            time for i in $(ls); do rm -f $i; done &
        )
        #echo "pwd out: $(pwd)"
    fi
done

Also,

Candidate 1) w3af temporary files
root@k:~/.w3af/tmp# df -vh ; rm -rf *; df -vh ← 7GB liberated (!)

Candidate 2) apt-cache

Step 1)du -sh /var/cache/apt/archives/ (5.7G /var/cache/apt/archives/ ← (!!!!) )
Step 2) apt-get clean
Step 3) du -sh /var/cache/apt/archives/ (116K /var/cache/apt/archives/← 6GB liberated (11GB on another box!) )

The "Usage" section should include:

• Downloading the repo
• Compiling with necessary documentation
• How to run and use the monitor

When running the program for first time, or running it after deleting the .owtf_monitor directory present in $HOME.

For the first time it fails in run, a panic occurs due to this database query returns the error.

On analyzing I found that the database contains the data, but still throws an error.

The functions have to be implemented to enable saving of config to database with different profile if asked.
If there is error found in loading of profile switch to default profile.

Currently go get github.com/owtf/health_monitor is not supported.

Logging has to be implemented in the api module

Cross-posted from owtf/owtf#423

Hi Folks, I don't have log output to show the incidence however I have had incidence atleast once when the OWTF scan process created so much of log files which resulted in 100% disk usage which causes issues towards the system stability.
A check to see how much is free storage and may be doing a tar.gz / tar.bz2 of the log files should help in long way as the data is anyways plain text.

Fix

Monitor size of proxy cache and log files and add a control method to backup the logs in a .tar.gz format to a directory of user's choice from the monitor's web interface.

In case of multiple profile, there should be an option to choose a profile on settings home page.

When SendModuleStatus is called by any module it is working perfectly. But if it called by webui, using api it does not sends data to the channel. The execution is blocked at the channel.