etsy / 411 Goto Github PK
View Code? Open in Web Editor NEWAn Alert Management Web Application
Home Page: https://demo.fouroneone.io
License: MIT License
An Alert Management Web Application
Home Page: https://demo.fouroneone.io
License: MIT License
composer.lock defines ecl url as './ecl' however that path doesn't exist, this causes composer install to fail.
"dist": { "type": "path", "url": "./ecl", "reference": "565c5929ab503b6241c9fd6ddc65ea478a817f1e", "shasum": null }
Hey,
i can't get this issue solved while trying to install 411.
core@ubuntu:/var/www$ 411/bin/create_site.php
PHP Fatal error: Uncaught exception 'PDOException' with message 'could not find driver' in /var/www/411/phplib/DB.php:49
Stack trace:
#0 /var/www/411/phplib/DB.php(49): PDO->__construct('sqlite:/var/www...', 'root', NULL, Array)
#1 /var/www/411/phplib/DB.php(35): FOO\DB::connect('sqlite:/var/www...', 'root', NULL)
#2 /var/www/411/phplib/411bootstrap.php(62): FOO\DB::init()
#3 /var/www/411/bin/create_site.php(7): require_once('/var/www/411/ph...')
#4 {main}
thrown in /var/www/411/phplib/DB.php on line 49
The queries used to generate the Dashboard data are pretty slow. Rewrite them so they're faster or experiment with querying ES instead.
Having a problem with the SSL connection, Elasticsearch is reporting an 'unknown CA' error and 411 has the error listed below.
If I create a simple ES PHP connector as follows, the connection works fine:
<?php
require 'vendor/autoload.php';
$hosts = ['https://localhost:9200'];
$myCert = '/home/mycert.pem';
use Elasticsearch\ClientBuilder;
$client = ClientBuilder::create()
->setHosts($hosts)
->setSSLVerification($myCert)
->build();
$params = [
'index' => 'logstash-2016.08.30',
'type' => 'logs',
'id' => 'AVbdvHEwAG-7iPeMOVRs'
];
$response = $client->get($params);
411 Configuration
# Elasticsearch configuration
$config['elasticsearch'] = [
# Configuration for the 411 Alerts index.
'alerts' => [
'hosts' => ['https://localhost:9200'],
'index_hosts' => [],
'ssl_cert' => '/home/mycert.pem',
'index' => null,
'date_based' => false,
'date_field' => 'alert_date',
'src_url' => null,
],
# Configuration for the logstash index that 411 queries.
'logstash' => [
'hosts' => ['https://localhost:9200'],
'index_hosts' => [],
'ssl_cert' => '/home/mycert.pem',
'index' => 'logstash',
'date_based' => true,
'date_field' => '@timestamp',
'src_url' => null,
],
];
411 Error
[Wed Aug 31 07:50:46.005023 2016] [:error] [pid 1598] [client 1.1.1.1:35031] except [NONE] Elasticsearch\\Common\\Exceptions\\NoNodesAvailableException: "No alive nodes found in your cluster" at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php:51] 0:[Elasticsearch\\ConnectionPool\\StaticNoPingConnectionPool->nextConnection() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php:71]] 1:[Elasticsearch\\Transport->getConnection() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php:89]] 2:[Elasticsearch\\Transport->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:228]] 3:[Elasticsearch\\Connections\\Connection->Elasticsearch\\Connections\\{closure}() called at [/var/www/411/vendor/react/promise/src/FulfilledPromise.php:25]] 4:[React\\Promise\\FulfilledPromise->then() called at [/var/www/411/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php:55]] 5:[GuzzleHttp\\Ring\\Future\\CompletedFutureValue->then() called at [/var/www/411/vendor/guzzlehttp/ringphp/src/Core.php:341]] 6:[GuzzleHttp\\Ring\\Core::proxy() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:283]] 7:[Elasticsearch\\Connections\\Connection->Elasticsearch\\Connections\\{closure}() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:159]] 8:[Elasticsearch\\Connections\\Connection->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php:99]] 9:[Elasticsearch\\Transport->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/AbstractEndpoint.php:79]] 10:[Elasticsearch\\Endpoints\\AbstractEndpoint->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Namespaces/BooleanRequestWrapper.php:34]] 11:[Elasticsearch\\Namespaces\\BooleanRequestWrapper::performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Namespaces/IndicesNamespace.php:919]] 12:[Elasticsearch\\Namespaces\\IndicesNamespace->existsTemplate() called at [/var/www/411/phplib/ESClient.php:68]] 13:[FOO\\ESClient->initializeIndex() called at [/var/www/411/phplib/ESClient.php:31]] 14:[FOO\\ESClient->__construct() called at [/var/www/411/phplib/REST/Dashboard.php:16]] 15:[FOO\\Dashboard_REST->GET() called at [/var/www/411/phplib/REST.php:107]] 16:[FOO\\REST->route() called at [/var/www/411/htdocs/api/dashboard.php:6]], referer: http://myserver.com/
There might be a bug on the dashboard API.
On my installation, the Crons are running, but i continue to get the Processor has not run in more than 20 mins! Error on the dashboard.
no_recent_cron comes from the /api/dashboard endpoint, which backs of
master/phplib/REST/Dashboard.php[line 81]
Looking in the database directly, the meta table, "last_cron_date" "2016"
$data['no_recent_cron'] = (int) $meta['last_cron_date'] < ($_SERVER['REQUEST_TIME'] - 20 * 60);
Since ($_SERVER['REQUEST_TIME'] - 20 * 60) returns a very big number, this statement will always be "TRUE" triggering the alert message.
To fix?
Hello,
I really like the project so far.
I was wondering if It's possible to create an alert using the API.
I already have custom app that can call a webhook when an alert is triggered. I'd like to add them into 411.
Edit : TLDR : Can I push an alert to 411 rather than pulling it from ES or Graphite ?
Hi,
I'm trying to use the following query:
type:authlog AND failure:* NOT ssh_interface:public
Results in the error Error: Catch all: Expected ":" or [a-zA-Z0-9._] but " " found.
Is this supposed to work with plain Lucene query syntax, or am I making horrible assumptions about ESquery?
Thanks :)
Firstly - Great project!
Re. config.php
I have a time based index 'logstash-apache-*' in ES. I have tried logstash, logstash-apache-, logstash-2016.08.12 in config.php but still get a Failure in the health screen. The alerts section works fine.
Any debug logging available?
'logstash' => [
'hosts' => ['http://192.168.1.196:9200'],
'index_hosts' => [],
'index' => 'logstash-apache-',
'date_based' => true,
'date_field' => '@timestamp',
'src_url' => null,
Curl request:
curl http://192.168.1.196:9200/logstash-apache-2016.08.14?pretty
{
"logstash-apache-2016.08.14" : {
"aliases" : { },
"mappings" : {
"default" : {
"_all" : {
"enabled" : true,
"omit_norms" : true
.....
Hi,
I've made a new install with the latest version, configure mysql instead of a previous Sqlite default install, and now i don't have an SQL syntax error with the LIMIT clause ! Multiple-indicies is OK too !
I've deleted the 411 indices in elasticsearch to start from a clean situation !
A search is configured to send notification once a day with the cron notation and is enabled (high priority) !
I don't know what i'm doing wrong, but i can't receive any email notification ???
If i execute the search everything is ok, and i receive the email notification !
My local smtp is up and running (Postfix) ...
Can someone help me please ?
Thanks for your help.
PS : The enable button is not so clear with the default colors (enable vs disable) !
We're seeing cron output every minute as a result of running cron.php & worker.php.
It might be an idea to silence the scripts (or have a switch to do this) and/or modify the setup instructions to include > /dev/null 2>&1
for the cron entry.
Cheers.
Here's the curl to reproduce the exception
curl 'https://demo.fouroneone.io/api/alert/bootstrap?query=state%3A(0+1)' --1.0 -H 'Host: demo.fouroneone.io' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://demo.fouroneone.io/alerts?query=state:(0+1)' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: 411sess=9a060399e2e44a4332b18f55098d0b8cf5d907fd5fbe7a2fe6705c95cb235db1%3A%7B%22nonce%22%3A%2263IlrPHsjCnCw3YEZ5p9%5C%2F2xvNViSmDMA%22%2C%22_%22%3A1474008875%2C%22id%22%3A2%7D' -H 'Connection: keep-alive'
Is it possible possible to alert on cardinality having some range. ex:
*| agg:terms field:event_data.TargetUserName| agg:card field:event_data.IpAddress
This returns the unique count of ip adresses per user. I want to trigger an alarm when the count is in range [x TO y].
Adding a new data source for an existing Search is tedious atm, see #3 for details. We should add support for registering multiple subtypes under a Search. Ideally, this would let you just modify a config file and have the new type up and running.
Hi,
In the initial setup guide, the cronjob configuration seems outdated.
I've followed the guide today, and got the system up and running, but cronjob didn't work
# Add the following line into your crontab.
$ crontab -u USER -e
/var/www/411/bin/cron.php > /dev/null 2>&1 && /var/www/411/bin/worker.php > /dev/null 2>&1
Running the CRON itself produces the following error
[+] Scheduler: 1475078702
except [NONE] Errno 2: "pcntl_exec(): Error has occurred: (errno 13) Permission denied" at [/var/www/411/phplib/Scheduler.php:44] 0:[pcntl_exec() called at [/var/www/411/phplib/Scheduler.php:44]] 1:[FOO\Scheduler->process() called at [/var/www/411/bin/cron.php:47]]
err [411_Scheduler] Scheduler error site:[1] ret:[256]
I had to change to this, which seems to make the cron functionality work
* * * * * php /var/www/411/bin/cron.php --backfill --date=/$(date +\%Y-\%m-\%d) --site=1
* * * * * php /var/www/411/bin/worker.php --site=1**
am i completely off, or is this correct?
in Slack.php:62
the CURL 'username' => Util::getSiteName(), attribute seems to cause problems.
I created a standard webhook to a slack-channel, and got a 400 Bad request back.
I Changed it to "webhookbot" and it startet to work.
After a fresh install with the latest update, cannot login too !
except [NONE] FOO\UnauthorizedException: "Invalid nonce specified" at [/var/www/411/phplib/REST.php:101] 0:[FOO\REST->route()
/api/data give the right hostname:8443
Can someone help please ?
Thanks
Hey,
I cannot generate a successful login.
I installed 411 on a host and then:
:/var/www/411$ sudo sqlite3 data.db < db.sql
:/var/www/411$ sudo bin/create_site.php
Creating new site
Site name: test
Hostname: test
From email: [email protected]
From Error email: [email protected]
Default To email: [email protected]
Site created! ID: 1
marius@rkv-c3s-selks2:/var/www/411$ sudo bin/create_user.php
Creating new user
Username: user
Real name: User User
Password:
Email: [email protected]
Admin (y/n): n
User created! ID: 1
So far, so good. Now I go to the site and voila:
I have the distinct feeling that the hostname needs to be correct. Is that so?
I checked out the DB:
sudo sqlite3 data.db
SQLite version 3.8.7.1 2014-10-29 13:59:56
sqlite> .tables
alert_logs groups report_targets search_targets users
alerts jobs reports searches
config lists search_filters sites
group_targets meta search_logs slogs
sqlite> select * from users;
1|1|user|User User|$2y$10$EvfB6.2XkRcuobAEPnNLKuu5IqLx0BH/aQKXkrkRS9uzmaNcGHy9.|[email protected]|0|{}|0|1471875162|1471875162
I keep the hash here since it's obviously just a test in the lab ;). Anyways, I don't get a successful login with this.
411 is using epoch_second pattern which isnt standard available in ES 1.5
I mentioned that alert and alert resolution have different subjects:
[411] xxxx xxxxx marked Resolved (Not an issue) --> Resolution
[411] Zoom --> Alert
Ideally we would have them match as far as search name. So in this example I would see:
[411] Zoom xxxx xxxxx marked Resolved (Not an issue) --> Resolution
[411] Zoom --> Alert
In both cases we have name of the search. This is very important for systems like PagerDuty - I could match [411] Zoom to be able to open/close PagerDuty tickets. Right now it is problematic as any resolution would close any ticket in PagerDuty since we do not have search name in the subject.
Running "requirejs:js" (requirejs) task
Error: ENOENT: no such file or directory, open '/411/htdocs/dist/js/views/searches/search/push.js'
In module tree:
main
app
views/searches/search/load
{ [Error: Error: ENOENT: no such file or directory, open '/411/htdocs/dist/js/views/searches/search/push.js'
In module tree:
main
app
views/searches/search/load
at Error (native)
]
originalError:
{ [Error: ENOENT: no such file or directory, open '/411/htdocs/dist/js/views/searches/search/push.js']
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/411/htdocs/dist/js/views/searches/search/push.js',
fileName: '/411/htdocs/dist/js/views/searches/search/push.js',
moduleTree: [ 'views/searches/search/load', 'app', 'main' ] } }
The command '/bin/sh -c grunt prod' returned a non-zero code: 1
What time does 411 sends daily roll up alert?
Is it possible to have time in CEST instead of GMT Please ?
Thanks for your help.
Hi,
It looks like jobs are not running on my system:
[411]$ /var/www/411/bin/cron.php && /var/www/411/bin/worker.php
[+] Scheduler: 1474926082
[+] Maintenance
[+] Search Health
[+] Rollups
[+] Searches
[+] Reports
[+] Summary
[+] Autoclose
[+] Cleanup
[+] Worker
[+] Worker: 1474926093
[+] Job count: 64
PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 1 near "LIMIT": syntax error in /var/www/411/phplib/DB.php:78
Stack trace:
#0 /var/www/411/phplib/DB.php(78): PDO->prepare('UPDATE jobs
S...')
#1 /var/www/411/phplib/Job.php(167): FOO\DB::query('UPDATE jobs
S...', Array, 0)
#2 /var/www/411/phplib/Worker.php(77): FOO\JobFinder::getAndLock('1', 1474926093)
#3 /var/www/411/bin/worker.php(34): FOO\Worker->processSite(Object(FOO\Site), 1474926093)
#4 {main}
thrown in /var/www/411/phplib/DB.php on line 78
err [411_Worker] Worker error site:[1] ret:[65280]
And I see this in the web UI:
Any ideas on why this is happening?
Thanks!
Andrew
Is this type of query supported by 411?
metrics.server_lag:[0 TO 86400]
It gives me an error. I assume it is either not supported or I need to use different syntax?
Thanks!
Some of the queries that 411 uses are expensive and the results never change. The Dashboard queries are a good example of this. Rather than recomputing this information every time someone loads the page, just generate it at the end of the day.
411's ESQuery syntax isn't 100% compatible with Lucene's query syntax - mostly because we don't currently make use of those features. Opening up this issue to keep track of all these inconsistencies so we can fix them.
List:
NOT
Using the demo site logging in as user / user I was able to just edit my user and grant admin access.
Hi,
I am trying to use the webhook capability but keep getting Target webhook: Remote server returned 0 on a slack webhook
Is it planned to manage users/groups from Active Directory please ?
Thanks for your help.
I keep getting the Apache2 Default Page.
All my configurations has the hostname and even setup the SetEnv as well in the
vi /var/www/411/411.conf
&
bin/create_site.php
please help
I receive the following error when trying to query field names that have a hyphen in them. Have tried a few ways around it i.e. escaping and such and only to be given other errors. Just curious if this is expected behaviour, if there is a way around it, or if we could get a bug fix. Almost all my fields have hyphens, works fine on any fields that don't. sad face.
Query given:
os-cpu-one:>10
Error Thrown:
Error: Catch all: Expected ":" or [a-zA-Z0-9._] but "-" found.
Please provide docker image for testing 411 locally.
Also, docker-compose would be nice.
Right now 411 only allows for, without changes to code outside config, searching a single index index prefix inside a single cluster.
It would be nice be able to configure multiple indexes or clusters then for each search specify against which indexes or clusters it should be executed against.
Within config.php, is it possible to create another section to query a logstash index of a different name?
I tried adding one but it doesn't come up as an option when creating a new Search.
Example:
# Configuration for the logstash index that 411 queries.
'logstash' => [
'hosts' => ['http://192.168.0.11:9200'],
'index_hosts' => [],
'index' => 'logstash-apache',
'date_based' => true,
'date_field' => '@timestamp',
'src_url' => null,
],
# Syslog configuration
'syslog' => [
'hosts' => ['http://192.168.0.11:9200'],
'index_hosts' => [],
'index' => 'syslog',
'date_based' => true,
'date_field' => '@timestamp',
'src_url' => null,
],
];
I am using MySQL and getting this error when I login to website:
except [NONE] PDOException: "SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ' "unixepoch") as date, action, COUNT(*) as count FROM alert_logs INNER J' at line 1" at [/var/www/411/phplib/DB.php:97] 0:[PDOStatement->execute() called at [/var/www/411/phplib/DB.php:97]] 1:[FOO\DB::query() called at [/var/www/411/phplib/REST/Dashboard.php:59]] 2:[FOO\Dashboard_REST->GET() called at [/var/www/411/phplib/REST.php:107]] 3:[FOO\REST->route() called at [/var/www/411/htdocs/api/dashboard.php:6]], referer: http://test.test.com/
Hey, have created a few users with "bin/create_user.php"
unfortunaly keep getting "Invalid credentials" errors on the login screen.
Could not find a way to check the logs or troubleshoot, any tips?
Regards,
AB
How does email notification actually work? I would like to use smtp on my CentOS 6.6 to relay messages to our primary email server. Is this possible or do I need to use Exim?
Thanks!
Andrew
It would be great if alerts page would auto refresh every x seconds/minutes/etc.
I think it is better to rename config.php into config_example.php. Otherwise every time I pull from GitHub my config.php is getting overwritten. Just a suggestion.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.