nextcloud / suspicious_login Goto Github PK

View Code? Open in Web Editor NEW

80.0 6.0 25.0 5.69 MB

Detect and warn about suspicious IPs logging into Nextcloud

License: GNU Affero General Public License v3.0

PHP 90.64% JavaScript 6.00% Vue 3.36%

nextcloud-app security intrusion-detection machine-learning deep-learning privacy

suspicious_login's Introduction

🔮 Nextcloud Suspicious Login Detection

Detect and warn about suspicious IPs logging into Nextcloud

The app is still under development, so it’s time for you to get involved! 👩‍💻

Installation

Nextcloud 25 and newer

The app is shipped and comes with the installation of Nextcloud Server. No additional steps are necessary.

Nextcloud 24 and older

Old versions of this app remain available through the app store. They can be installed through Nextcloud's app management UI.

Note

Newer versions of the app are not included in the app store since it is now a shipped app.

Releases and CHANGELOGs

As a shipped app:

changes are posted within the Nextcloud Server changelog.
releases are not posted in this GitHub repository, but they are tagged for code perusal.
it is automatically kept up-to-date with each Nextcloud Server release.

How it works

Data collection

Once this app is enabled, it will automatically start tracking (IP, uid) tuples from successful logins on the instance and feed them into the login_address table. This insert operation is executed for the majority of requests (client authenticate on almost all requests) and therefore has to be fast. In a background job, these rows will be transformed into an aggregated format that is suitable for the training of the neural net. The (IP, uid) tuple becomes (IP, uid, first_seen, last_seen, seen) so that we know which (IP, uid) tuple has been seen first and last. The aggregated data is a compressed format of the raw data. The original data gets deleted and thus the database does not need much space for the collected login data.

Neural net

When enough data is collected – which by default is 60 days (!) – a first training run can be started.

The app registers a background job that invokes the training once a day. As long as there isn't sufficient data, no trained model is generated.

Manual training

The training can also be invoked via the OCC command line tool:

php -f occ suspiciouslogin:train

This command uses several sensible default that should work for instances of any size. The --stats flag is useful to see the measured performance of the trained model after the training finishes. The duration of the training run depends on the size of the input training set, but is usually between two to 15 minutes.

The full list of parameters, their description and default values can be seen with

php -f occ suspiciouslogin:train --help

Hyper parameter optimization (optional)

To find the best possible parameters for the training it's possible to start a hyper parameter optimization run via the CLI:

php -f occ suspiciouslogin:optimize

This command uses the heuristic simulated annealing algorithm to find optimal parameter sets in the multidimensional parameter space. By default this will do 100 steps consisting of five training runs per step, hence this command might take a few days to execute on large instances. On smaller ones it will also take a few hours.

Login classification

As soon as the first model is trained, the app will start classifying (IP, uid) tuples on login. In contrast to the data collection it won't consider requests authenticated via an app password as suspicious. Should it detect a password login where the (IP, uid) is classified as suspicious by the trained model, it will add an entry to the suspicious_login table, including the timestamp, request id and URL.

Development setup

☁ Clone the app into the apps folder of your Nextcloud: git clone https://github.com/nextcloud/suspicious_login.git
💻 Run npm i or krankerl up to install the dependencies
🏗 To build the Javascript whenever you make changes, run npm run dev
☁ Enable the app through the app management of your Nextcloud or run krankerl enable
👍 Partytime! Help fix some issues and review pull requests

suspicious_login's People

Contributors

Stargazers

Watchers

suspicious_login's Issues

Validate the model with realistic and strict data sets

In (machine learning) theory, there should be no intersection between the training data set and the validation training set. However, in this application the classifier does not only have to classify IPs it hasn't seen before, but also ones it sees again. Therefore I chose to not only validate on data that has only been seen recently but also IPs there have historic and recent appearance.

These results might not be a real representation of the quality of the model, so I'd like to add a second validation that only uses true recent data.

postgres errors during cronjob

I noticed in my nextcloud log that an error is happening during the cronjob. It is happening because at some point the query INSERT INTO "oc_login_address_aggregated" ("uid", "ip", "seen", "first_seen", "last_seen") VALUES($1, $2, $3, $4, $5) tries to insert a duplicate value for (uid, ip) which violates a unique key defined on that table.
Am running nextcloud 16.0.1 on a linux server with nginx and postgres 10.8.

Error	core	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'SELECT "seen", "first_seen", "last_seen" FROM "oc_login_address_aggregated" WHERE ("uid" = ?) AND ("ip" = ?)' with params ["***redacted***", "***redacted***"]: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR: current transaction is aborted, commands ignored until end of transaction block

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 184:

    Doctrine\DBAL\Driver\AbstractPostgreSQLDriver->convertException("An exceptio ... k", Doctrine\DBA ... ]})

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 158:

    Doctrine\DBAL\DBALException::wrapException(Doctrine\DBA ... {}, Doctrine\DBA ... ]}, "An exceptio ... k")

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 938:

    Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Doctrine\DBA ... {}, Doctrine\DBA ... ]}, "SELECT \"se ... )", { 1: "***redacted*** ... "})

    /var/www/nextcloud/lib/private/DB/Connection.php - line 195:

    Doctrine\DBAL\Connection->executeQuery("SELECT \"se ... )", [ "***redacted*** ... "], [ 2,2], null)

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php - line 219:

    OC\DB\Connection->executeQuery("SELECT \"se ... )", { uid: "***redacted*** ... "}, [])

    /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php - line 214:

    Doctrine\DBAL\Query\QueryBuilder->execute()

    /var/www/nextcloud/apps/suspicious_login/lib/Service/ETLService.php - line 134:

    OC\DB\QueryBuilder\QueryBuilder->execute()

    /var/www/nextcloud/apps/suspicious_login/lib/BackgroundJob/ETLJob.php - line 51:

    OCA\SuspiciousLogin\Service\ETLService->extractAndTransform()

    /var/www/nextcloud/lib/public/BackgroundJob/Job.php - line 77:

    OCA\SuspiciousLogin\BackgroundJob\ETLJob->run(null)

    /var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php - line 59:

    OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})

    /var/www/nextcloud/cron.php - line 123:

    OCP\BackgroundJob\TimedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

Cased by Doctrine\DBAL\Driver\PDOException: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR: current transaction is aborted, commands ignored until end of transaction block

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 930:

    Doctrine\DBAL\Driver\PDOStatement->execute()

    /var/www/nextcloud/lib/private/DB/Connection.php - line 195:

    Doctrine\DBAL\Connection->executeQuery("SELECT \"se ... )", [ "***redacted*** ... "], [ 2,2], null)

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php - line 219:

    OC\DB\Connection->executeQuery("SELECT \"se ... )", { uid: "***redacted*** ... "}, [])

    /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php - line 214:

    Doctrine\DBAL\Query\QueryBuilder->execute()

    /var/www/nextcloud/apps/suspicious_login/lib/Service/ETLService.php - line 134:

    OC\DB\QueryBuilder\QueryBuilder->execute()

    /var/www/nextcloud/apps/suspicious_login/lib/BackgroundJob/ETLJob.php - line 51:

    OCA\SuspiciousLogin\Service\ETLService->extractAndTransform()

    /var/www/nextcloud/lib/public/BackgroundJob/Job.php - line 77:

    OCA\SuspiciousLogin\BackgroundJob\ETLJob->run(null)

    /var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php - line 59:

    OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})

    /var/www/nextcloud/cron.php - line 123:

    OCP\BackgroundJob\TimedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

Cased by PDOException: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR: current transaction is aborted, commands ignored until end of transaction block

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php - line 141:

    PDOStatement->execute(null)

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 930:

    Doctrine\DBAL\Driver\PDOStatement->execute()

    /var/www/nextcloud/lib/private/DB/Connection.php - line 195:

    Doctrine\DBAL\Connection->executeQuery("SELECT \"se ... )", [ "***redacted*** ... "], [ 2,2], null)

    /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php - line 219:

    OC\DB\Connection->executeQuery("SELECT \"se ... )", { uid: "***redacted*** ... "}, [])

    /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php - line 214:

    Doctrine\DBAL\Query\QueryBuilder->execute()

    /var/www/nextcloud/apps/suspicious_login/lib/Service/ETLService.php - line 134:

    OC\DB\QueryBuilder\QueryBuilder->execute()

    /var/www/nextcloud/apps/suspicious_login/lib/BackgroundJob/ETLJob.php - line 51:

    OCA\SuspiciousLogin\Service\ETLService->extractAndTransform()

    /var/www/nextcloud/lib/public/BackgroundJob/Job.php - line 77:

    OCA\SuspiciousLogin\BackgroundJob\ETLJob->run(null)

    /var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php - line 59:

    OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})

    /var/www/nextcloud/cron.php - line 123:

    OCP\BackgroundJob\TimedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

How reset suspicious login data/model ?

Hello,

On my Nextcloud instance 'trusted_proxies' and 'forwarded_for_headers' wasn't properly configured, so the IP recorded by suspicious login was always the same, the local address of my reverse proxy, and suspicious login have been trained with these wrong data for a while...

Since I fixed the issue, most of login on the instance are detected as suspicious.

Is there a way to reset the model and ignore all the data before a certain date ? Or more simply reset the application ?

Thanks in advance for your support.

occ suspiciouslogin:optimize --stats causes an exception to be thrown

Nextcloud version: 17.0.1
Suspicious login app version: 2.4.1

The app itself seems to works as expected, but the execution of the following command causes an exception to be thrown:

./occ suspiciouslogin:optimize --stats
An unhandled exception has been thrown:
TypeError: Argument 2 passed to OCA\SuspiciousLogin\Service\MLP\Optimizer::optimize() must be an instance of OCA\SuspiciousLogin\Service\IClassificationStrategy, null given, called in NEXTCLOUD_DOC_ROOT/apps2/suspicious_login/lib/Command/Optimize.php on line 56 and defined in NEXTCLOUD_DOC_ROOT/apps2/suspicious_login/lib/Service/MLP/Optimizer.php:151
Stack trace:
#0 NEXTCLOUD_DOC_ROOT/apps2/suspicious_login/lib/Command/Optimize.php(56): OCA\SuspiciousLogin\Service\MLP\Optimizer->optimize(100, NULL, Object(Symfony\Component\Console\Output\ConsoleOutput))
#1 NEXTCLOUD_DOC_ROOT/3rdparty/symfony/console/Command/Command.php(255): OCA\SuspiciousLogin\Command\Optimize->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 NEXTCLOUD_DOC_ROOT/3rdparty/symfony/console/Application.php(908): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 NEXTCLOUD_DOC_ROOT/3rdparty/symfony/console/Application.php(269): Symfony\Component\Console\Application->doRunCommand(Object(OCA\SuspiciousLogin\Command\Optimize), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#4 NEXTCLOUD_DOC_ROOT/3rdparty/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#5 NEXTCLOUD_DOC_ROOT/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 NEXTCLOUD_DOC_ROOT/console.php(97): OC\Console\Application->run()
#7 NEXTCLOUD_DOC_ROOT/occ(11): require_once('NEXTCLOUD_DOC_ROOT...')

Degraded performance after update from 17.0.0.8 to 17.0.0.9

I just noticed that the model performance got a lot worse with one of the recent updates on my personal production server. Going through the logs tells me the app version was always the same, but the server version changed the same day as the results get worse.

Add a cleanup job for old data

Data sets that are outdated (older than the max age, possible with a bit of padding to allow parameter changes) should be deleted to keep the db small. This can be done in a simple background job.

enhancement: right now the IP in "Neuer Anmeldeort erkannt" info is almost useless

He!

Form time to time i get the "Neuer Anmeldeort erkannt" information with the IP of the device that wanted to login, but for, ruff guess, 98% of all intenet users doesnt even know what an IP is. So the info is almost useless, when you dont know how to handle that info.

but what most people know, most of the time, is the network they belong to right now or their several devices at home, at work, etc.

so what about when having an suspicious login, we know the IP and do a quick whois on that target IP, grep the netname, organisation, org-name and maybe the address of the company and put it into the information email.

so users can better decide "is one of my devices right now logged into this network? and do i have to change my pw?"

regards
sash

Nextcloud 21 compatibility

As this is an important security feature, it really needs to be made available for NC21, too.

Compatibility PHP 7.1

I can't install plugin on my nextcloud, my version php installed is 7.1.26

Send an email

Notifications can be dismissed. This app should also send an email to the user if possible.

Could not deserialize persisted model

I have a lot of following errors on NC protocol:

"Could not deserialize persisted model 254: Missing class definition for unserialized object."

Here are the raw data:
{"reqId":"tDWLs8oLa6wApE1uFe9r","level":4,"time":"2021-04-12T10:19:43+00:00","remoteAddr":"192.168.1.101","user":"sync","app":"suspicious_login","method":"REPORT","url":"/remote.php/dav/addressbooks/users/sync/gemeinsam/","message":"Could not deserialize persisted model 254: Missing class definition for unserialized object.","userAgent":"macOS/11.2.3 (20D91) AddressBookCore/2452.4","version":"21.0.1.1","id":"60741ec2b4442"}

Steps to reproduce

Expected behaviour

Contact sync is ok. I have very much denied http request regarding to contact/calendar sync. I dont know, if this have something to do with each other.

Actual behaviour

This error in nc protocol

Server configuration

**Operating system: linuxserver/nextcloud docker image

**Web server: nginx (linuxserver/nextcloud docker image)

**Database: MariaDB (linuxserver/nextcloud docker image)

**PHP version: 7.4.15

**Nextcloud version: 21.0.1

**Updated from an older Nextcloud/ownCloud or fresh install: Updated from 20.xx

**Where did you install Nextcloud from: docker image

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.7.0
  - activity: 2.14.3
  - admin_audit: 1.11.0
  - bruteforcesettings: 2.1.0
  - calendar: 2.2.0
  - cloud_federation_api: 1.4.0
  - contacts: 3.5.1
  - contactsinteraction: 1.2.0
  - dashboard: 7.1.0
  - dav: 1.17.1
  - federatedfilesharing: 1.11.0
  - files: 1.16.0
  - files_pdfviewer: 2.1.0
  - files_rightclick: 1.0.0
  - files_sharing: 1.13.1
  - files_trackdownloads: 1.10.0
  - files_trashbin: 1.11.0
  - files_versions: 1.14.0
  - files_videoplayer: 1.10.0
  - firstrunwizard: 2.10.0
  - forms: 2.2.4
  - logreader: 2.6.0
  - lookup_server_connector: 1.9.0
  - mail: 1.9.5
  - nextcloud_announcements: 1.10.0
  - notifications: 2.9.0
  - oauth2: 1.9.0
  - password_policy: 1.11.0
  - photos: 1.3.0
  - previewgenerator: 3.1.1
  - privacy: 1.5.0
  - provisioning_api: 1.11.0
  - recommendations: 1.0.0
  - serverinfo: 1.11.0
  - settings: 1.3.0
  - support: 1.4.0
  - suspicious_login: 4.0.0
  - systemtags: 1.11.0
  - tasks: 0.13.6
  - text: 3.2.0
  - theming: 1.12.0
  - twofactor_backupcodes: 1.10.0
  - twofactor_totp: 6.0.0
  - updatenotification: 1.11.0
  - user_status: 1.1.1
  - viewer: 1.5.0
  - weather_status: 1.1.0
  - workflowengine: 2.3.0
Disabled:
  - comments
  - encryption
  - federation
  - files_external
  - sharebymail
  - spreed
  - survey_client
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "local_name:9443",
            "my_domain.me"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "21.0.1.1",
        "overwrite.cli.url": "https:\/\/local_name:9443",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "twofactor_enforced": "false",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "theme": "",
        "loglevel": 0,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpsecure": "tls",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "calendar",
            "bruteforcesettings"
        ],
        "default_phone_region": "DE"
    }
}

**Are you using external storage, if yes which one: no

**Are you using encryption: no

**Are you using an external user-backend, if yes which one: no

Client configuration

**Browser: Brave

**Operating system: MacOS

Logs

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"9bA52GmunmHWrJpsINnX","level":4,"time":"2021-04-12T10:31:17+00:00","remoteAddr":"192.168.1.101","user":"my_second_user","app":"suspicious_login","method":"REPORT","url":"/remote.php/dav/addressbooks/users/my_second_user/gemeinsam/","message":"Could not deserialize persisted model 254: Missing class definition for unserialized object.","userAgent":"macOS/11.2.3 (20D91) AddressBookCore/2452.4","version":"21.0.1.1"}
{"reqId":"9bA52GmunmHWrJpsINnX","level":2,"time":"2021-04-12T10:31:17+00:00","remoteAddr":"192.168.1.101","user":"my_second_user","app":"suspicious_login","method":"REPORT","url":"/remote.php/dav/addressbooks/users/my_second_user/gemeinsam/","message":"Could not predict suspiciousness: Could not load model  to classify UID sync and IP 192.168.1.101: Missing class definition for unserialized object.","userAgent":"macOS/11.2.3 (20D91) AddressBookCore/2452.4","version":"21.0.1.1"}
{"reqId":"bmpsLJ0UpDOFP1Rq8qKM","level":0,"time":"2021-04-12T10:31:20+00:00","remoteAddr":"192.168.1.101","user":"my_user","app":"suspicious_login","method":"PROPFIND","url":"/remote.php/dav/files/my_user/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.1.3git (build 4850) (Nextcloud)","version":"21.0.1.1"}
{"reqId":"bmpsLJ0UpDOFP1Rq8qKM","level":0,"time":"2021-04-12T10:31:20+00:00","remoteAddr":"192.168.1.101","user":"my_user","app":"contacts","method":"PROPFIND","url":"/remote.php/dav/files/my_user/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.1.3git (build 4850) (Nextcloud)","version":"21.0.1.1"}
{"reqId":"bmpsLJ0UpDOFP1Rq8qKM","level":0,"time":"2021-04-12T10:31:20+00:00","remoteAddr":"192.168.1.101","user":"my_user","app":"files_sharing","method":"PROPFIND","url":"/remote.php/dav/files/my_user/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.1.3git (build 4850) (Nextcloud)","version":"21.0.1.1"}

Add a --dry-run option to the train command

Sometimes it's interesting to experiment with new parameter sets. In that case I don't want the trained model to be used for login classification.

Lots of warnings for IPv6 addresses on local subnet

I installed version 1.0.0 on nextcloud 16 recently and I get a lot warnings all from the same IPv6 subnet, which happens to the local subnet the server is in:

fd5b:db8:beef:0::/64

This is rather annoying, because clients change their "interface identifier" part of their address frequently for privacy reasons.

Is there an option for whitelisting IP ranges (although this might have security implications if attackers get access to that net)?

Warning suspicious_login Could not predict suspiciousness: No models found

Hi,

I get this error "Warning | suspicious_login | Could not predict suspiciousness: No models found" a lot.

How to fix?

NC 17.0.1

Background job is executed but does not aggregate any data

I just noticed that oc_jobs says the job is being executed but the last duration was 1s (!) and oc_login_address contained 22k entries from a full week. This should not happen.

Show feedback for admins

As noted by @MorrisJobke at #31 (comment), it would be great to show a bit of feedback to the admin.

This can include

Is sufficient data available?
How much (aggregated) data is available?
When was the last training?
How good is the latest/current model?
Were there any alerts today/this week?

missing description what happens if a suspicious login is detected

Hi!
is a suspicious login blocked or does it send a warning or what ?

Can't enable the app

Hi Christoph,

I've got the same problem as described in #415.
I tried to install the app on NC 21 with a version below 4.0 and ran into an error (Error: There is no table with name 'xxx.oc_suspicious_login_model' in the schema.)

After waiting for version 4.0 and trying to remove and/or reinstall now, it throws the same error.
No difference between the browser-based installation or the cli-way with occ.

Do you have an idea how to fix this?

Kidn regards
Michael

Error: Undefined index: settings at /var/www/html/lib/private/AppFramework/Bootstrap/RegistrationContext.php#404
/var/www/html/lib/private/AppFramework/Bootstrap/RegistrationContext.php - line 404:

OC\Log\ErrorHandler::onError(8, "Undefined index: settings", "/var/www/ht ... p", 404, { apps: [],r ... "})

/var/www/html/lib/private/AppFramework/Bootstrap/Coordinator.php - line 143:

OC\AppFramework\Bootstrap\RegistrationContext->delegateContainerRegistrations([])

/var/www/html/lib/private/AppFramework/Bootstrap/Coordinator.php - line 86:

OC\AppFramework\Bootstrap\Coordinator->registerApps([ "suspicious_login"])

/var/www/html/lib/private/Installer.php - line 146:

OC\AppFramework\Bootstrap\Coordinator->runLazyRegistration("suspicious_login")

/var/www/html/core/Command/App/Install.php - line 68:

OC\Installer->installApp("suspicious_login")

/var/www/html/3rdparty/symfony/console/Command/Command.php - line 255:

OC\Core\Command\App\Install->execute(Symfony\Comp ... {}, Symfony\Comp ... {})

/var/www/html/3rdparty/symfony/console/Application.php - line 1009:

Symfony\Component\Console\Command\Command->run(Symfony\Comp ... {}, Symfony\Comp ... {})

/var/www/html/3rdparty/symfony/console/Application.php - line 273:

Symfony\Component\Console\Application->doRunCommand(OC\Core\Command\App\Install {}, Symfony\Comp ... {}, Symfony\Comp ... {})

/var/www/html/3rdparty/symfony/console/Application.php - line 149:

Symfony\Component\Console\Application->doRun(Symfony\Comp ... {}, Symfony\Comp ... {})

/var/www/html/lib/private/Console/Application.php - line 215:

Symfony\Component\Console\Application->run(Symfony\Comp ... {}, Symfony\Comp ... {})

/var/www/html/console.php - line 100:

OC\Console\Application->run()

/var/www/html/occ - line 11:

require_once("/var/www/html/console.php")

App raises fatal error about missing model on recent activations

On new installs, the app will raises a fatal error about it missing a model:

{"reqId":"ksqdCUucdhnLY2UHkYM1","level":4,"time":"2019-05-17T15:16:15+00:00","remoteAddr":"10.8.10.6","user":"admin","app":"suspicious_login","method":"POST","url":"\/index.php\/login?redirect_url=\/index.php\/settings\/admin\/logging","message":"No models found. Can't load latest","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko\/20100101 Firefox\/66.0","version":"16.0.1.1"}
{"reqId":"ksqdCUucdhnLY2UHkYM1","level":2,"time":"2019-05-17T15:16:15+00:00","remoteAddr":"10.8.10.6","user":"admin","app":"suspicious_login","method":"POST","url":"\/index.php\/login?redirect_url=\/index.php\/settings\/admin\/logging","message":"Could not predict suspiciousness: No models found","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko\/20100101 Firefox\/66.0","version":"16.0.1.1"}

This is obviously expected since there has not been enough time to amass data. Would it not make sense to adapt the severity of the message based on the presence of "young" training data ? Eg. something like:

detect the presence of data and its age
if no data or data is < 60 days old
=> warn
else (if no model and data is >= 60 days old)
=> assume that a model should have been trained and raise a (fatal) error.

This would be consistent when the "severity" of the message if the admin console:

No classifier model has been trained yet. This most likely means that you just enabled the app recently. Because the training of a model requires good data, the app waits until logins of at least 60 days have been captured.

Time of login ?

Is there a way to view the table of the suspicious login IP + Date + Time ?

reports invalid IP address

I installed the app today. I just got a warning about a suspicious login from 2003:d3:c719:5c00.95f2:6fbc:e6f:9da4, which doesn't seem to be a valid ipv6 address

Insufficient data: No recent data available

I did install the plugin on 2 sites. One did work as expected, but the second one is stuck.

the plugin was installed about 6 months ago (should be long enough)
Training data statistics: So far the app has captured 30013533 logins (including client connections), of which 37 are distinct (IP, UID) tuples.
php -f occ suspiciouslogin:train => Not enough data, try again later (Insufficient data: No recent data available)

Any help how to get out of this would be welcome. Any file or db table to delete ?

Clarify use case of app & recall/precision; ML model seems bad

Hi,
I have had this app installed for a while now, but cannot really figure out what it is supposed to be used for, for multiple reasons.

as far as I can tell, the app

i) collects (ip, uid) tuples of sucessful logins and trains a ML model from that
ii) checks if the login is classified as suspicious
iii) if yes, sends a notification to the account in question

However, really, what is the point of (iii)?
It does not block logins. And in fact, once an attacker gets into the account, they can just delete that notification. Which will then also disappear on all other devices. The only useful thing about it would be if 'suspicious'-classified logins could not delete any notifications from suspicious_login or could not login at all without 2FA or something.

2nd, about point ii). In my case it seems to flag basically EVERY* log-in with a new IP as suspicious, even if it comes from an IP address very close (numerically) to previous IP addresses. If I want it to do that, there is no need for any sort machine learning. It seems the model may not be 100% fit for purpose. I assume it also uses neither geographical information nor internet topology information, which would be very useful for a classification model pipeline (i.e. turn (ip, uid) into (ip, uid, lat, long) for training with geographic info), and would drastically reduce the amount of data required. Is there a command with which I can test the classifier on an (ip, uid) tuple of my choosing?

3rd, again about point ii). The security page lists two stats, first, recall ('fraction of all suspicious logins captured', TP/(TP+FN), precision ('fraction of logins classified as suspicious that are indeed suspicious', TP/(TP+FP). How can you make these stats? I would wager that 100% of the successful logins on my instance are legitmate so there are zero "actually suspicious" logins, so the real precision is zero. According to the repository description you are also not using failed logins. So where do you get authoritative/"supervised" information from? All you really know is (TP+FP) and (TN+FN).
So, what exactly do TP, FP, FN mean here? My best guess:

Assume G is the full address space, M: (ip,uid) -> {0, 1} is your classifier model.
My best guess is that you 1) assume all captured (successful) logins L are legitimate, and that 2) all** G \ L are suspicious, of which you pick a random set. You then run M(t) on the non-training part of set L (and get FP & TN), and run M(t) on a random sample of G \ L (and get TP & FN).

Again, is there a command with which I can test just the trained classifier on an (ip, uid) tuple of my choosing?

*"So far the app has captured 555049 logins (including client connections), of which 829 are distinct (IP, UID) tuples."

**or a certain subset with minimum distance to L according to some metric?

AddressBookCore - Missing class definition for unserialized object.

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

I have a lot of following errors on NC protocol:
"Could not deserialize persisted model 254: Missing class definition for unserialized object."

Steps to reproduce

Expected behaviour

Contact sync is ok. I have very much denied http request regarding to contact/calendar sync. I dont know, if this have something to do with each other.

Actual behaviour

This error in nc protocol

Server configuration

**Operating system: linuxserver/nextcloud docker image

**Web server: nginx (linuxserver/nextcloud docker image)

**Database: MariaDB (linuxserver/nextcloud docker image)

**PHP version: 7.4.15

**Nextcloud version: 21.0.1

**Updated from an older Nextcloud/ownCloud or fresh install: Updated from 20.xx

**Where did you install Nextcloud from: docker image

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.7.0
  - activity: 2.14.3
  - admin_audit: 1.11.0
  - bruteforcesettings: 2.1.0
  - calendar: 2.2.0
  - cloud_federation_api: 1.4.0
  - contacts: 3.5.1
  - contactsinteraction: 1.2.0
  - dashboard: 7.1.0
  - dav: 1.17.1
  - federatedfilesharing: 1.11.0
  - files: 1.16.0
  - files_pdfviewer: 2.1.0
  - files_rightclick: 1.0.0
  - files_sharing: 1.13.1
  - files_trackdownloads: 1.10.0
  - files_trashbin: 1.11.0
  - files_versions: 1.14.0
  - files_videoplayer: 1.10.0
  - firstrunwizard: 2.10.0
  - forms: 2.2.4
  - logreader: 2.6.0
  - lookup_server_connector: 1.9.0
  - mail: 1.9.5
  - nextcloud_announcements: 1.10.0
  - notifications: 2.9.0
  - oauth2: 1.9.0
  - password_policy: 1.11.0
  - photos: 1.3.0
  - previewgenerator: 3.1.1
  - privacy: 1.5.0
  - provisioning_api: 1.11.0
  - recommendations: 1.0.0
  - serverinfo: 1.11.0
  - settings: 1.3.0
  - support: 1.4.0
  - suspicious_login: 4.0.0
  - systemtags: 1.11.0
  - tasks: 0.13.6
  - text: 3.2.0
  - theming: 1.12.0
  - twofactor_backupcodes: 1.10.0
  - twofactor_totp: 6.0.0
  - updatenotification: 1.11.0
  - user_status: 1.1.1
  - viewer: 1.5.0
  - weather_status: 1.1.0
  - workflowengine: 2.3.0
Disabled:
  - comments
  - encryption
  - federation
  - files_external
  - sharebymail
  - spreed
  - survey_client
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "local_name:9443",
            "my_domain.me"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "21.0.1.1",
        "overwrite.cli.url": "https:\/\/local_name:9443",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "twofactor_enforced": "false",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "theme": "",
        "loglevel": 0,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpsecure": "tls",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "calendar",
            "bruteforcesettings"
        ],
        "default_phone_region": "DE"
    }
}

**Are you using external storage, if yes which one: no

**Are you using encryption: no

**Are you using an external user-backend, if yes which one: no

Client configuration

**Browser: Brave

**Operating system: MacOS

Logs

Web server error log

2021/04/12 12:13:31 [error] 3163#3163: *200267 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.101, server: my_domain.me, request: "PROPFIND /remote.php/dav/files/my_user/ HTTP/1.1", upstream: "https
://192.168.1.3:9443/remote.php/dav/files/ruba/", host: "nc.rbnet.me"
2021/04/12 12:13:35 [error] 3163#3163: *200396 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.101, server: my_domain.me, request: "OPTIONS /remote.php/dav/principals/users/sync/ HTTP/1.1", upstream: "https://192.168.1.3:9443/remote.php/dav/principals/users/my_second_user/", host: "my_domain.me"

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"9bA52GmunmHWrJpsINnX","level":4,"time":"2021-04-12T10:31:17+00:00","remoteAddr":"192.168.1.101","user":"my_second_user","app":"suspicious_login","method":"REPORT","url":"/remote.php/dav/addressbooks/users/my_second_user/gemeinsam/","message":"Could not deserialize persisted model 254: Missing class definition for unserialized object.","userAgent":"macOS/11.2.3 (20D91) AddressBookCore/2452.4","version":"21.0.1.1"}
{"reqId":"9bA52GmunmHWrJpsINnX","level":2,"time":"2021-04-12T10:31:17+00:00","remoteAddr":"192.168.1.101","user":"my_second_user","app":"suspicious_login","method":"REPORT","url":"/remote.php/dav/addressbooks/users/my_second_user/gemeinsam/","message":"Could not predict suspiciousness: Could not load model  to classify UID sync and IP 192.168.1.101: Missing class definition for unserialized object.","userAgent":"macOS/11.2.3 (20D91) AddressBookCore/2452.4","version":"21.0.1.1"}
{"reqId":"bmpsLJ0UpDOFP1Rq8qKM","level":0,"time":"2021-04-12T10:31:20+00:00","remoteAddr":"192.168.1.101","user":"my_user","app":"suspicious_login","method":"PROPFIND","url":"/remote.php/dav/files/my_user/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.1.3git (build 4850) (Nextcloud)","version":"21.0.1.1"}
{"reqId":"bmpsLJ0UpDOFP1Rq8qKM","level":0,"time":"2021-04-12T10:31:20+00:00","remoteAddr":"192.168.1.101","user":"my_user","app":"contacts","method":"PROPFIND","url":"/remote.php/dav/files/my_user/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.1.3git (build 4850) (Nextcloud)","version":"21.0.1.1"}
{"reqId":"bmpsLJ0UpDOFP1Rq8qKM","level":0,"time":"2021-04-12T10:31:20+00:00","remoteAddr":"192.168.1.101","user":"my_user","app":"files_sharing","method":"PROPFIND","url":"/remote.php/dav/files/my_user/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.1.3git (build 4850) (Nextcloud)","version":"21.0.1.1"}

Add the expected date, when the app is going to be active

Hi,

it would be nice if you could add the expected date, when the app is going to be active in the admin settings. (When the 60 days are going to be reached)

This would tell the admin the date when she should have a look on the security settings again.

Suggestion of the message (bold):

Ein Kriterienmodell wurde bis jetzt noch nicht erstellt. Dies bedeutet, dass die Anwendung höchstwahrscheinlich erst kürzlich aktiviert wurde. Um ein Kriterienmodell erstellen zu können, wird eine verlässliche Datenbasis benötigt. Aus diesem Grund wartet die Anwendung, bis Anmeldungen von mindestens 60 Tagen (voraussichtlich am 01.01.2021) erfasst wurden.

Many thanks

PHP Fatal error: causes update from 18.0.7 to 18.0.8 to fail

Nextcloud system becomes unusable:

A occ -h returns:
cf394fd77aca:/var/www/html$ occ -h -sh: occ: not found cf394fd77aca:/var/www/html$ ./occ -h PHP Fatal error: Declaration of Symfony\Component\Console\Input\ArrayInput::hasParameterOption($values, bool $onlyParams = false) must be compatible with Symfony\Component\Console\Input\InputInterface::hasParameterOption($values, $onlyParams = false) in /var/www/html/custom_apps/suspicious_login/vendor/symfony/console/Input/ArrayInput.php on line 202
Data log shows:
{"reqId":"UU9bPOOz4PvChBABRmpU","level":3,"time":"2020-08-31 17:35:21","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Declaration of Symfony\\Component\\Console\\Input\\ArrayInput::hasParamet erOption($values, bool $onlyParams = false) must be compatible with Symfony\\Component\\Console\\Input\\InputInterface::hasParameterOption($values, $onlyParams = false) at /var/www/html/custom_apps/suspicious_login/ven dor/symfony/console/Input/ArrayInput.php#202","userAgent":"--","version":"18.0.8.2"}
Going into the docker-session, one can disable the app with:
cf394fd77aca:/var/www/html$ ./occ app:disable suspicious_login suspicious_login disabled

Add icons for notifications

Currently the notifications do not have any icon. Would simply adding an icon via img/app.svg resolve this @nickvergessen?

The icon we want here is icon-error, that is: https://github.com/nextcloud/server/blob/master/core/img/actions/error.svg

(Otherwise for reference, @ChristophWurst mentioned the place the notifications are generated are at https://github.com/nextcloud/suspicious_login/blob/master/lib/Notifications/Notifier.php#L66-L73 )

what does mean this graphic and where are the configuration option?

I'm sorry but this is unclear to me.
What's recall?
precision is mentionned between '()' so I guess that points to a possible definition. But how is the graphic made exactly? There are apparently no options to configure what's a suspicious activity and what a normal activity.
The x axis is clear, it's date. But what's the y axis? %? if yes then it is crazy that my external instance of this screenshot is classifying everything as suspicious....

Make sure not to send out numerous notifications for the very same IP within a certain time span

As described in #38, there are cases where a user might have a client connected that uses the login password instead of the app password. Most likely, that account has been set up ages ago, because new client version will use app passwords now.

Anyway, we must not spam the user with notifications in the case of a (large) number of requests. One notification per IP and time span (two hours? a day?) is sufficient.

cc @rullzer as discussed

No login possible

Today i can't login with any my device, without changing anything.

I use the plugin since last year now and never get problems. Last weekend i update to latest version and to latest Nextcloud (19)

Edit: using the train command looks like fixed it

"Could not load model 143" warning flooding the logs

I have a similar issue to #158 with model 143. I can't find a model 143 in /appdata_xxxx/suspicious_login/models/143., but there is a 144 and 145.

Using the model in the console:

$ occ suspiciouslogin:predict -v llucax 192.168.10.10 143

In ModelPersistenceService.php line 138:
                                                    
  [OCA\SuspiciousLogin\Exception\ServiceException]  
  Could not load model 143                          
                                                    

Exception trace:
  at /var/www/nextcloud/apps/suspicious_login/lib/Service/ModelPersistenceService.php:138
 OCA\SuspiciousLogin\Service\ModelPersistenceService->load() at /var/www/nextcloud/apps/suspicious_login/lib/Service/EstimatorService.php:63
 OCA\SuspiciousLogin\Service\EstimatorService->predict() at /var/www/nextcloud/apps/suspicious_login/lib/Command/Predict.php:77
 OCA\SuspiciousLogin\Command\Predict->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:1012
 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:272
 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:148
 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:215
 OC\Console\Application->run() at /var/www/nextcloud/console.php:100
 require_once() at /var/www/nextcloud/occ:11

In Root.php line 205:
                                                     
  [OCP\Files\NotFoundException]                      
  /appdata_xxxxx/suspicious_login/models/143  
                                                     

Exception trace:
  at /var/www/nextcloud/lib/private/Files/Node/Root.php:205
 OC\Files\Node\Root->get() at /var/www/nextcloud/lib/private/Files/Node/Folder.php:138
 OC\Files\Node\Folder->get() at /var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFolder.php:76
 OC\Files\SimpleFS\SimpleFolder->getFile() at /var/www/nextcloud/apps/suspicious_login/lib/Service/ModelPersistenceService.php:135
 OCA\SuspiciousLogin\Service\ModelPersistenceService->load() at /var/www/nextcloud/apps/suspicious_login/lib/Service/EstimatorService.php:63
 OCA\SuspiciousLogin\Service\EstimatorService->predict() at /var/www/nextcloud/apps/suspicious_login/lib/Command/Predict.php:77
 OCA\SuspiciousLogin\Command\Predict->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:1012
 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:272
 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:148
 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:215
 OC\Console\Application->run() at /var/www/nextcloud/console.php:100
 require_once() at /var/www/nextcloud/occ:11

suspiciouslogin:predict [--v6] [--] <uid> <ip> [<model>]

(the xxxxxx is the masked ID, not sure if it would be a security or privacy issue if I share it)

I had some out of space issues just a few days ago, and I have the suspicion (pun intended) that it might have happened that the model had to be created in the disk but couldn't because of the lack of space (inodes in reality).

Could this have happened? And if so, would it be OK to remove the model from the DB (from table suspicious_login_model, something else?) or to just copy another model file as a hack? Or the non-existence of that file is normal and I should wait 60 days and that should be all?

In any case is not ideal to have the logs flooded with a warning that is not really a warning if it's expected behavior, so in any case I think there is something that can be improved here.

Can't download: curl timeout

Hello,

I'm trying to download this app, but this is the result:

cURL error 28: Operation timed out after 30000 milliseconds with 452166 out of 1195120 bytes received (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

I think the reason for this is the restricted way the targeted machine is connected to the net. Is there any other way to download this app, maybe as a zip file?

Thanks in advance.

Make Suspicious Login a Featured App

I've been using this app on my nextcloud server and I love it. Personally I think this app should be a "Featured App"
Big plus on the security side of things. Plus a lot of other campiness like Google, Microsoft, Instagram have this enabled by default.

Blocking Next cloud bookmark android app add bookmark API

Hi There,
Suspicious Login App is blocking Bookmark API call from the Android app. Below is the configuration:

android 9 (EMUI9.1.0)
Device Huawei p20.
Nextcloudpi v1.24.0
nextcloud 18.0.6
Runnning in docker container on raspbian lite on Rasberry Pi 4 4GB.

The error is given below:

{"reqId":"Xue@rz@O60kwebLtVEhNAwAAAAw","level":2,"time":"2020-06-15T18:32:16+00:00","remoteAddr":"192.168.1.1","user":"lukebeart","app":"suspicious_login","method":"GET","url":"/index.php/apps/bookmarks/public/rest/v2/bookmark?page=-1","message":"Could not predict suspiciousness: No models found","userAgent":"Dalvik/2.1.0 (Linux; U; Android 9; EML-L09 Build/HUAWEIEML-L09)","version":"18.0.6.0"}

I have checked for other devices its not blocking but some how raspbian lite on Rasberry Pi 4 4GB next cloud instance it is blocking that particular API call. Any reason you can find? App Code: Repo & Android App code: Repo. Create Bookmark API: Docs. I need to close this issue. Any help will be appriciated.
Cheers
Biswajit.

Use shared browserslist config

Use https://www.npmjs.com/package/@nextcloud/browserslist-config

Allow users to opt-out from login alerts

I can imagine that not all users care about these events and might want to disable the notifications for their account.

There is also the case where a user has clients connected that use the login password. In that case the classifier might very often warn about a login, though there was none. For flawless security we can't duck type a connection to being a client because attackers can also just use a Nextcloud client to access the victim's data. Hence the separation of web login and client access does not help here.

The proposed solution is a simple checkbox on the user's personal settings page that default to checked. If unchecked, the app will never send out notifications.

cc @rullzer as discussed

No models found

Steps to reproduce

Run ./occ suspiciouslogin:predict <uid> <ip>

In ModelPersistenceService.php line 91:
                   
  No models found  
                   

In QBMapper.php line 248:
                                                                                                                                                   
  Did expect one result but found none when executing: query "SELECT * FROM `*PREFIX*suspicious_login_model` ORDER BY `created_at` desc LIMIT 1";  
                                                                                                                                                   

suspiciouslogin:predict <uid> <ip> [<model>]

Expected behaviour

No idea, but not this.

Actual behaviour

In ModelPersistenceService.php line 91:
                   
  No models found  
                   

In QBMapper.php line 248:
                                                                                                                                                   
  Did expect one result but found none when executing: query "SELECT * FROM `*PREFIX*suspicious_login_model` ORDER BY `created_at` desc LIMIT 1";  
                                                                                                                                                   

suspiciouslogin:predict <uid> <ip> [<model>]

Server configuration detail

Operating system: Linux 5.0.21-1-pve #1 SMP PVE 5.0.21-2 (Wed, 28 Aug 2019 15:12:18 +0200) x86_64

Webserver: Apache/2.4.38 (Debian) (apache2handler)

Database: mysql 10.3.17

PHP version:

7.3.4-2
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, sodium, standard, apache2handler, pdlib, mysqlnd, PDO, xml, apcu, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, imap, intl, json, exif, msgpack, mysqli, pdo_mysql, pdo_pgsql, pdo_sqlite, pgsql, apc, posix, readline, redis, shmop, SimpleXML, smbclient, sockets, sqlite3, ssh2, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlrpc, xmlwriter, xsl, zip, Phar, memcached, libsmbclient, Zend OPcache

Nextcloud version: 16.0.4 - 16.0.4.1

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status

Array
(
[core] => Array
(
[INVALID_HASH] => Array
(
[core/js/mimetypelist.js] => Array
(
[expected] => dc48de7ad4baa030c5e563350c9a80b274bad783f6f5adbf1595ecef6c6a32e52890a24cb26cddb0aa20193ba52c001150c68d8bfb567f0aed566f4029a190a3
[current] => 6a05a7a2bd52651177dc54170f94c7ccf1966c421f30540be49d64770d71c71731052565c2f8e00406a579e7c78dad16673202051231626f1e6d4e656046e6fc
)

)

List of activated apps

Enabled:
 - accessibility: 1.2.0
 - activity: 2.9.1
 - admin_audit: 1.6.0
 - admin_notifications: 1.0.2
 - announcementcenter: 3.5.1
 - apporder: 0.7.1
 - audioplayer: 2.8.4
 - audioplayer_editor: 0.1.0
 - bookmarks: 2.1.1
 - calendar: 1.7.1
 - camerarawpreviews: 0.7.0
 - carnet: 0.17.0
 - checksum: 0.4.3
 - circles: 0.17.7
 - cloud_federation_api: 0.2.0
 - cms_pico: 0.9.8
 - comments: 1.6.0
 - contacts: 3.1.3
 - cospend: 0.1.0
 - dashboard: 6.0.0
 - data_request: 1.3.0
 - dav: 1.9.2
 - deck: 0.6.6
 - dicomviewer: 1.2.0
 - drawio: 0.9.3
 - drop_account: 0.1.0
 - dropit: 0.2.1
 - encryption: 2.4.0
 - event_update_notification: 0.3.4
 - external: 3.3.0
 - extract: 1.2.0
 - facerecognition: 0.5.7
 - federatedfilesharing: 1.6.0
 - federation: 1.6.0
 - files: 1.11.0
 - files_3d: 0.1.0
 - files_antivirus: 2.1.1
 - files_automatedtagging: 1.6.0
 - files_clipboard: 0.7.1
 - files_downloadactivity: 1.5.0
 - files_ebookreader: 0.0.1
 - files_external: 1.7.0
 - files_linkeditor: 1.0.10
 - files_markdown: 2.1.0
 - files_mindmap: 0.0.15
 - files_pdfviewer: 1.5.0
 - files_readmemd: 1.1.1
 - files_retention: 1.5.1
 - files_rightclick: 0.15.1
 - files_sharing: 1.8.0
 - files_texteditor: 2.8.0
 - files_trackdownloads: 1.5.0
 - files_trashbin: 1.6.0
 - files_versions: 1.9.0
 - files_videoplayer: 1.5.0
 - firstrunwizard: 2.5.0
 - flowupload: 0.1.2
 - forms: 1.0.3
 - gallery: 18.3.0
 - gpxedit: 0.0.11
 - gpxmotion: 0.0.9
 - gpxpod: 4.0.5
 - groupfolders: 4.1.0
 - hsts: 0.3.0
 - issuetemplate: 0.5.0
 - keeweb: 0.5.1
 - logreader: 2.1.0
 - lookup_server_connector: 1.4.0
 - mail: 0.17.0
 - maps: 0.1.2
 - metadata: 0.9.0
 - mindmaps: 0.1.0
 - music: 0.10.1
 - myscan: 0.0.1
 - news: 13.1.6
 - nextcloud_announcements: 1.5.0
 - notes: 3.0.2
 - notifications: 2.4.1
 - oauth2: 1.4.2
 - ocdownloader: 1.7.2
 - ocr: 4.4.6
 - ocsms: 2.1.3
 - ownpad: 0.6.13
 - passman: 2.3.2
 - password_policy: 1.6.0
 - passwords: 2019.9.0
 - phonetrack: 0.5.2
 - piwik: 0.5.3
 - polls: 0.10.2
 - preferred_providers: 1.4.0
 - previewgenerator: 2.1.0
 - printer: 0.0.1
 - privacy: 1.0.0
 - privatedata: 1.0.1
 - provisioning_api: 1.6.0
 - quickaccesssorting: 1.0.1
 - quicknotes: 0.1.9
 - quota_warning: 1.5.0
 - radio: 0.6.5
 - ransomware_protection: 1.4.0
 - recommendations: 0.4.0
 - registration: 0.4.7
 - scanner: 0.1.4
 - serverinfo: 1.6.0
 - sharebymail: 1.6.0
 - sharelisting: 0.2.0
 - sharerenamer: 2.7.1
 - sharingpath: 0.0.1
 - social: 0.2.101
 - socialsharing_diaspora: 1.0.4
 - socialsharing_email: 1.0.5
 - socialsharing_facebook: 1.0.4
 - socialsharing_googleplus: 1.0.4
 - socialsharing_twitter: 1.0.4
 - spreed: 6.0.4
 - survey_client: 1.4.0
 - suspicious_login: 1.0.0
 - systemtags: 1.6.0
 - tasks: 0.11.2
 - telephoneprovider: 1.0.2
 - terms_of_service: 1.2.2
 - text: 1.0.2
 - theming: 1.7.0
 - twofactor_admin: 0.4.1
 - twofactor_backupcodes: 1.5.0
 - twofactor_email: 1.0.0
 - twofactor_gateway: 0.14.1
 - twofactor_nextcloud_notification: 1.1.2
 - twofactor_totp: 3.0.1
 - updatenotification: 1.6.0
 - uploaddetails: 0.1.1
 - user_usage_report: 1.1.2
 - video_converter: 0.1.0
 - viewer: 1.1.0
 - workflow_pdf_converter: 1.1.0
 - workflowengine: 1.6.0
Disabled:
 - end_to_end_encryption
 - files_accesscontrol
 - files_inotify
 - files_snapshots
 - folderplayer
 - githubmergetracker
 - ojsxc
 - ownbackup
 - ransomware_detection
 - sociallogin
 - support
 - theming_customcss
 - twofactor_yubikey
 - user_ldap
 - w2g2
 - weather
 - workflow_script

Configuration (config/config.php)

{
    "log_type": "syslog",
    "logfile": "",
    "loglevel": 0,
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "nc.waimanu.io"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/nc.waimanu.io",
    "overwritehost": "nc.waimanu.io",
    "overwriteprotocol": "https",
    "overwritewebroot": "",
    "dbtype": "mysql",
    "version": "16.0.4.1",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.distributed": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "memcached_servers": [
        [
            "127.0.0.1",
            11211
        ]
    ],
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 6379
    },
    "htaccess.RewriteBase": "\/",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_smtpsecure": "tls",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpauth": 1,
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "25",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "theme": "",
    "mysql.utf8mb4": true,
    "log_rotate_size": 1073741824,
    "logtimezone": "Europe\/Berlin",
    "updater.release.channel": "stable",
    "maintenance": false,
    "trashbin_retention_obligation": "30,auto",
    "versions_retention_obligation": "365,auto",
    "enable_previews": true,
    "enabledPreviewProviders": [
        "OC\\Preview\\PNG",
        "OC\\Preview\\JPEG",
        "OC\\Preview\\GIF",
        "OC\\Preview\\BMP",
        "OC\\Preview\\XBitmap",
        "OC\\Preview\\Movie",
        "OC\\Preview\\PDF",
        "OC\\Preview\\MP3",
        "OC\\Preview\\TXT",
        "OC\\Preview\\MarkDown",
        "OC\\Preview\\Image",
        "OC\\Preview\\Photoshop",
        "OC\\Preview\\TIFF",
        "OC\\Preview\\SVG",
        "OC\\Preview\\OpenDocument"
    ],
    "ldapIgnoreNamingRules": false,
    "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
    "data-fingerprint": "14f9373638db7e4c620b53251289ac68",
    "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
    "filesystem_check_changes": 1,
    "has_rebuilt_cache": true,
    "app_install_overwrite": [
        "files_downloadactivity",
        "admin_notifications",
        "mindmaps",
        "privatedata",
        "files_clipboard",
        "githubmergetracker",
        "folderplayer",
        "documents",
        "defaultgroup",
        "myscan",
        "twofactor_email"
    ]
}

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: false

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

Client configuration

Browser: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0

Operating system: Arch Linux

Logs

Web server error log

2019-09-16T23:48:46.610 819698600530 apache2.service Nextcloud[12700] CRITICAL {suspicious_login} No models found. Can't load latest
2019-09-16T23:48:46.610 819698601274 apache2.service Nextcloud[12700] WARNING {suspicious_login} Could not predict suspiciousness: No models found

Nextcloud log

In ModelPersistenceService.php line 91:
                   
  No models found  
                   

In QBMapper.php line 248:
                                                                                                                                                   
  Did expect one result but found none when executing: query "SELECT * FROM `*PREFIX*suspicious_login_model` ORDER BY `created_at` desc LIMIT 1";  
                                                                                                                                                   

suspiciouslogin:predict <uid> <ip> [<model>]

Browser log

View list of IPs

This may be a dumb question, but, how do you view a list of the IPs listed as suspicious or not suspicious from the app?

Integrate IP geolocation into notifications

For now, my users are rather irritated than helped by the suspicious login notifications. It would be somewhat more useful if the notification cloud integrate geolocation and say something like "we detected a new login from location XXX" than just the raw IP address, or at least provide a link to a geolocation service that resolves the IP address to a location for you.

Use start time stamp for relative time frames

Long running tasks like the hyper parameter optimization query the database many times. If they run for a few hours or days, the time frames shift and therefore might produce distorted results.

thausends over thausends messeseges for suspicios_login

I have a connection from a synologic nas over webdav to the nextcloud server both have a fixed IP (IPv4) I have level 2 and level 4 messages?

What is the way to secure this IP from external or stop the million messages ?

Thanks for a answer
Günther

Use the initial state API to load admin stats

When implementing #37 I deliberately didn't use the Nextcloud 16 initial state API in order to make it possible to run the app on (patched) Nextcloud 15's. However, that API should be used once 16 is a hard requirement.

Same IPs marked as suspicious over several months

Hello,

My logs are getting spammed with warnings because of (legitimate) logins from the same user/ip over the course of several months.
I checked that new models are being trained and also trained them manually with no changes.

From the graphs the classifier believes that it has a recall of 100% and precision of 81%, when that is actually false given that no real suspicious login has happened (yet at least).

How can I ensure that the app recognises legitimate logins over suspicious ones? There are no links in the warning emails asking whether that was a legitimate login or not. How does the app learn which suspicious logins are actually suspicious?

Cache the trained model

For the classification – especially on login – we have to retrieve the trained model from the app data. This is an expensive operation. Since the model is immutable and small (a few kBs), we can just cache it locally and improve the login performance.

Add PHP 8 compatibility

The application is not compatible with PHP 8 and cannot be activated in the store:
PHP with a version lower than 7.4 is required.

Please add PHP 8 compatibility.

Note: The text is misleading. The application works with PHP 7.4 and not only with the lower version 7.3

GDPR Compliance, use hashed IP addresses / user names

Congratulations to another very interesting and sophisticated app in the nextcloud ecosystem! Thank you for your efforts.

But it might be objected that this app is not GDPR compliant at the actual development stage, regarding Art. 5(1)(c).

After what I could see from the database entries, this app saves usernames, IP addresses (both unhashed) and login dates. As long as it is technically possible to work with hashed IP addresses and probably hashed usernames, too, I do not think it is legally compliant in the means of ‘data minimisation’.

Give frequent but static IPs more weight

Expected behavior

This app should classify my (almost) static IP of my office as not suspicious.

Actual behavior

On my test instance, the trained model reaches a very high precision on IPs classifier as not suspicious. But since of all the rows it uses for training only a few have my static IP, it does not seem to optimize for that. While logins from my phone are treated as non-suspicious, logins from the static office IP are.

The training data (aggregated) looked like this:

two rows for the static IP (two distinct users)
~250 rows for the other IPs

-> hence it's not surprising that the neural net will more likely learn the patterns of the other IPs than the static one. I manually compare the IPs and the dynamic ones are indeed "different" to my static one.

Possible solution

I have to read up on this but I think we can give frequently seen IPs more weight so the net develops a bias for those.

Support IPv6

Right now the classifier only support IPv4. For IPv6 we'll probably need a second classifier as the structure (and thus bit vector size) is different.

Everything is flagged as suspicious

Hi,

I'm currently experiencing the issue, that every single connection is flagged as suspicious!
Even:

127.0.0.1
The IP of the initial setup
The Servers own IP

My logfile is literally multiple gigabytes large and almost exclusively full of lines like this:
{"reqId":"o20VWlrQIU1OTT0n90QI","level":2,"time":"2020-08-07T12:52:09+00:00","remoteAddr":"127.0.0.1","user":"server","app":"suspicious_login","method":"MKCOL","url":"/remote.php/dav/files/<redacted>","message":"Detected a login from a suspicious login. user=server ip=127.0.0.1 strategy=ipv4","userAgent":"curl/7.68.0","version":"18.0.7.1"}

The Graph in the security section telling me: "Yay 99% of my detected suspicious logins are actually suspicious" is so next to useless as it is literally telling me: 99% of your users get an unnecessary notification and have to reload the website just to continue doing their job (the amount of "warnings" causes nextcloud to invalidate the current token) ...

Nextcloud Version: 18.0.7
Addon Version: 3.1.0

Greetings

PS: this is not limited to curl! Chrome and firefox have the same issues.

Can't enable the app

When I click Enable I get the following error:

An error occured during the request. Unable to proceed
There is no table with name 'oc_spoody.oc_suspicious_login_model' in the schema

PS: spoody is my username on Nextcloud

nextcloud / suspicious_login Goto Github PK

suspicious_login's Introduction

🔮 Nextcloud Suspicious Login Detection

Installation

Nextcloud 25 and newer

Nextcloud 24 and older

Releases and CHANGELOGs

How it works

Data collection

Neural net

Manual training

Hyper parameter optimization (optional)

Login classification

Development setup

suspicious_login's People

Contributors

Stargazers

Watchers

Forkers

suspicious_login's Issues

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

How to use GitHub

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Suggestion of the message (bold):

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration detail

Client configuration

Logs

Expected behavior

Actual behavior

Possible solution

Recommend Projects

Recommend Topics

Recommend Org