inquest / omnibus Goto Github PK

• module will retrieve arbitrary HTML, text, or PDF from local or remote sources
• Leverage the InQuest iocextract library to parse out any IOCs and add as new artifacts where possible
  • https://github.com/InQuest/python-iocextract
• Use pyPDF to extract text from PDF docs for extraction

• File artifact will still be identified by its has but can then contain more additional data such as PE features automatically upon creation if file exists on disk
• If file not on disk, user can download it from VT (or other sources I'm working on adding for scanning / downloading files and reports)
• Users can run newly created YARA scan rule against file and store results
• Users can scan files against external services like VT, Hybrid Analysis, Anlyz, etc.
  • Modules for these need to be created

Create module to automatically discover subdomains of a given FQDN

• Module base created
• Added and tested func to get subs from VirusTotal
• Added and tested func to get subs from OTX
• Add code to create child artifacts from discovered subdomains

IPWhois module apparently requires ipaddr but doesn't install it in some cases and this exception was reported by a user. Need to investigate, potentially switch to different package version or method of getting whois info

omnibus >> whois 1.1.1.1

[!] Failed to load module (whois)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 1653, in onecmd_plus_hooks
stop = self.onecmd(statement)
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 1848, in onecmd
stop = func(statement)
File "omnibus-cli.py", line 634, in do_whois
result = self.dispatch.submit(self.session, 'whois', arg)
File "/root/omnibus/lib/dispatch.py", line 131, in submit
result = self.run(module, artifact)
File "/root/omnibus/lib/dispatch.py", line 167, in run
raise err
ImportError: No module named ipaddr
EXCEPTION of type 'ImportError' occurred with message: 'No module named ipaddr'

after running pip install ipaddr it gives the following:

omnibus >> whois 115.23.21.1
/usr/local/lib/python2.7/dist-packages/ipwhois/nir.py:443: UserWarning: NIRWhois._get_nets_krnic() has been deprecated and will be removed. You should now use NIRWhois.get_nets_krnic().
warn('NIRWhois._get_nets_krnic() has been deprecated and will be '
[!] Caught unhandled exception: 'str' object does not support item assignment
{
"asn": "4766"
}

Provide support for users to import & export artifacts and their enriched context via:

Export

• AWS SQS
• HTTP REST API Endpoints
• Local JSON files

Import

• Local text files
• Amazon SQS
• Local JSON files (Omnibus backup files)
• Arbitrary remote URL list file (e.g., blacklist files)
• IOC extraction from arbitrary HTML local file / remote website, local text file, (maybe leverage InQuest ioc_extract here)

Primary Question / Request

Do i need to setup a proxy?

[] Using configuration file (/home/obi/Downloads/omnibus-master/etc/omnibus.conf) ...
[] Debug: False
Welcome to the Omnibus shell! Type "session" to get started or "help" to view all commands.
omnibus >> new inquest.net
EXCEPTION of type 'ServerSelectionTimeoutError' occurred with message: '127.0.0.1:27017: [Errno 111] Connection refused'
To enable full traceback, run the following command: 'set debug true'
omnibus >>

What type of request is there?

• Breaking bug
• [x ] General error (non-fatal)
• Additional module (Create a pull request! :D)
• Use-case suggestions
• Workflow improvments

Please explain the bug in a clear and concise manner

When i startup omnibus i get a error 111.

Steps to Recreate (If Applicable)

Is this preventing you from using the application?

• Yes
• No

If you selected Yes, please explain why fully

Do you have any screenshots, log output, tracebacks, or otherwise to support your requests?

the FQDN artifact regex pattern doesn't properly recognize subdomains. this needs to be addressed so users can run modules against subdomains they add or ones discovered by other modules as children

To get a wider user base and ensure we can support both Python2.7 and Python3.6+, we'll need to make some changes:

• Ensure all dependencies from requirements.txt are Py3.6+ compatible
• Ensure all exceptions are handled correctly
• Handle print as print(foo) versus print foo
• Use fstrings and new format method in places where it makes sense:
• ...
• ... ongoing as I find more requirements

new: info(f'{artifact} tags: {result}')

original: info('%s tags: %s' % (artifact, result))

new: 
build_version = 'v{}.{}-{}_{}'.format(__app_major, rev_count.strip(), rev_hash.strip(), __app_stage)
app_version = 'v{}.{}_{}'.format(__app_major, rev_count.strip(), __app_stage)

old:
build_version = 'v%s.%s-%s_%s' % (__app_major, rev_count.strip(), rev_hash.strip(), __app_stage)
app_version = 'v%s.%s_%s' % (__app_major, rev_count.strip(), __app_stage)

Primary Question / Request

BTC addresses are being recognized and stored as usernames 😢

What type of request is there?

• Breaking bug
• General error (non-fatal)
• Additional module (Create a pull request! :D)
• Use-case suggestions
• Workflow improvements

Please explain the bug in a clear and concise manner

Steps to Recreate (If Applicable)

• start cli
• create new artifact with BTC addr 37KW5474sQmxcnqsc88zGWPL74a1m48iUj

omnibus >> new 37KW5474sQmxcnqsc88zGWPL74a1m48iUj
[~] Created new artifact (37KW5474sQmxcnqsc88zGWPL74a1m48iUj - user)
[~] Opened new session
Artifact ID: 1
omnibus >> blockchain 1
{
    "total_sent": 50000000,
    "total_received": 50000000,
    "final_balance": 0,
    "address": "37KW5474sQmxcnqsc88zGWPL74a1m48iUj",
    "hash160": "3dbf1968b3b1010523d62c818d76b75be81b3701",
    "txs": [
        { ...

Is this preventing you from using the application?

• Yes
• No

If you selected Yes, please explain why

Do you have any screenshots, log output, tracebacks, or otherwise to support your requests?

Creating a constants.py file to contain path to root dir, API keys file, app configuration file, app version info.

Found this snippet from the threatshell project by @tstallings at Salesforce and it's a great way to track versioning if users will be filing tickets. I modified the output a bit for build_version and app_version for our needs.
Users could reference the app_version provided and we'll know exactly what they are on for debugging.

import os
import subprocess

__app_stage = 'beta'
__app_major = '1.0'

__cwd = os.path.abspath(os.path.dirname(__file__))

root_path = os.path.abspath(os.path.join(__cwd, '..'))
api_conf = os.path.join(root_path, 'etc/apikeys.json')
app_conf = os.path.join(root_path, 'etc/omnibus.conf')

rev_hash = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"])
rev_count = subprocess.check_output(["git", "rev-list", "HEAD", "--count"])

build_version = 'v%s.%s-%s_%s' % (__app_major, rev_count.strip(), rev_hash.strip(), __app_stage)
app_version = 'v%s.%s_%s' % (__app_major, rev_count.strip(), __app_stage)

example output from my current local repo:

build: v1.0.125-e73bcf8_beta
app: v1.0.125_beta

Inside omnibus-cli.py and common.py, we'd then just do:

from constants import api_conf
from constants import app_conf
from constants import build_version
from constants import app_version

as needed for printing the current build_version, initializing DB classes with the config files, etc

Example of updated common.py

from constants import api_conf
from constants import app_conf

...

def get_option(section, name):
    config = ConfigParser.ConfigParser()
    if not os.path.exists(app_conf):
        error('configuration file %s does not exist!' % app_conf)
        return None
    config.read(app_conf)
    answer = None
    try:
        answer = config.get(section, name)
    except:
        pass
    return answer


def get_apikey(service):
    """ Read API key config file and return API key by service name """
    if os.path.exists(api_conf):
        api_keys = load_json(api_conf)
        if service in api_keys.keys():
            return api_keys[service]
    else:
        error('cannot find API keys file: %s' % api_conf)

Requires API Key:

• Yes
• No

Tasks

• Write module for IP, Domain, Hash queries
  • Hash will retrieve http traffic
  • IP will retrieve Passive DNS info
  • Domain gets Passive DNS info and Subdomains
• Create child artifacts found by results when applicable
• Test in development branch

• Update all modules so any Exceptions are caught in a way that is OK with Python3.x
  • and so it will past tests
    NO:

    pass

YES:

    logger.error('Caught exception in module {mod}: {err}'.format(mod=module_name, err=err)

• In modules, API keys are returned in Plugin.init. By doing this we have no real way to exit a module w/o raising an exception if an API key is invalid or doesn't exist. Move the API key check to one of the functions the key needs to use so we can log an error message to the user and return from the class gracefully w/o needing to raise an exception

Example:

class Plugin(object):
    def __init__(self, artifact):
        self.artifact = artifact
        self.artifact['data']['censys'] = None
        self.api_key = get_apikey('censys')
        self.headers = {'User-Agent': 'OSINT Omnibus (https://github.com/InQuest/Omnibus)'}

    def run(self):
        if self.api_key == '':
            error('API keys cannot be left blank | set all keys in etc/apikeys.json')
            return

        url = 'https://censys.io/api/v1/view/ipv4{0}'.format(self.artifact['name'])

        try:
            status, response = get(url, auth=(self.api_key['token'], self.api_key['secret']), headers=self.headers)
            if status:
                self.artifact['data']['censys'] = response.json()
        except Exception as err:
            warning('Caught exception in module {0}'.format(err)

def main(artifact):
    plugin = Plugin(artifact)
    plugin.run()
    return plugin.artifact

I see that you have a requirement specifically on cmd2 version 0.8.2 in your requirements.txt file. I recommend that you increase it to version 0.8.9 in order to pick up numerous bugfixes while maintaining full compatibility with the previous version you were depending upon.

Primary Question / Request

What type of request is there?

• [x ] Breaking bug
• General error (non-fatal)
• Additional module (Create a pull request! :D)
• Use-case suggestions
• Workflow improvments

Please explain the bug in a clear and concise manner

On startup omnibus shows several user warnings referring to cmd2.py. Cmd2 seems properly installed.
Program then won't return any output and doesn't terminate. Ctrl+c must be used to return to prompt but help and exit are not even working

Steps to Recreate (If Applicable)

Install and run omnibus then attempt to run any command

Is this preventing you from using the application?

• [ x] Yes
• No

If you selected Yes, please explain why fully

No output is returned on any command

Do you have any screenshots, log output, tracebacks, or otherwise to support your requests?

` ██████╗ ███╗   ███╗███╗   ██╗██╗██████╗ ██╗   ██╗███████╗
██╔═══██╗████╗ ████║████╗  ██║██║██╔══██╗██║   ██║██╔════╝
██║   ██║██╔████╔██║██╔██╗ ██║██║██████╔╝██║   ██║███████╗
██║   ██║██║╚██╔╝██║██║╚██╗██║██║██╔══██╗██║   ██║╚════██║
╚██████╔╝██║ ╚═╝ ██║██║ ╚████║██║██████╔╝╚██████╔╝███████║
 ╚═════╝ ╚═╝     ╚═╝╚═╝  ╚═══╝╚═╝╚═════╝  ╚═════╝ ╚══════╝
             https://github.com/InQuest/omnibus


[*] Using configuration file (/home/lifehacker42/security/omnibus/etc/omnibus.conf) ...
[*] Debug: False
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:3999: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'terminator' on contained expression
  lambda x: x[0].strip())('args') + blankLineTerminator)('statement')
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4006: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'terminator' on contained expression
  ignore=do_not_parse).setParseAction(lambda x: x[0].strip())('suffix') +
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4014: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'command' on contained expression
  pyparsing.Optional(terminator_parser) + after_elements)
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4014: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'args' on contained expression
  pyparsing.Optional(terminator_parser) + after_elements)
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4016: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'terminator' on contained expression
  blankLineTerminationParser = blankLineTerminationParser.setResultsName('statement')
Welcome to the Omnibus shell! Type "session" to get started or "help" to view all commands.
omnibus >>`

Added new section to config file:

[proxy]
# set 'use' to True to enable proxy for all HTTP requests
use = False

# proxy must be a SOCKS5 proxy
# proxy must be in host:port format (do NOT add protocol prefix)
http = 127.0.0.1:9050
https = 127.0.0.1:9050

This is ideally meant to be used with Tor, but doesn't have to be. Just any SOCKS5 proxy that can support http and https as the socks5 proxy.

http.py was re-worked to read this configuration data and use these proxies for all web requests if use = True. If the http/https values are invalid, you will receive an Error and the HTTP request (and the module using it) will fail until you address the config error.

I'm encountering an error when running most commands. Such as in the sample: omnibus>> new inquest.net
ResponseError: unknown command 'SCAN'
EXCEPTION of type 'ResponseError' occurred with message: 'unknown command 'SCAN''

This will go up on rtfd after completion.
I have a almost completed version locally, just need to migrate.

• Requires API key
• Use endpoint to check threat lists of IP artifacts
• Use endpoint "/api/pastries/{IP}"

Case management system

• Allow users to open cases and add artifacts to those cases
• Cases can be switched between and worked on by themselves
• Cases can have notes added, references, etc.
• Allow users to perform bulk analysis tasks against all artifacts within a case

Hi,

I clone this tool in my repo (Kali) and try to test it. Requirements.txt are fully installed, I can create a new session but when I try to create an artifact to investigate on, I have the following message :
EXCEPTION of type 'ConnectionError' occurred with message: 'Error 111 connecting to 127.0.0.1:6379. Connection denied.'

The error seems to come from pymongo. I tried to install mongodb via apt-get install. Install is OK, service starts well but always the same message. Ports are open and redirected, I don't understand where the error come from.

Any help is welcome...

Thank you!!

By default the dnsresolve module keeps the trailing periods on discovered records and digits in front of MX records. These should be stripped so they can be created as new artifacts when found

• Migrate all documentation to readthedocs.io
• Convert Markdown docs to RST/Sphinx doc format
  • not the quickest task but I'm getting there

See comment below

Hi,
I'm currently testing your tool and it seems really powerful, thanks for your work.

When I try to run the Blockchain module, it seems there is an error. I have the following output from Omnibus :
omnibus >> blockchain 2
[!] Unknown command

Artifact 2 is a blockchain adress.

Thanks for your help!

In current master branch, the code re-use in Dispatch.py is insane. This has cut the file size down in over half it's size, the workflow makes much more sense, avoid code re-use, and is much easier to follow overall

I've reworked this so all modules and machines go to a single submit function within Dispatch:

• if no argument is required for module, run the module and return results
• attempt to lookup artifact by session ID if provided, otherwise use artifact by name
• verify artifact exists in DB
• populate list of modules to run against artifact (singular if it isnt a machine, multiple if it is)
• send these modules to execute via self.run against the artifact and return results
  • same as before; gets modular pointer via importlib, runs the module and returns results
• call self.save_results
  • to update any existing documents and create any children documents
• create nice JSON data for command line output and return this to omnibus-cli

Provide support for URLs and expand modules to investigate them, add them as children, etc.

Better error handling within modules. Since they are imported libraries, it's difficult to pinpoint Python exceptions or other errors.

By leveraging cmd2's support for argparse we can get rid of a lot of commands and add argument support to many existing ones. For example, we'll now have support for commands like: tags --view inquest.net, tags --add inquest.net these are my tags, https://inquest.net, artifact --new inquest.net, artifact --view inquest.net, etc etc.

The parsers will go in their own library that we import into omnibus-cli so it doesn't look like garbage inside that script. lets add them to lib/parsers.py

Example of the tags parser:

# do_tags parsere
tags_parser = argparse.ArgumentParser()
id_tags = tags_parser.add_mutually_exclusive_group()
manage_tags = tags_parser.add_mutually_exclusive_group()

manage_tags.add_argument('-a', '--add', action='store_true', help='add tag')
manage_tags.add_argument('-v', '--view', action='store_true', help='view tags')

id_tags.add_argument('-l', '--last', action='store_true', help='use last created artifact')
id_tags.add_argument('-n', '--name', action='store', help='artifact name')
id_tags.add_argument('-i', '--id', action='store', help='artifact session ID')

tags_parser.add_argument('tags', nargs='?', help='comma separated tags')

and we'd port tags to look something like this:

    @cmd2.with_argparser(tags_parser)
    def do_tags(self, args):
        """Manage artifact tags"""
        artifact = self.parse_identifier(args)
        if artifact is None:
            return

        _type = detect_type(artifact)

        if args.view:
            result = self.db.get_value(_type, {'name': artifact}, 'tags')
            info('%s tags: %s' % (artifact, result))
            return

        elif args.add:
            if args.tags == '':
                error('Tags not specified')
            else:
                tags = args.tags

                new_tags = []
                if ',' in tags:
                    for t in tags.split(','):
                        t = t.strip()
                        new_tags.append(t)
                else:
                    new_tags = tags

                if self.db.exists(_type, {'name': artifact}):
                    self.db.update_one(_type, {'name': artifact}, {'tags': new_tags})
                    success('Added tags to artifact (%s: %s)' % (artifact, new_tags))
                else:
                    warning('Failed to find artifact in MongoDB. Run "new <artifact name>" before using the tags command')

        else:
            info('Run "tags --help" or "help tags" for all available commands')