inquest / omnibus Goto Github PK
View Code? Open in Web Editor NEWThe OSINT Omnibus (beta release)
License: MIT License
The OSINT Omnibus (beta release)
License: MIT License
Create module to automatically discover subdomains of a given FQDN
IPWhois module apparently requires ipaddr but doesn't install it in some cases and this exception was reported by a user. Need to investigate, potentially switch to different package version or method of getting whois info
omnibus >> whois 1.1.1.1
[!] Failed to load module (whois)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 1653, in onecmd_plus_hooks
stop = self.onecmd(statement)
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 1848, in onecmd
stop = func(statement)
File "omnibus-cli.py", line 634, in do_whois
result = self.dispatch.submit(self.session, 'whois', arg)
File "/root/omnibus/lib/dispatch.py", line 131, in submit
result = self.run(module, artifact)
File "/root/omnibus/lib/dispatch.py", line 167, in run
raise err
ImportError: No module named ipaddr
EXCEPTION of type 'ImportError' occurred with message: 'No module named ipaddr'
after running pip install ipaddr it gives the following:
omnibus >> whois 115.23.21.1
/usr/local/lib/python2.7/dist-packages/ipwhois/nir.py:443: UserWarning: NIRWhois._get_nets_krnic() has been deprecated and will be removed. You should now use NIRWhois.get_nets_krnic().
warn('NIRWhois._get_nets_krnic() has been deprecated and will be '
[!] Caught unhandled exception: 'str' object does not support item assignment
{
"asn": "4766"
}
Provide support for users to import & export artifacts and their enriched context via:
Export
Import
Do i need to setup a proxy?
[] Using configuration file (/home/obi/Downloads/omnibus-master/etc/omnibus.conf) ...
[] Debug: False
Welcome to the Omnibus shell! Type "session" to get started or "help" to view all commands.
omnibus >> new inquest.net
EXCEPTION of type 'ServerSelectionTimeoutError' occurred with message: '127.0.0.1:27017: [Errno 111] Connection refused'
To enable full traceback, run the following command: 'set debug true'
omnibus >>
When i startup omnibus i get a error 111.
the FQDN artifact regex pattern doesn't properly recognize subdomains. this needs to be addressed so users can run modules against subdomains they add or ones discovered by other modules as children
To get a wider user base and ensure we can support both Python2.7 and Python3.6+, we'll need to make some changes:
print(foo)
versus print foo
new: info(f'{artifact} tags: {result}')
original: info('%s tags: %s' % (artifact, result))
new:
build_version = 'v{}.{}-{}_{}'.format(__app_major, rev_count.strip(), rev_hash.strip(), __app_stage)
app_version = 'v{}.{}_{}'.format(__app_major, rev_count.strip(), __app_stage)
old:
build_version = 'v%s.%s-%s_%s' % (__app_major, rev_count.strip(), rev_hash.strip(), __app_stage)
app_version = 'v%s.%s_%s' % (__app_major, rev_count.strip(), __app_stage)
BTC addresses are being recognized and stored as usernames ๐ข
37KW5474sQmxcnqsc88zGWPL74a1m48iUj
omnibus >> new 37KW5474sQmxcnqsc88zGWPL74a1m48iUj
[~] Created new artifact (37KW5474sQmxcnqsc88zGWPL74a1m48iUj - user)
[~] Opened new session
Artifact ID: 1
omnibus >> blockchain 1
{
"total_sent": 50000000,
"total_received": 50000000,
"final_balance": 0,
"address": "37KW5474sQmxcnqsc88zGWPL74a1m48iUj",
"hash160": "3dbf1968b3b1010523d62c818d76b75be81b3701",
"txs": [
{ ...
Creating a constants.py file to contain path to root dir, API keys file, app configuration file, app version info.
Found this snippet from the threatshell project by @tstallings at Salesforce and it's a great way to track versioning if users will be filing tickets. I modified the output a bit for build_version and app_version for our needs.
Users could reference the app_version provided and we'll know exactly what they are on for debugging.
import os
import subprocess
__app_stage = 'beta'
__app_major = '1.0'
__cwd = os.path.abspath(os.path.dirname(__file__))
root_path = os.path.abspath(os.path.join(__cwd, '..'))
api_conf = os.path.join(root_path, 'etc/apikeys.json')
app_conf = os.path.join(root_path, 'etc/omnibus.conf')
rev_hash = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"])
rev_count = subprocess.check_output(["git", "rev-list", "HEAD", "--count"])
build_version = 'v%s.%s-%s_%s' % (__app_major, rev_count.strip(), rev_hash.strip(), __app_stage)
app_version = 'v%s.%s_%s' % (__app_major, rev_count.strip(), __app_stage)
example output from my current local repo:
build: v1.0.125-e73bcf8_beta
app: v1.0.125_beta
Inside omnibus-cli.py and common.py, we'd then just do:
from constants import api_conf
from constants import app_conf
from constants import build_version
from constants import app_version
as needed for printing the current build_version, initializing DB classes with the config files, etc
Example of updated common.py
from constants import api_conf
from constants import app_conf
...
def get_option(section, name):
config = ConfigParser.ConfigParser()
if not os.path.exists(app_conf):
error('configuration file %s does not exist!' % app_conf)
return None
config.read(app_conf)
answer = None
try:
answer = config.get(section, name)
except:
pass
return answer
def get_apikey(service):
""" Read API key config file and return API key by service name """
if os.path.exists(api_conf):
api_keys = load_json(api_conf)
if service in api_keys.keys():
return api_keys[service]
else:
error('cannot find API keys file: %s' % api_conf)
Requires API Key:
Tasks
pass
YES:
logger.error('Caught exception in module {mod}: {err}'.format(mod=module_name, err=err)
Example:
class Plugin(object):
def __init__(self, artifact):
self.artifact = artifact
self.artifact['data']['censys'] = None
self.api_key = get_apikey('censys')
self.headers = {'User-Agent': 'OSINT Omnibus (https://github.com/InQuest/Omnibus)'}
def run(self):
if self.api_key == '':
error('API keys cannot be left blank | set all keys in etc/apikeys.json')
return
url = 'https://censys.io/api/v1/view/ipv4{0}'.format(self.artifact['name'])
try:
status, response = get(url, auth=(self.api_key['token'], self.api_key['secret']), headers=self.headers)
if status:
self.artifact['data']['censys'] = response.json()
except Exception as err:
warning('Caught exception in module {0}'.format(err)
def main(artifact):
plugin = Plugin(artifact)
plugin.run()
return plugin.artifact
I see that you have a requirement specifically on cmd2
version 0.8.2 in your requirements.txt file. I recommend that you increase it to version 0.8.9 in order to pick up numerous bugfixes while maintaining full compatibility with the previous version you were depending upon.
On startup omnibus shows several user warnings referring to cmd2.py. Cmd2 seems properly installed.
Program then won't return any output and doesn't terminate. Ctrl+c must be used to return to prompt but help and exit are not even working
Install and run omnibus then attempt to run any command
No output is returned on any command
` โโโโโโโ โโโโ โโโโโโโโ โโโโโโโโโโโโโ โโโ โโโโโโโโโโโ
โโโโโโโโโโโโโโ โโโโโโโโโโ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโ
โโโ โโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโ
โโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโ
โโโโโโโโโโโโ โโโ โโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโ โโโ โโโโโโ โโโโโโโโโโโโโโโ โโโโโโโ โโโโโโโโ
https://github.com/InQuest/omnibus
[*] Using configuration file (/home/lifehacker42/security/omnibus/etc/omnibus.conf) ...
[*] Debug: False
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:3999: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'terminator' on contained expression
lambda x: x[0].strip())('args') + blankLineTerminator)('statement')
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4006: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'terminator' on contained expression
ignore=do_not_parse).setParseAction(lambda x: x[0].strip())('suffix') +
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4014: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'command' on contained expression
pyparsing.Optional(terminator_parser) + after_elements)
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4014: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'args' on contained expression
pyparsing.Optional(terminator_parser) + after_elements)
/home/lifehacker42/.local/lib/python2.7/site-packages/cmd2.py:4016: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'statement' on And expression collides with 'terminator' on contained expression
blankLineTerminationParser = blankLineTerminationParser.setResultsName('statement')
Welcome to the Omnibus shell! Type "session" to get started or "help" to view all commands.
omnibus >>`
Added new section to config file:
[proxy]
# set 'use' to True to enable proxy for all HTTP requests
use = False
# proxy must be a SOCKS5 proxy
# proxy must be in host:port format (do NOT add protocol prefix)
http = 127.0.0.1:9050
https = 127.0.0.1:9050
This is ideally meant to be used with Tor, but doesn't have to be. Just any SOCKS5 proxy that can support http and https as the socks5 proxy.
http.py was re-worked to read this configuration data and use these proxies for all web requests if use = True
. If the http/https values are invalid, you will receive an Error and the HTTP request (and the module using it) will fail until you address the config error.
I'm encountering an error when running most commands. Such as in the sample: omnibus>> new inquest.net
ResponseError: unknown command 'SCAN'
EXCEPTION of type 'ResponseError' occurred with message: 'unknown command 'SCAN''
This will go up on rtfd after completion.
I have a almost completed version locally, just need to migrate.
Case management system
Hi,
I clone this tool in my repo (Kali) and try to test it. Requirements.txt are fully installed, I can create a new session but when I try to create an artifact to investigate on, I have the following message :
EXCEPTION of type 'ConnectionError' occurred with message: 'Error 111 connecting to 127.0.0.1:6379. Connection denied.'
The error seems to come from pymongo. I tried to install mongodb via apt-get install. Install is OK, service starts well but always the same message. Ports are open and redirected, I don't understand where the error come from.
Any help is welcome...
Thank you!!
By default the dnsresolve module keeps the trailing periods on discovered records and digits in front of MX records. These should be stripped so they can be created as new artifacts when found
See comment below
Hi,
I'm currently testing your tool and it seems really powerful, thanks for your work.
When I try to run the Blockchain module, it seems there is an error. I have the following output from Omnibus :
omnibus >> blockchain 2
[!] Unknown command
Artifact 2 is a blockchain adress.
Thanks for your help!
In current master branch, the code re-use in Dispatch.py is insane. This has cut the file size down in over half it's size, the workflow makes much more sense, avoid code re-use, and is much easier to follow overall
I've reworked this so all modules and machines go to a single submit
function within Dispatch:
self.run
against the artifact and return results
self.save_results
Provide support for URLs and expand modules to investigate them, add them as children, etc.
Better error handling within modules. Since they are imported libraries, it's difficult to pinpoint Python exceptions or other errors.
By leveraging cmd2's support for argparse we can get rid of a lot of commands and add argument support to many existing ones. For example, we'll now have support for commands like: tags --view inquest.net
, tags --add inquest.net these are my tags, https://inquest.net
, artifact --new inquest.net
, artifact --view inquest.net
, etc etc.
The parsers will go in their own library that we import into omnibus-cli so it doesn't look like garbage inside that script. lets add them to lib/parsers.py
Example of the tags parser:
# do_tags parsere
tags_parser = argparse.ArgumentParser()
id_tags = tags_parser.add_mutually_exclusive_group()
manage_tags = tags_parser.add_mutually_exclusive_group()
manage_tags.add_argument('-a', '--add', action='store_true', help='add tag')
manage_tags.add_argument('-v', '--view', action='store_true', help='view tags')
id_tags.add_argument('-l', '--last', action='store_true', help='use last created artifact')
id_tags.add_argument('-n', '--name', action='store', help='artifact name')
id_tags.add_argument('-i', '--id', action='store', help='artifact session ID')
tags_parser.add_argument('tags', nargs='?', help='comma separated tags')
and we'd port tags to look something like this:
@cmd2.with_argparser(tags_parser)
def do_tags(self, args):
"""Manage artifact tags"""
artifact = self.parse_identifier(args)
if artifact is None:
return
_type = detect_type(artifact)
if args.view:
result = self.db.get_value(_type, {'name': artifact}, 'tags')
info('%s tags: %s' % (artifact, result))
return
elif args.add:
if args.tags == '':
error('Tags not specified')
else:
tags = args.tags
new_tags = []
if ',' in tags:
for t in tags.split(','):
t = t.strip()
new_tags.append(t)
else:
new_tags = tags
if self.db.exists(_type, {'name': artifact}):
self.db.update_one(_type, {'name': artifact}, {'tags': new_tags})
success('Added tags to artifact (%s: %s)' % (artifact, new_tags))
else:
warning('Failed to find artifact in MongoDB. Run "new <artifact name>" before using the tags command')
else:
info('Run "tags --help" or "help tags" for all available commands')
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.