inquest / threatkb Goto Github PK

We need a task workflow for artifacts that aren't easily parseable. Tasks are a generic concept and we'll need a new table with the following: title, owner, state, task text, final_artifact_text.

We need a CRUD API for tasks.

Pull state from cfg_states for state. The task details modal should have comments in it.

Also, we need the ability to convert the final_artifact_text into another artifact. For example, a task item might be to look at a potentially malicious DNS host. The owner of the task can do their research, conclude the DNS host is malicious, and click a button to convert the dns host into an official c2dns entry.

Get this in an endpoint and connect to UI or reimplement in JS:

#!/usr/bin/env python

import sys

for arg in sys.argv[1:]:
    result = []
    for c in arg:
        upper, lower = ("%02x" % ord(c.upper())), ("%02x" % ord(c.lower()))
        if upper == lower:
            result.append(upper)
        elif upper[1] == lower[1] and upper[0] != lower[0]:
            result.append("[%s%s]%s" % (upper[0], lower[0], lower[1]))
        elif upper[0] == lower[0] and upper[1] != lower[1]:
            result.append("%s[%s%s]" % (upper[0], upper[1], lower[1]))
        else:
            result.append("%s|%s"% (upper, lower))
    print "".join(result)

they need to be positive and negative files

Bug Repro: Edit a signature, click add tag, it falls into some refresh loop.

Instead of hard-coded columns in the artifact tables we should give the users the ability to dynamically create metadata fields (EX: subcategory1, subcategory2, file type, etc).

Field types should include:
singletext - standard text input
multitext - textarea text input
singlechoice - user must select a single option, options to be provided when the field is defined
multichoice - user can select multiple options, options to be provided when the field is defined
boolean - yes or no flat
date - a date
integer - a number
user - user can select from a dropdown of the users registered in threatkb

Fields can be:
required/not required

Admins should be able to create new field types and then assign them to artifacts. Once assigned, they should appear in the artifact detail modal.

Have some default settings

clean corpus files
navigation header image
base path for file upload
file preprocessor for upload
path to yara

Create a new configuration table for signature category to id range mapping. Call the model CfgCategoryRangeMapping and the table cfg_category_range_mapping.

id - int, pk
category - varchar, uniq
range_min - int, unsigned
range_max - int, unsigned
current - int, unsigned

Also, create a front end router/controller/service/link on the UI for CRUD operations on the new entity.

Be sure to create the migration and commit it also.

Add pagination with a default of 25 items.

Leave a comment in the existing IP/DNS with the reference link.

• High granularity.
• Keep a history of times. Warn if time has gone up by __%.
• Have the ability to view a report of slow running sigs.

To include:

• release notes
• artifact export
• configurable start/end dates
• configureable states

To use instead of hard tables.

Use something like this to add functionality that will allow the team to tag signatures, c2 dns, and c2 ip addresses.
http://mbenford.github.io/ngTagsInput/

The tags should be displayed in the table view page.

Create an auto-incrementing revision number on the signatures entity.

Consider for example the C2 IPs for WannaCry from https://www.lastline.com/blog/wannacry-ransomware/:

The Amazon IPs for example can be quick to recycle, resulting in a false positive opportunity. We want auto aging of C2 artifacts on the following passive attribute:

• Hard coded date or time delta.

The following are active measures, that we'll save for manual / net-assess:

• Custom check for existence of C2 endpoint.
• Generic check for change in OS or port fingerprint etc.
• Reverse DNS lookups (how many domains are hosted on this IP).
• Port scan baseline and change detection.

Normal users can only see artifacts assigned to them. Admins can see it all. We'll need this sooner than later to support working with non core team members.

Tasks
Releases
Import
Artifacts

• C2 IP
• C2 DNS
• Yara Rules
  Administration
• States
• Reference Text
• Categories
• Key/VAlue Settings
• Users

User would specify a list of strings and a variable name template / type / optional suffix, and get a list of ordered / named variables in return:

IN:
    A.com
    B.com
    C.com

TEMPLATE:
    $sXXX

TYPE:
    (regex|string) (/|")

OPTIONAL SUFFIX:
    "nocase ascii wide"

OUT:
    $S001 = "A.com" nocase ascii wide
    $S002 = "B.com" nocase ascii wide
    $S003 = "C.com" nocase ascii wide

In the case of regex, instead of string. You'd have:

OUT:
    $S001 = /A.com/ nocase ascii wide
    $S002 = /B.com/ nocase ascii wide
    $S003 = /C.com/ nocase ascii wide

Do, like now

Create a comments entity that will allow the team to create comments against the c2dns, c2ip, and signature artifacts.

id - int, pk
user_id - int, foreign_key to kb_users
date_created - datetime, autogenerated
comment - varchar

There needs to be rest endpoints to pull all comments for a particular artifact.

Dont forget to create the necessary migrations and commit it.

One textbox for writing a single string: input.
One text area that takes raw content, to which the string is applied using yara_rule.match

single output, match or no match.

Add functionality that would allow users to filter specifically (and only) on the signature column. If we can make this generic, that would be ideal.

EX: state:"Ready For Production" would filter on the state column only and look for the text "Ready For Production". category:"Malicious Content" would filter only on the category column and look for the text "Malicious Content".

@pedramamini

• Go to users, edit one, hit cancel. User list is gone. Done
• Import functionality seems to hang when you hit "extract" ... need a spinner? Done
• Same feeling when uploading files. Don't spin too much time on a spinner though, I'm sure @ofzza can whip that one up fast. Done
• When editing a signature, the required field indicator vanishes. The form can't be submitted until the required fields are filled, but they are no longer marked. Description, reference text and reference link is what I experienced. Fixed. The red * will remain even after you type now.
• Can't extract artifacts from a list of domains. List of IPs is working, list of domains is failing to be recognized (I wasn't giving the http:// prefix, is that required?) The http:// prefix is required otherwise we will get tons of false positives. Another option is to anchor on TLDs. Currently the regex to extract is 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*(),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'.

^^ radio button options for specific import types

Comments are one way we can track this. But I'd like this to track reference timestamps as well. This will allow us to build a timeline for each artifact. First seen and by who, then how it spreads.

For example, AWS ranges. DNS whitelist.

Ability to create, suspend, and edit users. Not sure about adding delete support, as it will break data integrity.

Accept a file upload that will be scanned with all yara sigs and show the resulting hits (-s option). Discard the file after scan is complete.

• Attaching files to a signature is not working.
• For importing signatures, must look at these fields and update accordingly (when

		Revision = "2"
		Last_Revision_Date = "09/29/2015"

• Add signature name to confirmation dialog for deletion: "Are you sure you want to inactivate this signature?"
• Edit a signature, click add tag, it falls into some refresh loop.
• Move non-editable fields to the top of the dialogue as a header?
  
• Move sig name field up, just ender eventid.
• Can't do a release:

Traceback (most recent call last):
  File app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File utils.py", line 228, in decorated_view
    return func(*args, **kwargs)
  File __init__.py", line 31, in wrapped
    return f(*args, **kwargs)
  File releases.py", line 92, in create_release
    release.release_data = release.get_release_data()
  File releases.py", line 112, in get_release_data
    if parser.parse(ip["date_modified"]) > datetime.datetime.now():
TypeError: list indices must be integers, not str

• Importing fails if rule isn't on its own line, lets add a case to parse this correctly too:

}rule FC_Multiple_Embedded_OLE_Objects
{

• Include counts on pages: states, categories.