punch-cyber / yaraguardian Goto Github PK

It will be useful to have a sort of validator during import (or during export) that reports warning/errors in the rules.
Something like run Yara against the rule and catch the warnings and the errors:

Add dropdown indication on edit / search features

Develop a method of preserving embedded comment content from rules during parsing (requires enhancement of plyara)

Finish testing modules

Update long IN list query statements to use CTE optimization or temporary tables instead

Received error message:

{"success_count":143,"message":"unknown text at include; token of type ID","last_success":"Trojan_DNS_Calc_String"}

Rule in question contained the following strings that likely caused the error:

strings:
    $a = "%c%c%c%c%c.exe"
    $b = "~dfds3.reg"

condition:
    all of them

Cleanup multiple import statements on export of rules

Hi,
we are using a custom script to download and keep updated yara rules from different github repos.
Each repo is stored in particular sub-folders under the main folder that we want to bulk upload in YaraGuardian with the UploadDirectory command.
The actual options are:

• bulk import everything having a single source option for everything
• import each sub-folder separately

Can be useful to add an option the the command that permit to use sub-folders name as source?

By default all emails come from webmaster@localhost this doesn't work well for some mail gateways.

Configuring DEFAULT_FROM_EMAIL in settings.py does change this from address, however should probably be configured from the upstream settings like all other settings.

Identify how to properly handle rules with external variables

Adding tags to apply to search result does not work if there is a space in the field. This occurred when I tried to add 2 tags but forgot to separate them with a comma.

The tool reported that the tags were applied successfully, but the tags were not added when the rules were reviewed again.

Merge button currently disabled. Update logic and enable.

The ability to make bulk changes to a yara file (or set or yara files) at the time of ingestion would make for a more efficient workflow.

Example - I am uploading a series of yara rules regarding PDFs - I'd like to make bulk changes to the name and metadata tags at time the files are uploaded. It makes for an easier workflow than uploading and then going in to find them based on upload time or a similar attribute.

Requests via pagination buttons for rule list are blocked due to attempting to request content via HTTP when it should be over HTTPS.

Serving static files with current docker configuration appears problematic without an Nginx proxy. This can be confusing or difficult for new users who would like to just try out YaraGuardian. Recommend update to allow for simple configs.

Project lacks documentation for using YaraGuardian with Docker. This can be potentially confusing for new users.

Add capability to automatically monitor and ingest github rule repositories.

When following the registration link sent via the registration token email, the webpage redirects to the login page. This means one would need to login before using the registration link.

[03/Aug/2017 16:00:41] "GET /register/ HTTP/1.1" 302 0
[03/Aug/2017 16:00:42] "GET /login/?next=/register/ HTTP/1.1" 200 3381

created_after, created_before, modified_after, modified_before query filters not working properly.

UI currently uses Angular 1.x and should be updated / modernized.

Command line management tools currently do not work:

process_parsed_rules() missing 1 required positional argument: 'owner'

Capability for users to submit rulesets to a context in which they are not the owner or admin.

*** Status placeholder already exists for this ("pending" & "rejected") - logic just needs to be finished ***

Update YaraGuardian to leverage the now maintained version of plyara instead of managing own.

Enforce rule dependencies to be exported before the dependent rules

1. Add category and source 'required' toggles
2. Display, enable, or disable elements based on group permissions
3. Restructure UI to make more manageable and visually appealing

Add capability to transfer rules from one ruleset context to another

External variable references are currently detected as dependencies. Needs to be updated.

Currently using bower. Migrate to yarn.

Will be amazing if it is possible to enable yara testing on a specified folder of files and send the results to the owner via email or just notify when it is done processing.

Hey!
I tried install your repo on a clean ubuntu16 machine,
I ran sudo ./install.sh - without and errors.
after I run collect static I get this permission error
(YaraGuardian) matand@ubuntu:/YaraGuardian$ python manage.py createsuperuser
/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/psycopg2/init.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.
""")
Username: matand
Email address: 1
Error: Enter a valid email address.
Email address:
Password:
Password (again):
Superuser created successfully.
(YaraGuardian) matand@ubuntu:/YaraGuardian$ python manage.py collectstatic
/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/psycopg2/init.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.
""")

You have requested to collect static files at the destination
location as specified in your settings:

/home/matand/YaraGuardian/static

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Copying '/home/matand/YaraGuardian/stylesheets/app.css'
Traceback (most recent call last):
File "manage.py", line 8, in
execute_from_command_line(sys.argv)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 189, in handle
collected = self.collect()
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
handler(path, prefixed_path, storage)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 354, in copy_file
self.storage.save(prefixed_path, source_file)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/core/files/storage.py", line 49, in save
return self._save(name, content)
File "/home/matand/YaraGuardian/.venv/lib/python3.5/site-packages/django/core/files/storage.py", line 264, in _save
fd = os.open(full_path, flags, 0o666)
PermissionError: [Errno 13] Permission denied: '/home/matand/YaraGuardian/static/app.css'

Thanks

Allow group owners to specify which status that member rule submissions go to. Add dropdown on status selection for owner/admins

I followed the installation instructions for a Development environment, but when I go to run the first command python manage.py migrate, I receive a traceback that throws AttributeError: module 'django.contrib.auth.views' has no attributes 'password_reset_confirm'

Any help here would be great 👍

Full traceback:

(YaraGuardian) zeroharbor :: /opt/YaraGuardian ‹master*› » python3.5 manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/base.py", line 361, in execute
    self.check()
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/base.py", line 390, in check
    include_deployment_checks=include_deployment_checks,
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 65, in _run_checks
    issues.extend(super()._run_checks(**kwargs))
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/management/base.py", line 377, in _run_checks
    return checks.run_checks(**kwargs)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/urls/resolvers.py", line 398, in check
    for pattern in self.url_patterns:
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/urls/resolvers.py", line 579, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/site-packages/django/urls/resolvers.py", line 572, in urlconf_module
    return import_module(self.urlconf_name)
  File "/home/adam/.local/share/virtualenvs/YaraGuardian-6kWGavhf/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/opt/YaraGuardian/YaraGuardian/urls.py", line 45, in <module>
    auth_views.password_reset_confirm, {'template_name': 'prelogin/ResetPassword.html'},
AttributeError: module 'django.contrib.auth.views' has no attribute 'password_reset_confirm'

Currently any rules imported using management commands are specified as active. Add option to allow a different status to be specified.

File names for upload do not display immediately.

docker build . cmd output:

Sending build context to Docker daemon   2.37MB
Step 1/12 : FROM ubuntu:16.04
 ---> 4a689991aa24
Step 2/12 : MAINTAINER Adam Trask ”[email protected]”
 ---> Using cache
 ---> 3bcebe9f88ac
Step 3/12 : ENV LANG='C.UTF-8' LC_ALL='C.UTF-8' LANGUAGE='C.UTF-8' NODE_VERSION='8.0.0'
 ---> Using cache
 ---> aa6469c49987
Step 4/12 : ENV API_DIR='/usr/local/YaraGuardian' API_USER='YaraManager' API_GROUP='YaraManager'
 ---> Using cache
 ---> c8e7fa61c80c
Step 5/12 : ADD . ${API_DIR}
 ---> Using cache
 ---> 2211e8de80bc
Step 6/12 : RUN apt-get update   && apt-get -y install software-properties-common   && apt-add-repository -y multiverse   && apt-get update   && apt-get upgrade -y   && echo "Installing prerequisite packages..."   && apt-get -y install     curl     git     libpq-dev     npm     python3     python3-dev     python3-setuptools   && easy_install3 pip   && apt-get clean -y   && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Using cache
 ---> a58747f972d0
Step 7/12 : WORKDIR ${API_DIR}
 ---> Using cache
 ---> 432e3b9e88e6
Step 8/12 : RUN echo "Installing NodeJS version ${NODE_VERSION}"   && npm cache clean -f   && npm install -g n   && n ${NODE_VERSION}   && ln -sf /usr/local/n/versions/node/${NODE_VERSION}/bin/node /usr/bin/node   && echo "Installing python requirements..."   && pip3 install pipenv   && pipenv install --deploy --system   && echo "Installing front-end components"   && npm install yarn -g   && yarn   && yarn webpack   && python3 manage.py collectstatic --noinput   && rm -rf /usr/local/n
 ---> Using cache
 ---> 0af428884f8b
Step 9/12 : RUN groupadd -r ${API_USER}   && useradd -r -g ${API_GROUP} ${API_USER}   && chown -R ${API_USER}:${API_GROUP} ${API_DIR}
 ---> Using cache
 ---> 4ca1b12def0b
Step 10/12 : EXPOSE 8080
 ---> Using cache
 ---> 1098b35d97b3
Step 11/12 : USER ${API_USER}
 ---> Using cache
 ---> abf0d30edd56
Step 12/12 : CMD PYTHONUNBUFFERED=1 gunicorn -k gevent --bind=0.0.0.0:8080 --access-logfile - --error-logfile - YaraGuardian.wsgi:application
 ---> Using cache
 ---> 0a88da07ffb9
Successfully built 0a88da07ffb9

docker-compose run web yarn cmd output fails with error:

Starting yaraguardian_db_1 ... done
yarn install v1.10.1
warning Skipping preferred cache folder "/home/YaraManager/.cache/yarn" because it is not writable.
warning Selected the next writable cache folder in the list, will be "/tmp/.yarn-cache-999".
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
error Could not write file "/usr/local/YaraGuardian/yarn-error.log": "EACCES: permission denied, open '/usr/local/YaraGuardian/yarn-error.log'"
error An unexpected error occurred: "EACCES: permission denied, mkdir '/usr/local/YaraGuardian/node_modules'".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: ENOENT: no such file or directory, open '/home/YaraManager/.yarnrc'

Large rule files are being reported as parsing rules inconsistently, but parse correctly when split up across multiple files.

Requesting others who experience similar behavior add their observations to help track down possible root cause.

Logic hashing is too simplistic and prone to inaccuracies. Update method.

Create ability to have tables and stats be sortable so user can more efficiently normalize rulesets.

Attached valid rules cause parser to fail:

parse_error_rules.txt

Newer versions of plyara have different output and semantics - will need to adjust YaraGuardian accordingly.

Plyara doesn't parse rules with an empty quotation mark ("") correctly.

eg: pe.sections[0].name == ""

Hello folks, thanks for the project. I noticed an enhancement request for Yara Testing (ie scanning) and would like to add some requests/suggestions on top of that around how the file scan tab might look and function which I think could be particularly useful.

First there could be a simple file system browser, with nested folder expansion, to navigate files to scan. Another file system browser that shows the structured Yara rule collection. Another area would have a text editor for live Yara rule modification, which could save or discard changes, etc. A last area could show the hit locations, string matches, debug info, etc, after a scan.

The Yara rules could be selected/deselected or marked for negative (to filter results) on scans against the scan file browser chosen directories or files. The results could be reflected in the file system browser frame via highlighting and filtering, along with the detailed scan result area.

For example, if there are 1000 files in the scan file browser and simple “PDF” and “OFFICE” yara sigs are selected in the rule browser, then only those files with matches would appear in the file browser. Ie, all JPEGS or JARS or whatever else would be grey or gone. If then an “RTF” signature is marked negative then the OFFICE files that are actually RTF would go away as well. If the RTF rule is then modified in the editor frame to account for some files that were missed, perhaps some syntax validation and color coding would be helpful.

The file system browser areas could allow for file system manipulations. For example, now that we have PDF and OFFICE files without RTFs identified via scan, the user could copy the identified files to a new directory, maybe even zip them up, open in notepad, or whatever. Likewise with the particular Yara rules.

Thanks for your consideration.

Installing on to a Ubuntu 16.04 vps.
Run install.sh
install pip requirements
setup config.json

not using venv

python manage.py migrate

ubuntu@yara-guardian:~/YaraGuardian$ python manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 398, in execute
    self.check()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 426, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 75, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 23, in check_resolver
    for pattern in resolver.url_patterns:
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 33, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 417, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 33, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 410, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/ubuntu/YaraGuardian/YaraGuardian/urls.py", line 3, in <module>
    import YaraGuardian.API.urls
  File "/home/ubuntu/YaraGuardian/YaraGuardian/API/urls.py", line 7, in <module>
    import YaraGuardian.API.rules
  File "/home/ubuntu/YaraGuardian/YaraGuardian/API/rules.py", line 3, in <module>
    from rules.REST_views import (RulesetsListingView,
ImportError: No module named REST_views

Feature you would like to see: Ability to add upstream sources of yara signatures from popular repositories.

Why you need it: Other people write much better yara rules than me :)

How it should work: Unclear at this point. More of an idea I had while talking with a co-worker today. But if you could add a git repo as a source of yara files - and even better, if it could automatically update on a schedule - that would be awesome!

First repo that made me think of this idea:

• https://github.com/Neo23x0/signature-base