directory support about evtx HOT 7 CLOSED

This could be incorporated with #37. But, should we include a source of which records come from which files?

from evtx.

RE: #37 (comment)

"Is it recursive?"

• I think recursion should be expected if a directory is specified. I am okay with an .evtx extension filter. That being said, an option switch to add recursion is not a large effort.

"How does one know where errors occurred?"

• We can log error to stdout with the filename in which it occurred

"In the common use case where one would like to save each .evtx to a respective .json file, how is this done?"

• If you batch file processing, which I have been doing, you can output to whichever name you like. For processing a directory of files I would generally expect a single "concatenated" file. For JSON output, it would be nice to have an option that would append a _source (or similarly named attribute) that describes its source. Not sure how this should be handled for XML though. I like the idea of single files rather than multiple files for ingestion into other types of back ends.
• What are your thoughts on this?

"If iterating over a whole directory and writing to files, how do is progress reflected?"

• I generally just output to STDERR when a file is processed or errors out. I don't think a "progress bar" is a requirement. Plus, evtx_dump is so fast that its not a nuisance to not know whats left :P

from evtx.

+1 Directory support, with recursion option, and ability to maintain "source" of the event log. This is very helpful when processing artifacts in batch.

from evtx.

@dnides I've talked with @forensicmatt with this on slack, and since this can behave somewhat unexpectedly in some use cases, such as XML output, and also files which may have different settings (like different ANSI encodings) we've opted to refrain from implementing this for now.

I'm not sure I see why this is preferable to a very simple python/shell script.
The unix philosophy on this is to make sure the tool does one thing well, and chain little tools together.

For example, this does just the exact same thing you've asked for (on a unix machine):
find . -name "*.evtx" | xargs -L1 evtx_dump -o jsonl

It even summarizes the stderr nicely:

I'm sure powershell has an simple alternative (I'm not very proficient in powershell).

If you want to modify and add metadata for the records, you could something like this (python extension is still multi-threaded):

import os
import sys
from evtx import PyEvtxParser

def evtx_collect(root):
    with open('out.json', 'wt') as j:
        for r, d, files in os.walk(root):
            for f in files:
                if f.endswith(".evtx"):
                    path = os.path.join(r, f)
                    print(path)
                    p = PyEvtxParser(path)
                    try:
                        for record in p.records_json():
                            # Do something with the JSON
                            j.write(record['data'])
                            j.write('\n')
                    except RuntimeError as e:
                        print(e, file=sys.stderr)

from evtx.

Thanks and understood on directory/recursion part. How about at minimum adding the source /path (i.e. C:\windows\system32\winevt_logs\Application.evtx) of the event log to the XML and JSON output? This way, when a bunch of logs are processed, and then later analyzed we know where each came from.

from evtx.

@dnides This isn't entirely trivial, since there isn't always a path (if loading the file from a stream), and since the path from evtx_dump is discarded very early on in favor of an opaque data stream (which allows stuff like the python extension to work), so we could consider this in a separate issue, but I can't promise I could support that within evtx_dump.

Internally we have a batch ingestion pipeline which receives the JSON logs from PyEvtxParser and adds _source and a bunch of other metadata fields, but again this is somewhat out of scope for evtx_dump.

As for directory processing, with the merging of #40, I'll add some documentation for batch processing and will close this issue.

from evtx.

I've merged #40, released 0.4.0 with said support, and have added notes on batch processing.
If there are any further concerns, please open another issue :)

(from the README):

evtx_dump can be combined with fd for convinient batch processing of files:

`fd -e evtx -x evtx_dump -o jsonl` will scan a folder and dump all evtx files to a single jsonlines file.
`fd -e evtx -x evtx_dump -f "{.}.xml` will create an xml file next to each evtx file, for all files in folder recursively!

from evtx.