malice-plugins / fileinfo Goto Github PK
View Code? Open in Web Editor NEWMalice File Info Plugin (libmagic, exiftool, TRiD and ssdeep)
License: MIT License
Malice File Info Plugin (libmagic, exiftool, TRiD and ssdeep)
License: MIT License
Newer Exiftool version (I've tested v11.59) has better file detection capabilities
Hi,
Thanks for this awesome tool, really makes my work a lot easier.
I ran into an issue while trying to automate the web service, it happens when I submit the same file multiple times, not simultaneously:
fileinfo_1 | 2018/07/06 15:44:28 cannot open; magic mime db is already open
any help would be highly appreciated ๐
If I delete the user in the dockerfile
2018/09/08 16:15:52 failed to index malice/fileinfo results: failed to update sample with id: beT3uWUBI6b13m1ZR4ow: elastic: Error 400 (Bad Request): Limit of total fields [1000]
in index [malice] has been exceeded [type=illegal_argument_exception]
All plugins are up to date.
Scanning putty.exe (7afb56dd48565c3c9804f683c80ef47e5333f847f2d3211ec11ed13ad36061e1) results in the following error:
>> docker -D run --rm -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`:/malice/samples --network="host" malice/fileinfo putty.exe
...
#### Exiftool
| Field | Value |
|-------------|----------------------|
| error | exit status 1 |
...
Some of the other files i tested worked as expected.
Docker version:
Docker version:
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:49:01 2018
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false
Docker info (with some info removed):
Docker info:
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 26
Server Version: 18.09.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Linux Mint 19
OSType: linux
Architecture: x86_64
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
Hi,
I ran into this weird bug when sending multiple API requests concurrently, this makes one request affect the response of other requests.
This can be reproduced using Apache Benchmark:
ab -n 10000 -c 1 -p post_data.txt -T "multipart/form-data; boundary=f" http://localhost:3993/scan
the post_data.txt
file holds the form data with the TXT file.
Send another request providing a PE file, while ab
is running:
$ curl -i -X POST -H "Content-Type: multipart/form-data" -F "[email protected]" http://localhost:3993/scan
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Mon, 20 Aug 2018 10:20:26 GMT
Content-Length: 1380
{
"magic":{
"mime":"text/plain",
"description":"**ASCII text, with very long lines, with no line terminators**"
},
"ssdeep":"6144:WmNxBCD65shMJmpdFvuKmZEPi4rj2/kH+/MH:WoWD0shvdhqEa4HOmIMH",
"trid":[
"**82.7% (.EXE) Win32 Executable Microsoft** Visual Basic 6 (82067/2/8)",
"6.6% (.DLL) Win32 Dynamic Link Library (generic) (6578/25/2)",
"4.5% (.EXE) Win32 Executable (generic) (4508/7/1)",
"2.0% (.EXE) OS/2 Executable (generic) (2029/13)",
"2.0% (.EXE) Generic Win/DOS Executable (2002/3)"
],
"exiftool":{
"CharacterSet":"Unicode",
"CodeSize":"307200",
"Comments":"Scribblative",
"CompanyName":"NCH Software",
"EntryPoint":"0x11b8",
"ExifToolVersionNumber":"11.06",
"FileDescription":"Redcrested2",
"FileFlags":"(none)",
"FileFlagsMask":"0x0000",
"FileOS":"Win32",
"FileSize":"316 kB",
"FileSubtype":"0",
"FileType":"Win32 EXE",
"FileTypeExtension":"exe",
"FileVersion":"1.00",
"FileVersionNumber":"1.0.0.0",
"ImageVersion":"1.0",
"InitializedDataSize":"24576",
"InternalName":"Infraglacial2",
"LanguageCode":"Chinese (Traditional)",
"LinkerVersion":"6.0",
"MIMEType":"application/octet-stream",
"MachineType":"Intel 386 or later, and compatibles",
"OSVersion":"4.0",
"ObjectFileType":"Executable application",
"OriginalFileName":"Infraglacial2.exe",
"PEType":"PE32",
"ProductName":"Touch",
"ProductVersion":"1.00",
"ProductVersionNumber":"1.0.0.0",
"Subsystem":"Windows GUI",
"SubsystemVersion":"4.0",
"UninitializedDataSize":"0"
}
}
As you can see according to Magic it's a text file while Trid and Exiftool say it's PE, Magic is the only service that is effected by this as you can see.
If we stop AB from making requests, fileinfo processes the file fine and Magic detects it as PE as intended
Thank you again for making this tool, any help would be appreciated ๐
I need to decide how to render that info the the user.
I will most likely output to the JSON/Markdown table letting them know the file has caused a timeout.
A little birdie told me that this happens a lot at 'scale'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.