evilsocket / ergo-pe-av Goto Github PK

Hi, while I run sudo pip3 install -r requirements.txt I get an error with Lief:

sudo pip3 install -r requirements.txt
Requirement already satisfied: ergo-ai in /usr/local/lib/python3.7/dist-packages/ergo_ai-1.11.0-py3.7.egg (from -r requirements.txt (line 1)) (1.11.0)
Collecting lief (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/cd/0e/0d6f3357975dd1530aeb4b4a84a99d493775391758378fb5109f47b613f9/lief-0.9.0.zip
Requirement already satisfied: flask in /usr/lib/python3/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (1.0.2)
Requirement already satisfied: keras in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (2.2.4)
Requirement already satisfied: matplotlib in /usr/lib/python3/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (3.0.2)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (0.24.2)
Requirement already satisfied: pydot in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (1.4.1)
Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (0.9.0)
Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (0.0)
Requirement already satisfied: tensorflow in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (1.13.1)
Requirement already satisfied: terminaltables in /usr/local/lib/python3.7/dist-packages (from ergo-ai->-r requirements.txt (line 1)) (3.1.0)
Requirement already satisfied: numpy>=1.9.1 in /usr/lib/python3/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (1.16.2)
Requirement already satisfied: scipy>=0.14 in /usr/lib/python3/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.7/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (1.0.8)
Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (2.9.0)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (1.12.0)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (3.13)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from keras->ergo-ai->-r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: python-dateutil>=2.5.0 in /usr/lib/python3/dist-packages (from pandas->ergo-ai->-r requirements.txt (line 1)) (2.7.3)
Requirement already satisfied: pytz>=2011k in /usr/lib/python3/dist-packages (from pandas->ergo-ai->-r requirements.txt (line 1)) (2019.1)
Requirement already satisfied: pyparsing>=2.1.4 in /usr/lib/python3/dist-packages (from pydot->ergo-ai->-r requirements.txt (line 1)) (2.2.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sklearn->ergo-ai->-r requirements.txt (line 1)) (0.21.2)
Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: tensorflow-estimator<1.14.0rc0,>=1.13.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (1.13.0)
Requirement already satisfied: wheel>=0.26 in /usr/lib/python3/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (0.32.3)
Requirement already satisfied: termcolor>=1.1.0 in /usr/lib/python3/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: tensorboard<1.14.0,>=1.13.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (1.13.1)
Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (0.7.1)
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (1.21.1)
Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (0.2.2)
Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow->ergo-ai->-r requirements.txt (line 1)) (3.8.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn->ergo-ai->-r requirements.txt (line 1)) (0.13.2)
Requirement already satisfied: mock>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow->ergo-ai->-r requirements.txt (line 1)) (3.0.5)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/lib/python3/dist-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow->ergo-ai->-r requirements.txt (line 1)) (0.14.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/lib/python3/dist-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow->ergo-ai->-r requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from protobuf>=3.6.1->tensorflow->ergo-ai->-r requirements.txt (line 1)) (40.8.0)
Building wheels for collected packages: lief
Running setup.py bdist_wheel for lief ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-iq06all0/lief/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-4vma_dud --python-tag cp37:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/lief
copying lief/init.py -> build/lib.linux-x86_64-3.7/lief
running build_ext
Url: https://github.com/lief-project/LIEF/releases/download/0.9.0/lief-0.9.0-py3.7-linux.egg
HTTP Error 404: Not Found
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-iq06all0/lief/setup.py", line 285, in
'sdist': lief_sdist
File "/usr/lib/python3/dist-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 188, in run
self.run_command('build')
File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3.7/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 78, in run
_build_ext.run(self)
File "/usr/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/tmp/pip-install-iq06all0/lief/setup.py", line 215, in build_extension
self._install_from_source_package()
File "/tmp/pip-install-iq06all0/lief/setup.py", line 265, in _install_from_source_package
raise Exception("Unable to find {}".format(url))
Exception: Unable to find /root/lief-0.9.0-py3.7-linux.egg

Failed building wheel for lief
Running setup.py clean for lief
Failed to build lief
Installing collected packages: lief
Running setup.py install for lief ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-iq06all0/lief/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-rceiz0nd/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/lief
copying lief/init.py -> build/lib.linux-x86_64-3.7/lief
running build_ext
Url: https://github.com/lief-project/LIEF/releases/download/0.9.0/lief-0.9.0-py3.7-linux.egg
HTTP Error 404: Not Found
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-iq06all0/lief/setup.py", line 285, in
'sdist': lief_sdist
File "/usr/lib/python3/dist-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib/python3.7/distutils/command/install.py", line 589, in run
self.run_command('build')
File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3.7/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 78, in run
_build_ext.run(self)
File "/usr/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/tmp/pip-install-iq06all0/lief/setup.py", line 215, in build_extension
self._install_from_source_package()
File "/tmp/pip-install-iq06all0/lief/setup.py", line 265, in _install_from_source_package
raise Exception("Unable to find {}".format(url))
Exception: Unable to find /root/lief-0.9.0-py3.7-linux.egg

----------------------------------------

Command "/usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-iq06all0/lief/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-rceiz0nd/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-iq06all0/lief/

I'm using the latest version of kali Linux and I'm in ergo-pe-av directory

Hello,
I've spotted an issue in encode_pe function:

try:
        ep_offset = pe.entrypoint - pe.optional_header.imagebase
        ep_bytes  = [int(b) for b in raw[ep_offset:ep_offset+64]]
    except Exception as e:
        log.warning("can't get entrypoint bytes from %s: %s", filepath, e)

The ep_offset is actually an RVA, it needs an extra step to get the offset in file as explained below: https://stackoverflow.com/questions/33724306/calculating-the-file-offset-of-a-entry-point-in-a-pe-file

Hello,
I don't see any setup.py in the repo, how can I use the project directly by calling the python scripts ? Where's the ergo entrypoint ?
Thanks,

Dear Sir I want to generate the result of your experiment. Kindly guide me how can I do this?

In your blog at
https://www.evilsocket.net/2019/05/22/How-to-create-a-Malware-detection-system-with-Machine-Learning/#Features-Engineering
in "Feature Engineering" section you have the following instructions

"Assuming you have a folder containing malicious samples in the pe-malicious subfolder and clean ones in pe-legit (feel free to give them any name, but the folder names will become the labels associated to each of the samples), you can start the encoding process to a dataset.csv file that our model can use for training with"

Do I need to place the malicious files in this folder?

please let me know how can I generate the result?

It will be nice to have the malicious/benign data.csv so as the clean data.csv

Hello,
I'm playing with the ergo relevance command, and I'm wondering what the value means.
I'm guessing the higher the percentage the better the feature, but what does negative value mean ? Does positive mean more likely to be a clean sample and negative a malware (or the opposite, depending on labels order) ?

Dear Sir, do you still have the data.csv anyway?