jinmo / idapkg Goto Github PK

Can you please add a license to the repository?
The absence of a license makes usage and contribution tricky.

While I was installing idapkg in IDA 7.2, I ran into some error with the following error messages:

Downloading idapkg...
Generating initial config at C:\<omit>\idapkg\config.json
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python27\Lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python27\Lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<string>", line 2, in b
  File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\__init__.py", line 4, in <module>
  File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\commands.py", line 4, in <module>
  File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\package.py", line 22, in <module>
  File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\env.py", line 79, in <module>
  File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\env.py", line 35, in __load_version_from_ida
  File "C:\Program Files\IDA 7.2\python\ida_kernwin.py", line 895, in get_kernel_version
    return _ida_kernwin.get_kernel_version(*args)
RuntimeError: Function can be called from the main thread only

I think it is similar to this issue

I think it's because IDA 7.2 introduced checks for IDA API functions to make sure they can only be called from the main thread.

IDAPython: all functions not marked as THREAD_SAFE in the C++ SDK, will now check that they are being called from the main thread, avoiding possible corruption or crashes

idapkg's uninstaller.py suffers a traditional encoding issues while concatenating the string:

............<omitted>
  File "C:/<omitted>/idapkg-master/uninstaller.py", line 32, in <module>
    update_pythonrc()
  File "C:/<omitted>/idapkg-master/uninstaller.py", line 12, in update_pythonrc
    sep_with_ver = SEP[0] + __version__
TypeError: can't concat str to bytes

Using the code sep_with_ver = SEP[0] + __version__.encode() will fix the issue ( I'm not sure if this is the best solution though ).

I'm gonna remove dependencies or embed it to make the installation part completely offline, and make it minimal, while I'm not sure if it's possible to bundle virtualenv.

Package actions

• ifred -> chooser + use ifred if present

Finding IDAUSR bases

• lief -> kaitai struct
• capstone -> minimal disassembler

I think it would be neat to install packages directly from github (or other git repos).

ENV

OS: Windows 10
Python: 3.8.5
IDA version: 7.5
IDAPython: v7.4.0
idapkg : 0.1.4

Details

During the installation, the install command shows the following error message:

idapkg version 0.1.4
Will install virtualenv at 'C:\\Users\\bruce30262\\idapkg\\python' since pip module is not found...
Downloading virtualenv from 'https://files.pythonhosted.org/packages/57/6e/a13442adf18bada682f88f55638cd43cc7a39c3e00fdcf898ca4ceaeb682/virtualenv-20.0.21-py2.py3-none-any.whl' ...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 90, in prepare_virtualenv
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "<string>", line 2, in install
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\main.py", line 87, in init_environment
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 110, in prepare_virtualenv
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 108, in handler
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 108, in <listcomp>
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 106, in <lambda>
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 67, in _install_virtualenv
  File "<frozen zipimport>", line 259, in load_module
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmp68526rtt.zip\virtualenv\__init__.py", line 3, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmp68526rtt.zip\virtualenv\run\__init__.py", line 5, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmp68526rtt.zip\virtualenv\run\app_data.py", line 6, in <module>
ModuleNotFoundError: No module named 'appdirs'

The ImportError message will be shown while importing the following modules:

• appdirs
• filelock
• six

All error can be resolved by using pip3 install <module>

Side note: idapkg cannot find pip on my machine because this line of code

After that, the install script will emit another error:

Creating environment using virtualenv...
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 90, in prepare_virtualenv
    raise ImportError()
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "<string>", line 2, in install
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\main.py", line 87, in init_environment
    prepare_virtualenv(wait=True)
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 110, in prepare_virtualenv
    __work(handler) if not wait else handler()
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 108, in handler
    def handler(): return ([task()
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 108, in <listcomp>
    def handler(): return ([task()
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 106, in <lambda>
    tasks.insert(0, lambda: _install_virtualenv(path))
  File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 71, in _install_virtualenv
    virtualenv.create_environment(path, site_packages=True)
AttributeError: module 'virtualenv' has no attribute 'create_environment'

The error seems to have something to do with this issue

Don't get me wrong ! I've been playing this plugin for a while and it really is an amazing project :) 👍
However when I was testing the plugin, I encountered some problem and have to find a way to re-install idapkg completely.

To be brief, I have two version of IDA in my computer (Windows 10), and when I was testing the plugin, it kind of screwed up the variable in config.json ( I think it's idausr_native_bases ), making me unable to open ida ( it kept crashing ). After I installed idapkg, I first open version A's ida.exe, then open version B's ida64.exe ( both of them worked perfectly ). And then I found I was unable to open version A's ida64.exe and version B's ida.exe ( both of them crashed ). It took me quite a while to find the root cause and have to delete config.json & re-install idapkg to fix the issue.

This make me think that the project should provide a way to uninstall the plugin (and document it in the manual), in case the user have decided not to use the plugin anymore. As far as I understand it has something to do with idapythonrc.py, and it's located in C:\Users\<username>\AppData\Roaming\Hex-Rays\IDA Pro\idapythonrc.py (Windows) , which is not an easy place for user to reach/discover. It would be nice if we can use a single command in IDA's python shell to remove the plugin completely.

Currently idapkg adds package root to sys.path, and I'm not sure if it's a good way.

Some path candidates that can be added to sys.path (but not sure what's best):

- idapkg/packages
- idapkg/packages/<package root> (current)
- idapkg/packages/<package root>/python

IDAPython adds <IDA installation path>/python to sys.path by default, and some plugins use this scheme in deployment. (e.g. Ghidra <-> IDA converter) One can just move python/* to package root, but I'm not sure if this is a good way. (It works, and doesn't conflict commonly, but if some files like setup.py exists and some package uses import setup or etc, it does matter.)

Here, I like the approach of idaenv which tries to wrap all packages as a python (ex. IDArling supports idaenv). Using setup.py or other setuptools/distutils module enables reusing python's package management system. (One can pip install ./idarling/ or etc though)

Package schema is a subject that improvements can be made on and on, so I made this thread.

This is inspired by Pipfile.lock.

Overall structure

Required:
+   info.json

Optional for IDA:
+   plugins/
       ...
+   procs/
       ...
+   loaders/
       ...
+   til/ sig/ ids/
       ...

Optional for package:
+   README.md

package-id/info.json

{
	// package name
	"name": "Community Headers",
	// version used in package management (required)
	"version": "0.0.1",
        "description": "",
        "homepage": "https://example.com",
	"installers": ["installer.py"],
	"dependencies": {
		"other-plugin-id": ">=1.0"
	}
}

NOTE

• "installer" must be python script, and is executed after downloading package. When an exception is thrown, the installation aborts and the package is not installed.
• "dependencies" is dependency list between packages. Will use semantic_version package which implements semver 2.0.
• The plugin directory will be added to sys.path after loading.

I was thinking about installing required pip packages, but installer can do that. Since there are many implementation for pip packaging, I'm not sure if I should expose a pip-installing function from package manager.

Maybe a website for validating the scheme and hosting some packages would be good.

In commands.py, remote() has the following usage comment:

Find a remote package from given repos.
................(omitted)...................
:type repo: str or list(str) or None
:returns: None if package is not found, else InstallablePackage instance.
:rtype: InstallablePackage

The type of repo should be "list(str) or None", as stated in install().

Am too lazy to send a PR

ENV

OS: Windows 10
Python: 3.8.5
IDA version: 7.5
IDAPython: v7.4.0
idapkg : 0.1.4

Detail

While installing idapkg, the install command shows the following warning message:

idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:100: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:100: SyntaxWarning: "is" with a literal. Did you mean "=="?

It seems to install idapkg successfully, however the warning messages' kind of annoying

I think the SyntaxWarning message was introduced in Python 3.8
Changing is to == should fix the problem

I'm going to write a plugin combining which internally uses angr for some analysis, because the angr is dropping Python 2 support, so I decided to use a package system for easier installation. However the angr is recommended to be installed in a separate virtualenv. But after going through the doc of idapkg, it seems that the whole idapkg is sharing one single virtualenv.
Can we add options to somewhere like info.json in order to make some packages use their own env?

Currently, idapkg uses idapythonrc.py to bootstrap idapkg. This structure makes moving idapkg itself to idapkg/packages possible. (Though ~/idapkg/config.json directory must be hardcoded for now)

Doing this gives some benefits: upgrading idapkg easier. Then it can make one more improvement: making the installation script inlined. Currently, idapkg.py does this.

Starting from version 0.10.0 LIEF had dropped the support for Python 2.7. Since idapkg will install the latest version of LIEF this will results in installation error if the user is using IDA before version 7.4 ( = no Python 3 support ).

Right now a dirty fix is to bind the version of LIEF to 0.9.0 ( python -m pip install lief==0.9.0 will resolve the issue ). However I think it is necessary to find an alternative package for LIEF ( e.g. katai struct like you mentioned in #10 ). The package should be Python2/3 compatible if possible, since not everyone has the latest version of IDA :P

I was testing changing IDAUSR variable to add directories for plugins/ procs/ loaders/ but it didn't work since IDA cached the list after calling get_ida_subdirs() or etc. I was annoyed since it's not exposed to user, making it impossible to invalidate cache without directly modifying the memory with ctypes. This completely breaks loading of processor/loader since idapkg is loaded after caching the environments.

Wrapping processors/loaders with native .py/.dll solves all of these issue, but needs extra work. e.g. Symbolic linking would not work since it modifies the path. This breaks relative imports.

So my plan is: Writing wrapper function with global constructor, like this.

__attribute__((constructor))
void init() {
  init_processor(); // copies LPH structure from wrapped dll
  init_loader();
  init_plugin();
}

define_wrapper(processor_t, LPH, processor);
define_wrapper(loader_t, LDSC, loader);
define_wrapper(plugin_t, PLUGIN, plugin);

Python wrapper:

import os

filename = '<wrapper path>'
dirname = os.path.dirname(filename)
orig_cwd = os.getcwd()
try:
    os.chdir(dirname)
    execfile(filename, {__file__: filename})
except:
    pass
finally:
    os.chdir(orig_cwd)

Indeed IDAUSR's feature is rich, and I need to implement these:

Not copyable (path-sensitive):
plugins/
procs/
loaders/

Copyable:
ids/
til/
sig/

Personally I don't like copying files, but there's no symlink support without Administrator privileges for Windows, and non-NTFS directories (but possible in NTFS, and w/ admin privilege).

Unless, I can use hardcoded offset for IDA (or use FLIRT for IDA binary?). I'm uploading PoC.

As titled, I found that I cannot login to idapkg.com ( it says "Network error" on the web page ). Also I wasn't able to access the repo page ( e.g. https://idapkg.com/p/PythonEditor ). Is everything OK ?

Normally, the author can bundle native libraries as dependencies, but it'll occupy space too much when it comes to python plugin. So I was just thinking about virtualenv-based python package management. Here's some problems:

1. pip doesn't support in-process packaging

So pip recommends programmers to call pip as separate process. IDA uses python interpreter installed into system, so maybe I can find and use the interpreter by determining python.dll used in the process. (sys.executable is set to ida.exe in IDAPython. This breaks virtualenv calls in IDAPython)

Solved.

2. Versioning

I think that PEP440 or semantic versioning would be good.

Env

• IDA version 7.2
• Windows 10 64 bit
• idapkg version 0.1.1

Detail

I've uploaded a loader for IDA Pro: https://idapkg.com/p/mclf_loader
In case you want the zip file: mclf_loader.zip

When I copy the command in the web page and execute it, I got the following error messages:

Python>pkg.install('mclf_loader', repo='https://api.idapkg.com')
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\commands.py", line 41, in install
    spec = _parse_spec(spec)
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\commands.py", line 21, in _parse_spec
    semantic_version.Spec(version)
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 505, in __init__
    subspecs = [self.parse(spec) for spec in specs_strings]
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 511, in parse
    return tuple(SpecItem(spec_text) for spec_text in spec_texts)
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 511, in <genexpr>
    return tuple(SpecItem(spec_text) for spec_text in spec_texts)
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 425, in __init__
    kind, spec = self.parse(requirement_string)
  File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 440, in parse
    raise ValueError("Invalid requirement specification: %r" % requirement_string)
ValueError: Invalid requirement specification: u'_loader'

BTW if I place the mclf_loader dir under the idapkg/packages folder, it works perfectly fine.