quarkslab / quokka Goto Github PK

View Code? Open in Web Editor NEW

162.0 10.0 10.0 186.61 MB

Quokka: A Fast and Accurate Binary Exporter

Home Page: https://quarkslab.github.io/quokka/

License: Apache License 2.0

CMake 4.81% Python 39.67% Dockerfile 0.35% C++ 54.68% C 0.49%

ida-plugin reverse-engineering binary-exporter binexport

quokka's Introduction

Quokka

image generated by DALL-E

Introduction
Installation
Usage
Building
Documentation
FAQ

Introduction

Quokka is a binary exporter: from the disassembly of a program, it generates an export file that can be used without the disassembler.

The main objective of Quokka is to enable to completely manipulate the binary without ever opening a disassembler after the initial step. Moreover, it abstracts the disassembler's API to expose a clean interface to the users.

Quokka is heavily inspired by BinExport, the binary exporter used by BinDiff.

Installation

Python plugin

The plugin is built in the CI and available in the registry.

It should be possible to install directly from PIP using this kind of commmand:

$ pip install quokka-project

IDA Plugin

Note: The IDA plugin is not needed to read a Quokka generated file. It is only used to generate them.

Quokka is currently compatible with IDA 7.3+

The plugin is built on the CI and available in the Release tab.

To download the plugin, get the file named quokka_plugin**.so.

Usage

Export a file

!!! note

This requires a working IDA installation.

Either using command line:

$ idat64 -OQuokkaAuto:true -A /path/to/hello.i64

Note: We are using idat64 and not ida64 to increase the export speed because we don't need the graphical interface.

Using the plugin shortcut inside IDA: (by default) Alt+A

Load an export file

import quokka

# Directly from the binary (requires the IDA plugin to be installed)
ls = quokka.Program.from_binary("/bin/ls")

# From the exported file
ls = quokka.Program("ls.quokka",  # the exported file 
                    "/bin/ls")    # the original binary

Building

Build

user@host:~/quokka$ cmake -B build \ # Where to build
                          -S . \ # Where are the sources
                          -DIdaSdk_ROOT_DIR:STRING=path/to/ida_sdk \ # Path to IDA SDK 
                          -DCMAKE_BUILD_TYPE:STRING=Release \ # Build Type

user@host:~/quokka$ cmake --build build --target quokka_plugin -- -j

To install the plugin:

user@host:~/quokka$ cmake --install build

In any case, the plugin will also be in build/quokka-install. You can copy it to IDA's user plugin directory.

user@host:~/quokka$ cp build/quokka-install/quokka*64.so $HOME/.idapro/plugins/

For more detailed information about building, see Building

Documentation

Documentation is available online at documentation

FAQ

You can see a list of questions here FAQ

quokka's People

Stargazers

Watchers

Forkers

gmh5225 peiga clayne icodein patacca wideglide dominikbayerl ognz 0xsunsama bwry

quokka's Issues

Error: "Unable to find the chunk requested because its a super chunk" on quokka file loading.

Hi !

I am encountering an issue when load quokka files generated from kernel modules. The export seems to be performed correctly but the loading fails with the exception:

Unable to find the chunk requested because its a super chunk

Here is a simple modules to reproduce: pktflow.ko.zip

from quokka import Program

p = Program("pktflow.ko.Quokka", "pktflow.ko")

Tested with:

IDA Pro 8.2
Quokka: 0.5.4

Align on the caching design choice

We should enforce on the python API the same deisgn choices over caching/uncaching that we used on python-binexport/qbindiff.

The design choices can be summarized in the following points:

Function objects are eagerly loaded in the constructor of Program and never deallocated
BasicBlock objects are not cached by default. In order to access them in a cached way there are two ways: either with the context manager (with statement) or by manually calling Function.preload() and Function.unload() methods.
Everything below basic block will be cached by default (Instruction, Operands, ...)

Dependencies too restrictives

Quokka setup.py defines the following dependencies:

capstone>=4.0.2,<5,
networkx>=2.4,<3,
protobuf>=3.12.2,<4,
pypcode>=1.1.1,<2,

As capstone 5.0.0 is now released, it makes quokka incompatible with latest capstone version. I am wondering why we purposely restricted the maximum version. I suggest:

removing max version enforcement (unless truly incompatible)
removing pypcode as a mandatory dependency. It brings a lot of additional dependencies and is not used in most Quokka usages. Thus I'd suggest moving it as extra_require, and importing it when being used (and failing nicely if it is not installed).

Redesign the architecture recognition module in the python bindings

As it is right now the whole architecture handling code that mostly resides in the analysis folder is rarely (if not never) being used in other parts of the python bindings and it is not easily extendible with other architectures.
There are some of the issues with its current implementation:

The base QuokkaArch class contains some fields that are either not common in all architectures (there is no inst_pointer register in ppc) or not easily defined (like compared_mnemonics)
We aim to use QuokkaArch as an interface to a generic architecture but at the same time we are building the various architecture specific subclasses directly from capstone, so it doesn't hold much additional information.
Imho it makes more sense to get the calling convention from the binary itself instead of defining them per system/architecture
The Enviornment and Replacer classes are never being used

As things are right now it's not really working as intended, either we drop it and just rely on capstone architecture or we fix it to have a proper interface to a generic architecture

Move the IDA SDKs out of the repository

In order to make the repository lighter it would be better to move the IDA SDKs out of the repository.
The SDKs are encrypted and used only for tests in the github CI, so they are not useful to the users.

Possible solutions are:

Using git LFS
using another repository

Quokka does not support Python 3.12 (because of capstone)

Due to a problem in capstone, quokka installation fails with Python 3.12 and 3.11.

In Python 3.12

➜ python --version
Python 3.12.2
➜ pip install quokka-project
+ quokka-project==0.5.5
➜ python -c 'import quokka'  
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./python312/lib/python3.12/site-packages/quokka/__init__.py", line 24, in <module>
    import quokka.analysis
  File "./python312/lib/python3.12/site-packages/quokka/analysis/__init__.py", line 19, in <module>
    from quokka.analysis.arch import (
  File "./python312/lib/python3.12/site-packages/quokka/analysis/arch.py", line 19, in <module>
    import capstone
  File "./python312/lib/python3.12/site-packages/capstone/__init__.py", line 379, in <module>
    import distutils.sysconfig
ModuleNotFoundError: No module named 'distutils'

This is resolved in capstone-engine/capstone@d63211e but not yet available on PyPi.

With Python 3.11

➜ python -c 'import quokka'   
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./capstone/python311/lib/python3.11/site-packages/quokka/__init__.py", line 24, in <module>
    import quokka.analysis
  File "./capstone/python311/lib/python3.11/site-packages/quokka/analysis/__init__.py", line 19, in <module>
    from quokka.analysis.arch import (
  File "./capstone/python311/lib/python3.11/site-packages/quokka/analysis/arch.py", line 19, in <module>
    import capstone
  File "./python311/lib/python3.11/site-packages/capstone/__init__.py", line 380, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

This is solved by installing setuptools.

pip install setuptools

However, this will also fail with the following error on Mac Mx:

➜ python -c 'import quokka'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./python311/lib/python3.11/site-packages/quokka/__init__.py", line 24, in <module>
    import quokka.analysis
  File "./python311/lib/python3.11/site-packages/quokka/analysis/__init__.py", line 19, in <module>
    from quokka.analysis.arch import (
  File "./python311/lib/python3.11/site-packages/quokka/analysis/arch.py", line 19, in <module>
    import capstone
  File "./capstone/python311/lib/python3.11/site-packages/capstone/__init__.py", line 425, in <module>
    _cs = _load_lib(_path)
          ^^^^^^^^^^^^^^^^
  File "./capstone/python311/lib/python3.11/site-packages/capstone/__init__.py", line 398, in _load_lib
    return ctypes.cdll.LoadLibrary(lib_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(./python311/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib, 0x0006): tried: './python311/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')),  './python311/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))

This can be solved by compiling the capstone bindings ourselves:

gh repo clone capstone-engine/capstone
cd capstone
./make.sh
cp ./libcapstone.5.dylib .venv/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib

Finally, this work

➜ python -c 'import quokka' && echo "GOOD"
GOOD

Of note, this last issue is fixed in : capstone-engine/capstone#2066

Please release the ida plugin for macOS(x64/arm64)

quokka exporting data references from non-exported segments (src/segment.cpp)

I'm encountering an issue when using Quokka v0.5.5 (a3de4c3) with quarkslab/qbindiff@69bed0a. Essentially the symptom is that Quokka bails when trying to access a reference to a segment that wasn't exported. The backtrace can be found at the end of this issue.

Specifically, the following code from Quokka iterates through all the segments in a database and collects them into an array.

quokka/src/Segment.cpp

Lines 79 to 100 in a3de4c3

 int ExportSegments(quokka::Quokka* proto) { 

 Timer timer(absl::Now()); 

 QLOG_INFO << "Start to export segments"; 

 std::vector<Segment> segments; 

 segments.reserve(get_segm_qty()); 

 segment_t* seg = get_first_seg(); 

 while (seg != nullptr) { 

 if (is_visible_segm(seg) && not is_ephemeral_segm(seg->start_ea)) { 

 segments.emplace_back(seg); 

 } 

 seg = get_next_seg(seg->start_ea); 

 } 

 WriteSegments(proto, segments); 

 QLOG_INFO << absl::StrFormat("Segments exported (took %.2fs)", 

 timer.ElapsedSeconds(absl::Now())); 

 return eOk; 

 }

Segments are only collected by this ExportSegments function if they are "visible" (is_visible_segm) and not "ephemeral" (is_ephemeral_segm). However, header segments (SFL_HEADER) that are loaded by the PE loader can be both "ephemeral" (SFL_LOADER|SFL_DEBUG) and referenced by an instruction (perhaps other formats too, as it would depend on the loader used to build the database).

So, if a database containing a segment with both the SFL_HEADER and SFL_LOADER flags is exported by Quokka and a user of that exported data attempts to enumerate instructions that reference an SFL_HEADER segment, the quokka.Program.get_segment method will raise an exception when trying to call quokka.Segment.in_segment. The exception it raises is uncaught by qbindiff which'll cause qbindiff to abort when it tries to gather references with qbindiff.loader.backend.InstructionBackendQuokka._cast_references.

I temporarily fixed the fragility of qbindiff by catching the exception "inside" this loop, but the proper fix would be to include is_header_segm as one of the checks in ExportSegments from quokka. To reproduce this, you might be able to build a database for a portable executable file which should load a segment for the PE header labeled as "HEADER". If you examine the segment_t.flags for that "HEADER" segment, it should have both the SFL_LOADER and SFL_HEADER flag set.

    def _cast_references(
        self, references: list[quokka.types.ReferenceTarget]
    ) -> list[ReferenceTarget]:
...
        ret_ref: list[ReferenceTarget] = []
        for ref in references:
            match ref:
                case quokka.data.Data():
...
                    try:
                        value = ref.value
                    except Exception:
                        logging.warning("Skipping missing reference for address {:#x}".format(ref.address), exc_info=True)
                    else:
                        ret_ref.append(Data(data_type, ref.address, value))

This following is the uncaught exception in qbindiff being raised by Quokka. It's also probably worth formatting the address that is being raised by quokka/program.py:325 as hexadecimal using "{address:#x}" too, so that it's not being emitted in decimal.

Traceback (most recent call last):                                                        
  File "/usr/lib/python3.12/site-packages/quokka/addresser.py", line 65, in file
    segment = self.program.get_segment(offset)                   
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                            
  File "/usr/lib/python3.12/site-packages/quokka/program.py", line 325, in get_segment
    raise KeyError(f"No segment has been found for address 0x{address}")                                                                                                            
KeyError: 'No segment has been found for address 0x268435968'                             
                                                                                          
The above exception was the direct cause of the following exception:

Traceback (most recent call last):                                                        
  File "/home/user/.local/bin/qbindiff", line 8, in <module>                                                                                                                        
    sys.exit(main())                                                                      
...
  File "/usr/lib/python3.12/site-packages/qbindiff/visitor.py", line 263, in visit_instruction
    callback(program, instruction, collector) 
  File "/usr/lib/python3.12/site-packages/qbindiff/features/artefact.py", line 72, in visit_instruction
    for ref_type, references in instruction.references.items():
                                ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/qbindiff/loader/instruction.py", line 65, in references
    return self._backend.references
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/qbindiff/loader/backend/quokka.py", line 277, in references
    ref[convert_ref_type(ref_type)] = self._cast_references(references)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/qbindiff/loader/backend/quokka.py", line 227, in _cast_references
    ret_ref.append(Data(data_type, ref.address, ref.value))
                                                ^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/quokka/data.py", line 99, in value
    address = self.program.addresser.file(self.address)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/quokka/addresser.py", line 67, in file
    raise quokka.NotInFileError("Unable to find the segment") from exc
quokka.exc.NotInFileError: Unable to find the segment

support for 32-bit addressing on posix platforms

diff --git a/cmake/FindIdaSdk.cmake b/cmake/FindIdaSdk.cmake
index 4d3d9d7..4377020 100644
--- a/cmake/FindIdaSdk.cmake
+++ b/cmake/FindIdaSdk.cmake
@@ -52,6 +52,8 @@ find_path(IdaSdk_DIR NAMES include/pro.h
 set(IdaSdk_INCLUDE_DIRS ${IdaSdk_DIR}/include)
 set(IdaSdk_MODULE_DIRS ${IdaSdk_DIR}/module)
 
+set(EA64 ON CACHE BOOL "64-bits addressing")
+
 find_package_handle_standard_args(
         IdaSdk FOUND_VAR IdaSdk_FOUND
         REQUIRED_VARS IdaSdk_DIR
@@ -71,7 +73,12 @@ else ()
 endif ()
 
 if (UNIX)
-    set(IdaLib ${IdaSdk_DIR}/lib/x64_linux_gcc_64/libida64.so)
+    if (${EA64})
+        set(IdaLib ${IdaSdk_DIR}/lib/x64_linux_gcc_64/libida64.so)
+    else ()
+        set(IdaLib ${IdaSdk_DIR}/lib/x64_linux_gcc_32/libida.so)
+    endif ()
+
     if (APPLE)
         set(IdaSdk_PLATFORM __MAC__)
     else ()
@@ -108,10 +115,15 @@ endif ()
 function(ida_common_target_settings t)
     # Add the necessary __IDP__ define and allow to use "dangerous" and standard
     # file functions.
-    target_compile_definitions(${t} PUBLIC
-            ${IdaSdk_PLATFORM} __X64__ __IDP__ USE_DANGEROUS_FUNCTIONS
-            USE_STANDARD_FILE_FUNCTIONS __EA64__)
-
+    if (${EA64})
+        target_compile_definitions(${t} PUBLIC
+                ${IdaSdk_PLATFORM} __X64__ __IDP__ USE_DANGEROUS_FUNCTIONS
+                USE_STANDARD_FILE_FUNCTIONS __EA64__)
+    else ()
+        target_compile_definitions(${t} PUBLIC
+                ${IdaSdk_PLATFORM} __X64__ __IDP__ USE_DANGEROUS_FUNCTIONS
+                USE_STANDARD_FILE_FUNCTIONS)
+    endif ()
     target_include_directories(${t} PUBLIC ${IdaSdk_INCLUDE_DIRS})
 endfunction()
 
@@ -121,9 +133,15 @@ function(_ida_plugin name link_script)  # ARGN contains sources
     ida_common_target_settings(${name})
 
     # Rename the plugin to have the proper naming scheme for IDA
-    set_target_properties(${name} PROPERTIES
-            PREFIX ""
-            OUTPUT_NAME ${name}${PROJECT_VERSION_MAJOR}${PROJECT_VERSION_MINOR}64)
+    if (${EA64})
+        set_target_properties(${name} PROPERTIES
+                PREFIX ""
+                OUTPUT_NAME ${name}${PROJECT_VERSION_MAJOR}${PROJECT_VERSION_MINOR}64)
+    else ()
+        set_target_properties(${name} PROPERTIES
+                PREFIX ""
+                OUTPUT_NAME ${name}${PROJECT_VERSION_MAJOR}${PROJECT_VERSION_MINOR}32)
+    endif ()
 
     if (UNIX)
         if (APPLE)

	int ExportSegments(quokka::Quokka* proto) {
	Timer timer(absl::Now());
	QLOG_INFO << "Start to export segments";

	std::vector<Segment> segments;
	segments.reserve(get_segm_qty());

	segment_t* seg = get_first_seg();
	while (seg != nullptr) {
	if (is_visible_segm(seg) && not is_ephemeral_segm(seg->start_ea)) {
	segments.emplace_back(seg);
	}

	seg = get_next_seg(seg->start_ea);
	}

	WriteSegments(proto, segments);
	QLOG_INFO << absl::StrFormat("Segments exported (took %.2fs)",
	timer.ElapsedSeconds(absl::Now()));

	return eOk;
	}