quarkslab / quokka Goto Github PK
View Code? Open in Web Editor NEWQuokka: A Fast and Accurate Binary Exporter
Home Page: https://quarkslab.github.io/quokka/
License: Apache License 2.0
Quokka: A Fast and Accurate Binary Exporter
Home Page: https://quarkslab.github.io/quokka/
License: Apache License 2.0
Hi !
I am encountering an issue when load quokka files generated from kernel modules. The export seems to be performed correctly but the loading fails with the exception:
Unable to find the chunk requested because its a super chunk
Here is a simple modules to reproduce: pktflow.ko.zip
from quokka import Program
p = Program("pktflow.ko.Quokka", "pktflow.ko")
Tested with:
In order to make the repository lighter it would be better to move the IDA SDKs out of the repository.
The SDKs are encrypted and used only for tests in the github CI, so they are not useful to the users.
Possible solutions are:
We should enforce on the python API the same deisgn choices over caching/uncaching that we used on python-binexport/qbindiff.
The design choices can be summarized in the following points:
Function
objects are eagerly loaded in the constructor of Program
and never deallocatedBasicBlock
objects are not cached by default. In order to access them in a cached way there are two ways: either with the context manager (with
statement) or by manually calling Function.preload()
and Function.unload()
methods.Instruction
, Operands
, ...)As it is right now the whole architecture handling code that mostly resides in the analysis
folder is rarely (if not never) being used in other parts of the python bindings and it is not easily extendible with other architectures.
There are some of the issues with its current implementation:
QuokkaArch
class contains some fields that are either not common in all architectures (there is no inst_pointer
register in ppc) or not easily defined (like compared_mnemonics
)QuokkaArch
as an interface to a generic architecture but at the same time we are building the various architecture specific subclasses directly from capstone, so it doesn't hold much additional information.Enviornment
and Replacer
classes are never being usedAs things are right now it's not really working as intended, either we drop it and just rely on capstone architecture or we fix it to have a proper interface to a generic architecture
I'm encountering an issue when using Quokka v0.5.5 (a3de4c3) with quarkslab/qbindiff@69bed0a. Essentially the symptom is that Quokka bails when trying to access a reference to a segment that wasn't exported. The backtrace can be found at the end of this issue.
Specifically, the following code from Quokka iterates through all the segments in a database and collects them into an array.
Lines 79 to 100 in a3de4c3
Segments are only collected by this ExportSegments
function if they are "visible" (is_visible_segm
) and not "ephemeral" (is_ephemeral_segm
). However, header segments (SFL_HEADER
) that are loaded by the PE loader can be both "ephemeral" (SFL_LOADER|SFL_DEBUG
) and referenced by an instruction (perhaps other formats too, as it would depend on the loader used to build the database).
So, if a database containing a segment with both the SFL_HEADER
and SFL_LOADER
flags is exported by Quokka and a user of that exported data attempts to enumerate instructions that reference an SFL_HEADER
segment, the quokka.Program.get_segment
method will raise an exception when trying to call quokka.Segment.in_segment
. The exception it raises is uncaught by qbindiff
which'll cause qbindiff
to abort when it tries to gather references with qbindiff.loader.backend.InstructionBackendQuokka._cast_references
.
I temporarily fixed the fragility of qbindiff
by catching the exception "inside" this loop, but the proper fix would be to include is_header_segm
as one of the checks in ExportSegments
from quokka
. To reproduce this, you might be able to build a database for a portable executable file which should load a segment for the PE header labeled as "HEADER". If you examine the segment_t.flags
for that "HEADER" segment, it should have both the SFL_LOADER
and SFL_HEADER
flag set.
def _cast_references(
self, references: list[quokka.types.ReferenceTarget]
) -> list[ReferenceTarget]:
...
ret_ref: list[ReferenceTarget] = []
for ref in references:
match ref:
case quokka.data.Data():
...
try:
value = ref.value
except Exception:
logging.warning("Skipping missing reference for address {:#x}".format(ref.address), exc_info=True)
else:
ret_ref.append(Data(data_type, ref.address, value))
This following is the uncaught exception in qbindiff
being raised by Quokka. It's also probably worth formatting the address that is being raised by quokka/program.py:325
as hexadecimal using "{address:#x}
" too, so that it's not being emitted in decimal.
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/quokka/addresser.py", line 65, in file
segment = self.program.get_segment(offset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/quokka/program.py", line 325, in get_segment
raise KeyError(f"No segment has been found for address 0x{address}")
KeyError: 'No segment has been found for address 0x268435968'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/.local/bin/qbindiff", line 8, in <module>
sys.exit(main())
...
File "/usr/lib/python3.12/site-packages/qbindiff/visitor.py", line 263, in visit_instruction
callback(program, instruction, collector)
File "/usr/lib/python3.12/site-packages/qbindiff/features/artefact.py", line 72, in visit_instruction
for ref_type, references in instruction.references.items():
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/functools.py", line 995, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/qbindiff/loader/instruction.py", line 65, in references
return self._backend.references
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/functools.py", line 995, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/qbindiff/loader/backend/quokka.py", line 277, in references
ref[convert_ref_type(ref_type)] = self._cast_references(references)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/qbindiff/loader/backend/quokka.py", line 227, in _cast_references
ret_ref.append(Data(data_type, ref.address, ref.value))
^^^^^^^^^
File "/usr/lib/python3.12/site-packages/quokka/data.py", line 99, in value
address = self.program.addresser.file(self.address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/quokka/addresser.py", line 67, in file
raise quokka.NotInFileError("Unable to find the segment") from exc
quokka.exc.NotInFileError: Unable to find the segment
Quokka setup.py defines the following dependencies:
As capstone 5.0.0 is now released, it makes quokka incompatible with latest capstone version. I am wondering why we purposely restricted the maximum version. I suggest:
Due to a problem in capstone, quokka
installation fails with Python 3.12 and 3.11.
➜ python --version
Python 3.12.2
➜ pip install quokka-project
+ quokka-project==0.5.5
➜ python -c 'import quokka'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./python312/lib/python3.12/site-packages/quokka/__init__.py", line 24, in <module>
import quokka.analysis
File "./python312/lib/python3.12/site-packages/quokka/analysis/__init__.py", line 19, in <module>
from quokka.analysis.arch import (
File "./python312/lib/python3.12/site-packages/quokka/analysis/arch.py", line 19, in <module>
import capstone
File "./python312/lib/python3.12/site-packages/capstone/__init__.py", line 379, in <module>
import distutils.sysconfig
ModuleNotFoundError: No module named 'distutils'
This is resolved in capstone-engine/capstone@d63211e but not yet available on PyPi.
➜ python -c 'import quokka'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./capstone/python311/lib/python3.11/site-packages/quokka/__init__.py", line 24, in <module>
import quokka.analysis
File "./capstone/python311/lib/python3.11/site-packages/quokka/analysis/__init__.py", line 19, in <module>
from quokka.analysis.arch import (
File "./capstone/python311/lib/python3.11/site-packages/quokka/analysis/arch.py", line 19, in <module>
import capstone
File "./python311/lib/python3.11/site-packages/capstone/__init__.py", line 380, in <module>
import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
This is solved by installing setuptools
.
pip install setuptools
However, this will also fail with the following error on Mac Mx:
➜ python -c 'import quokka'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./python311/lib/python3.11/site-packages/quokka/__init__.py", line 24, in <module>
import quokka.analysis
File "./python311/lib/python3.11/site-packages/quokka/analysis/__init__.py", line 19, in <module>
from quokka.analysis.arch import (
File "./python311/lib/python3.11/site-packages/quokka/analysis/arch.py", line 19, in <module>
import capstone
File "./capstone/python311/lib/python3.11/site-packages/capstone/__init__.py", line 425, in <module>
_cs = _load_lib(_path)
^^^^^^^^^^^^^^^^
File "./capstone/python311/lib/python3.11/site-packages/capstone/__init__.py", line 398, in _load_lib
return ctypes.cdll.LoadLibrary(lib_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
return self._dlltype(name)
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(./python311/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib, 0x0006): tried: './python311/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), './python311/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))
This can be solved by compiling the capstone bindings ourselves:
gh repo clone capstone-engine/capstone
cd capstone
./make.sh
cp ./libcapstone.5.dylib .venv/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib
Finally, this work
➜ python -c 'import quokka' && echo "GOOD"
GOOD
Of note, this last issue is fixed in : capstone-engine/capstone#2066
diff --git a/cmake/FindIdaSdk.cmake b/cmake/FindIdaSdk.cmake
index 4d3d9d7..4377020 100644
--- a/cmake/FindIdaSdk.cmake
+++ b/cmake/FindIdaSdk.cmake
@@ -52,6 +52,8 @@ find_path(IdaSdk_DIR NAMES include/pro.h
set(IdaSdk_INCLUDE_DIRS ${IdaSdk_DIR}/include)
set(IdaSdk_MODULE_DIRS ${IdaSdk_DIR}/module)
+set(EA64 ON CACHE BOOL "64-bits addressing")
+
find_package_handle_standard_args(
IdaSdk FOUND_VAR IdaSdk_FOUND
REQUIRED_VARS IdaSdk_DIR
@@ -71,7 +73,12 @@ else ()
endif ()
if (UNIX)
- set(IdaLib ${IdaSdk_DIR}/lib/x64_linux_gcc_64/libida64.so)
+ if (${EA64})
+ set(IdaLib ${IdaSdk_DIR}/lib/x64_linux_gcc_64/libida64.so)
+ else ()
+ set(IdaLib ${IdaSdk_DIR}/lib/x64_linux_gcc_32/libida.so)
+ endif ()
+
if (APPLE)
set(IdaSdk_PLATFORM __MAC__)
else ()
@@ -108,10 +115,15 @@ endif ()
function(ida_common_target_settings t)
# Add the necessary __IDP__ define and allow to use "dangerous" and standard
# file functions.
- target_compile_definitions(${t} PUBLIC
- ${IdaSdk_PLATFORM} __X64__ __IDP__ USE_DANGEROUS_FUNCTIONS
- USE_STANDARD_FILE_FUNCTIONS __EA64__)
-
+ if (${EA64})
+ target_compile_definitions(${t} PUBLIC
+ ${IdaSdk_PLATFORM} __X64__ __IDP__ USE_DANGEROUS_FUNCTIONS
+ USE_STANDARD_FILE_FUNCTIONS __EA64__)
+ else ()
+ target_compile_definitions(${t} PUBLIC
+ ${IdaSdk_PLATFORM} __X64__ __IDP__ USE_DANGEROUS_FUNCTIONS
+ USE_STANDARD_FILE_FUNCTIONS)
+ endif ()
target_include_directories(${t} PUBLIC ${IdaSdk_INCLUDE_DIRS})
endfunction()
@@ -121,9 +133,15 @@ function(_ida_plugin name link_script) # ARGN contains sources
ida_common_target_settings(${name})
# Rename the plugin to have the proper naming scheme for IDA
- set_target_properties(${name} PROPERTIES
- PREFIX ""
- OUTPUT_NAME ${name}${PROJECT_VERSION_MAJOR}${PROJECT_VERSION_MINOR}64)
+ if (${EA64})
+ set_target_properties(${name} PROPERTIES
+ PREFIX ""
+ OUTPUT_NAME ${name}${PROJECT_VERSION_MAJOR}${PROJECT_VERSION_MINOR}64)
+ else ()
+ set_target_properties(${name} PROPERTIES
+ PREFIX ""
+ OUTPUT_NAME ${name}${PROJECT_VERSION_MAJOR}${PROJECT_VERSION_MINOR}32)
+ endif ()
if (UNIX)
if (APPLE)
Hello,
When I tried to access the corresponding pcode instructions of a block, I encounter the following error :
>>> import quokka
>>> p = quokka.Program('x86-gcc-7-O0_fips.so.Quokka', 'x86-gcc-7-O0_fips.so')
>>> for ffa, ff in p.items():
... for (addr, bb) in ff.blocks.items():
... print(bb.pcode_insts)
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/usr/lib/python3.11/functools.py", line 1001, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/rcohen/Documents/ressources/experiments/current/sog/HermesSim/binaries/Dataset-1/openssl/venv/lib/python3.11/site-packages/quokka/block.py", line 263, in pcode_insts
return pypcode_decode_block(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rcohen/Documents/ressources/experiments/current/sog/HermesSim/binaries/Dataset-1/openssl/venv/lib/python3.11/site-packages/quokka/backends/pypcode.py", line 142, in pypcode_decode_block
first_instruction: Optional[quokka.Instruction] = next(block.instructions, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'ValuesView' object is not an iterator
I'm using the latest version of Quokka (0.5.5) and the version 2.1.0 of pypcode. Do you have any idea in order to solve the problem ?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.