jarryshaw / pypcapkit Goto Github PK

Python-based Comprehensive Network Packet Analysis Library

Home Page: https://jarryshaw.github.io/PyPCAPKit/

License: BSD 3-Clause "New" or "Revised" License

Python 99.93% Makefile 0.06% Shell 0.01% Batchfile 0.01%

computer-networking pcap-analyzer pcap-parser network network-security network-tools packet-analyser packet-analysis packet-analyzer packet-crafting

pypcapkit's Introduction

PyPCAPKit - Comprehensive Network Packet Analysis Library

For any technical and/or maintenance information, please kindly refer to the Official Documentation_.

The PyPCAPKit project is an open source Python program focus on network packet parsing and analysis, which works as a comprehensive PCAP file extraction, construction and analysis library.

The whole project supports Python 3.6 or later.

About

PyPCAPKit is a comprehensive Python-native network packet analysis library, with DictDumper as its formatted output dumper.

Unlike popular PCAP file extractors, such as Scapy, DPKT, PyShark, and etc, pcapkit is designed to be much more comprehensive, which means it is able to provide more detailed information about the packet, as well as a more Pythonic interface for users to interact with.

Module Structure

In pcapkit, all files can be described as following eight parts.

Interface (pcapkit.interface)

User interface for the pcapkit library, which standardises and simplifies the usage of this library.
Foundation (pcapkit.foundation)

Synthesises file I/O and protocol analysis, coordinates information exchange in all network layers, as well as provides the foundamental functions for pcapkit.
Protocols (pcapkit.protocols)

Collection of all protocol family, with detailed implementation and methods.
Utilities (pcapkit.utilities)

Auxiliary functions and tools for pcapkit.
CoreKit (pcapkit.corekit)

Core utilities for pcapkit implementation, mainly for internal data structure and processing.
ToolKit (pcapkit.toolkit)

Auxiliary tools for pcapkit to support the multiple extraction engines with a unified interface.
DumpKit (pcapkit.dumpkit)

File output formatters for pcapkit.
Constants (pcapkit.const)

Constant enumerations used in pcapkit for protocol family extraction and representation.

Engine Comparison

Due to the general overhead of pcapkit, its extraction procedure takes around 0.2 milliseconds per packet, which is already impressive but not enough comparing to other popular extration engines availbale on the market, given the fact that pcapkit is a comprehensive packet processing module.

Additionally, pcapkit introduced alternative extractionengines to accelerate this procedure. By now pcapkit supports Scapy, DPKT, and PyShark.

Test Environment

Operating System	macOS Ventura 13.4.1
Chip	Apple M2 Pro
Memory	16 GB

Test Results

Engine	Performance (ms per packet)
`dpkt`	0.010390_056723
`scapy`	0.091690_233567
`pcapkit`	0.200390_390390
`pyshark`	24.682185_018351

Installation

Note

pcapkit supports Python versions since 3.6.

Simply run the following to install the current version from PyPI:

pip install pypcapkit

Or install the latest version from the gi repository:

git clone https://github.com/JarryShaw/PyPCAPKit.git
cd pypcapkit
pip install -e .
# and to update at any time
git pull

And since pcapkit supports various extraction engines, and extensive plug-in functions, you may want to install the optional ones:

# for DPKT only
pip install pypcapkit[DPKT]
# for Scapy only
pip install pypcapkit[Scapy]
# for PyShark only
pip install pypcapkit[PyShark]
# and to install all the optional packages
pip install pypcapkit[all]
# or to do this explicitly
pip install pypcapkit dpkt scapy pyshark

For CLI usage, you will need to install the optional packages:

pip install pypcapkit[cli]
# or explicitly...
pip install pypcapkit emoji

pypcapkit's People

Contributors

Stargazers

Watchers

pypcapkit's Issues

Invalid .pyup.yml detected

The bot encountered an error in your .pyup.yml config file:

  in "<unicode string>", line 9, column 13:
          update: False
                ^

You can validate it with this online YAML parser or by taking a look at the Documentation.

InfoClass instanciation calls `os.stat`, ruining performance

(thanks for having written the best PCAP parsing library out there !)

The bug

Problem: pcap parsing is slow, it takes 1s to parse a single .pcap file with 1 packet.

When parsing packet layers (our use case is TCP reassembly with pcapkit.extract().reassembly.tcp), InfoClass objects (Info) are instanciated quite a large number of times, as expected.

In the initialization code of said classes ( https://github.com/JarryShaw/PyPCAPKit/blob/master/src/corekit/infoclass.py#L55 ) , a call to the validations.dict_check is usually performed.

validations.dict_check itself calls inspect.stack() (here: https://github.com/JarryShaw/PyPCAPKit/blob/master/src/utilities/validations.py#L128 ) , and here lies the problem : inspect.stack() performs causes os.stat to be called.

See the bottom of the following trace graph,

python3 -m cProfile -o /tmp/parse_pcap.pstats ./my-script.py
gprof2dot -f pstats /tmp/parse_pcap.pstats | dot -Tpng -o /tmp/parse_pcap.png

System information
A clear and concise description of your system information.

OS Version:CYGWIN_NT-10.0 3.0.7(0.338/5/3) 2019-04-30 18:08 unknown unknown Cygwin (uname -srvpio)
Python Version: 3.6.8
Python Implementation: CPython [GCC 7.4.0] on cygwin

Traceback stack
Run program again with PCAPKIT_DEVMODE=true set to provide the traceback stack.

It's not a crash, but to reproduce this I just had to ctrl-C since 90% of the code execution is spent there :)

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/cProfile.py", line 161, in <module>
    main()
  File "/usr/lib/python3.6/cProfile.py", line 154, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib/python3.6/cProfile.py", line 20, in runctx
    filename, sort)
  File "/usr/lib/python3.6/profile.py", line 64, in runctx
    prof.runctx(statement, globals, locals)
  File "/usr/lib/python3.6/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)

  File "redacted.py", line 316, in operate
    p = pcap.PCAP(filename = self.sf.get_pcap_path(query))
  File "redacted/pcap.py", line 90, in __init__
    'tcp': True,
  File "/usr/lib/python3.6/site-packages/pcapkit/interface/__init__.py", line 131, in extract
    trace_byteorder=trace_byteorder, trace_nanosecond=trace_nanosecond)
  File "/usr/lib/python3.6/site-packages/pcapkit/foundation/extraction.py", line 533, in __init__
    self.run()                      # start extraction
  File "/usr/lib/python3.6/site-packages/pcapkit/foundation/extraction.py", line 261, in run
    self.record_frames()            # read frames
  File "/usr/lib/python3.6/site-packages/pcapkit/foundation/extraction.py", line 376, in record_frames
    self._read_frame()
  File "/usr/lib/python3.6/site-packages/pcapkit/foundation/extraction.py", line 612, in _read_frame
    return self._default_read_frame()
  File "/usr/lib/python3.6/site-packages/pcapkit/foundation/extraction.py", line 629, in _default_read_frame
    layer=self._exlyr, protocol=self._exptl, nanosecond=self._nnsec)
  File "/usr/lib/python3.6/site-packages/pcapkit/protocols/pcap/frame.py", line 158, in __init__
    self._info = Info(self.read_frame())
  File "/usr/lib/python3.6/site-packages/pcapkit/corekit/infoclass.py", line 56, in __new__
    self.__dict__.update(__read__(dict_))
  File "/usr/lib/python3.6/site-packages/pcapkit/corekit/infoclass.py", line 38, in __read__
    __dict__[key] = Info(value)
  File "/usr/lib/python3.6/site-packages/pcapkit/corekit/infoclass.py", line 55, in __new__
    dict_check(dict_)
  File "/usr/lib/python3.6/site-packages/pcapkit/utilities/validations.py", line 128, in dict_check
    func = func or inspect.stack()[2][3]
  File "/usr/lib/python3.6/inspect.py", line 1501, in stack
    return getouterframes(sys._getframe(1), context)
  File "/usr/lib/python3.6/inspect.py", line 1478, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/usr/lib/python3.6/inspect.py", line 1452, in getframeinfo
    lines, lnum = findsource(frame)
  File "/usr/lib/python3.6/inspect.py", line 768, in findsource
    file = getsourcefile(object)
  File "/usr/lib/python3.6/inspect.py", line 693, in getsourcefile
    if os.path.exists(filename):
  File "/usr/lib/python3.6/genericpath.py", line 19, in exists
    os.stat(path)
KeyboardInterrupt
> /usr/lib/python3.6/genericpath.py(19)exists()
-> os.stat(path)
(Pdb)

Expected behavior

Ideally, pcapkit would not call posix.stat so much because it would chose not to rely on inspect features. That would make it significantly faster, probably ?

But I guess there's a reason for inspect to be used here, as well as for the whole https://github.com/JarryShaw/PyPCAPKit/blob/master/src/utilities/validations.py file. And now that I see https://github.com/JarryShaw/PyPCAPKit/blob/master/src/utilities/validations.py#L10 , I'm starting to think you foresaw this potential problem.

May I suggest moving func = func or inspect.stack()[2][3] after the inner check of each validation ? So that it's only called if the check failed. (Otherwise func seems not to be used, but I might be wrong)

Thanks for reading, and thanks again for the (awesome) lib !

ipv4_reassembly

Describe the bug
I' reading packets with dpkt like this:

import dpkt
import pcapkit
from pcapkit.reassembly import IPv4_Reassembly
ipv4_reassembly = IPv4_Reassembly()
f = open(r'test.pcapng','rb')
pcap = dpkt.pcapng.Reader(f)
buffer = []
for timestamp, buf in pcap:
    eth = dpkt.ethernet.Ethernet(buf)
    if isinstance(eth.data, dpkt.ip.IP):
        ip = eth.data
        if ip.p == dpkt.ip.IP_PROTO_UDP:
            udp = ip.data
            frag, defrag = pcapkit.toolkit.dpkt.ipv4_reassembly(eth)
            if frag:
                buffer = ipv4_reassembly(defrag)

System information

OS Version: [Windows 10]
Python Version: [3.8,7]
Python Implementation: [CPython]

Traceback stack
Run program again with PCAPKIT_DEVMODE=true set to provide the traceback stack.

pcapkit.utilities.exceptions.IntError: Function call expected integral number, NotImplementedType got instead.
During handling of the above exception, another exception occurred:
pcapkit.utilities.exceptions.FragmentError: 'Malformed fragment object: Function call expected integral number, NotImplementedType got instead.'

With

buffer = []
for timestamp, buf in pcap:
    eth = dpkt.ethernet.Ethernet(buf)
    if isinstance(eth.data, dpkt.ip.IP):
        ip = eth.data
        if ip.p == dpkt.ip.IP_PROTO_UDP:
            udp = ip.data
            frag, defrag = pcapkit.toolkit.dpkt.ipv4_reassembly(eth)
            if frag:
                defrag['num'] = 0
                buffer = ipv4_reassembly(defrag)

I got:
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "", line 10, in
UnboundLocalError: local variable 'TDL' referenced before assignment

Expected behavior
Defragmented packet(s).

Comma "," appears when parsing from pcap to JSON

Hi,
IN a Python3 program, I parse multiple pcap file in JSON using :

for ip in ip_list:
pcap_to_json = pcapkit.extract(fin=(ip+".pcap"), fout=(ip+"out.json"), format='json', extension=False)

Describe the bug
After the first file (10.10.0.51 in annexe) all the JSON files have a "," added at the second line which is not supposed to happen.

Expected behavior
no comma in the second line of JSON files

Additional context
Here is a zip containing examples:
scan_21-05-2020_15-14-13.zip

System information

OS Version: Ubuntu 20.04
Python Version: 3.8

Thank you for your help !

data.tcp_reassembled_data() returns stupid form of string instead of normal bytes.

Describe the bug
data.tcp_reassembled_data() returns completly antihuman form of string.

System information
A clear and concise description of your system information.

OS Version: Windows 10 19536.1000 64-bit
Python Version: 3.8.1
Python Implementation: Cpython

Expected behavior
b'\x41\x41\x41\x41\x41\x41\x41\x41'

Additional context
Format of '41:41:41:41:41:41:41:41' is much harder to process than normal bytes.

C Arrays Format Parser

Hello,
It will be nice if the module could parse as in "C Arrays" format from a TCP / UDP Stream (as Wireshark give this feature in the GUI version, adding photos).

https://imgur.com/a/rpEnkLB

Thank you for your help :)

Defragmentaion of IPv4/IPv6 Pakets..

Describe the bug
Not working defragmentation of IP Pakets...

System information

OS Version: Windows 10
Python Version: 3.8
Python Implementation: CPython

Traceback stack

testpacks = pcapkit.foundation.extraction.Extractor(r'test_ipv4.pcap',engine=pcapkit.DPKT,ipv4=True,verbose=True,nofile=True,store=False)
 - Frame   1: Ethernet:IP:UDP
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
UnboundLocalError: local variable 'TDL' referenced before assignment

For ipv6:
testpacks = pcapkit.foundation.extraction.Extractor(r'ipv6_test.pcap',engine=pcapkit.DPKT,ipv6=True,verbose=True,nofile=True,store=False) 
 - Frame   1: Ethernet:IP6:UDP
 - Frame   2: Ethernet:IP6:UDP
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
AttributeError: 'IP6FragmentHeader' object has no attribute 'nh'


```**Expected behavior**
dict with defragmented packets

Just a question

I just have a question, is it possible to use this to parse domains out of PCAP data?

object_hook() missing 1 required positional argument

Describe the bug
Function object_hook() requires a positional argument

System information

OS Version: ubuntu 19.04
Python Version: 3.7
Python Implementation: pip

Traceback stack

$ pcapkit-cli file.pcap --format json
Traceback (most recent call last):
  File "/home/me/test/venv/bin/pcapkit-cli", line 8, in <module>
    sys.exit(main())
TypeError: object_hook() missing 1 required positional argument: 'o'

Expected behavior
Packet in JSON format

Question: how to process realtime pcap stream

Thanks for fixing #155.

I want to capture http traffic from a pod in kubernetes. For this I am running tcpdump in the pod and I am receiving the live pcap stream as stdout on my local computer and want to pipe it to a python script that will log the http requests and responses.

Is it possible PyPCAPKit will be reading pcap from stdin or a pipe and fire a callback each time a reassembled packet is found?

Follow TCP stream

It would be better to have a functionality to dump all the TCP streams in a PCAP file (similar to Wireshark's "Follow TCP stream"), which presents users with a set of conversations (TCP payloads). For better performance, streams could be dumped all at once instead of one by one.

pcapkit.extraction not working

Describe the bug
A clear and concise description of what the bug is.
Attempting to follow the test file "test_extraction.py" and receiving the error ValueError: too many values to unpack (expected 2)

System information
A clear and concise description of your system information.

OS Version: Windows 10 19045.3324
Python Version: 3.10
Python Implementation: pycharm

Traceback stack

Traceback (most recent call last):
File "C:\Users----\PycharmProjects\----\----\test.py", line 8, in
pcapkit.extract('in.pcap', 'out.json', 'json')
ValueError: too many values to unpack (expected 2)

Note the ---- are to remove PI from file path

Expected behavior
Upon running this test program I expect the provided sample pcap file to be extracted to JSON as it is shown in the "sample" directory in GitHub

Additional context
Add any other context about the problem here.

Import Error

Import error in python 3.8 for latest version of pcapkit

Traceback (most recent call last):

File "C:\Users\MPN8KOR\Documents\venv_venv3.8\Lib\site-packages\pcapkit\untitled2.py", line 1, in
import pcapkit

File "C:\Users\MPN8KOR\Documents\venv_venv3.8\lib\site-packages\pcapkit_init_.py", line 83, in
from pcapkit.foundation.registry import *

File "C:\Users\MPN8KOR\Documents\venv_venv3.8\lib\site-packages\pcapkit\foundation_init_.py", line 13, in
from pcapkit.foundation.extraction import Extractor

File "C:\Users\MPN8KOR\Documents\venv_venv3.8\lib\site-packages\pcapkit\foundation\extraction.py", line 22, in
from pcapkit.corekit.infoclass import Info

File "C:\Users\MPN8KOR\Documents\venv_venv3.8\lib\site-packages\pcapkit\corekit_init_.py", line 15, in
from pcapkit.corekit.infoclass import Info

File "C:\Users\MPN8KOR\Documents\venv_venv3.8\lib\site-packages\pcapkit\corekit\infoclass.py", line 25, in
class Info(collections.abc.Mapping[str, VT], Generic[VT]):

TypeError: 'ABCMeta' object is not subscriptable

`follow_tcp_stream` (and flow tracing in general) doesn't work

Describe the bug

def main():
    try:
        streams = follow_tcp_stream(fin="test.pcap")
    except Exception as e:
        traceback.print_exc()
        return

Traceback (most recent call last):
  File "c:\Users\tobyx\Desktop\prox\main.py", line 51, in main
    streams = follow_tcp_stream(fin="test.pcap")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\interface\misc.py", line 80, in follow_tcp_stream
    extraction = Extractor(fin=fin, fout=None, format=None, auto=True, extension=extension,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\extraction.py", line 812, in __init__
    self.run()    # start extraction
    ^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\extraction.py", line 448, in run
    self.record_frames()
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\extraction.py", line 599, in record_frames
    self._exeng.read_frame()
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\engines\pcap.py", line 175, in read_frame
    ext._trace.tcp(data_tf_tcp)
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\traceflow\traceflow.py", line 336, in __call__
    self.dump(packet)
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\traceflow\tcp.py", line 59, in dump
    output = self.trace(packet, output=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\foundation\traceflow\tcp.py", line 112, in trace
    fpout=self._foutio(fname=f'{self._fproot}/{label}{self._fdpext or ""}', protocol=packet.protocol,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\dumpkit\pcap.py", line 70, in __init__
    super().__init__(fname, protocol=protocol, byteorder=byteorder, nanosecond=nanosecond, **kwargs)
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\dictdumper\dumper.py", line 168, in __init__
    self._dump_header(**kwargs)  # initialise output file
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\dumpkit\pcap.py", line 103, in _dump_header
    packet = Header(
             ^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\protocols\protocol.py", line 520, in __init__
    self.__post_init__(file, length, **kwargs)  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\protocols\misc\pcap\header.py", line 261, in __post_init__
    _data = self.pack(**kwargs)
            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\protocols\protocol.py", line 259, in pack
    return self.__header__.pack(packet)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\protocols\schema\schema.py", line 544, in pack
    temp = field.pack(value, packet)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tobyx\Desktop\prox\venv\Lib\site-packages\pcapkit\corekit\fields\field.py", line 185, in pack
    return struct.pack(self.template, pre_processed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: argument for 's' must be a bytes object

System information
A clear and concise description of your system information.

OS Version: Windows 11
Python Version: 3.11.6 (tried 3.9, same)
Python Implementation: CPython

Traceback stack
Already did

Expected behavior
Doesn't crash and performs as expected.

Additional context
N/A

packet.analyse() fails: UnicodeDecode error

Describe the bug
Showing error 'charmap' codec can't decode byte 0x81

Environment information
Python: 3.8.0

Traceback stack
Traceback (most recent call last):
File "pcapkit-demo.py", line 8, in
analysis = pcapkit.analyse(fileObj, length=None)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 205: character maps to

Expected behavior
It should analyze the packets

Additional context
[email protected]
+91 8097872267

Add Example to print reassembled byte stream

I would like to see an example on how to do reassembly on a TCP byte stream coming from a PCAP file and print the reassembled version

UDP response messages (DNS responses) are skipped when ipv4 reassembly is set to true.

Describe the bug
I created a pcap using wireshark, it has some DNS quarries and its responses, but with ipv4 reassembly set to true, dns responses are skipped when i enumerate it.

System information

OS Version: Windows 11 Pro 23H2
Python Version: 3.11
Python Implementation: PyPy
PyPcapKit Version: 1.3.1.post9

Traceback stack
When run program with `PCAPKIT_DEVMODE=1:

C:\Python311\Lib\site-packages\pcapkit_init_.py:0: DevModeWarning: development mode enabled
PCAPKIT_DEVMODE=1
[INFO] 05/14/2024 04:25:19 PM - IPv4 reassembly enabled
[INFO] 05/14/2024 04:25:19 PM - IPv6 reassembly enabled
[INFO] 05/14/2024 04:25:19 PM - TCP reassembly enabled
[WARNING] 05/14/2024 04:25:19 PM - EOF reached
[WARNING] 05/14/2024 04:25:19 PM - EOF reached
C:\Python311\Lib\site-packages\pcapkit\utilities\warnings.py:54: ExtractionWarning: EOF reached
warnings.warn(message, category, stacklevel)

Expected behavior
It should give both DNS request and responses after reassembly.

Additional context
It worked fine for other pcaps, i created two of them on two different PCs using wireshark. PC1 used DNS over TCP and PC2 used DNS over UDP, PC2 case was creating this issue of skipping responses after reassembly.

Following is the extraction setting
self.extraction = extract( fin=inpuct_pcap, store=False, ip=True, reasm_strict=True, tcp=True, reassembly=True, nofile=False, )
Enumeration is done as:
for _, stream in enumerate(self.extraction.reassembly.ipv4)

I have attached the pcap here secureProfileDNSOverUdp.zip

Unable to execute pcapkit at all.

Describe the bug
After installing through pip or through git, trying to run the command "pcapkit" gives a "permission denied" error. Attempting to run it with elevated credentials gives a "command not found"

System information
A clear and concise description of your system information.

OS Version: macOS Mojave 10.14.6
Python Version: 3.7

Expected behavior
Run the script

Additional context
Have verified the correct path is added for the library.

module 'pcapkit.utilities' has no attribute 'stacklevel'

Describe the bug

Cannot run cli or import pcapkit in python script.

Traceback (most recent call last):
  File "./venv/bin/pcapkit-cli", line 5, in <module>
    from pcapkit.__main__ import main
AttributeError: module 'pcapkit.utilities' has no attribute 'stacklevel'

System information

OS Version: macOS Mojave 10.15.6
Python Version: 3.8.5
Python Implementation: CPython

Traceback stack

/Users/ohoh/hehe/venv/lib/python3.8/site-packages/pcapkit/__init__.py:0: DevModeWarning: development mode enabled
  PCAPKIT_DEVMODE=true
Traceback (most recent call last):
  File "./venv/bin/pcapkit-cli", line 5, in <module>
    from pcapkit.__main__ import main
  File "/Users/ohoh/hehe/venv/lib/python3.8/site-packages/pcapkit/__init__.py", line 82, in <module>
    import pcapkit.all
  File "/Users/ohoh/hehe/venv/lib/python3.8/site-packages/pcapkit/all.py", line 20, in <module>
    from pcapkit.utilities import *  # pylint: disable=redefined-builtin
AttributeError: module 'pcapkit.utilities' has no attribute 'stacklevel'

Expected behavior

It works

Additional context

Only pip install pypcapkit dpkt scapy pyshark worked. All other with brackets reported a "no matches found: pypcapkit[xxx]"

Unable to run pcapkit.extract

Describe the bug
When I try to execute the test program text_extractor.py I get the following error message:

Traceback (most recent call last):
  File "~/PyPCAPKit/test/test_extractor.py", line 5, in <module>
    plist = pcapkit.extract(fin='../sample/in.pcap', fout='../sample/out.plist', format='plist')
TypeError: 'prec' is an invalid keyword argument for this function

System information
A clear and concise description of your system information.

OS Version: Ubuntu 20.04.5 LTS
Python Version: Python 3.10.11
Python Implementation: CPython

Traceback stack

~/pcapkit/__init__.py:0: DevModeWarning: development mode enabled
  PCAPKIT_DEVMODE=true
Traceback (most recent call last):
  File "~/PyPCAPKit/test/test_extractor.py", line 5, in <module>
    plist = pcapkit.extract(fin='../sample/in.pcap', fout='../sample/out.plist', format='plist')
  File "~/PyPCAPKit/pcapkit/interface/core.py", line 117, in extract
    return Extractor(fin=fin, fout=fout, format=format,
  File "~/PyPCAPKit/pcapkit/foundation/extraction.py", line 635, in __init__
    self.run()    # start extraction
  File "~/PyPCAPKit/pcapkit/foundation/extraction.py", line 342, in run
    self.record_frames()
  File "~/PyPCAPKit/pcapkit/foundation/extraction.py", line 490, in record_frames
    self._exeng.read_frame()
  File "~/PyPCAPKit/pcapkit/foundation/engines/pcap.py", line 139, in read_frame
    frame = Frame(ext._ifile, num=ext._frnum+1, header=self._gbhdr.info,
  File "~/PyPCAPKit/pcapkit/protocols/protocol.py", line 497, in __init__
    self.__post_init__(file, length, **kwargs)  # type: ignore[arg-type]
  File "~/PyPCAPKit/pcapkit/protocols/misc/pcap/frame.py", line 349, in __post_init__
    self._info = self.unpack(length, _read=_read, **kwargs)
  File "~/PyPCAPKit/pcapkit/protocols/misc/pcap/frame.py", line 191, in unpack
    return self.read(length, **kwargs)
  File "~/PyPCAPKit/pcapkit/protocols/misc/pcap/frame.py", line 212, in read
    with decimal.localcontext(prec=64):
TypeError: 'prec' is an invalid keyword argument for this function

Expected behavior
The program should run correctly and generate the out.plist file.

Additional context
Add any other context about the problem here.

Dynamic code execution is always an issue and should never be used.

Describe the bug
The library contains exec call

System information
A clear and concise description of your system information.

OS Version: any
Python Version: any
Python Implementation:any

Traceback stack
Was not executed.

Expected behavior
Must not contain any dynamic code execution.

Additional context
Dynamic code execution is always a security issue.

Import crash related to #113

Unfortunately, the fix for #113 seems to have introduced another issue that appears when PyPCAPKit is used in a Python 3.7 environment. The following is on Python 3.7.0, Windows 10, CPython:

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.34.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pcapkit
Traceback (most recent call last):

  File "X:\test\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-2-fdafce80c9fe>", line 1, in <module>
    import pcapkit

  File "X:\test\venv\lib\site-packages\pcapkit\__init__.py", line 83, in <module>
    from pcapkit.foundation.registry import *

  File "X:\test\venv\lib\site-packages\pcapkit\foundation\__init__.py", line 13, in <module>
    from pcapkit.foundation.extraction import Extractor

  File "X:\test\venv\lib\site-packages\pcapkit\foundation\extraction.py", line 424
    if (parent := os.path.split(fout)[0]):
               ^
SyntaxError: invalid syntax

Unknown file format

Describe the bug
I am trying to parse a pcap-ng file, but when running the parser it says it doesn't support the file format.
Is this a bug or it doesn't support pcap-ng?

System information
A clear and concise description of your system information.

OS Version: Ubuntu 18.04
Python Version: 3.6
Python Implementation: CPython

Traceback stack

Traceback (most recent call last):
pcapkit.utilities.exceptions.FileError: [Errno 5] Unknown file format: 'in.pcap'

Expected behavior
I should have the pcap exported to json format

Additional context
File version:
in.pcap: pcap-ng capture file - version 1.0

Command in python script:
json = pcapkit.extract(fin='in.cap', fout='out.json', format='json', extension=False)
I also tried with a new capture (with tcpdump) and I got a very different error with the same command above and with the ppcapkit-cli command as well.

Traceback (most recent call last):
  File "/usr/local/bin/pcapkit-cli", line 10, in <module>
    sys.exit(main())
TypeError: object_hook() missing 1 required positional argument: 'o'

How to obtain tpc trace flow

Hi,
Describe the bug
I have been trying for a while to trace TCP flows using TraceFlow but without success.
I did so:
extractor = pcapkit.extract(fin='in.pcap', fout='out.json', format='json', extension=False, protocol='tcp', trace=True, trace_fout="trace_out")
but trace_out file is not created, only out.json

Expected behavior
New file trace_out containing all TCP flows

Additional context
I attach the file in.pcap
in.zip

Maybe I'm doing something wrong.
Can anyone provide me a simple example to do this operation?

Thx a lot

Unable to install from source

Describe the bug
When I tried to install current version with pip install -e . command. I got the error message:

TypeError: expected str, bytes or os.PathLike object, not NoneType [end of output]

System information

OS Version: Ubuntu 22.04.2 LTS
Python Version: Python 3.10.6
Python Implementation: CPython

**Traceback stack**
   Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/home/user/workspace/PyPCAPKit/setup.py", line 105, in <module>
          setup(
        File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/user/workspace/PyPCAPKit/setup.py", line 92, in run
          refactor(os.path.join(self.install_lib, 'pcapkit'))
        File "/usr/lib/python3.10/posixpath.py", line 76, in join
          a = os.fspath(a)

Expected behavior
I expected the library to be installed.

Additional context

Import of PCAPKit crashes

In a clean virtual environment running Python 3.9.7, importing PyPCAPKit fails:

c:\Workspace\temporary>C:\Python39\python.exe -m venv venv

c:\Workspace\temporary>venv\Scripts\activate.bat

(venv) c:\Workspace\temporary>pip install pypcapkit[scapy]
Collecting pypcapkit[scapy]
  Using cached pypcapkit-0.16.1-py3-none-any.whl (406 kB)
Collecting aenum
  Using cached aenum-3.1.11-py3-none-any.whl (131 kB)
Collecting tbtrim>=0.2.1
  Using cached tbtrim-0.3.1-py2.py3-none-any.whl (4.4 kB)
Collecting chardet
  Using cached chardet-5.0.0-py3-none-any.whl (193 kB)
Collecting dictdumper~=0.8.0
  Using cached dictdumper-0.8.4.post2-py2.py3-none-any.whl (18 kB)
Collecting scapy
  Using cached scapy-2.4.5-py2.py3-none-any.whl
Installing collected packages: tbtrim, dictdumper, chardet, aenum, scapy, pypcapkit
Successfully installed aenum-3.1.11 chardet-5.0.0 dictdumper-0.8.4.post2 pypcapkit-0.16.1 scapy-2.4.5 tbtrim-0.3.1
WARNING: You are using pip version 21.2.3; however, version 22.2.1 is available.
You should consider upgrading via the 'c:\Workspace\temporary\venv\Scripts\python.exe -m pip install --upgrade pip' command.

(venv) c:\Workspace\temporary>python
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pcapkit
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() got an unexpected keyword argument 'ABOR'
>>>

System information
A clear and concise description of your system information.

OS Version: Windows 10
Python Version: 3.9.7
Python Implementation: CPython

Expected behavior
Importing the module succeeds

Reassembly not working since 0.16.3

Describe the bug
The example for reassembly in the documentation and test/test_reassembly.py do not work for the 1.x.x versions. Latest working version seems to be 0.16.3.

System information
A clear and concise description of your system information.

OS Version: Ubuntu 22.04.3 LTS
Python Version: Python 3.11.4 (main, Jun 7 2023, 12:45:48) [GCC 11.3.0] on linux
Python Implementation: CPython

Traceback stack

$ PCAPKIT_DEVMODE=true python3 test_reassembly.py
/home/martin/edc/network/.venv/lib/python3.11/site-packages/pcapkit/__init__.py:0: DevModeWarning: development mode enabled
  PCAPKIT_DEVMODE=true
Frame   1: Ethernet:IPv4:TCP
Frame   2: Ethernet:IPv4:TCP
Frame   3: Ethernet:IPv4:TCP
Frame   4: Ethernet:IPv4:TCP:Raw
..snip..
Frame 744: Ethernet:IPv4:TCP:Raw
Frame 745: Ethernet:IPv4:TCP
Frame 746: Ethernet:IPv4:TCP:Raw
Traceback (most recent call last):
  File "/home/martin/edc/network/PyPCAPKit/test/test_reassembly.py", line 19, in <module>
    for datagram in extraction.reassembly.tcp:  # type: ignore[union-attr]
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/martin/edc/network/.venv/lib/python3.11/site-packages/pcapkit/foundation/extraction.py", line 227, in reassembly
    data = ReassemblyData(
           ^^^^^^^^^^^^^^
NameError: name 'ReassemblyData' is not defined. Did you mean: 'ReassemblyManager'?

Expected behavior
No error, working tests.

Additional context
test/test_reassembly.py relies on ../sample/test.pcap which does not exist. I provided my own test.pcap.

test/ directory seems to be more like examples than tests. Might be worth creating a unit test set, or at least check manually that the examples are still working. Another non-working example is test/test_http.py which relies on ../sample/http.pcap that also does not exist.

`pcapkit-cli -j capture.pcap` - invalid json output

error message parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line ..., column ... when trying to parse with a tool like jq

pcapkit-vendor causes PermissionError writing to site-packages

pcapkit-vendor causes a write to site-packages, which should be read-only, and is read-only on distro.

System information
Using https://build.opensuse.org/package/show/home:jayvdb:py-new/python-pypcapkit
Python 3.8

Traceback stack

/usr/bin/pcapkit-vendor
Ethertype IEEE 802 Numbers
Operation Codes [RFC 826][RFC 5494]
ESP Transform Suite IDs
Cipher IDs
HIP NAT Traversal Modes
Registration Types
HTTP/2 Error Code
Option Classes
IPv4 Router Alert Option Values
TOS (DS Field) Reliability
QS Functions
TaggerID Types
Authentication Types
Priority levels defined in IEEE 802.1p.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pcapkit/vendor/__main__.py", line 30, in run
    vendor()
  File "/usr/lib/python3.8/site-packages/pcapkit/vendor/default.py", line 243, in __init__
    with open(os.path.join(ROOT, '..', 'const', STEM, FILE), 'w') as file:
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.8/site-packages/pcapkit/vendor/../const/hip/cipher.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/lib/python3.8/site-packages/pcapkit/vendor/__main__.py", line 32, in run
    warnings.warn(error, VendorRuntimeWarning)
TypeError: expected string or bytes-like object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/pcapkit-vendor", line 11, in <module>
    load_entry_point('pypcapkit==0.14.5', 'console_scripts', 'pcapkit-vendor')()
TypeError: expected string or bytes-like object

Expected behavior
If there is a need to write custom code to the site-packages, that should be done in setup.py

Error running test_reassembly.py

running the test_reassembly.py test fails seemingly independently of the used PCAP file.

System information
A clear and concise description of your system information.

OS Version: manjaro
Python Version: 3.10.2
Python Implementation: CPython

Traceback (most recent call last):
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/foundation/extraction.py", line 531, in record_frames
    self._read_frame()
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/foundation/extraction.py", line 871, in _read_frame
    return self._dpkt_read_frame()
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/foundation/extraction.py", line 1167, in _dpkt_read_frame
    flag, data = tcp_reassembly(packet, count=self._frnum)
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/toolkit/dpkt.py", line 204, in tcp_reassembly
    ip = packet['ip']
  File "/home/jo/Desktop/tmp/PyPCAPKit/.env/lib/python3.10/site-packages/dpkt/dpkt.py", line 188, in __getitem__
    raise KeyError(kls)
KeyError: 'ip'

Expected behavior
run without error

As far as I have investigated this comparison if dd.__class__ == kls: causes the problem because it fails to identify the ip class.

maybe the DPKT has change the API and caused the test to fail.

I doesn't crash if the engine is changed to SCAPY

Can run test_reassembly.py

I am trying to run test_reassembly.py and get the folloing error:

System information
A clear and concise description of your system information.

OS Version: manjaro
Python Version: 3.10.2
Python Implementation: CPython

  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/foundation/extraction.py", line 531, in record_frames
    self._read_frame()
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/foundation/extraction.py", line 874, in _read_frame
    return self._default_read_frame()
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/foundation/extraction.py", line 902, in _default_read_frame
    frame = Frame(self._ifile, num=self._frnum+1, proto=self._dlink,
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/protocols/protocol.py", line 307, in __init__
    self.__post_init__(file, length, **kwargs)  # type: ignore[arg-type]
  File "/home/jo/Desktop/tmp/PyPCAPKit/pcapkit/protocols/misc/pcap/frame.py", line 279, in __post_init__
    self._prot = header.network
AttributeError: 'Header' object has no attribute 'network'

'bs4' dependency not installed via pip

Describe the bug
After installing the latest release using "pip install pypcapkit" into a virtualenv, I get the following "module not found" error when importing pcapkit:
ModuleNotFoundError: No module named 'bs4'

Running "pip install bs4" resolves this.

System information
A clear and concise description of your system information.

OS Version: macOS High Sierra 10.13.6
Python Version: 3.6.5
Python Implementation: CPython

NameError: name 'packet' is not defined

import pcapkit

pcapfile='adsb.pcap'

extraction=pcapkit.extract(fin=pcapfile,store=False,nofile=True,tcp=True,strict=True)

NameError: name 'packet' is not defined

the call stack:

File "C:\Users\Root\Anaconda3\lib\site-packages\pcapkit\interface_init_.py", line 128, in extract
trace=trace, trace_fout=trace_fout, trace_format=trace_format)
File "C:\Users\Root\Anaconda3\lib\site-packages\pcapkit\foundation\extraction.py", line 498, in init
self.run() # start extraction
File "C:\Users\Root\Anaconda3\lib\site-packages\pcapkit\foundation\extraction.py", line 256, in run
self.record_frames() # read frames
File "C:\Users\Root\Anaconda3\lib\site-packages\pcapkit\foundation\extraction.py", line 360, in record_frames
self._read_frame()
File "C:\Users\Root\Anaconda3\lib\site-packages\pcapkit\foundation\extraction.py", line 579, in _read_frame
return self._default_read_frame()
File "C:\Users\Root\Anaconda3\lib\site-packages\pcapkit\foundation\extraction.py", line 621, in _default_read_frame
flag, data = ipv4_reassembly(packet)
NameError: name 'packet' is not defined

jarryshaw / pypcapkit Goto Github PK

pypcapkit's Introduction

PyPCAPKit - Comprehensive Network Packet Analysis Library

About

Module Structure

Engine Comparison

Test Environment

Test Results

Installation

pypcapkit's People

Contributors

Stargazers

Watchers

Forkers

pypcapkit's Issues

Recommend Projects

Recommend Topics

Recommend Org