synchronizing / mitm Goto Github PK

View Code? Open in Web Editor NEW

121.0 121.0 17.0 265 KB

👨🏼‍💻 ‎‎‎‏‏ A customizable man-in-the-middle TCP intercepting proxy.

Home Page: https://synchronizing.github.io/mitm/

License: MIT License

Python 100.00%

asyncio mitm proxy python

mitm's People

Contributors

Stargazers

Watchers

Forkers

stdedos zentooo krumboychev itspectre pbehnke noob-z swavan cchinnasamy limingzxf scott91e1 jimanx2 flikhamud45 planetben simrit1 4v4loon ongyuann proxygang

mitm's Issues

Problem working with multiple protocols

Discussed in #28

^{Originally posted by macruz21 April 13, 2023}
Hello,
First of all congratulations for your code. I found it very elegant. I am using it for several things that I have in mind, but I have found a problem in the code. When you use for to go through all the possible protocols, there should be a break when a protocol doesn't throw the "InvalidProtocol" exception, because otherwise it makes multiple connections until it reaches the last protocol that is actually executed.
In this way, when it reaches the first protocol that works, it exits the for.
Thank you very much for your code, I am enjoying a lot with it.
Greetings.

TypeError: ClassTask.init() got an unexpected keyword argument 'run_forever'

Hello, here I am again!

EDIT: If I knew how to fix this I'd make a PR, sorry in advance!

Code (from examples):

from mitm import MITM, protocol, middleware, crypto

mitm = MITM(
    host="127.0.0.1",
    port=8888,
    protocols=[protocol.HTTP],
    middlewares=[middleware.Log],
    buffer_size=8192,
    timeout=5,
    ssl_context=crypto.mitm_ssl_default_context(),
    start=False,
)
mitm.start()

Output error:

Traceback (most recent call last):
  File "c:\Users\Slimakoi\Desktop\Coding\test\falling_new.py", line 3, in <module>
    mitm = MITM(
  File "C:\Program Files\Python310\lib\site-packages\mitm\mitm.py", line 65, in __init__
    super().__init__(
TypeError: ClassTask.__init__() got an unexpected keyword argument 'run_forever'

Make installing certificates easier.

A few issues/discussion posts have been opened regarding mitm's certificates & and its use with Chrome. It would be a nice addition to have an easy method for installing certificates on different machines.

AttributeError: module 'mitm.crypto' has no attribute 'mitm_ssl_context'

Code (from examples):

from mitm import MITM, protocol, middleware, crypto

mitm = MITM(
    host="127.0.0.1",
    port=8888,
    protocols=[protocol.HTTP],
    middlewares=[middleware.Log],
    buffer_size=8192,
    timeout=5,
    ssl_context=crypto.mitm_ssl_context(),
    start=False,
)
mitm.start()

Error:

C:\Users\Slimakoi\Desktop\Coding>main.py
Traceback (most recent call last):
  File "C:\Users\Slimakoi\Desktop\Coding\main.py", line 10, in <module>
    ssl_context=crypto.mitm_ssl_context(),
AttributeError: module 'mitm.crypto' has no attribute 'mitm_ssl_context'

Not Compatible with PyInstaller

locally the .exe works fine, on other system its just returning this:

Traceback (most recent call last):
  File "pbr\version.py", line 451, in _get_version_from_pkg_resources
  File "pkg_resources__init.py", line 342, in get_provider
  File "pkg_resources__init.py", line 886, in require
  File "pkg_resources__init.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'mitm' distribution was not found and is required by the application

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<dist\obf\main.py>", line 3, in <module>
  File "<frozen main>", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 352, in exec_module
  File "mitm__init.py", line 15, in <module>
  File "pbr\version.py", line 485, in release_string
  File "pbr\version.py", line 495, in semantic_version
  File "pbr\version.py", line 458, in _get_version_from_pkg_resources
  File "pbr\packaging.py", line 872, in get_version
Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name mitm was given, but was not able to be found.
[12828] Failed to execute script 'main' due to unhandled exception!

command used to create .exe:
pyarmor-7 pack -e " --onefile --icon main_free.ico" main.py

Create a test suite for the project.

A testing suite needs to be built for the project. I'm currently unsure how to go about this, and so any suggestions are welcomed.

I've tried to use Pytest for this, but I've had major issues booting up the server and having it run in the background before tests.

Basic Sample throws error

Python: 3.8
Fresh install of mitm

Using the basic example on the readme, I get the following error when trying to run it:
"partially initialized module 'mitm' has no attribute 'MITM' (most likely due to a circular import)

Not sure what's going on here, since I just installed the package and tried running the sample.

Contact?

G'day, I'm looking at using this for a project I have in mind, I just want to ask some questions about this repo.

If you want to contact me, chuck me a message on Twitter: @mitchnaake
I would send you a pm, but I am unable for some odd reason.

I think that you might find my project quite interesting :)

Have a good one, mate!

Performance bogs down with normal web use.

G'day,

I tried using the proxy as a normal HTTPs proxy for normal web-browsing. It seems like it struggles with a backlog of requests and does things sequentially.

I'm not sure if it's built for this kind of purpose, but it's what I intend on using it for so any help in getting it to run slightly smoother would be of great help!

Cheers,

Mitch

MITM doesn't function as intended in some situations

I'm using something similar to the sample python to run MITM under Windows 10.

from mitm import MITM, CertificateAuthority, middleware, protocol
from pathlib import Path

# Loads the CA certificate.
path = Path("")
ca = CertificateAuthority.init(path=path)

# Starts the MITM server.
mitm = MITM(
    host="127.0.0.1",
    port=8888,
    protocols=[protocol.HTTP],
    middlewares=[middleware.Log],
    certificate_authority=ca,
)
mitm.run()

The issue is that, in some situations, MITM doesn't complete the round trip. I can't tell where it's failing.

The sequence of operation is:

Launch the proxy
Make sure the cert is installed in the Windows cert store as a root CA for the machine
Launch the browser and navigate to a site by entering a url

On my home system, this issue happens when I change the configuration to use middleware.HTTPLog (at the bottom is a log for that). Using middleware.Log seems to work as expected.

On my work system, it doesn't seem to matter which logger is used. The work system is behind a corporate firewall. I do have https_proxy, http_proxy, and no_proxy set and they are known to be working with other python scripts.

The log from the home system, using middleware.HTTPLog:

(venv) C:\Xfer\mitm>python start-mitm.py
2023-09-30 19:28:28 INFO     MITM server started on ←[1m127.0.0.1:8888←[0m.
2023-09-30 19:28:37 INFO     Client ←[1m127.0.0.1:49865←[0m has connected.
2023-09-30 19:28:37 INFO     Client 127.0.0.1:49865 to mitm:

→ CONNECT accounts.google.com:443 HTTP/1.1
→ Host: accounts.google.com:443
→ Proxy-Connection: keep-alive
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

2023-09-30 19:28:37 INFO     Client ←[1m127.0.0.1:49865←[0m has connected to server ←[1m172.217.14.205:443←[0m.
2023-09-30 19:28:37 INFO     Client 127.0.0.1:49865 to 172.217.14.205:443:

→ POST /ListAccounts?gpsia=1&source=ChromiumBrowser&json=standard HTTP/1.1
→ Host: accounts.google.com
→ Connection: keep-alive
→ Content-Length: 1
→ Origin: https://www.google.com
→ Content-Type: application/x-www-form-urlencoded
→ Sec-Fetch-Site: none
→ Sec-Fetch-Mode: no-cors
→ Sec-Fetch-Dest: empty
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
→ Accept-Encoding: gzip, deflate, br
→ Accept-Language: en-US, en;q=0.9
→
→

2023-09-30 19:28:41 INFO     Client ←[1m127.0.0.1:49869←[0m has connected.
2023-09-30 19:28:41 INFO     Client 127.0.0.1:49869 to mitm:

→ CONNECT ouroborus.org:443 HTTP/1.1
→ Host: ouroborus.org:443
→ Proxy-Connection: keep-alive
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

2023-09-30 19:28:41 INFO     Client ←[1m127.0.0.1:49870←[0m has connected.
2023-09-30 19:28:41 INFO     Client 127.0.0.1:49870 to mitm:

→ CONNECT ouroborus.org:443 HTTP/1.1
→ Host: ouroborus.org:443
→ Proxy-Connection: keep-alive
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

2023-09-30 19:28:41 INFO     Client ←[1m127.0.0.1:49870←[0m has connected to server ←[1m69.163.216.180:443←[0m.
2023-09-30 19:28:41 INFO     Client ←[1m127.0.0.1:49869←[0m has connected to server ←[1m69.163.216.180:443←[0m.
2023-09-30 19:28:41 INFO     Client 127.0.0.1:49869 to 69.163.216.180:443:

→ GET / HTTP/1.1
→ Host: ouroborus.org
→ Connection: keep-alive
→ sec-ch-ua: "Google Chrome";v="117", "Not;A=Brand";v="8", "Chromium";v="117"
→ sec-ch-ua-mobile: ?0
→ sec-ch-ua-platform: "Windows"
→ Upgrade-Insecure-Requests: 1
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
→ Accept: text/html, application/xhtml+xml, application/xml;q=0.9, image/avif, image/webp, image/apng, */*;q=0.8, application/signed-exchange;v=b3;q=0.7
→ Sec-Fetch-Site: none
→ Sec-Fetch-Mode: navigate
→ Sec-Fetch-User: ?1
→ Sec-Fetch-Dest: document
→ Accept-Encoding: gzip, deflate, br
→ Accept-Language: en-US, en;q=0.9

2023-09-30 19:28:49 INFO     Client ←[1m127.0.0.1:49874←[0m has connected.
2023-09-30 19:28:49 INFO     Client 127.0.0.1:49874 to mitm:

→ CONNECT optimizationguide-pa.googleapis.com:443 HTTP/1.1
→ Host: optimizationguide-pa.googleapis.com:443
→ Proxy-Connection: keep-alive
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

2023-09-30 19:28:49 INFO     Client ←[1m127.0.0.1:49874←[0m has connected to server ←[1m142.251.33.74:443←[0m.
2023-09-30 19:28:49 INFO     Client 127.0.0.1:49874 to 142.251.33.74:443:

→ POST /v1:GetModels?key=AIzaSyBOti4mM-6x9WDnZIjIeyEU21OpBXqWBgw HTTP/1.1
→ Host: optimizationguide-pa.googleapis.com
→ Connection: keep-alive
→ Content-Length: 560
→ Content-Type: application/x-protobuf
→ X-Client-Data: COvxygE=
→ Sec-Fetch-Site: none
→ Sec-Fetch-Mode: no-cors
→ Sec-Fetch-Dest: empty
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
→ Accept-Encoding: gzip, deflate, br
→ Accept-Language: en-US, en;q=0.9
→
→
→☻ ♫
→ ♫2_
→ Ytype.googleapis.com/google.internal.chrome.optimizationguide.v1.PageEntitiesModelMetadata↕♂→
→        ♫
→  ♫
→ ► ♫2a
→ Ytype.googleapis.com/google.internal.chrome.optimizationguide.v1.SegmentationModelMetadata↕♦J☻►♥
→¶ ♫
→ § ♫2a
→ Ytype.googleapis.com/google.internal.chrome.optimizationguide.v1.SegmentationModelMetadata↕♦J☻►♥
→↨ ♫2a
→ Ytype.googleapis.com/google.internal.chrome.optimizationguide.v1.SegmentationModelMetadata↕♦J☻►♥
→← ♫2a
→ Ytype.googleapis.com/google.internal.chrome.optimizationguide.v1.SegmentationModelMetadata↕♦J☻►♥↑♠*♣en-US♠☻

2023-09-30 19:29:11 INFO     Client ←[1m127.0.0.1:49878←[0m has connected.
2023-09-30 19:29:11 INFO     Client 127.0.0.1:49878 to mitm:

→ GET http://www.gstatic.com/generate_204 HTTP/1.1
→ Host: www.gstatic.com
→ Proxy-Connection: keep-alive
→ Pragma: no-cache
→ Cache-Control: no-cache
→ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
→ Accept-Encoding: gzip, deflate
→ Accept-Language: en-US, en;q=0.9

2023-09-30 19:29:11 INFO     Client ←[1m127.0.0.1:49878←[0m has connected to server ←[1m142.250.217.67:80←[0m.
Traceback (most recent call last):
  File "C:\Xfer\mitm\start-mitm.py", line 16, in <module>
    mitm.run()
  File "C:\Xfer\mitm\venv\lib\site-packages\toolbox\asyncio\pattern.py", line 134, in run
    self._loop.run_until_complete(self._task)
  File "C:\Python310\lib\asyncio\base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "C:\Python310\lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Python310\lib\asyncio\base_events.py", line 600, in run_forever
    self._run_once()
  File "C:\Python310\lib\asyncio\base_events.py", line 1860, in _run_once
    event_list = self._selector.select(timeout)
  File "C:\Python310\lib\asyncio\windows_events.py", line 439, in select
    self._poll(timeout)
  File "C:\Python310\lib\asyncio\windows_events.py", line 788, in _poll
    status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)
KeyboardInterrupt
^C

Not decoding requests

Hey, I'm using your example in the Middleware section in the readme of the project.

But I'm only getting following :

py main.py
2021-11-09 18:27:17 INFO     Booting up server on 127.0.0.1:8888.
2021-11-09 18:27:18 INFO     Client 127.0.0.1:62708 has connected.
2021-11-09 18:27:19 INFO     Successfully closed connection with 127.0.0.1:62708.

When running the following script:

import requests

proxies = {"http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888"}
requests.get("https://httpbin.org/anything", proxies=proxies, verify=False)

I'd like to be able to see the headers, the content, etc of the request

Work needed for caching?

Hi, thanks for this wonderful library!
It's rare to see such compact and elegant async code.

I've been wondering whether it would be possible to add caching to mitm.
Middlewares would not work for this, since don't have the capability of fundamentally altering the data flow.

So my question is: Do you see an elegant way of adding such functionality?

The basic issue is that we currently cannot initiate a (fake/cached) server response when handling the request.

One possible option would be to check and prevent sending data in https://github.com/synchronizing/mitm/blob/master/mitm/mitm.py#L137.

I.e. something like

if data:
    writer.write(data)

Thus the request would never get sent to the true destination if a middleware blanks out the request data.
A second middleware would then need to figure out how and when to return the cached data.

But that's of course not very robust and might break on all sorts of edge cases.

Perhaps you have a better idea?

Circular import error

G'day!

I just got around to trying the 1.3.0 release. I created a fresh project on PyCharm, using Python 3.10 - When running the following code:

from mitm import MITM, CertificateAuthority, middleware, protocol
from pathlib import Path

# Loads the CA certificate.
path = Path("")
ca = CertificateAuthority.init(path=path)

# Starts the MITM server.
mitm = MITM(
    host="127.0.0.1",
    port=8888,
    protocols=[protocol.HTTP],
    middlewares=[middleware.Log],
    buffer_size=8192,
    timeout=5,
    ca=ca,
)
mitm.run()

It throws this error:

Traceback (most recent call last):
  File "/Users/myname/PycharmProjects/ComputerScience/misc/mitm.py", line 1, in <module>
    from mitm import CertificateAuthority, middleware, protocol
  File "/Users/myname/PycharmProjects/ComputerScience/misc/mitm.py", line 1, in <module>
    from mitm import CertificateAuthority, middleware, protocol
ImportError: cannot import name 'CertificateAuthority' from partially initialized module 'mitm' (most likely due to a circular import) (/Users/myname/PycharmProjects/ComputerScience/misc/mitm.py)

Use without having to use verify=False

Hello, I wanted to know if it was possible to use this project without having to use verify=False.
I heard this was possible by installing a certificate.
Not using verify=False while doing requests will make my program crash because of SSL errors

Deal with hanging connections and unknown protocols.

As of right now mitm does not deal with hanging connections and unknown protocols very well. httpq will hang if the client never provide the correct bytes:

mitm/mitm/mitm.py

Lines 117 to 121 in 5b9ae63

 # Read request from client until body. 

 req = httpq.Request() 

 while req.step_state() != httpq.state.BODY: 

 data = await reader.read(self.config.buffer_size) 

 req.feed(data)

Probable solution:

(a) Check if client.at_eof directly on the while loop, and
(b) Read up to n bytes. If we don't have a valid HTTP first line by then, the client is sending some other protocol.

How to make it work for https

for me, mitm just works with http packets and not https.

do i need a certificate or something?

thanks in advice :D

Can I selectively try intercepting HTTPS and when not possible just transparently proxy it so HTTPS doesn't fail somehow? I don't understand why it doesn't work yet.

Hello. While HTTP works HTTPS doesn't and I had used charles proxy so far so I know what I want but sadly I don't know much about the terms.

In charles proxy, I had a client certificate and SSL Proxying switch. In that program, I didn't have a way to selectively do SSL proxying. I could either use it globally or not. When enabled it could decode HTTPS stuff and show me more information on some HTTPS requests but only if I had used this patch (and the client certificate comes from here too): https://github.com/SciresM/3DS-SSL-Patch
(Well, most importantly, HTTPS for 3DS would work fine with all that)

If I didn't use that thing, I would instead see these same HTTPS requests fail (EVEN IF I NEVER TOUCHED THE REQUEST OR RESPONSE MYSELF IN ANY WAY) and their method become CONNECT instead of the normal GET, POST etc.

I get that it became CONNECT but why does it even fail since there is a tunnel now and proxy is doing its job as expected? I don't know. I just though, if it cannot decrypt then it should just let it pass without touching the request and response data and everything would be okay right? Well... I am not sure how it really works out but it didn't happen. In the end I even got the status code but there was an alert code in TLS info section: "unknown_ca (48) - CA certificate could not be matched with a known, trusted CA"

Yeah, the SSL patch mentioned that. It disables the verification: "This is a patch for the 3DS SSL module to disable Root CA Verification."

But I still don't get it. Why can't the mitm proxy just use CONNECT and step away? Why 3DS won't trust it no matter what? What even is proxy doing wrong? A professional proxy would work, so what is wrong with my mitm proxy? I don't want to break HTTPS as a whole, I just want to capture IF I CAN CAPTURE but if I can't, I want HTTPS to work transparently.

For charles, I added this certificate as "client certificate". Well for mitm, I didn't even see a way to specify a client certificate to begin with. So I added it as a certificate_authority. Now obviously HTTPS is broken. Well, even if I left it at default, HTTPS was broken anyway.

Can it handle my use case? Is this behaviour possible? Do I need some weird middleware to do it manually?

I actually wanted to change URL of where the request would go for HTTP freely while trying to decrypt HTTPS to see its endpoint but not touching HTTPS requests and responses (well, only block them or let them pass with CONNECT) but I can't do it due to this HTTPS issue.

Sorry for the wall of text where I describe how badly I am confused. Here is what I used:

import OpenSSL
from mitm import MITM, protocol, middleware

with open("ClCertA.p12", "rb") as f: # This file is linked above.
    p12 = OpenSSL.crypto.load_pkcs12(f.read(), b"3ds") # my version is 23.1.1 (OpenSSL.__version__)

mitm = MITM(
    host="192.168.1.114",
    port=8080,
    protocols=[protocol.HTTP],
    middlewares=[middleware.Log], # middleware.HTTPLog used for the example below.
    certificate_authority = p12.get_ca_certificates(),
)

mitm.run()

	# Read request from client until body.
	req = httpq.Request()
	while req.step_state() != httpq.state.BODY:
	data = await reader.read(self.config.buffer_size)
	req.feed(data)