Built-in TLS Decryption about pcapdroid HOT 59 CLOSED

The necessary upstream project nitmproxy is now supporting non HTTP data, I'm currently testing it. If it works, a data logger must be still added, but I then we are almost there.

from pcapdroid.

• I haved removed the commons-io dependency.
• conscrypt-openjdk-uber must be excluded when using the library under Android, it is only required in Java-SE, Android is already using this internally as crypto provider.
• bcpkix-jdk15to18 and bcprov-jdk15to18 are needed for creating the MITM certificates.
• Under Android the barchart-udt-bundle dependency must be excluded, it is optional and not supported and not needed. I have patched the latest version of littleproxy to make it optional under Android with a merge request and it was already merged.
• I guess, I cannot influence the LittleProxy maintainer to remove guava or commons-lang3, these 2 libraries are convenient, but usually only a subset of the functions are needed.

I will check with my Android test project how much is left after using the Android shrinker. Usually almost all from the utilitiy libraries and major parts of bouncy castle should get removed resulting in just a few HD floppy disks.

from pcapdroid.

The Android internal SSL Engine seems to have some issues with TLS 1.3 and ALPN. I have patched netty to be able to use the BouncyCastle TLS implementation which is working for the above mentioned issue. I will wait until this gets accepted to have it available as published dependency.

from pcapdroid.

The nitmproxy maintainer seems not to handle this. So, I started on my own, I think I'm on the right track now, actually it seems simple, let's see how it works.

from pcapdroid.

Tried to export SSL keylog file but the exported .txt file is empty.

The keylog must be exported after the HTTPS connection is established.

The ability to inspect the full HTTP payload from PCAPdroid is now implemented, check out #107 (comment) .

A lock icon now reports the state of the decryption.

• gray closed: decryption is in progress
• gray open: the connection is not encrypted
• green open: the connection was decrypted
• orange closed: decryption not supported (e.g. QUIC or encrypted protocol like telegram/whatsapp)
• red closed: the decryption failed (e.g. due to a TLS error)

from pcapdroid.

Great! I think LittleProxy should be run as a thread in PCAPdroid and communicate with it via the socks5 interface (will I will take care of the socks5 migration asap). This will allow PCAPdroid to carry on the TLS decryption asynchronously while processing the network packets in parallel.

I've added a roadmap for this feature in the issue description above, we can work on a branch until it is complete. You can proceed with the integration. Once the embedded LittleProxy works correctly, I will take care of the final step to show the decrypted data in PCAPdroid/in the PCAP. For this I will need a call API in LittleProxy-mitm to read the decrypted data back in PCAPdroid. The API should expose: the decrypted data byte[], the direction of the data (from/to the internet) and the connection identifier (the 5 tuple, or at least proto, dst_ip, src_port, dst_port)

from pcapdroid.

I have checked the sizes after shrinking:

• BouncyCastle is just using 261 kb
• guava was reduced to 146 kb
• netty is 812 kb
• LittleProxy-mitm itself is 71 kb

from pcapdroid.

Filtering the HTTP request seems to be possible with LittleProxy. I hope the filter is the one seeing the plain text. How will you map the plain text to the original requests? I can record the plain text, but somehow then later you have to know which plain text corresponds to which encrypted text.

from pcapdroid.

I have checked the sizes after shrinking

1.3 MB is a good result, I think we can settle with this. Great work!

Filtering the HTTP request seems to be possible with LittleProxy. I hope the filter is the one seeing the plain text. How will you map the plain text to the original requests? I can record the plain text, but somehow then later you have to know which plain text corresponds to which encrypted text.

The idea is the following: when LittleProxy is started I will register a callback (via a java interface). Whenever the proxy decrypts a packet, it will call the callback with the decrypted data and the connection information (original 5 tuple). With the 5 tuple I can retrieve the original connection and set the plain text data.

In b029895 I've implemented the SOCKS5 proxy client. This replaces the old "tunnel" mode. You should now be able to redirect the TCP packets to LittleProxy by using SOCKS5.

Edit: another step is required in order to make this work with an embedded LittleProxy. Since LittleProxy will make connections on his own, we must exclude these connections from the VPN, otherwise they will go back into the VPNService in a loop. To do this, the VpnService protect api must be called on every socket open by LittleProxy. In order to keep LittleProxy clean, I'd suggest to do this by registering a callback at LittleProxy startup via a java interface. Of course I'm assuming that LittleProxy provides an api to extract the raw socket from netty.

from pcapdroid.

I will have a look into this. My first concern is if LittleProxy is actually supporting SOCKS5. I'm not familiar with the protocol, but what I see is that it needs a HTTP CONNECT call to trigger the proxy.

from pcapdroid.

Whatever is supported by conscrypt using the Java SSL engine can be supported. If this is not on the roadmap of both support will only be possible with external SOCKS based proxy tools supporting quic.

from pcapdroid.

I assume that it only handles TCP, so the protocol is not needed. The direction of the data is determined by the callback used (onForwardRequest/onForwardResponse), so we should now have everything needed in the log handler.

from pcapdroid.

I'm waiting for the next release of it and then will create a pull request draft.

from pcapdroid.

I'm waiting on chhsiao90/nitmproxy#82 If the socket must be protected this seems to be a requirement, if there is no reaction on this, I could look into this. The problem is that the socket identifier must be a long, but this identifier is not revealed by netty.

from pcapdroid.

With 6f42015 the built-in decryption is now integrated. The decryption requires the installation of the PCAPdroid-mitm addon as a separate apk. I've decided to separate it from the PCAPdroid app because of the size (the addon apk is about ~36 MB) and because it relies on the third-party chaquopy gradle plugin, which is closed source and it has build-time dependencies which complicate the build.

This is just an initial integration and there are a lot of things to improve.

from pcapdroid.

Is the chaquopy approach raising any security concerns? I.e. because it is closed source, is it safe to used it without any risk of hidden code?

These concerns are legit for any closed source software. chaquopy is quite a complicate plugin so it's not easy to review it, but given that it widely known, it has a public github page for issues and a clear monetization plan, I think we can trust it. One drawback though is that the license code cannot be make public, so people trying to build the PCAPdroid-mitm module from source will run in "demo" mode.

In short, it does its job and it does it well, although it's not open source. Unless other equivalent foss solutions are implemented, this is what we have now and I'm very satisfied with the result.

from pcapdroid.

Thank you for this useful package!

I have a question which, I guess, is similar to the one by @kenduq. How does one decrypt the PCAP file with the help of the new plugin? I installed the plugin but I couldn't see any changes in the app itself. Are decryption keys saved separately somewhere?

from pcapdroid.

In latest debug version i can see the app crashing and also hanging when "Full payload" toggle is turned on.

Should be fixed in fe26ecc. New debug apk.

from pcapdroid.

Might it be possible to add an option to drop UDP traffic on port 443? This usually prevents apps from using QUIC, and instead makes these apps fallback to TCP

I read that this fallback behavior is implemented by chrome. Is this part of the standard or it only applies to some apps? Please provide more info in a separate issue and also possibly an example on how it helps with decryption

Not sure if this is part of a standard. I guess some firewall configurations don't allow UDP on port 443, and thus a fallback is usually needed.

From my personal experience analysing thousands of Android apps, blocking UDP on port 443 works pretty well.

I've opened a new issue #213.

There might be some complications with some Private DNS solutions

Private DNS should user TLS on port 443. Do any of them use UDP on port 443?

Actually, no. I just checked that. Just wasn't sure!

EDIT: This has been added to PCAPdroid now.

from pcapdroid.

With the purpose of minimizing the dependencies, here is a quick overview of the LittleProxy-mitm depedencies (max depth 3):

- koh-osug/LittleProxy-mitm
  - bcpkix-jdk15to18
  - bcprov-jdk15to18
  - conscrypt-openjdk-uber
  - commons-io
  - slf4j-log4j12
  - mrog/LittleProxy
    - guava
    - commons-lang3
    - netty-all
    - barchart-udt-bundle (UDT transport protocol)
    - slf4j-api

@koh-osug can you please clarify the need for the additional dependencies of LittleProxy-mitm over LittleProxy? E.g. jdk15to18 and conscrypt. Regarding the project structure, I think it's better to keep LittleProxy-mitm in a separate code base, unless it requires a tight integration with PCAPdroid.

from pcapdroid.

I'm investigating this one as alternative: https://github.com/chhsiao90/nitmproxy
It supports socks and can also be used for SMTP / IMAP with TLS then.

from pcapdroid.

I had an issue with nitmproxy because the addresses received from SOCKS have been IP addresses, no hostnames anymore. I added a handler to extract the Server Name Indication and this looks good now. I created a fork and check its stability. LittleProxy might be maintained, but it is focusing too much on HTTP, nitmproxy supports more protocols and has the MITM feature is included. I have tested in under Linux and by connecting from Android to it.

from pcapdroid.

Have you tested it with TLS 1.3 and HTTP2? A SOCKS proxy opens up more possibilities so I also encourage this choice.

from pcapdroid.

I have tested the page "https://tools.keycdn.com/http2-test" with FF usign the HTTP2 protocol. The web page is working.
I used the web page https://resources.duda.co/tls-1-3 to check for TLS 1.3. This was also working.
Maybe some unit test could be added, too.

from pcapdroid.

Great! Please provide a pull request/branch to review when ready! If you want a more interactive technical discussion you can reach me on the telegram group

from pcapdroid.

Sure, will do. I have an issue with a specific domain. I'm in contact with the developer and I'm analyzing this. It is not a problem in a PC environment, but the same code is not working on Android. Most likely the Android SSLEngine stack is behaving differently. But the website seems to use a strange combination of TLS protocols, I could not detect this with any other web page.

from pcapdroid.

The nitmproxy can now transparently detect HTTP/HTTPS and supports also at least Android 8. I have also some older phones which I can check. A new API was added to plugin loggers I will add an AIDL interface around this so that PCAPdroid can call this.

from pcapdroid.

The android emulator is a convenient tool to test stuff on different SDK versions and devices. I see that the API is HTTPS oriented, does it provide a way to handle non HTTPS traffic? Also I think it could be better to run the proxy as a thread in PCAPdroid, to save IPC complexity and overhead.

from pcapdroid.

OK, not having it as separate app makes it simpler. Non HTTP is still on the TO DO list.

from pcapdroid.

Hi!
I am very excited to see that "Built-in TLS Decryption "!
In addition to supporting TLS1.3, have you considered supporting the quic protocol? It will be very common in the future.

from pcapdroid.

@emanuele-f Hi emanuele! Any update on this topic?

from pcapdroid.

@koh-osug any update on this? Can I help with the tests?

from pcapdroid.

Sorry, taking longer than expected. I tested SMTP, IMAP and HTTPS. Looks good. You can test the latest version from it from the command line, too: https://github.com/chhsiao90/nitmproxy
But it was necessary to include a dependency to Conscrypt. Android is using it internally, but only later Android versions are supporting ALPN which is necessary for HTTP/2. May idea to use BouncyCastle does not work, because BouncyCastle does not supporting TLS renegotiation which is used by some pages. This means that if all binaries of Conscrypt are included 8 MB are added. If different versions are created, e.g. supporting x86, armv7, armv8 as separate packages only 2,x MBs should be needed in addition.
The loggers might not yet be good enough for the purpose of PCACdroid, see HTTPListener and FowradListener, used for non HTTP. The request, the response, the data are logged with different callbacks, it might be problematic to assemble it again, I assuem to need to source, destination IP and port together with the data in one chunk.

from pcapdroid.

PCAPdroid has minSdkVersion 21, on which android version the built-in Conscrypt provides ALPN support (links appreciated)? The loggers callbacks can be adapted at any time (e.g. via a fork), that's not a problem.

Ps. Here is the link to for BouncyCastle missing TLS renegotiation support: bcgit/bc-java#593

from pcapdroid.

The APLN support requires a backport from a later Java 9 version to Android (OpenJDK 9+181). The important method is getApplicationProtocol. It is still missing in Android 9 but exists in Android 10, i.e. all version below Android 10 have to use a bundled Conscrypt version.
Maybe it would be possible to remove support for HTTP/2 from nitmproxy for Android versions < 10, I have not yet tested it, because the project is focusing on Desktop Java so far.

from pcapdroid.

Thanks for the clarification. 2 MB overhead is ok, but 8 seems too much to me. Anyway, please note that if BouncyCastle is integrated into PCAPdroid, it must meet the f-droid policy, so it is either built as a submodule or it is downloaded from a trusted repository.

from pcapdroid.

I have raised a pull request in nitmproxy: chhsiao90/nitmproxy#73 to provide the connection context with the log handler. source/dest IP + port + transmitted data would be sufficient, I assume?

from pcapdroid.

New nitmproxy version was released, I started with the gradle integration and shrinking the apk size.

from pcapdroid.

I added a draft: #87 What is the netxt step to integrate the nitmproxy listeners?

from pcapdroid.

Thanks! If that's not an issue, please base your branch on dev rather than master. I will merge dev into master just before the release to allow other users to test it and update the translations. Regarding the integration, I will need any useful info in order to integrate your libraries, e.g. on how to start the proxy and examples on how to integrate with it. If you already know what to modify in PCAPdroid, you can provide a initial integration, otherwise I can integrate it and commit into your branch.

from pcapdroid.

If you can point me to the integration point where the data, IP address, port, etc., is read and written to the pcap I can start to develop a stub.

from pcapdroid.

• The proxy should be started/stopped as a thread in CaptureService.java. You can check how the mHttpServer is handled to do the same.
• For the time being, you can direct the packets to the proxy by configuring the SOCK5 proxy into the PCAPdroid settings. Later we will add a dedicate option for this built-in proxy. With this setup you should have a working integration, but without the ability to get the packets payload data.
• (edit) You will need to call the protect function as explained in #57 (comment) to make this work.
• The last step is to show the decrypted payload data in PCAPdroid by using the callbacks. We can check this later.

from pcapdroid.

Regarding the protect method, I think BackendChannelBootstrap.java should be modified to allow the call of an external function call after connection (e.g. protect) as shown in https://stackoverflow.com/questions/43463044/protecting-socket-when-using-netty4-on-android-with-vpnservice

from pcapdroid.

I have added an issue to chhsiao90/nitmproxy#82 for the protect method.

from pcapdroid.

Good, it's better to have a generic callback which is called every time a connection is open. In our case we will call protect, other projects can use it for other stuff.

from pcapdroid.

@kaoh any update on this? It would be great to have this feature in the next PCAPdroid major release.

from pcapdroid.

I'm looking again at this problem as this feature is something users are demanding. The Java libraries evaluated so far are not actively developed and in my opinion they are not mature enough to carry on this task.

The only tool which I trust can do the job is mitmproxy. In #123 I've implemented a plugin which can pass the HTTP and websockets decrypted data to PCAPdroid. My idea to implement the built-in decryption is to bundle mitmproxy with its dependencies so that PCAPdroid can start/stop it at will and receive the decrypted data. This may either be implemented directly in PCAPdroid or into a separate app. The only obstacle to this is python. It seems like building python for Android is not easy and most projects on this matter are now abandoned.

The most promising one seems to be chaquopy, although closed source. I've start experimenting with it but faced dependency issues. Another candidate is python-for-android.

In short, I'm willing to implement a very basic app with mitmproxy which can be controlled by PCAPdroid. I'm opting for a separate apk right now both for convenience and because in my tests bundling python with mitmproxy requires about 10 MB of overhead per ABI (so x4) in the apk size.

from pcapdroid.

These are good news. Is the chaquopy approach raising any security concerns? I.e. because it is closed source, is it safe to used it without any risk of hidden code?

from pcapdroid.

So for now, is there any option to produce decryptable or decrypted pcap file for analysis in wireshark?
I tried to install PCAPdroid-mitm app on my emulator but emulator can not see the app. Install was via adb install .apk

from pcapdroid.

I tried to install PCAPdroid-mitm app on my emulator but emulator can not see the app

This is not released yet. You either need to build PCAPdroid from source (dev branch) or install a pre-built debug apk. Here is a pre-built apk for the latest commit. Then you should enable the "TLS decryption (preview)" option in the PCAPdroid settings.

How does one decrypt the PCAP file with the help of the new plugin?

This is still a work in progress, at the moment the app shows the decrypted HTTP request data (see this screenshot). I'm implementing the ability to show the full HTTP request/response in the app. See below for decryption keys.

Are decryption keys saved separately somewhere

I've just added (in 12727bd) the ability to export the decryption keys, to be used in Wireshark. Be sure to install the latest debug apk (link above), then you can export them (the capture must be running):

This link explains how to load them in Wireshark. A more practical way to get a decryptable pcap will be available with #185

from pcapdroid.

Tried to export SSL keylog file but the exported .txt file is empty.
Option of showing the full HTTP request/response in the app works fine.

from pcapdroid.

In latest debug version i can see the app crashing and also hanging when "Full payload" toggle is turned on.

from pcapdroid.

Consider also creating whitelist of apps that may be captured.

I guess then the whitelist structure may be like that:

package.name, confirmed_version

I don't know what's more reliable for version detection - versionCode or versionName, just because probably average user won't be able to extract versionCode, but versionName finally just a string, which anybody can fill with anything.

i.e. when loading list of target apps you can mark them:

• everything with targetSdk less than android 7 will be "green" or somehow marked as "decryptable"
• apps which are detected in whitelist and with versionName/versionCode lower/equal to value from whitelist also become "green"
• apps which are detected from whitelist by package name, but with version newer than confirmed in whitelist become "orange", or somehow "may be decryptable"
• others will be "red" or somehow "unconfirmed"

The idea is that amount of directly decryptable apps will be small and finally they won't update each day. So there will be added software confirmed by community, so I'm not sure that this list will be updated very often. I guess also it's worth to make minimal in-app update of this list to once per week or even less (but with ability to manually update).

Also one more notice is to avoid adding to list ROM-specific apps, i.e. something with package com.android.browser and etc. Just because depending on ROM it may be completelly different apps

from pcapdroid.

Today I tested the beta TLS decryption with a patched Signal apk and it worked in PCAPdroid. I could see decrypted TLS packets. But in wireshark via the http server and also after saving the pcap in pcapdroid and opening this file afterwards in wireshark I could only see encrypted TLS packets.
Is saving or serving decrypted packages via http not implemented yet?
Is there currently a way to extract the decrypted pcap?

Edit: I noticed that I can export the SSLKEYLOG. However that crashes the app with the debug release. If I see correctly this should be fixed with 6854284
Could you provide a new debug release?

from pcapdroid.

Could you provide a new debug release?

Try this one. The keylog file is now exported when the capture is stopped. If you still experience the crash, please report the stack trace

from pcapdroid.

Could you provide a new debug release?

Try this one. The keylog file is now exported when the capture is stopped. If you still experience the crash, please report the stack trace

Thank you very much and also for the fast response. Worked like a charm.
Might I suggest including the sslkeylog in the pcap that can be saved in the app and also the one streamed via http/udp?
https://wiki.wireshark.org/TLS#embedding-decryption-secrets-in-a-pcapng-file

from pcapdroid.

Thank you very much and also for the fast response. Worked like a charm. Might I suggest including the sslkeylog in the pcap that can be saved in the app and also the one streamed via http/udp?

This will be implemented in #185

from pcapdroid.

This is a great addition to PCAPdroid, and I look forward to using it for my research.

Might it be possible to add an option to drop UDP traffic on port 443? This usually prevents apps from using QUIC, and instead makes these apps fallback to TCP.

There might be some complications with some Private DNS solutions -- in this case, one might just let the user know that this must be disabled?

from pcapdroid.

Might it be possible to add an option to drop UDP traffic on port 443? This usually prevents apps from using QUIC, and instead makes these apps fallback to TCP

I read that this fallback behavior is implemented by chrome. Is this part of the standard or it only applies to some apps? Please provide more info in a separate issue and also possibly an example on how it helps with decryption

There might be some complications with some Private DNS solutions

Private DNS should user TLS on port 443. Do any of them use UDP on port 443?

from pcapdroid.