If I run all commands and replace localhost with the domain of my server. Is it routing it external so I can configure it to download the certificates? Also does this work on cellular?

Trying to get this running per the instructions in the readme. Its failing on Start. I had to make one modification to the code that I believe to be unrelated. Lines 8 and 9 of adblock.py have been modified as follows:
#import re2
import re as re2

This was required as re2 will not compile for raspberry pi.
Platform RPi 4, 8gig
Raspian Buster, 32bit

pi@servinator:~/mitm-adblock $ ./go -d

• Starting proxy server on port 8118...
• Dumping data to flows/log-1592095799.flows...
  Traceback (most recent call last):
  File "/home/pi/.local/bin/mitmdump", line 6, in
  from mitmproxy.tools.main import mitmdump
  File "/home/pi/.local/lib/python3.7/site-packages/mitmproxy/tools/main.py", line 22, in
  from ._main import * # noqa
  File "/home/pi/.local/lib/python3.7/site-packages/mitmproxy/tools/_main.py", line 17, in
  from mitmproxy import proxy
  File "/home/pi/.local/lib/python3.7/site-packages/mitmproxy/proxy/init.py", line 2, in
  from .root_context import RootContext
  File "/home/pi/.local/lib/python3.7/site-packages/mitmproxy/proxy/root_context.py", line 4, in
  from mitmproxy.proxy import protocol
  File "/home/pi/.local/lib/python3.7/site-packages/mitmproxy/proxy/protocol/init.py", line 36, in
  from .http2 import Http2Layer
  File "/home/pi/.local/lib/python3.7/site-packages/mitmproxy/proxy/protocol/http2.py", line 7, in
  from h2 import connection
  File "/home/pi/.local/lib/python3.7/site-packages/h2/connection.py", line 23, in
  from .events import (
  File "/home/pi/.local/lib/python3.7/site-packages/h2/events.py", line 14, in
  from .settings import ChangedSetting, _setting_code_from_int
  File "/home/pi/.local/lib/python3.7/site-packages/h2/settings.py", line 25, in
  class SettingCodes(enum.IntEnum):
  File "/home/pi/.local/lib/python3.7/site-packages/h2/settings.py", line 60, in SettingCodes
  ENABLE_CONNECT_PROTOCOL = SettingsFrame.ENABLE_CONNECT_PROTOCOL
  AttributeError: type object 'SettingsFrame' has no attribute 'ENABLE_CONNECT_PROTOCOL'

When I'm routing Youtube traffic through the ad-blocker it buffer a ton. It kinda works in bursts. I think it's something to do with concurrency?

I'm running it on Ubuntu Server 22.04 with 8 GB RAM and a 4c/4t CPU.
Am I doing something wrong or is this expected behavior?

Thanks!

OS Debian 9.0
mitmproxy 0.18.2-6
Python 2.7.13

# ./go --help
* Starting proxy server on port 8118...
Loading script: adblock.py
Script error: Traceback (most recent call last):
  File "adblock.py", line 10, in <module>
    from mitmproxy.protocol.http import HTTPResponse
ImportError: cannot import name HTTPResponse

I know that this codebase appears to block based on network traffic by useing regexes to filter out URLs that are associated with ad serving, but is there, in this code base or any other, references for how one would identify elements on pages, once rendered, that would need to be removed from the page in order to drop all ads off a page's rendered DOM?

Get this error message if a browser, in my case Firefox on a Debian (testing) machine tries to communicate with mitmproxy/mitmdump

File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3/dist-packages/mitmproxy/script/concurrent.py", line 27, in run fn(*args) File "adblock.py", line 77, in request if rules.should_block(req.url, options): File "/usr/local/lib/python3.7/dist-packages/adblockparser/parser.py", line 346, in should_block if self._is_blacklisted(url, options): File "/usr/local/lib/python3.7/dist-packages/adblockparser/parser.py", line 363, in _is_blacklisted self.blacklist_with_options File "/usr/local/lib/python3.7/dist-packages/adblockparser/parser.py", line 380, in _matches if general_re and general_re.search(url): File "pattern.pxi", line 635, in re2.PythonRePattern.search OverflowError: Python int too large to convert to C ssize_t

-> Got this issue while starting mitmdump via ./go (but there's no difference with the -c parameter)
-> Every request leads to this error and the browser timeouts afterwards.
-> Running mitm-adblock on Debian Buster in a LXC on a Turris Omnia.
-> for testing purposes I'm just using the easylist from 20th of May.

EDIT: Rewording and adding info

Its slow and it only blocks youtube ads most of the time and when a youtube ad is blocked, youtube tries to display the video then it shows a black video with no audio then gives the skip ad option to skip it

Building wheel for pyre2 (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/opt/[email protected]/bin/python3.9 /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/tmpo7ongjl1
cwd: /private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-req-build-nn6svfgl
Complete output (95 lines):
running bdist_wheel
running build
running build_ext
-- The CXX compiler identification is AppleClang 12.0.5.12050022
-- The C compiler identification is AppleClang 12.0.5.12050022
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find pybind11 (missing: pybind11_DIR)
-- Fetching pybind11 from github
-- pybind11 v2.6.1
-- Found PythonInterp: /usr/local/Cellar/[email protected]/3.9.7/bin/python3.9 (found version "3.9.7")
-- Found PythonLibs: /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/lib/libpython3.9.dylib
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Performing Test HAS_FLTO_THIN
-- Performing Test HAS_FLTO_THIN - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Cython: /private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-build-env-ez2pl7jy/overlay/bin/cython
-- Could NOT find re2 (missing: re2_DIR)
-- Trying PkgConfig
CMake Error at /private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-build-env-ez2pl7jy/overlay/lib/python3.9/site-packages/cmake/data/CMake.app/Contents/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
Call Stack (most recent call first):
/private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-build-env-ez2pl7jy/overlay/lib/python3.9/site-packages/cmake/data/CMake.app/Contents/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
/private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-build-env-ez2pl7jy/overlay/lib/python3.9/site-packages/cmake/data/CMake.app/Contents/share/cmake-3.21/Modules/FindPkgConfig.cmake:70 (find_package_handle_standard_args)
src/CMakeLists.txt:44 (find_package)

-- Configuring incomplete, errors occurred!
See also "/private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-req-build-nn6svfgl/build/temp.macosx-11-x86_64-3.9/CMakeFiles/CMakeOutput.log".
See also "/private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-req-build-nn6svfgl/build/temp.macosx-11-x86_64-3.9/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in
main()
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 248, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 221, in build_wheel
return self._build_with_temp_dir(['bdist_wheel'], '.whl',
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
self.run_setup()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 150, in run_setup
exec(compile(code, file, 'exec'), locals())
File "setup.py", line 116, in
setup(
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "setup.py", line 108, in build_extension
subprocess.check_call(
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-req-build-nn6svfgl', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/6q/k62zqc0x2_1gdjsb77y6q0cr0000gn/T/pip-req-build-nn6svfgl/build/lib.macosx-11-x86_64-3.9/', '-DPYTHON_EXECUTABLE=/usr/local/Cellar/[email protected]/3.9.7/bin/python3.9', '-DSCM_VERSION_INFO=0.3.6', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-GNinja']' returned non-zero exit status 1.

ERROR: Failed building wheel for pyre2
Failed to build pyre2
ERROR: Could not build wheels for pyre2 which use PEP 517 and cannot be installed directly

I would prefer not BLOCK hosts with NX or non-routable address like 0.0.0.0

I want targeted ad hosts to be redirected to an ip address. This ip address lives on the same VPS as mitmproxy. Or is it a MitmProxy feature (seems yes)?

This will answer for ANY host, with the mitmproxy generated certs for the targeted host, and ALWAYS return http(s) code 200 for any requested content: image, video, document -- while serving a modicum of content to satisfy the ad receiver

Targeted hosts are redirected by a third party DNS service devices employ via DoH (DNS over HTTPS) or DoT.

muahahaha

Firstly, thank you very much epitron and davizucon for this project. It's increasingly relevant these days.

Secondly, I'm trying to run Adblock.py with MITMProxy 4.0.4 under python 3.7 and have been encountering issues. As a complete Python noob, I've done a couple of very minor modifications to address error messages as they come up:

• changed "import mitmproxy.protocol.http" to "mitmproxy.http" to fix HTTPResponse import error.
• deleted "from netlib.odict import ODictCaseless" to fix import error message
• changed "def request(context, flow):" to "def request(flow):" to fix "TypeError: request() missing 1 required positional argument: 'flow'"

I'm still left with the error message:

  File "adblock.py", line 79, in request
    if rules.should_block(req.url, options):
NameError: name 'rules' is not defined

I'm not sure how to proceed from there, after everything I've tried to somehow define the 'rules' object. Also not sure if the minor changes I've detailed above have broken anything.

Thanks again.
Cheers.

Hi. You are being consumed (and slightly refactored) by a new project using AGPL-3. Can you add a license file (either same or similar)? I didn't see anything buried in the source tree, but maybe I missed something... You can see the latest (alpha) code here: https://github.com/freepn/freepn-proxy

I included the whole adblock source tree (from an updated branch) as a a git subtree so all the commits are preserved.

I have tried setting it up on a linux machine but i get errors when installing python dependencies.

Please make a documentation for it