kaliiiiiiiiii / cdp-socket Goto Github PK
View Code? Open in Web Editor NEWsocket for handling chrome-developer-protocol connections
License: MIT License
socket for handling chrome-developer-protocol connections
License: MIT License
High memory consumption There is something wrong with the class, if a program runs in a loop and downloads data non-stop, the program's memory consumption increases every second.
Sample program:
from selenium_driverless import webdriver
from selenium_driverless.types.by import By
import asyncio
async def main():
options = webdriver.ChromeOptions()
async with webdriver.Chrome(options=options) as driver:
while True:
await driver.get('https://pudelek.pl')
await driver.sleep(0.5)
source = await driver.page_source
await asyncio.sleep(3)
asyncio.run(main())
After several minutes of running the program, the memory usage of the python.exe process increases by several MB, and after several dozen minutes even up to 2 GB.
Python 3.10.6
Windows 11
Chrome 119.0.6045.124
"C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\webelement.py", line 382,
in text return await self.get_property("textContent") File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\webelement.py", line 371,
in get_property return await self.execute_script(f"return obj[arguments[0]]", name) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\webelement.py", line 913,
in execute_script return await self.exec(script, *args, max_depth=max_depth, serialization=serialization, File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\deserialize.py", line 202,
in exec res = await self.exec_raw(script, *args, max_depth=max_depth, File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\deserialize.py", line 152,
in exec_raw raise e File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\deserialize.py", line 145,
in exec_raw res = await self.target.execute_cdp_cmd("Runtime.callFunctionOn", args, timeout=timeout) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\types\target.py", line 1125,
in execute_cdp_cmd result = await self.socket.exec(method=cmd, params=cmd_args, timeout=timeout) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\socket.py", line 85,
in exec res = await asyncio.wait_for(self._responses[_id], timeout=timeout) File "C:\Users\R\AppData\Local\Programs\Python\Python310\lib\asyncio\tasks.py", line 445,
in wait_for return fut.result()
cdp_socket.exceptions.CDPError: {'code': -32600, 'message': 'Duplicate id in protocol request'}
You suggested possible issue is here:
CDP-Socket/src/cdp_socket/socket.py
Lines 71 to 77 in 4813479
Full error:
Task was destroyed but it is pending!
task: <Task pending name='Task-247'
coro=<SingleCDPSocket._handle_callback.<locals>.async_handle() done,
defined at C:\Users\PC\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\socket.py:178>
wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]>>
import asyncio
from selenium_driverless import webdriver
from selenium_driverless.scripts.network_interceptor import NetworkInterceptor
options = webdriver.ChromeOptions()
async def task(driver):
await driver.get('https://github.com', wait_load=False)
async def RequestBlocker():
async with await webdriver.Chrome(options=options) as driver:
while True:
await driver.get('https://google.com')
driver3 = await driver.new_window('tab')
async with NetworkInterceptor(driver3) as interceptor:
await task(driver3)
await asyncio.sleep(2)
async for data in interceptor:
if data.request.url == "https://collector.github.com/github/collect":
await data.fail_request('TimedOut')
print('Request Blocked')
break
await driver3.close()
asyncio.run(RequestBlocker())
Minimal code to reproduce:
import asyncio
from selenium_driverless import webdriver
from selenium_driverless.scripts.network_interceptor import NetworkInterceptor
async def main():
async with await webdriver.Chrome() as driver:
while True:
tab = await driver.new_window('tab')
async with NetworkInterceptor(tab) as interceptor:
asyncio.ensure_future(tab.get('https://httpbin.org/get', wait_load=False))
async for data in interceptor:
await data.fail_request('TimedOut')
print('Request Blocked')
break
await tab.close()
asyncio.run(main())
OS: Windows 10 22H2
Chrome Version: 124.0.6367.62
selenium_driverless.version: 1.9.1
cdp_socket.__version: 1.2.7
Python version: 3.11
EDIT by @kaliiiiiiiiii
File "/lib/python3.10/site-packages/cdp_socket/utils/utils.py", line 119, in launch_chrome
os.makedirs(data_dir_path, exist_ok=True)
PermissionError: [Errno 13] Permission denied: '/lib/python3.10/site-packages/cdp_socket/files/tmp'
CDP-Socket/src/cdp_socket/utils/utils.py
Lines 116 to 119 in 4bae35c
CDP-Socket/src/cdp_socket/utils/utils.py
Lines 60 to 61 in 4bae35c
quickfix:
def cdp_sock_path():
return os.environ.get("HOME", "/tmp") + "/.cache/cdp-socket"
similar issue: kaliiiiiiiiii/Selenium-Driverless#122
these belong to a separate file, something like requirements-dev.txt
Lines 5 to 7 in 9188210
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.