I am a software engineer especialist in web scraping and web automation.
📫 How to reach me
- on email [email protected]
- on telegram @heraldodev
Python client and types generator for the Chrome DevTools Protocol (CDP)
License: MIT License
I am a software engineer especialist in web scraping and web automation.
📫 How to reach me
Looking through pypi.org, I stumbled upon this package:
https://pypi.org/project/python-recentx/
Which seems to contain a copy of this project (python-cdp) under a different name. It looks suspicious to me. Is that PyPI package related to python-cdp project?
Hi,
All the python classes have the from_json
method. The ones that represent CDP data types also have a to_json
method, but the events don't have. I would like a complete coverage of json conversion for the python objects. I think the generation script can be modified to achieve this requirement.
At the moment, I don't see this to be a difficult task. I am happy to make the contribution myself.
GG for the project
async with target_session.wait_for(cdp.page.DomContentEventFired):
does not catch an event on tab navigation using a python types generated by cdpgen
.
last used protocol commit: ChromeDevTools/devtools-protocol@5caaeb9
but seems like this bug came a few commits before.
using a README.md example code:
import asyncio
# from pycdp import cdp # here is an import from default pycdp types
import cdp # here we imported cdp which was generated by cdpgen
from pycdp.browser import ChromeLauncher
from pycdp.asyncio import connect_cdp
async def main():
chrome = ChromeLauncher(
binary='/usr/bin/google-chrome', # linux path
args=['--remote-debugging-port=9222', '--incognito']
)
# ChromeLauncher.launch() is blocking, run it on a background thread
await asyncio.get_running_loop().run_in_executor(None, chrome.launch)
conn = await connect_cdp('http://localhost:9222')
target_id = await conn.execute(cdp.target.create_target('about:blank'))
target_session = await conn.connect_session(target_id)
await target_session.execute(cdp.page.enable())
await target_session.execute(cdp.page.navigate('https://chromedevtools.github.io/devtools-protocol/'))
# you may use "async for target_session.listen()" to listen multiple events, here we listen just a single event.
async with target_session.wait_for(cdp.page.DomContentEventFired):
dom = await target_session.execute(cdp.dom.get_document())
node = await target_session.execute(cdp.dom.query_selector(dom.node_id, 'p'))
js_node = await target_session.execute(cdp.dom.resolve_node(node))
print((await target_session.execute(cdp.runtime.call_function_on('function() {return this.innerText;}', js_node.object_id, return_by_value=True)))[0].value)
await target_session.execute(cdp.page.close())
await conn.close()
await asyncio.get_running_loop().run_in_executor(None, chrome.kill)
asyncio.run(main())
In Chrome 128 cdp.dom.get_document() freezes when network.enable().
We need to abstract event-loop dependent features so we can reuse common code between asyncio and twisted clients.
Is there an option to properly handle experimental (and not optional) fields (that may or may not be present in implementation as far as I understand that correctly)?
Currently it evaluates directly as json['field'] which throws 'key not found' exception, breaking the whole thing down.
after executing a cdpgen --browser-protocol devtools-protocol/browser_protocol.json --js-protocol devtools-protocol/js_protocol.json --output cdp/
command, script does not create a util.py
which used in cdp/accessibility.py
.
Used environment:
MacOS
Poetry
Here is my traceback:
Traceback (most recent call last):
File "/Users/turbokach/Dev/betbot_cdp/script.py", line 2, in <module>
import cdp
File "/Users/turbokach/Dev/betbot_cdp/cdp/__init__.py", line 6, in <module>
from . import (accessibility, animation, audits, background_service, browser, css, cache_storage, cast, console, dom, dom_debugger, dom_snapshot, dom_storage, database, debugger, device_orientation, emulation, event_breakpoints, fetch, headless_experimental, heap_profiler, io, indexed_db, input_, inspector, layer_tree, log, media, memory, network, overlay, page, performance, performance_timeline, profiler, runtime, schema, security, service_worker, storage, system_info, target, tethering, tracing, web_audio, web_authn)
File "/Users/turbokach/Dev/betbot_cdp/cdp/accessibility.py", line 12, in <module>
from .util import event_class, T_JSON_DICT
ModuleNotFoundError: No module named 'cdp.util'
Hello, I'm experimenting with this library and I'm trying to use the aiohttp-jinja2
package alongside it but the strict version locks make pip deny the installation.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
aiohttp-jinja2 1.6 requires aiohttp>=3.9.0, but you have aiohttp 3.8.3 which is incompatible.
I was wondering, is there a specific reason why the dependencies are version locked?
Currently pyproject.toml
contains:
[tool.poetry.dependencies]
python = "^3.8, <3.12"
deprecated = "1.2.9"
inflection = "0.4.0"
aiohttp = "3.8.3"
but for more compatibility across libraries would it be possible to change it to something like this?
[tool.poetry.dependencies]
python = "^3.8, <3.12"
deprecated = "^1.2.9"
inflection = "^0.4.0"
aiohttp = "^3.8.3"
If there aren't any specific reasons for this then I will make a merge request so the library is more flexible with other packages. Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.