Giter VIP home page Giter VIP logo

chrome-devtools-java-client's Introduction

Chrome DevTools Java Client

Description

Chrome DevTools Java Client is a DevTools client - in Java. (: It can be used for instrumenting, inspecting, debuging and profiling Chromium, Chrome and other Blink-based browsers. [1]

For more information on DevTools, see https://chromedevtools.github.io/devtools-protocol/.

v1.3.6 tested on Google Chrome Version 67.0.3396.87. Protocol files from dev-tools-protocol#1aa7b31ca7

v2.1.0 tested on Google Chrome Version 76.0.3809.100. Protocol files from dev-tools-protocol#e1fb93bd76

v3.0.0 tested on Google Chrome Version 86.0.4240.111. Protocol files from dev-tools-protocol#fcb68d10bc

v4.0.0 tested on Google Chrome Version 90.0.4430.212. Protocol files from dev-tools-protocol#987bbb1124

[1] https://chromedevtools.github.io/devtools-protocol/.

Usage

Add the following dependency to your pom.xml:

<dependency>
  <groupId>com.github.kklisura.cdt</groupId>
  <artifactId>cdt-java-client</artifactId>
  <version>4.0.0</version>
</dependency>

You can use following code, taken from, LogRequestsExample:

package com.github.kklisura.cdt.examples;

import com.github.kklisura.cdt.launch.ChromeLauncher;
import com.github.kklisura.cdt.protocol.commands.Network;
import com.github.kklisura.cdt.protocol.commands.Page;
import com.github.kklisura.cdt.services.ChromeDevToolsService;
import com.github.kklisura.cdt.services.ChromeService;
import com.github.kklisura.cdt.services.types.ChromeTab;

/**
 * Log requests example with DevTools java client.
 *
 * <p>The following example will open chrome, create a tab with about:blank url, subscribe to
 * requestWillBeSent event and then navigate to github.com.
 *
 * @author Kenan Klisura
 */
public class LogRequestsExample {
  public static void main(String[] args) {
    // Create chrome launcher.
    final ChromeLauncher launcher = new ChromeLauncher();

    // Launch chrome either as headless (true) or regular (false).
    final ChromeService chromeService = launcher.launch(false);

    // Create empty tab ie about:blank.
    final ChromeTab tab = chromeService.createTab();

    // Get DevTools service to this tab
    final ChromeDevToolsService devToolsService = chromeService.createDevToolsService(tab);

    // Get individual commands
    final Page page = devToolsService.getPage();
    final Network network = devToolsService.getNetwork();

    // Log requests with onRequestWillBeSent event handler.
    network.onRequestWillBeSent(
        event ->
            System.out.printf(
                "request: %s %s%s",
                event.getRequest().getMethod(),
                event.getRequest().getUrl(),
                System.lineSeparator()));

    network.onLoadingFinished(
        event -> {
          // Close the tab and close the browser when loading finishes.
          chromeService.closeTab(tab);
          launcher.close();
        });

    // Enable network events.
    network.enable();

    // Navigate to github.com.
    page.navigate("http://github.com");

    devToolsService.waitUntilClosed();
  }
}

For more examples, see examples.

Known-issues

API hangs (ie when printing PDFs)

What: If you're using cdt-java-client before version 2.1.0 for printing to PDF or using API which requests large amount of data, API may hang if the requesting data exceeds 4MB.

Why: This is due to underlying WebSocket library having 4MB buffer for receiving data from browser.

How to fix: With the version 2.1.0 and above, this buffer was increased to 8MB and can be further increased if necessary by setting the appropriate configuration property.

Debugging chrome

In order to debug chrome when using this library, set the logger com.github.kklisura.cdt.launch.chrome.output to DEBUG level. See ChromeLoggingExample for more information. Be sure to remove or turn the logger off, when done.

Running unit tests

make verify

Sonar analysis

make sonar-analysis

Download latest protocol

Run following:

make download-latest-protocol

This will download browser_protocol.json and js_protocol.json (protocol definitions) from https://github.com/ChromeDevTools/devtools-protocol repo.

Update protocol (generate Java files from protocol definitions)

Make sure you have correct or latest protocol definitions. See Download latest protocol on how to update protocol definitions to latest version.

Run following:

make update-protocol

This will build the tools for parsing and generating Java files, cdt-java-protocol-builder project. The input for this tool are protocol definitions files: browser_protocol.json and js_protocol.json. The generated Java files will be present in cdt-java-client project. After building Java files, the cdt-java-client will be compiled. If everything goes successfully, consider the protocol updated. :)

Updating copyright license header

To go over each module and each source java file to update copyright license header, run:

make update-copyright-license-header

License

Chrome DevTools Java Client is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

chrome-devtools-java-client's People

Contributors

dadza avatar ddrozdov avatar dependabot[bot] avatar dunaj avatar kklisura avatar konikos avatar rjkip avatar sinaa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

chrome-devtools-java-client's Issues

Actions on page elements

Hi and thanks for your work on this library. Is there a way to interact with the content of a page e.g. click on an element, find elements etc...? like with the following snippet for Selenium

WebElement el = driver.findElementById("dateDecided"); el.click(); el.submit();

Unable to set timezone using Emulation

I was trying out the Emulation interface to set the timezone in the Chrome tab. However, Either I am not using it properly or it doesn't work.

Emulation emulation = chromeDevToolsService.getEmulation();
emulation.setTimezoneOverride("inccu"); 

The string is taken from here as mentioned here

I verified using this code in chrome JS console. My machine is UTC timezone and I tried to change the timezone to Asia/Calcutta Asia/Kolkata using above API :

var date = new Date();
console.log(date);

The time still shows the same as the host OS

java.lang.IllegalStateException: Text message handler not found.

Every time I will new a WebSocketService Instance, add new addMessagehandler, finally close this websocketservice.Usually message will handle success, the exception blow will recurs once in hundreds

Exception:

09-23 15:42:43.809 [Grizzly(2)]  ERROR c.g.k.cdt.services.impl.WebSocketServiceImpl :190 | Error in web socket session.
java.lang.IllegalStateException: Text message handler not found. Session: 'TyrusSession{uri=ws://127.0.0.1:8088/debugProxy/platform, id='3b24ba39-788e-4e43-afe8-40212e42c996', endpointWrapper=TyrusEndpointWrapper{endpointClass=null, endpoint=com.github.kklisura.cdt.services.impl.WebSocketServiceImpl$1@4511f172, contextPath='ws://127.0.0.1:8088/debugProxy/platform', endpointPath=null, encoders=[CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.NoOpTextCoder, coder=org.glassfish.tyrus.core.coder.NoOpTextCoder@49a2bf04, type=class java.lang.String}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.NoOpByteBufferCoder, coder=org.glassfish.tyrus.core.coder.NoOpByteBufferCoder@50da60f9, type=class java.nio.ByteBuffer}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.NoOpByteArrayCoder, coder=org.glassfish.tyrus.core.coder.NoOpByteArrayCoder@4ef4f0c, type=class [B}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.ToStringEncoder, coder=org.glassfish.tyrus.core.coder.ToStringEncoder@4e3fa482, type=class java.lang.Object}], decoders=[CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$BooleanDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$BooleanDecoder@123978de, type=class java.lang.Boolean}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$ByteDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$ByteDecoder@11e71fc8, type=class java.lang.Byte}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$CharacterDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$CharacterDecoder@6f8eb5d7, type=class java.lang.Character}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$DoubleDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$DoubleDecoder@5ca61710, type=class java.lang.Double}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$FloatDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$FloatDecoder@5006bd2, type=class java.lang.Float}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$IntegerDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$IntegerDecoder@2cb02de5, type=class java.lang.Integer}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$LongDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$LongDecoder@368a15a9, type=class java.lang.Long}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.PrimitiveDecoders$ShortDecoder, coder=org.glassfish.tyrus.core.coder.PrimitiveDecoders$ShortDecoder@5af88aff, type=class java.lang.Short}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.NoOpTextCoder, coder=org.glassfish.tyrus.core.coder.NoOpTextCoder@5ca03489, type=class java.lang.String}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.NoOpByteBufferCoder, coder=org.glassfish.tyrus.core.coder.NoOpByteBufferCoder@7152f5fd, type=class java.nio.ByteBuffer}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.NoOpByteArrayCoder, coder=org.glassfish.tyrus.core.coder.NoOpByteArrayCoder@17939123, type=class [B}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.ReaderDecoder, coder=org.glassfish.tyrus.core.coder.ReaderDecoder@9224e5c, type=class java.io.Reader}, CoderWrapper{coderClass=class org.glassfish.tyrus.core.coder.InputStreamDecoder, coder=org.glassfish.tyrus.core.coder.InputStreamDecoder@54fb378d, type=class java.io.InputStream}]}}'.
	at org.glassfish.tyrus.core.TyrusEndpointWrapper.onMessage(TyrusEndpointWrapper.java:883)
	at org.glassfish.tyrus.core.TyrusWebSocket.onMessage(TyrusWebSocket.java:216)
	at org.glassfish.tyrus.core.frame.TextFrame.respond(TextFrame.java:139)
	at org.glassfish.tyrus.core.ProtocolHandler.process(ProtocolHandler.java:807)
	at org.glassfish.tyrus.client.TyrusClientEngine$TyrusReadHandler.handle(TyrusClientEngine.java:747)
	at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter$ProcessTask.execute(GrizzlyClientFilter.java:476)
	at org.glassfish.tyrus.container.grizzly.client.TaskProcessor.processTask(TaskProcessor.java:114)
	at org.glassfish.tyrus.container.grizzly.client.TaskProcessor.processTask(TaskProcessor.java:91)
	at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleRead(GrizzlyClientFilter.java:272)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
	at java.lang.Thread.run(Thread.java:748)

Headers on page.navigate

It doesn't seem possible to pass headers to page.navgiate, am I missing something, or is it simply not possible yet? (In my case Im specifically looking for the possibility of passing Basic Auth headers)

Use newer/unimplemented API functionality?

Is there an easy way to use an API which is currently not implemented in this library?

I'm interested in the Overlay.highlightNode but would like to call it with a selector instead of an ID. According to the documentation this is possible with the most recent version, but this library currently does not allow to give a selector:

image

If it makes sense I could alternatively see whether I can provide a PR.

PS: Thanks for this, it's very handy for me!

network.setCookie does not work

network.setCookie does not work or usage is not intended like this:

ChromeService chromeService = new ChromeServiceImpl(9222);
ChromeTab tab = chromeService.getTabs().get(0);
ChromeDevToolsService devToolsService = chromeService.createDevToolsService(tab);
Network network = devToolsService.getNetwork();
network.enable();
Boolean result = network.setCookie("some", "test", null, "example.com", "/", true, true, null, -1.0, CookiePriority.MEDIUM);

result is true.

network.getAllCookies();

returns added cookie, however when inspecting requests in the browser from the network tab, cookie is not present. cookie is also not present in Application / Storage / Cookies under example.com domain. There are other cookies present at example.com domain.

Not working with JDK 11 (SOLVED)

Hi, thank you for this wonderful project.
I've been using it for some month, but today I had to switch to JDK 11 and it stopped working.
After some research I was able to let it work again.
It was necessary to upgrade javassist to the latest version: javassist-3.24.1-GA
instead of the not working (with JDK 11) version: javassist-3.22.0-GA
which is referenced in the project pom.xml file
Just to let you know.
Thank you again.

Config for ChromeDevToolsService not used

When getting an instance of the ChromeDevToolsService from the ChomeService the provided configuration is not used.

chromeService.createDevToolsService(tab, devToolsConfig);

Instead of using the passed configuration, it always uses a new instance of the configuration, see:
https://github.com/kklisura/chrome-devtools-java-client/blob/master/cdt-java-client/src/main/java/com/github/kklisura/cdt/services/impl/ChromeServiceImpl.java#L179

As a workaround it is possible to set the config on the returned service with reflection.

printToPDF support

Nice work 👏

Would it be possible to support printToPDF I was looking into the code but couldn't figure out how to add this functionality.

can't handle with more than 6 concurrent requests

I have the following issue:
we use docker container with chromium(we add it with apk) to generate pdf. after i generate and download pdf through endpoint i close chrome tab via interceptor after complition. here is the following code which i invoke in interceptor

public void closeChromeTab(String url) {
final ChromeService service = ChromeServiceHolder.CHROME_SERVICE.getService();

    final List<ChromeTab> chromeTabs = service.getTabs()
            .stream()
            .filter(chromeTab -> chromeTab.getUrl().equals(url) || chromeTab.getUrl().equals(ABOUT_PAGE))
            .collect(Collectors.toList());

chrome.tabs.forEach(service::close tab)

it's work fine till i create <= 5 simultaneous requests. approximately when the load become 6-8 simultaneous request tabs stop closing (i see that in container with top command)

cannot evaluate com.sun.proxy.$proxy tostring()

I have imported "cdt-java-client" module in my java project and am trying to print pdfs.
But it receive the following exception in the debug mode.
No Exception is thrown in the logs and an empty pdf is generated.
Can someone please guide:

Method threw '.ChromeDevToolsInvocationException' exception. Cannot evaluate com.sun.proxy.$Proxy589.toString()

Return DOM elements from Javascript script.

Im running a simple Javascript which returns some DOM nodes. I can see that these DOM nodes are represented as hash maps for some reason in Java. Is there any way to address this, so that we have a proper DOM node representation in java for DOM nodes returned by Javascript.

Get screenshot of a particular element

Hi,

I see that chrome dev tools has an option in commands menu to take screenshot of a specific element. I didn't find this in any of the APIs provided by dev tools protocol. I tried using the Page.captureScreenshot with the required viewport that I want, but its not working when the element is outside of view-port and requires scroll. Requesting your help on, to take screenshots of elements outside of viewport (which are visible after desired scrolling).

Ability to access websocket directly

I am one of developers of the Selenoid project which is a Selenium implementation running browsers in Docker containers. In Selenoid we have a feature to provide access to Chrome devtools API. However compared to local Chrome startup - user don't need to manually launch Chrome with your client. Instead after launching Chrome with Selenium - user can directly access Chrome Devtools web socket. It would be great to provide some utility method or constructor to initialize your client having a direct ws:// URL.

I created an example of how we are using your client (this is mainly a copy-paste from your code). Basically the following code snippet works:

        String webSocketURL = "ws://some-url";
        WebSocketService webSocketService = WebSocketServiceImpl.create(new URI(webSocketURL));
        CommandInvocationHandler commandInvocationHandler = new CommandInvocationHandler();
        Map<Method, Object> commandsCache = new ConcurrentHashMap<>();
        ChromeDevToolsService devtools =
                ProxyUtils.createProxyFromAbstract(
                        ChromeDevToolsServiceImpl.class,
                        new Class[] {WebSocketService.class, ChromeDevToolsServiceConfiguration.class},
                        new Object[] {webSocketService, new ChromeDevToolsServiceConfiguration()},
                        (unused, method, args) ->
                                commandsCache.computeIfAbsent(
                                        method,
                                        key -> {
                                            Class<?> returnType = method.getReturnType();
                                            return ProxyUtils.createProxy(returnType, commandInvocationHandler);
                                        }));
        commandInvocationHandler.setChromeDevToolsService(devtools);

Having a public method doing the same in your code would be great. I could provide a respective PR. What do you think?

Backwards / forwards compatibility with Chrome

Could we clarify what the backwards compatibility is for this project with older versions of Chrome? It seems that this project needs updating reasonably regularly as the Chrome API changes, e.g. #60, but we use it on servers that may not have Chrome updated on the same frequency.

I'm also wondering if the library is, or whether we could make it, more robust in the face of future API, like new enum values (#60 again)?

CDT not handling Preflight InitiatorType/ResourceType

I started getting this error popping up often while handling network events. Seems to be a new InitiatorType/ResourceType thats not currently handled by the current code.

Error while processing event Network.requestWillBeSent
com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.kklisura.cdt.protocol.types.network.InitiatorType` from String "preflight": value not one of declared Enum instance names: [other, preload, script, SignedExchange, parser]
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.github.kklisura.cdt.protocol.events.network.RequestWillBeSent["initiator"]->com.github.kklisura.cdt.protocol.types.network.Initiator["type"])
	at com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1535)
	at com.fasterxml.jackson.databind.DeserializationContext.handleWeirdStringValue(DeserializationContext.java:910)
	at com.fasterxml.jackson.databind.deser.std.EnumDeserializer._deserializeAltString(EnumDeserializer.java:255)
	at com.fasterxml.jackson.databind.deser.std.EnumDeserializer.deserialize(EnumDeserializer.java:179)
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:287)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:287)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1608)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1292)
	at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.readJsonObject(ChromeDevToolsServiceImpl.java:355)
	at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.lambda$handleEvent$1(ChromeDevToolsServiceImpl.java:303)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Error while processing event Network.responseReceived
com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.kklisura.cdt.protocol.types.network.ResourceType` from String "Preflight": value not one of declared Enum instance names: [XHR, Media, WebSocket, Script, Stylesheet, Manifest, Other, Ping, EventSource, CSPViolationReport, Fetch, Document, TextTrack, SignedExchange, Font, Image]
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.github.kklisura.cdt.protocol.events.network.ResponseReceived["type"])
	at com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1535)
	at com.fasterxml.jackson.databind.DeserializationContext.handleWeirdStringValue(DeserializationContext.java:910)
	at com.fasterxml.jackson.databind.deser.std.EnumDeserializer._deserializeAltString(EnumDeserializer.java:255)
	at com.fasterxml.jackson.databind.deser.std.EnumDeserializer.deserialize(EnumDeserializer.java:179)
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:287)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1608)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1292)
	at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.readJsonObject(ChromeDevToolsServiceImpl.java:355)
	at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.lambda$handleEvent$1(ChromeDevToolsServiceImpl.java:303)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

profiler.takePreciseCoverage() is Failing with Chrome 78.0.3904.87

Chrome version 78.0.3904.87.

Start failing when

List scriptCoverages = profiler.takePreciseCoverage()

com.github.kklisura.cdt.services.exceptions.ChromeDevToolsInvocationException: Failed sending web socket message.
	at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(ChromeDevToolsServiceImpl.java:175)
	at com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(CommandInvocationHandler.java:87)

When Chrome crashes

Sometimes Chrome crashes, and yet there is no way to know it, devtools client just hangs waiting forever for a response that will never come. Is this the expected behaviour?

exception when running on Mac / Ubuntu

hi, hope anybody out there can help me with my first trial of the DevTools client on java. Unfortunately I'm not familiar with AccessController Classes - I get the following exception (moreless 1:1 the example on the page)

Thanks a lot
Martin

try {
			launcher = new ChromeLauncher();
			// Launch chrome either as headless (true) or regular (false).
			ChromeArguments args=null;
			if (browserparameters!=null&&browserparameters.size()>0)
			{
				args=ChromeArguments.defaults(false).additionalArguments(browserparameters).build();
				
			}else args=ChromeArguments.defaults(false).build();
			
			log.info("launcher args={} isalive={}",args,launcher.isAlive());
		    
		    chromeService = launcher.launch(args);
		    log.info("chromservice version={}",chromeService.getVersion());
		    // Create empty tab ie about:blank.
		    ChromeTab tab = chromeService.createTab();

		    // Get DevTools service to this tab
		    devToolsService = chromeService.createDevToolsService(tab);

		}  catch (Exception e) {
			StringWriter errors = new StringWriter();
			e.printStackTrace(new PrintWriter(errors));
			log.error("exception e: {} s: {}", e, errors);
			this.setAppstate(AppState.AS_ERROR);
			this.setLasterrormsg(e.toString());
		}

I'm using MacOS but same occurs if I run the app on ubuntu. Maybe relevant: in both ways I'm non root.

[2020-07-23 07:23:52,317]-[Grizzly(1)] INFO  com.github.kklisura.cdt.services.impl.WebSocketServiceImpl - Connected to ws ws://localhost:59668/devtools/page/C2468662CE12D1DCC3790331D02EA3AE
[2020-07-23 07:23:52,341]-[netty task-4-2] ERROR com.github.kklisura.cdt.services.utils.ProxyUtils - Failed creating proxy from abstract class
java.lang.NullPointerException: null
	at io.be1.quattro//javassist.util.proxy.SecurityActions.setAccessible(SecurityActions.java:103)
	at io.be1.quattro//javassist.util.proxy.DefineClassHelper.toClass3(DefineClassHelper.java:151)
	at io.be1.quattro//javassist.util.proxy.DefineClassHelper.toClass2(DefineClassHelper.java:134)
	at io.be1.quattro//javassist.util.proxy.DefineClassHelper.toClass(DefineClassHelper.java:95)
	at io.be1.quattro//javassist.util.proxy.FactoryHelper.toClass(FactoryHelper.java:131)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass3(ProxyFactory.java:530)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass2(ProxyFactory.java:515)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass1(ProxyFactory.java:451)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass(ProxyFactory.java:422)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.create(ProxyFactory.java:698)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.create(ProxyFactory.java:683)
	at io.be1.quattro//com.github.kklisura.cdt.services.utils.ProxyUtils.createProxyFromAbstract(ProxyUtils.java:76)
	at io.be1.quattro//com.github.kklisura.cdt.services.impl.ChromeServiceImpl.createDevToolsService(ChromeServiceImpl.java:186)
	at io.be1.quattro//com.github.kklisura.cdt.services.impl.ChromeServiceImpl.createDevToolsService(ChromeServiceImpl.java:161)
	at io.be1.quattro//io.be1.quattro.digitalsignage.DigitalSignage.initComponents(DigitalSignage.java:550)
	at io.be1.quattro//io.be1.quattro.digitalsignage.DigitalSignage.start(DigitalSignage.java:187)
	at io.be1.quattro//io.be1.quattro.AppManager.initializeApp(AppManager.java:395)
	at io.be1.quattro//io.be1.quattro.AppManager.lambda$configure$5(AppManager.java:491)
	at io.be1.quattro//org.jooby.Route$Handler.handle(Route.java:1760)
	at io.be1.quattro//org.jooby.internal.RouteImpl.handle(RouteImpl.java:280)
	at io.be1.quattro//org.jooby.internal.RouteChain.next(RouteChain.java:262)
	at io.be1.quattro//org.jooby.Route$Chain.next(Route.java:2163)
	at io.be1.quattro//org.jooby.Route$After.handle(Route.java:2002)
	at io.be1.quattro//org.jooby.internal.RouteImpl.handle(RouteImpl.java:280)
	at io.be1.quattro//org.jooby.internal.RouteChain.next(RouteChain.java:262)
	at io.be1.quattro//org.jooby.Route$Chain.next(Route.java:2163)
	at io.be1.quattro//org.jooby.internal.pac4j2.Pac4jGrantAccessAdapter.adapt(Pac4jGrantAccessAdapter.java:224)
	at io.be1.quattro//org.pac4j.core.engine.DefaultSecurityLogic.perform(DefaultSecurityLogic.java:142)
	at io.be1.quattro//org.jooby.internal.pac4j2.Pac4jSecurityFilter.handle(Pac4jSecurityFilter.java:257)
	at io.be1.quattro//org.jooby.internal.RouteImpl.handle(RouteImpl.java:280)
	at io.be1.quattro//org.jooby.internal.RouteChain.next(RouteChain.java:262)
	at io.be1.quattro//org.jooby.Route$Chain.next(Route.java:2163)
	at io.be1.quattro//org.jooby.internal.HttpHandlerImpl.handle(HttpHandlerImpl.java:496)
	at io.be1.quattro//org.jooby.internal.netty.NettyHandler.channelRead0(NettyHandler.java:274)
	at io.be1.quattro//io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.be1.quattro//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.be1.quattro//io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:56)
	at io.be1.quattro//io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:365)
	at io.be1.quattro//io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
	at io.be1.quattro//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
	at io.be1.quattro//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.be1.quattro//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:832)
[2020-07-23 07:23:52,342]-[netty task-4-2] ERROR io.be1.quattro.digitalsignage.DigitalSignage - exception e: java.lang.RuntimeException: Failed creating proxy from abstract class s: java.lang.RuntimeException: Failed creating proxy from abstract class
	at io.be1.quattro//com.github.kklisura.cdt.services.utils.ProxyUtils.createProxyFromAbstract(ProxyUtils.java:82)
	at io.be1.quattro//com.github.kklisura.cdt.services.impl.ChromeServiceImpl.createDevToolsService(ChromeServiceImpl.java:186)
	at io.be1.quattro//com.github.kklisura.cdt.services.impl.ChromeServiceImpl.createDevToolsService(ChromeServiceImpl.java:161)
	at io.be1.quattro//io.be1.quattro.digitalsignage.DigitalSignage.initComponents(DigitalSignage.java:550)
	at io.be1.quattro//io.be1.quattro.digitalsignage.DigitalSignage.start(DigitalSignage.java:187)
	at io.be1.quattro//io.be1.quattro.AppManager.initializeApp(AppManager.java:395)
	at io.be1.quattro//io.be1.quattro.AppManager.lambda$configure$5(AppManager.java:491)
	at io.be1.quattro//org.jooby.Route$Handler.handle(Route.java:1760)
	at io.be1.quattro//org.jooby.internal.RouteImpl.handle(RouteImpl.java:280)
	at io.be1.quattro//org.jooby.internal.RouteChain.next(RouteChain.java:262)
	at io.be1.quattro//org.jooby.Route$Chain.next(Route.java:2163)
	at io.be1.quattro//org.jooby.Route$After.handle(Route.java:2002)
	at io.be1.quattro//org.jooby.internal.RouteImpl.handle(RouteImpl.java:280)
	at io.be1.quattro//org.jooby.internal.RouteChain.next(RouteChain.java:262)
	at io.be1.quattro//org.jooby.Route$Chain.next(Route.java:2163)
	at io.be1.quattro//org.jooby.internal.pac4j2.Pac4jGrantAccessAdapter.adapt(Pac4jGrantAccessAdapter.java:224)
	at io.be1.quattro//org.pac4j.core.engine.DefaultSecurityLogic.perform(DefaultSecurityLogic.java:142)
	at io.be1.quattro//org.jooby.internal.pac4j2.Pac4jSecurityFilter.handle(Pac4jSecurityFilter.java:257)
	at io.be1.quattro//org.jooby.internal.RouteImpl.handle(RouteImpl.java:280)
	at io.be1.quattro//org.jooby.internal.RouteChain.next(RouteChain.java:262)
	at io.be1.quattro//org.jooby.Route$Chain.next(Route.java:2163)
	at io.be1.quattro//org.jooby.internal.HttpHandlerImpl.handle(HttpHandlerImpl.java:496)
	at io.be1.quattro//org.jooby.internal.netty.NettyHandler.channelRead0(NettyHandler.java:274)
	at io.be1.quattro//io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.be1.quattro//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.be1.quattro//io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:56)
	at io.be1.quattro//io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:365)
	at io.be1.quattro//io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
	at io.be1.quattro//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
	at io.be1.quattro//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.be1.quattro//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.NullPointerException
	at io.be1.quattro//javassist.util.proxy.SecurityActions.setAccessible(SecurityActions.java:103)
	at io.be1.quattro//javassist.util.proxy.DefineClassHelper.toClass3(DefineClassHelper.java:151)
	at io.be1.quattro//javassist.util.proxy.DefineClassHelper.toClass2(DefineClassHelper.java:134)
	at io.be1.quattro//javassist.util.proxy.DefineClassHelper.toClass(DefineClassHelper.java:95)
	at io.be1.quattro//javassist.util.proxy.FactoryHelper.toClass(FactoryHelper.java:131)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass3(ProxyFactory.java:530)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass2(ProxyFactory.java:515)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass1(ProxyFactory.java:451)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.createClass(ProxyFactory.java:422)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.create(ProxyFactory.java:698)
	at io.be1.quattro//javassist.util.proxy.ProxyFactory.create(ProxyFactory.java:683)
	at io.be1.quattro//com.github.kklisura.cdt.services.utils.ProxyUtils.createProxyFromAbstract(ProxyUtils.java:76)
	... 31 more

Chrome does not always stop

First of all, I would like yo thank all the contributors of this project which is awesome!

I recently integrated it to a backend project to generate some PDF files via the print to PDF feature.

I noticed that the Chrome (Chromium indeed, it runs on a Ubuntu server) process does not always stops at the end of the PDF printing. I see that with the ps command. On these servers, Chromium is only used to generate PDF files.

Most of the time, the Chrome process is stopped because it remains less processes than the times it ran. I also have some automated tests that run this feature and most of the time, it remains no Chrome process at the end.

My code is inspired from the print page to PDF example. I noticed that it does not close the tab and the launcher whereas the example in the main readme page (log request example) does it:

    network.onLoadingFinished(
        event -> {
          // Close the tab and close the browser when loading finishes.
          chromeService.closeTab(tab);
          launcher.close();
        });

I assume that I should close it at the end and that the print to PDF example code should also close the tab and the launcher, right?

I added this piece of code at the end of the PDF printing. I also upgraded the library to 2.1.0 (from 2.0.0). This new version has been running for a few days ago and as of now, all Chrome processes were closed, even if it did not run many times. Will this be enough to fix the issue?

Update protocol failure

I have followed your instructions on downloading latest protocol and updating the protocol.

I am seeing the following errors when performing make update-protocol

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.RuntimeException: Class java/lang/UnknownError could not be instrumented.
	at org.jacoco.agent.rt.internal_290345e.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:139)
	at org.jacoco.agent.rt.internal_290345e.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:100)
	at org.jacoco.agent.rt.internal_290345e.PreMain.createRuntime(PreMain.java:55)
	at org.jacoco.agent.rt.internal_290345e.PreMain.premain(PreMain.java:47)
	... 6 more
Caused by: java.lang.NoSuchFieldException: $jacocoAccess
	at java.base/java.lang.Class.getField(Class.java:1999)
	at org.jacoco.agent.rt.internal_290345e.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:137)
	... 9 more
FATAL ERROR in native method: processing of -javaagent failed
/bin/sh: line 1: 88784 Abort trap: 6           /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java -javaagent:/Users/danny/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=/Users/danny/Desktop/chrome-devtools-java-client/cdt-java-protocol-builder/target/jacoco.exec -jar /Users/danny/Desktop/chrome-devtools-java-client/cdt-java-protocol-builder/target/surefire/surefirebooter6850608009943420711.jar /Users/danny/Desktop/chrome-devtools-java-client/cdt-java-protocol-builder/target/surefire/surefire11171973420015458850tmp /Users/danny/Desktop/chrome-devtools-java-client/cdt-java-protocol-builder/target/surefire/surefire_14730910830068431712tmp

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ctd-java-client-root 1.0-SNAPSHOT .................. SUCCESS [  0.207 s]
[INFO] cdt-protocol-parser 1.0-SNAPSHOT ................... SUCCESS [  5.678 s]
[INFO] cdt-java-protocol-builder 1.0-SNAPSHOT ............. FAILURE [  4.926 s]
[INFO] cdt-java-client 3.0.0-SNAPSHOT ..................... SKIPPED
[INFO] cdt-examples 1.0-SNAPSHOT .......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.019 s
[INFO] Finished at: 2020-12-04T10:48:01Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project cdt-java-protocol-builder: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :cdt-java-protocol-builder
make: *** [build-all-modules] Error 1

Impossible to create a pdf using #onLoadEventFired

I faced with a problem using the primer of pdf generation using #onLoadEventFired event.
The problem is in page generation using ReactJS. Components rendering takes for about a 2 seconds but the PDF output looks like a loading page:

image

As a workaround, we are creating the CustomEvent at #componentDidMount ReactJS component lifecycle step with name, for instance "mounted":
... event = new CustomEvent(typeArg, customEventInit); dispatchEvent(event) ...

This event is logging at the absolutely correct time to the page for a generation. Nevertheless, no one event of Page instance handles this custom event. Moreover, I've tried to handle this event using DOM events. Unfortunately, no one handler helps me.
I've been searching a lot for a decision and found that using Puppeteer we may use #exposeFunction method of page object to handle simply custom events.

Could you please help me to find the correct solution?
I've used the code sample that you suggested for pdf generation.

Thank you.

dom.getAttributes return value

Hello

I've just come across this project and it's been very helpful thus far. Just a question regarding the getAttributes(Integer nodeId) method on the dom though - is there a specific reason it returns a list of strings, which are, alternately, attribute names and then their values?

Error when deploy due to glassfish Tyrus

Hi @kklisura,

Thank you for your great project. However I failed to deploy and get below exception trace:

javax.servlet.ServletException: java.lang.InstantiationException: org.glassfish.tyrus.server.TyrusServerConfiguration
at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:88)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5196)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InstantiationException: org.glassfish.tyrus.server.TyrusServerConfiguration
at java.lang.Class.newInstance(Class.java:427)
at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:74)
... 8 more

Caused by: java.lang.NoSuchMethodException: org.glassfish.tyrus.server.TyrusServerConfiguration.()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.newInstance(Class.java:412)
... 9 more

After checking your source code, I figure out that you only need Glassfish Tyrus to run UnitTest.
So can you add <scope>test</scope> into your pom.xml file? Or make a separated module for testing.

Thank you in advance.

ChromeDevToolsServiceConfiguration creates unused ExecutorService

Creating an instance of the ChromeDevToolsServiceConfiguration also results in the creation of an ExecutorService (through the DefaultEventExecutorService). That executor is not used in case the service is cached (and not shut down).

Some possible solutions:

  • Don't create a default DefaultEventExecutorService in the ChromeDevToolsServiceConfiguration, but only if required and no custom one is specified in the config.
  • Have a Supplier as part of the config, which is only invoked if required.

Vulnerability with tyrus-container-grizzly-client

A scan with org.owasp:dependency-check-maven:4.0.2 gives a message.

One or more dependencies were identified with known vulnerabilities in cdt-java-client:

tyrus-client-1.13.1.jar (cpe:/a:oracle:oracle_client:1.13.1, org.glassfish.tyrus:tyrus-client:1.13.1) : CVE-2006-0550


See the dependency-check report for more details.

Wait for page load - JS page

There is currently no support for waiting on page load based on pages that written in any async framework like Angular.
Is it possible to add support for such applications?

For example with ChromeDriver it is possible to do this with an ExpectedCondition

Logging is broken.

Hi,

It seems like something is causing com.github.kklisura.cdt.services.impl.WebSocketServiceImpl` from respecting Slf4j settings, as it is always logging with the DEBUG level, regardless of Slf4j settings, filling up the log with a lot of needless noise.

I am unsure what is causing this.

Kind regards,
Mikkel Løkke

chromeProcess termination with exit code 1

@kklisura Thanks again for this wonderful project. 👍

Sometimes the chromeProcess in ChromeLauncher terminates with exit code 1 when waiting for Page onLoadEvent to occur.

As of now, It is not possible to know when this happens.

Can you expose a method in ChromeLauncher to address this.

public boolean hasExited() {
    try {
        chromeProcess.exitValue();
        return true;
    } catch(IllegalThreadStateException e) {
        return false;
    }
}

New release?

Is it possible to create a new release of this library? The last release is more than 1 year old and the protocol has been updated since.

Settings resetted?

Hi,
I need for my project the setting "Autoplay policy" unfortunately this setting is reset every time.
Is there a way to change this promatically?

On the other hand, the settings should be taken over from the normal browser.

ChromeDevTools hangs indefinately when running inside a Docker Container.

Ok, this is a wierd one. I've spent an entire day trying to figure out what's going on.
I am using ChromedevTools to get the source code of a page once I get a "firstMeaningfulPaint".

The code snippet is as follows:

    final ChromeLauncher launcher = new ChromeLauncher();
    final ChromeService chromeService = launcher.launch(chromeArguments);

    log.info(String.format("Entering chromeStrategy for %s", link.getUrl()));
    final Instant startTime = Instant.now();

    final ChromeTab tab = chromeService.createTab();
    final ChromeDevToolsService devToolsService = chromeService.createDevToolsService(tab);

    final var latch = new CountDownLatch(1);

    final var page = devToolsService.getPage();
    page.setLifecycleEventsEnabled(true);
    page.onLifecycleEvent(x -> {
        log.info(String.format("Got event from chrome: %s", x.getName()));
        if (x.getName().equals(LATCH_NAME)) {
            latch.countDown();
        }
    });

    page.enable();

    final var dom = devToolsService.getDOM();
    dom.enable();

    final var network = devToolsService.getNetwork();
    network.enable();
    network.setBypassServiceWorker(true);
    network.setCacheDisabled(true);

    var blah = Try.of(() -> page.navigate(link.getUrl()))
            .andThenTry(() -> latch.await(10, TimeUnit.SECONDS))
            .mapTry(x -> dom.getOuterHTML());

    chromeService.closeTab(tab);
    launcher.close();
    devToolsService.waitUntilClosed();

Now here comes the really strange part. When I try this from my local machine, either with gradle or IntelliJ it works fine. When I try it in a docker container it dies after a few tries.

Yourkit reports the following stackframes, which don't really mean that much to me:

jdk.internal.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:194
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() AbstractQueuedSynchronizer.java:885
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) AbstractQueuedSynchronizer.java:1039
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) AbstractQueuedSynchronizer.java:1345
java.util.concurrent.CountDownLatch.await() CountDownLatch.java:232
com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl$InvocationResult.waitForResult(long, TimeUnit) ChromeDevToolsServiceImpl.java:396
com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(String, Class, MethodInvocation) ChromeDevToolsServiceImpl.java:138
com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(Object, Method, Object[]) CommandInvocationHandler.java:80
com.sun.proxy.$Proxy129.getOuterHTML()

Mostly it hangs, but sometimes it crashes with the following stacktrace:

java.lang.IllegalStateException: The connection has been closed.
    at org.glassfish.tyrus.core.TyrusSession.checkConnectionState(TyrusSession.java:530) ~[tyrus-core-1.13.1.jar:na]
    at org.glassfish.tyrus.core.TyrusSession.getBasicRemote(TyrusSession.java:212) ~[tyrus-core-1.13.1.jar:na]
    at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.send(WebSocketServiceImpl.java:114) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(ChromeDevToolsServiceImpl.java:135) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(CommandInvocationHandler.java:80) ~[cdt-java-client-1.3.3.jar:na]
    at com.sun.proxy.$Proxy129.getOuterHTML(Unknown Source) ~[na:na]

Whether it works, hangs or crashes does not seem to be deterministic, or rather I have not been able to determine the cause. Any help is appreciated. I'm using the openjdk:11.0.1-slim Docker image.

Lot of temp file system usage

Hi,

I observed that cdt stores some files at the temp directory of user. This usage is huge and is not cleaned up after CDT session. Eventually it eats up a lot of disk space and causing other processes to stop working some times if the disk has less space.

image

Is there any plan to clean up those files ?

Support for Browsing context

Would be great if it supports creating browsing context and tab from it as https://devdocs.io/puppeteer/ does.

try (ChromeDevToolsService parentDevToolService = ChromeServiceExt
.createDevToolsService(chromeService.getVersion().getWebSocketDebuggerUrl())){}

public static ChromeDevToolsService createDevToolsService(String webSocketURL) {
try {
WebSocketService webSocketService = WebSocketServiceImpl.create(new URI(webSocketURL));
CommandInvocationHandler commandInvocationHandler = new CommandInvocationHandler();
Map<Method, Object> commandsCache = new ConcurrentHashMap<>();
ChromeDevToolsService devtools =
(ChromeDevToolsService) ProxyUtils.createProxyFromAbstract(
ChromeDevToolsServiceImpl.class,
new Class[] {WebSocketService.class, ChromeDevToolsServiceConfiguration.class},
new Object[] {webSocketService, new ChromeDevToolsServiceConfiguration()},
(unused, method, args) ->
commandsCache.computeIfAbsent(
method,
key -> {
Class<?> returnType = method.getReturnType();
return ProxyUtils.createProxy(returnType, commandInvocationHandler);
}));
commandInvocationHandler.setChromeDevToolsService(devtools);
return devtools;
}catch(Exception e) {
throw new SnapshotGenerationException("Couldn't create ChromeDevToolsService for chrome webSocketDebuggerUrl ",e);
}
}

 static ChromeTab createTab(ChromeService chromeService, ChromeDevToolsService parentDevToolService,
		String browserContext) {
	ChromeTab tab;
	String targetId = parentDevToolService.getTarget().createTarget(SnapshotGenConstants.ABOUT_BLANK, null, null,browserContext, null, null, null);
	tab = chromeService.getTabs().stream().filter(chromeTab -> chromeTab.getId().equals(targetId))
			.findFirst().get();
	return tab;
}

Exception when Launching Chrome, after it crashed.

I'm not really sure how to reproduce this, but I suspect that what is happening is something like this:

Process launches.
Pages are loaded.
Chrome Crashes.
Another page is trying to load:

This happens:

java.lang.IllegalArgumentException: Hook previously registered
    at java.base/java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:72) ~[na:na]
    at java.base/java.lang.Runtime.addShutdownHook(Runtime.java:215) ~[na:na]
    at com.github.kklisura.cdt.launch.ChromeLauncher$ShutdownHookRegistry.register(ChromeLauncher.java:445) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.launch.ChromeLauncher.launchChromeProcess(ChromeLauncher.java:283) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.launch.ChromeLauncher.launch(ChromeLauncher.java:143) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.launch.ChromeLauncher.launch(ChromeLauncher.java:157) ~[cdt-java-client-1.3.3.jar:na]

Previous to that, the following happened:

java.lang.IllegalStateException: The connection has been closed.
    at org.glassfish.tyrus.core.TyrusSession.checkConnectionState(TyrusSession.java:530) ~[tyrus-core-1.13.1.jar:na]
    at org.glassfish.tyrus.core.TyrusSession.getBasicRemote(TyrusSession.java:212) ~[tyrus-core-1.13.1.jar:na]
    at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.send(WebSocketServiceImpl.java:114) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(ChromeDevToolsServiceImpl.java:135) ~[cdt-java-client-1.3.3.jar:na]
    at com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(CommandInvocationHandler.java:80) ~[cdt-java-client-1.3.3.jar:na]
    at com.sun.proxy.$Proxy122.getResourceTree(Unknown Source) ~[na:na]

After these two exceptions have been thrown the application must be restarted to recover.

Web socket client not annotated with @ClientEndpoint

Tomcat 9, Java 8, getting the following error

javax.websocket.DeploymentException: Cannot use POJO class [com.github.kklisura.cdt.services.impl.WebSocketServiceImpl$1] as it is not annotated with @ClientEndpoint
at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServer(WsWebSocketContainer.java:122)
at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.connect(WebSocketServiceImpl.java:86)
at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.create(WebSocketServiceImpl.java:76)
at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.lambda$new$3(ChromeServiceImpl.java:94)
at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.createDevToolsService(ChromeServiceImpl.java:176)
at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.createDevToolsService(ChromeServiceImpl.java:161)

Hang due to WebSocket closing due to buffer overflow

If the WebSocket closes, chrome-devtools-java-client just hangs as it does not listen to or respond to the socket close event.

This causes an issue when printing large PDFs and requesting the result back as base64, as the WebSocket implementation has a maximum message size of 4MB. I have written this up https://superuser.com/questions/1472051/headless-chrome-hangs-when-trying-to-print-page-as-pdf/1483423#1483423

I note there is a TODO in place of the onClose implementation but I'm not sure how to implement it so that we can bubble the error back up, or where it should bubble back up to! I'd be happy to contribute a patch with a little guidance, or patiently await a patch if it's out of my league.

Handling the SIGTERM

Hi,
Do chrome processes spawned by CDT have any provision to handle SIGTERM received by JVM running them ?
I found this during my test that as soon as the SIGTERM is received by the main process , the chrome windows immediately get killed even though there is a SIGTERM handler in the main thread.

How to avoid hang on extracting HTML content from non HTML page (solved)

I was having a problem with this command when browsing to content that happened to be XML - the evaluation function hung indefinitely.

devToolsService.getRuntime().evaluate("document.documentElement.outerHTML")

Here is a work around:

Object contentType = devToolsService.getRuntime().evaluate("document.contentType");
if((contentType != null) && "text/html".equalsIgnoreCase(contentType.toString())) {
   devToolsService.getRuntime().evaluate("document.documentElement.outerHTML")
}

Failed while waiting for chrome to start: Timeout expired! Chrome output:

I am trying to run the example in a docker container but the chrome process is closes immediately,

23:06:44.769 [main] INFO  io.micronaut.runtime.Micronaut - Startup completed in 17ms. Server Running: http://623714484c50:8080
23:06:55.620 [pool-2-thread-3] INFO  c.g.k.cdt.launch.ChromeLauncher - Launching chrome process /usr/bin/chromium with arguments {disable-software-rasterizer=null, disable-dev-shm-usage=null, user-data-dir=/tmp/cdt-user-data-dir3872197855285647012}
23:06:55.714 [pool-2-thread-3] INFO  c.g.k.cdt.launch.ChromeLauncher - Closing chrome process...
23:06:55.714 [pool-2-thread-3] INFO  c.g.k.cdt.launch.ChromeLauncher - Chrome process closed.
..
..
..
Caused by: com.github.kklisura.cdt.launch.exceptions.ChromeProcessTimeoutException: Failed while waiting for chrome to start: Timeout expired! Chrome output: 
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Exception.<init>(Exception.java:66)
	at java.lang.RuntimeException.<init>(RuntimeException.java:62)
	at com.github.kklisura.cdt.launch.exceptions.ChromeProcessException.<init>(ChromeProcessException.java:35)
	at com.github.kklisura.cdt.launch.exceptions.ChromeProcessTimeoutException.<init>(ChromeProcessTimeoutException.java:35)
	at com.github.kklisura.cdt.launch.ChromeLauncher.waitForDevToolsServer(ChromeLauncher.java:346)
	at com.github.kklisura.cdt.launch.ChromeLauncher.launchChromeProcess(ChromeLauncher.java:280)
	at com.github.kklisura.cdt.launch.ChromeLauncher.launch(ChromeLauncher.java:143)
	at com.github.kklisura.cdt.launch.ChromeLauncher.launch(ChromeLauncher.java:157)

this is my launcher code (the only thing different from the example)

        final ChromeLauncher launcher = new ChromeLauncher();
        ChromeArguments args = ChromeArguments.defaults(true)
                .additionalArguments("disable-software-rasterizer", null)
                .additionalArguments("disable-dev-shm-usage", null).build();
        chromeService = launcher.launch(args);

Starting headless chromium works as expected

chromium --no-sandbox --headless --disable-gpu --disable-software-rasterizer --disable-dev-shm-usage  --remote-debugging-port=9222
[1217/230628.349717:ERROR:gpu_process_transport_factory.cc(967)] Lost UI shared context.
Failed to create secure directory (/home/appuser/.config/pulse): No such file or directory

DevTools listening on ws://127.0.0.1:9222/devtools/browser/628c3a40-6646-49d4-bfd5-7e21df027bef

This is the dockerfile I am using it plain debian:stretch with the default chromium. Nothing fancy.

FROM debian:stretch-slim

COPY --from=pdfgenerator /app/pdfgenerator

RUN apt-get update && apt-get install -y \
    chromium \
    fonts-liberation2 \
    fonts-roboto \
    --no-install-recommends \
     && rm -rf /var/lib/apt/lists/*


# Add chromium user
RUN groupadd -r appuser && useradd -r -g appuser -G audio,video appuser \
    && mkdir -p /app/temp && chown -R appuser:appuser /app

USER appuser

ENV CHROME_PATH=/usr/bin/chromium
WORKDIR /app

#ENTRYPOINT ["chromium-browser", "--headless", "--disable-gpu", "--disable-software-rasterizer", "--disable-dev-shm-usage"]
CMD ["./pdfgenerator"]

Maybe you have an idea what it might be, but I'll investigate some more in the following days.

Way to force close the tab or chrome window

Hi,

I recently came across the issue where the chrome tab just hangs and becomes non-responsive. In such cases the close() on chrome tab doesn't work.
However, I can manually go to the task manager of chrome and kill the window from there.

Is it possible to provide a way to force close the window or tab ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.