webmetrics / browsermob-proxy Goto Github PK

The HAR produced advertised itself as a 1.1 one, but included serverIPAddress which was introduced in 1.2.

Currently there seems to be a bug when browsing to SSL pages with the browsermob proxy, see this discussion on the Google groups page for more info:
https://groups.google.com/d/topic/browsermob-proxy/hCgODJFb494/discussion

I use the Apache HttpComponents to communicate with the REST interface of the Proxy. When I start a new proxy using a URL generated like the next code fragment, the HttpComponents url encode the httpProxy parameter.

URIBuilder builder = new URIBuilder();
builder.setScheme("http")
    .setHost(browserMobHost)
    .setPort(this.browserMobPort)
    .setPath("/proxy")
    builder.setParameter("httpProxy", "172.16.96.10:8001");

However the REST-Interface does not decode the parameter thus resulting in an ArrayIndexOutOfBoundsException in BrowserMobHttpClient (Line 935). I believe that the URL-Parameters should be decoded in the REST Interface, however I am unsure if this can have negative side effects for people using the REST interface without URL-Encoded parameters.

I ran mvn install and see it created a target folder. I don't see a bin folder. I tried
java -jar browsermob-proxy-2.0-beta-3-SNAPSHOT.jar

and got this error

Failed to load Main-Class manifest attribute from
browsermob-proxy-2.0-beta-3-SNAPSHOT.jar

Some environments might require setting the HTTP proxy of the Apache HttpClient instance used by the BrowserMobHttpClient class. It would be great if users were allowed to set it.

Hi

Would be nice to have a API call ( GET /proxy/list ) for getting the list of all proxies available [{'port' :9091}, {'port' :9092}... ] so that in the creation process of a proxy on the same port it's easier to program a failure
Is anything planned related to this?

Thanks

We in Yandex (http://www.yandex.com) are running WebDriver autotests on remote browsers.
In ProxyServer class there is hardcoded value of proxy host.
It doesn't work for us.

Our remote browser tries to use localhost proxy, but the proxy server runs not on the same machine as browser.
I have attached patch, that fixes isshue.
Please review it and apply.

We at Yandex hope you fix this issue.
Your tool is very helpfull for us.

--- a/src/main/java/org/browsermob/proxy/ProxyServer.java
+++ b/src/main/java/org/browsermob/proxy/ProxyServer.java
@@ -16,6 +16,8 @@ import org.openqa.selenium.Proxy;
 import java.util.Date;
 import java.util.Map;

+import java.net.InetAddress;
+

 public class ProxyServer {
     private static final HarNameVersion CREATOR = new HarNameVersion("BrowserMob Proxy", "2.0");
@@ -61,7 +63,7 @@ public class ProxyServer {
     public org.openqa.selenium.Proxy seleniumProxy() {
         Proxy proxy = new Proxy();
         proxy.setProxyType(Proxy.ProxyType.MANUAL);
-        String proxyStr = String.format("localhost:%d", getPort());
+        String proxyStr = String.format("%s:%d", InetAddress.getLocalHost().getCanonicalHostName(),  getPort());
         proxy.setHttpProxy(proxyStr);
         proxy.setSslProxy(proxyStr);

It looks like BrowserMob proxy rewrites the user agent field to insert "BrowserMob RBU." As a proxy, it shouldn't be touching the user agent field. It's fair game to insert an X-* request header, but even that I think is unnecessary.

I have been experimenting with using browsermob proxy and have done the following:

(using Mac Pro w/ Lion ... java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03-383-11A511) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-383, mixed mode)

downloaded the latest binaries from https://browsermob.s3.amazonaws.com/proxy/browsermob-proxy-2.0-SNAPSHOT-bin.zip
unpacked
run bin/browsermob-proxy

curl -i -X POST http://localhost:8080/proxy #=> proxy: 9091
curl -i -X PUT http://localhost:8080/proxy/9091/har

As this point I get a stack trace like this....

2011-09-02 14:24:09.157:WARN::/proxy/9091/har
java.lang.RuntimeException: Sitebricks received a null reply from the resource.
at com.google.sitebricks.headless.ReplyBasedHeadlessRenderer.render(ReplyBasedHeadlessRenderer.java:25)
at com.google.sitebricks.routing.WidgetRoutingDispatcher.bindAndReply(WidgetRoutingDispatcher.java:90)
at com.google.sitebricks.routing.WidgetRoutingDispatcher.dispatch(WidgetRoutingDispatcher.java:73)
at com.google.sitebricks.DebugModeRoutingDispatcher.dispatch(DebugModeRoutingDispatcher.java:63)
at com.google.sitebricks.SitebricksFilter.doFilter(SitebricksFilter.java:41)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at com.google.sitebricks.HiddenMethodFilter.doFilter(HiddenMethodFilter.java:74)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1323)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:474)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:935)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:404)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:870)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:346)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:807)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:520)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:528)
at java.lang.Thread.run(Thread.java:680)

The documentation suggests that this is all that should be required to start capturing data, but clearly I'm missing something or I've gotten some wires crossed somewhere. Any ideas?

Currently there seems to be a bug when naviating to certain URL with the browsermob proxy,
see this topics on the Google groups page for more info:
http://groups.google.com/group/browsermob-proxy/browse_thread/thread/f84a95f263474f38

Hi, the link to beta5 in the page mentioned above does not work -> link points to beta7

cheers,
andy

I want to test/track requests (in my special case AJAX and image requests on localhost:8080) of a web application running on localhost. Those requests are not possible to catch with any interceptors as browsermob skips them, see this discussion for more details:

https://groups.google.com/forum/?fromgroups#!topic/browsermob-proxy/eV6SqG65S5E/discussion

When using "localhost" (after removing the proxy restriction), Lookup tries to do a DNS A lookup using the search domain from resolv.conf, which will fail in most cases for obvious reasons. It seems to ignore /etc/hosts as a first lookup step.

Trying to use 127.0.0.1 instead fails as well, since HTTPClient does a reverse lookup and sends "localhost" again to BrowserMobHostNameResolver.

The only workaround I've found is to use a remap rule, like so:

proxy.start()
proxy.remapHost("localhost", "127.0.0.1")

This bypasses the broken DNS lookup.

Currently the headers are captured before the changes made by [autoBasicAuthorization()](http://static.browsermob.com/api/com/browsermob/api/HttpClient.html#autoBasicAuthorization(java.lang.String, java.lang.String, java.lang.String) take place.
It would be nice if the headers captured by the proxy reflect the changes made by any header manipluation.

Please see: Google Groups Discussion

Could you place put an example how to manipulate the content with the HttpResponseInterceptor?

It's not clear how to replace the HttpEntity.

In this Example I like to convert all Text to uppercase:

@Test
public void shoudConvertEverythingToUpperCase() throws ClientProtocolException, IOException
{
    final DefaultHttpClient defaultHttpClient = new DefaultHttpClient();

    defaultHttpClient.addResponseInterceptor(new HttpResponseInterceptor() {

        @Override
        public void process(final HttpResponse response, final HttpContext context) throws HttpException, IOException
        {
            final HttpEntity entity = response.getEntity();

            final HttpEntity upperCaseEntity = makeAllUppercase(entity);

            response.setEntity(upperCaseEntity);
        }

        private HttpEntity makeAllUppercase(final HttpEntity entity)
        {
            // how to uppercase everything and return the cloned HttpEntity
            return null;
        }
    });

    final HttpResponse httpResponse = defaultHttpClient.execute(new HttpGet("http://opensource.webmetrics.com/browsermob-proxy/"));

    assertTrue(StringUtils.isAllUpperCase(EntityUtils.toString(httpResponse.getEntity())));
}

When I load the HARs I capture with browsermob now I get the following sorts of errors in the harviewer page:

HAR Validation
Sum of request timings doesn't correspond to the total value: http://www.amazon.co.uk/ (request.time: 1505 vs. sum: 1284), request#: 0, parent page: www.amazon.co.uk
HAR Validation
Sum of request timings doesn't correspond to the total value: http://z-ecx.images-amazon.com/images/G/01/browser-scripts/uk-site-wide-1.2.6/site-wide-2650665772.css._V165058414_.css (request.time: 280 vs. sum: 272), request#: 1, parent page: www.amazon.co.uk
HAR Validation
Sum of request timings doesn't correspond to the total value: http://z-ecx.images-amazon.com/images/G/01/nav2/gamma/websiteGridCSS/websiteGridCSS-websiteGridCSS-9085.css._V165356162_.css (request.time: 560 vs. sum: 555), request#: 2, parent page: www.amazon.co.uk
HAR Validation
Sum of request timings doesn't correspond to the total value: http://g-ecx.images-amazon.com/images/G/02/x-locale/common/transparent-pixel._V167145160_.gif (request.time: 270 vs. sum: 269), request#: 3, parent page: www.amazon.co.uk
HAR Validation
Sum of request timings doesn't correspond to the total value: http://g-ecx.images-amazon.com/images/G/02/gno/images/orangeBlue/navPackedSprites-UK-15._V202471918_.png (request.time: 590 vs. sum: 566), request#: 4, parent page: www.amazon.co.uk
HAR Validation
Sum of request timings doesn't correspond to the total value: http://g-ecx.images-amazon.com/images/G/02/x-site/BlackFriday2011/uk_xsite_bf_lozenge._V164928606_.png (request.time: 468 vs. sum: 415), request#: 5, parent page: www.amazon.co.uk
HAR Validation
Sum of request timings doesn't correspond to the total value: http://g-ecx.images-amazon.com/images/G/02/uk-jw/homepage/UK_WAT_BrandStore-Perfect_ROTO._V177647797_.png (request.time: 413 vs. sum: 367), request#: 6, parent page: www.amazon.co.uk

Note that they recently updated their version to be stricter on timings -see below link

http://code.google.com/p/harviewer/source/browse/trunk/webapp/ReleaseNotes.txt

• Validation extended (negative timings not allowed and summary of timings must be correct)

Hi,

is it possible to update the dependencies, so they are more streamlined
with the latest selenium versions. This concerns especially the
httpclient. Especially the usage of th Commons HttpClient should be
updated to the ApacheHttpComponents, since the Commons HttpClient is dead
(http://hc.apache.org/httpclient-3.x/)

Also the Selenium API should be updated to a later version.

I'm using browsermob proxy through the REST API.

I noticed that onContentLoaded is never set in the har that we get from browsermob.

I read the source code as well and i didn't see anything about setting it in the har.

There is setters and accessors for onContentLoad but I can't see when they are (or have to be) triggered.

Hi guys,

I`ve tried to use SSL connection, however I got problems:

• under FF3: Secure Connection Failed page with message 'Error code: ssl_error_internal_error_alert'
• under IE8: There is a problem with this website's security certificate page, where every time I need to manually 'Continue to this website (not recommended).'

To be clear:

• original browsermob-proxy-2.0-beta-3 used (any changes in pom.xml dependencies)
• JDK 1.7.0_01 used (with JDK 1.5 I got compile errors - even that you recommend to use bouncycastle with bcprov-jdk15)
• certicate cybervillainsCA.cer imported to both browsers

Any ideas what to do else?

The proxy currently doesn't work on Java 7 when interacting with certain browsers, most notably Firefox. It appears each of the browsers negotiates their SSL/TLS stuff a bit differently. And my best guess is Firefox requests a key length > 1024 bits, which results in the proxy failing the request because it's unable to create a DH keypair. I don't see the problem with Java 6, which should have had the same cryptography export restrictions, so I don't totally get what's going on. A sample trace looks like:

19:59:11.201 DEBUG  [SeleniumProxyHandler$SslRelay1-1] org.browsermob.proxy.jetty.http.HttpConnection.handleNext(HttpConnection.java:991) >05> javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

I want to be able to make assertions against the raw responses sent over the wire when performing web testing.

ProxyServer has a setCaptureContent that should capture response bodies to the "text" and "encoding" fields of the HAR content entry, but in fact nothing is captured.

I am using browsermob proxy to capture the network traffic for further analysis but I do not get to see the full http traffic. Unfortunately, it misses out 0*0 image/gif requests. Tried whitelisting but that didn't help.

http://sa.abc.co.uk/bbc/test/s?name=beta.xinga.page&pal_route=service_show&ml_name=barlesque&app_type=web&language=en-GB

Appreciate any quick help.

Thanks,
Putta

Appears to be a bug in the pom file:

    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-io</artifactId>
        <version>1.3.2</version>
    </dependency>

Should be:

    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>1.3.2</version>
    </dependency>

Error:
Some projects fail to be resolved
Impossible to resolve dependencies - unresolved dependency: org.apache.commons#commons-io;1.3.2: java.text.ParseException: inconsistent module descriptor file found in 'http://.../nexus/content/groups/public/org/apache/commons/commons-io/1.3.2/commons-io-1.3.2.pom': bad organisation: expected='org.apache.commons' found='commons-io';

WHEN ProxyServer is configured to use an upstream HTTP and SSL proxy:
final ProxyServer proxyServer = new ProxyServer(browserMobProxyPort);
Map<String,String> options = new HashMap<String, String>();
options.put("httpProxy", upstreamProxyHost + ":" + upstreamProxyPort);
proxyServer.start();
proxyServer.setOptions(options);

THEN
A java.net.UnknownHostException is thrown when I request a SSL connection from the browser:

10:30:28,758 DEBUG HttpConnection:990 - EXCEPTION
EOFException()
at
org.browsermob.proxy.jetty.http.HttpRequest.readHeader(HttpRequest.java:
222)
at
org.browsermob.proxy.jetty.http.HttpConnection.readRequest(HttpConnection.java:
861)
at
org.browsermob.proxy.jetty.http.HttpConnection.handleNext(HttpConnection.java:
907)
at
org.browsermob.proxy.jetty.http.HttpConnection.handle(HttpConnection.java:
832)
at
org.browsermob.proxy.jetty.http.SocketListener.handleConnection(SocketListener.java:
245)
at
org.browsermob.proxy.jetty.util.ThreadedServer.handle(ThreadedServer.java:
357)
at org.browsermob.proxy.jetty.util.ThreadPool
$PoolThread.run(ThreadPool.java:534)
10:30:28,793 INFO JavaScriptErrorHuntingSmokeTest:? - Testing
testOpenHomePage: https://www.pcisecuritystandards.org/
10:30:28,793 INFO JavaScriptErrorHuntingSmokeTest:? - Testing
testOpenHomePage: https://www.pcisecuritystandards.org/
10:30:28,802 DEBUG HttpConnection:110 - new HttpConnection:
Socket[addr=/127.0.0.1,port=64962,localport=4449]
10:30:28,802 DEBUG HttpConnection:973 - REQUEST from
[email protected]:4449:
CONNECT www.pcisecuritystandards.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101
Firefox/8.0
Proxy-Connection: keep-alive
Host: www.pcisecuritystandards.org

10:30:28,802 DEBUG HttpServer:876 - Contexts at www.pcisecuritystandards.org:443:
[/=[HttpContext[/,/]]]
10:30:28,802 DEBUG HttpServer:890 - Try HttpContext[/,/],0
10:30:28,803 DEBUG HttpContext:1508 - Handler
org.browsermob.proxy.BrowserMobProxyHandler in HttpContext[/,/]
10:30:28,803 DEBUG HttpContext:1513 - Handled by
org.browsermob.proxy.BrowserMobProxyHandler in HttpContext[/,/]
10:30:28,803 DEBUG HttpInputStream:231 - resetStream()
10:30:28,804 DEBUG HttpConnection:1077 - RESPONSE:
HTTP/1.1 500 java%2Enet%2EUnknownHostException%3A+www
%2Epcisecuritystandards%2Eorg
Date: Fri, 02 Dec 2011 09:30:28 GMT
Server: Jetty/5.1.x (Windows 7/6.1 amd64 java/1.6.0_29
Connection: close
Content-Type: text/html
Content-Length: 1342

The bin/browsermob-proxy script should be executable on *nix platforms by default. It currently has permissions of 664.

Randomly, I'm getting a null entry in the entries array.

I tried to check the same URL each hour with a document that wasn't changing at all, and the bug is still there, totally randomly.

........
null,{
"timings":{"wait":39,"connect":0,"blocked":0,"receive":1,"send":0,"dns":0},
"response":{
"statusText":"OK","headersSize":0,"httpVersion":"HTTP","bodySize":2056,"cookies":[],"content":{"size":2056,"mimeType":"image/jpeg"},
"status":200,
"redirectURL":"",
"headers":[{"name":"Server","value":"nginx/1.0.2"},{"name":"Date","value":"Thu, 12 Jan 2012 23:10:04 GMT"},{"name":"Content-Type","value":"image/jpeg"},{"name":"Content-Length","value":"2056"},{"name":"Last-Modified","value":"Fri, 09 Sep 2011 12:48:15 GMT"},{"name":"Connection","value":"keep-alive"},{"name":"Expires","value":"Sun, 26 Feb 2012 23:10:04 GMT"},{"name":"Cache-Control","value":"max-age=3888000"},{"name":"Accept-Ranges","value":"bytes"}]},
"cache":{},
"time":40,
"pageref":"4f0f6848a11460d343000001"
....

Any idea?

My code is location, the proxy is remote，how can I use the Function server.remaphost，server.rewriteurl, server.setCaptureContent，server.newHar(initialPageRef)，server.newPage(pageRef) by java code or command line,

thanks

If you produce a har file for http://www.adobe.com, the resulting file is invalid. Specifically, the entry for for http://www.adobe.com/favicon.ico. For instance, it is missing the mandatory 'timings' section.

Files need to be valid even if the response was something unexpected.

timings [object] - Detailed timing info about request/response round trip.

Here is the entry as dumped from PHP.

  [65]=>
  array(6) {
    ["time"]=>
    int(0)
    ["request"]=>
    array(8) {
      ["method"]=>
      string(3) "GET"
      ["headers"]=>
      array(0) {
      }
      ["queryString"]=>
      array(0) {
      }
      ["cookies"]=>
      array(0) {
      }
      ["bodySize"]=>
      int(0)
      ["url"]=>
      string(32) "http://www.adobe.com/favicon.ico"
      ["httpVersion"]=>
      string(4) "HTTP"
      ["headersSize"]=>
      int(0)
    }
    ["response"]=>
    array(9) {
      ["content"]=>
      array(2) {
        ["size"]=>
        int(0)
        ["mimeType"]=>
        string(0) ""
      }
      ["headers"]=>
      array(0) {
      }
      ["status"]=>
      int(-999)
      ["cookies"]=>
      array(0) {
      }
      ["bodySize"]=>
      int(0)
      ["httpVersion"]=>
      string(4) "HTTP"
      ["headersSize"]=>
      int(0)
      ["statusText"]=>
      string(11) "NO RESPONSE"
      ["redirectURL"]=>
      string(0) ""
    }
    ["startedDateTime"]=>
    string(28) "2012-07-11T14:16:04.928+0000"
    ["pageref"]=>
    string(15) "adobe home page"
    ["cache"]=>
    array(0) {
    }
  }

It'd probably be more appropriate if the proxy creation status code returned 201. This is in keeping with many other ReST APIs.

I'm seeing an exception thrown when running with "--help" on both beta-6 and the beta-7 snapshot. Example below:

nirvdrum@melchior /opt/browsermob-proxy $ ./bin/browsermob-proxy --help
INFO 04/24 00:48:52 o.b.p.Main           - Starting BrowserMob Proxy version 2.0-beta-7-SNAPSHOT
Option                                  Description                            
------                                  -----------                            
-?, --help                              This help text                         
--port [Integer]                        The port to listen on (default: 8080)  
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:

1) No implementation for java.lang.Integer annotated with @com.google.inject.name.Named(value=port) was bound.
  while locating java.lang.Integer annotated with @com.google.inject.name.Named(value=port)
    for parameter 0 at org.browsermob.proxy.guice.JettyServerProvider.(JettyServerProvider.java:16)
  at org.browsermob.proxy.guice.JettyModule.configure(JettyModule.java:10)

1 error
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
    at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
    at com.google.inject.Guice.createInjector(Guice.java:95)
    at com.google.inject.Guice.createInjector(Guice.java:72)
    at com.google.inject.Guice.createInjector(Guice.java:62)
    at org.browsermob.proxy.Main.main(Main.java:32)

Hi,

I'm using browsermob for a long time and I think it's really great!

But :)

I like to use Selenium Grid for cross browser testing. So, I run a browsermob standalone proxy on a node. I like to connect from my testscript to the running proxy, because I like to use the features to read out the HAR.

Steps I took (node: 192.168.1.103) :

1. sh browsermob-proxy -port 9090
2. curl -X POST http://localhost:9090/proxy
3. curl -X PUT -d 'initialPageRef=Foo' http://localhost:9090/proxy/9091/har

How can we use the embedded browsermob functionality to walk through the har file and validate the requests ?

Thanks,
Roy

dns resolution error，why？

hi,

I will refer to the google groups post I started: https://groups.google.com/group/browsermob-proxy/browse_thread/thread/87b8242b970d7d9?hl=en

To summarize:
We have a veryvery secured network and only a limited range of ports are open.

It would be very nice if browsermob will restart the port counting once we create a new proxy.

Thanks,
R

Hi,

I have defined the following host definition in /etc/hosts
127.0.0.1 localhost mycomputer.mycompany.org

Then I tried to use BrowserMob Proxy but name resolution failed:

15:30:17.973 DEBUG o.b.proxy.jetty.http.HttpContext - Handler org.browsermob.proxy.BrowserMobProxyHandler in HttpContext[/,/]
15:30:17.981 DEBUG o.b.p.http.BrowserMobHttpClient$1 - Get connection: HttpRoute[{}->http://mycomputer.mycompany.com:8082], timeout = 60000
15:35:47.826 DEBUG o.b.p.http.BrowserMobHttpClient$1 - Released connection is not reusable.
INFO 09/21 13:35:47 o.b.p.h.BrowserMobH~ - java.net.UnknownHostException: mycomputer.mycompany.com when requesting http://mycomputer.mycompany.com:8082/xxxxxxxxx

Thanks

Hi,

I just cloned the read only repo (git clone https://github.com/webmetrics/browsermob-proxy.git) and I am trying to build it, but I am not able due to the following error:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building BrowserMob Proxy
[INFO] task-segment: [compile]
[INFO] ------------------------------------------------------------------------
[INFO] [enforcer:enforce {execution: enforce-maven}]
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 14 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Compiling 444 source files to D:\Users\MyUser\My Documents\Projects\browsermob-proxy\target\classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,25] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,58] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,59] not a statement
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,66] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,26] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,50] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,59] not a statement
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,68] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[161,4] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[161,33] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[161,49] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,4] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,11] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,17] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,24] not a statement
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,39] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[173,8] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[175,8] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[175,32] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[178,8] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[178,35] '.class' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[178,39] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[182,8] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[182,26] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,8] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,15] illegal start of expression
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,27] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,36] ';' expected
[ERROR] \Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[190,1] reached end of file while parsing
[INFO] 29 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure

\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,25] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,58] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,59] not a statement
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[131,66] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,26] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,50] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,59] not a statement
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[156,68] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[161,4] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[161,33] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[161,49] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,4] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,11] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,17] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,24] not a statement
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[172,39] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[173,8] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[175,8] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[175,32] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[178,8] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[178,35] '.class' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[178,39] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[182,8] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[182,26] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,8] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,15] illegal start of expression
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,27] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[186,36] ';' expected
\Users\MyUser\My Documents\Projects\browsermob-proxy\src\main\java\org\browsermob\proxy\bricks\ProxyResource.java:[190,1] reached end of file while parsing

[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3 seconds
[INFO] Finished at: Mon Jan 23 17:02:18 CET 2012
[INFO] Final Memory: 21M/68M
[INFO] ------------------------------------------------------------------------

I am using JDK 1.6.0_30 and maven 2.2.1.

Any ideas?

Thanks

M.

Hi,

I would appreciate an update of the selenium-api dependency since the integrated version is quite out of date and many features and bugfixes had been implemented since then.

Is this possible?

So far, very nice tool!

Olmo

I've been trying to add headers to the proxy using CURL and other methods, but cannot figure out precisely how to format the query. Unfortunately, the docs can't clarify further for me. I presume this is simply a minor issue on my behalf, so I wrote up a quick test script to show what I've been trying. If you can please point me in the right direction, I'd appreciate it.

The script is here: https://gist.github.com/1812932

When testing apps which rely on external services or AJAX it'd be really useful to be able to set limits for latency or bandwidth using a regex so I could simulate e.g. really slow search results or external JS loading rather than outright failures. From the user's perspective this seems like it should be as simple as PUT /proxy/[port]/limit gaining the same regex support which the white/blacklist interface has.

Whenever i enabled HAR File Recording all pages requested through the proxy started to load very slow. I traced the problem back to the BrowserMobHttpClient (line 411).

UserAgentInfo uai = p.parse(userAgent);

This took about 5 seconds for each request, and then throwing an IOException.

I don't know if this is relevant but I have to use an upstream proxy to access the internet. The problem happened with all pages I tried (e.g. google.com, facebook.com, ...) both if i use the proxy via the REST API or start it from Java directly (in a selenium test). As I didn't need the browser I just removed the parsing of the UA string and it works fine and speedy again.

Hi,

i sometimes get the following exception: i am using a snapshot version cloned from the git repo on 02.02.2012

java.lang.ArrayIndexOutOfBoundsException: 88
at java.util.ArrayList.add(Unknown Source)
at org.browsermob.core.har.HarLog.addEntry(HarLog.java:36)
at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:512)
at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:393)
at org.browsermob.proxy.http.BrowserMobHttpRequest.execute(BrowserMobHttpRequest.java:149)
at org.browsermob.proxy.BrowserMobProxyHandler.proxyPlainTextRequest(BrowserMobProxyHandler.java:270)
at org.browsermob.proxy.selenium.SeleniumProxyHandler.handle(SeleniumProxyHandler.java:185)
at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1509)
at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1461)
at org.browsermob.proxy.jetty.http.HttpServer.service(HttpServer.java:892)
at org.browsermob.proxy.jetty.http.HttpConnection.service(HttpConnection.java:815)
at org.browsermob.proxy.jetty.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.browsermob.proxy.jetty.http.HttpConnection.handle(HttpConnection.java:832)
at org.browsermob.proxy.jetty.http.SocketListener.handleConnection(SocketListener.java:245)
at org.browsermob.proxy.jetty.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.browsermob.proxy.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

If I read the content of the response from the inside of the interceptor callback, I then receive the following error:
org.openqa.selenium.WebDriverException: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 1031; received: 0

If I only keep a reference on the HttpResponse object then try to read it after the response is completely processed, I get the following error:
java.lang.RuntimeException: java.io.IOException: Attempted read from closed stream.

Maybe should you wrap the response object to allow reading the stream several times.

Thanks

Please implement such important feature!

It'd probably be more appropriate if the proxy destruction operation returned a status code of 204. This is in keeping with many other ReST APIs.

I am interested to start using proxy but I don't know how to set up proxy .So can anyone help me out to setup proxy and how to use it.i will be pleased if somebody gives me any kind of documentation for the same.

As well as the options to control network speed and latency - an option to control jitter and packet loss ratio would be very useful to use the tool to simulate mobile network environments for actual mobile devices connecting through your wireless card or simulating it with user-agents being changed in the browser.

I wanted to test front-end handling of server failures but it appears that as of the current HEAD this feature cannot be completely configured using the REST interface - note the HTTP status below:

$ curl -X POST -d port=9091 http://localhost:8080/proxy
{"port":9091}
$ curl -X PUT -d 'regex=^.*/api/v1/.*$' -d status=503 http://localhost:8080/proxy/9091/blacklist
$ curl -si --proxy 127.0.0.1:9091 'http://127.0.0.1:8000/en/api/v1/…'
HTTP/1.1 200 OK
Date: Thu, 05 Jan 2012 16:18:06 GMT
Server: Jetty/5.1.x (Mac OS X/10.7.2 x86_64 java/1.6.0_29
Content-Type: text/plain
Content-Length: 0

This does provide a good place for me to test my handling of empty responses, though

Issue created from Patrick Lightbody's request.

The following test code will fail at line 38 due to a concurrent modification exception. This requires developers to implement the HttpRequestInterceptor interface to do this type of processing.

import java.util.Iterator;
import java.util.List;

import org.browsermob.core.har.Har;
import org.browsermob.core.har.HarEntry;
import org.browsermob.core.har.HarLog;
import org.browsermob.core.har.HarNameValuePair;
import org.browsermob.proxy.ProxyServer;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;

public class SeleniumProxyTesting {

    @Test
    public void testYahoo() throws Exception {
        ProxyServer server = new ProxyServer(9000);
        server.start();
        server.newHar("yahoo.com");

        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability(CapabilityType.PROXY, server.seleniumProxy());
        WebDriver driver = new FirefoxDriver(capabilities);
        driver.get("http://www.yahoo.com");

        boolean found = false;
        Har har = null;

        //This should never complete in this example
        while(!found) {
            har = server.getHar();
            HarLog harLog = har.getLog();
            List <HarEntry> entries = harLog.getEntries();
            for (Iterator<HarEntry> i = entries.iterator(); i.hasNext() && !found;) {
                for(HarNameValuePair queryData :  i.next().getRequest().getQueryString()) { //CME occurrs here
                    found = (queryData.getName().compareToIgnoreCase("FOO_BAR") == 0) && (queryData.getValue().contains("FOO_BAZ"));
                }
            }
        }
    }
}

When deleting cookies while running browsermob proxy in beta5/6, the cookie value is cached and appended on the next refresh of page. We tested through a webserver requesting the cookie value, deleting all cookies and refreshing the page; the cookie is deleted but the cookie info is saved in the headers then being appended with a comma on refresh . This behavior does not occur for every single cookie but we are able to reproduce it enough times to believe it is an issue with the proxy. First request with proxy cookieID=7029831823, 2nd request with cookies deleted through proxy: cookieID=3717988751,cookieID=7029831823. Reverting to beta4 solves this issue.

What's the timeline for this http request manipulation Rest API?

Note: I am looking for the feature of manipulating HTTP response body before a page renders in browsers.

I have a browsermob proxy running within our corporate network. To access the internet we need to go through a proxy.

Currently you cannot tell browsermob to use a proxy so it is unusable in this setup.

Please add the ability to specify a proxy server to forward requests to.