opendiffy / diffy Goto Github PK

Hi,

Is there any documentation for using diffy on Thrift comparison?
I couldn't find any Thrift related examples from README.
I have some thrift IDLs. I used apache thrift code generator to generate the Java files. Then used "javac" and "jar" command to combine those java class files into a single JAR archive file.

Tried to run diffy with this command
...
-service.protocol=thrift
-allowHttpSideEffects=true
-enableThriftMux='true'
-thrift.jar='/path/to/thrift.jar'
-thrift.serviceClass=$SomeServiceName

Then I got such error:
Error in custom provider, ai.diffy.lifter.ThriftLifter$InvalidServiceException: Invalid service: ....

I noticed that the filenames returned empty in this function:
https://github.com/opendiffy/diffy/blob/master/src/main/scala/ai/diffy/proxy/ThriftDifferenceProxy.scala#L37

Must the fat jar that feeds diffy be generated by Scrooge?

Really appreciate your time here 😊

From: twitter-archive/diffy#76
@pedromss: Mind if we move the conversation here.

I am not able to get this to work at all.

First of all, curl response header below will indicate that my services are working.

$ curl -IsSk https://localhost:50111

HTTP/1.1 200 OK
Date: Tue, 14 Jul 2020 03:49:31 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Last-Modified: Thu, 19 Mar 2020 12:46:05 GMT
ETag: "1b3-5a1348d58c64f"
Accept-Ranges: bytes
Content-Length: 435
Vary: Accept-Encoding
Cache-Control: max-age=2592000
Expires: Thu, 13 Aug 2020 03:49:31 GMT
Connection: close
Content-Type: text/html; charset=UTF-8


$ curl -IsSk https://localhost:50120


HTTP/1.1 200 OK
Date: Tue, 14 Jul 2020 03:49:50 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Last-Modified: Thu, 19 Mar 2020 12:47:14 GMT
ETag: "1b3-5a13491710daf"
Accept-Ranges: bytes
Content-Length: 435
Vary: Accept-Encoding
Cache-Control: max-age=2592000
Expires: Thu, 13 Aug 2020 03:49:50 GMT
Connection: close
Content-Type: text/html; charset=UTF-8

$ curl -IsSk https://localhost:50122


HTTP/1.1 200 OK
Date: Tue, 14 Jul 2020 03:49:55 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Last-Modified: Thu, 19 Mar 2020 12:47:14 GMT
ETag: "1b3-5a13491710daf"
Accept-Ranges: bytes
Content-Length: 435
Vary: Accept-Encoding
Cache-Control: max-age=2592000
Expires: Thu, 13 Aug 2020 03:49:55 GMT
Connection: close
Content-Type: text/html; charset=UTF-8

I run the diffy instance like this:

docker run --rm \
  -p 8880:8880 -p 8881:8881 -p 8888:8888 \
  diffy/diffy \
    -candidate=localhost:50111 \
    -master.primary=localhost:50120 \
    -master.secondary=localhost:50122 \
    -responseMode='primary' \
    -service.protocol=https \
    -serviceName="Tier-Service" \
    -proxy.port=:8880 \
    -admin.port=:8881 \
    -http.port=:8888 \
    -rootUrl=localhost:8888 \
    -summary.email='[email protected]' \
    -summary.delay="5"

Please note that I am not running on daemon mode. The command above outputs the following lines in the logs.

03:55:35.532 [main] INFO com.twitter.util.logging.Slf4jBridgeUtility$ - org.slf4j.bridge.SLF4JBridgeHandler installed.
03:55:37.399 [main] INFO ai.diffy.Main$ - Process started
03:55:38.291 [main] INFO ai.diffy.Main$ - Serving admin http on 0.0.0.0/0.0.0.0:8881
03:55:42.513 [main] WARN com.twitter.finagle.util.DefaultTimer - Can not service-load a timer. Using JavaTimer instead.
03:55:42.538 [main] INFO com.twitter.finagle - Finagle version 19.8.0 (rev=91a366ca88ba7231eeced7987f3835ff1eb0a0c1) built at 20190805-173757
03:55:48.836 [main] WARN ai.diffy.DiffyServiceModule$ - Malformed resource mapping: [Ljava.lang.String;@48c4245d. Should be <pattern>;<resource-name>
03:55:50.794 [main] INFO com.twitter.finagle - candidate: name resolution failed  (local dtab: Dtab())
java.lang.IllegalArgumentException: requirement failed: You must specify host and port in hosts: localhost:50111:443
        at scala.Predef$.require(Predef.scala:281)
        at com.twitter.finagle.util.InetSocketAddressUtil$.$anonfun$parseHostPorts$3(InetSocketAddressUtil.scala:38)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
        at scala.collection.TraversableLike.map(TraversableLike.scala:237)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
        at com.twitter.finagle.util.InetSocketAddressUtil$.parseHostPorts(InetSocketAddressUtil.scala:37)
        at com.twitter.finagle.InetResolver.$anonfun$bind$1(InetResolver.scala:166)
        at com.twitter.util.Try$.apply(Try.scala:26)
        at com.twitter.finagle.InetResolver.bind(InetResolver.scala:166)
        at com.twitter.finagle.BaseResolver.eval(Resolver.scala:171)
        at com.twitter.finagle.Client.newService(Client.scala:48)
        at com.twitter.finagle.Client.newService$(Client.scala:47)
        at com.twitter.finagle.Http$Client.newService(Http.scala:184)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:108)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:88)
        at ai.diffy.proxy.DifferenceProxy.$init$(DifferenceProxy.scala:57)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.<init>(HttpDifferenceProxy.scala:93)
        at ai.diffy.proxy.DifferenceProxyModule$.providesDifferenceProxy(DifferenceProxy.scala:30)
        at ai.diffy.proxy.DifferenceProxyModule$$$FastClassByGuice$$16957c87.invoke(<generated>)
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:84)
        at com.twitter.inject.app.internal.InstalledModules$.create(InstalledModules.scala:38)
        at com.twitter.inject.app.App.loadModules(App.scala:146)
        at com.twitter.inject.app.App.loadModules$(App.scala:141)
        at ai.diffy.MainService.loadModules(Main.scala:10)
        at com.twitter.inject.app.App.main(App.scala:55)
        at com.twitter.inject.app.App.main$(App.scala:54)
        at com.twitter.inject.server.TwitterServer.main(TwitterServer.scala:173)
        at com.twitter.inject.server.TwitterServer.main$(TwitterServer.scala:172)
        at ai.diffy.MainService.main(Main.scala:10)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.twitter.app.App.$anonfun$nonExitingMain$4(App.scala:364)
        at scala.Option.foreach(Option.scala:274)
        at com.twitter.app.App.nonExitingMain(App.scala:363)
        at com.twitter.app.App.nonExitingMain$(App.scala:344)
        at ai.diffy.MainService.nonExitingMain(Main.scala:10)
        at com.twitter.app.App.main(App.scala:333)
        at com.twitter.app.App.main$(App.scala:331)
        at ai.diffy.MainService.main(Main.scala:10)
        at ai.diffy.Main.main(Main.scala)
03:55:50.968 [main] INFO com.twitter.finagle - primary: name resolution failed  (local dtab: Dtab())
java.lang.IllegalArgumentException: requirement failed: You must specify host and port in hosts: localhost:50120:443
        at scala.Predef$.require(Predef.scala:281)
        at com.twitter.finagle.util.InetSocketAddressUtil$.$anonfun$parseHostPorts$3(InetSocketAddressUtil.scala:38)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
        at scala.collection.TraversableLike.map(TraversableLike.scala:237)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
        at com.twitter.finagle.util.InetSocketAddressUtil$.parseHostPorts(InetSocketAddressUtil.scala:37)
        at com.twitter.finagle.InetResolver.$anonfun$bind$1(InetResolver.scala:166)
        at com.twitter.util.Try$.apply(Try.scala:26)
        at com.twitter.finagle.InetResolver.bind(InetResolver.scala:166)
        at com.twitter.finagle.BaseResolver.eval(Resolver.scala:171)
        at com.twitter.finagle.Client.newService(Client.scala:48)
        at com.twitter.finagle.Client.newService$(Client.scala:47)
        at com.twitter.finagle.Http$Client.newService(Http.scala:184)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:108)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:88)
        at ai.diffy.proxy.DifferenceProxy.$init$(DifferenceProxy.scala:58)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.<init>(HttpDifferenceProxy.scala:93)
        at ai.diffy.proxy.DifferenceProxyModule$.providesDifferenceProxy(DifferenceProxy.scala:30)
        at ai.diffy.proxy.DifferenceProxyModule$$$FastClassByGuice$$16957c87.invoke(<generated>)
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:84)
        at com.twitter.inject.app.internal.InstalledModules$.create(InstalledModules.scala:38)
        at com.twitter.inject.app.App.loadModules(App.scala:146)
        at com.twitter.inject.app.App.loadModules$(App.scala:141)
        at ai.diffy.MainService.loadModules(Main.scala:10)
        at com.twitter.inject.app.App.main(App.scala:55)
        at com.twitter.inject.app.App.main$(App.scala:54)
        at com.twitter.inject.server.TwitterServer.main(TwitterServer.scala:173)
        at com.twitter.inject.server.TwitterServer.main$(TwitterServer.scala:172)
        at ai.diffy.MainService.main(Main.scala:10)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.twitter.app.App.$anonfun$nonExitingMain$4(App.scala:364)
        at scala.Option.foreach(Option.scala:274)
        at com.twitter.app.App.nonExitingMain(App.scala:363)
        at com.twitter.app.App.nonExitingMain$(App.scala:344)
        at ai.diffy.MainService.nonExitingMain(Main.scala:10)
        at com.twitter.app.App.main(App.scala:333)
        at com.twitter.app.App.main$(App.scala:331)
        at ai.diffy.MainService.main(Main.scala:10)
        at ai.diffy.Main.main(Main.scala)
03:55:51.028 [main] INFO com.twitter.finagle - secondary: name resolution failed  (local dtab: Dtab())
java.lang.IllegalArgumentException: requirement failed: You must specify host and port in hosts: localhost:50122:443
        at scala.Predef$.require(Predef.scala:281)
        at com.twitter.finagle.util.InetSocketAddressUtil$.$anonfun$parseHostPorts$3(InetSocketAddressUtil.scala:38)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
        at scala.collection.TraversableLike.map(TraversableLike.scala:237)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
        at com.twitter.finagle.util.InetSocketAddressUtil$.parseHostPorts(InetSocketAddressUtil.scala:37)
        at com.twitter.finagle.InetResolver.$anonfun$bind$1(InetResolver.scala:166)
        at com.twitter.util.Try$.apply(Try.scala:26)
        at com.twitter.finagle.InetResolver.bind(InetResolver.scala:166)
        at com.twitter.finagle.BaseResolver.eval(Resolver.scala:171)
        at com.twitter.finagle.Client.newService(Client.scala:48)
        at com.twitter.finagle.Client.newService$(Client.scala:47)
        at com.twitter.finagle.Http$Client.newService(Http.scala:184)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:108)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:88)
        at ai.diffy.proxy.DifferenceProxy.$init$(DifferenceProxy.scala:59)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.<init>(HttpDifferenceProxy.scala:93)
        at ai.diffy.proxy.DifferenceProxyModule$.providesDifferenceProxy(DifferenceProxy.scala:30)
        at ai.diffy.proxy.DifferenceProxyModule$$$FastClassByGuice$$16957c87.invoke(<generated>)
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:84)
        at com.twitter.inject.app.internal.InstalledModules$.create(InstalledModules.scala:38)
        at com.twitter.inject.app.App.loadModules(App.scala:146)
        at com.twitter.inject.app.App.loadModules$(App.scala:141)
        at ai.diffy.MainService.loadModules(Main.scala:10)
        at com.twitter.inject.app.App.main(App.scala:55)
        at com.twitter.inject.app.App.main$(App.scala:54)
        at com.twitter.inject.server.TwitterServer.main(TwitterServer.scala:173)
        at com.twitter.inject.server.TwitterServer.main$(TwitterServer.scala:172)
        at ai.diffy.MainService.main(Main.scala:10)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.twitter.app.App.$anonfun$nonExitingMain$4(App.scala:364)
        at scala.Option.foreach(Option.scala:274)
        at com.twitter.app.App.nonExitingMain(App.scala:363)
        at com.twitter.app.App.nonExitingMain$(App.scala:344)
        at ai.diffy.MainService.nonExitingMain(Main.scala:10)
        at com.twitter.app.App.main(App.scala:333)
        at com.twitter.app.App.main$(App.scala:331)
        at ai.diffy.MainService.main(Main.scala:10)
        at ai.diffy.Main.main(Main.scala)
03:55:51.474 [main] INFO ai.diffy.Main$ - Resolving Finagle clients before warmup
03:55:51.510 [main] INFO com.twitter.finagle - candidate resolved to Failed(java.lang.IllegalArgumentException: requirement failed: You must specify host
and port in hosts: localhost:50111:443)
03:55:51.515 [main] INFO com.twitter.finagle - primary resolved to Failed(java.lang.IllegalArgumentException: requirement failed: You must specify host an
d port in hosts: localhost:50120:443)
03:55:51.520 [main] INFO com.twitter.finagle - secondary resolved to Failed(java.lang.IllegalArgumentException: requirement failed: You must specify host
and port in hosts: localhost:50122:443)
03:55:51.544 [main] INFO ai.diffy.Main$ - Done resolving clients: [candidate, primary, secondary].
03:55:52.092 [main] INFO ai.diffy.workflow.Workflow - Scheduling ai.diffy.workflow.FunctionalReport at 2020-07-14 03:55:52 +0000
03:56:00.990 [main] INFO com.twitter.finatra.http.routing.HttpRouter - Adding routes
GET     /api/1/overview
GET     /api/1/report
GET     /api/1/endpoints
GET     /api/1/endpoints/:endpoint/stats
GET     /api/1/endpoints/:endpoint/fields/:path/results
GET     /api/1/endpoints/:endpoint/fields/:path/results/:index
GET     /api/1/requests/:id
GET     /api/1/clear
POST    /api/1/email
GET     /api/1/info
GET     /
GET     /css/:*
GET     /scripts/:*
03:56:01.035 [main] INFO ai.diffy.Main$ - Warming up.
03:56:01.434 [main] INFO ai.diffy.Main$ - http server started on port: 0.0.0.0/0.0.0.0:8888
03:56:01.445 [main] INFO ai.diffy.Main$ - admin http server started on port 8881
03:56:01.472 [main] INFO ai.diffy.Main$ - diffy started.
03:56:01.473 [main] INFO ai.diffy.Main$ - Startup complete, server awaiting.
03:56:01.477 [main] INFO com.twitter.inject.server.Awaiter$ - Awaiting 2 awaitables:
com.twitter.finagle.server.ListeningStackServer$$anon$1
com.twitter.finagle.server.ListeningStackServer$$anon$1

Although it mentions Startup complete You can see from the output that there were several errors (extracted below) (P.S a side question, why does the log indicate that it is adding some different routes?)

03:55:42.513 [main] WARN com.twitter.finagle.util.DefaultTimer - Can not service-load a timer. Using JavaTimer instead.
03:55:51.028 [main] INFO com.twitter.finagle - secondary: name resolution failed  (local dtab: Dtab())
java.lang.IllegalArgumentException: requirement failed: You must specify host and port in hosts: localhost:50122:443
03:55:51.510 [main] INFO com.twitter.finagle - candidate resolved to Failed(java.lang.IllegalArgumentException: requirement failed: You must specify host
and port in hosts: localhost:50111:443)
03:55:51.515 [main] INFO com.twitter.finagle - primary resolved to Failed(java.lang.IllegalArgumentException: requirement failed: You must specify host an
d port in hosts: localhost:50120:443)
03:55:51.520 [main] INFO com.twitter.finagle - secondary resolved to Failed(java.lang.IllegalArgumentException: requirement failed: You must specify host

Now, doing the following curl

curl http://localhost:8880

outputs the following in the log.

04:00:33.273 [finagle/netty4-2-3] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /172.17.0.
1:40082, downstream address: n/a, label: ).
java.lang.IllegalArgumentException: requirement failed: You must specify host and port in hosts: localhost:50120:443
        at scala.Predef$.require(Predef.scala:281)
        at com.twitter.finagle.util.InetSocketAddressUtil$.$anonfun$parseHostPorts$3(InetSocketAddressUtil.scala:38)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
        at scala.collection.TraversableLike.map(TraversableLike.scala:237)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
        at com.twitter.finagle.util.InetSocketAddressUtil$.parseHostPorts(InetSocketAddressUtil.scala:37)
        at com.twitter.finagle.InetResolver.$anonfun$bind$1(InetResolver.scala:166)
        at com.twitter.util.Try$.apply(Try.scala:26)
        at com.twitter.finagle.InetResolver.bind(InetResolver.scala:166)
        at com.twitter.finagle.BaseResolver.eval(Resolver.scala:171)
        at com.twitter.finagle.Client.newService(Client.scala:48)
        at com.twitter.finagle.Client.newService$(Client.scala:47)
        at com.twitter.finagle.Http$Client.newService(Http.scala:184)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:108)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:88)
        at ai.diffy.proxy.DifferenceProxy.$init$(DifferenceProxy.scala:58)
        at ai.diffy.proxy.SimpleHttpsDifferenceProxy.<init>(HttpDifferenceProxy.scala:93)
        at ai.diffy.proxy.DifferenceProxyModule$.providesDifferenceProxy(DifferenceProxy.scala:30)
        at ai.diffy.proxy.DifferenceProxyModule$$$FastClassByGuice$$16957c87.invoke(<generated>)
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:84)
        at com.twitter.inject.app.internal.InstalledModules$.create(InstalledModules.scala:38)
        at com.twitter.inject.app.App.loadModules(App.scala:146)
        at com.twitter.inject.app.App.loadModules$(App.scala:141)
        at ai.diffy.MainService.loadModules(Main.scala:10)
        at com.twitter.inject.app.App.main(App.scala:55)
        at com.twitter.inject.app.App.main$(App.scala:54)
        at com.twitter.inject.server.TwitterServer.main(TwitterServer.scala:173)
        at com.twitter.inject.server.TwitterServer.main$(TwitterServer.scala:172)
        at ai.diffy.MainService.main(Main.scala:10)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.twitter.app.App.$anonfun$nonExitingMain$4(App.scala:364)
        at scala.Option.foreach(Option.scala:274)
        at com.twitter.app.App.nonExitingMain(App.scala:363)
        at com.twitter.app.App.nonExitingMain$(App.scala:344)
        at ai.diffy.MainService.nonExitingMain(Main.scala:10)
        at com.twitter.app.App.main(App.scala:333)
        at com.twitter.app.App.main$(App.scala:331)
        at ai.diffy.MainService.main(Main.scala:10)
        at ai.diffy.Main.main(Main.scala)

I tried on the https as well, which gives the following error. So although my app is served over https it seems to still use http to initiate the propagation.

$ curl -k https://localhost:8880


curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Currently, I'm able to get only those requests which show a genuine difference.
Is there a way to fetch information about individual requests, irrespective of whether there is a diff or not, possibly using the /api/1/requests/:id endpoint? Moreover, how can we get the request ID for every request?

20:18:08.088 [finagle/netty4-2-6] INFO com.twitter.finagle - FailureAccrualFactory marking connection to "diffyproject.appspot.com:443" as dead. Remote Address: Inet(diffyproject.appspot.com/67.228.235.91:443,Map())

Is there a way to remove "OrderingDifference" noise from diffy ?

(i dont mean manually pressing the grey dot next to it on each diff)

I'm trying to export the report via the email extension provided by diffy.
After reading all the documentation and most of the code, I could figure out that specifying notifications.targetEmail flag would automate an email from Diffy <[email protected]>

Here is the docker command being used:

docker run --rm -d --name diffy-staging-clarisights \
  -ti -p 8880:8880 -p 8881:8881 -p 8888:8888 \
  diffy/diffy \
    -candidate=localhost:9000 \
    -master.primary=localhost:9001 \
    -master.secondary=localhost:9001 \
    -service.protocol=http \
    -serviceName="Production-Test" \
    -proxy.port=:8880 \
    -admin.port=:8881 \
    -http.port=:8888 \
    -rootUrl=localhost:8888 \
    -notifications.targetEmail="[email protected]" \
    -notifications.delay=10.minutes

Assuming that a SMTP service (namshi/smtp image) listening on localhost:25 accessible by the container. (I'm running both of them in a K8s pod). However, I'm not able the diffy is having issues sending the email. Is there a workaround to exporting data from diffy?

I run my Diffy jar .The develop Diffy script :

java -jar /Users/huangwenjiao/.Trash/diffy/diffy/target/scala-2.12/diffy-server.jar
-candidate= 'testdnapp.bitkinetic.com:9052'
-master.primary= 'devdnapp.bitkinetic.com:9052'
-master.secondary= 'prednapp.bitkinetic.com:9052'
-service.protocol=https
-serviceName=My-Service
-proxy.port=:8780
-admin.port=:8781
-http.port=:8789
-rootUrl='localhost:8789'
-summary.email="[email protected]"

And I can open http：//localhost:8770 in my browser.
But when I send my resquest ,it shown Exception propagated to the default monitor .And there is nothing difference in http：//localhost:877 .
Exception propagated to the default monitor (upstream address: /127.0.0.1:56232, downstream address: n/a, label: ).
ai.diffy.proxy.DifferenceProxy$NoResponseException$: No responses provided by diffy
at ai.diffy.proxy.DifferenceProxy$NoResponseException$.(DifferenceProxy.scala)
at ai.diffy.proxy.DifferenceProxy$.(DifferenceProxy.scala:36)
at ai.diffy.proxy.DifferenceProxy$.(DifferenceProxy.scala)
at ai.diffy.proxy.SimpleHttpDifferenceProxy$.$anonfun$httpSideEffectsFilter$1(HttpDifferenceProxy.scala:59)
at com.twitter.finagle.Filter$$anon$16.apply(Filter.scala:406)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.ServiceProxy.apply(ServiceProxy.scala:12)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.tracing.AnnotatingTracingFilter.apply(TraceInitializerFilter.scala:140)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.tracing.ServerDestTracingFilter.apply(DestinationTracing.scala:38)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.service.DeadlineFilter.apply(DeadlineFilter.scala:265)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.filter.ExceptionSourceFilter.apply(ExceptionSourceFilter.scala:50)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(MkJvmFilter.scala:30)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.tracing.AnnotatingTracingFilter.apply(TraceInitializerFilter.scala:140)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.http.filter.HttpNackFilter.$anonfun$apply$1(HttpNackFilter.scala:159)
at com.twitter.util.Local.let(Local.scala:4979)
at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
at com.twitter.finagle.http.filter.HttpNackFilter.apply(HttpNackFilter.scala:159)
at com.twitter.finagle.http.filter.HttpNackFilter.apply(HttpNackFilter.scala:113)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.http.filter.PayloadSizeFilter.apply(PayloadSizeFilter.scala:123)
at com.twitter.finagle.http.filter.PayloadSizeFilter.apply(PayloadSizeFilter.scala:47)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.server.BackupRequest$$anon$1$$anon$2.apply(BackupRequest.scala:31)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.http.filter.ServerContextFilter.$anonfun$apply$1(ContextFilter.scala:15)
at com.twitter.util.Local.let(Local.scala:4979)
at com.twitter.finagle.context.MarshalledContext.letLocal(MarshalledContext.scala:157)
at com.twitter.finagle.context.MarshalledContext.let(MarshalledContext.scala:104)
at com.twitter.finagle.http.codec.HttpContext$.read(HttpContext.scala:90)
at com.twitter.finagle.http.filter.ServerContextFilter.apply(ContextFilter.scala:15)
at com.twitter.finagle.http.filter.ServerContextFilter.apply(ContextFilter.scala:12)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:13)
at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$3(HttpServerTraceInitializer.scala:20)
at com.twitter.finagle.http.TraceInfo$.$anonfun$letTraceIdFromRequestHeaders$1(TraceInfo.scala:105)
at com.twitter.util.Local.let(Local.scala:4979)
at com.twitter.finagle.context.MarshalledContext.letLocal(MarshalledContext.scala:157)
at com.twitter.finagle.context.MarshalledContext.let(MarshalledContext.scala:90)
at com.twitter.finagle.tracing.Trace$.letId(Trace.scala:104)
at com.twitter.finagle.http.TraceInfo$.letTraceIdFromRequestHeaders(TraceInfo.scala:103)
at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$2(HttpServerTraceInitializer.scala:20)
at com.twitter.util.Local.let(Local.scala:4979)
at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
at com.twitter.finagle.tracing.Trace$.letTracer(Trace.scala:124)
at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$1(HttpServerTraceInitializer.scala:20)
at com.twitter.finagle.Filter$$anon$16.apply(Filter.scala:406)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.filter.MonitorFilter.apply(MonitorFilter.scala:66)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.http.filter.DtabFilter.apply(DtabFilter.scala:25)
at com.twitter.finagle.http.filter.DtabFilter.apply(DtabFilter.scala:12)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
at com.twitter.finagle.http.codec.ResponseConformanceFilter$.apply(ResponseConformanceFilter.scala:23)
at com.twitter.finagle.http.codec.ResponseConformanceFilter$.apply(ResponseConformanceFilter.scala:18)
at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
at com.twitter.finagle.http.codec.HttpServerDispatcher.dispatch(HttpServerDispatcher.scala:42)
at com.twitter.finagle.http.codec.HttpServerDispatcher.dispatch(HttpServerDispatcher.scala:23)
at com.twitter.finagle.http.exp.GenStreamingSerialServerDispatcher.$anonfun$dispatchAndHandleFn$2(GenStreamingSerialServerDispatcher.scala:79)
at com.twitter.util.Local.let(Local.scala:4979)
at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
at com.twitter.finagle.http.exp.GenStreamingSerialServerDispatcher.$anonfun$dispatchAndHandleFn$1(GenStreamingSerialServerDispatcher.scala:77)
at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1798)
at com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:250)
at com.twitter.util.Promise$FutureTransformer.k(Promise.scala:250)
at com.twitter.util.Promise$Transformer.apply(Promise.scala:225)
at com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:101)
at com.twitter.util.Promise$WaitQueue$$anon$1.run(Promise.scala:96)
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:167)
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:126)
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:243)
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:78)
at com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:96)
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:797)
at com.twitter.util.Promise.update(Promise.scala:769)
at com.twitter.util.Promise.setValue(Promise.scala:745)
at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)
at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:169)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at com.twitter.finagle.netty4.http.handler.BadRequestHandler.channelRead(BadRequestHandler.scala:41)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at com.twitter.finagle.netty4.http.handler.UriValidatorHandler$.channelRead(UriValidatorHandler.scala:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

In the expanded response view I get diffs on URLs that are the same. It only happens to values that are URLs.

The following is a detailed error message, please help me, is it wrong with my packaged order: sbt ~package

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: com/twitter/app/Flags
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: com.twitter.app.Flags
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

Hello,

Is there a way to display headers for the POST request that resulted in differences?

Thank you

@puneetkhanduri

I often encounter diffy proxy processing downstream three returns to normal, get raw_diff data is normal, but the report does not display any results, the data obtained by the api layer is {}
I started the multiple Diffy instance on the same host, bound to a different port, and there is no error message.

When I kill all the diffy instances, restarting one diffy instance, then the report will show the right diff result.

on the same host, start multiple diffy instances for service, will it affect each other?

Suffering from the above problems, I often run 10 times, 5-6 reports have no data, I went into diffy / compare and diffy / analysis to println debugging information, as follows:

Thank you very much for your guidance.

Hi,

I liked the diff idea. But got blocked at very beginning because I am not sure how we can pass different requestHeader/Cookies to primary, secondary and candidate deployment?

Note: For real testing, there is need to pass different secureToken to each deployment.

Hello! I'm looking at this tool right know for using it.

I think it would be awesome and pretty useful if you can have some way of making a set of requests to test your api. I mean, if I understand correctly, supposing you want to make N requests, based in a file with a pool of good data as query params to test the api well...Well, you will have to make a script with a for loop doing a curl for example, based on a file...Or using something like postman for the same...But if you add a param for example when you initialize diffy, and do this for loop in diffy, making the requests, then diffy would be 100% auto-sufficient for this kind of typical tests.

Using docker, wanting to compare my local dev instance with production, running:

docker run -d --name diffy-01 \
  -p 8880:8880 -p 8881:8881 -p 8888:8888 \
  diffy/diffy \
    -candidate=localhost:3000 \
    -master.primary=www.myproductionenv.com \
    -master.secondary=www.myproductionenv.com \
    -service.protocol=http \
    -serviceName="Test" \
    -proxy.port=:8880 \
    -admin.port=:8881 \
    -http.port=:8888 \
    -rootUrl=localhost:8888

Then: curl localhost:8880/my/path/to/page

I get nothing. No results in the diffy comparison window. No errors. No logging. No network problems.

@puneetkhanduri,

I have been getting HTTP status code (400) for our application.

Logs:
09:26:46.165 [finagle/netty4-18] INFO ai.diffy.lifter.HttpLifter - Return(Response("HTTP/1.1 Status(400)"))
09:26:46.165 [finagle/netty4-18] INFO ai.diffy.lifter.HttpLifter - Return(Response("HTTP/1.1 Status(400)"))
09:26:46.165 [finagle/netty4-18] INFO ai.diffy.lifter.HttpLifter - Return(Response("HTTP/1.1 Status(400)"))

Command:
java -jar diffy-server.jar -master.primary=prod.XXXXXX.com:443 -master.secondary=prod.xxxxxxx.com:443 -candidate=feature.xxxxxx.com:443 -service.protocol=https -serviceName=prod -proxy.port=:55545 -admin.port=:55546 -allowHttpSideEffects=true -https.port=:55547 -rootUrl=xxx.xxxxxx.com:55547 -log.append=true

Curl -l "http://xxx.xxxxxx.com:55545/request"

Request not hit to our servers (candidate,primary and secondary).

@puneetkhanduri
I try to use diffy with thrift protocol. but the Scrooge which diffy use can not parse apache offical thrift which can parse by apache thrift 0.9.2 normally.

the thrift idl define like this:
include "common.thrift"
struct Demo {
common.DataType datatype = DataType.JSON
}

when I run diffy. it raise errors like these:

Caused by: com.twitter.scrooge.frontend.FileParseException: Exception parsing: aweme.thrift
Caused by: com.twitter.scrooge.frontend.TypeNotFoundException: DataType

@puneetkhanduri can you help me? thank you very much.

btw: another problem when parse thrift file by scrooge like this. can we resolve the limitation of the Scrooge library we use to parse the thrift into AST ?
The offical apache thrift support it.

hi,
I pulled the latest code. But I have an error when i just use one request. What does the error means? How can i fix it？ Thank you！

18:25:37.902 [finagle/netty4-8] INFO ai.diffy.lifter.HttpLifter - Return(Response("HTTP/1.1 Status(200)"))
18:25:38.954 [finagle/netty4-8] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /10.10.35.183:50718, downstream address: n/a, label: ).
java.lang.StackOverflowError: null
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)
at java.util.regex.Pattern$Branch.match(Pattern.java:4604)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)
at java.util.regex.Pattern$Branch.match(Pattern.java:4604)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)
at java.util.regex.Pattern$Branch.match(Pattern.java:4604)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)
at java.util.regex.Pattern$Branch.match(Pattern.java:4604)

I try to check log in admin tool. But it show You have not configured a logging handler implementation for TwitterServer. . How to configure logging handler in diffy server ?

I am trying to run diffy against webservers that sit behind a Traefik reverse proxy. It seems like Finarta resolves the hosts of the servers on startup, but what is being resolved is the proxy server and not the downstream servers I am trying to test with Diffy. Is it possible to configure the Diffy proxy somehow to point to the downstream servers of the reverse proxy?

I'm not really a Scala expert, but if I am understanding this correctly and this can be fixed, I'd be happy to do so with some guidance.

Hi Punith,
I wanted to use DIFFY in testing my RESTful Service, could you please let me know how can I setup DIFFY on wondows machine. I could not find any useful link from internet.

Regards
Shyju

In order to use the Dashboard behind an api gateway, that forward the path to the underneath service, we should be able to specify in the dashboard html the apiRoot, and prefix it to the scripts (.js/.css) files path.

In our use case, we have the apigateway that proxy the requests and redirect it to the underneath service. We have configured the apiRoot variable to /diffyservice, which works ok for the calls made by the js scripts, but the Dashboard html is not reflecting the same behaviour.

I suggest to add the {{apiRoot}} to the scripts path in order to fix it.

I can create the PR if needed.

Hi,
we want to use diffy in our project.
For this we need an additional function to log all incoming requests.
Does diffy has this function or can it be implemented?

Hi there, I just switched from the old diffy to this one, with the same config. I see the requests coming and everything is green, but the admin page is broken, I don't see the graphics and I got warnings about linter:

@sky-eduardolopes

sky-eduardolopes @sky-eduardolopes Jun 25 13:51
on another note, hoping someone can assist on this: in my tests I'm getting a response larger than the limit and causing a 500. I get " Throw(com.twitter.finagle.http.TooLongMessageException: Response entity too large" on the logs. Looking at a related issue I saw a suggestion to set "com.twitter.finagle.Http.param.MaxRequestSize" when starting the service. Since my error is due to response size I set two parameters when starting diffy, as such: "java -jar -Dcom.twitter.finagle.Http.param.MaxRequestSize=15728640.bytes -Dcom.twitter.finagle.Http.param.MaxResponseSize=15728640.bytes /root/Diffy/diffy-server.jar". Does anyone know how to make Diffy accept a larger response?

Puneet Khanduri @puneetkhanduri Jun 25 19:27
@sky-eduardolopes : thanks for your kind words. The flags you are already setting should be sufficient if your payload is less than 15 MB. Can you confirm the size of your payload?

sky-eduardolopes @sky-eduardolopes Jun 25 19:30
Content-Length: 6454768
Due to the lack of a responseMode I'm still using a quite old version of the openDiffy, before the isotope...
Could this be the reason for those flags not applying?

sky-eduardolopes @sky-eduardolopes Jun 25 19:37
Due to the lack of a responseMode I'm still using a quite old version of the openDiffy, before the isotope...

actually one that I patched with theresponseMode commit from twitter/diffy

sky-eduardolopes @sky-eduardolopes Jun 25 20:32
FYI I had asked on the finagle gitter before asking here and this is what they told me:
"those are Stack params which are not read from System properties. They’re generally set as code via the Finagle client"

External link within project.

We don't understand how to run tests inside internal closed networks without Internet Access. What for do you use this link?

Hi,

We are interested into using Diffy with the flow where traffic is directly shadowed from load balancer to Diffy machine.

However, for such flow we need to have healtcheck/endpoint where it would be possible to indicate that traffic should be stopped shadowed to Diffy machine, without turning Diffy itself (for analyzing logs, after traffic is stopped)

I could not find it, but maybe Diffy already has such functionality? Can you advise on that?

I'm trying to set up diffy to test an HTTPS API.
Here is the Docker command I'm running:

$ docker run --rm -d --name diffy-test \
  -ti -p 8880:8880 -p 8881:8881 -p 8888:8888 -p 123:123 \
  diffy/diffy \
    -candidate='https://<candidate>.com' \
    -master.primary='https://<prod>.com' \
    -master.secondary='https://<prod>.com' \
    -service.protocol=https \
    -serviceName="Beta-Testing" \
    -proxy.port=:8880 \
    -admin.port=:8881 \
    -https.port=:123 \
    -rootUrl=localhost:8888

I'm sending traffic on 8880 but there is no diff on the Web UI.
What am I missing here?
What is the correct way to setup diffy with HTTPS?

I have tried a different variation of Diffy (on Windows, on centos, on docker) and every time results are the same - it crashes after starting to shadow traffic from load balancer (~150 req/s) or generating it with Tsung to Diffy (even ~1 req/s). Version that has been mainly used while trying to set up flow is 19.11.8.00 but with others, the same issue happened.
However, with the old version from https://github.com/twitter-archive/diffy/tags, this scenario does not happen, but other issues occur there (not all headers are caught, etc) and we would like to use mentioned or a newer version of the tool. Can please advise how to solve this?

Also, as I have understood and found out this could be closely related to the issues in the admin panel:

Please find attached Diffy logs with the issue below:
Diffy_logs.txt

My builds have been erroring out since yesterday.

Apparently, the summary.email flag is now mandatory and this was breaking my builds.
Since the container image hosted on Docker Hub just has a single image (latest tag), I'm unable to use a previous version of Diffy.

Breaking changes should be communicated to the users who depend on the project.
I request the authors to follow some sort of release cycle and/or software versioning to provide developers the safety and freedom to use a suitable copy of the project.

I am finding that I cannot use diffy because of this error

14:04:35.474 [finagle/netty4-2-3] WARN com.twitter.finagle - Unhandled exception in connection with <candidate-endpoint>, shutting down connection io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1329) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ... 20 common frames omitted Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1499) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1513) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1397) ... 24 common frames omitted Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:259) at sun.security.validator.Validator.validate(Validator.java:262) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ... 33 common frames omitted Caused by: java.security.cert.CertPathValidatorException: validity check failed at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141) at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ... 39 common frames omitted Caused by: java.security.cert.CertificateExpiredException: NotAfter: Tue Jul 07 10:35:38 EDT 2020 at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629) at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ... 44 common frames omitted 14:04:35.475 [finagle/netty4-2-3] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1329) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ... 20 common frames omitted Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1499) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1513) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1397) ... 24 common frames omitted Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:259) at sun.security.validator.Validator.validate(Validator.java:262) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ... 33 common frames omitted Caused by: java.security.cert.CertPathValidatorException: validity check failed at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141) at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ... 39 common frames omitted Caused by: java.security.cert.CertificateExpiredException: NotAfter: Tue Jul 07 10:35:38 EDT 2020 at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629) at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ... 44 common frames omitted

It seems that the certificate expired July 7th, 2020, which was 15 days ago. Will this issue be resolved?

when ran the Main.scala, I got the error "Error:(8, 17) object thriftscala is not a member of package ai.diffy" ,and i could not find thriftscala.

Step1:clone the code
Step2:cd diffy
Step3:./sbt assembly

then it failed with some errors:

How to get the diffy-server.jar, please? I do not want to build project by myself.

According to RFC-2616, HTTP header names are case-insensitive. Diffy should then by default and/or controlled via an option, make case-insensitive comparisons of header names.

This problem showed up for us doing a comparison of a Phoenix app that replaces a Rails app. Phoenix by default returns header names in all lower-case whereas Rails uses title-case.

Currently, I see only Primitive Difference being flagged. Is there a feature which can:

1. detect other differences like ordering changes in a list?
2. show aggregated view of results where we get % of the difference for some fields(keys).
3. show metrics for some selected fields.
4. record the request patterns.

Also, is there an article where I can read about how it works internally? Does it provide any other API which can be used for maybe storing the diffy results.

@puneetkhanduri
I found one new bug about diffy when I use it with thrift.
when I send or receive binary types through Thrift, my bytebuffer has garbage prepended to it. This does not happen with TCompactProtocol.

I have a very basic service:

namespace java com.xorlev.heimdall.core.thrift

struct GetResult {
1: optional binary value;
}

exception HeimdallException {
1: string message;
}
service HeimdallShard {
GetResult GetStreamAdV2(1: string key) throws(1: HeimdallException ex);
}

but the response like this: It include some dirty data.

so I guest the TBinaryProtocol.readBinary() has some bug?

@puneetkhanduri thank you very much for your any help!
other the same problem: twitter/scrooge#231

I have some base service use thrift protocol , like hiveserver2。I want to test when basic service upgrade, is there any problem with my program。

I run diffy with
java -jar diffy-server.jar -candidate=192.168.1.2:10000 -master.primary=192.168.1.3:10000 -master.secondary=192.168.1.4:10000 -service.protocol=thrift -thrift.jar='/opt/hbase/lib/hbase-thrift-1.4.10.jar' -serviceName=Fancy-Service -proxy.port=:8880 -admin.port=:8881 -http.port=:8888 -rootUrl="localhost:8888" -summary.email="[email protected]" -summary.delay="5" -allowHttpSideEffects=true -thrift.serviceClass=test-service -enableThriftMux=true

but get a error:

1. Error in custom provider, ai.diffy.lifter.ThriftLifter$InvalidServiceException: Invalid service:

Maybe there's something wrong with the thrift.jar and thrift.serviceClass configuration, but I don't know how to fix it

Really appreciate your time here

Can three different versions of servers be deployed on a single machine with this service?
java -jar diffy-server.jar
-candidate=172.16.3.1:9992
-master.primary=172.16.3.2:9990
-master.secondary=172.16.3.3:9991
-service.protocol=http
-serviceName=My-Service
-proxy.port=:8880
-admin.port=:8881
-http.port=:8888
-rootUrl='localhost:8888'

hi:

I have followed these steps in starting the diffy:

1. git clone https://github.com/opendiffy/diffy.git
2. cd diffy
3. ./example/run.sh start

And then the error happend. The log showed this :

Build Diffy
Build primary, secondary, and candidate servers
Deploy Diffy
Error: Unable to access jarfile ./target/scala-2.12/diffy-server.jar
Deploy primary, secondary, and candidate servers
!!!failed to start!!!
!!!failed to start!!!
!!!failed to start!!!
Wait for server to deploy
Send some traffic to your Diffy instance
Your Diffy UI can be reached at http://localhost:8888

I could not find the diffy-server.jar in any directory of the diffy project
I send 'http://localhost:8888' in browser but it didn't work

where is the diffy-server.jar?
How can i deal with it??

How can I set up 3 IP to a same HOSTNAME?

For example :

-candidate='test.domain.com' \
-master.primary='test.domain.com' \
-master.secondary='test.domain.com' \

The real candidate ip is 192.168.244.128, the master.primary ip is 192.168.244.129, the master.secondary ip is 192.168.244.120.

I get an error while server startup. Looks like its not overriding the port that I specify. It seems to be using localhost:10162:443 instead of localhost:10162. Am I doing anything wrong? Or is that a bug?

Release version: 19.11.8.00

Command:

java -jar diffy-server.jar \
    -candidate=localhost:10162 \
    -master.primary=localhost:10161 \
    -master.secondary=localhost:10161 \
    -service.protocol=https \
    -serviceName=my-api \
    -proxy.port=:9090 \
    -admin.port=:8881 \
    -https.port=:8888 \
    -rootUrl="localhost:8888" \
    -summary.email="[email protected]" \
    -summary.delay="5"

Error:

java.lang.IllegalArgumentException: requirement failed: You must specify host and port in hosts: localhost:10162:443
	at scala.Predef$.require(Predef.scala:281)
	at com.twitter.finagle.util.InetSocketAddressUtil$.$anonfun$parseHostPorts$3(InetSocketAddressUtil.scala:38)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
	at com.twitter.finagle.util.InetSocketAddressUtil$.parseHostPorts(InetSocketAddressUtil.scala:37)
	at com.twitter.finagle.InetResolver.$anonfun$bind$1(InetResolver.scala:166)
	at com.twitter.util.Try$.apply(Try.scala:26)
	at com.twitter.finagle.InetResolver.bind(InetResolver.scala:166)
	at com.twitter.finagle.BaseResolver.eval(Resolver.scala:171)
	at com.twitter.finagle.Client.newService(Client.scala:48)
	at com.twitter.finagle.Client.newService$(Client.scala:47)
	at com.twitter.finagle.Http$Client.newService(Http.scala:184)
	at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:119)
	at ai.diffy.proxy.SimpleHttpsDifferenceProxy.serviceFactory(HttpDifferenceProxy.scala:99)
	at ai.diffy.proxy.DifferenceProxy.$init$(DifferenceProxy.scala:55)
	at ai.diffy.proxy.SimpleHttpsDifferenceProxy.<init>(HttpDifferenceProxy.scala:104)
	at ai.diffy.proxy.DifferenceProxyModule$.providesDifferenceProxy(DifferenceProxy.scala:28)
	at ai.diffy.proxy.DifferenceProxyModule$$$FastClassByGuice$$16957c87.invoke(<generated>)
	at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
	at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
	at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
	at com.google.inject.Guice.createInjector(Guice.java:99)
	at com.google.inject.Guice.createInjector(Guice.java:84)
	at com.twitter.inject.app.internal.InstalledModules$.create(InstalledModules.scala:38)
	at com.twitter.inject.app.App.loadModules(App.scala:146)
	at com.twitter.inject.app.App.loadModules$(App.scala:141)
	at ai.diffy.MainService.loadModules(Main.scala:10)
	at com.twitter.inject.app.App.main(App.scala:55)
	at com.twitter.inject.app.App.main$(App.scala:54)
	at com.twitter.inject.server.TwitterServer.main(TwitterServer.scala:173)
	at com.twitter.inject.server.TwitterServer.main$(TwitterServer.scala:172)
	at ai.diffy.MainService.main(Main.scala:10)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.twitter.app.App.$anonfun$nonExitingMain$4(App.scala:364)
	at scala.Option.foreach(Option.scala:274)
	at com.twitter.app.App.nonExitingMain(App.scala:363)
	at com.twitter.app.App.nonExitingMain$(App.scala:344)
	at ai.diffy.MainService.nonExitingMain(Main.scala:10)
	at com.twitter.app.App.main(App.scala:333)
	at com.twitter.app.App.main$(App.scala:331)
	at ai.diffy.MainService.main(Main.scala:10)

The request is like this : http://localhost/signature/signature/get-signature-image-by-id?accountId=82963051108122962&id=1581862551226133507, will download a picture. If candicate and primary are same, it works. If not , will crash.

Will this feature merged into twitter/diffy ever be ported into opendiffy/diffy?

twitter-archive/diffy#48

I understand that it goes against the project philosophy, but it helps a lot when testing low traffic systems for which we allow side effects (eg. POST requests)

If there would be millions of requests to Diffy proxy, how to scale Diffy ?

Is it possible to start multiple Diffy servers by using same http.port so all requests could be load balanced but the result is still aggregated (not sure if the volume of the result would blow out service behind http.port as well) ?

In the admin UI, there is an option for /admin/logging endpoint. But when I go here I get the message:

You have not configured a logging handler implementation for TwitterServer.

Please add a dependency on one of: twitter-server-logback-classic, twitter-server-slf4j-jdk14, or twitter-server-slf4j-log4j12.

If I am running Diffy through Docker image, is it possible to do this? If so, how?

When I start screenshot test of my site I get a picture with error "CheckUrl ETIMEDOUT" instead of screenshots