If this is the first time you set up this prototype, I encourage you to skip the quick start and read the description below.

1. Start Mountebank:

docker run -p 2525:2525 -p 4545:4545 -p 4546:4546 -d jkris/mountebank:latest --allowInjection

2. Configure Mountebank imposters:

cd mountebank_config/curl
curl -X PUT -d @initial_setup.json http://localhost:2525/imposters

3. Start the Hystrix Dashboard:

docker run -d -p 8081:9002 --name hystrix-dashboard mlabouardy/hystrix-dashboard:latest

4. Start the prototype application:

cd Prototype
./gradlew jettyRun

5. Monitor circuit health through Hystrix Dashboard at http://localhost:8081/hystrix

Add the hystrix stream: http://<ip_address_of_host_machine>:8080/HystrixPrototype/hystrix.stream

6. Putting load on the prototype:

ab -n 10000 -c 10 http://localhost:8080/HystrixPrototype/v1/hello

Watch the Dashboard to see the requests going through. Go to Simulating slow response from the DOC for further testing.

The purpose of this prototype is to gain experience with Hystrix as a circuit breaker pattern in distributed architecture. We achieve this by:

1. Wrapping HTTP calls to downstream services in Hystrix commands.
2. Using Mountebank to control latency of downstream services (dependent-on components).
3. Using JMeter or Apache Benchmark to put load on the prototype.

The prototype is a very basic implementation of a web application, which uses Hystrix Commands to wrap HTTP requests. The web server exposes Hystrix metrics which can be used for monitoring the HTTP requests and the state of the circuit breaker in a Hystrix Dashboard.

The Hystrix commands will execute 2 HTTP GET requests to two different URLs, which should be controlled by Mountebank and can be manipulated by sending HTTP requests to the Mountebank server.

The following sections will explain the setup of the specific components.

• Setting up and configuring Mountebank
• Setting up Hystrix Dashboard
• Import the Prototype in Eclipse
• Running the prototype Java application
• Putting load on the prototype application
• Project Overview

Find links to the resources used in this manual in the references section

• Eclipse with Jetty or Tomcat support.
• Docker for running Mountebank and Hystrix Dashboard.
• CURL or Postman for configuring Mountebank.
• JMeter or Apache Benchmark for putting load on the prototype.

This guide assumes that the developer has knowledge about eclipse and project handling.

Mountebank imposters are used to simulate real dependent-on components (DOC) to the prototype. A docker image can be used for setting up the Mountebank server:

docker run -p 2525:2525 -p 4545:4545 -p 4546:4546 -d jkris/mountebank:latest --allowInjection

Access the mountebank server at: http://localhost:2525

Use Postman or cURL to configure Mountebank as described below:

• Open Postman and import the Postman collection:
  • Click the Import button located in the upper left side of the window and import the collection located here:
    • mountebank_config/postman/HystrixPrototype.postman_collection
  • You should now see 5 imported HTTP requests
• Send the request named [Stub] Initial setup to setup Mountebank the first time

Open a terminal in the root directory of the project and run the following commands:

cd mountebank_config/curl
curl -X PUT -d @initial_setup.json http://localhost:2525/imposters

You should now have created two stubs on ports 4545 and 4546:

• Quotes service stub: http://localhost:4545/quotes
• Time service stub: http://localhost:4546/time

Install a Hystrix Dashboard by executing following docker command:

docker run -d -p 8081:9002 --name hystrix-dashboard mlabouardy/hystrix-dashboard:latest

Find more information about the Hystrix Dashboard in the References section.

Verify that the hystrix dashboard works by accessing http://localhost:8081/hystrix

The prototype application is located in the Prototype directory. The application uses Gradle as the build and dependency management tool and can be imported into eclipse in two ways:

• Import gradle project
• Import existing project

Using the eclipse gradle-plugin to import the project, gradle will automatically take care of dependencies and you will be able to run gradle commands directly from eclipse.

Open eclipse:

• Help -> Eclipse Marketplace
• Search for Buildship Gradle Integration 2.0 and install it

When eclipse has restarted, import the project:

• File -> Import
• Gradle -> Existing Gradle Project
• Browse for the Prototype projects root dir and click Open
• Click Finish

If you don't want to install the gradle plugin in eclipse, the project can be imported in the traditional way. Before importing the project into eclipse, you need to generate the eclipse .project file. This is done using the gradle wrappe - run the following command that will setup an eclipse project:

# change to the project root directory
cd Prototype
./gradlew eclipse

If you are on a windows machine, use the gradlew.bat file instead.

Open eclipse:

• File -> Import
• General -> Existing projects into Workspace
• Browse for the Prototype projects root dir and click Open

You should now have a HystrixPrototype project in the project explorer.

The prototype application can be run from within eclipse or from the command line using gradle.

Running from eclipse requires a Jetty plugin, which can be installed in eclipse using this guide: http://eclipse-jetty.github.io

Once the Jetty plugin is installed do the following in eclipse:

• Right click on the project in the Project Explorer
• Run As -> Run Configuration
• Right click Jetty Webapp and select New
• To tell eclipse where to find the webapp dir, click the Scan button
• Make sure you select port 8080 for your application
• Click Run

The endpoint for accessing the server endpoint is: http://localhost:8080/v1/hello. This of course depends on the port you configured in the run configuration in eclipse.

# change to the project root directory
cd Prototype
./gradlew jettyRun

This uses the project files and deploys them to Jetty. Running this way will make the endpoint available here: http://localhost:8080/HystrixPrototype/v1/hello

Notice that running from command line makes the prototype available on a different URL than Eclipse!

The application should now be running via eclipse or command line. The application should be available through one of the following URLs depending on how you started it:

• Eclipse: http://localhost:8080/v1/hello
• Command Line: http://localhost:8080/HystrixPrototype/v1/hello

Try accessing it through Postman or curl:

curl http://localhost:8080/v1/hello
# or
curl http://localhost:8080/HystrixPrototype/v1/hello

The output should look similar to this:

Quites command secceded : true
Time command succeded   : true
Time  : Server time is 15:54:13
Quote : Program testing can be used to show the presence of bugs, but never to show their absence! - Edsger Dijkstra

• Quites command secceded shows true if the Hystrix command requesting the Quites service stub succeeded and false if the request failed and the command returned the fallback message.
• Time command succeded acts like above but for the Time service stub.
• Time displays the message being returned by the Time service stub or the fallback.
• Quote acts like above but for the Quites service stub.

Now that the Hystrix Dashboard and Mountebank stubs are in place and you have the prototype running on a Jetty server, it's time to add some load on the application.

Access the hystrix dashboard http://localhost:8081/hystrix and add the hystrix stream URL depending on how you started the prototype.

• Eclipse: http://<ip_address_of_host_machine>:8080/hystrix.stream
• Command Line: http://<ip_address_of_host_machine>:8080/HystrixPrototype/hystrix.stream

Notice that because the dashboard is running in a docker container and the prototype application sending the data stream is running on the host machine, you must use the IP address of the host machine when adding the stream URL on the dashboard page.

Click the Monitor Stream button. Try sending some requests to the prototype URL and you should see two circuits (Command1 and Command2) both receiving requests and both in Closed state.

If you see the message Unable to connect to Command Metric Stream. try sending a few requests to the prototype application (/v1/hello) as it does not start the stream before the first request has been received. If it is still not working, your application might not be started or you added the wrong hystrix.stream URL.

To put load on the prototype use Apache Benchmark or JMeter.

The project contains two configurations located in the JMeter dir. Which one you should use depends on how you started the prototype.

• Eclipse: eclipse_started.jmx (use this if you startet the prototype from eclipse)
• Command Line: cmdline_started.jmx (use this if you startet the prototype from command line)

Both configs are configured with 10 Threads and 2000 Loop Count per thread resulting in 20000 requests. The only difference is the HTTP request path they use (http://localhost:8080/v1/hello or http://localhost:8080/HystrixPrototype/v1/hello).

• Open JMeter
• File -> Open (browse to the desired .jmx file)
• Click the green arrow to Start the load test

To use Apache Benchmark run one of the following commands:

# Run this command of the prototype was started using eclipse
ab -n 20000 -c 10 http://localhost:8080/v1/hello
# or
# Run this command of the prototype was started from command line
ab -n 20000 -c 10 http://localhost:8080/HystrixPrototype/v1/hello

This will start the test using 10 Threads and 2000 Loop Count.

Run the test using one of the methods described above while you watch Hystrix Dashboard. You should be able to see the requests being porcessed by the Hystrix Commands and that the circuit remain closed at all times.

Now that everything is set up and we are able to put load on the prototype, it's time to simulate slow response from the DOCs.

To simulate slow response from the two DOC send a PUT using Postman or CURL. This will configure the DOC stubs to with a latency of 1500 milliseconds.

Send a PUT request to Mountebank by selecting [Stub] Both services slow response and click Send.

cd mountebank_config/curl
curl -X PUT -d @both_services_slow_response.json http://localhost:2525/imposters

Now rerun the test from before and take a look at the Dashboard. This time you will see the percentage rising and the circuits opening. You will also notice that you still receive 200 OK responses in your test. This means that Hystrix has detected the slow response, opened its circuits and returning the fallback response insteam.

There are additional stub configurations which will simulate slow response from one of the two DOCs or from both of them. Use Postman or CURL to play around with Hystrix and see what happens if you change configuration while testing.

• The QuotesCommand.java and TimeCommand.java classes inherits form HystrixCommand and are responsible for sending the HTTP requests to the DOC (Mountebank stubs) and for performing the fallback in case the HTTP request fails. They also contain configurations for the circuit breakers and URL to the DOC.

• PoolingConnectionManager.java contains thread pool settings for the HTTP client library used in the HystrixCommand classes.

• CloseableHttpClientFactory.java contains timeout values for the HTTP client used for accessing the DOC in the HystrixCommand classes.

• The web.xml is located in the webapp directory and contains the servlets being loaded by the web server and on which URI path the servlets should handle requests from. The Hystrix matrics stream and prototype application servlets are defined in this file.

• Eclipse https://eclipse.org/
• Eclipse Jetty Integration http://eclipse-jetty.github.io/
• Hystrix Wiki https://github.com/Netflix/Hystrix/wiki
• Hystrix Dashboard https://github.com/Netflix/Hystrix/wiki/Dashboard
• Docker https://docs.docker.com/
• Mountebank http://www.mbtest.org/
  • Getting started http://www.mbtest.org/docs/gettingStarted
• Mountebank Docker Image https://hub.docker.com/r/jkris/mountebank/
• JMeter http://jmeter.apache.org/
• Apache Benchmark https://httpd.apache.org/docs/2.4/programs/ab.html
• Postman https://www.getpostman.com/
• CURL https://curl.haxx.se/docs/manpage.html
• Gradle https://gradle.org/
• Gradle plugin for Eclipse http://www.vogella.com/tutorials/EclipseGradle/article.html