agourlay / cornichon Goto Github PK

This test fails

"do not trip on duplicate" in {
        val input: JValue =
          ("name" → "bob") ~
            ("age", 50) ~
            ("brother" →
              (("name" → "john") ~ ("age", 40))) ~
              ("friend" →
                (("name" → "john") ~ ("age", 30)))

        val expected = ("name" → "bob") ~ ("age", 50) ~ ("brother" → ("age", 40)) ~ ("friend" → (("name" → "john") ~ ("age", 30)))

        removeFieldsByPath(input, Seq("brother.name")) should be(expected)```

Duplicate nested fields confuse removeFieldsByPath as it is not really exploring the jsonPath but merely relying on content equality.

Can be used to fail faster.

And I create_user
Then assert status_is(201)

The status assertion can be removed by adding the expects arg. to the POST.

def create_user = effectful(
    title = s"create a user",
    effect =
      s ⇒ 
        http.Post(
          url = "/users",
          payload = ???,
          params = Seq.empty,
          headers = Seq.empty,
         expects = "201"
        )(s)
  )

Split the RunnableStep concept into assertStep and effectStep

• effectStep is a function from Session to Session
• assertStep[A] is a function from Session to Assertion[A]

When two scenarios have the same name, the error is not very helpful:

[info] DeferredAbortedSuite:
[info] Exception encountered when attempting to run a suite with class name: org.scalatest.DeferredAbortedSuite *** ABORTED ***
[info]   Exception encountered when attempting to run a suite with class name: org.scalatest.DeferredAbortedSuite (ScalatestIntegration.scala:28)

Provide hooks offering a subscription to Feature's results to enable users to build specific post processing action.

The ignoring param in the DSL should allow to ignore nested key using .

e.g. ignoring = "company", "employee.adress"

Then assert is redundant.

The usage of Then should be enough to infer that an assertion follows.

It should look more natural, something like:

Then X is Y where X is a generic extraction and Y a value.

e.g Then status is 200

Requires chaining of infix notation : obj method arg method arg method arg ...

Make it possible to feed a Scalacheck Generator to a custom mapper.

"user-name" → GenMapper(Gen.alphaStr)

Offer better typed JsonPath using Dynamic Trait

root.Episodes(0).Released instead of "Episodes[0].Released"

Current implem. should be improved...

@agourlay

For instance

And assert response body with transformation is '<film-title>'

The transformation should be explained in terms of JsonPath for more clarity.

In this case a proper CornichonError should be returned to the user and not a technical exception Future Timeout.

exception thrown 'java.util.concurrent.TimeoutException: Futures timed out after [2000 milliseconds]
[info]      at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
[info]      at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
[info]      at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
[info]      at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169)
[info]      at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
[info]      at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)
[info]      at scala.concurrent.Await$.result(package.scala:190)
[info]      at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(HttpService.scala:34)
[info]      at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(HttpService.scala:33)
[info]      at cats.data.Xor.flatMap(Xor.scala:98)
[info]      at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5.apply(HttpService.scala:33)
[info]      at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5.apply(HttpService.scala:32)

Currently Gql JSON can only be used within the body assertion.

It should be possible to use also in the payload of the HTTP effects steps.

beforeEachScenario and afterEachScenario are treated as regular steps, this means if it will simply exit at the first failed step without trying to run the cleanup steps in afterEachScenario.

Use json4s everywhere.

It should be possible to run a single scenario within a feature from the CLI.

Currently the wrapper steps like Eventually or RetryMax causing series of steps being replayed upon errors are displaying all the logs from failed run which makes debugging difficult.

Ideally only the first error, the last error and the number of errors in between should be reported.

Debug steps should not require a specific keyword.

No one wants to see placeholders in steps logs.

save_body_key("addresses[0].id" -> "customer-address-id") does not work, as the JsonPath "addresses[0].id" is not properly evaluated.

(FYI: body.path("addresses[0].id").is("test") does work)

https://github.com/travisbrown/circe

Maybe it has everything necessary to replace the mix of Json4s and SprayJson

It should print as well the inital array on which the transformation was applied.

The current reporting is not really helpful while debugging

Then assert response body 'results' array size is '1' *** FAILED ***
         expected result was:
         'true'
         but actual result is:
         'false'

Fix this asap, or you will have no chance

@agourlay

It is already possible to ignore Features and Scenarios by passing ignore = true to the DSL builder.

It would be nice to do it just with an annotation as well.

class CornichonExamplesSpec extends CornichonFeature {

  lazy val feature =
    @Ignore
     Feature("Cornichon feature example") {

      @Ignore
      Scenario("demonstrate CRUD features") {
      ...
      }

Will test the JSON equality for the two bodies and display a nice diff in the error.

When I get("/superheroes/Batman")
When I get("/superheroes/batman")
Then assert lastBodies(2).ignoring(root.name).areEquals

It could potentially accept 'n' requests.

The usage of specific keyword between Given/And/Then and the step definition is not clear.

Currently cornichon offers I, a, assert.

Those can't be combined and have side effects on the steps title.

The main usage for those keywords is actually to make a clean "infix notation" possible.

A better alternative would be to have as a minimal step definition Given/And/Then directly followed by a step definition.

Linking words could be added in between if desired.

The key must be there but the value can be discarded.

Metaspace is never garbage collected.

JSON is cumbersome to write, it would be interesting to also support the GraphQL query AST as input.

 {
       id: 1
       name: "door"
       items: [
         # pretty broken door
         {state: Open, durability: 0.1465645654675762354763254763343243242}
         null
         {state: Open, durability: 0.5, foo: null}
       ]
}

ref sangria-graphql/sangria@1998e0f#diff-cf4a54c5a5ef49edc6332de99a35377aR34

It is quite noisy, easy to forget and scary for non Scala dev

https://github.com/ktoso/sbt-jmh

With Scalatest 2.2.6 the scenarios seem to be executed in // within a feature.

It breaks CornichonExamplesSpec as the state on the server is globally shared.

Having all the doc inside the readme file is becoming cumbersome.

Investigate sbt-microsite https://github.com/47deg/sbt-microsites

@agourlay, thank's for putting this together! You have asked for feedback, so here we go:

• First, I don't fully understand what additional features chrnichon offers over using ScalaTest (e.g. using a FeatureSpec) in combination with the akka-http-testkit
• Second, using snake_case is not idiomatic Scala, you better switch to camelCase

Parse String input to JSON at compile time using a macro to fail faster.

The input String is likely to contain placeholders so it can not be a simple JSON parse.

exemple of invalid JSON prior placeholder resolution:

{
  "a number" : <number-placeholder>
}

Making sure those examples compile is cumbersome.
Integrate with Tut https://github.com/tpolecat/tut

Each step should log it's execution time

When I Get /superheroes/random ( executed in 233 ms )

The number of spaces in hardcoded in the log statement.

It should be generated depending of the level of nesting.

def log(input: String, depth: Int)

Say we have a endpoint to create a user with an email respecting the constraint that all email must be unique.

Given I Post("/user", "{ "email" : "<random-string>"}

<random-string> is a builtin markup in the resolver machanism.

Now what happens if we write

Concurrently(3, timeout){
  Given I Post("/user", "{ "email" : "<random-string>"}
}

A user could expect different behaviours according to the semantic of the random values.

1 - Random values are identical accross concurrent executions

Used to check that only one call should pass, in our case testing that the the email constraint is applied concurrently.

2 - Random values are different accross concurrent executions

Used to verify that an endpoint behaves correctly under load.

The current implementation uses identical random values accross concurrent executions:

• it is not clear reading the doc that it is the case
• it should also be possible to configure it to have different random values otherwise it can be quite limiting.

The error reporting of the HttpService is not proper.

When the underlying call times out an expection is thrown and bubble up in the Engine.

exception thrown 'java.util.concurrent.TimeoutException: Futures timed out after [4 seconds]
[info]      at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
[info]      at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
[info]      at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
[info]      at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
[info]      at scala.concurrent.Await$.result(package.scala:190)
[info]      at com.github.agourlay.cornichon.http.HttpService$$anonfun$withoutPayload$1$$anonfun$apply$9$$anonfun$apply$10$$anonfun$apply$11$$anonfun$apply$12.apply(HttpService.scala:47)
[info]      at com.github.agourlay.cornichon.http.HttpService$$anonfun$withoutPayload$1$$anonfun$apply$9$$anonfun$apply$10$$anonfun$apply$11$$anonfun$apply$12.apply(HttpService.scala:46)
[info]      at cats.data.Xor.flatMap(Xor.scala:98)

Use proper HttpCornichonError instead with explicit error message.

It could hold previous values and offer an API to retrieve them.

• <name> takes the last element
• <name[0]> takes the head
• <name[1]> takes the second element in the list

New values would be appended to the tail.

Currently it is necessary to provide a full mapping for a custom mapper.
view-key + key-in-session + extract-fct

It creates a unique binding.

What if most of the object in session have the same shape, you would still have to create all bindings.

• product-id -> product -> v \ id
• cart-id -> cart -> v \ id
• order-id -> order -> v \ id

The goal of this issue is to design a new custom mapper that accepts generic/regex like binding.

*-id -> * -> v \ id

Make sure that all error messages contain a formatted JSON output.

For instance status_is shows the JSON inlined

The engine should expose a clear API enabling users to definer their own steps (regular and wrapper).

Offer a RepeatDuring bloc that accepts a duration as an argument and repeat the execution of the inner steps during the specified time.

RepeatDuring(3.seconds) {
  When I get("/superheroes/Batman")
  Then assert status.is(200)
}

Equivalent of a nested filter on JArray.

"remove field in each element of an array" in {
  val p1 = ("name" → "bob") ~ ("age", 50)
  val p2 = ("name" → "jim") ~ ("age", 40)
  val p3 = ("name" → "john") ~ ("age", 30)

  val input = JArray(List(p1, p2, p3))
  val expected = JArray(List(JObject(JField("name", "bob"), JField("name", "jim"), JField("name", "john"))))

  removeFieldsByPath(input, Seq(JsonPath.root.age)) should be(expected)
}

Really usefull to finally ignore the imdbRating of all episodes in the OpenMovieDatabase integration test.

The repeat block does not appear in the execution log.

The current implementation is a bit hacky and just repeats a seq of Steps.

Ideally the Repeat block should be clearly visible in the logs with a proper identation.

When using a json path like this:

Then assert body.path("masterData / current / masterVariant / availability")

the error message is not very helpful:

[info] DeferredAbortedSuite:
[info] Exception encountered when attempting to run a suite with class name: org.scalatest.DeferredAbortedSuite *** ABORTED ***
[info]   com.github.agourlay.cornichon.json.JsonPathParsingError:
[info]   ...
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0

Propose a DSL block that generates a standard CRUD test from:

• url with resource
• creation and created payload.
• update payload
• delete by id
• could be also generated from JsonSchema