Just a crazy idea. It could be nice to have a simple CLI in this package, which would be able to:

• dredd-template init - Initialize a new hooks project (interactively prompting for information)
• dredd-template update - Update existing hooks project from the upstream contents of dredd-hooks-template repository
• dredd-template test - Optional, convenience command wrapping executing of the tests

Obviously, the biggest challenge would be the update command. However, the update command would be the reason for existence of this tool in the first place. Currently, getting updates from upstream is quite painful and manual process. If we have more hook repositories, it can be a real bottle neck.

The update command would probably try to use git under the hood to evaluate the changes. It would take care of language placeholders and would be able to keep language-specific code snippets in place. It would be up to the maintainer to fix them (or add new ones) if the tests do not pass after update. The tool wouldn't try to be super-intelligent, but it would rather try to automate as much of the manual, clearly automatable work as possible.

Use @announce instead of @debug as @debug is possibly buffering or otherwise tempering with the output and tests even though valid may fail.

In #42 there's a new approach to how the template should be used. The helper CLI takes care of copying the feature files and replaces Dredd path in the feature files to its actual path in local node_modules.

This could be either made completely explicit by putting ./node_modules/.bin/dredd into the feature files directly, or perhaps the steps.js could take care of it somehow. In such case the CLI wouldn't need to modify the files during copy in any way.

Related to #3

I fixed this bug a while back with the php hooks (Go is probably an offender at the moment). Nothing in the test suite validates that a hook server's implementation will work with dredd's --hooks-worker-handler-port. Making this as a reminder to myself to make a future PR to add this test. Do we have any idea if other hook servers respect this? I am assuming ruby and python are good since you guys developed them. That would leave the Go and Perl ones potentially if thats the case.

The #42 introduces an 'upgrade' command. We should somehow verify it works in the smoke test. It is necessary to publish the package for the first time for the 'upgrade' command to work, and perhaps also Semantic Release should be employed here as well before we dig into this.

This template was updated with some minor changes recently. We should update the local copies in each hooks repo to contain those changes.

• Ruby
• Python - apiaryio/dredd-hooks-python#28
• PHP
• Go
• Perl

Related to: #9, #10, #12, #13, #14

Running bundle exec cucumber outputs

$ bundle exec cucumber
Feature: Execution order

Killed: 9

on my computer (Mac). I debugged that it's caused by following lines in features/support/env.rb:

...
Before do
  puts "Killing server..."
  system "for i in `ps axu | grep 'server.rb' | grep ruby | awk '{print $2}'`; do kill -9 $i; done > /dev/null 2>&1"
  puts "Killing handler..."
  system "for i in `ps axu | grep 'dredd-hooks' | grep ruby | awk '{print $2}'`; do kill -9 $i; done > /dev/null 2>&1"
  ...

If I comment out the two "killing" lines, I can run Cucumber without problems, but then it seems to hang in the Feature: TCP server and messages / Scenario: Message exchange for event beforeEach / And I send a newline character as a message delimiter to the socket part:

Feature: TCP server and messages

  Scenario: TCP server                                       # features/tcp_server.feature:3
      Killing server...
      Killing handler...
    When I run `dredd-hooks-ruby` interactively              # vendor/bundle/ruby/2.0.0/gems/aruba-0.6.2/lib/aruba/cucumber.rb:113
    And I wait for output to contain "Starting"              # vendor/bundle/ruby/2.0.0/gems/aruba-0.6.2/lib/aruba/cucumber.rb:131
    Then It should start listening on localhost port "61321" # features/step_definitions/dredd_steps.rb:13

  Scenario: Message exchange for event beforeEach                       # features/tcp_server.feature:8
      Killing server...
      Killing handler...
    Given I run `dredd-hooks-ruby` interactively                        # vendor/bundle/ruby/2.0.0/gems/aruba-0.6.2/lib/aruba/cucumber.rb:113
    When I wait for output to contain "Starting"                        # vendor/bundle/ruby/2.0.0/gems/aruba-0.6.2/lib/aruba/cucumber.rb:131
    And I connect to the server                                         # features/step_definitions/dredd_steps.rb:18
    And I send a JSON message to the socket:                            # features/step_definitions/dredd_steps.rb:22
      """
      {"event": "beforeEach", "uuid": "1234-abcd", "data": {"key":"value"}}
      """
    And I send a newline character as a message delimiter to the socket # features/step_definitions/dredd_steps.rb:27

Nothing gets printed since that moment for very long time and I have to eventually interrupt Cucumber with Ctrl+C.

Taking this out of https://github.com/apiaryio/dredd-hooks-template/pull/65/files#r290878471 where @kylef proposes:

Maybe in this case if a process doesn't gracefully exit from SIGTERM that should be a test failure too (instead of having to SIGKILL it).

This is a proposal to add a feature, which ensures the hooks handlers can react to SIGTERM correctly (SIGINT could be tested too).

Downsides of introducing this is that it would be slightly harder to write a hooks handler (the author already needs to know TCP, now they'd need to know what signals are and how they work) and that SIGTERM (or any alternative to it) doesn't exist on Windows, which would make it complicated for those either aiming for Windows support (#21) or developing the hooks handler on Windows (no way to test locally that it works).

The whole hooks template and especially the Ruby hooks are in a somewhat hard-to-resolve state: apiaryio/dredd#917 I've written 5 lines of Ruby in my whole life and putting the template and the Ruby hooks into shape feels overwhelming to me. I'd like to port dredd-hooks-template to Node.js instead of using Ruby.

Why?

• Because I'm able to maintain it in that case and to give it a full support, as well as anyone else in the company or new hires or Dredd contributors, as Dredd itself is in JavaScript.

• Second, if the test suite itself is in Ruby, it brings Ruby dependencies to all of the other projects. That's an issue for Ruby hooks, where it constraints Ruby hooks' own dependencies, and it also feels strange (read as "makes CI setup complicated") in projects like Go, Rust, Python, etc. Unlike Ruby, thanks to web development, today Node.js is a common "buddy" of other languages, living alongside other-language projects, and people are used to work with it. Even hardcore backend Python people know how to npm install things.

  If you don't know what I mean by the "buddy" thing, see e.g. this:

  

• The Node.js hooks are a native part of Dredd, so there can never be a collision like Ruby hooks / Ruby hooks template dependencies. Node.js is mutually exclusive to additional languages supported by Dredd.

Currently there's dredd-hooks-{{mylanguage}} in the feature files. It could be changed to something better conveying the fact that the whole string is a placeholder (not just a part of it) and that it represents a path to the hooks handler executable under test. Related to #4

...but this thing is not documented anywhere.
See this code

Perhaps it's unnecessary to be so explicit? What about something like

And a file "server.js" with a server responding on "/message" with "Hello World!" 

The steps can implement it as a black box then.

...which most people don't have/do not want to have.

Make install it locally and use that instance.

Once apiaryio/dredd#680 is implemented, the template should not force the hook handlers to implement their own globbing anymore.

The process of making changes to the test suite and propagating made changes to all hook handler implementations was designed in discussions in apiaryio/dredd-hooks-ruby#26 (comment) and mentioned in #17. It should be codified in the README of this repository.

See apiaryio/dredd-hooks-ruby#26 for details.

...because most of the time it will be a locally bin script

Starting with version v3.4.1, Dredd supports Windows. Of course, it would be great if this test suite supports Windows as well and if the hook handlers at least give a clear information on whether they work on Windows or not (AppVeyor can be easily used for testing). How do we want to coordinate this? Any ideas around this topic?

@w-vi @ddelnano @gonzalo-bulnes @ungrim97 @snikch @netmilk