it would be handy if I could run the tests without also running all the 'feedback' cases, my current hacky solution is:

> rm test_cases/feedback*
> node test -e stage
> git checkout test_cases/feedback*

maybe we could add an 'exclude' flag to the cli?

Persist test results on server and email to pelias team.

Tests are very New-Yorky. Not that it's a bad thing to have adequate coverage of the greatest city in the world, but we should expand the test set to be more broadly representative and globally inclusive.

One of our test cases needs a little help, and all that's required is knowledge of some place on Earth that uses postal codes. We're hoping it can be someone from our community!

🌍 What are we testing?

Much of the data we import into Pelias has postal code information, but not all of it. Unfortunately this presents us with a problem: users can perform searches for an address with a postalcode, and we might know about that address, but we might not know its postcode.

This happens fairly frequently right now, and we don't expect to have perfect data everywhere, well...ever. Fortunately, Pelias should be able to find the place a user is looking for, even if they type in the postalcode but its not in the data. To ensure this works, we have a test case for it. But right now it's broken. :(

📈 What changed?

In short, open data got better! A new data source was added for the part of Florida that includes the current test, and it adds postalcodes! Overall this is great, but this specific test requires that we don't have the postalcode for an address, or it doesn't work. We need a new test case.

💌 How to help out

We need people that can find an address in our data that doesn't have a postal code associated with it, and also happen to know what the postal code for that address should actually be. It doesn't have to be in the USA (there are postalcodes all over the world!).

The fastest way to find areas you might know about that have good data coverage might be to look at the OpenAddresses dot map. It's at the bottom of the page and shows a dot for every single address OpenAddresses knows about. We import all OpenAddresses data so it's also a good indicator of our coverage.

After that, try running some queries using our compare app. Here's a link to the address we're currently using. Try typing in some addresses you know about (without the postalcode).

If Pelias doesn't know about the address at all, none of the results will match what you typed in. But if we do know about it, a couple of the results (hopefully at the top, but maybe not) will match. Read through each one, and see if it has a postalcode property with the correct postalcode. If any of them do, then it's the same problem of open data being too good for this test. 🎉. So try another one and hopefully you can find an address that doesn't have a postalcode.

One important sidenote: please don't use any other geocoders that use proprietary data to find a good test case. There's a famous one who's name starts with G and ends with oogle that you may have heard about. Using proprietary data to make open data better is against the terms of service of those tools. It's also not really in the spirit of open data.

💻 What the updated test will look like?

Our tests are written in JSON, so updating them doesn't require writing any code. In fact for this test, you won't need to change the structure of the test at all, just update some of the values.

The test case currently looks like this:

 {
      "id": 2,
      "status": "fail",
      "user": "Julian",
      "type": "dev",
      "notes": "Desired document has no zipcode, but should be 33801. If we ever get better data, find a new test case",
      "in": {
        "text": "440 S Combee Rd, Lakeland, FL 33801"
      },
      "expected": {
        "properties": [
          {
            "name": "440 South Combee Road",
            "country_a": "USA",
            "country": "United States",
            "region": "Florida",
            "region_a": "FL",
            "county": "Polk County",
            "locality": "Crystal Lake",
            "housenumber": "440",
            "street": "South Combee Road"
          }
        ]
      }

Once you've found an address that works, update each of the fields so that it reflects the new address. Pay special attention to the text field (the actual query the test will run), the notes (which is just a helpful sentence for anyone reading the test, and of course the properties section which lists a bunch of fields for the actual address.

Keep the same format, with the postalcode in the query text but not in the properties section. IF you have to remove a field like region or region_a because the address you found doesn't have it, that's ok.

We're happy to help verify that a particular address will work for this test, but we want to make sure someone outside our team submits the actual PR. We get to do enough of them and we'd rather someone newer to the open source community gets the experience! 🏁

as per pelias/api#82

Thanks to the great work of @vesameskanen some time ago, we have the ability to write tests that require the results of a query to be within a certain distance of a given point!

What do we want?

More tests that check the lat/lon of results! 🌐

When do we want it?

Whenever it makes sense. At this point, we don't have any hard and fast rules for when a test should look at the lat/lon of results. However there are definitely cases where they aren't helpful. Many of our tests are specifically for labels, for example, and adding lat/lon requirements might be overspecification.

A great case is whenever we have had a bug where a result who's name, label, etc look correct but the result is not in the right place. This test for San Francisco's centroid being on land, not in the ocean is a great example.

Some ideas

• We have tests for places like "Union Square", which are all over the place. We currently use the city name to differentiate, but coordinates could help as well.
• Some of our tests deal with restaurant chains like McDonalds, of which there are many all over the place. A perfect case where lat/lon requirements might be helpful.

Test Syntax

As a reminder here's the syntax of a test with a lat/lon:

  {
      "id": 1,
      "status": "fail",
      "user": "Julian",
      "in": {
        "text": "san francisco",
        "boundary.country": "USA"
      },
      "expected": {
        "distanceThresh": 1000,
        "properties": [
          {
            "region": "California",
            "name": "San Francisco"
          }
        ],
        "coordinates": [
          [ -122.431272, 37.778008 ]
        ]
      }
    },

The coordinates section in the expected section is an array of coordinates (which themselves are arrays). The coordinates are in [longitude, latitude] format with each point represented as a number (no north/south/east/west or degrees/minutes/seconds needed). The distanceThresh value is the maximum distance allowed between the expected point and the result, in meters. distanceThresh can also go at the root level of the test case at the top of the file, and then it will apply to all tests that don't override it.

when an endpoint other than /search is used there is no indication of that in the cli reporter which makes debugging difficult and error-prone.

eg. the following result was for "suggest/coarse"

  ✔ [4] "{"lat":40.744243,"lon":-73.990342,"input":"DiDi dump","details":false}"
  ✘ regression [5] "{"lat":40.744243,"lon":-73.990342,"input":"new york","details":false}": No matching result found.
  ✔ [6] "{"lat":40.744243,"lon":-73.990342,"input":"30 West 26th Street","details":true}"

There are some dataset available via creative commons such as http://dev.maxmind.com/geoip/geoip2/geolite2/ which can be used to generate testcases which cover every city in the world.

I marked a test as 'status:fail' and now it's showing as an 'improvement'?

  ✔ improvement [2] "{"input":"bronx","layers":"admin2"}"

just wondering why errors are yellow and regressions are red?

I would expect errors to be red because they are serious problems, the minor regressions such as changes in ordering of results should be yellow.

would anyone object to me changing red/yellow around?

• /autocomplete + focus.point + text
• /search + focus.viewport + text
• /reverse + point
• /reverse + point away fromNYC
• /reverse + point away fromNYC
• test for privacy headers: maybe check that there is a private flag sent back as part of the response?

This package was initialized without jshint; add it, and make sure that everything passes its tests.

Right now it appears our acceptance test library is wholly home-grown, and while it works nice I wonder if we might want to switch to a widely used testing library. Mocha comes to mind first for me, but I do not have any strong preference.

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

Your Greenkeeper Bot 🌴

Each execution of the test suite must store results along with API build id.
We then need a tool for keeping track of what has changed from one build to the next in each environment (dev/stage/prod). We need a clear indication of progress (working tests are incrementing) and possible regression (things that used to work now don't).

Each test case should be flagged with one of the following types:

• working : actual == expected
• pending : actual != expected, pending investigation / fix
• placeholder : data is not in our data sources, or unclear if it should actually work (keep around for documentation purposes)

And here's what we need to know:

• which tests have been added to the suite since previous execution
• which tests used to fail and are now passing --- YAY
• which tests used to pass and are now failing --- ALERT about this
• which tests were changed from pending to working

Typical work flow:

1. Begin working on an issue by selecting existing pending test cases and making them as working
2. Write code until all working tests are passing in your local environment
   • run tests on dev and validate all working tests are passing
   • if any pending tests show up as passing mark them as working
3. Promote code to staging
   -run tests and validate all working tests are passing
   • if any pending tests show up as passing mark them as working
   • do NOT merge to prod until all working tests are passing
4. Promote code to prod
   • run tests and validate all working tests are passing
   • if any pending tests show up as passing, investigate and verify again on staging, this shouldn't ever happen (staging==prod)
   • do NOT merge to prod until all working tests are passing

as per https://travis-ci.org/pelias/acceptance-tests/builds/69300265

Create distance calculator for acceptance tests that can be used to validate that a result is within x km of a point

First result is in Brazil, but it should be 1. instead of 2.

Test for landmarks such as Eiffel Tower, Golden Gate Bridge, Union Square

Related to #83
Related to #80
Related to #79

libpostal integration is going to completely up end how we search/geocode so a thorough examination of acceptance-tests results should be performed.

Here's the plan:

• Add region tag to tests/suites that will indicate which regions need to be loaded in order for those tests to pass as expected.

{
      "in": {
        "input": "search text"
      },
      "user": "sevko",
      "id": 2,
      "regions": [ "london", "new york", "berlin" ]
      ...
}

• Allow user to run tests with an array of loaded regions and all tests where region requirements are met should be executed.
• As before allow running the entire set of tests when planet is loaded and no regions are specified.
• Move tests expected to fail into separate pull requests, where each one documents a bug or shortcoming of our system.
• Move pending tests, currently in light blue, to a separate PR as well. Revisit them to fill in result details and turn them into legitimate tests.
• Once all non-binary state tests have been moved out, update the framework to support pass/fail/warning states. Some warnings could be:
  • found but not in the expected position
  • found with N (N=1,?) field difference
  • ??? other ideas
• Create new branch, production, and make the .travis.yaml file run all tests against production instance
• Change .travis.yaml in master branch to create a local vagrant instance with configured regions and run acceptance tests applicable to that region only.

Testing the overall functionality of a geocoder is not like testing many other software projects.
While a geocoder's interface is as simple as a single function in a programming language (in our
case: send a string as input, and get GeoJSON back), the underlying complexity is a lot different.

Worse, there is lots of data involved, and at an acceptance-test level it's impossible to separate
correctness of the data and correctness of the code. Both have to work together to produce a correct
result. With a geocoder using open data we don't even have full control over our data. It can change
at any time without action on our part and changes we suggest to make things better may not stick.

So it's very clear that unit-test style tests will not be completely sufficient for testing a
geocoder. We will need something else as well.

What do we need?

What problems have to be solved to accurately and completely test a geocoder? At the very least, a
solution has to have the following properties:

Scales well to thousands of tests or more

We are literally testing the entire planet, so there are going to be a lot of tests.

Accepts that some tests will always be failing, and still provides value

A build with 10,000 tests that fails if any one tests fails will never pass, and simply getting
one bit (pass or fail) of information back from running tests will not tell us enough.

Gracefully degrades when data or code changes

Many teams hate writing tests because they end up spending too much time updating tests to pass when
their newly written or refactored code already works. We have that problem, and the problem of
previously passing tests spontaneously failing because of changing data.

Handles tests of low quality

We aren't going to write 10,000 tests ourselves. The majority of them will hopefully come from
non-technical users not involved in our project at all. Some of these tests will be bad but we won't
have time to improve them all.

What we should build?

The acceptance tests we should keep around

The first part is close to what we already have: a very closely watched set of acceptance tests. We
had about 300 of these tests until recently, when many of them were pruned. I think we should get
this down to about 50.

We should carefully craft and maintain all of these tests. Each one should very specifically test one
particular feature or bit of functionality. If two tests do almost the same thing we should get rid
of one of them. When data changes we should be responsible for manually updating the tests. The idea
is that we have so few that this isn't a huge issue.

Peter's Guide to Writing Solid Tests is certainly the starting point of our high standards for these tests.

Fuzzy testing suite

The new part we need to build is, for lack of a better name, a fuzzy testing suite. This will have
more in common with the tests for, say, a statistical machine translation test suite than a
unit-test suite.

This fuzzy testing suite will have the thousands and thousands of user-generated tests, and at any
given time a bunch of them will be failing. We will almost never update the expectations of these
tests, and as our most strongly cherished rule, we will NEVER change the input string for these
tests. If we think of a new input string that's even slightly different than one we already have,
we'll just add another test.

How to make the tests valuable

After collecting all of these tests, we'll have to do a bit of work to make tooling to even get any
value out of them.

First, instead of the main output of a test suite being simply pass/fail, the output should be given
as a score. The score for the test suite will simply be the sum of the scores from all the tests,
and for each test it will be based on how many of the expectations of the test passed. This gives
our tests a little bit more granularity. The simplest implementation would give one point for each
expectation, so a test that expected certain values for two admin names and one name field could get
three points if all three matched.

The test suite score could be reported as a total score or a percentage of the maximum possible
sore, I'm not sure which yet. Percentages seem popular for this sort of thing.

The other thing we'll want is some sort of tool set for easily viewing differences in test failures
between two builds. This would be used to see how certain code changes affect our success rate in
the tests, and occasionally could be used to update tests if we feel the need.

It sounds like this should be a diff-like utility, and if we can have our test suite output simply
be readable by diff itself, or a thin layer over diff, that saves us time building stuff and
learning a new tool.

Unsolved problems

There's a few other questions and problems we have that this plan doesn't directly solve. I'm open
to suggestions, pleas feel free to offer any thoughts!

Builds break sometimes

Sometimes something is wrong with our infrastructure and a bunch of tests are failing for a reason
unrelated to the data or code. Ideally we'd have a way to filter out these situations so they don't
cause us too much wasted effort figuring out why all our tests are broken.

Reporting pass/fail to travis, etc

While our test suite will not work in terms of a simple test/fail, lots of tools we might want to
use like bash scripts and TravisCI still do. How do we map our test output to a pass/fail when we
need to? One idea I had would be to count any test suite as a pass as long as none of the tests that
passed in the last build fail (no new regressions), but this requires that our testing tools keep
track of previous runs, which would be a bit of work.

Insight from tests changing over time

@riordan made a great, brief comment to me today about how older feedback tests and test results
could be used as snapshots of the progress of our geocoder. We could compare test results for the
same tests over time pretty easily.

Another related question is what to do as our test suite ages: a 2 year old test to find a building
that was demolished last year probably isn't useful. Maybe we assign more weight to newer tests, or
something similar.

would it be possible to test for the nonexistence of a document for a specific query?

in some cases it is handy to ensure that a record which was erroneously returned in the past is not returned again in the future.

some examples:

• imagine the case where a specific record is removed due to profanity, could we hit the /doc endpoint to ensure that it is not found?
• if a record was added to the wrong layer, could we test that it is now indexed on the correct layer & also that it no longer appears in the incorrect layer?
• if a record keeps appearing in results when it shouldn't, can we ensure that after the dev work is complete that this issue will not regress?

we'll clearly never be able to source a list of all the things that shouldn't be returned for each query but in some cases copying the 1-2 test cases from the bug reports to the regression tests will ensure that the reported error doesn't come back.

As specified in the README, the way to configure API keys for servers to run acceptance tests is to put them in a mapzen section of pelias.json. Obviously, this is no longer the most relevant name.

{
  "mapzen": {
    "api_key": {
      "search.mapzen.com": "search-XXXXXX"
    }
  },
  "acceptance-tests": {
    "endpoints": {
      "prod": "http://search.mapzen.com/v1/"
    }
  }
}

We should add an alternate key name (supporting the old key for backwards compatibility), that is more relevant going forward.

Connects pelias/pelias#703

Add tests for

• Mexico City
• Pärnu, Estonia

Both have had encoding problems in the past for geonames results. Have been fixed with WOF import.

Diacriticals should be flattened to ease comparisons where applicable

diff --git a/test/test.js b/test/test.js
index 740c064..0d6879b 100644
--- a/test/test.js
+++ b/test/test.js
@@ -33,6 +33,11 @@ tape( 'equalProperties() works.', function ( test ){
       expected: {a: 1},
       actual: {a: 1},
       match: true
+    },
+    {
+      expected: {a: 1},
+      actual: {a: 1, b: 2},
+      match: true
     }
   ];

  total:     19
  passing:   19
  duration:  110ms

I would suggest using an npm module for this rather than rolling your own eg:
https://www.npmjs.com/package/deep-equal

### search
  ✘ regression [2] "{"text":"brooklyn, ny"}": score 3 out of 4
  diff:
    label
      expected: Brooklyn, Kings County, NY, USA
      actual:   Brooklyn, New York, NY, USA
  ✘ regression [7] "{"text":"billerica"}": score 1 out of 2
  diff:
    label
      expected: Billerica, Middlesex County, MA, USA
      actual:   Billerica, River Pines, MA, USA
  ✘ regression [8] "{"text":"billerica, ma"}": score 1 out of 2
  diff:
    label
      expected: Billerica, Middlesex County, MA, USA
      actual:   Billerica, River Pines, MA, USA
  ✘ regression [10] "{"text":"chelsea, new york"}": score 1 out of 2
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [11] "{"text":"soho, new york"}": score 2 out of 4
  diff:
    label
      expected: Soho, New York County, NY, USA
      actual:   Soho, Brooklyn, NY, USA
    label
      expected: SoHo, Manhattan, NY, USA
      actual:   Soho, Brooklyn, NY, USA
  ✘ regression [1425586777012:3] "{"text":"paris"}": score 1 out of 2
  diff:
    label
      expected: Paris, France
      actual:   Paris, Paris-4E-Arrondissement, France
  ✘ regression [1425586777012:6] "{"text":"london"}": score 3 out of 6
  diff:
    label
      expected: London, Greater London, United Kingdom
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, Middlesex, Canada
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, United Kingdom
      actual:   London, City of Westminster, United Kingdom

### San Francisco
  ✘ regression [5] "{"text":"san francisco"}": score 2 out of 3
  diff:
    region
      expected: California
      actual:   Atlántida

### quattroshapes popularity
  ✘ regression [1] "{"layers":"coarse","text":"chelsea"}": score 3 out of 4
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [3] "{"layers":"coarse","text":"ridgewood"}": score 3 out of 4
  diff:
    label
      expected: Ridgewood, Queens County, NY, USA
      actual:   Ridgewood, Bergen County, NJ, USA
  ✘ regression [1-1] "{"layers":"coarse","text":"chelsea, ny"}": score 1 out of 2
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [3-1] "{"layers":"coarse","text":"ridgewood, ny"}": score 1 out of 2
  diff:
    label
      expected: Ridgewood, Queens County, NY, USA
      actual:   Ridgewood, Bergen County, NJ, USA

### Brooklyn
  ✘ regression [3] "{"text":"brooklyn","layers":"locality,localadmin","boundary.country":"USA"}": score 2 out of 3
  diff:
    region
      expected: New York
      actual:   Iowa
  ✘ regression [7] "{"text":"brooklyn","layers":"locality,localadmin"}": score 2 out of 3
  diff:
    region
      expected: New York
      actual:   Victoria

### autocomplete admin areas
  ✘ regression [2] "{"text":"new york"}": score 7 out of 8
  diff:
    label
      expected: New York, USA
      actual:   New York, Brooklyn, NY, USA
  ✘ regression [7] "{"text":"wales"}": score 1 out of 2
  diff:
    label
      expected: Wales, United Kingdom
      actual:   Powys, United Kingdom
  ✘ regression [11] "{"text":"madrid"}": score 4 out of 5
  diff:
    layer
      expected: locality
      actual:   venue

### admin lookup
  ✘ regression [17] "{"text":"ringland","layers":"neighbourhood"}": score 5 out of 6
  diff:
    county
      expected: Newport City
      actual:
  ✘ regression [18] "{"text":"california","layers":"region"}": score 8 out of 9
  diff:
    source
      expected: whosonfirst
      actual:   geonames
  ✘ regression [5] "{"point.lat":51.515049,"point.lon":-0.093229,"layers":"coarse"}": score 2 out of 3
  diff:
    region
      expected: London
      actual:
  ✘ regression [8] "{"text":"crown heights","layers":"coarse"}": score 6 out of 8
  diff:
    label
      expected: Crown Heights, Kings County, NY, USA
      actual:   Crown Heights, Poughkeepsie, NY, USA
    county
      expected: Kings County
      actual:   Dutchess County

### address parsing
  ✘ regression [3] "{"text":"450 w 37th st, new york, ny 11232"}": score 13 out of 14
  diff:
    unexpected result found
  ✘ regression [4] "{"text":"starbucks 10010"}": score 10 out of 14
  diff:
    name
      expected: Starbucks
      actual:   100 East 22nd Street
    housenumber
      expected: 304
      actual:   100
    street
      expected: Park Avenue South
      actual:   East 22nd Street
    label
      expected: Starbucks, Manhattan, NY, USA
      actual:   100 East 22nd Street, Manhattan, NY, USA

### address matching
  ✘ regression [2] "{"text":"507 hackney rd"}": score 6 out of 7
  diff:
    name
      expected: Hackney Cycles
      actual:   507 Hackney Road
  ✘ regression [3] "{"text":"49 Kay Street"}": score 2 out of 3
  diff:
    unexpected result found

Aggregate test results
Pass: 129
Fail: 24
Placeholders: 0
Regressions: 25
Took 81119ms
Test success rate 86%

The tests are hitting
curl http://192.168.29.76:9200//search?text=Tajura%27%20wa%20an%20Nawahi%20al%20Arba%2C%20LY&sources=wof

when my index name is pelias, how can I change that?

I get the error message 'No result found' when a property does not match the expected.

address type
  ✔ [1] "{"input":"102 Fleet Street","lat":51.53177,"lon":-0.06672}"
  ✔ [2] "{"input":"Shepherd and Flock","lat":51.219671,"lon":-0.779389}"
  ✘ [3] "{"input":"22 Moor Park Lane","lat":51.219671,"lon":-0.779389}": No result found.

It might be nicer to have a more descriptive error in this case such as 'Property mismatch'

This issue consists of two closely-related parts. Firstly, the test output is excessively noisy:

> pelias-acceptance-tests@0.0.0 test /home/sevko/src/mapzen/acceptance-tests
> mocha -R spec tests



  GET /search

    1) [id:0] query: { input: 'brooklyn' } (top 3)

    2) [id:1] query: { input: 'brooklyn, ny' } (top 3)

    3) [id:2] query: { input: 'new york' } (top 3)

    4) [id:3] query: { input: 'philadelphia' } (top 3)

    5) [id:4] query: { input: 'new york, ny' } (top 3)

    6) [id:5] query: { input: '130 dean street brooklyn, ny' } (top 1)

    ✓ [id:6] query: { input: 'billerica' } (top 1) (54ms)

    ✓ [id:7] query: { input: 'billerica, ma' } (top 1) (90ms)

    7) [id:8] query: { input: '15 call street billerica, ma' } (top 1)

    8) [id:9] query: { input: 'union sq' } (top 3)

    9) [id:10] query: { input: 'union square' } (top 1)

    ✓ [id:11] query: { input: 'portland' } (top 1) (59ms)

    ✓ [id:12] query: { input: 'paris' } (top 1) (73ms)

    10) [id:13] query: { input: 'france' } (top 3)

    11) [id:14] query: { input: 'prospect park' } (top 1)

    ✓ [id:15] query: { input: 'london' } (top 1) (57ms)


  5 passing (7s)
  11 failing

  1) GET /search [id:0] query: { input: 'brooklyn' } (top 3):
     Error: timeout of 2000ms exceeded
      at null.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/node_modules/mocha/lib/runnable.js:159:19)
      at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)

  2) GET /search [id:1] query: { input: 'brooklyn, ny' } (top 3):
     Error: Expected result not in top 3
      at Object.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/tests/searchTest.js:81:19)
      at Test.assert (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/lib/test.js:221:6)
      at assert (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/lib/test.js:132:12)
      at /home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/lib/test.js:129:5
      at Test.Request.callback (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/node_modules/superagent/lib/node/index.js:746:30)
      at Test.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/node_modules/superagent/lib/node/index.js:135:10)
      at Test.emit (events.js:95:17)
      at IncomingMessage.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/node_modules/superagent/lib/node/index.js:938:12)
      at IncomingMessage.emit (events.js:117:20)
      at _stream_readable.js:943:16
      at process._tickCallback (node.js:419:13)

  3) GET /search [id:2] query: { input: 'new york' } (top 3):
     Error: Expected result not in top 3
      at Object.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/tests/searchTest.js:81:19)
      at Test.assert (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/lib/test.js:221:6)
      at assert (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/lib/test.js:132:12)
      at /home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/lib/test.js:129:5
      at Test.Request.callback (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/node_modules/superagent/lib/node/index.js:746:30)
      at Test.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/node_modules/superagent/lib/node/index.js:135:10)
      at Test.emit (events.js:95:17)
      at IncomingMessage.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/node_modules/supertest/node_modules/superagent/lib/node/index.js:938:12)
      at IncomingMessage.emit (events.js:117:20)
      at _stream_readable.js:943:16
      at process._tickCallback (node.js:419:13)

  4) GET /search [id:3] query: { input: 'philadelphia' } (top 3):
     Error: timeout of 2000ms exceeded
      at null.<anonymous> (/home/sevko/src/mapzen/acceptance-tests/node_modules/mocha/lib/runnable.js:159:19)
      at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)

/* etc */

The stack traces at the end make it difficult to view the failing tests at the top of the output, and there's some additional text in those (like the time the query took) that might not be worth displaying. I think the traces should definitely go, and while we remove them, we should consider just writing our own output generator here instead of using mocha. Could look something like this:

✘ [id 9] 'brooklyn': timeout of 2000ms exceeded
✘ [id 9] 'new york, ny': not in top 10 results
✘ [id 9] 'new york, ny': not in top 10 results
✓ [id 10] 'union sq': no `out` object specified
✓ [id 10] 'union square'

I'm suggesting that since it'll give us fine-tuned control over the output format in the long run, and would remove auto-injected mocha text.

This issue is a great example of where there are little or no street numbers in our database for a query but we we have the street geometry from openstreetmap, we should return the street itself.

pelias/pelias#93

Add checks for each request roundtrip time to not exceed a predetermined threshold. Be able to specify different thresholds for each environment. Report warning if average latency reaches N% of threshold. Report regression if average latency is >= threshold.

A recent test run failed due to 2x tests marked as "status": "fail" reported as regressions by the reporter.

GET /search
  ✘ [5] "{"input":"new york, new york"}": Result found, but not in top 3. (4)
  ✔ improvement [11] "{"input":"soho, new york"}"
  ✘ [18] "{"input":"newark airport"}": No matching result found.

admin lookup
  ✔ improvement [2] "{"input":"bronx","layers":"admin2"}"

    {
      "id": 5,
      "status": "fail",
      "issue": "https://github.com/pelias/api/issues/149",

    {
      "id": 18,
      "user": "sevko",
      "status": "fail",
      "issue": "https://github.com/pelias/api/issues/151",

ticket to keep track of which cases need to be updated due to the changed made in https://github.com/pelias/wof-pip-service/pull/36

✘ regression [4] "{"text":"starbucks 10010"}": score 12 out of 13
  diff:
    neighbourhood
      expected: Midtown South
      actual:   

After a fresh clone and an npm install a file called .validate.json appears in my working directory.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    .validate.json

Version 2.9.5 of jshint just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As jshint is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Unit tests weren't initially necessary due to the simplicity of the package, but, especially with the implementation of #21 , has grown rather more complex (logic-wise). Add some unit tests.