shanejansen / touchstone Goto Github PK
View Code? Open in Web Editor NEWTouchstone is a testing framework for your services that focuses on component, end-to-end, and exploratory testing.
Touchstone is a testing framework for your services that focuses on component, end-to-end, and exploratory testing.
Various CI servers such as Jenkins and Team City support test report logs in particular formats that can be parsed and read by the server in order to identify and track tests, allowing e.g. "4 out of 12 done, no failures" type indicators on the builds.
Supporting this for a variety of CI servers probably requires test result reporting to be pluggable.
See also #14 re. potentially needing to refactor output anyway.
MySql mock method does_not_exist
has a misleading error message: "was found 1 time(s) but expected any time(s)"
The Touchstone develop console lacks many common console/shell features such as command history via up and down keys, jumping over words by holding control or alt/option while pressing left or right, home and end or other shortcuts to the beginning and end of a line, holding alt or option while pressing delete/backspace to delete entire words, etc.
It would be great if some Python library for console apps could supply these features for Touchstone without having to reimplement them. Bonus points if tab completion is also possible using such a system.
I have an app that has a queue "all" on which it receives from multiple routing keys, "route.a", "route.b", "route.c". (This helps with ordering of different messages that affect the same object, when the app comes back up after downtime. Arguably the root problem is the app's structure but it's a legacy app we're hoping to first get Touchstone tests for then use those tests when refactoring or replacing. Then again: Rabbit plainly allows different sources to use their own routing keys and one recipient to route them into the same queue.) This gives me Touchstone failures, 'This exchange "my.exchange", routing-key "route.b" combination is not defined. Check your "rabbit.yml" defaults.' (and same for route.c) even though I have this in my.exchange:
- name: all
routingKey: route.a
- name: all
routingKey: route.b
- name: all
routingKey: route.c
The MongoDB mock will not clear the collections created by the service if they are not included in the defaults file.
It should clear any collections that the service created that are left out of the mock specification:
collections
is not definedcollections
is an empty array (really just a case of the below)Some services need supporting images which do no include mocks. Add ability to specify a list of "supporting" Docker images to spin up when Touchstone is started
When running Touchstone Develop with snapshot_databases: true
, after running an individal test, the database is no longer reset when running mocks reset
or running all tests.
I'm not certain whether the reset stops happening or whether the snapshot gets overwritten (which would start getting reset to after the test state).
Rather than 'TS_ANY' string, which could (though it's unlikely) be in a user data string, consider a technique like https://stackoverflow.com/q/23703863 or https://stackoverflow.com/q/1677726
This may or may not impact #22
Usually when a service fails to pass the healthcheck, --log-services
provides useful info. However, sometimes the log file is empty and to see its output I have to catch the container with a Docker attach command in the fleeting window during which it is live. It seems as though Touchstone connects to the container after the initial docker run command to get the logs. I would expect it to direct the container output to a file as part of the initial container run/start command, guaranteeing all output is logged.
I believe the order of effects when starting up the mocks and services with mysql set to snapshot DBs is:
(The same order is used without snapshotting DBs, with similar questions/issues but it's eclipsed by the migrations getting wiped so with migrations one pretty much always wants to snapshot DBs.)
This has a couple limitations/oddities:
In other words, you can have migrations or defaults but effectively it's hard to use both, e.g. to populate default data in tables the service creates.
It would make sense in my opinion to defer running the defaults until after the services start, to the same point when the DB is snapshotted (presumably before the snapshot, although at that point as long as the defaults don't contain nondeterministic values/behavior there is only performance difference between "defaults once then snapshot" versus "snapshot migrations, reapply snapshot in place of wiping DB and reapply defaults after each wipe or snapshot application".)
In the MySQL mock, if you omit a field it can be any value including null
, and if you give it the value None
then the mock checks that it IS NULL
.
Sometimes I need to confirm that the field has a non-null value, but allow that value to be anything. I.e. IS NOT NULL
. This seems like a use case for validation.ANY
; that currently tries to see if the field = 'TS_ANY'
.
See also #23
In touchstone.yml
, services: None
should pass with 0 test executions
The slowest part of Touchstone tests is spinning up everything in Docker. While it's unlikely anything can be done about that for the services themselves, the mocks could likely be made long-running, provided some way to sandbox multiple instances of mock usage in the same container. Instead of spinning them up on demand Touchstone would have a client-server model in which an instance is running/managing mock containers and other instances connect to that.
Implementing this for Rabbit should be pretty easy. Virtual hosts! ${TS_RABBITMQ_VHOST}
= randomized ID
MySQL could probably implement this as multiple databases. Multiple databases are already supported, only, their names are hardcoded. So it looks like this…
mysql.yml
databases:
- name: a
statements:
# any statements, or maybe migrations handle the main DB
- name: b
statements:
# lots of statements
properties file
spring.datasource.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/a
spring.datasource-b.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/b
mysql.yml is the same
However, instead of using those names, a randomized ID is generated and used, and a variable TS_MYSQL_DB_<capitalized_name>
is set to that ID.
properties file
spring.datasource.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/${TS_MYSQL_DB_A}
spring.datasource-b.url=jdbc:mysql://${TS_MYSQL_HOST}:${TS_MYSQL_PORT}/${TS_MYSQL_DB_B}
If Wiremock supports using multiple ports and treating them like virtual hosts, then this is trivial, like Rabbit but without any change required for service configuration.
If not, there are a couple options:
Instead of aiming at http://${TS_HTTP_HOST}:${TS_HTTP_PORT}/
, aim at http://${TS_HTTP_HOST}:${TS_HTTP_PORT}/${TS_HTTP_ROOT}
and a randomized ID (followed by /
) for TS_HTTP_ROOT is used to sandbox all the mocks for a given instance. Plus, most services should be pointed at ${TS_HTTP_URL}
anyway, which would get the root added to it without the service needing to be reconfigured at all!
Create a sandboxing root as above on the Wiremock side; however, for ease of use…
Stand up a simple routing server that sends each port to its respective root without needing to add the root to the URL.
My thinking on MongoDB mocks is that, since it's a database, it should largely parallel MySQL (except where the difference between the two is relevant, if there are any).
Do any services use migrations on a MongoDB? Presumably they don't need them to create collections as collections are schema-less or not strongly typed or however you want to put it, but theoretically a service could set up initial data. (As with #46, it might make sense to allow defaults to be applied by Touchstone after the service rather than before it.)
This scenario seems relatively unlikely given the use cases for Mongo, and therefore low priority; however, I wanted to bring up matching MySQL behavior/options in general.
In touchstone.yml
, a service can be defined with dockerfile
. Support ability to specify docker_image
instead
Sometimes when a container has catastrophically failed, Touchstone goes to stop it as part of network shutdown and spits out an error saying no such container. This isn't very detailed and isn't obvious to all developers that the real issue is the container died when it attempted to start up. (In contrast, a healthcheck never passing spits out a whole stack trace where all we need to know is which container's healthcheck never became healthy.)
Would it be possible to catch the Docker missing container error and provide a message saying which mock or service died or wasn't started, and ensure this does not block shutting down the rest of the network?
(Relatedly I would expect a similarly informative error if the Docker command to build or to start/run the container in the first place errors; it's been a while since I had that happen so I don't recall what it looks like, though I think it may be reproducible by attempting to run while the app's port is already allocated to another app or instance.)
When a database assertion fails and it prints which SQL was not successful, the values are placeholders (as they should be for the whole string escaping logic and such) so only the fields are shown; would it be possible to either substitute the values into the placeholders when printing this output, or else add the list/map of values used?
Touchstone will not start if Docker isn't running. It would be helpful if Touchstone automatically started Docker for you
Exactly what it says on the tin; I've got a payload json comparison where validation.ANY works for strings but for numbers (integers, specifically, not sure if that's a different type) I have to hardcode the number in order to pass.
The code doesn't have any obvious (to me) problems that would cause this…
When looking at this be aware of #23
At work, my local machine is able to make service-to-service REST calls as intended referencing other containers through their name. Unfortunately it isn't working in CI.
It turns out this is an oddity in our machines, not CI. As far as I have been able to figure out, Docker's DNS settings will use --hostname
to resolve the host by name for that container itself, but the proper way to set it for the rest of the network appears to be --network-alias
For some reason hostname works on our local, but whatever we've got on CI appears to be more strict.
Touchstone is using --hostname
; I believe it should be updated to use --network-alias
for maximum compatibility.
A workaround for now is to add --network-alias <your service name>
to docker_options in your touchstone.yml
file.
Rabbit payload assertion failures show the expected and list of actual messages, which is generally clear enough but it wouldn't hurt to show the routing key and exchange that the assertion is checking.
Rabbit message count assertion without the payload print "expected N does not match actual M," which borders on meaningless without further contextual info.
Adding routing key and exchange to both would be very handy.
Added Redis to supported mocks
In 4fe636c#diff-f00ff2c353e1ab98d15ad58bb6a842e49495c52107bc800ddce3783d412b1dcbL49-R59 a bug was corrected around expecting 0 calls to a REST API. The fix was to change not n
, which is also True
for 0
rather than only None
, to n is None
, which is False
for 0
.
(The MySQL mock didn't seriously need this since it has row_does_not_exist
; however, other mocks would be helpful to have a similar fix applied.)
All the code I could find where the logic should be corrected are:
One thing Touchstone is great at, is setting up a test for a flow that spans multiple services. Drop a rabbit message to service-a for input, service-a talks to service-b, assert that service-b sends a rabbit message for output.
If the flow involves REST calls to both services, however, we are limited to hitting one service via self.service_url
; so for instance you could not POST to service-a for input and then GET service-b for output (to assert on) in the same test.
This could be made possible with a self.services
hashmap, self.services['service-name']
Currently, Touchstone must be completely shutdown and restarted when mock's defaults are changed in develop mode. This significantly slows down development especially when integrating TS with an existing service. Support ability to:
Right now, in order to give tests enough time for Rabbit messages to be processed, a hardcoded delay is introduced. Set too low and the tests become flakey, set higher and they are more reliable but take longer. It would be best if instead there was a timeout, either at the test level or the assertion level, for trying the assertions until they either pass or have continued failing till the timeout is hit.
Since Touchstone currently prints the test failure from within the assertion on failure, some refactoring will be needed to prevent printing assertion failures over and over.
I can think of a couple different possible solutions. They aren't mutually exclusive and could be used together to get the advantages of both.
Most test frameworks I'm familiar with throw an exception on failed assertions, where the exception message describes the failure and the assertion exception class contains an expected value and an actual value.
This leaves it up to the test framework (or a reporting plugin for the framework; see #15) how to print the message and the comparison of expected vs actual.
Cypress assertions do not actually fail until they've tried until a time limit is hit. Touchstone assertions could do likewise. There could be a default time limit for the test with a possibility to override it for individual assertions.
Add support for additional Docker images to run along side mocks and services. This is important if a service has a dependency on an unsupported mock
Several mocks have a feature where one can set the num_times
to None
to assert at least once but no specific number of times matched (>0, >=1).
I can't recall for certain but I think this predates the JSON APIs and validation.ANY
.
It would make more sense to use ANY
instead of None
.
Further, if None
were no longer used for this purpose it would make it easier to avoid bugs conflating 0
and None
such as #44
My suggestion around backwards compatibility, to avoid confusion (old Touchstone tests use None
to mean any, but a newcomer would likely expect it to mean 0
once ANY
can mean any), would be to make it an error to use None
. This is backwards incompatible, but fails explicitly and clearly rather than behaving unexpectedly. In my experience most tests are using either the default (implicitly 1 time) or 0 (asserting something is not there), only a few special cases are using fuzziness such as "assert 3 calls to some URL matching this wildcard pattern" or "assert this was called, but assert it was called any number of times".
While we're at it it may make sense to try to centralize more of the num times logic. As I recall we had an issue about correcting a 0
vs None
bug re. equals vs. is
, it had to be corrected separately in each mock.
Depending on the developer machine (and not the project), sometimes mocks can take longer than their health check time to finish spinning up. It would be helpful to check for a user-level or system-level configuration that can override the health check timing for the mocks, either all mocks or specific ones.
Example:
---
databases:
- name: myapp
collections:
- name: users
documents:
- { "firstName": "John", "lastName": "Smith" }
- >
{
"firstName": "Jane",
"lastName": "Brown"
}
My thinking on MongoDB mocks is that, since it's a database, it should largely parallel MySQL (except where the difference between the two is relevant, if there are any).
In MySQL assertions we have four cases:
None
, asserts field IS NULLvalidation.ANY
, asserts field IS NOT NULLI know Mongo assertions have the first two, maybe the first three. Is there a way to implement all four in MongoDB queries?
When building the HTTP mock, "urlPattern" is used by default. This can be confusing because characters need to be escaped when not expected. "url" should be the default and "urlPattern" should be an option
Services can point to localhost
/host.docker.internal
to reach other services when running in Touchstone Develop. However, under Touchstone Run, the services are given randomized ports and are not reachable. This makes HTTP calls (and other direct, non-Rabbit communication) between two or more services impossible.
My thinking on MongoDB mocks is that, since it's a database, it should largely parallel MySQL (except where the difference between the two is relevant, if there are any).
MySQL does not require anything in the defaults file (beyond the existence of a particular DB for the service to connect to), as the actual tables created by the service can be used by Touchstone on the fly without Touchstone needing to set anything up before their use.
This can be contrasted with, say, in Rabbit the messages are consumed or lost unless Touchstone creates shadow queues ahead of time, hence Touchstone must know about routing in the defaults file.
In Mongo I believe MySQL could be paralleled: services can automatically create their databases and collections, it is possible to query them on the fly without needing information about them in the defaults file.
(Touchstone already parallels MySQL with wiping the entire Mongo DB, not just what's in the defaults, thereby ensuring no state carries over between tests and it must be explicitly set in defaults or per-test mocks setup calls, as is best practice for testing in general. See also #47 )
Some mocks provide an indication of the actual data, e.g. rabbit's list of published messages. MySQL one cannot tell from the Touchstone failure whether a field didn't match and if so which, vs if there are no records.
Could MySQL be updated to query against the primary keys (preferably just looking those up automatically) and compare the fields?
(Arguably related to #38 which is seeing expected field values)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.