Compatibility Acceptance Tests about graphql-spec HOT 6 CLOSED

We started to work on compatibility acceptance tests here:

https://github.com/graphql-cats/graphql-cats

If you are working on a GraphQL implementation, it would be great if you could share your opinion and join us.

from graphql-spec.

I'm going to close this issue since graphql-cats is well underway!

from graphql-spec.

Thanks, for opening this. I'm renaming the task to use the terminology we have been using to talk about this.

We agree that this is really important, but are not yet actively working on it.

from graphql-spec.

do you already have an idea which form it will take?

Sort of.

I think it will specify a schema to implement, along with a data set to use. Ideally this schema is as small as possible while still illustrating every feature of GraphiQL and supporting all of the acceptance tests. Ideally this schema is written in the GraphQL Type DSL so it is easy to understand.

Then there should be a list of "tests" where each test includes the following:

• Description: what is this test testing and why
• Query: a document to submit to the GraphQL service
• Response: The response expected from the GraphQL service

Then optionally also include Variables and OperationName when those are relevant to a test.

This form should allow us to test everything from parser corner cases and proper detection and reporting of syntax errors, to validation errors for all the rules we specify, to execution behavior.

from graphql-spec.

I guess implementing something like this can be a challenging task, considering that it suppose to test implementations written in different languages.

Ideally, the form I wrote above only needs to be written once and can leverage the fact that the GraphQL language is represented as a string independent of any one backend language and the output can be serialized to JSON.

However the provided schema still needs to be implemented by that backend. Either by interpreting the GraphQL Type DSL or by manually implementing it. Hopefully that is not too much of a burden.

However overall I would certainly rate this task as "challenging" :D

from graphql-spec.

Thanks for the reply! Glad to hear that you already thought about it. The approach you described sounds good to me. I was also thinking about something like this.

About the schema, reference implementation now has a schema materialiser, which also can be implemented in other languages. Resulting schema does not contain the resolve method, but even in this form it can be used for many tests that do not require execution, like query validation and parsing or introspection. As far as I saw, even execution tests have very similar resolve implementations, which are in most cases fall in one of these categories:

• successful resolve with null
• successful resolve with not-null value
• successful resolve of promise with null
• successful resolve of promise with not-null value
• exception is thrown
• promise was resolved with failure

Maybe we can create an enum of possible implementation behaviours and then enhance textual schema representation to include this information in some form. This will allow schema materialiser to create simple implementations of resolve method. This means that a test schema can be completely generated from textual description, just like test cases. What do you think about this idea?

from graphql-spec.