Comments (6)
This is a good question, and it remains an open question for me.
(FWIW, basic pagination is documented at https://github.com/shurcooL/githubv4#pagination, but it doesn't cover pagination across more than one resource that this issue is asking about.)
So far in the few places where I need to do pagination across more than one resource type per query, I've been able to get by with only doing pagination at the top level, where it's most important. For the inner levels, I just request the maximum single page size with a "TODO" to figure out a good approach, and that's getting me by for now (enough that this isn't a high priority for me). You can see an example here:
It seems to me that it could be implemented by doing nested iteration. That is, paginate across all entries in the inner-most resource type first, then go to next higher level page, and repeat until all pages are visited. Something like:
for range top level pages {
for range low level pages {
// perform query
}
}
If someone finds good example code, I think this would be worth documenting to save others the effort of figuring it out.
One more thought: this question is applicable to GraphQL as a whole, so perhaps people have found solutions for this in other languages that may be helpful to find.
from graphql.
@dmitshur thats really surprising to hear - seems like this would be a pretty typical thing with complex graphql APIs.
As it stands right now i'm thinking i'm going to have to make a choice between 2 options both of which require a 2nd API call that is JIT (just in time) at its usage:
Adding an additional function on the client where you pass in the top level resource you want to fill something like client.GetTools(*services[0])
OR
adding a function instead of using a field on that struct IE services[0].Tools()
instead of services[0].Tools
which would be a structure function that can then make that additional API call. But i think the really disgusting part about this is that the structure would also need a reference to the client to make the additional API call which would need to be held as a pointer on each of the returned structures or the user would end up needing to pass the client to the tools function like services[0].Tools(client)
which doesn't feel great either.
Definitely seems like a rock and a hard place to implement nested pagination with this framework. /cry
from graphql.
If anyone stumble across this - this is how i've implemented it in our OpsLevel go client
- i made structs that implement nodes and pageInfo data which the toplevel resource uses
- I added "hydrate" functions on those structs
- I then implement a "hydrate" function on the top level resource to call all the nested resources "hydrate" functions. Passing in the client that was used to make the initial API call
- every "hydrate" call appends to the existing struct's node list so the final structure the end user gets has all the nested resource lists fully paginated out and has access to all the data. They just incur
1..N
extra API calls depending on how many pages each resource that is paginated has.
from graphql.
@dmitshur is there any way a field can be included on query struct but ignored when this library generates the graphql query - similar to the json tag omitempty
Something like
type StuffQuery struct {
StuffPages []StuffPage `graphql:"stuff"`
Stuff []Stuff `graphql:",ignored"`
}
This would allow me to get the raw data back from the server into the field StuffPages
but then use my pagination processing to handle pagination and then fill the Stuff
field so that this is the field the enduser interacts with which has all the pageinfo, cursor, hasNextPage etc removed and they just get the final array of Stuff
from graphql.
After some digging it appears not - https://github.com/shurcooL/graphql/blob/master/internal/jsonutil/graphql.go#L257
seems like thats the area which would need to be refactored to support this. Seems like this would be a good feature to add to support writing easy to access structures of paginated data. Thoughts @dmitshur?
from graphql.
Following this thread on Twitter, I came up with the following:
https://gist.github.com/myitcv/f7da6cb073de3c48c9f88b6b95225b4c
The overall goal was to extract:
- all discussions
- all comments from those discussions
- all replies to those comments
for a given GitHub repository, minimising the number of calls to the GitHub GraphQL endpoint as far as possible.
This felt like incredibly hard work, not to mention error-prone.
I would welcome any thoughts on a better approach.
from graphql.
Related Issues (20)
- query with circular dependencies leads to infinite recursion HOT 1
- Any interest in native go type support? HOT 1
- I want to add where condition in the query structure, how to do this
- How to define struct HOT 3
- Query and Mutation methods ignore query content when it's wrapped in an interface HOT 3
- Support uuid type HOT 1
- Help: Unable to unmarshal server reply containing a map into `map[string]interface{}` HOT 4
- "Variable \\\"$evar\\\" is never used. HOT 2
- [Feature reqeust] Add possiblity to load query from string / .gql file HOT 4
- Question: How to implement an enumeration type HOT 1
- Nested struct tag not applied to query HOT 2
- Using alternative rest clients
- OAuth2 Bearer Token with Customer HttpClient HOT 1
- Please make "constructQuery" exported HOT 3
- Allow building mutations from maps instead of structs
- I love this library but it seems unmaintained? HOT 1
- FaunaDB graphql _id defined as _ID not exported HOT 4
- Message: x-hasura-admin-secret but not found,
- Return more verbose error to the caller HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from graphql.