See https://github.com/vega/editor/issues/604

See https://github.com/vega/editor/tree/server. I want to run this at vega.now.sh.

Somehow, I cannot sign in on safari. After going through the login process on github, I don't get signed in on the editor.

https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/

We have a dependency on now to build the backend but as stated in https://www.npmjs.com/package/now it is deprecated so if anything breaks in future because of now we wont have a update for it.

We have setup instructions in two places right now

Overview

Pagination in GraphQL isn't done using page numbers like in a standard API endpoint which looks something like this: https://example.com/?page=1. In graphQL, we use cursor which a base64 encoded string or offset (this method is not available in octokit/grapql). Please read about these methods here - https://graphql.org/learn/pagination/#pagination-and-edges.
Therefore, we might need to resort to store cursors on front-end to paginate.

Working

I will first explain what queries and its corresponding outputs mean.

Input Query

{
  user(login: "algomaster99") {
    gists(first: 1, orderBy: {field: CREATED_AT, direction: DESC}) {
      edges {
        cursor
        node {
          name
          description
        }
      }
      pageInfo {
        startCursor
        hasPreviousPage
        endCursor
        hasNextPage
      }
      totalCount
    }
  }
}

Output

{
  "data": {
    "user": {
      "gists": {
        "edges": [
          {
            "cursor": "Y3Vyc29yOnYyOpK5MjAxOS0wOC0zMFQwMToyNjo0NSswNTozMM4F17K9",
            "node": {
              "name": "19adeb7cfaffadeefda5b39a412f317c",
              "description": "Latest"
            }
          }
        ],
        "pageInfo": {
          "startCursor": "Y3Vyc29yOnYyOpK5MjAxOS0wOC0zMFQwMToyNjo0NSswNTozMM4F17K9",
          "hasPreviousPage": false,
          "endCursor": "Y3Vyc29yOnYyOpK5MjAxOS0wOC0zMFQwMToyNjo0NSswNTozMM4F17K9",
          "hasNextPage": true
        },
        "totalCount": 4
      }
    }
  }
}

Explanation of output

• Edges: It is an array of all the gists fetched according to the condition (first: 1, orderBy: {field: CREATED_AT, direction: DESC}). Fetching of one gist is enforced by: first: 1

  • Cursor: This is a base64 encoded string which is, I think, a unique way to index gists.
  • Node: This is a gist object. The name and description attributes correspond to the ID and title of the gists.

• PageInfo: This gives the details about the pages.

  • Start cursor: Shows the cursor of the first gist.
  • End cursor: Shows the cursor of the last gist.
  • hasPreviousPage: Checks if there are any gist before the startCursor.
  • hasNextPage: Checks if there are any gists after the endCursor.

All gists corresponding to cursors between startCursor and endCursor are displayed in edges.

• Total count: Shows the number of gists.

Solution

To implement pagination, let us consider the following assumptions first:

• Number of gist in one page: 2 (I am taking my case specifically 😅)

1. When we open the Gist modal on front-end, an initial request will be made to the back-end to get the following data:-

• Total edges as an array of cursors.
• First page data (the details of first two gists).

"edges": [
          {
            "cursor": "Y3Vyc29yOnYyOpK5MjAxOS0wOC0zMFQwMToyNjo0NSswNTozMM4F17K9",
            "node": {
              "name": "19adeb7cfaffadeefda5b39a412f317c",
              "description": "Latest",
              "files": [
                {
                  "name": "spec.json"
                }
              ]
            }
          },
          {
            "cursor": "Y3Vyc29yOnYyOpK5MjAxOS0wOC0yOFQxMjowNjoyOSswNTozMM4F1z-E",
            "node": {
              "name": "e5cf1cc1e12a40bc0a69cf77c934f239",
              "description": "It works so fine!",
              "files": [
                {
                  "name": "test.vl.json"
                }
              ]
            }
          },
          {
            "cursor": "Y3Vyc29yOnYyOpK5MjAxOS0wOC0yM1QwNzo1Mjo1OCswNTozMM4F1g8C",
            "node": {
              "name": "6ee25328dd6aa5c86476b0fa5d4a85b9",
              "description": "Google Summer of Code 2019 - Final Evaluation",
              "files": [
                {
                  "name": "GSOC19.md"
                }
              ]
            }
          },
          {
            "cursor": "Y3Vyc29yOnYyOpK5MjAxOC0xMi0xNVQyMTowNToxOSswNTozMM4FlBQJ",
            "node": {
              "name": "830edacfc6b36fcf5604332cf7b8cbc1",
              "description": "Key generation using RSA algorithm",
              "files": [
                {
                  "name": "key_generation.py"
                }
              ]
            }
          }
        ],
"totalCount": 4

• An array of all cursors will be sent.
• The ID, files, etc. of first two gists will be sent to display the information for the first page.

2. Since there are 4 gists in total, the pagination component will look like this:
   | ⬅️ | 1 (selected) | 2 | ➡️ |

3. Clicking on 2 will find the corresponding cursor from the array and send a request to back-end. For example, assume the array is as follows: ['1', '2', '3', '4']. Since we have two gist on the first page, the second page will have the following two gist viz 3 and 4. So the nextCursor will be 3.

The query can look like this:

{
  user(login: "algomaster99") {
    gists(first: 2, after: "2" orderBy: {field: CREATED_AT, direction: DESC})
...

This will get 3 and 4.

One problem

The above solution works well for people who have 100 gist. first attribute in the query allows to fetch only 100 gists at a time. Therefore, we will have cursors for all 100 gists.

Question Why getting all cursors are important?
If someone from 1st page goes to 103rd page (suppose), we will need the 206th cursor to get contents of that page.

We could probably just pass the auth token to the editor frontend and make the backend stateless. This would mean that we could do the graphql queries from the frontend directly.

See https://github.com/vega/vega-embed/blob/master/package.json#L50 for an example.

The advantage would be that we can share the same dev config across all projects.

We can search over the following things for vega and vega-lite visualization:-

• Description of the visualization it can be of the visualization or the gist.
• Type of graph
• Search a specific user's visualization

Elasticsearch DB

• This DB doesn't support relations between any fields. One wouldn't be able to link user and its gist. This will be a problem when we are searching for visualization according to a user.
• It also doesn't have an updated ODM. Making search queries will be difficulty and be a little long. Current version of elasticsearch is 7.0.0 while its ODM was updated a year ago for 5.0.0.
• This repository's file simply makes api calls to GitHub's api to retrieve a user and the clicked GistID.
• Searching can be performed like they have done in this file. The queries written in there are quite long to write. Also, it is good for textual search over random data. It can't search for visualization according to users. Our data is very organized and can be simply put in a schema.
• ElasticSearch does not provide any authentication or access control functionality.
• I think it would have been a good choice if had to just search over random texts.

PostgreSQL vs. MongoDB

• They both are pretty good with MongoDB having a famous ODM called Mongoose.
• I am inclined towards MongoDB since it deals with JSON objects which need not be serialized before rendering it in an API view. Hence, shortening the process.
• Since our schema definition isn't really perfect for gists because we don't know yet what users would search over, MongoDB is a better choice since it stores document like data and doesn't necessarily require a predefined structure. So later on we can add more fields in a gist so that it can be easily indexed and searched over.

Add a link to https://vega.github.io/editor-backend/ to https://vega.now.sh/

We can add favicon using this link but Express has different instructions to set it up. It suggests to install serve-favicon and have the favicon.ico file as a static file in the repository. Should we download it and keep at as an asset in the project?

Move setup from https://github.com/vega/editor/tree/server

Instead of npm.

When one logs in for the first time, a cookie is set with expiry 1 month. And whenever one re-visits after some time, let's say 5 days, the TTL resets because disableTouch is by default false which is the expected behaviour.
Although the cookie stored in the browser doesn't show any changes or increment in the expiry date which needs to be fixed.

So we can fetch more than the first 20 gists.