microlinkhq / mql Goto Github PK

View Code? Open in Web Editor NEW

54.0 54.0 10.0 1.4 MB

Microlink Query Language. The official HTTP client to interact with Microlink API for Node.js, browsers & Deno.

Home Page: https://microlink.io/mql

License: MIT License

JavaScript 84.11% HTML 4.78% TypeScript 11.11%

deno http-client javascript nodejs

mql's People

Contributors

Stargazers

Watchers

Forkers

zanachka serebano a1ip delacruz-dev forkkit goleary pyoner schniz isabella232 mohinsandhi

mql's Issues

support for embed content

const apiUrl = getApiUrl(apiEndpoint, input)
  const { body, headers } = await got(apiUrl, { encoding: null })
  const contentType = headers['content-type'].toLowerCase()

  jsome(headers)

  const isUTF = contentType.includes('utf')
  if (!isUTF) return termImg(body)

  const isText = contentType.includes('text/plain')
  jsome(isText ? body.toString() : JSON.parse(body.toString()))

Nested query results

Context: I'm using mql via the https://microlink.io/ service (Node.js API).

I'm looking for a way to put custom query results in a nested structure.

Eg instead of getting a result like:

{
  title: '',
  description: ',
  lang: 'en',
  author: null,
  publisher: '',
  ...
  someCustomData: '',
  moreCustomData: ''
}

It'd be great to nest the custom data something like this:

{
  title: '',
  description: ',
  lang: 'en',
  author: null,
  publisher: '',
  ...
  myData: {
    someData: '',
    moreData: ''
  }
}

On https://microlink.io/docs/mql/getting-started/overview there is an example that displays nesting of queries, like this:

const mql = require('@microlink/mql')

const twitter = (username) =>
  mql(`https://twitter.com/${username}`, {
    data: {
      stats: {
        selector: '.ProfileNav-list',
        attr: {
          tweets: {
            selector: '.ProfileNav-item--tweets .ProfileNav-value',
            attr: 'data-count',
          },
          followings: {
            selector: '.ProfileNav-item--following .ProfileNav-value',
            attr: 'data-count',
          },
          favorites: {
            selector: '.ProfileNav-item--favorites .ProfileNav-value',
            attr: 'data-count',
          },
        },
      },
    },
  })

And the result is nested under stats. Since it seems the nested queries iterate over the parent selector, I did this:

{
  data: {
    // This works
    jsonld: {
      selectorAll: 'script[type="application/ld+json"]',
      attr: 'html',
    },
    // This seems to be solid
    myTitle: {
      selector: 'title',
      attr: 'text',
    },
    meta: {
      selector: '*',
      attr: {

        // This works mostly but sometimes the result is padded with a bunch of css selectors etc.
        title: {
          selector: 'title',
          attr: 'text',
        },
        ogTitle: {
          selector: 'meta[property="og:title"]:not([content=""])',
          attr: 'content',
        },
        twitterTitle: {
          selector: 'meta[name="twitter:title"]:not([content=""])',
          attr: 'content',
        },
        description: {
          selector: 'meta[name="description"]:not([content=""])',
          attr: 'content',
        },
        ogDescription: {
          selector: 'meta[property="og:description"]:not([content=""])',
          attr: 'content',
        },
        twitterDescription: {
          selector: 'meta[name="twitter:description"]:not([content=""])',
          attr: 'content',
        },

        // This always returns null, while putting it at the top level works, see above
        jsonld: {
          selectorAll: 'script[type="application/ld+json"]',
          attr: 'html',
        },
      },
    },
  },
}

But as you can see in the comments above, there are some issues with this approach.

Is there a better way to do this?
Why is the JSON+LD selector always empty when used in the nested way?

If this is not the way to do this, I'll rename this issue a feature request and just use top level queries for now.

Big thanks for an awesome product!

Better types

There are some TODO's to improve types detection based on the context environment

TypeScript declaration file

Hey there, as I've mentioned to you in support, we're using TypeScript & currently it's not possible to import this module in strict mode, as there are no type definitions.

It would be ideal if the library could expose a index.d.ts file so that TypeScript users can use this. As a bonus, this would allow users of VS Code to get autocomplete suggestions even if they're using JavaScript.

I've written the following declaration file for our project, which could be a good starting point for an index.d.ts file:

/// <reference types="node" />

declare module '@microlink/mql' {
  export type WaitUntilEvent =
    | 'load'
    | 'domcontentloaded'
    | 'networkidle0'
    | 'networkidle2'

  export type ScreenshotOptions = Partial<{
    background: string
    browser: 'light' | 'dark'
    click: string | string[]
    deviceScaleFactor: number
    disableAnimations: boolean
    emulation: string
    fullPage: boolean
    hasTouch: boolean
    height: number
    hide: string | string[]
    isLandscape: boolean
    isMobile: boolean
    scrollTo: string
    type: 'jpeg' | 'png'
    waitFor: number | string
    waitUntil: WaitUntilEvent | WaitUntilEvent[]
    width: number
  }>

  export type MqlQueryResponseType =
    | 'author'
    | 'date'
    | 'description'
    | 'image'
    | 'title'
    | 'url'
    | 'lang'
    | 'publisher'

  export type MqlQueryOptions = Partial<{
    attr: string | string[]
    selector: string | string[]
    selectorAll: string | string[]
    type: MqlQueryResponseType
  }>

  export interface MqlQuery {
    [field: string]: MqlQueryOptions
  }

  export type MicrolinkApiOptions = Partial<{
    audio: boolean
    data: MqlQuery
    embed: string
    filter: string
    force: boolean
    headers: object
    meta: boolean
    palette: boolean
    prerender: boolean | 'auto'
    proxy: string
    screenshot: boolean | ScreenshotOptions
    ttl: string | number
    url: string
    video: boolean
  }>

  export type MqlOptions = Partial<{
    apiKey: string
    retry: number
    cache: Map<string, any>
    timeout: number
  }>

  export interface ImageInfo {
    width: number
    height: number
    type: string
    size: number
    size_pretty: string
  }

  export interface PlayableMediaInfo extends ImageInfo {
    duration: number
    duration_pretty: string
  }

  export type MqlResponseData = Partial<{
    author: string
    date: string
    description: string
    video: string
    lang: string
    publisher: string
    title: string
    url: string
    image: ImageInfo
    logo: ImageInfo
    video: PlayableMediaInfo
    audio: PlayableMediaInfo
  }>

  export type MqlStatus = 'success' | 'fail'

  export interface MqlResponse {
    status: MqlStatus
    data: MqlResponseData
    response: Response
  }

  declare function mql(url: string, opts?: MqlOptions & MicrolinkApiOptions): Promise<MqlResponse>

  declare namespace mql {
    export class MicrolinkError extends Error {}
  }

  export = mql
}

I've written this based on the documentation on the Microlink website. There were a couple of cases where the documentation was incorrect (eg. the cache option), so I've used a best guess based on the examples provided on the website.

I haven't massively tested this beyond our pretty simple use case, but as far as I know this is working for us.

There are a bunch of APIs that I didn't expose in here (getApiUrl, fetchFromApi, etc.), as I don't see us using these at the moment. A "official" index.d.ts file should of course include these.

Thanks a lot for this service, it's super helpful!

React | Failed to compile | Module parse failed: Unexpected token (13:40)

Hi everyone,

I am trying to implement the Microlink/SDK into a project and I am experiencing a problem when I compile (cf . below)

Screenshot

After getting this error, I updated all the Dependencies and DevDependencies but it didn't help.
Here is the list, maybe something is wrong there:

  "dependencies": {
    "@babel/core": "^7.8.4",
    "@babel/plugin-proposal-class-properties": "^7.8.3",
    "@babel/plugin-transform-react-constant-elements": "^7.8.3",
    "@babel/preset-env": "^7.8.4",
    "@babel/preset-react": "^7.8.3",
    "@microlink/demo-links": "^1.0.12",
    "@microlink/react": "^5.1.0",
    "@storybook/addon-storyshots": "^5.3.13",
    "@storybook/react": "^5.3.13",
    "babel-eslint": "^10.0.3",
    "babel-plugin-transform-react-pure-class-to-function": "^1.0.1",
    "babel-plugin-transform-react-remove-prop-types": "^0.4.24",
    "core-js": "^3.2.1",
    "cross-fetch": "^3.0.4",
    "enzyme": "^3.11.0",
    "enzyme-adapter-react-16": "^1.15.2",
    "eslint-plugin-jsx-a11y": "^6.2.3",
    "husky": "^4.2.3",
    "jest-environment-enzyme": "^7.1.2",
    "jest-enzyme": "^7.1.2",
    "jest-styled-components": "^7.0.0",
    "ky": "^0.17.0",
    "ky-universal": "^0.5.0",
    "moment": "^2.24.0",
    "oidc-client": "^1.9.1",
    "prettier-standard": "^16.1.0",
    "react": "^16.10.1",
    "react-dom": "^16.10.1",
    "react-redux": "^7.1.1",
    "react-router": "^5.1.2",
    "react-router-dom": "^5.1.2",
    "redux-thunk": "^2.3.0",
    "rollup": "^1.31.1",
    "rollup-plugin-babel": "^4.3.3",
    "rollup-plugin-commonjs": "^10.1.0",
    "rollup-plugin-filesize": "^6.2.1",
    "rollup-plugin-node-resolve": "^5.2.0",
    "rollup-plugin-replace": "^2.2.0",
    "rollup-plugin-terser": "^5.2.0",
    "rollup-plugin-visualizer": "^3.3.1",
    "standard": "^14.3.1",
    "styled-components": "^5.0.1",
    "unfetch": "^4.1.0"
  },
  "devDependencies": {
    "@types/jest": "^24.0.18",
    "@types/node": "^12.7.10",
    "@types/react": "^16.9.4",
    "@types/react-dom": "^16.9.1",
    "@types/react-redux": "^7.1.4",
    "@types/react-router-dom": "^5.1.0",
    "babel-jest": "^24.9.0",
    "jest": "^24.9.0",
    "node-sass-chokidar": "^1.3.5",
    "npm-run-all": "^4.1.5",
    "react-scripts-ts": "^3.1.0",
    "ts-jest": "^24.1.0",
    "tslint": "^5.20.0",
    "tslint-config-prettier": "^1.18.0",
    "tslint-react": "^4.1.0",
    "typescript": "^3.6.3"
  }

I also updated the ky and ky-universal modules, as it seems the error happens at this point, but no effect-.

Context of Use

I try to build a website where users could paste an url link into an input field and the microlink render a preview, here is how I call the module, base on your CodeSandBox example for React:

const LinkPreview = () => (
  <Microlink url={this.state.titleValue === '' ? 'https://microlink.io' : this.state.titleValue} />
);

I am compiling with yarn start and running on a local server, not sure if it's the right way.
I tried to compile with yarn build and I get the same message.

Do you have any idea what I am doing wrong ? :-)

Thank you in advance for your help!

An in-range update of ky is breaking the build 🚨

The dependency ky was updated from `0.11.0` to `0.11.1`.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

ky is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for v0.11.1

Fix TypeScript 3.5 compatibility 93d2faf

v0.11.0...v0.11.1

Commits

The new version differs by 3 commits.

07838c7 0.11.1
93d2faf Fix TypeScript 3.5 compatibility
5a4b4b0 Enable the repo sponsor button

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Add `cache` for browser side

Incorrect use of `module` field in `package.json`

Currently the module field in package.json is pointing to the browser only implementation of this library meaning modern bundling tools that bundle for Node.js are importing the wrong code.

Also the module field should be used only for ESModule compliant code, which looking at the source code this is not the case.

My recommendation would be to just remove module field from package.json.

Does not work for instagram links, always returns login url

I have been using mql Pro with user agent and proxy. It never works to fetch for the instgram URL and always gives the response.

lang: "en"
author: null
title: "Login • Instagram"
publisher: "instagram.com"
image: Object {…}
date: "2024-03-10T21:06:13.000Z"
description: "Welcome back to Instagram. Sign in to check out what your…interests have been capturing & sharing around the world."
url: "https://www.instagram.com/accounts/login/"
logo: Object {url: "https://static.cdninstagram.com/rsrc.php/v3/yI/r/VsNE-OHk_8a.png", type: "png", size: 1772, …}

No response types

Hello! Thanks for your work on this library. On the current version 0.10.34, there do not seem to be any response types. Do you have any suggestions for how to type the response in typescript? thanks so much!

Code sample:

   const { status, data } = await mql("https://google.com", {
      meta: true,
    });

status and data give errors like: Property data does not exist on type {}.

convert snake case props into camel case

just doing this

https://github.com/Kikobeats/to-query/blob/master/map.js#L6

the bad thing: it adds more bytes into the build.

Add `number` Data Shape

An in-range update of ky-universal is breaking the build 🚨

The dependency ky-universal was updated from `0.2.0` to `0.2.1`.