letsvalidate / api Goto Github PK

API that uncovers the technologies used on websites and generates thumbnail from screenshot of website

thumbnail analyze api screenshot chrome capture-screenshots generating-thumbnails chrome-headless headless-chrome rest-api

api's People

Contributors

Stargazers

Watchers

Forkers

bussiere alonronin yonatanlebo aubreyhewes chrisemoulton swaylin zichoolelonggui shamarkanda verceti-ke geekwolverine

api's Issues

some pages appear elongated

Hello!

For most sites API works excellent, but we found site embertribe.com, which screenshot appears elongated.
https://api.letsvalidate.com/v1/thumbs?full=true&quality=80&url=embertribe.com

Could you please check, whether it's possible to fix somehow.

Thank you.

Add option to prettify JSON output

First off, this thing is pretty cool. Second, it would be awesome to have an option to output the JSON responses prettified.

So instead of just being able to receive:

{"url":"https://hackcabin.com/","originalUrl":"http://hackcabin.com","applications":[{"name":"Amazon Cloudfront","confidence":"100","version":"","icon":"Amazon-Cloudfront.svg","website":"http://aws.amazon.com/cloudfront/","categories":["CDN"]},{"name":"Amazon S3","confidence":"100","version":"","icon":"aws-s3.svg","website":"http://aws.amazon.com/s3/","categories":["Miscellaneous"]},{"name":"Google Analytics","confidence":"100","version":"UA","icon":"Google Analytics.svg","website":"http://google.com/analytics","categories":["Analytics"]},{"name":"Hugo","confidence":"100","version":"0.21","icon":"Hugo.png","website":"http://gohugo.io","categories":["CMS","Blogs"]}]}

We would be able to have returned from the API this with use of the option:

{
  "url": "https://hackcabin.com/",
  "originalUrl": "http://hackcabin.com",
  "applications": [
    {
      "name": "Amazon Cloudfront",
      "confidence": "100",
      "version": "",
      "icon": "Amazon-Cloudfront.svg",
      "website": "http://aws.amazon.com/cloudfront/",
      "categories": [
        "CDN"
      ]
    },
    {
      "name": "Amazon S3",
      "confidence": "100",
      "version": "",
      "icon": "aws-s3.svg",
      "website": "http://aws.amazon.com/s3/",
      "categories": [
        "Miscellaneous"
      ]
    },
    {
      "name": "Google Analytics",
      "confidence": "100",
      "version": "UA",
      "icon": "Google Analytics.svg",
      "website": "http://google.com/analytics",
      "categories": [
        "Analytics"
      ]
    },
    {
      "name": "Hugo",
      "confidence": "100",
      "version": "0.21",
      "icon": "Hugo.png",
      "website": "http://gohugo.io",
      "categories": [
        "CMS",
        "Blogs"
      ]
    }
  ]
}

Please disable website cache

I scan a weather site time by time. It looks like the websites are cached. Is it possible to disable the cache?

502 error when setting width, height and size at the same time

Like the title says, if you set the width, height and size at the same time when doing a request to /v1/thumbs it just gives a 502 error, the same goes for very large pages like http://www.eveandersson.com/pi/digits/1000000.

Full mode produces - "code":"004": An internal error accoured.

Hello,

First of all thanks a lot for your API, it's amazing.
I've started receiving error with parameter full=true
{"error":{"code":"004","message":"An internal error accoured."}}

There are several examples:

Could you please check whether it's a bug or my incorrect usage of API.
Best wishes.

Extract data from parsed structured data.
e.g.

<script type="application/ld+json">
{
    "@context": "http://schema.org",
    "@type": "Organization",
    "name": "Let's Validate",
    "url": "https://www.letsvalidate.com/",
    "logo": "https://www.letsvalidate.com/img/logo.png",
    "email": "[email protected]",
    "description": "Site launch checklist checker",
    "sameAs": [
        "https://www.facebook.com/letsvalidate",
        "https://m.me/letsvalidate",
        "https://twitter.com/letsvalidate",
        "https://plus.google.com/+letsvalidate",
        "https://vk.com/letsvalidate"
    ]
}
</script>

Extract data from parsed meta data of twitter card.
e.g.

<meta name="twitter:site" content="@letsvalidate">

Prerender web apps before extracting data.
Deeply crawl, e.g. contact page
Maybe to use Google knowledge graph too?

the endpoint:

https://api.letsvalidate.com/v1/contacts?url=docker.com&prettify=true

result:

{
  "url": "https://www.docker.com/",
  "originalUrl": "http://docker.com",
  "contacts": {
    "email": null,
    "fax": null,
    "tel": null,
    "socials": [
      {
        "domain": "twitter.com",
        "id": null,
        "name": "docker",
        "confidence": 100,
        "url": "http://twitter.com/docker"
      },
      {
        "domain": "youtube.com",
        "id": null,
        "name": "dockerrun",
        "confidence": 100,
        "url": "http://www.youtube.com/user/dockerrun"
      },
      {
        "domain": "facebook.com",
        "id": null,
        "name": "docker.run",
        "confidence": 100,
        "url": "https://www.facebook.com/docker.run"
      }
    ]
  }
}

@jhabdas What do you think, is it worth to implement it or already available such api?

No sources included

There was probably a mistake when uploading the repository.
There is no LICENSE file (you should always have a good license!), and I can't see the source code.

forbidden words error

https://api.letsvalidate.com/v1/thumbs/?url=https://en.wikipedia.org/wiki/cashmeran&viewport=1200x900&full=true

returns an error:

{
  "error": {
    "code": "016",
    "message": "The url contains forbidden words"
  }
}

The forbidden word seems to be cashmeran, but with capital C it all of a sudden is allowed...

https://api.letsvalidate.com/v1/thumbs/?url=https://en.wikipedia.org/wiki/Cashmeran&viewport=1200x900&full=true