yhat / scrape Goto Github PK

View Code? Open in Web Editor NEW

1.5K 41.0 100.0 10 KB

A simple, higher level interface for Go web scraping.

License: BSD 2-Clause "Simplified" License

Go 100.00%

scrape's Introduction

scrape

A simple, higher level interface for Go web scraping.

When scraping with Go, I find myself redefining tree traversal and other utility functions.

This package is a place to put some simple tools which build on top of the Go HTML parsing library.

For the full interface check out the godoc

Sample

Scrape defines traversal functions like Find and FindAll while attempting to be generic. It also defines convenience functions such as Attr and Text.

// Parse the page
root, err := html.Parse(resp.Body)
if err != nil {
    // handle error
}
// Search for the title
title, ok := scrape.Find(root, scrape.ByTag(atom.Title))
if ok {
    // Print the title
    fmt.Println(scrape.Text(title))
}

A full example: Scraping Hacker News

package main

import (
	"fmt"
	"net/http"

	"github.com/yhat/scrape"
	"golang.org/x/net/html"
	"golang.org/x/net/html/atom"
)

func main() {
	// request and parse the front page
	resp, err := http.Get("https://news.ycombinator.com/")
	if err != nil {
		panic(err)
	}
	root, err := html.Parse(resp.Body)
	if err != nil {
		panic(err)
	}

	// define a matcher
	matcher := func(n *html.Node) bool {
		// must check for nil values
		if n.DataAtom == atom.A && n.Parent != nil && n.Parent.Parent != nil {
			return scrape.Attr(n.Parent.Parent, "class") == "athing"
		}
		return false
	}
	// grab all articles and print them
	articles := scrape.FindAll(root, matcher)
	for i, article := range articles {
		fmt.Printf("%2d %s (%s)\n", i, scrape.Text(article), scrape.Attr(article, "href"))
	}
}

scrape's People

Stargazers

Watchers

Forkers

tanitall aosmith kandangjamur rui-santos andradeandrey llp-full-stack zeb209 geekcheng codevlabs ealliaume mrkeng mazharul dotpot code-kshetra iarnaud drooids kylemclaren liujianping schuka-kirill insider19 li5ch angrilove modulexcite andeya h2object magastzheng ezeql flyeven pombredanne sh1nu11bi collects ghostbite reinhardhsu gozes lawrencecraft tommady eazynow jochasinga rlugojr hoai tpltnt jwasinger kerasking xiris zhujinlong wangxiaowang digideskio restocks cophy08 antrampa wanghuahui psidium hemel-cse pkothbauer budiboi22 strogo soar0603 russellwmy hyhlinux cognifly joonhochoi gplume jxub nghiatranuit negz5 joelself xiangxd dbs6713 forging2012 cesardarinel alexleethinker sysbot 82ndairbornediv priestd09 misbahkhan rchirakk nkatarmal djfordz cchunkiat swizzley xsoer ykankaya adamluzsi-backup swainl randolphchina hswuhao zhengger alexlombry codelingobot hubitor feynmanium webscrapist pickkaa antichown btautist icodein wjt2015 mediumboost iq-scm himbarbuana

scrape's Issues

FindAll doesn't return nested matching nodes

The function FindAll bails from the recursion immediately when the node is a match. Thus, if we have HTML like the following:

<div class="bigbird">
  <div class="container">
    <div class="bigbird">
    </div>
  </div>
</div>

and you issue a search like this:

nodes := scrape.FindAll(rootDivNode, By.Class("bigbird"))

your nodes array will be 1 rather than 2.

I'll submit a PR for this shortly, but I wanted to get your advice: do you want to keep the old functionality and introduce a new function like FindAllDescending() for compatibility purposes or is it best to patch the first method and potentially break the interface?

Return the first-child

How can I return the first child element?
Trying to return the first-child td.

<table id="myTable">
  <thead>
    <tr>
      <th>1</th>
      <th>2</th>
      <th>3</th>      
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>this is returned</td>
      <td>not returned</td>
      <td>not returned</td>
    </tr>
    <tr>
      <td>this is returned</td>
      <td>not returned</td>
      <td>not returned</td>
    </tr>
    <tr>
      <td>this is returned</td>
      <td>not returned</td>
      <td>not returned</td>
    </tr>
  </tbody>
</table>

package main

func main() {
        matcher := func(n *html.Node) bool {
                if n.Parent != nil && n.Parent.Parent != nil && n.Parent.Parent.Parent != nil {
                        return scrape.Attr(n.Parent.Parent.Parent, "id") == "myTable"
                }
                return false
        }
        
        matches := scrape.FindAll(root, matcher)
        fmt.Printf("len: %d\n", len(matches))
        for i, m := range matches {
                fmt.Printf("%2d %s\n", i, scrape.Text(m))
        }
}

How to run it?

I try your code but on https://play.golang.org i take this error:

/tmp/sandbox817629291/main.go:7:2: cannot find package "golang.org/x/net/html" in any of:
    /usr/src/go/src/golang.org/x/net/html (from $GOROOT)
    /go/src/golang.org/x/net/html (from $GOPATH)
/tmp/sandbox817629291/main.go:8:2: cannot find package "golang.org/x/net/html/atom" in any of:
    /usr/src/go/src/golang.org/x/net/html/atom (from $GOROOT)
    /go/src/golang.org/x/net/html/atom (from $GOPATH)

And i try on my cmd:

scrape.go:7:2: cannot find package "golang.org/x/net/html" in any of:
        C:\Go\src\golang.org\x\net\html (from $GOROOT)
        ($GOPATH not set)
scrape.go:8:2: cannot find package "golang.org/x/net/html/atom" in any of:
        C:\Go\src\golang.org\x\net\html\atom (from $GOROOT)
        ($GOPATH not set)

How to use debpkg: usage help please

I have no idea about golang, yet googled it, did this:

go build
got debpkg

rizvan@debian:~/build/debpak$ ./debpak -deb-version "stable" -pkg gedit

2021/03/19 10:26:29 Finished walking 811 dependencies, 625 of which were dups.
{"name":"gedit","no-autogen":"true","sources":[{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gcc-8/gcc-8-base_8.3.0-6_amd64.deb","sha256":"1b00f7cef567645a7e695caf6c1ad395577e7d2e903820097ebd3496ddcfcc84","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gcc-8/libgcc1_8.3.0-6_amd64.deb","sha256":"b1bb7611f3372732889d502cb1d09fe572b5fbb5288a4a8b1ed0363fecc3555a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/glibc/libc6_2.28-10_amd64.deb","sha256":"6f703e27185f594f8633159d00180ea1df12d84f152261b6e88af75667195a79","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libf/libffi/libffi6_3.2.1-9_amd64.deb","sha256":"d4d748d897e8e53aa239ead23a18724a1a30085cc6ca41a8c31b3b1e1b3452f4","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/u/util-linux/libuuid1_2.33.1-0.1_amd64.deb","sha256":"90b90bef4593d4f347fb1e74a63c5609daa86d4c5003b14e85f58628d6c118b2","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/u/util-linux/libblkid1_2.33.1-0.1_amd64.deb","sha256":"0b15f3eb3cf2fbe540f99ae1c9fd5ec1730f2245b99e31c91755de71b967343a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pcre3/libpcre3_8.39-12_amd64.deb","sha256":"5496ea46b812b1a00104fc97b30e13fc5f8f6e9ec128a8ff4fd2d66a80cc6bee","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libs/libselinux/libselinux1_2.8-1+b1_amd64.deb","sha256":"05238a8c13c32418511a965e7b756ab031c140ef154ca0b3b2a1bb7a14e2faab","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/u/util-linux/libmount1_2.33.1-0.1_amd64.deb","sha256":"b8b28669dc4995a7a48d47d9199d1806d4fce9c4051277279d4dcc514c086ba3","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/z/zlib/zlib1g_1.2.11.dfsg-1_amd64.deb","sha256":"61bc9085aadd3007433ce6f560a08446a3d3ceb0b5e061db3fc62c42fbfe3eff","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/glib2.0/libglib2.0-0_2.58.3-2+deb10u2_amd64.deb","sha256":"9b2d2c420beed1bb115b05c7766e981eab3865f9e9509d22fc621389614d2528","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/dconf/libdconf1_0.30.1-2_amd64.deb","sha256":"22775563fd803db3dafe4fcc93950f72acf04e3d87b51b3dd5c107b21105a5ff","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/dconf/dconf-service_0.30.1-2_amd64.deb","sha256":"1adc68353e17f12ceb3f2e01bb0cb4e5d11b547b9436a89fa0209c46cf028c51","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/dconf/dconf-gsettings-backend_0.30.1-2_amd64.deb","sha256":"8dd9f676ed51db557cfdbb107542bf5406627dc1c83ded565149f02abb60e268","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/e/expat/libexpat1_2.2.6-2+deb10u1_amd64.deb","sha256":"d60dee1f402ee0fba6d44df584512ae9ede73e866048e8476de55d9b78fa2da1","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/b/bzip2/libbz2-1.0_1.0.6-9.2~deb10u1_amd64.deb","sha256":"238193cbaa71cc5365ef2aa5ad45de8521ac38dd54f4ab53bafa7de15046fa89","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/x/xz-utils/liblzma5_5.2.4-1_amd64.deb","sha256":"292dfe85defad3a08cca62beba85e90b0231d16345160f4a66aba96399c85859","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/attr/libattr1_2.4.48-4_amd64.deb","sha256":"4ba903c087f2b9661e067ca210cfd83ef2dc1a162a15b8735997bfa96ac8e760","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/acl/libacl1_2.2.53-4_amd64.deb","sha256":"ca1b512a4a09317018408bbb65ee3f48abdf03dcb8da671554a1f2bd8e5d4de4","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/t/tar/tar_1.30+dfsg-6_amd64.deb","sha256":"8afffcf03195b06b0345a81b307d662fb9419c5795e238ccc5b36eceea3ec22f","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/dpkg/dpkg_1.19.7_amd64.deb","sha256":"4eaa78124b0a5495fd06afefc79f7c96a7f7795c6b6a45349ac4173b0c9b7362","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/perl/perl-base_5.28.1-6+deb10u1_amd64.deb","sha256":"41191859a20c49a1f629ecd8899576631858676c4079f7bbff3808f5376e3680","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/debconf/debconf_1.5.71_all.deb","sha256":"27a3138139b8cce80a814db942688413d94c8f538651a9944d616962a8f71fae","dest":"debs"},{"type":"file","url":"http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl1.1_1.1.1d-0+deb10u5_amd64.deb","sha256":"1741ec08b10caa4d3c8a165768323a14946278a7e6fb9cd56ae59cf4fe1ef970","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3.7/libpython3.7-minimal_3.7.3-2+deb10u2_amd64.deb","sha256":"783822ae63e3bbdadeabfc8121fc920652248acf9de6be9b1bb42d2a4a0baaf1","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3.7/python3.7-minimal_3.7.3-2+deb10u2_amd64.deb","sha256":"c3f234591469e8fc3eaac9cb6a20732b10e4392d29cf8c743216c6ea841a2133","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3-defaults/python3-minimal_3.7.3-1_amd64.deb","sha256":"9c937923b35ac24f5cb6be81626f00dd6b810fc0889e5b3b08b7ffc9d179ff1b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/db5.3/libdb5.3_5.3.28+dfsg1-0.5_amd64.deb","sha256":"c7f0e9a423840731362ee52d4344c0bcf84318fbc06dad4fefe0e61d9e7062bc","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/m/mpdecimal/libmpdec2_2.4.2-2_amd64.deb","sha256":"9ca85e6e2645a5e660431294320658ec7a2910d9fed90ca4e648c1211a2b844b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/n/ncurses/libtinfo6_6.1+20181013-2+deb10u2_amd64.deb","sha256":"7f39c7a7b02c3373a427aa276830a6e1e0c4cc003371f34e2e50e9992aa70e1a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/n/ncurses/libncursesw6_6.1+20181013-2+deb10u2_amd64.deb","sha256":"7dffe9602586300292960f2e3cf4301acfc64a91aed6fa41ea2e719ae75788b3","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/t/texinfo/install-info_6.5.0.dfsg.1-4+b1_amd64.deb","sha256":"1b29164d4254cf34084fcd3b8ee2c4782874386e2a1c5099a1f5b55475052815","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/r/readline/readline-common_7.0-5_all.deb","sha256":"153d8a5ddb04044d10f877a8955d944612ec9035f4c73eec99d85a92c3816712","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/r/readline/libreadline7_7.0-5_amd64.deb","sha256":"01e99d68427722e64c603d45f00063c303b02afb53d85c8d1476deca70db64c6","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/s/sqlite3/libsqlite3-0_3.27.2-3+deb10u1_amd64.deb","sha256":"19268b796e62f754400c67c69cb759220089cf10aaa5dfd72a84ab1a818caa08","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/m/mime-support/mime-support_3.62_all.deb","sha256":"776efd686af26fa26325450280e3305463b1faef75d82b383bb00da61893d8ca","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3.7/libpython3.7-stdlib_3.7.3-2+deb10u2_amd64.deb","sha256":"bfa1a449fcd229456f73b6fb6af24aeb9ad3d6b998b650b7e5615b8101e5d3c0","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3-defaults/libpython3-stdlib_3.7.3-1_amd64.deb","sha256":"4f8883d378e698aa89b7bd4b68ce8e7cca01c961d3df87fafe4c079bb4668f5b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3.7/python3.7_3.7.3-2+deb10u2_amd64.deb","sha256":"82cc1d080c6921b84ff4177889f6dfc4315dd8c2f9ce31dda5fa083629d48c32","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3-defaults/python3_3.7.3-1_amd64.deb","sha256":"eb7862c7ad2cf5b86f3851c7103f72f8fa45b48514ddcf371a8e4ba8f02a79e5","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gedit/gedit-common_3.30.2-2_all.deb","sha256":"d6fec42b316afc4eb8ba3a3df6b4401f19aa56366925d173abc4cf332840b8ea","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gobject-introspection/libgirepository-1.0-1_1.58.3-2_amd64.deb","sha256":"6db170195856b430d580e6ecf528b2efaf66233a98884368658fbef5abe4eaa5","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gobject-introspection/gir1.2-glib-2.0_1.58.3-2_amd64.deb","sha256":"21843844415a72bda1f3f524dfcff94550128b082f31b850f5149552ae171264","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/atk1.0/libatk1.0-data_2.30.0-2_all.deb","sha256":"cf0c94611ff2245ae31d12a5a43971eb4ca628f42e93b0e003fd2c4c0de5e533","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/atk1.0/libatk1.0-0_2.30.0-2_amd64.deb","sha256":"51603cc054baa82cee4cd50ac41578266e1321ef1c74bccbb78a3dcf1729d168","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/atk1.0/gir1.2-atk-1.0_2.30.0-2_amd64.deb","sha256":"1c606506d2f73c4f933c231f724f0cd4cf6135fabe00f369f93717ccce435eb4","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/fonts-dejavu/fonts-dejavu-core_2.37-1_all.deb","sha256":"58d21a255606191e6512cca51f32c4480e7a798945cc980623377696acfa3cfc","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/t/ttf-bitstream-vera/ttf-bitstream-vera_1.10-8_all.deb","sha256":"328def7f581bf94b3b06d21e641f3e5df9a9b2e84e93b4206bc952fe8e80f38a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/fonts-liberation/fonts-liberation_1.07.4-9_all.deb","sha256":"c936aebbfd0af7851399ae5ab08bb01744f5e3381f7678fb87cc77114f95ef53","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/c/coreutils/coreutils_8.30-3_amd64.deb","sha256":"ae6e5cd6e9aaf74d66edded3931a7a6c916625b8b890379189c75574f6856bf4","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/s/sensible-utils/sensible-utils_0.0.12_all.deb","sha256":"2043859f8bf39a20d075bf52206549f90dcabd66665bb9d6837273494fc6a598","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/u/ucf/ucf_3.0038+nmu1_all.deb","sha256":"d02a82455faab988a52121f37d97c528a4f967ed75e9398e1d8db571398c12f9","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/fontconfig/fontconfig-config_2.13.1-2_all.deb","sha256":"9f5d34ba20eb156ef62d8126866a376be985c6a83fdcfb33f12cd83acac480c2","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libp/libpng1.6/libpng16-16_1.6.36-6_amd64.deb","sha256":"82a252478465521cde9d5af473df01ed79f16e912effc5971892a574e9113500","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/freetype/libfreetype6_2.9.1-3+deb10u2_amd64.deb","sha256":"93f009440fd1ffcc4b3afdbc413eccc1d8101145a262ca0d0c305fc7029f2417","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/fontconfig/libfontconfig1_2.13.1-2_amd64.deb","sha256":"6766d0bcfc615fb15542efb5235d38237ccaec4c219beb84dbd22d1662ccea8f","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pixman/libpixman-1-0_0.36.0-1_amd64.deb","sha256":"4382ebfc5c52623d917dc0f63c22fbf7a791d00f5b303cd56a44bf9616fa5fbe","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libx11/libx11-data_1.6.7-1+deb10u1_all.deb","sha256":"02f795889390fa0e1f29c6ecdd4a30cd0aae39c0c6b1379410055404b0897c66","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxau/libxau6_1.0.8-1+b2_amd64.deb","sha256":"a7857b726c3e0d16cda2fbb9020d42e024a3160d54ef858f58578612276683e8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libb/libbsd/libbsd0_0.9.1-2_amd64.deb","sha256":"0827321e85d36200759e3ec621fc05154c752534c330ffc5472ad75bbb8eb913","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxdmcp/libxdmcp6_1.1.2-3_amd64.deb","sha256":"ecb8536f5fb34543b55bb9dc5f5b14c9dbb4150a7bddb3f2287b7cab6e9d25ef","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxcb/libxcb1_1.13.1-2_amd64.deb","sha256":"87d9ed9340dc3cb6d7ce024d2e046a659d91356863083715d2c428a32e908833","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libx11/libx11-6_1.6.7-1+deb10u1_amd64.deb","sha256":"f9d62eaa734828d4282fe4c17613c1a688af4cccbe2819eb691b3aaf615e882f","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxcb/libxcb-render0_1.13.1-2_amd64.deb","sha256":"7bd78eb3d27de76d43185d68914e938d60f233737f7a05586888072695cab6fb","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxcb/libxcb-shm0_1.13.1-2_amd64.deb","sha256":"a7a9927c9b656c253fe6f61497b94aa7332e2270cc30ca67c2925a3ecb61d742","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxext/libxext6_1.3.3-1+b2_amd64.deb","sha256":"724901105792e983bd0e7c2b46960cd925dd6a2b33b5ee999b4e80aaf624b082","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxrender/libxrender1_0.9.10-1_amd64.deb","sha256":"3ea17d07b5aa89012130e2acd92f0fc0ea67314e2f5eab6e33930ef688f48294","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/c/cairo/libcairo2_1.16.0-4+deb10u1_amd64.deb","sha256":"228e8af1f2ea388367c26a64a4491ee3f758d9e61a73f421bb60a03c07b30d2b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/c/cairo/libcairo-gobject2_1.16.0-4+deb10u1_amd64.deb","sha256":"c1e74a6852ad0b299b9197ca00ba091ef681538325d1f81680b4deb29adf094c","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gobject-introspection/gir1.2-freedesktop_1.58.3-2_amd64.deb","sha256":"d7decfe38fa1401f0ab54eb7923882316d71d537da0be3eb1438c6f06e726071","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gdk-pixbuf/libgdk-pixbuf2.0-common_2.38.1+dfsg-1_all.deb","sha256":"1310e3f0258866eb4d0e95f140d5d9025cf6be1e3e2c375f4a426ccc2e78cf68","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libj/libjpeg-turbo/libjpeg62-turbo_1.5.2-2+deb10u1_amd64.deb","sha256":"b6cbc7d722cbf697cedbcd9b8b209f8cfa05f147fba4061adf2fcee6cc64c556","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/j/jbigkit/libjbig0_2.1-3.1+b2_amd64.deb","sha256":"9646d69eefce505407bf0437ea12fb7c2d47a3fd4434720ba46b642b6dcfd80f","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libw/libwebp/libwebp6_0.6.1-2_amd64.deb","sha256":"7d9cb5e08149327731e84380e454a56f148c517ec2ecad30900c6837d0b1b76a","dest":"debs"},{"type":"file","url":"http://security.debian.org/debian-security/pool/updates/main/libz/libzstd/libzstd1_1.3.8+dfsg-3+deb10u2_amd64.deb","sha256":"3c1d6fdaeb4a2357c27b3ac7a488eee562fef2961972098be386b5618e6d20ce","dest":"debs"},{"type":"file","url":"http://security.debian.org/debian-security/pool/updates/main/t/tiff/libtiff5_4.1.0+git191117-2~deb10u2_amd64.deb","sha256":"fc8ee9e6b088cf408c9234e707ed8aa18cbe1e811e93c8a63744bf65be1f7bc9","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gcc-8/libstdc++6_8.3.0-6_amd64.deb","sha256":"5cc70625329655ff9382580971d4616db8aa39af958b7c995ee84598f142a4ee","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/i/icu/libicu63_63.1-6+deb10u1_amd64.deb","sha256":"603f929d1ae548a8faa1f892ae93e623bde97de7ddbe4a796618c735ca7ff8b8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxml2/libxml2_2.9.4+dfsg1-7+deb10u1_amd64.deb","sha256":"86b720e28f05e862a86ce3270b080fbbb9b2e8e770f27ae18f9ef300ab6c5c5b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/s/shared-mime-info/shared-mime-info_1.10-1_amd64.deb","sha256":"6a19f62c59788ba3a52c8b08750a263edde89ac98e63c7e4ccfb14b40eafaf51","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gdk-pixbuf/libgdk-pixbuf2.0-0_2.38.1+dfsg-1_amd64.deb","sha256":"90e1842771968ffae4b4c28f1ad6a8bf77ff3a57616b799abed93354b860edc8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gdk-pixbuf/gir1.2-gdkpixbuf-2.0_2.38.1+dfsg-1_amd64.deb","sha256":"64d9e66e272541f790eafeb2a3ddea9868f9e7987e048d290f323cce3aaaa9ed","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/fontconfig/fontconfig_2.13.1-2_amd64.deb","sha256":"efbc7d9a8cf245e31429d3bda3e560df275f6b7302367aabe83503ca734ac0fd","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/f/fribidi/libfribidi0_1.0.5-3.1+deb10u1_amd64.deb","sha256":"9844b02a3bfa8c9f89a077cc5208122f9245a6a6301cbf5fdc66b1a76f163c08","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libd/libdatrie/libdatrie1_0.2.12-2_amd64.deb","sha256":"7159a08f4a40f74e4582ebd62db0fb48b3ba8e592655ac2ab44f7bfacbca12f3","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libt/libthai/libthai-data_0.1.28-2_all.deb","sha256":"267d6b251f77c17fb1415ac0727675cb978c895cc1c77d7540e7133125614366","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libt/libthai/libthai0_0.1.28-2_amd64.deb","sha256":"40e7fbd1ed27185879836b43fb8a739c8991a6d589fef9fb2b3b63e188a537ae","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pango1.0/libpango-1.0-0_1.42.4-8~deb10u1_amd64.deb","sha256":"efd677c77cb5e89dd94a6f981c7dd4e705e393b61ba4fd3610009d2ca140fa11","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/graphite2/libgraphite2-3_1.3.13-7_amd64.deb","sha256":"f79bfdcfe09285cccee68c070171888b98adbf3e7bd3e8f6afcb6caef5623179","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/h/harfbuzz/libharfbuzz0b_2.3.1-1_amd64.deb","sha256":"aee1dd6f9884c1acdd1b6d6f49bd419235decd00f49cd927e4be4c37af2ecdab","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pango1.0/libpangoft2-1.0-0_1.42.4-8~deb10u1_amd64.deb","sha256":"4724bb2dfa25beb9c1a978de0c5780f6d3ce5aa03232e47c11b297b4899a6073","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pango1.0/libpangocairo-1.0-0_1.42.4-8~deb10u1_amd64.deb","sha256":"3ca2e8aaa6c061f91ae05163ce31b3bc63a611c63c1a43ccf855330e505825f0","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/x/xft/libxft2_2.3.2-2_amd64.deb","sha256":"cd71384b4d511cba69bcee29af326943c7ca12450765f44c40d246608c779aad","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pango1.0/libpangoxft-1.0-0_1.42.4-8~deb10u1_amd64.deb","sha256":"a50fdb33d39ea966288308a32a12cd434f30bf0b7c45bd33a764268250677066","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pango1.0/gir1.2-pango-1.0_1.42.4-8~deb10u1_amd64.deb","sha256":"36e4f995659cc24f73104042736e7a20a3d2c353385b2e702db44dfb804e5d32","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtk+3.0/gtk-update-icon-cache_3.24.5-1_amd64.deb","sha256":"ca87a8eaa7a662049e2a95f3405d8affb7715a9dbdcba6fa186ae0bcc8981847","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/h/hicolor-icon-theme/hicolor-icon-theme_0.17-2_all.deb","sha256":"20304d34b85a734ec1e4830badf3a3a70a5dc5f9c1afc0b2230ecd760c81b5e0","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libc/libcroco/libcroco3_0.6.12-3_amd64.deb","sha256":"1acb00996b7477687e4f3f12de7fbf4b635866a6167671f2201ea3e67af05336","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libr/librsvg/librsvg2-2_2.44.10-2.1_amd64.deb","sha256":"181188485d646e0ac29e79df67d8fa3ca7a984bb65024b06b36e917b4e282e21","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libr/librsvg/librsvg2-common_2.44.10-2.1_amd64.deb","sha256":"c873d99436da50dfcc23104d827bd73e5063d9ee5742f39ffeb44ba1145af5e1","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/adwaita-icon-theme/adwaita-icon-theme_3.30.1-1_all.deb","sha256":"698b3f0fa337bb36ea4fe072a37a32a1c81875db13042368677490bb087ccb93","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libg/libgpg-error/libgpg-error0_1.35-1_amd64.deb","sha256":"996b67baf6b5c6fda0db2df27cce15701b122403d0a7f30e9a1f50d07205450a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libg/libgcrypt20/libgcrypt20_1.8.4-5_amd64.deb","sha256":"1b826517b328e29a441cc89e5c427896182ffc946713329f50accc8417856136","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/l/lz4/liblz4-1_1.8.3-1_amd64.deb","sha256":"826203ecea7e8cb87aebfbb7bd2afc9f7e519f4c0f578c0404e21416572d1005","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/s/systemd/libsystemd0_241-7~deb10u6_amd64.deb","sha256":"94d76153b6417fafb142103b794b28c53ad640673c7e742b13bd1bb321099049","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/dbus/libdbus-1-3_1.12.20-0+deb10u1_amd64.deb","sha256":"e394bd35626e3ccf437e1e7776e6573636e6413b0ebe2483bd54ac243eed1007","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/at-spi2-core/libatspi2.0-0_2.30.0-7_amd64.deb","sha256":"8ff1ab1508799679e3209188c5f4765c3da16a7876e220c49d805ef03cced397","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/at-spi2-atk/libatk-bridge2.0-0_2.30.0-5_amd64.deb","sha256":"52ed3333fd0e1430b573343fc65d594a075ee5f493b8cbff0f64d5f41f6f3f8f","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/l/lcms2/liblcms2-2_2.9-3_amd64.deb","sha256":"6dd806a326519b98ed9e54b184b4da2d256c4d516e75d0a38f2f6059e14eb325","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/s/systemd/libudev1_241-7~deb10u6_amd64.deb","sha256":"a9bff5fb4ecd3dc429a79831daf33b5ee027bf13ed832e169ea6b770a5b333d6","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/c/colord/libcolord2_1.4.3-4_amd64.deb","sha256":"2fd78fc761cc8465702ce4ec03bc6922b172e47f524c7c64312dcf2ad0db1489","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/avahi/libavahi-common-data_0.7-4+b1_amd64.deb","sha256":"388bc38faf2ad715b9c383daa7c358c656c05de33886c2da7e3641db6bdf5512","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/avahi/libavahi-common3_0.7-4+b1_amd64.deb","sha256":"82ee379d69de764c2fc1663535e43fabad7e31247a6fae7d492b5a0446a730c3","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/avahi/libavahi-client3_0.7-4+b1_amd64.deb","sha256":"ab80126c56e9aa8fd3b2ef5991b547bd15eb41cd78919fa55c55d221a9d349e9","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gmp/libgmp10_6.1.2+dfsg-4_amd64.deb","sha256":"d9c9661c7d4d686a82c29d183124adacbefff797f1ef5723d509dbaa2e92a87c","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/n/nettle/libnettle6_3.4.1-1_amd64.deb","sha256":"5a384c773ae68b0c7905ecc0abf5e45925794b679674866d7783d88786ffb0d2","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/n/nettle/libhogweed4_3.4.1-1_amd64.deb","sha256":"a938f22b6cead37c5f980a59330e71e2df1df4af890ea6b3432485c0da96ea58","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libu/libunistring/libunistring2_0.9.10-1_amd64.deb","sha256":"bc3961271c9f78e7ef93dec3bf7c1047f2cde73dfc3e2b0c475b6115b76780f8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libi/libidn2/libidn2-0_2.0.5-1+deb10u1_amd64.deb","sha256":"13c3129c4930cd8b1255dbc5da7068c036f217218d1017634b83847a659fad16","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/p11-kit/libp11-kit0_0.23.15-2+deb10u1_amd64.deb","sha256":"02b2f15ad9cd2ead38dbeb85cdba65d8e4d44155495110231504cd0de1f16f83","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libt/libtasn1-6/libtasn1-6_4.13-3_amd64.deb","sha256":"2771ea1ba49d30f033e67e708f71da9b031649c7c13d2ce04cb3ec913ac3b839","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gnutls28/libgnutls30_3.6.7-4+deb10u6_amd64.deb","sha256":"7f9096293466c2f69c83566175e34b4af4a64096d7de5bc96a65cfabe47aabe8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/e/e2fsprogs/libcom-err2_1.44.5-1+deb10u3_amd64.deb","sha256":"e5ea8e6db9453ed13199f4cbfe8e29d76c579eb6f678ab9bb4bebd7d12c1936e","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/k/keyutils/libkeyutils1_1.6-6_amd64.deb","sha256":"0c199af9431db289ba5b34a4f21e30a4f1b6c5305203da9298096fce1cdcdb97","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/k/krb5/libkrb5support0_1.17-3+deb10u1_amd64.deb","sha256":"2d90547b3509aae6cdedbad35661d0cac362490cb9dd74eb9d0891c7ca8aee43","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/k/krb5/libk5crypto3_1.17-3+deb10u1_amd64.deb","sha256":"6c7105878258ef2c05966e42bfdfffa8a339901a208192d4b0387b7b32a27c75","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/k/krb5/libkrb5-3_1.17-3+deb10u1_amd64.deb","sha256":"9e758deb40cc7e421527e0fcee7598a7eefe625bc591f464ea014a0591920373","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/k/krb5/libgssapi-krb5-2_1.17-3+deb10u1_amd64.deb","sha256":"e3ee98ec6cae4ee207fdfda18f77e05d558840e680af252056a02952b2d71fc8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/c/cups/libcups2_2.2.10-6+deb10u4_amd64.deb","sha256":"62ea6f113f85d575fa293904cd48c4b644a616225a52c124c1e8e8e4810a20d6","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libe/libepoxy/libepoxy0_1.5.3-0.1_amd64.deb","sha256":"968295ae7382be0fc06e535f2a1408f54b0b29096e0142618d185da1c7a42ed0","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtk+3.0/libgtk-3-common_3.24.5-1_all.deb","sha256":"1e1c979ec882542ce09b40c0f7246a7f348b42d9bec6f31eb2614a8ddccd4874","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/j/json-glib/libjson-glib-1.0-common_1.4.4-2_all.deb","sha256":"c27dbb0cf9c73e2a09d5c774fb46ecf6d2b634facaf3b37b20a4654d9c549187","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/j/json-glib/libjson-glib-1.0-0_1.4.4-2_amd64.deb","sha256":"58f872df6bc521a7ef4990c2a4b3264b1a1fab15440297a7e92ef88067e308ed","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/glib-networking/glib-networking-common_2.58.0-2+deb10u2_all.deb","sha256":"ef84896340440b25871c1e3f1d102837290db24382ad40ac74c769a70b1203be","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libp/libproxy/libproxy1v5_0.4.15-5+deb10u1_amd64.deb","sha256":"4dc8ecc24bcc164169f593ab1c387eacea383e3da8c46d5c0d5877fe7da1b967","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/glib-networking/glib-networking-services_2.58.0-2+deb10u2_amd64.deb","sha256":"d4e2ec66aeb3e4a35d80710c69eebdb949bc4731d4b77e3beb1447988f554dbd","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gsettings-desktop-schemas/gsettings-desktop-schemas_3.28.1-1_all.deb","sha256":"a75aed8781a781c4b819b2d1e952791b123580b1a02a4bb35fdbbba2e3ab8310","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/glib-networking/glib-networking_2.58.0-2+deb10u2_amd64.deb","sha256":"6906c9748b7f5f909649b20bf97496aaddfe507c8bc146dc96310cfad3bc8fbb","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libp/libpsl/libpsl5_0.20.2-2_amd64.deb","sha256":"290fc88e99d21586164d51f8562c3b4c6a3bfabdbb626d91b6541896d76a582b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libs/libsoup2.4/libsoup2.4-1_2.64.2-2_amd64.deb","sha256":"db9918e3937eb4f92068665a9b42ea33b0860da602fa5c2f0e80e5cb15a556c4","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libs/libsoup2.4/libsoup-gnome2.4-1_2.64.2-2_amd64.deb","sha256":"33c571659e0fe2ba55214d2c68b15d883215c6c0e08e6037173da92585f9a623","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libr/librest/librest-0.7-0_0.8.1-1_amd64.deb","sha256":"17d25479dd8fb0bfc7fd92ca92d7c063e9d0a22f43cb90e2de243b89111cde93","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/w/wayland/libwayland-client0_1.16.0-1_amd64.deb","sha256":"826fdd1a6a5ffa01415f138e238da858aae22ac4f4835cedfecab76dd0dcb01b","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/w/wayland/libwayland-cursor0_1.16.0-1_amd64.deb","sha256":"eee990ea0ad68ac409986ebf92106b8deada54cc2cfd19293177f5f938c35690","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/w/wayland/libwayland-egl1_1.16.0-1_amd64.deb","sha256":"a021e9aa9a92270fa259211a0ca69b5e8428f32c6e800a4a93f0766b0a48a5c6","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxcomposite/libxcomposite1_0.4.4-2_amd64.deb","sha256":"043c878356954f4521c401b160d554809115c472ca384d9f793c1c7542316eb9","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxfixes/libxfixes3_5.0.3-1_amd64.deb","sha256":"3b307490c669accd52dc627ad4dc269a03632ca512fbc7b185b572f76608ff4e","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxcursor/libxcursor1_1.1.15-2_amd64.deb","sha256":"5c5c3c5020b3e963afcf45af21ad8c0c14375ae35f6c649a05a22790503bf24c","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxdamage/libxdamage1_1.1.4-3+b3_amd64.deb","sha256":"e9539838d47cb10b4273c320f8e885ef85df7bd3a95f0ea9bcbc144db82c03ae","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxi/libxi6_1.7.9-1_amd64.deb","sha256":"fe26733adf2025f184bf904caf088a5d3f6aa29a8863b616af9cafaad85b1237","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxinerama/libxinerama1_1.1.4-2_amd64.deb","sha256":"f692c854935571ee44fe313541d8a9f678a4f11dc513bc43b9d0a501c6dff0bd","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/x/xkeyboard-config/xkb-data_2.26-2_all.deb","sha256":"17d21564c940dd8d89e0a1b69d6fea0144d057e4698902378f5c83500612b779","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxkbcommon/libxkbcommon0_0.8.2-1_amd64.deb","sha256":"a93729f1d325598ad9c6a7ffe00c464fbe276181a3a124855041c1e303175f0c","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libx/libxrandr/libxrandr2_1.5.1-1_amd64.deb","sha256":"8fdd8ba4a8ad819731d6bbd903b52851a2ec2f9ef4139d880e9be421ea61338c","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtk+3.0/libgtk-3-0_3.24.5-1_amd64.deb","sha256":"e652e04b04cc8a67c24c5773180a7fdd65a6cfc55a2777722e80825a56a33729","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtk+3.0/gir1.2-gtk-3.0_3.24.5-1_amd64.deb","sha256":"e5168d6d46991d26ff101923f18953a51d34e8f14aa904f12680910a831c3ca6","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtksourceview3/libgtksourceview-3.0-common_3.24.9-2_all.deb","sha256":"b030014065aee4bfd61d550e64eede40bf3ec3e1aa3e3c69b7a9ae04f6c998e2","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtksourceview3/libgtksourceview-3.0-1_3.24.9-2_amd64.deb","sha256":"0711617abfbb2438ee09f9b6040cf30c0fb9a66ab65a2a4f070e2d5cc8b7b6a3","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gtksourceview3/gir1.2-gtksource-3.0_3.24.9-2_amd64.deb","sha256":"351731e69b74b3f290599739ca191f37cc4fbab04dd2b3198563c692bc132fc9","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libp/libpeas/libpeas-common_1.22.0-4_all.deb","sha256":"9d2454567e6cbf1f7ad5234f1a9c0cc830e288fd34ea0deba75190b2b257a99a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/python3.7/libpython3.7_3.7.3-2+deb10u2_amd64.deb","sha256":"94c34bee59dfd4f21a249d0a8efb999dbeff5d2b63a172b10b7548c22be93073","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libp/libpeas/libpeas-1.0-0_1.22.0-4_amd64.deb","sha256":"13577818e6b04d4c586425c500a5cc93d280ff1fe0d27ef3fe958b5f132c8f39","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libp/libpeas/gir1.2-peas-1.0_1.22.0-4_amd64.deb","sha256":"729f9f8738f571746944624218be4fc34a606b279f6df2b0ff911c59f5a94f1f","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/i/iso-codes/iso-codes_4.2-1_all.deb","sha256":"72d0589fc065005574232bf103c7d34acf5e4c805f2b0d19e6aeb639438ff253","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/e/emacsen-common/emacsen-common_3.0.4_all.deb","sha256":"7b7533b27f0421177993c7835b6bd8fcdb67b9a48223964d5134639af3b4ae8a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/libt/libtext-iconv-perl/libtext-iconv-perl_1.7-5+b7_amd64.deb","sha256":"c9193a375cbe39a4c00980213a9a169a4dd06802e412b36d9176e0104023afa6","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/d/dictionaries-common/dictionaries-common_1.28.1_all.deb","sha256":"bb0a56551b282dddf2e72b08d1500ee8490cbdcabcbb6f54972f852a3e4813af","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/aspell/libaspell15_0.60.7~20110707-6_amd64.deb","sha256":"9a569b91a33ebd569d37c93dd65879c05c57c062f30a0e5399e658709a4342a8","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/aspell/aspell_0.60.7~20110707-6_amd64.deb","sha256":"c1f2ca2479928e4897382335f42a8a7ddd8dd767f525bb24c0739631042840a0","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/a/aspell-en/aspell-en_2018.04.16-0-1_all.deb","sha256":"22bf952370705a762564fb1a8d32250bd580cb9769c05b01a2fd99b5bc1b00af","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/h/hunspell/libhunspell-1.7-0_1.7.0-2_amd64.deb","sha256":"2e720b31a0d28bb6d55f6cce777719d33a171eaba5098c8f878dc16542d6d194","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/e/enchant/libenchant1c2a_1.6.0-11.1+b1_amd64.deb","sha256":"42916be9a85eb409e210d2bfc58cd48a07968a7e303ad4b7d9f462aec4a2dc7c","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gspell/libgspell-1-common_1.6.1-2_all.deb","sha256":"0d7ed2ae8bd35128ac33721ba6051fffc83f4b2da9d460dd92a83b7c50bb4c61","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gspell/libgspell-1-1_1.6.1-2_amd64.deb","sha256":"857f95309ecbb20993110153d6973a91201d5d45433fbe292f8f58650c7ee1d7","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pygobject/python3-gi_3.30.4-1_amd64.deb","sha256":"539d7f81bc80152194b2dccdc4559ccf637eb692b260e0e522bb77240f30fd3a","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pycairo/python3-cairo_1.16.2-1+b1_amd64.deb","sha256":"f0a9601192c5180d1664606cfe894eacd572deabe29230f6c9c2d8feb2247e84","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/p/pygobject/python3-gi-cairo_3.30.4-1_amd64.deb","sha256":"fe855b275c76e58b18f3150d7cd8881b123627acf9e91e58528a175eb6f6d58d","dest":"debs"},{"type":"file","url":"http://ftp.us.debian.org/debian/pool/main/g/gedit/gedit_3.30.2-2_amd64.deb","sha256":"39a55ce1a39d73d0f090f865bb0f9be9f78369c785f823973959c44b51ae9465","dest":"debs"}]}rizvan@debian:~/build/debpak$ 
rizvan@debian:~/build/debpak$ ls
debpak  README.md  scraper.go

How can you get the raw html for a node?

I would like to the the full html data for a node, is there a way to do it without iterating through all nodes and reconstructing it?

XPath support

Hi,

Chromium has a feature where you can right-click an element and copy the XPath.

If scrape would support XPath, it would be amazingly easy to copy the XPath from the desired page and use that in Go to retrieve that element.

Quick overview of XPath.

Cheers!

add more examples

My 2 cents:

https://github.com/ezeql/gin-scrape-example

( Enhancement | Suggestion ) Maybe using gobuilder.me to build packages for various OS's.

I submitted it to gobuilder.me and it finished the build in a few seconds.

The result is packages for nearly every used operating system, ready to go.

Suggestions:

Maybe a webhook, to automatically build it when a new commit occurs
.gobuilder.yml file to customize the build process.

More information on both of them is available here.

I am not affiliated, nor promoting Gobuilder for comercial purposes. 😉

Congratulations on getting onto the front page of Hacker News :).

How to find nested element within a class

I have following node:


<ul class="list clearfix">
 <li> 
     <div>
          <a href=www.example.com/asda">
                     <img src="..sadsada./ssa/3.jpg">
         </a>
      </div>
   </li>
 <li> 
     <div>
          <a href=www.example.comsdsds/sds">
                     <img srr="..sadsada./ssa/2.jpg">
         </a>
      </div>
   </li>
 <li> 
     <div>
          <a href=www.example.com/sdds">
                     <img src="..sadsada./ssa/1.jpg">
         </a>
      </div>
   </li>
</ul>

How do I get all the img src?

matcher := func(n *html.Node) bool {

		if n.DataAtom == atom.A && n.Parent != nil && n.Parent.Parent != nil && n.Parent.Parent.Parent != nil && n.Parent.Parent.Parent.Parent != nil {

			return scrape.Attr(n.Parent.Parent.Parent.Parent, "class") == "list clearfix"
		}
		return false
	}

	images := scrape.FindAll(root, matcher)

Doesn't work for me.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.