heq is a command-line tool for extracting structured data as JSON from HTML using concise expressions, akin to jq. Additionally, heq serves as a Python library, facilitating the efficient scraping of HTML content through its jq-inspired DSL based on XPath and CSS selectors.
pip install heqheq depends on the following Python packages:
- lxml
- parsimonious
$ curl -s https://news.ycombinator.com/ | heq '$`tr.athing` / {
title: `(./td[@class="title"]//a)[1]`.text,
link: `(./td[@class="title"]//a)[1]`@href,
}'
[
{
"title": "Teachable Machine",
"link": "https://..."
},
{
"title": "They correctly predicted a Nobel Prize winning discovery. And no one cared [video]",
"link": "https://..."
},
...
]
$ cat << 'EOF' | heq '$`div.product` / {name: $`h2.name`.text}'
<body>
<div id="header">Welcome to Our Store!</div>
<div class="product">
<h2 class="name">Widget A</h2>
<p class="price">$10</p>
<ul class="features"><li>Durable</li><li>Lightweight</li></ul>
<a href="/products/widget_a">Details</a>
</div>
<div class="product">
<h2 class="name">Gadget B</h2>
<p class="price">$20</p>
<ul class="features"><li>Compact</li><li>Energy Efficient</li></ul>
<a href="/products/gadget_b">Details</a>
</div>
</body>
EOFOutput:
[
{"name": "Widget A"},
{"name": "Gadget B"}
]$ cat expr.heq
$`div.product` / {
name: $`.name`.text,
price: $`.price`.text,
features: $`li` / text,
url: $`a`@href
}
$ cat << 'EOF' | heq -f expr.heq
(The same HTML as above)
EOFOutput:
[
{
"name": "Widget A",
"price": "$10",
"features": ["Durable", "Lightweight"],
"url": "/products/widget_a"
},
{
"name": "Gadget B",
"price": "$20",
"features": ["Compact", "Energy Efficient"],
"url": "/products/gadget_b"
}
]from heq import extract, xpath
html = '''<body>
<div class="product">
<h2 class="name">Widget A</h2>
<p class="price">$10</p>
<ul class="features"><li>Durable</li><li>Lightweight</li></ul>
</div>
<div class="product">
<h2 class="name">Gadget B</h2>
<p class="price">$20</p>
<ul class="features"><li>Compact</li><li>Energy Efficient</li></ul>
</div>
</body>'''
expr = xpath("//div[@class='product']") / {
'name': xpath(".//h2[@class='name']").text,
'price': xpath(".//p[@class='price']").text,
'features': xpath(".//li") / {
'feature': xpath('.').text
}
}
print(extract(expr, html))Output:
[{'name': 'Widget A',
'price': '$10',
'features': [{'feature': 'Durable'}, {'feature': 'Lightweight'}]},
{'name': 'Gadget B',
'price': '$20',
'features': [{'feature': 'Compact'}, {'feature': 'Energy Efficient'}]}]
<S> ::= <expr>
<expr> ::= <selector_lit> '/' <term>
| <term>
<term> ::= <dict_lit> | <dottext> | <atattr> | <filter>
<filter> ::= 'text' | <attr_lit>
<dict_lit> ::= '{' ((<dict_field_value> ',')* <dict_field_value>)? '}'
<dict_field_value> ::= <dict_field> ':' <expr>
<dottext> ::= <selector_lit> '.text'
<atattr> ::= <selector_lit> <attr_lit>
<selector_lit> ::= <css_lit> / <xpath_lit>
<css_lit> ::= '$' <backtick_lit>
<xpath_lit> ::= <backtick_lit>
<attr_lit> ::= '@' <ident_with_hyphen>
heq has the concept of context DOM tree. This is the DOM tree against which XPath expressions or CSS selectors are evaluated. Initially, it is set to the root tree, and it changes as the / operator is applied, to each of the elements.
Available syntactic constructs and their semantics are as follows:
- Value Forms
{key: expression}: Evaluates to a dictionary.keyis a string without quotes andexpressionis an expression.text: Evaluates to a string representing the text content of the context DOM tree.@attr: Evaluates to the value associated with the attributeattrof the context DOM tree.<selector>.text: Evaluates to a string representing the text content of the element(s) selected by the specified selector.<selector>@attr: Evaluates to a string representing the value associated with the attributeattrof the first element selected by the specified XPath expression.
- Selectors
`<xpath>`: Selects elements by evaluating the XPath against the context DOM tree.$`<css_selector>`: Selects elements by evaluating the CSS selector against the context DOM tree.
- Mapping Against Query Results
<selector> / <value_form>: First, evaluates the selector to obtain a list of elements. Then, for each element, thevalue_formis evaluated with the element as the new context DOM tree. The entire expression evaluates to an array.
<body>
<div id="header">Welcome to Our Store!</div>
<div class="product">
<h2 class="name">Widget A</h2>
<p class="price">$10</p>
<ul class="features"><li>Durable</li><li>Lightweight</li></ul>
<a href="/products/widget_a">Details</a>
</div>
<div class="product">
<h2 class="name">Gadget B</h2>
<p class="price">$20</p>
<ul class="features"><li>Compact</li><li>Energy Efficient</li></ul>
<a href="/products/gadget_b">Details</a>
</div>
</body>All example expressions below use the HTML above for their evaluations.
{ header: `//div[@id="header"]`.text }
evaluates to:
{ "header": "Welcome to Our Store!" }Expression Breakdown:
`//div[@id="header"]`: This XPath literal selects the div element withid="header"..text: This part extracts the text content of the selected element.- The entire expression is wrapped in
{}to create a JSON object with"header"as the key.
`//div[@id="header"]`.text
evaluates to:
"Welcome to Our Store!"`//div[@class="product"]` / `.//a`@href
evaluates to:
["/products/widget_a", "/products/gadget_b"]Expression Breakdown:
//div[@class="product"]: Selects all div elements withclass="product"./: This operator is used to apply the following expression to each selected element.`.//a`@href: For each div, this extracts the href attribute of the first a element found within.
`//div[@class="product"]` / {
name: `.//h2[@class="name"]`.text,
price: `.//p[@class="price"]`.text,
features: `.//li` / text,
url: `.//a`@href
}
evaluates to:
[
{
"name": "Widget A",
"price": "$10",
"features": ["Durable", "Lightweight"],
"url": "/products/widget_a"
},
{
"name": "Gadget B",
"price": "$20",
"features": ["Compact", "Energy Efficient"],
"url": "/products/gadget_b"
}
]$`div.product` / {
name: $`h2.name`.text,
price: $`p.price`.text,
features: $`li` / text,
url: $`a`@href
}
evaluates to:
[
{
"name": "Widget A",
"price": "$10",
"features": ["Durable", "Lightweight"],
"url": "/products/widget_a"
},
{
"name": "Gadget B",
"price": "$20",
"features": ["Compact", "Energy Efficient"],
"url": "/products/gadget_b"
}
]This example performs the same extraction as the previous example, but using CSS selectors in place of XPath expressions.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent