mozilla / fathom Goto Github PK
View Code? Open in Web Editor NEWA framework for extracting meaning from web pages
Home Page: http://mozilla.github.io/fathom/
License: Mozilla Public License 2.0
A framework for extracting meaning from web pages
Home Page: http://mozilla.github.io/fathom/
License: Mozilla Public License 2.0
There's really no need to write code against the Corpus Framework anymore. Training in-browser gives better performance and stronger sources of signal with less setup.
…for symmetry with facts.get('smoo').
Implement clustering on DOM nodes. A cluster of paragraphs might be the body text of an article. A cluster of links might be a nav element.
It is possible to define rulesets which run very slowly on some webpages.
This currently causes performance problems in Firefox (I'm currently testing on 58).
I think we should be able to mitigate this problem if Fathom supported a way to timeout ruleset execution after X ms.
In my head, I'd like something like a callback system where I can pass in a function that accepts the current runtime in ms of the ruleset. Returning a non-zero value should cause the fathom ruleset to abort early.
Is it possible to use this for EPUB
files? The EPUB format is a subset of XHTML, zipped and with a bit of metadata thrown in. I am trying to strip out unnecessary content e.g. copyright pages, content pages etc. for conversion of ebooks to audiobooks (where such pages are redundant), is this a suitable use case?
Once https://bugzilla.mozilla.org/show_bug.cgi?id=1398734 is done, we'll be able to move our labeling UI from its current home in a separate Fathom devtools panel to a subpane of the Inspector panel. This will greatly speed the labeling experience, relieving people of pogo-sticking between the two panes.
If we want to make arbitrary logical expressions on LHSs go fast, this is one way. Rete trees are a sort of generalized trie in which LHS conditions are the trie nodes, getting more specific as you go away from the root. Then fnodes get tagged onto the trie nodes where they match, so, when executing a rule, you don't need to look aside to any indices or anything. It trades memory for speed.
Just saw the lightning talk at #MozAloha - very cool!
Is it possible to write Fathom rules to detect nested class names? Perhaps even with additional CSS Selector constraints?
Would it be possible to add a simple example rule set for parsing data from nested elements with particular class names?
For example, I'd like to write rules like the nested class name rules for microformats2 parsing:
http://microformats.org/wiki/microformats2-parsing#parse_an_element_for_class_microformats
Or at least rules like:
.h-entry .p-name -> title = textContent
Happy to help get something simple working and iterate from there.
Not everything has a toString() yet: rules in particular.
We use jsdom for our test cases (and, by accident, effectively recommend it for people who need to feed fathom a string rather than a predisgested DOM object). But jsdom is pretty slow, and the DOM API itself is not great. cheerio has been requested by one group inside Mozilla. There's also domjs and domino. Write pro/con lists and choose one. Here's a starting set of goals. We can edit it as needed.
Obviously, that name is ridiculously unwieldy, but the need stands. We'll want to scale the emitted score of a rule by a constant, automatically, while training (and perhaps in the final, tuned ruleset). Right now, this could be done by interposing a type, but it will probably prove pragmatic to let us do it more concisely. We could either add this complication to the runtime or let the compilation pass interpose a made-up type implicitly.
I'm trying to play w/ <meta name="keywords"/>
and using open graph tags... Does Fathom support returning multiple matching elements?
For example, can I return all matching meta[property='og:video:tag']
elements?
<meta name="keywords" content=""double dream hands" "john jacobson" music dance fun "sprint guy"">
...
<meta property="og:video:tag" content="double dream hands">
<meta property="og:video:tag" content="john jacobson">
<meta property="og:video:tag" content="music">
<meta property="og:video:tag" content="dance">
<meta property="og:video:tag" content="fun">
<meta property="og:video:tag" content="sprint guy">
Plus http://ogp.me/#no_vertical
Ultimately, I think we want to return an array of keywords (either from meta keywords, or og:article:tag
, og:book:tag
, or og:video:tag
).
The other use case would be potentially returning an array of og:image
tags, per http://ogp.me/#array. (Currently ignoring the problem of trying to extract adjacent og:image
and optional og:image:width
and og:image:height
elements):
<meta property="og:image" content="http://example.com/rock.jpg" />
<meta property="og:image:width" content="300" />
<meta property="og:image:height" content="300" />
<meta property="og:image" content="http://example.com/rock2.jpg" />
<meta property="og:image" content="http://example.com/rock3.jpg" />
<meta property="og:image:height" content="1000" />
Still early days, but it'd be nice to see some code coverage numbers and a "shame badge" on the README (especially if we're going to start trying to use this for Activity Stream).
We need node.js 6.0 or better for ES6 support.
Takes a predicate that can further specify which nodes to select
Example:
type(‘priceish’).when(fnode => fnode.element.tagName.len > 5)
Maintain an array of predicates in the higher level Lhs class. when function in Lhs adds predicate to the array. Other Lhs subclasses check that all predicates in the array are satisfied before returning the element.
It's shorter and more accurate.
This will doubtless challenge and inform our design. A "should-work" sketch is already in test.js.
Add a compilation pass for rulesets. This will bring several improvements:
I see us inserting a CompiledRuleset class chronologically between Ruleset and BoundRuleset. Callers can compile a ruleset manually if they wish, or they can just call Ruleset.against() as now, and the compile will happen implicitly (and then get thrown away).
Clustering has a lot of hard-coded coefficients at the moment: the all-caps ones in fathom.utils.distance(). I picked those based on contrived test cases, not real-world examples, so it is likely that users will want to tweak them. (We may also change the default ones, in which case users would need a way to lock in values they like.) Design a way for callers of clusters() to specify their own coefficient values to customize the behavior of distance measurement.
We need a proper renderer in order to take advantage of data like element position.
make debugtest
target if necessary)Partway through implementing #18, I noticed that it takes more than simply HTML to reconstitute the geometry of a page. That is, a mere headless browser does not suffice to deliver the additional signal we wanted. Let's choose a solution.
See https://github.com/mozilla/fathom/wiki/PageGeometryCaptureSolutions for a more editable, history-tracking place to do it.
Over in fathom-trainees there's a recommendation to return not-quite-0 and not-quite 1 values from rules to accommodate the trainer. Fathom should probably be exporting these as MIN_SCORE
and MAX_SCORE
constants for rulesets to use so that they don't have to copy the values themselves.
Fathom would like "metarules": the ability to turn certain rules on or off based on [yet-to-be-determined criteria, likely including URL].
(Until then, you can do it manually by just writing a little JS code to affect which rules get passed into the ruleset. Rules are unordered in Fathom, so you can just toss things in in any order the logic deems convenient. But it would be great to make such imperative ruleset-building unneeded.)
This could be really useful if you can extract names, proper nouns with a bit of NLP. Also email addresses, social media accounts. Are any of these items on the road map?
I'd like to remove the verbose scoreUpTo and getScore and say max (or at least maxScore) and score instead, but I don't want to confuse people. If max(8) on the LHS means "the top 8" and max(8) on the RHS means "score at most 8", that might be confusing.
What about max(8) on the LHS meaning "no score over 8" (which we don't have to implement until we need it) and top(8) on the LHS meaning "the top 8"? The other semantic we'll need is over(8).
under(8) on the rhs might be a nice symmetry with over(5) on the lhs.
How does Chrome identify what's a distraction and what's not? Summarize on a new ChromeDistractionFreeBrowsing page.
Something like type('smoo').biggestCluster() on the LHS which emits the most populous cluster of smoos. We ought to be able to have it tolerate the occasional low-scoring sibling by messing with the stride-node cost coefficient.
And/or we could have one the emits the max-scoring cluster, by some metric.
This isn't gospel; please apply design thought.
Update Readbility with the changes Mozilla made to Arc90 Labs' original work.
Once we can surface the min/max of a ranker declaratively, we can have the executor work out the most efficient execution path. This will make rulesets like Jared's title-finding one as efficient as a hard-coded approach. For example, we can execute the highest-possible-scoring rules first and, if they succeed and emit score S, we don't need to execute the rules whose max is lower than S and who emit the same type. Of course, this assumes we care about only the max of a set of rules, so we'll want to bring yankers into the ruleset and make them declarative so the global optimizer can know when such an assumption is valid. Consequently, we'll stop returning the entire knowledgebase and start returning instead a map of yankers to their output. For example (and don't pay attention to the syntax)…
ruleset:
...
type(blah) -> {}
...
yank("maxBlah", max(blah)),
yank("cf", cluster(foo)), // You can add more yankers to a ruleset later by just appending them. Later yankers of the same name override previous ones.
yank("bar", all(bar))
That will emit something like…
{ maxBlah => a fathom node (fnode),
cf => a bunch of fnodes or maybe a cluster object,
bar => all fnodes of type "bar" so you can do your own yanking after the fact }
I'd like to be able to keep API docs in one place—in the source code—but still pull them out into HTML docs that slap more prose and context around them. I would like to avoid the common antipattern where the docs consist of an alphabetical list of routines, leaving the poor newcomer to wander linearly through them, trying to piece together an overview. Add contenders below.
Sphinx integration is nice because Sphinx is amazing, offering freeform docs in restructured text, indexing and glossary building, a good markup language with arbitrary nesting of constructs parsed in an unambiguous way, output of numerous formats, a decade plus of maturity, and the fact that I know it like the back of my hand.
The nice thing about this is that some JS tooling can consume it, like editors and Google's Closure Compiler (which uses type annotations to hint to the optimizer).
Perhaps this should wait until we have a concrete use case, but being able to AND types together on the left-hand side of rules would make the language much more powerful. ORs could happen, too, but they're already expressible by simply adding a second rule with the same right-hand side.
Did OmniWeb attempt to identify the main content of a page? If so, how? Make an OmniWeb page and write the answer there.
utils.js includes a reference to jsdom over here:
https://github.com/mozilla/fathom/blob/master/utils.js#L430
That should be moved over into a separate test module that isn't included in the official fathom package as that ends up causing issues when bundling fathom in a webpack-ified bundle.
staticDom
isn't used by any parts of fathom itself - just the testsuite.
The fathom-webextension works around this by adding an ignore directive to workaround this for now:
https://github.com/crankycoder/fathom-webextension/blob/master/src/webpack.config.js#L71
I'm pretty sure the content at https://fathom.readthedocs.io/en/latest/ is unrelated to this library.
Write an and() combinator for the LHS. This, plus negation, should make us as powerful as Prolog.
…if this proves useful.
Document all the shiny new 2.0 stuff. Pull from the outline of my Hawaii talk.
First of all, thanks for this amazing and well thought out library.
From reading the jsdom docs, it appears that there was an api update that broke compatibility with pre api 10 versions of jsdom.
Using the latest jsdom (11.2.0), I wasn't able to get the code at https://mozilla.github.io/fathom/using.html#rules-sides-and-flows to work. Moreover, even after tweaking it for the new api, it breaks unexpectedly when querying facts against a boundedruleset.
I haven't investigated it in detail but it appears to be breaking at lhs.js
const matches = ruleset.doc.querySelectorAll(this.selector);
where jsdom used to expose querySelectorAll directly. For completeness, this is my stacktrace:
Uncaught TypeError: ruleset.doc.querySelectorAll is not a function
at DomLhs.fnodes (/Users/Ying/Code/sfpc/newspaper-headlines/node_modules/fathom-web/lhs.js:88:37)
at InwardRule.results (/Users/Ying/Code/sfpc/newspaper-headlines/node_modules/fathom-web/ruleset.js:315:37)
at BoundRuleset._execute (/Users/Ying/Code/sfpc/newspaper-headlines/node_modules/fathom-web/ruleset.js:215:31)
at BoundRuleset.get (/Users/Ying/Code/sfpc/newspaper-headlines/node_modules/fathom-web/ruleset.js:129:40)
at request (/Users/Ying/Code/sfpc/newspaper-headlines/index.js:30:21)
at ReadStream.<anonymous> (/Users/Ying/Code/sfpc/newspaper-headlines/node_modules/cached-request/lib/cached-request.js:249:15)
at emitNone (events.js:110:20)
at ReadStream.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1047:12)
at _combinedTickCallback (internal/process/next_tick.js:102:11)
fnodes @ VM255 lhs.js:88
results @ VM249 ruleset.js:315
_execute @ VM249 ruleset.js:215
get @ VM249 ruleset.js:129
request @ VM60 index.js:30
(anonymous) @ VM63 cached-request.js:249
emitNone @ VM27 events.js:110
emit @ VM27 events.js:207
endReadableNT @ VM52 _stream_readable.js:1047
_combinedTickCallback @ VM39 next_tick.js:102
_tickCallback @ VM39 next_tick.js:161
I'm using node v8.1.3.
I tried to see what the query might look like on latest jsdom, but ran into some issues. I'd be happy to elaborate on it further.
For now, I downgraded jsdom to 8.5.0. If this is documented somewhere already, sorry for the repost. Otherwise, I can make a PR to the README to specify the dependency.
"Node" already means "DOM node". Use another word for the node proxies Fathom scribbles its bookkeeping information on.
…from index.js and anywhere else they persist. Move them into docs or personal design notes.
Refactor the executor to plan not in terms of types added or emitted but in terms of types potentially changed from LHS to RHS. This, described in comments on erikrose's inferrer-generalization branch, is the generalized method which both provides optimal execution plans for the and() combinator (as in situations like and(type('a')) -> type('a')
) and still gets the right answer for simple rules that have a single type on the LHS.
ESLint is objectively better (and more configurable).
Plus, it found these:
➜ fathom git:(master) ✗ npm run lint
> @ lint /Users/pdehaan/dev/github/fathom
> eslint .
/Users/pdehaan/dev/github/fathom/fathom.js
6:7 error 'flatMap' is defined but never used no-unused-vars
10:7 error 'jsdom' is defined but never used no-unused-vars
222:44 error 'isBlock' is not defined no-undef
243:10 error 'paragraphish' is defined but never used no-unused-vars
247:24 error 'str' is defined but never used no-unused-vars
/Users/pdehaan/dev/github/fathom/test/test.js
20:40 error 'node' is defined but never used no-unused-vars
35:30 error 'node' is defined but never used no-unused-vars
36:28 error 'node' is defined but never used no-unused-vars
✖ 8 problems (8 errors, 0 warnings)
And my .eslintrc file looks like this:
env:
es6: true
mocha: true
node: true
extends:
- eslint:recommended
This is pretty verbose, certainly too long for an arrow function embedded in a ruleset:
function someRanker(node) {
return [{scoreMultiplier: 3,
element: node.element, // unnecessary, since this is the default
flavor: 'texty',
notes: {suspicious: true}}];
}
Once we've written some more test cases, observe what patterns shake out, and add shortcuts. I could see these taking the form of simple factored-up functions or of introspection in score() (for example, detecting iterables vs. plain objects or integer scores vs. objects).
Jared did a sketch of some extractors for Universal Search but based them on a much simpler, non-scoring, short-circuiting design with mixed rank and yank phases: https://github.com/JaredKerim-Mozilla/fathom/blob/master/fathom.js#L66. Port them to Fathom's current design, and bend our design to make them possible, if necessary. Short-circuiting isn't a blocker and would probably uglify our design. It's likely we can instead make the ranker functions more declarative and make the rule engine smart enough not to execute rules that couldn't possible win.
In 1.0 and 2.0, all scores on a type are multiplied together. This ticket is a parking place for discussion around allowing other kinds of operations. There are downsides to this, like not fitting certain machine-learning models as well. It could also make us care even more that rules execute in order and not in parallel. However, one approach might be to have have rules lay down the component numbers and an aggregate function do the combination operations.
And do or delete or file them.
Cool project!
Have you considered ways to determine weights automatically? Once you have a set of rules, it'd be awesome to be able to learn weights by providing a set of documents with extractions and labelling them as either correct or incorrect. This then becomes a fairly standard supervised learning problem to estimate the weights.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.