It takes 3 minutes and 12 seconds on my 2017 MacBook Pro to serialize https://stuffdutchpeoplelike.com/product/stroopwafel-teething-toy-bijtring/ (from the devtools panel, though that shouldn't matter). During the process, Firefox maxes out 1 entire CPU core and also sucks up an additional 5GB RAM. This makes corpus collection more of a drag than it needs to be. Profile and see what's taking so long. Reducing RAM use would be a bonus.

We're typically interested in preserving the URLs of captured pages, for instance to ensure we don't overrepresent one site. We've done this manually, but it takes time, and sometimes we forget or mess it up or lose the list. Let's embed the URL in the page automatically, in a place invisible to the viewer and unlikely to be mistaken for original page content. Data attribute on the root tag? fathom-url? I think there was also a convention freeze-dry used to store the original img src URL. We could follow that instead.

We don't want to silently and without visible penalty drop any target elements on the floor. For example, if the only input to a ruleset is dom('<input type='submit'>) but some should-be candidates are marked up as <button>, those candidates won't be vectorized. They will thus be considered non-targets, and the ruleset that consequently fails to find them will be rewarded for that rather than penalized, in terms of trainer output metrics. We currently rely on ruleset authors to be vigilant and catch this by looking at the per-page metrics, for example. That's not ideal, especially in applications with multiple targets per page, where per-page metrics are not very useful.

I can see at least 2 possible fixes:

• If the dom() selector is too tight and one or more target elements are consequently not vectorized, show a warning in the Vectorizer UI.
• Always vectorize all target elements. That might involve more plumbing around in Fathom itself, but it would give a better result, in that the premature pruning of target elements would show up in the fathom-train accuracy numbers—no special-purpose UI to check.

Bianca thought she had to wait until the progress bar completed. Ideas:

• Move from a progress bar to a barber pole (which I don't think <meter> helps us with), and have the start button turn into a stop or pause button when clicked. (I've wanted a pause button several times during debugging—or when getting on a plane.)
• Literally say that the values update in realtime. (But users don't read.)
• Make the coeffs and other output stuff pulse or wiggle or throb or something.

Please come up with something better than these. :-)

The Vectorizer currently populates the dropdown with the features specified in trainees.js, but it does not populate the Viewport Size fields which are also specified in trainees.js. It'd be helpful if that were autopopulated as well.

If an error occurs in the process of vectorizing a page, the Vectorizer will happily move on to the next page, fail, move on to the next page, fail, and continue like this for all of the pages. Instead we should stop the Vectorizer so the user is more obviously alerted that something bad has happened.

rough measurements points to updateProgress() accounting for 9% of runtime.
i suspect there's some quick wins that can be had there.

Parallelizing the Vectorizer (PR #55 ) allowed us to drastically reduce the amount of time it takes to vectorize a large number of samples, however I think we may need to revisit the maxTabs value initially set, as empirical evidence suggests that 16 (our current value) tabs causes "Unable to connect" errors, 24 tabs causes even more and 8 tabs cause none.

See this discussion for more details.

In our case, I think the first bullet point for what is wrong applies: the server is receiving too many requests (or something like that) simultaneously.

Subjectively, it seems like training ran faster when the ruleset was hard-coded into FathomFox, rather than each annealer iteration being kicked off by a message send to a separate webext. Benchmark this. If it's true, do larger chunks of work in fathom-trainees to avoid messaging overhead. One strategy could be to factor the async annealer out into Fathom itself and call out to it from fathom-trainees (to avoid fathom-trainees itself become too large and a maintenance burden for forkers).

FYI: The following changes were made to this repository's wiki:

• defacing spam has been removed

• Restricting write access to contributors is strongly encouraged. Please make that change (documentation).

These were made as the result of a recent automated defacement of publically writeable wikis.

STR:

1. Visit https://www.amazon.com/Nintendo-Switch-Neon-Blue-Red-Joy/dp/B01MUAGZ49/ref=sr_1_2?ie=UTF8&qid=1535647997&sr=8-2&keywords=switch&th=1
2. Open the devtools and use the inspector to inspect a DOM element
3. Click the "Fathom" tab

The panel appears blank for me on the latest Nightly:

I see some CSP issues in my browser console:

Content Security Policy: The page’s settings blocked the loading of a resource at inline (“style-src”).
fathomfox
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“style-src”).
fathomfox:1:1
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“style-src”). fathomfox 

I also get a weird error from "devtoolsPanel.js" about "Simmer being undefined", but the UUID in the error doesn't match FathomFox in about:support so it's probably not related?

[Affected versions]:

• Firefox Nightly 67.0a1
• FathomFox 2.3

[Affected Platforms]:

• Windows 10

[Prerequisites]:

• Latest FathomFox add-on is installed
• Have a batch of links:
  http://www.agame.com/game/paws-to-beauty-christmasedition
  http://www.4j.com/Death-Row
  http://8iz.com/game/build-a-bridge

[Steps to reproduce]:

1. Reach Corpus Collector page in Firefox.
2. Copy-paste the batch of links in the Corpus Collector.
3. Click on "Download" button,
4. Observe the downloaded pages.

[Expected results]:

• All pages are downloaded.

[Actual results]:

• http://www.4j.com/Death-Row page is skipped and only 2 pages are downloaded.

[Notes]:

• If http://www.4j.com/Death-Row is not part of an batch (downloaded alone), the download is made successfully.
• When opening http://www.4j.com/Death-Row in a new tab, a popup is displayed on top of the page. Maybe this has something to do with the issue.
• Screenshot of the issue attached

In mozilla/fathom#117, @danielhertenstein added a new fathom-serve tool that spins up an HTTPS server to serve local frozen sample pages. This is used for FathomFox's Vectorizer among other things.
For Firefox to load a page from an HTTPS web server, the server has to have a certificate that is trusted. We are currently using a self-signed certificate and asking the developer to trust us by clicking through a warning in Firefox the first time a page is served during that session. Can we add our server certificate to the list of trusted certificates in the temporary profile used by FathomFox?

Here's the warning displayed currently:

I talked to Dana Keeler, a security engineer who works on security certificates, and here's what she suggested:

the easiest way would probably be to use certutil to add the certificate to the temporary profile's cert9.db as a trust anchor

Here's a further breakdown of the steps:

1. brew install nss
2. Create a Firefox profile directory
3. Use NSS' certutil CLI to trust the server certificate in the newly created profile
4. Pass web-ext run the path to that profile directory

As part of this issue, we should also include a brief explanation in the docs about why we have to use HTTPS in the first place[1], as it is a lot more effort.

[1]: Why HTTPS rather than HTTP?
Using the freeze-dry library to freeze an HTML page inlines all of its resources, making the HTML file very large--too large to store such files on GitHub.
This is a problem, because we want to keep these files associated with their Fathom rulesets. After some thinking, we converged on Git LFS as a solution, which lets us store large files elsewhere and keep references to those files in a GitHub repo.
This required moving the resources out of the HTML document, keeping the HTML document in the GitHub repo, and its resources in Git LFS (@danielhertenstein created the fathom-extract CLI to do this).
By moving resources outside the page, directories or otherwise, we encountered CORS restrictions related to how file:/// URLs are handled. Basically, when a resource is requested from another file, that's treated as a CORS request, and CORS requests may only use the HTTPS scheme.

• add a notification immediately after hitting save to indicate that it's working, something like "Preparing download, please wait..." should suffice
• hitting return while the labelling field has focus should submit
• there should be some way to view existing labels; even if it's a simple console query, or a line you can copy+paste into the console to generate the list

Rather than always defaulting to 1024x768 for the Viewport Size in the Vectorizer, pull the default from the viewportSize of the selected trainee.

Refactor the FathomFox trainer and fathom-trainees so we can explore alternate success predicates in the trainee extensions. For example, rather than being hard-coded to run a ruleset and report whether the returned item is the one labeled, the trainee should be able to supply arbitrary code to compare the returned node's CSS background image with a set of URLs stored somewhere.

As well as serving short-term goals, this will let us experiment with alternate success functions (#12) without having to be architecture astronauts.

The Evaluate button is still useful as a debugging tool until #39 is implemented, but the "Train on the tabs in this window" and Pause buttons are attractive nuisances that don't even train Fathom 3 rulesets properly (for instance, they aim for per-page accuracy, meaning they won't work for applications that find multiple occurrences of the same type of entity). Remove them and any code that goes dead as a result (all the training code in FathomFox, for starters).

It may improve iframe serialization, which is supposed to work but, in my experience, does not happen at all.

[Affected versions]:

• Firefox Nightly 67.0a1
• FathomFox 2.3

[Affected Platforms]:

• Windows 10

[Prerequisites]:

• Latest FathomFox add-on is installed
• Have a batch of links:
  http://www.agame.com/game/paws-to-beauty-christmasedition
  http://www.4j.com/Death-Row
  http://8iz.com/game/build-a-bridge
  http://www.addictinggames.com/puzzle-games/creepy-halloween-difference-game.jsp
  https://ovh-cdn.amt-games.com/web/custom/bfg.html
  https://bigdino.com/play/1100/Rats-Invasion
  https://www.bubblebox.com/play/1720/ducklife.htm
  https://www.newworldempires.com/index.php
  http://www.cargames.com/Sniper-Assassin-4
  https://www.bgames.com/sport-games/darts--beer/

[Steps to reproduce]:

1. Reach Corpus Collector page in Firefox.
2. Copy-paste the batch of links in the Corpus Collector.
3. Click on "Download" button,
4. Observe the downloaded pages from the download folder.
5. After all downloads are done, delete the links from Corpus Collector page.
6. Paste other links and click on Download button.

[Expected results]:

• All pages are downloaded and no dupes are created

[Actual results]:

• Duplicates are created after multiple uses of the same Corpus Collector tab

[Notes]:

• This scenario can be observed using the same links but before every use, the downloaded HTML files must be deleted from the download location. Just repeat steps 5 and 6.
• Screenshot of the issue attached

Video of the issue can be found HERE.

When we were capturing login-form samples, some forms were in a menu that appeared only when we hovered over an element. This made it impossible to capture the page with the form showing. Add a key shortcut that triggers a freeze so we can grab such things.

[Affected versions]:

• Firefox Nightly 67.0a1
• FathomFox 2.3.1

[Affected Platforms]:

• Windows 10

[Prerequisites]:

• Latest FathomFox add-on is installed
• Link with issues:
  • https://www.glassesusa.com/

[Steps to reproduce]:

1. Open the above link into Firefox.
2. Click on "My Account" button.
3. Right-click on the Username field and Inspect Element.
4. Inspected element is selected.
5. Reach the Fathom tab from DevTools.
6. Type something into "Label" field.
7. Observe the "Label" field.

[Expected results]:

• The characters written are displayed.

[Actual results]:

• "Label" field is cleared when Tab Title is automatically changed

[Notes]:

• Screen recorder of the issue attached:

Attempting to train a ruleset via fathom-train with a vectors.json file containing feature vectors with null values results in an error similar to this one:

bdanforth ~/Projects/price-tracker (fathom3) $ fathom-train src/extraction/fathom/vectors/vectors_training_image.json -a src/extraction/fathom/vectors/vectors_validation_image.json -s
Traceback (most recent call last):
  File "/Users/bdanforth/.pyenv/versions/3.6.5/bin/fathom-train", line 11, in <module>
    sys.exit(main())
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/fathom_web/train.py", line 289, in main
    x, y, num_yes = tensors_from(training_data['pages'], shuffle=True)
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/fathom_web/train.py", line 38, in tensors_from
    return tensor(xs), tensor(ys), num_targets
  File "/Users/bdanforth/.pyenv/versions/3.6.5/lib/python3.6/site-packages/fathom_web/train.py", line 14, in tensor
    return torch.tensor(some_list, dtype=torch.float)
TypeError: must be real number, not NoneType

It'd be helpful if FathomFox's Vectorizer warned the user when it outputs a vectors.json file that contains null values in one or more feature vectors. This warning could also offer suggestions for possible causes, some of which include:

• A name mismatch between the name value passed into a rule function and the name of the rule in the list of [ruleName, coefficient] tuples referenced in the ruleset object.
• A score callback might be failing to return a number.
• A corner case of a DOM or CSSOM specification; e.g. innerText could return null instead of the empty string in Firefox.

The main motivations are…

• We could use atomicity between the two. For example, a lot of the code FathomFox runs, indirectly, lives in Fathom's trainees.js module. Adding a FathomFox feature often means changing FathomFox and that.
• We already have Fathom's Python-based CLI tools in the Fathom repo, and releasing 2 things from 1 repo has gone just fine for several iterations.

Bonus: It'll make testing fathom-vectorize easier, which otherwise has to explicitly check out a bunch of repos, just hoping that all the master branches are compatible.

At the moment, fathom-train reports training and validation accuracy but not testing accuracy. Design a way to feed a trained set of coeffs and biases into the trainer and measure the accuracy that results.

A significant time bottleneck in the ruleset development workflow is waiting for pages to vectorize. The vectorizer processes each page serially when the pages could be processed in parallel. The vectorizer is implemented by the CorpusCollector class which extends the PageVisitor class. The likely starting place is PageVisitor's visitAllPages() function.

Currently, the success function—that is, the test that tells us whether a Fathom-selected node is the right one—is simple equality: did it choose the same node as the human labeled? We'd like to crowdsource labeling, but I doubt the crowd will be as strict in their choice of nodes as ruleset authors. For example, I can imagine them selecting either the inner or outer div when trying to find the price node in <div><div>$34.95</div></div>. I doubt they will stick closely to rubrics even if we define them.

Thus, we should explore alternate, more forgiving success functions. Perhaps any node of equal or nearly equal dimensions and placement should be accepted. Perhaps any node with equivalent innerText should be accepted. Perhaps there should be a fuzzy acceptance based on how far off they are. Perhaps success functions should be pluggable in trainees.js, based on the needs of the ruleset or individual out() rule.

Write up some prospective success functions, and see how close you can get to rubric-strict training accuracy with slightly faulty labeling.

Accuracy as a statistic doesn't paint the whole picture; it would be nice to have an idea of the error bars. Toward that, see if the Trainer has all the data it needs to compute a CI as it does, perhaps a binomial proportion confidence interval.

You can't train rulesets with multiple out() rules without going in and editing fathom-trainees in between runs, because currently the ruleset key is assumed to be the same as the out() key, which is the same as the element-labeling key. Break that equivalence.

We could expose a list of all the out() rules in the menu. We could even use <select> groups to group them by ruleset. I imagine there'd usually be only one ruleset anyway, though. The out() key could remain the same as the labeling key.

Some pages, like https://tutorials-raspberrypi.com/google-chrome-for-raspberry-pi/, generate the spinny pizza of death—but only during automated corpus capture, not when freezing using the dev panel.

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Edit: Related

In light of the helpful new fathom-serve and fathom-list CLIs added to fathom-web, we should recommend use of these tools in the Vectorizer page.

I believe currently only fathom-list has been released in the latest version 3.1 of fathom-web. But that will probably change soon.

We should also update the Base URL text value to be https://localhost:8000/training/.

When testing the Evaluate page on the Price Tracker ruleset, @biancadanforth found that no BAD labels were applied to pages evaluated using the title type (see here). I believe there are no labels applied because the ruleset does not find any candidate elements when evaluating the ruleset (see my last comment in the pull request referenced above).

Tied to this issue is this issue @erikrose worked on.