When a new bookmark is created by the user, it should be put into the db as well.

Often, document.body.title is not available, so the url is taken instead of a title field in the search result elements.

However for the same pages, if i use document.title everything works fine and it gives me the correct title.

I assume that changing the request to document.title (or asking for both and take the longer one) could solve the issue.

In order not to have performance issues when a user uses his browser, we could delay certain processes to be executed while a user is idle. (i.e. deletion of old data, adding new pages to index or such stuff)

User Story

As a user I want to filter by source I came from. Meaning I want to do a search like "I came from Facebook.com and the article had the words 'test' in it"

Implementation ideas

Seems like this could be relatively straight forward thing to do, as the historyAPI provides this information.
So we could just query it for a url, add it to the visit doc and integrate it into the index over the visits (if that is the way we structure our indexes)

User Story

As a user i want to be able to quickly blacklist a page via the drop down.

Implementation ideas:

When a user opens the popup, there is a button: "Quick Blacklist"
When a user clicks on it, two selections appear "Whole Domain" "Just this page"?
For now, it could just store the either the cut-off domain-string "www.times.com" respectively the url "www.times.com/testpage.html". Later maybe even some type "url" "domain" etc.
I would cut off the http/https though, so that the pages are not stored irrespectively of the protocol they are opened with.

It just happened that the download ran twice. This is the screenshot of the import view:

###here is the error log:

Page 1:

Page 2:

@swissums will take a deeper look into the code base now and may actually find the cause of it.
@poltak: I sent him your PR thread around the importing of urls, so he can understand your reasoning better. (Thanks again for the extensive documentation around it)

These are the things that still need to be done in order to make the upgraded WebMemex extension releasable:

• Update Documentation & GitHub Readmes & Roadmap
• Update Designs of Settings
• Update Designs of worldbrain.io Homepage
• Find new name for WebMemex
• Add Blacklist function (UI and logic) (thanks @poltak)
• Add delete function to remove page from DB and connect to UI
• Add Batch Download (in progress)
• Add UI for Import (in progress)
• Support Web Apps (in progress)
• Natural Language Support (in progress)
• Importing bookmarks (in progress)
• Infinite Scroll
• Implement new search
• Store only one version per page (make no duplicates in DB/Index)
• Document data model
• Remove component that shows pages before and after search result
• Only show latest visit in defined time range when searching
• Importing data from old extension and put it into pouchDB

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

When searching, the result list always shows all the visit of a page object in the list.
To me, it is unnecessarily overwhelming and it would be enough if only the latest visit is shown for any given time range.

So if a user uses a time filter like climate change before:"2 weeks ago it should show only one result per page object, with the latest being before 2 weeks ago, even if he visited it 1000 times before.

When I open a task on Asana and then go to the next one in the list, it changes the url for that task without reloading the whole page.

Problem is now, that it creates a visit item, which is missing the title and does not get the archived version stored.
Furthermore, it seems as if page data is not fetched again, as I also could not search for words from those tasks in Asana.

When working on web-apps (Asana, Facebook, Twitter, Github) I regularly experience lags & freezes.

My preliminary assumption is that the storing processes take too much resources and that it has mainly to do with the freeze dry page-archiving.

Needs to be investigated and fixed before the next release.

Placeholder for now, will put more information here as soon as needed.

In omnibar.js

The

let browserName
;(async () => {
    const browserInfo = await browser.runtime.getBrowserInfo()
    browserName = browserInfo.name
})()

Throws an Error in chrome as the function browser.runtime.getBrowserInfo() isn't supported. see this link

I don't fully understand why the name is needed?

Background Information

The old extension stores the data about websites (title, keyword etc) to local storage whereas the new extension has it stored in PouchDB also with a different data model.

When we update the code from the old extension to the new one, we have to make sure, that all the previous data is transferred to the new DB and data model and entries deduplicated (1 page object per URL)

Previous work to understand old data model:
WorldBrain/Legacy-Research-Engine#101

Caution: In some extensions there is about 1GB of data, whereas there will be many duplicates in the system. Means the import process may need to be executed in batches, as well as to check for each element before it is stored, if there is already one available. We should not add duplicates.
Also we need to make sure we have a visit items for each entry.

Inspiration for that process could be gotten from @poltak's PR: #25

Also in the WebMemex upstream there has work been happening towards a simpler data model, which should be taken into consideration: WebMemex/webmemex-extension#101

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

With upcoming imports feature comes bookmark docs. These need to somehow fit into the overview search.

Current bookmark doc structure looks like:

interface IBookmarkDoc {
  _id: string; // 'bookmark/:timestamp/:nonce'
  title: string;
  url: string;
  page: { _id: string };
}

In the context of imports in the _id, :timestamp is set to the BookmarkTreeNode's dateAdded field while :nonce is the id field (more info on BookmarkTreeNodes in web extension API). Setting the _id like this makes the _id idempotent, in the same way visit and page docs are generated in the context of imports.

Bookmark docs will also be made whenever a user creates a new bookmark (can use browser.bookmarks.onCreated event). This is unimplemented as of yet; feel free to.

This structure is subject to change; I roughly came up with this based on the current visit doc structure, and what's available in BookmarkTreeNodes, and would very much appreciate some critique.

The most important thing is that it references a page doc, much like a visit doc, so any given page doc can have both visit and bookmark docs pointing to it. Current search logic seems to get visit docs then map their associated page doc data to them for displaying in the overview. Rough idea now is to add another db.find query for bookmark docs in the similar way, but then visits and bookmarks need to be merged someway in before passing them to the container as UI state (preferably done in a Redux selector).

Some comments from @oliversauter relating to how the UI could show this:

why dont we handle a bookmark item, if it does not have a visit item, as if it were a visit item that has a small ribbon, indicating it to be a bookmark? And if it is a visit item that also has a bookmark, we give it a ribbon as well?

This would need some way to relate bookmark docs to visit docs in the search logic.

The aim is to design a weighted model to boost certain search results based on features that are not provided implicitly by the stored documents. Some features that can be used:

• Boost certain websites based on the frequency of visiting that particular website
• Explicitly ask a user for favourite websites
• Websites visited more recently can be given a higher priority.

More criteria can be added so all suggestions are welcome 😄

@poltak please advise where such a provision can be placed in the code.

User Story

As a user, I want to filter my search results by bookmarks.

Implementation ideas

We could just have a filteroption as a flag in the UI that lets a person do that.
This task both contains UI and business logic work.
UI: the flag
Logic: filtering out results that are bookmarks.

May worth looking into the mockups for the new overview design and how the filter could fit in there: https://freehand.invisionapp.com/freehand/document/m7HtV3RYd

As discussed, we need a new search implementation before the re-launch.
A bit dependent on the work of @bwbroersma.

Plan is to get everything else finished first and then either be able to already implement Benjamin's work or having to temporarily resort to another library.

I got a bit confused when using the deletion prompt after choosing to blacklist domains/urls.

Its a bit counterintuitive that the prompt appears in the middle of the page, also i could imagine that we will run into some problems on some pages, that disallow/block CSS injections?

May be possible to just use the whole popup, looking something like this?

Problem:

Currently, if visiting a web app (Facebook, Twitter, GitHub) only the first page opened will be indexed.

Solution ideas:

There are some ways to detect state changes in the web app/history to redo the scraping of the page.

In the old Research-Engine code, we did it with this method here.
https://github.com/WorldBrain/Research-Engine/blob/master/src/js/get_tab.js

The downside of this approach is that it does not detect reloads of the same page, but it captures all other actions. Might be ok, not to detect reloads though.

Others are: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webNavigation/onHistoryStateUpdated

Related Discussions

WebMemex/webmemex-extension#50

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

User Story:

As a user I want to open the search results page with a custom shortcut

Current State:

By default the extension opens up the search results page in a new tab, which is not preferable, because many users already have a "new tab" replacement.
This setting has to be removed from manifest.

Implementation ideas:

In the options page, there should be the ability to define a custom shortcut for opening the search page.

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

Just had a call with Alexander Praetorius, who gave me a lot of good tips for libraries we can use instead of pouchDB and adapters to use for DAT.

Just leaving them here, so we can discuss about them, also related to the sharing network:

https://github.com/beakerbrowser/hashbase (https://hashbase.io/)
https://github.com/mafintosh/hypercore
https://github.com/mafintosh/hypervision
https://github.com/mafintosh/hyperdb (alternatively: https://github.com/beakerbrowser/injestdb)
https://github.com/levelgraph/levelgraph as a graph database (alternatively gun.js.org, however the latter is a separate application which does not integrate so)

hyperdb, injestdb,levelgraph, hypercore and hashbase modularly integrate/build on top of each other very well with leveldb, so we can use it in the browser's native database + on the server as well.

There are a lot of 'data fetch time out' errors in the import process.
Sometimes a google search url is successful then the other one again not.

I assume that it might have something to do with with the way the timer is set?
After what time does it time out the request?

User Story

As a user I want to delete a page from the DB

Implementation ideas

For now, we could simply delete all occurrences of the page, when a user clicks on the small trash icon in the overview.
Maybe implementing a confirmation dialogue would be good. But one that can be set to not pop up everytime. "Do not show this message again"

Optional, if not to hard to implement:
The popup gives the choice between all visits and page object or just the current visit.

Related Work

In this thread over at WebMemex source there has already been some work done.
One might borrow from the code there.

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

User Story

As a User, I want to search in the address bar (and search box in overview) with time filters and put in NLP queries like:
climate change after:"10 days ago" before:"yesterday"

Current Status:

This is currently implemented in the Research-Engine here and there might be a lot to get inspiration from:
https://github.com/WorldBrain/Research-Engine/blob/master/src/js/background.js#L302
https://github.com/WorldBrain/Research-Engine/blob/master/src/js/queryparser.js

Implementation Ideas:

Ideally the NLP date filter translate also in filled out date pickers in the overview

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

Hey team.

We need to find a new name for the WebMemex before we release it.
It was a temporary name, as long as there was still a lot of overlap with Gerben's direction, and before we release the new version.

Any ideas are welcome to be listed here. Whoever is in, I would invite you to have a video chat for this brainstorming session.

Here are some resource Links:

• Preact
• Vue.js
• Cycle.js
• Choo.io
• Overview of React vs Preact vs Deku here
• Problems with React and Cylce.js/Elm as alternative here

Loading browsing history to a graph db can be cool and provide nice visualization.

Here is an example https://jesusbarrasa.wordpress.com/2016/09/30/quickgraph4-explore-your-browser-history-in-neo4j/

User Story

As a user I want to quickly blacklist the current URL and/or domain. Also when choosing to blacklist one or the other, I want to delete all occurrences in the DB with that domain/url.

Current State

With the PRs #5 and #3 by @poltak we implemented the ability to blacklist items when going to the settings page.
This requires 3 additional steps though and a common use case is that a user wants to blacklist the currently visited domain/url.

Implementation ideas

Just throwing ideas in the room how it could be done, happy for some new input.

Stage 1:

A drop down-menu like in the (Re)search-Engine extension when clicking on the icon in the address bar.
There is a button called "Blacklist current page", when clicked, the input switches to 2 options:

• "Current URL"
• "Current Domain"

Note: clicking on the icon of the WebMemex at the moment, opens the search results page, which should not happen in that case.

Stage 2 (Optional for now):

When choosing either of those, another switch happens with a new Y/N choice
"Do you want to delete all entries for your choice from the database?"
For this part, there has been some work done, which still needs completion in: WebMemex/webmemex-extension#67
Although here it could be that the code would need some more review/rewrite before being used.

My suggestion is therefore first starting with the Stage 1 and then as an optional thing add the deletion for all entries.

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

When searching it always shows 2 pages that have been visited before and 2 pages that have been visited after. From a design and UX perspective it is not a worked out feature yet and needs some more work.

Hence, I would suggest we remove/disable the feature for now.

All work relating to this will go into the feature/blacklist branch.

There is now a working blacklist UI that has been worked on in another fork of the upstream webmemex-extension repo. This is yet to be merged upstream, but is now merged into the feature/blacklist branch on this repo.

Will continue work on this branch with the intent to use the data stored from that UI to match against visited sites and ignore the tracking of page visits if the visited page's URL matches any values in the stored blacklist. The tracking of page visits should be in the src/activity-logger module.

User story:

As a user I want to pause my entire recording of my history for a period of time.

Implementation idea:

There could be a button in the drop down, letting people chose to pause the recording for a chosen time. They may have a drop down with the selections: 5;10;20;30;60;120;180 min and a small confirm button
When people activate it, there could be a system popup, indicating that the recording is now paused for X minutes.
It pops up again, when the recording is back on again. This popup though should be one a user can only remove by actively clicking on it. So not a temporary one that automatically disappears after X seconds.
As long as the pausing is activated, there might be a count-down timer (in min) that indicated how long it will still be active. It could be at the place in the drop down, where the choice to activate pausing was before.

Also a user should be able to stop the pausing by just clicking on a small X icon next to the count-down timer.

User Story

As a user, I want to read an article later offline.

User Story

As a user I want to see when I visited a search result, specifically if I filter the results by time, as developed in #23

Implementation ideas

Without time filter:

With time filter:

In the Research-Engine extension, the code for the time representation is in the function suggestionsComplete(). Might offer inspiration:
https://github.com/WorldBrain/Research-Engine/blob/master/src/js/background.js#L166

Gerben provided a nice deduplication framework, where we can flexibly define the rules on when and if to store duplicates.

I may be wrong about that, but It seems to me that we currently won't need to store a new object for every visit, as long as there is no versioning.
It also would make it simpler if we just had one version per page and could index all of them, instead of worrying which versions to index. We may want to store and search multiple versions later though.

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

As a user I want to export the data from the extension, save it as a JSON (?) file and import it again.
This would make it possible to do testings with larger data sets.

Somehow related to the work done in #19, at least the import part. Since this feature will have to come anyhow, we should take both use cases into account: manual upload & transfer from old extension.

Performance Issues:

Import:

• The layout is throttling 14.3% of the total time to finish an import. Which means that react is re-rendering more than it needs for some reason.

• Also the Layout Tree which directly manipulates individual layouts is throttling 25.8% of the total load time.

• The dispatchOnMessage callback is throttling 48.9% of the cpu as well. After some research I believe this has to do with the way that import is communicating with the client side. From what I understand import is updating a global redux state to pass the info down to the child component (is this correct?)

Some Suggestions to (maybe) help solve this issue:

• I believe mapStateToProps is being called too many times or unnecessarily somewhere.

• Don’t print every single url that is being saved or maybe have this as a default of hidden.

• Make the client-side import page more visual and have a status bar that only shows changes of 25%- 50%-75%-100% instead of every single item change.

• Use a web worker to push bigger amounts of data via transferable objects see.

Search

Getting the same with Layout Tree eating up lots of performance 50.3%… I am not sure what is causing this.

User Story:

As a user I want to see all my archived versions

Implementation Ideas

Could be done with a simple flag in the form of a checkbox that lets users filter for all page docs with a freeze-dry attachment.

To be implemented after #42

It seems as if there are still quite some issues with freeze dry as seen here, here and here.

My suggestion, if it is feasible, is to add a feature flag for the archiving feature, which people have to activate in the settings first. There should also be a note to mark the feature as experimental.

In case of feature being turned off, it should also remove the list entry "Archive current page" from the popup.

When working on web-apps (Asana, Facebook, Twitter, Github) I regularly experience lags & freezes.

My preliminary assumption is that the storing processes take too much resources and that it has mainly to do with the freeze dry page-archiving.

Needs to be investigated and fixed before the next release.

Windows and Linux have the combination CTRL + Y reserved for the redo functionality which is pretty useful. The shortcut for the overview page should be changed to some other combination.

When you make the browser smaller this occurs:

User Story

As a user I want to have quick access to the main features of the tool.
Including:

• Search Overview [for now just a link to it, later maybe even quick search]
• Settings
• Help
• Importing of History and Bookmarks
• Notifications [later]
• Quick Blacklist [later]

Implementation

Basically a drop down popup maybe even similar to the LastPass dropdown with a lot of space and te ability to extend features (like notifications and quick blacklist and maybe later also quick search)

Option to chose between a Full HTML archive or "Read later version"

Read later version would be just the text of the page put into a standardised view:
Allows saving space, Better readability bc.removing ads and clutter.

A bit like the app readablity already provided:

Background

Fairly new archive button on the popup allows a keepFreezeDry flag to be set on a page doc that matches the webpage currently active in a tab. This flag allows the page doc to be skipped during the freeze-dry cleanup cron job.

When the popup loads (whenever clicked), the initial archive button state is disabled until it has time to check for the matching page doc.

Problem

That matching page doc check is currently being done via an un-indexed pouchdb-find query on _id and url fields (getPageDocId fn in archive-button.js). This works fine with small amounts of data, but once the extension amounts a bit of a history, things get very slow and means the archive button is not useful (stays disabled for a long time), and resources will be used up by the popup script trying to find a matching page in all that unordered data.

Ideas

Creating apouchdb-find index for this query doesn't seem possible; index never gets used in the query. Query uses _id and url, which the index should reflect, but attempts on different combinations of those fields have not produced any solutions.

Depending on what the new search index will afford, it may be able to utilise that.

Every page visited by the user should get processed by the activity-logger module, which lives in the background script of the extension. It will create a page stub for every visited page (as long as not blacklisted, paused, or page is a special non-loggable - extension pages, etc.). Maybe there's some way to get the popup to check some state that activity-logger has set in its page visit event?

tl;dr version:

• on popup script load, using the data from the current browser tab, we need to find the best matching page doc stored in the DB
• in a lot of cases this page doc would have been created by activity-logger in the background script on visit to that page

User Story:

As a User I want to quickly search via the omnibar, but also see more than 5 results, without having to retype everything.

Current State:

In the omnibar-dropdown, there are currently only 5 results visible. Obviously this is not enough, hence we have the overview for results now.
But currently a user has to go to the overview and type the query again, when he does not find a good result.

Work already done: WebMemex/webmemex-extension#63

Implementation Ideas:

Replace the first (or last) result of the omnibar with "Show more results". When clicking, it directs the user to the overview page with the same query.

The settings dashboard UI still needs some overhaul so it fits into the general UI of the dashboard.

Elements/pages involved:

• Sidebar

• needs the ability for elements and sub elements (to directly get to a sub-section of the settings page)

• Settings
  Needs good components to seperate different parts of it

• Import process
  As seen in https://github.com/WorldBrain/WebMemex/tree/feature/imports, also needs some rework.

Expected behaviour:

When clicking on a date field and choosing nothing, then there should not be any filtering.

Actual behaviour

When clicking on a date field and choosing nothing, it fills the filter with null as recognised in the change of the url to: /overview.html?startDate=null&endDate=null
Then it applies its filters and removes all the results.
Problem is, without reloading the page or removing the querystring from the url, you cannot reverse the selection of null

Freeze dry pages take up a lot of space (100kb-2MB per page, EDIT: with photos the amount is pretty unlimited, just had a version with 45MB) and should not be saved for each and every visit of a page, and also not for every page.

I suggest 3 options on how to implement, ideas welcome:

1. User can define in the settings that Freeze Dry Versions are made for each page they bookmark via the browser. For that we have to listen to the bookmark event.

2. They can "Archive Current Page" via a button in the Dropdown, not the same as bookmarks though.

3. We automatically store the last 100 pages as freeze dry, and if they are not bookmarked or manually archived, we delete the freeze dry version.

User Story:

As a user I want to opt in to send WorldBrain crash reports

Might be an extension for the work @JessicaML and @V-Ho do with the notifications / dashboard

Problem

When a blacklist a page is not recorded, there would be a gap in the trails between visits.
So we might have to add the id of the previous visit to the next recorded visit item, or interrupt the trail.