currently it just assumes its a sqlite file but it could be a JSON file/JSONL lines

once you get to about 8-10 databases, it becomes faster to use cachew

could probably create an example database by copying a current db and dropping a bunch of the sensitive data, just to make sure everythings still working

allows any files that are merged to be moved to another directory (e.g. to ~/.cache, /tmp), making periodic cleanups easier

Ideally, can it merge histories from all of them? But at least, can I point ffexport toward those profiles to save separately in some way?

Might be potentially interesting for Promnesia. Although also possible that the utility is very marginal and traversing history by timestamps is good enough.

• firefox: moz_historyvisits.from_visit. On mobile the field is present but seems to always be NULL
• chrome: visits.from_visit

Perhaps belongs to metadata, but also means that we'd need to keep original visit ID from the sqlite database to match the visit. And even worse, when all visits from different historic exports are merged in a single stream, the ids don't make sense anymore (they are 'internal' to a specific export in general). Although this would be possible to workaround if we somehow remap the ids in browserexport itself

As discussed here, is possible to take the merged data and convert it back into a new database to use, but supporting every browser would probably be a lot of code

Maybe support Chrome/Firefox, so at least you could merge data from browsers you use less into your 'main' (but this also promotes a silo, and pushes you towards firefox/chrome)

However,

• I expect this would become a lot of code
• you'd lose some data since this doesn't extract all attributes from every table, and we don't have some data, so we'd probably end up writing dummy values? Which then pollute the database even more for future exports...
• re-normalizing all the data is an annoying task for every different schema.
• the browser databases remove history over time anyways, so theres a limit, so you cant just merge millions of entries into your database to have history going back for years -- this is the whole reason browserexport exists, to save your old data.

Relatively low priority as I can interact with this programmatically by writing my own promnesia sources, but I can see this feature being useful for others...

Feels like a pretty difficult problem to solve, adds much more complexity to browserexport, when otherwise this is more functional -- taking databases, merging them and extracting visits

doing some testing in the vm related to #44

it was an experiment and it hasnt really panned out, so can add a deprecation notice and remove it an a year or so with 0.4.0

Support reading from compressed files -- not sure which format yet

would need to copy shm and wal from the live database and then

would need to set a checkpoint to vaccum the files
https://stackoverflow.com/questions/19574286/how-to-merge-contents-of-sqlite-3-7-wal-file-into-main-database-file

https://github.com/seanbreckenridge/ffexport/blob/6304e403af8b0ea84357d93fe1d88953fdbc6caf/ffexport/merge_db.py#L29-L38

https://github.com/seanbreckenridge/ffexport/blob/6304e403af8b0ea84357d93fe1d88953fdbc6caf/ffexport/parse_db.py#L100-L108

Simplify this into an additional SQL query that does the read and merge simultaneously in SQL, instead of python

promnesia supports firefoxmobile

Apparently you can use termux to backup that database periodically, may require root?

Not sure, haven't been able to figure it out myself yet, pretty new to termux -- if anyone has any info on this, would appreciate a comment

Hi. i have a bunch of cent browser plugins soon i am moving to a new gmail / i have a bunch of plugins , a lot of them.
I need to make a backup in the form of links.
example picture, just a list of my extensions as links from chrome store.

would be nice to re-use detection/parse/serialization mechanisms,

would rather duck-type stuff with some protocols than do a bunch of classes

over time, the amount of databases/space needed increases and may get harder to remove 'redundant' backups, like described by bleanser in #26

allowing this to read from JSON files (the output from running browserexport merge ~/data/browsing/* --json >/tmp/dump.json can end up compressing the data quite a lot

Essentially, just need to install these browsers on Windows and see where they typically install to, add the corresponding paths to each Browser file in the data_directories function

json lines would make reading from large JSON exports more memory efficient/allow it to start yielding lines before parsing the whole (sometimes half a gig) file

supporting the gziped versions of both of these would be nice as well as it drastically reduces the size of the json files as theyre just text with a lot of repeated property names

probably would just be run on my machine

• opens browser application in the background (and possibly goes to some URL?)
• saves the browser database to a temp directory
• parses the browser history

reports if the parse failed (or if the recent browser URL or database could not be found)

use emitted set instead of reading all the data into memory concurrently

As mentioned in #16

may be nice to offer a basic interface to search the dumped JSON data, not everyone is able to do some list comprehension in python

Currently you can dump it to JSON, and thats machine consumable, but end users of this don't have a great way to search this... could create a basic HTML file which searches the JSON using some javascript?

could be in JS or Python, doesn't really matter

Is the search queries you type into the menu bar; has an input and a use count, and a place ID

for most items, this would be null, so maybe store this differently than in Visits? or merge it differently into the interactive portion, and provide another merge library func which merges individual moz_inputhistory wit moz_historyvisits from one database, and then merges those together into another 'merge' lib function.

gives access to these in both inspect and merge

This is of low priority, it doesn't really add something that valuable to the Visit enum, its quite distinct. Not personally that interested in the data myself

its getting to be a bit unreadable with the amount of browsers

Seems that in ubuntu 22.04 the default firefox installation in via Snap, so the profile dir is in ~/snap/firefox/common/.mozilla/firefox 👀

I might fix it myself later, just leaving it here in case someone else is looking -- likely needs a fix somewhere around this line, e.g. if could take a sequence of paths and go through them

Btw IIRC it's been similar for chromium for a while, but haven't used chrome for ages, so not sure

https://github.com/karlicoss/bleanser

to help remove old databases which don't add any unique results

Doesnt seem that it'd be useful here, since we're already reading from a database (the firefox history database). Caching that info to another cachew database wouldn't make much sense.

Cant cache the live firefox history file because that keeps changing, so the only place cachew would improve any performance would be if we were spending a long time in merge_visits. But that doesnt even do any IO, its just a loop with a set, so doubtful.

For reference:

[ ~ ] $ time sh -c  'HPI_LOGS=debug python3 -c "from my.browsing import history; x = list(history())"'
[DEBUG   2020-09-05 03:07:21,267 my.browsing __init__.py:681] using inferred type <class 'ffexport.model.Visit'>
[D 200905 03:07:21 save_hist:66] backing up /home/sean/.mozilla/firefox/lsinsptf.dev-edition-default/places.sqlite to /tmp/tmpxvxci5yl/places-20200905100721.sqlite
[D 200905 03:07:21 save_hist:70] done!
[D 200905 03:07:21 merge_db:48] merging information from 2 databases...
[DEBUG   2020-09-05 03:07:21,303 my.browsing __init__.py:728] using /tmp/browser-cachw/homeseandatafirefoxdbsplaces-20200828223058.sqlite for db cache
[DEBUG   2020-09-05 03:07:21,303 my.browsing __init__.py:734] new hash: cachew: 0.7.0, schema: {'url': <class 'str'>, 'visit_date': <class 'datetime.datetime'>, 'visit_type': <class 'int'>, 'title': typing.Union[str, NoneType], 'description': typing.Union[str, NoneType], 'preview_image': typing.Union[str, NoneType]}, hash: 1598653858
[DEBUG   2020-09-05 03:07:21,310 my.browsing __init__.py:761] old hash: cachew: 0.7.0, schema: {'url': <class 'str'>, 'visit_date': <class 'datetime.datetime'>, 'visit_type': <class 'int'>, 'title': typing.Union[str, NoneType], 'description': typing.Union[str, NoneType], 'preview_image': typing.Union[str, NoneType]}, hash: 1598653858
[DEBUG   2020-09-05 03:07:21,310 my.browsing __init__.py:764] hash matched: loading from cache
[DEBUG   2020-09-05 03:07:22,083 my.browsing __init__.py:728] using /tmp/browser-cachw/tmptmpxvxci5ylplaces-20200905100721.sqlite for db cache
[DEBUG   2020-09-05 03:07:22,083 my.browsing __init__.py:734] new hash: cachew: 0.7.0, schema: {'url': <class 'str'>, 'visit_date': <class 'datetime.datetime'>, 'visit_type': <class 'int'>, 'title': typing.Union[str, NoneType], 'description': typing.Union[str, NoneType], 'preview_image': typing.Union[str, NoneType]}, hash: 1599300441
[DEBUG   2020-09-05 03:07:22,085 my.browsing __init__.py:761] old hash: None
[DEBUG   2020-09-05 03:07:22,085 my.browsing __init__.py:770] hash mismatch: computing data and writing to db
[D 200905 03:07:22 parse_db:69] Parsing visits from /tmp/tmpxvxci5yl/places-20200905100721.sqlite...
[D 200905 03:07:22 parse_db:88] Parsing sitedata from /tmp/tmpxvxci5yl/places-20200905100721.sqlite...
[D 200905 03:07:28 merge_db:60] Summary: removed 91,787 duplicates...
[D 200905 03:07:28 merge_db:61] Summary: returning 98,609 visit entries...
sh -c   7.46s user 0.19s system 99% cpu 7.711 total
[ ~ ] $ time sh -c 'HPI_LOGS=debug python3 -c "from my.browsing import history; x = list(history())"'
[D 200905 03:07:48 save_hist:66] backing up /home/sean/.mozilla/firefox/lsinsptf.dev-edition-default/places.sqlite to /tmp/tmpsvri7hr8/places-20200905100748.sqlite
[D 200905 03:07:48 save_hist:70] done!
[D 200905 03:07:48 merge_db:48] merging information from 2 databases...
[D 200905 03:07:48 parse_db:69] Parsing visits from /home/sean/data/firefox/dbs/places-20200828223058.sqlite...
[D 200905 03:07:48 parse_db:88] Parsing sitedata from /home/sean/data/firefox/dbs/places-20200828223058.sqlite...
[D 200905 03:07:49 parse_db:69] Parsing visits from /tmp/tmpsvri7hr8/places-20200905100748.sqlite...
[D 200905 03:07:49 parse_db:88] Parsing sitedata from /tmp/tmpsvri7hr8/places-20200905100748.sqlite...
[D 200905 03:07:50 merge_db:60] Summary: removed 91,787 duplicates...
[D 200905 03:07:50 merge_db:61] Summary: returning 98,609 visit entries...
sh -c   1.65s user 0.10s system 99% cpu 1.759 total

First run is 7 seconds, with a cached cachew hit for the backed up database. Second is reading from both of them directly, which takes 1.6 seconds.

For reference, this is how I modified my.browsing from HPI

diff --git a/my/browsing.py b/my/browsing.py
index 9f44322..af66530 100644
--- a/my/browsing.py
+++ b/my/browsing.py
@@ -25,17 +25,25 @@ import tempfile
 from pathlib import Path
 from typing import Iterator, Sequence
 
-from .core.common import listify, get_files
+from .core.common import listify, get_files, mcachew
 
 
+from .kython.klogging import LazyLogger, mklevel
 # monkey patch ffexport logs
 if "HPI_LOGS" in os.environ:
-    from .kython.klogging import mklevel
     os.environ["FFEXPORT_LOGS"] = str(mklevel(os.environ["HPI_LOGS"]))
 
+logger = LazyLogger(__name__, level="info")
 
-from ffexport import read_and_merge, Visit
+CACHEW_PATH = "/tmp/browser-cachw"
+
+# create cache path
+os.makedirs(CACHEW_PATH, exist_ok=True)
+
+from ffexport import Visit
 from ffexport.save_hist import backup_history
+from ffexport.parse_db import read_visits
+from ffexport.merge_db import merge_visits
 
 @listify
 def inputs() -> Sequence[Path]:
@@ -60,7 +68,20 @@ def history(from_paths=inputs) -> Results:
     import my.browsing
     visits = list(my.browsing.history())
     """
-    yield from read_and_merge(*from_paths())
+    # only load items that are in the config.export path using cachew
+    # the 'live_file' is always going to be uncached
+    db_paths = list(from_paths())
+    tmp_path = db_paths.pop()
+    yield from merge_visits(*map(_read_history, db_paths), _read_history(tmp_path))
+
+
+def _browser_mtime(p: Path) -> int:
+    return int(p.stat().st_mtime)
+
+@mcachew(hashf=_browser_mtime, logger=logger, cache_path=lambda db_path: f"{CACHEW_PATH}/{str(db_path).replace('/','')}")
+def _read_history(db: Path) -> Iterator[Visit]:
+    yield from read_visits(db)
+
 
 def stats():
     from .core import stat

create an enum/some helper function to handle expanding path/sending warnings when not present in a dispatch dict like

{linux: ~/something,
mac: ~/something
...
}

Most of that functions body would just be the dict and the call

also could probably make them all debug

add info to readme

librewolf
\AppData\Roaming\librewolf\Profiles\

floorp
\AppData\Roaming\Floorp\Profiles\

firefox - in the program it has a dot in the .mozzila here is without it maybe it changed already
\AppData\Roaming\Mozilla\Firefox\Profiles\

and '*/places.sqlite' //like in firefox