obsolescence warnings from the MSVC2019 IDE today already:

2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3270: There was a mismatch between the processor architecture of the project being built "MSIL" and the processor architecture of the reference "tessnet2_32, Version=2.0.4.0, Culture=neutral, PublicKeyToken=1550524b5ba07e29, processorArchitecture=x86", "x86". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and references, or take a dependency on references with a processor architecture that matches the targeted processor architecture of your project.
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.CSharp.CurrentVersion.targets(133,9): warning MSB3884: Could not find rule set file "AllRules.ruleset".
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(148,60,148,82): warning CS0618: 'StandardAnalyzer.StandardAnalyzer()' is obsolete: 'Use StandardAnalyzer(Version) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneSimilarityQueries.cs(93,25,93,34): warning CS0618: 'TokenStream.Next()' is obsolete: 'The returned Token is a "full private copy" (not re-used across calls to Next()) but will be slower than calling Next(Token) or using the new IncrementToken() method with the new AttributeSource API.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(175,36,175,124): warning CS0618: 'IndexWriter.IndexWriter(string, Analyzer, IndexWriter.MaxFieldLength)' is obsolete: 'Use IndexWriter(Directory, Analyzer, MaxFieldLength)'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(210,44,210,91): warning CS0618: 'IndexReader.Open(string, bool)' is obsolete: 'Use Open(Directory, bool) instead. This method will be removed in the 3.0 release.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(216,17,216,21): warning CS0618: 'Hits' is obsolete: 'see Searcher.Search(Query, int), Searcher.Search(Query, Filter, int) and Searcher.Search(Query, Filter, int, Sort)'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(216,29,216,64): warning CS0618: 'Searcher.Search(Query)' is obsolete: 'Hits will be removed in Lucene 3.0. Use Search(Query, Filter, int) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(221,21,221,24): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(221,32,221,35): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(253,44,253,91): warning CS0618: 'IndexReader.Open(string, bool)' is obsolete: 'Use Open(Directory, bool) instead. This method will be removed in the 3.0 release.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(259,17,259,21): warning CS0618: 'Hits' is obsolete: 'see Searcher.Search(Query, int), Searcher.Search(Query, Filter, int) and Searcher.Search(Query, Filter, int, Sort)'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(259,29,259,64): warning CS0618: 'Searcher.Search(Query)' is obsolete: 'Hits will be removed in Lucene 3.0. Use Search(Query, Filter, int) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(264,21,264,24): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(264,32,264,35): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(318,48,318,95): warning CS0618: 'IndexReader.Open(string, bool)' is obsolete: 'Use Open(Directory, bool) instead. This method will be removed in the 3.0 release.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(322,21,322,25): warning CS0618: 'Hits' is obsolete: 'see Searcher.Search(Query, int), Searcher.Search(Query, Filter, int) and Searcher.Search(Query, Filter, int, Sort)'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(322,33,322,66): warning CS0618: 'Searcher.Search(Query)' is obsolete: 'Hits will be removed in Lucene 3.0. Use Search(Query, Filter, int) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(327,25,327,28): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(327,36,327,39): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(351,44,351,91): warning CS0618: 'IndexReader.Open(string, bool)' is obsolete: 'Use Open(Directory, bool) instead. This method will be removed in the 3.0 release.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(359,17,359,21): warning CS0618: 'Hits' is obsolete: 'see Searcher.Search(Query, int), Searcher.Search(Query, Filter, int) and Searcher.Search(Query, Filter, int, Sort)'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(359,29,359,57): warning CS0618: 'Searcher.Search(Query)' is obsolete: 'Hits will be removed in Lucene 3.0. Use Search(Query, Filter, int) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(363,21,363,24): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneIndex.cs(363,32,363,35): warning CS0618: 'Hit' is obsolete: 'Use TopScoreDocCollector and TopDocs instead. Hits will be removed in Lucene 3.0.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(647,17,647,114): warning CS0618: 'PriorityQueue.Insert(object)' is obsolete: 'use InsertWithOverflow(Object) instead, which encourages objects reuse.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(703,29,703,56): warning CS0618: 'IndexReader.Open(string)' is obsolete: 'Use Open(Directory, boolean) instead. This method will be removed in the 3.0 release.'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(726,38,726,66): warning CS0618: 'IndexSearcher.IndexSearcher(string)' is obsolete: 'Use IndexSearcher(Directory, bool) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(728,13,728,17): warning CS0618: 'Hits' is obsolete: 'see Searcher.Search(Query, int), Searcher.Search(Query, Filter, int) and Searcher.Search(Query, Filter, int, Sort)'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(728,25,728,47): warning CS0618: 'Searcher.Search(Query)' is obsolete: 'Hits will be removed in Lucene 3.0. Use Search(Query, Filter, int) instead'
2>W:\...qiqqa\Utilities\Language\TextIndexing\LuceneMoreLikeThis.cs(823,29,823,38): warning CS0618: 'TokenStream.Next()' is obsolete: 'The returned Token is a "full private copy" (not re-used across calls to Next()) but will be slower than calling Next(Token) or using the new IncrementToken() method with the new AttributeSource API.'
2>W:\...qiqqa\Utilities\GUI\Brainstorm\Nodes\NodeControlSceneData.cs(13,16,13,44): warning CS0169: The field 'NodeControlSceneData.current_scale_at_last_resize' is never used
2>W:\...qiqqa\Utilities\Internet\GoogleScholar\GoogleScholarScrapePaperSet.cs(16,24,16,40): warning CS0414: The field 'GoogleScholarScrapePaperSet.paper_set_source' is assigned but its value is never used
2>  Utilities -> W:\...qiqqa\Utilities\bin\Debug\Utilities.dll

HTR:

• pick a PDF with bibtex where Google Scholar will find a BibTex record too.

• if the Sniffer DOES NOT have Wizard turn on yet:

  • open the Sniffer.
  • Turn on (enable) the Wizard
  • exit/close Sniffer

• open the Sniffer

• Google Scholar will dig up the BibTex, Qiqqa Sniffer will snarfle that BibTex and dump it in the top-right corner box, turning it green.

• ... no try to edit something in there...

The moment you change something, the Google Scholar snarfle -> Wizard action will quickly jitter the screen a bit and your edit will be ERASED.

Same process can be used to nuke existing BibTeX. Which IS SUPPOSED to be correct.

Hence: Wizard action should NEVER overwrite existing BibTeX!

Considering

See discussion at #15: as James emphasized backwards compatibility there, it triggered this thought process:

1. normally, you test for that kind of thing, using stuff like qiqqa test databases and running multiple versions of qiqqa in parallel plus checking if upgrading/downgrading via setup.exe actually flies (version checks or otherwise in there might block downgrade paths; currently the Qiqqa installer mentions everything as if it were an upgrade all the time, but that's okay as it allows moving back & forth that way without much trouble.

2. then there's the choice of how far we take this:

   Does 'backwards compatibility' mean:

   1. when you upgrade, then downgrade, after the downgrade all your latest activity after the upgrade makes it through? ("full continuation across versions" - tough to accomplish / near impossible), or

   2. when you upgrade, then downgrade, you get 'rolled back' to the time you were upgrading? ("rollback to previous version's SNAPSHOT"?)

   3. when you upgrade (or downgrade, your major data sources are kept intact (using option 1 or 2 above), while the derivative data, e.g. search index caches, are rebuilt on every data format change: if we upgrade Lucene (#23), or downgrade it as we downgrade Qiqqa, we MUST regenerate the index as the Lucene file format has changed significantly. ("regenerate what you can that's unnoticeable")

   4. or you use a 'weaker' compatibility definition, where 'derivative data' includes BrainStorms, Expedition sessions, etc.: the Qiqqa Configuration and Qiqqa Library Database plus PDF collection (and very probably the OCR text output) are then considered sources which must be kept backwards compatible using option 1 or 2 above, while the rest is marked as 'incompatible' and regenerated upon user action/request: this would mean BrainStorms and such would not always survive past an upgrade or downgrade where the brainstorm file format changes. ("regenerate as much as you can")

Depending on the choice made above (do we go with option 1,2,3 or 4?), there's the technical side of it:

1. is an extremely tough nut to crack under all circumstances and there's the question of new versions introducing data that old versions don't carry (take #27 which currently is only noticeable to hackers like me until extra work will make it visible via CSL, but how 'faithful' should the backwards compatible move be: kill all the new URI info? You get the point. Some degree of "metadata loss" is always lurking when downgrading.)

2. snapshotting can be done by

   1. creating a (archived?) backup of the Qiqqa files/libraries and then migrating the files themselves, but that only works out well for upgrades: downgrades need to unpack that archive or they'll handle more-or-less-visibly incompatible data files. This would require more coding.

   2. using (file format) version identifiers in the Qiqqa data file system, such that every version of Qiqqa knows "where to find his own": upgrades then go and look for older versions to migrate upwards when "their own" doesn't exist yet, while downgrades can simply revert to looking at their own data "that was before".

3. regenerate-what-you-can is the same as option 2, except you exchange extra time for code by regenerating indexes/caches instead of adding cod to migrate them to the new file format/version.

4. regenerate-even-more is like 3, and thus 2, except you don't code upgrade migration code for some of the Qiqqa derivative data files, e.g. brainstorms, expeditions, ...

Preference

My personal preference would be to go with option 2.ii - snapshotting with version in path (though do we need to snapshot/copy the entire bunch of metadata files as there are (or will be) interrelations, or do we snapshot/copy single files? My pref is the first: one snapshot per version, upgrade what you need upgraded).

Add to that as much option 3 as we can get away with (Lucene search index regeneration, OCR cache regeneration, i.e. re-run OCR task when we change the formats there, or the OCR engine maybe: #35 + #34, autotags, ...)

It's very simple and thus least error prone, which is handy to have for a path less traveled: back-pedaling on a Qiqqa software update/upgrade.

The 'negative' consequences is the user disk slowly filling with Qiqqa library/config snapshots, but that's a 'risk' I am very much okay with: here a 20K+ articles library clocks in at LESS THAN 50MByte of metadata, SQlite3 library main database included.

Of course, it would get worse than that when we change the OCR text cache, but that can very probably be coped with nicely by blowing away the entire OCR cache and regenerating it from scratch.

The (Lucene) Search Index is also quite another matter as that one clocks in at about ❗️700MByte❗️ for that same 20K+ article library, hence we should consider the Lucene 'database' as 'separate' from the rest of the Qiqqa library metadata (.library, .autotags, folder watcher, you name it)

Suggested storage layout

- documents     -- fixed hashed PDF store, no changes expected
- index         -- Lucene search index store; only migrate on Lucene changes
  - *           -- v79 and before Lucene v2.9.x search db files
  - v82/*       -- Lucene DB for when upgraded. Use Qiqqa version, 
                -- which is fastest changing
  - v97/*       -- future Lucene DB, yet another upgrade there
                -- (which happened with Qiqqa v97 release then)
- *             -- v79 and before Qiqqa library metadata: autotags, library, etc.
- v82/*         -- (v81 was experimental) for every Qiqqa version with ...
- v83/*         -- ... *any* format change, there's a version-named ...
- v84/*         -- ... directory for the metadata so you can rollback.
... etc. ...

+1

@jimmejardine: I won't be working on Qiqqa all the time, more in bursts (on and off), so it might be handy to collect a few folks to run the show, but if you like, you can put me on the list.

"object reference not set" hence a null pointer dereference somewhere :-(

It's not handling some modern (protected) PDFs very well.

BTW: is pdfdraw.exe this one: https://github.com/flexpaper/pdfdraw ? 🤔

[EDIT] Nope, it's MuPDF+pdfdraw.c; see ./libs/3rdparty/pdfdraw.c, ./libs/3rdparty/mupdf.changes.txt and ./libs/3rdparty/mupdf.zip. MuPDF is an open source lib available from Artifex, while the source repo is also available on GitHub at https://github.com/ArtifexSoftware/mupdf . pdfdraw.c (and possibly the other files as well) look like patched versions of files which are contained in mupdf.zip hence I expect them to exist in the ArtifexSoftware repo -- then the question becomes how to transfer the edits listed in mupdf.changes.txt and to see if those changes are still relevant...

1. 'Save Preferences' doesn't seem to work; every time Qiqqa opens another Sniffer, the standard tabs are opened and not the set which has been selected in this Preferences dialog.

2. 'PDFF' at the end of the tooltip for the Save button.

How to go there?
Qiqqa::Home > Configuration > Support Functions > LibraryDB Explorer 🍰

🎉 Revision: 6dcc971
DBExplorer severely enhanced:

• now supports wildcards in query parameters (% and _, but also * and ?, which are the MSDOS wildcards which translate directly to the SQL wildcards)
• now supports GETting multiple records.
• when GETting multiple records, DBExplorer not only prints the BibTeX for each record, but also the identifying fingerprint, verification MD5 and most importantly: the PARSED BibTeX (iff available) and BibTeX parse error diagnostics report.
• when GETting multiple records, the DBExplorer output is also dumped to the file Qiqqa.DBexplorer.QueryDump.txt in the Qiqqa Library base directory. A previous DBExplorer query report dump will be REPLACED.
• an extra input field has been added which allows the user to specify a maximum number of records to fetch: this speeds up queries and their reporting when working on large libraries with query criterai which would produce 1000nds of records if left unchecked.

This allows to use the DBExplorer as a rough diagnostics tool to check the library internals, including a way to find erroneous/offending BibTeX entries which may cause havoc in Qiqqa elsewhere.

servers don't necessarily return a clean "text/html" or "application/pdf".

See also: https://getsatisfaction.com/qiqqa/topics/qiqqa-premium-windows-qiqqa-does-not-recall-qiqqa-window-maximized-window-size

Qiqqa v79 Premium / Windows 10: qiqqa does not remember/restore qiqqa window maximized size, hence qiqqa does not correctly restore window size on app restart.

How to reproduce:

• start qiqqa (premium)
• maximize qiqqa window
• exit qiqqa
• restart qiqqa

This will correctly position the qiqqa window in the top/left screen corner, but the qiqqa window size will be wrong (too small) and not maximized.

(emphasis on the and there as you can store the window position+size, but when you don't store 'maximized/minimized' states as well, you will not recover the non-maximized window size when user is in the habit of maximizing/un-maximizing app window depending on user work activity.)

Aside question: is this purely a premium bug? (I don't remember if window position/size recall is a premium feature and am too lazy to check right now... ;-) )

[EDIT 2019/07] As the open source version of Qiqqa includes premium features, this is still applicable.

Qiqqa can load Qiqqa backups, but this is something entirely different: this is about recovery of Qiqqa libraries as they exist on your disk(s) — that's the qiqqa.library SQlite database file + documents tree + misc files:

Fundamental goal

We want to be able to recover and/or merge:

• Qiqqa intranet libraries (your local copies/mirrors(?) of your remote Qiqqa libraries)
• Qiqqa libraries where Qiqqa previously crashed or otherwise failed to perform and you threw your hands up, re-installed Qiqqa and started a new library afresh, keeping the old one around in hopes of one day being able to reload/recover that work: all the collected BibTeX info, annotations, etc.

Technically this means...

... that Qiqqa should be enabled to load any arbitrary Qiqqa.library file (an SQLite3 database BTW) and import / mix & merge(🤔) that data with the currently active Qiqqa library.

If that Qiqqa.library file is part of a directory tree, e.g.

$ ls base/INTRANET_0D4E8DC2-412B-461D-B359-63E40F1359B9/ 
-->
./documents/       <-- the PDF dir-tree kept by Qiqqa
./index/                 <-- the Lucene context text search kept by Qiqqa
./Qiqqa.aitag_blackwhitelists
./Qiqqa.autotags
./Qiqqa.expedition
./Qiqqa.library        <-- the main Qiqqa library DB
./Qiqqa.recently_read
...etc...

then Qiqqa should be able to load/import/merge that data into the currently active Library.

IFF possible, any record with fingerprint F should be matched with any PDF with fingerprint F located in the current library, thus having the ability to augment/recover metadata for your PDFs.

That assumes you imported all PDF files from that other library first via WatchFolders first but that's just how I go about it these days; the PDFs are in the new lib, but getting the old metadata in there with them is still a heck of a job. 😞

Additional wishes

1. Import/recover from a DB dump, i.e. create entire Qiqqa library record entries with fingerprint hashes and everything from augmented BibTex bib imports and dumps as produced by the tools at https://github.com/GerHobbelt/qiqqa-revengin : I have several such dumps from crashed/corrupted Qiqqa databases in text form: the metadata collected in there is still quite valuable as it is a lot of work to get it back any other way.

2. As a corollary this would also mean I'ld finally be able to sanity check my Qiqqa libraries: 'recovery' can also be applied to already loaded libraries where these failures have been observed with my own Qiqqa libraries:

   • PDFs have been downloaded into the library, but Qiqqa crashed before the accompanying record was updated in the library. In other words: detect unregistered PDF files in a Qiqqa library and add them to the database after all + report when such PDF files are found during a sanity check scan.

   • when importing a Qiqqa library (or what remains of it) DO NOT loose already present BibTeX metadata: do not overwrite but merge/append the metadata — this means we'll probably end up with another category of BibTeX records: any such merge which cannot be completed automatically should be hand-checked by the user via the Sniffer process. In the set "missing/skipped/auto/manual" I guess a fifth category is called for: "merge conflict". 🤔

   • when a sanity check reveals the metadata to be unparsable BibTeX (heck, we've quite a few of those already) the user should be able to go through just those records and patch them as desired — my preference is to have an augmented Sniffer window/process where I can filter/select just those records (6th category?? 🤔 😱 ) and edit them, not just in RAW form as Qiqqa currently supports but also in 'parsed BibTeX' form, the same way you can see and edit BibTeX entries in the document properties pane in the main view.

3. Export/dump an augmented BibTeX bib-like dump which includes fingerprints, source paths (note commit SHA-1: 84fe992 !), etc. so external tools/scripts can process these sets.

4. 🤔🤔 I guess that also means adding a CSL style which includes these items: I had a need for answering "where did I get that, back then?!" quite a few times the last couple of years. As Zotero is still, ah, unsatisfactory and the rest, well, crap, for doing stuff like that in bursts I'm stuck with Qiqqa.

</end of rant; taking my blue meds>

pops up dialog "There were problems...."

WatchFolder doesn't check if that particular hash is already queued and since OCR is done using multiple threads, you get trouble reports like these:

Report 1

The following files caused problems while being imported into Qiqqa:

Path already exists

W:\Sopkonijn!QIQQA-pdf-watch-dir!1!from-old-drive\B6318A57-2EF4-470D-A012-2332EEAE1754\documents\9\9B2CDB9E42AE3844E0DB5452A41E164E7D35473.pdf

Report 2

The following files caused problems while being imported into Qiqqa:

The process cannot access the file 'D:\Qiqqa\base\ocr\72\72276A1C84D037602145F4F1B49BABE5DCF43DE1.pagecount.0.txt' because it is being used by another process.

W:\Sopkonijn!QIQQA-pdf-watch-dir!1!from-old-drive\B6318A57-2EF4-470D-A012-2332EEAE1754\documents\7\72276A1C84D037602145F4F1B49BABE5DCF43DE1.pdf

Report 3 (multiple entries in a single file)

The following files caused problems while being imported into Qiqqa:

The process cannot access the file 'D:\Qiqqa\base\ocr\89\895F94F9442B89D50B3E5DCA4EEF3D4C61140DC.pagecount.0.txt' because it is being used by another process.

W:\Sopkonijn!QIQQA-pdf-watch-dir!1!from-old-drive\B6318A57-2EF4-470D-A012-2332EEAE1754\documents\8\895F94F9442B89D50B3E5DCA4EEF3D4C61140DC.pdf

The process cannot access the file 'D:\Qiqqa\base\ocr\89\899D82475A2E3919E6C9692AEC6EDEFDC99AAB2.pagecount.0.txt' because it is being used by another process.

W:\Sopkonijn!QIQQA-pdf-watch-dir!1!from-old-drive\B6318A57-2EF4-470D-A012-2332EEAE1754\documents\8\899D82475A2E3919E6C9692AEC6EDEFDC99AAB2.pdf

The process cannot access the file 'D:\Qiqqa\base\ocr\8A\8AFD2FF5FB5F23DC5832FD47E2CEC7ECC63A693.pagecount.0.txt' because it is being used by another process.

W:\Sopkonijn!QIQQA-pdf-watch-dir!1!from-old-drive\B6318A57-2EF4-470D-A012-2332EEAE1754\documents\8\8AFD2FF5FB5F23DC5832FD47E2CEC7ECC63A693.pdf

Path already exists

W:\Sopkonijn!QIQQA-pdf-watch-dir!1!from-old-drive\B6318A57-2EF4-470D-A012-2332EEAE1754\documents\8\8BF42674A38DCA6EF37B1ADD8577E77D79AD20A3.pdf

Exception thrown: Insufficient memory to continue the execution of the program

Running Qiqqa in the MSVC debugger and having a quick look at the memory footprint (snapshot inspection), it turns out that in this case the LDAStuff.DocProbability and LDAStuff.TopicProbability both are huge: Count > 3.5 million, Size > 70MB each.

The next memory hog (?) seems to be a WPF component: WpfTap.WpfVisualTreeService --> Dictionary component @ Count ~ 24K and Size ~ 35MB :: I bet that's the visual list of the entire library.

However, the LDAStuff is odd to me as I haven't used Qiqqa Expedition or anything else, except the Sniffer during this session.

Related material: https://academia.stackexchange.com/questions/41688/how-to-represent-and-cite-a-patent-using-bibtex

Goal

when using @datasheet, etc. Qiqqa should recognize it sufficiently to provide a default set of fields in the editor like it does for the @article and other standard BibTeX types.

Note the old comment about QiqqaOCR at https://github.com/GerHobbelt/qiqqa-open-source/blob/master/QiqqaOCR/README.md :

A separate .exe file had to be made because Tesseract had a memory leak, and so when spawned from within Qiqqa would slowly use up all RAM.

It would be worth checking at some point if they have fixed it so that this abomination might be removed. :)

DevTip: Also do global search for QiqqaOCR and pdfdraw as there are such beauties as these lingering in the codebase:

• 485: int SECONDS_TO_WAIT = 60; // MAKE SURE THIS NUMBER IS LARGER THAN THE NUMBER IN THE ACTUAL QiqqaOCR so that QiqqaOCR has time to finish up...!
• 525: int SECONDS_TO_WAIT = 210; // MAKE SURE THIS NUMBER IS LARGER THAN THE NUMBER IN THE ACTUAL QiqqaOCR so that QiqqaOCR has time to finish up...!
• 90: // Support windows-level error reporting - helps suppressing the errors in pdfdraw.exe and QiqqaOCR.exe
• 91 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms680621%28v=vs.85%29.aspx

🎉 👍 Got an experimental Qiqqa binary built using Microsoft Visual Studio 2019 today (2019/august/1).

Also using Syncfusion 17 instead of Syncfusion 14 as that is the release currently available as main choice at Syncfusion for MSVC2019 and I want to use my 'regular' MSVC version instead of having to dual-install or otherwise run an old MSVC setup in parallel on an already pretty loaded dev box.

happy news: despite many warnings and errors at the start of this activity a few days ago, I now have a working binary (at least, it works for me 😋 🤡 ) and able to modify the source code to hunt down the major showstopping Qiqqa bugs that block me from actually using my PDF collection the way I need!

From the debug stats log (v80 github release on Windows 10/64):

20190730.192655 ERROR [39] There was an error while resizing a PDF page image

Utilities.GenericException: Error while rasterising page 1 at 187.5dpi of 'D:\Qiqqa\base\Guest\documents\2\26ABFE782ECFA6E4ACC9F93138A353B6F3126.pdf' --->
System.Runtime.InteropServices.SEHException: External component has thrown an exception.
at Utilities.PDF.Sorax.SoraxDLL.SPD_GetPageBitmap(IntPtr hDoc, IntPtr hDC, Int32 nPage, Int32 nRot, Single fDPI)
at Utilities.PDF.Sorax.SoraxPDFRendererDLLWrapper.GetPageByDPIAsImage_LOCK(HDOCWrapper hdoc, Int32 page, Single dpi)
--- End of inner exception stack trace ---
at Utilities.PDF.Sorax.SoraxPDFRendererDLLWrapper.GetPageByDPIAsImage_LOCK(HDOCWrapper hdoc, Int32 page, Single dpi)
at Utilities.PDF.Sorax.SoraxPDFRendererDLLWrapper.GetPageByHeightAsImage(String filename, String pdf_user_password, String pdf_owner_password, Int32 page, Double height)
at Utilities.PDF.Sorax.SoraxPDFRenderer.GetPageByHeightAsImage(Int32 page, Double height)
at Qiqqa.Documents.PDF.PDFControls.PDFRendererControlStats.ResizedPageImageItemThreadEntry(Object arg)

Found while going through WebSearcherPreferenceManager.cs as part of work done on #44

Assumption

SCIVERSE is probably https://en.wikipedia.org/w/index.php?title=SciVerse&redirect=no --> Elsevier --> ScienceDirect --> https://www.elsevier.com/search-results?query=YOUR-SEARCH-TERMS-HERE&labels=all&page=1

https://github.com/tesseract-ocr/tesseract

Some times you want to go through the documents which have their BibTex metadata NOT set up 'Manually' or 'Auto' or 'Missing': then you're looking for the inverse of what the current radio button set is offering.

Side Note on UX/UI/Workflow

🤔 Not sure how to present this in a (novice-)user friendly way, but I do know I need and have needed this ability quite a few times: the Sniffer also serves as a very handy 'paging through the collection while updating or checking the metadata' type of 'library browser'. (Not its original purpose I suppose but that's a big part of my usual Qiqqa workflow: pick a subset of documents via Search, then use the Sniffer to go through them quickly as there you have both PDF preview and a (rough-ish, BibTeX) metadata view while you can zip through the set faster than when you hop down the selection list that resulted from the Search action alone.

HTR (1 example; this happens everywhere a Browser is shown in Qiqqa):

• Open Sniffer on document
• Click "Browsing" tab
• See the cruft below

Ditto for other tabs

It "helps" to have a slow Internet connection, e.g. Inet over mobile Portable Hotspot out in the boondocks. This cruft remains as long as there's arrived no web page (server response), which can sometimes take forever (mobile traffic isn't always that reliable, particularly outside the Big Cities 😉)

Exported set of themes for the given 'Expedition' run on a library (Expedition set to 15 themes, only Use Tags, not AutoTags):

1. learning
2. learning
3. learning
4. help
5. learning; education
6. manual
7. datasheet
8. help
9. manual
10. help
11. education
12. patent
13. learning
14. help
15. help

Note the duplicate entries. See also screenshot below: see left panel there

.

While pre-open-source, qiqqa supported WEB libraries - those that you could sync via the Qiqqa.com website. Support for these has been removed from the open source version,

Instead, if users want to sync their libraries across computers, or share their libraries with other users, they first need to get themselves a dropbox/googledrive account, and in it create a folder and then create an INTRANET library that uses that folder as its sync point. Access to that library is then controlled using dropbox/googledrive permission management.

There is surely a better name for INTRANET library now. I would suggest SHARED library...

had plenty trouble today with .NET serialization while working on #40 and Visual Studio Designer.

Pretty frustrating, hence it's high time for the Dandelion Patch:

While I don't mind working in MSWord, it's much easier to track progress and review edit diffs when the documentation source is plaintext, e.g. MarkDown. GitHub can turn that into a website using GitHub Pages and then the manual is available online.

There's tools out there that can turn the same docs into a PDF (GitBooks, pandoc, ...) so anyone who wants it in PDF format can have it too.

Suggested: create GitHub Pages from master branch /docs/ directory; IME this is easier to work with than having a separate gh_pages branch for it.

I've had this problem many times over the years with a 20K+ docs db. (using v76-80 (github release))

Sniffer layout looks a big b0rked, etc.

[EDIT]

Turns out Qiqqa cannot parse this (legal) BibTeX concatenation + string macro:

month=may # "~3"

and looses data due to the parse. (When you edit this BibTeX record in any way in Qiqqa, that month field will be damaged and only list month="may" afterwards!

Original issue text

An example (produced by enhanced DBExlorer code -- see commit 6dcc971):

BibTeX Formatted:
    @misc{graham1955audio
    ,	title	= {Audio frequency transformer}
    ,	author	= {Graham, Cecil R}
    ,	year	= {1955}
    ,	month	= {may}
    ,	publisher	= {Google Patents}
    ,	note	= {US Patent 2,707,773}
    }

BibTeX RAW FMT:
    misc(graham1955audio) - 6 fields

BibTeX Parse Diagnostics:
    
    Errors:
    
      Expecting a field name but nothing was acceptable, so moving onto next whitespace
    m, Cecil R},   year={1955},   month=may # "~3",   publisher={Google Patents},   
                                            ^
      Expecting a field name but nothing was acceptable, so moving onto next whitespace
     Cecil R},   year={1955},   month=may # "~3",   publisher={Google Patents},   no
                                            ^
    

BibTeX RAW INPUT:
    @misc{graham1955audio,
      title={Audio frequency transformer},
      author={Graham, Cecil R},
      year={1955},
      month=may # "~3",
      publisher={Google Patents},
      note={US Patent 2,707,773}
    }

@jimmejardine : was this once done intentionally?

HTR:

• open Sniffer
• click "Browser" tab
• enter search term (see screenshot example: "rs485")
• Qiqqa jumps back to Google Scholar tab

This does not happen for other tabs", e.g. "Arxiv" remains active and shows the search result. Ditto for "Google.com" but since #44 has existed for me for 'forever' (ever since I started using Qiqqa many years ago), I had kinda hoped "Browser" tab was an alias for "Google.com" tab.

At least, this is a little confusing, particularly when you have 'https://google.com/' open in "Browser" tab at the time.

(User activity where this happens: when I hunt for PDF files outside Google Scholar using search criteria shaped like "pdf ".)

I've seen all sorts of things go wrong with PDFs, but these should be dealt with more or less sensibly (as far as you can deal sensibly with a garbage input).

At least Qiqqa & QiqqaOCR MUST NOT crash, lock up or otherwise b0rk on bad input PDFs.

Debug log extract:

20190804.204323 ERROR [PDFTextExtractor] There was a problem while running OCR with parameters: GROUP "D:\Qiqqa\base\Guest\documents\9\97F3C6565FB76E8DF535D150CB43D08EC2E62517.pdf" 301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320 "C:\Users\Ger\AppData\Local\Temp\\TempFile.dd76b1ad-93ef-44fe-93e0-f5d0fa5f25ec.txt" "" ""
20190804.204323 INFO  [PDFTextExtractor] Parameters: GROUP "D:\Qiqqa\base\Guest\documents\9\97F3C6565FB76E8DF535D150CB43D08EC2E62517.pdf" 301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320 "C:\Users\Ger\AppData\Local\Temp\\TempFile.dd76b1ad-93ef-44fe-93e0-f5d0fa5f25ec.txt" "" ""
20190804.204323 INFO  [PDFTextExtractor] --- Standard output:
20190804.204322 INFO  [Main] Logging initialised
20190804.204322 INFO  [Main] Starting the text extract thread
20190804.204323 INFO  [Main] Both text extract and OCR have exited, so exiting
20190804.204323 ERROR [Main] There was an error in QiqqaOCR:
--- Parameters ---
GROUP D:\Qiqqa\base\Guest\documents\9\97F3C6565FB76E8DF535D150CB43D08EC2E62517.pdf 301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320 C:\Users\Ger\AppData\Local\Temp\\TempFile.dd76b1ad-93ef-44fe-93e0-f5d0fa5f25ec.txt   
--- Exception ---
System.Exception: We have no wordlist to write!
   at QiqqaOCR.TextExtractEngine.MainEntry(String[] args, Boolean no_kill) in W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\tooling\qiqqa\QiqqaOCR\TextExtractEngine.cs:line 111
   at QiqqaOCR.Program.Main(String[] args) in W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\tooling\qiqqa\QiqqaOCR\Program.cs:line 46


System.Exception: We have no wordlist to write!
   at QiqqaOCR.TextExtractEngine.MainEntry(String[] args, Boolean no_kill) in W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\tooling\qiqqa\QiqqaOCR\TextExtractEngine.cs:line 111
   at QiqqaOCR.Program.Main(String[] args) in W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\tooling\qiqqa\QiqqaOCR\Program.cs:line 46

--- Standard error:

20K PDF library. Coming from a v79 commercial install, this library has suffered badly from #16 in the past and a recompiled Qiqqa (with #11 fixed and #13 partly fixed) would now (#14) finally attempt to recreate that Lucene-backed search index, only to end up as 'Not Responding...' and spitting out several MBytes of logfile output carrying a zillion lines like these:

20190802.104554 INFO [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] Indexing document E6B963888DF9A4CCD5E2CD7647BFE94F692DF1

20190802.104554 INFO [PDFTextExtractor] PDFOCR:297 page(s) to textify and 1254 page(s) to OCR. (1/1551)

HTR:

• have a directory tree with many files and directories, which is watched by Qiqqa for new PDFs to occur.
• Quit Qiqqa

Logging will report something like this:

20190814.130928 [Q] DEBUG [Daemon.Maintainable:GeneralTaskDaemon.DoMaintenance] [187.176M] 
      Directory.GetFiles took 300433 ms

(augmented Qiqqa @ commit XXX)

Note the 300 seconds it took GetFiles() to complete.

Qiqqa reacts badly to broken and protected PDFs (e.g. several ON-SEMI/Fairchild datasheets)

Finding these in the library can be done using external tools (see https://github.com/GerHobbelt/qiqqa-revengin reverse engineering work in the past few years and some of the scripts I've written for this purpose: https://github.com/GerHobbelt/qiqqa-revengin/blob/master/deprotect_all_directories.sh et al).

However, IMHO, it's much better to have this type of diagnostic built into Qiqqa, as I then don't need to maintain a second bunch of tools and codebase to R/W the Qiqqa library database.

It would also allow for behaviour like this:

1. import a PDF, which turns out to be protected (sample PDF will be uploaded later)
2. discover this (and maybe others like it) in Qiqqa via new menu/command (feature request)
3. dump those files to a work directory where they may be processed using qpdf or similar tools. (modify using external toolchain)
4. import the 'deprotected' PDF files, have Qiqqa fingerprint them as usual
5. have Qiqqa link/relate both protected and deprotected copy of the PDF to the same BibTeX / library info!
   • Idea: step 5 can be done by giving the dumped PDF a special name which includes the original PDF's fingerprint, which can then be discovered on import of the deprotected copy.

Same type of behaviour as #17, either due to dropping a ton of PDFs in the Watch Folder or resetting / changing the Watch Folder, while pointing to a large set of PDFs.

For a 20K+ library, this can take ages and keeps going in the background "almost indefinitely" even after the user closed Qiqqa: this is undesirable behaviour.

The Qiqqa.log logfile gets littered with reams of this stuff:

20190802.180827 INFO [11] FolderWatcher file_system_watcher_Created
20190802.180827 INFO [11] FolderWatcher file_system_watcher_Changed
20190802.180827 INFO [11] FolderWatcher file_system_watcher_Changed
20190802.180827 INFO [11] FolderWatcher file_system_watcher_Created
20190802.180827 INFO [11] FolderWatcher file_system_watcher_Changed
20190802.180827 INFO [11] FolderWatcher file_system_watcher_Changed
20190802.180827 INFO [Main] Waiting for Maintainable Qiqqa.Common.GeneralTaskDaemonStuff.GeneralTaskDaemon:DoMaintenance to terminate.
20190802.180827 INFO [11] FolderWatcher file_system_watcher_Created
20190802.180827 INFO [21] FolderWatcher file_system_watcher_Changed
20190802.180827 INFO [21] FolderWatcher file_system_watcher_Changed

As reported by Tiffany here: https://getsatisfaction.com/qiqqa/topics/pdf-content-not-visible

I am using v. 78. I have a large library. Many of my pdfs display perfectly but some do not. If I have over the document, I still get the quick view. However, when I open the document none of the pages display, nor are they visible in the preview tab. The documents open fine in an external viewer.

Related issue report: https://getsatisfaction.com/qiqqa/topics/pdfs_stop_displaying_blank_pages

The only remedy is to reboot or kill -9 Qiqqa et al, i.e. open the Windows Task Manager as Adminstrator and end all Qiqqa processes. ('end process tree')

v79/v80 Qiqqa only records local file paths as source locs, but it is very handy to have the source URL for any PDF.

About updating that URL: I regularly import PDFs from my local Download directory, but when I go to the Qiqqa Sniffer, I might want not just want to grab the BibTeX from there, but also the source URL of the PDF.

Getting the source loc info out of Qiqqa is another matter still...

depends on #2

Double-click the XAML files and find that many fail to render with a reported Exception like Brush-blablabla-Blue-blalblBLA cannot be found, yadayada.

So we can go through the set which has their DownloadLocation set up incorrectly / not at all.

Given the code for Library.IsBusyAddingPDFs I assume Qiqqa has more multithreading risks like that (a timeout/delay check isn't going to fly at all times and with a large library over here, those 3 seconds isn't going to cut it, resulting in a Qiqqa lock up some of the time)

Random lock up behaviour like this strongly hints towards a race condition somewhere in the code, so I guess we'll have to go through the interlocking/thread interaction inside Qiqqa. 🙉 🙈 😄

Observations

1. These lockups almost always happen when you download a new/existing(?) PDF from the Sniffer into Qiqqa while both Qiqqa.exe and QiqqaOCR.exe are running.

2. Dialing down the number of QiqqaOCR.exe instances to 1(ONE) on my i7 dev box seems to reduce the chance of lockup like this happening, but it certainly is still present.

3. Using Library.IsBusyAddingPDFs in the Infrequent Background Task code seems to have minor negative = worse impact.

4. My work on SHA-1: 8a1d766 (Fixing #17) seems to have a strong detrimental = much worse effect as the number of lockup occurrences has increased notably since that change.

   However, since that commit fixed #17 and thus prevents me from short-circuiting to the Windows Task Manager kill -9=end process tree decision, this "much worse" is quite possibly purely psychological as Qiqqa is running much longer now, per session.

From Qiqqa forum:

Axel Sonntag: 2 months ago

Change the style of autogenerated bibtex codes

The bibtex sniffer is a great feature of qiqqa, but I manually have to change each bibtext code because I use a different format, e.g. instead of "smith1999gear", I would like to customize the autogenerated code to something like "Smith1999".

Having this customizability would improve my efficiency a lot!

Many thanks for considering...

HTR:

• open Expedition on a large library
• Refresh
• Hit Update / Look for citations to start that background process
• Wait a bit to see it work: statusbar at bottom is updated with messages
• Quit Qiqqa.

Qiqqa takes a long time before actually stopping in the background - sufficiently long that I kill -9 terminated it using Windows Task Manager.

Seems related to #17, #20 et al

Now that I have access to the Qiqqa source code and have been able to rebuild the binary and extend its logging, I find that quite a lot of my troubles in the past years is due to Qiqqa not coping well with all kinds of broken/b0rked PDF files in the Qiqqa libraries:

1. #10: several PDFs caused Qiqqa to run indefinitely after closing it: every time I had to open the Windows Task Manager and KILL thr Qiqqa process or process tree to make it stop. If I didn't do that, Qiqqa would report it's already running when you restart it, necessitating a reboot. Instead, I've executed the Windows equivalent of kill -9 every time I exit/stop Qiqqa.

2. #16: Qiqqa failed on several occasions with my large PDF collection, causing a permanent and total failure in its search feature, i.e. the Lucene database got nuked/b0rked. All subsequent searches in Qiqqa would deliver ZERO results, quickly.

   • Reindexing via the Qiqqa Tools panel would have no effect.

     Tools > Qiqqa Configuration > Troubleshooting > Rebuild Library Search Indices
     

   • Manually deleting all the Lucene DB files in base/Guest/index/ would also be to no avail.

   • Reconstructing the Library by importing the PDF files in tiny batches via the Directory Watch feature of Qiqqa would result in 'semi-random behaviour': it now turns out to be highly dependent on which PDF files got loaded first: as soon as an offending PDF (to be uploaded later) got included in the library, the Lucene-backed search facility would break down and stop to function.

   Note: Pending investigation suspects #11 at least; at the time of this writing #11 has been fixed and this was a required first step towards making the Lucene-backed search feature work and (re)generate a working search index once again.

3. When using the sniffer (Yay! 😄 Superb Feature!) to fetch additional documents (PDFs), sometimes you'll observe a load failure, where

   • the document occurs as pure white multi-page document with no content at all, or
   • the document would render as a single pure-white page document with no content at all, or
   • the PDF download/fetch operation would lock up and you'ld have to kill -9 Qiqqa to stop it. Depending on the alignment of the planets, you'll be able to restart Qiqqa with a functioning or broken 'search' feature then. Waiting on http://website/path.../file.pdf would be shown forever in the status line at the bottom of the main window.

4. There's no way to dig out these b0rked PDFs from the library and 'select all' the discovered culprits to apply some chosen user activity (delete PDF + library entry, export/dump to diagnostics directory, ...?what you want?...)

   • This is filed as #12, by the way.

Did a tree compare after I ran into a few Code Analysis reports from MSVS: it looks like those source trees are VERY similar and one may have been a copy of the other at some time.

The question is which of the few differences in the source codes there is leading, i.e. which one can be thrown away?

My initial bet is on the Utilities/... one as this is quite Qiqqa UI specific stuff IMHO, but then the Brainstorm material originated from earlier days, right?...