eyalroz / removedupes Goto Github PK

In setAppStringPref(), we're using BiDiMailUI instead of RemoveDupes as the global object. That's due to a mis-copy-paste.

I don't see any prefs for skipping "special" folders and other folder options and I don't see the pref for time resolution like in the old <=68 only version. Are all addon prefs supposed to now be accessible via the toolbar button?
Sorry if known bugs but I didn't see these explicitly in the issue list (except for one that might be referring to this).

Also, is there any explanation for the "Set original message folders(s) for the next duplicate search"? I don't understand what this means.

I built your "development" branch and don't see any prefs when I install it. Thought maybe it fixed the missing prefs but didn't see any prefs and the button never appeared.

I've been working on another project here https://github.com/smitgd/addon-copyfolder-tb-esr68 and wanting to adapt it to tb 78 like you have so I've been using yours as an example. The copyfolder addon is a lot simpler than yours but has some in common. I think between your removedupes and the copyfolder addon, they might provide a good workaround for a long standing and difficult to fix tb bug https://bugzilla.mozilla.org/show_bug.cgi?id=538375

A user reports:

Since the recent update to removedups (looks like it was pushed on Aug
8, my Thunderbird got it on Aug 11, and I only restarted it last night
and noticed the problem) the addon is no longer able to remove duplicate
messages with the "Delete Permanently" option. Looking in the error
console, it says that that this is because of the error:

 NS_ERROR_XPC_NOT_ENOUGH_ARGS: Not enough arguments [nsIStringBundle.formatStringFromName]

pointing at line 138 of remove-dupes-common.jz. Looking there I see it
was modified to remove the final (third) argument to
formatStringFromName(), with the comment "The third argument is dropped
sometime between TB 68 and TB 78". The problem is that I (like many
other people using a distro-built Linux version of Thunderbird rather
than an auto-update directly from thunderbird.net) am still running TB
68...

It would be useful if the same message sent to multiple mailing/distribution lists could be identified for removal.
Perhaps matching on the message subject and contents would be sufficiently unique.

Why this addon installation requires full access to anything on my computer ?!?

If a message is missing a necessary field (and we are not accepting substitutes for that fields, e.g. Thunderbird's generated MD5 hash for message IDs), we currently treat this field as identical (hash-wise) with other messages'. For message IDs, that's through a use of '' (not such a good idea in itself); for Subject and Author, we're placing null in a string (so probably a '(null)'. I'm assuming this isn't relevant to other fields.

This should be at most an option. By default, if a necessary field is missing (and it's part of the comparison criteria) - we must assume there is some unique value for that field, unknown to us - and thus never mark this message as a duplicate of any other one - even if the other message is octet-for-octet identical, on headers and body.

The status flags comparison criteria is missing from the prefs dialog, and only appears in the menubutton. Whoops!

Hello!
Cannot configure, because there is no "Options" page available for this add-on... Therefore it cannot find duplicates correctly, can you help?

This add on never finds any duplicates when there are some in my In box. At this rate I could delete manually and forget this add on....
Why can it not find the obvious duplicate emails in my In box???
I am using TB 77.7.1, FF 88.0
POP!_OS 20.04LTS

It works very well, but I have some comments.
A little bit funny for me, that in the window titled 'Duplicate messages Deletion" I check mark what I want to keep, and not what I want to delete.
Also, my default (but configurable) option and setting would be "Keep All Copies" and not "Keep First Copy" as it is now.
Maybe this default could be set in the dropdown menu under the Comparison Criteria.
I can't see where I clicked on the right side, all four keep options are equally gray, only the check marks change.
This can be confusing in case of many lines, you have to look through every line what is check marked.
Therefore the use of mixed check marks could also be indicated, for example as "mixed copy"
Also, if all keep All copies is set, OK will not work.

Thanks a lot for making this extension, just left a positive review!

Just wanted to let you know that the two links on https://addons.thunderbird.net/nb-NO/thunderbird/addon/removedupes/ in the sentence "(As Thunderbird is undergoing massive changes under the hood, always check out the releases page on the website's homepage for a beta supporting the latest Thunderbird version. You are also welcome to report issues.)" goes to another extension (bidimailui).

Also, it seems like many of the negative reviews are from users who haven't read the documentation. Perhaps it could help to add a very visible link to the FAQ at the top of that page?

Hello everybody,
I love this add-on and I even installed Thunderbird after switching to Outlook because I missed this add-on. And no, I am not lying. Why I moved to Outlook? it is mandatory at work and sooner or later you don't want to use two programs anymore.

Anyway, I have found that I have quite a few duplicates from backups, mailbox problems etc. But they are spread across 4 different accounts. Would it be possible to have this add-on looked in all accounts synced in Thunderbird? The data is on the computer anyway. And all work individually. But I do not like the idea of having to download all messages from all accounts to my local folder, then search for duplicates and then having to re-upload all mails to all accounts (This is how I lost quite a few mails a few years ago).

Any help would be very much appreciated.

Thank you very much and best regards
Alex

The lower entry includes all other Add-ons and is also connected to the keyboard shortcut "g".

Thunderbird V68.12.1 (32-Bit), German

I have non-exact match items, but for example the subjects and date match.

https://addons.thunderbird.net/en-US/thunderbird/addon/remove-duplicates had a separate dialogue for changing the duplicate criteria options.

Right now, I have everything duplicated :/

This:

  cycleDirectionSettings : function() {
    var messagePane = document.getElementById("messagepane");
    var body = messagePane.contentDocument.body;
    // ... etc etc

used to work (and maybe still works in some contexts), but it sometimes, or always, fails:

TypeError: messagePane.contentDocument is null

Note: This code is also used in forceDirection().

Hi! It looks like Thunderbird is no longer hosting the link to download your extension. Could you please offer an alternate location ?

2008.zip

Problem with many same emails
It doesn't find any duplicates

After changing criteria, removing messageid, it found some but not all.
EDIT: WORKED!
Author, Recipient, Time(in secs), Subject

chrome.manifest has the following entries:

style chrome://removedupes/content/removedupes-dialog.tb68.xul chrome://removedupes/skin/style.css appversion>=68.0
style chrome://removedupes/content/removedupes-dialog.xul chrome://removedupes/skin/style.css

these are invalid, as there is no file named style.css in the (classic) skin folder. Also, the dialog itself has a style attribute which loads the actually-relevant css file, removedupes-dialog.css.

Hello Eyal,

I have been using your plugin for months with great satisfaction. But recently (I don't know exactly when, maybe about 2 weeks ago), it stopped finding any duplicate message. Typically, I've got :

• mails sent to me and to a mailing list I belong to,
• mails I send to a mailing list that I also belong to.

Before, these duplicates were all found.

Here are the configs I use (the first 2 are not managed by me, this is why they are not the latest version of TB):
ThB 60.2.1 64b / Linux / Plugin 0.4.1
ThB 78.5.0 32b / Win10 / Plugin 0.5.1b2
ThB 78.5.1 32b / Win10 / Plugin 0.5.1b2

For the Win10 configs, our IT department asked us to enable the option "security.tls.version.enable-deprecated". For the rest, I think that we are using a standard config.

Of course, I can provide you with any other information that you would need.

Cheers,
Didier Pieroux

Hi,
first of, thank you for this great Add-On. It's actually the main reason I am still using Thunderbird.

What I have found is that my entire Thunderbird locks up when I use your add-on on large folders.
for example, I just tried to compared a folder with 45000 mails with one with 50000 mails. Thunderbird stops reacting after a few seconds and then has to be shut down (or shuts down itself after a while).

I am guessing there is some issue in how Remove Duplicates tries to compare and use memory. I would not mind waiting 5 min or more. I have time. But it is a bit of a problem that it will always crash Thunderbird. Everytime.

I hope you can find a fix. Let me know if I can supply you with some addition information :-)

Best regards
Alex

I had occasion to attempt to use the older release today only to find that the menu item was missing and the add-ons manager reported it was incompatible.

I clicked on the homepage link which to me to the github page. After looking things over I downloaded the update and installed it. Unfortunately I am getting an incompatible message for this one as well. Reading the details I see "There's also a SeaMonkey version, though not as frequently tested." which leads me to wonder if I installed the wrong version. I downloaded the one here:

https://github.com/eyalroz/removedupes/releases/download/v0.5/removedupes_0.5_tbird.xpi

Do I need to download another version for SeaMonkey 2.53.5.1 from elsewhere?

Hi...

I tried to send an email to the link you provided but it bounced. That address is on:
https://github.com/eyalroz/removedupes/

We are using TB 78.7.1 on Kubuntu Linux 20.04. (We have been using Thunderbird for at least 15 years and have always been trying to solve this problem, but as of a couple months ago we made a big jump to new workstations with new Thunderbird and so I would really like to find an answer to this as I have to otherwise manually "walk" through the entire IMAP directory structure for each seat!)

We have a directory structure probably up to 12 levels deep in some places and, in total, containing many tens of thousands of emails.

I am NOT concerned about finding or removing duplicate emails.

I only want to build .msf files. But...

I found a years-old search result saying that a then-existing (but not now?) "find-duplicates" add-on would, as a by-product of the process, build all the .msf files in the user's TB profile, including for a new TB installation.

On our system, the "/public" files are on our IMAP server and each user's Thunderbird profile contains only .msf files.

We need all the .msf files to be present in each person's TB profile so that other add-ons, such as Nostalgy++ or QuickFolders, can "see" the directory structure and mailbox file names and thus can file into those mailboxes.

I have not been able to find any Thunderbird automated method to build all the .msf files. The only method I have found is to manually use the GUI to "open" each folder and thus expose the mailboxes -- which then causes Thunderbird to create the .msf files.

So, I am wondering... in the process of doing its job, does your program create .msf files for the entire IMAP structure?

(I don't care about message counts of the files IN the mailboxes. I just want the .msf files to be created FOR Each mailbox.)

Thanks,

Jay Smith

'Remove Duplicate' works OK on my IMAP Inbox, and says 'No duplicates found' in several local folders. However, it fails in two local folders. One is named '2011' and contains 887 messages. The other is '2009', with 197 messages. In both cases, after about five seconds a window title appeared for about a second, saying "Mozilla Thunderbird (not responding)". No dialog appeared. I found the following in the error console:

Uncaught InternalError: too much recursion
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
removedupes.js:409:28
    RegExpGlobalMatchOpt self-hosted:2033
    Symbol.match self-hosted:1997
    match self-hosted:2931
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    next self-hosted:1099
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
Uncaught InternalError: too much recursion
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
removedupes.js:409:28
    RegExpGlobalMatchOpt self-hosted:2033
    Symbol.match self-hosted:1997
    match self-hosted:2931
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    next self-hosted:1099
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
Uncaught InternalError: too much recursion
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
removedupes.js:409:28
    RegExpGlobalMatchOpt self-hosted:2033
    Symbol.match self-hosted:1997
    match self-hosted:2931
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    next self-hosted:1099
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
Uncaught InternalError: too much recursion
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
removedupes.js:409:28
    RegExpGlobalMatchOpt self-hosted:2033
    Symbol.match self-hosted:1997
    match self-hosted:2931
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    next self-hosted:1099
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332

Thunderbird 78.4.1
RemoveDupes 0.5
Windows 10 Home 1903

When I upgraded TB to v78 (currently 78.0.1), it disabled the extension and showed a warning message "Remove Duplicate Messages (Alternate) is incompatible with Thunderbird 78.0.1.". I use TB on macOS 10.14 in case it matters.

Thanks for your work!

This extension is very useful but occasionally misses duplicates when filtered on the body.
When filtering on the message ID, the Duplicate Message Deletion window will show
several messages with slightly different number of lines. Usually only off by one or two.
Diffs of the message bodies does not show any differences in the body lines.

As a sample of the messages:
https://mail.openjdk.java.net/pipermail/kulla-dev/2020-September/002590.html
https://mail.openjdk.java.net/pipermail/compiler-dev/2020-September/015078.html
https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-September/069405.html

Hello Eyal,
.
Unfortunately, your programme did not find any duplicates, although I even duplicated another e-mail to be sure.




.
How can I install your beta version?
.
Best regards
.
[email protected]
.

I'm trying to search for duplicates in an Archives folder with many sub-folders but I keep getting the message above "Not searching special folders". This is not a search folder this is a regular folder in an email address called Archives.

Oddly, in another email address, searching the Archives and it's subfolders. works fine

Recently released Thunderbird 78.0 has a new compose interface. I believe this is what broke the extension.
There are no RTL buttons when you compose a message.
See attached screenshot.

Machine details:
Windows 10 Pro (Version 2004 19041.388)
Thunderbird 78.0
BiDi UI version: 0.10.1b2

Thunderbird 78.4.3, Plugin 0.5, several IMAP accounts

When I fire up the search for duplicates on folders containing different number of messages the plugin stops at one point to give a response at all.

The problem is not directly tied to the number of messages alone.
Some folders with ~20000 messages work, another with 12000 works not, thus content and or size of the messages seems to play a role.

I noticed after starting a dupe-search on a larger folder that Tunderbird gets unresponsive after ~5-6 seconds (windows renders the unresponsive message):

If this happens there comes no response from the plugin at all - neither mentioning "no dupes" nor the dupes dialogue.

It looks like the plugin hits a timeout and gets terminated if the search takes too long. I noticed this also in earlier versions (years ago). Because of that I switched to the competitor "Remove Duplicates" plugin that was working in TB 68 for a while.

Is there a way to avoid this behaviour? Maybe by splitting the search into smaller chunks that take less time?

I got this console output:

Uncaught InternalError: too much recursion
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
removedupes.js:409:28
    RegExpGlobalMatchOpt self-hosted:2033
    Symbol.match self-hosted:1997
    match self-hosted:2931
    stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
    sillyHash chrome://removedupes/content/removedupes.js:511
    populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
    next self-hosted:1099
    processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332

While Writing this I notice it is coupled/duplicate of #26

At some point between v68 and v78 (probably 78 itself), all .xul files got renamed into .xhtml, and we must now only load/inject .xhtml files. Of course, for continued v68 compatibility, we still need the .xul's.

Thunderbird has provided - for quite some time now - services with easy lazy getters of references to several popular services - in Services.jsm, (all-Mozilla-relevant) and MailServices.jsm (Mail-andnews-relevant). Let's use those whenever relevant instead of defining our own getters and referencers.

Add-on installs on daily without problem But:

1. If TB is restarted add-on's ICON (in top RH corner) is not shown and no entries in tools menu. Diable and re-enable the add-on and the icon appears and tools menu has entries for "Remove Duplicates"
2. Add-on finds and marks all duplicate files in folder but the delete does nothing and error message displayed "Failed erasing messages"
   js console log attached
   console-export-2020-12-30_21-26-58.txt

When I tell it to delete duplicates by moving it to the trash folder, it actually moves it to the Local Trash folder. Emails are then not deleted and get redownloaded. Even though I told Thunderbird where the trash bin was.

I recently needed to run a massive de-duplication of emails (Over 16k duplicates in 20k emails due to a technical issue).
FYI: It's a newly installed version of Thunderbird (v78.7) running on Windows 10.

The 1st attempt worked brilliantly and did a good job of clearing out the duplicates.

I then needed to re-download a few emails which were still being held on our ISPs servers. I knew this would re-introduce a few duplicates, but I wasn't concerned as the 1st clean-up had gone so well.

However, when trying to run the de-duplication a 2nd time (to remove these newly downloaded duplicates), it said there were "No duplicates found".

This is odd as I can literally see them on screen (See the screenshot below), and haven't changed any settings since the last scan was run.

I tried restarting Thunderbird just in case it was a glitch, but the add-on still says it can't find any duplicates... even though I'm looking directly at them.

Everything is exactly the same, including the time.

Any ideas?

Instead of surrounding every debug-logging with preprocessor directives, we could centralize them in one place - a wrapper for console.log() within the RemoveDupes object - and have that wrapper be aware of which DEBUG_XYZ values were set on compilation. Specifically, those could be strings in a set / entries in a dictionary, which would be looked up before actually calling console log; and the wrapper would be invoked with the relevant string.

In the past, Seamonkey and Thunderbird had an interface for arrays of objects with basic instrumentation to recast them as different types - nsISupportsArray. This no longer exists, it was completely supplanted by nsIArray, see this bug page. We can now drop the code for the old kind of arrays.

The link indicated in the README.md file and the repository front page (i.e. remove-duplicate-messages-alte) seems to not work at this time.

Maybe can this issue be related to the recent changing of name? Or does the extension been removed?

Hello,

I've installed the addon to my thunderbird (V89.0b1, 64bit) running on Windows 10 pro.
But now, I'm unable to find either the button to add to Thunderbird or find the context menu option. Both of them are not showing up:

What do I do now to use the extension?

Thank you

Hey. Excellent software. Can you add a killer feature? Filter duplicates by message body using regular expression? One more condition. It would be just great. But I hope you have time for this. Regards, Oleg.

It used to be somewhat of a big deal to get access to the console - but it hasn't been already for quite a while. So instead of calling RemoveDupes.JSConsole.logStringMessage() (and defining it) - let's just use console.log().

I got an error "Uncaught TypeError: folderMessageHdrsIterator is undefined" when I ran Remove Duplicate Messages on a folder on an Exchange server (outlook.office.com). The add-on works with out any error on my another account using IMAP. Is this add-on supposed to work with an Exchange server?

Using Thunderbird 78.7.0 (64-bit) on macOS 11.2

Setting extensions.removedupes.skip_special_folders to false does not seem to have an effect on removedupe's behavior.

The reaction is the same as with extensions.removedupes.skip_special_folders = true

Regards
Heribert

We have this line in removedupes.js:

      var GetSelectedMsgFolders = GetSelectedMsgFolders();

bad idea; it's confusing. Let's rename the variable.

As suggested I installed 0.4.1 suiterunner on my SeaMonkey 2.53.5.1 x64 linux here. After a restart I now have the "Remove duplicates..." menu item on my mail folders. When I try to use it I am seeing inconsistent results. On some I see some quick status messages and then a pop-up window "No duplicates found." On others I see a final status message "Duplicate message search complete." but no pop-up window of any kind. These are the folders I suspect I have duplicates in. One of these has 34280 messages and the other has 13441 messages. Both of these are for POP accounts. Can I get this to work? Dave

This extension is now the only dupe remover to work with TB 78, after Dillinger "retired" his extension. It seems it's about time to drop the "(Alternate)" from the extension's name and claim the position of the main extension with this functionality.

I'm getting this Uncaught InternalError: too much recursion. It's folder with quite a lof of mail ca. 40.000. But sometimes it also works.
Effect is, nothing happens, no duplicate window opens nor message no duplicates is shown

Uncaught InternalError: too much recursion
stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
sillyHash chrome://removedupes/content/removedupes.js:519
populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332
removedupes.js:409:28
RegExpGlobalMatchOpt self-hosted:2033
Symbol.match self-hosted:1997
match self-hosted:2931
stripAndSortAddresses chrome://removedupes/content/removedupes.js:409
sillyHash chrome://removedupes/content/removedupes.js:519
populateDupeSetsHash chrome://removedupes/content/removedupes.js:631
next self-hosted:1099
processMessagesInCollectedFoldersPhase2 chrome://removedupes/content/removedupes.js:332

Sometime between TB 68 and 78, the menubutton XUL element was reworked, to use the is= attribute. For our menubutton to continue working in 78 and later, we must add is="toolbarbutton-menu-button" to its attributes.

Intro:
This is a continuation of an earlier ticket:

Would it be possible to add a new feature where you can select if you want to keep/delete an email based on whether it's plain text or HTML?

Please open a new issue about that.

Originally posted by @eyalroz in #43 (comment)

Summary:
During a recent issue, I noticed I had duplicates which I could not filter out accurately as 1 version of an email would be Plain Text and 1 version would be in HTML.

To try to solve this, I tried to "sort" the duplicates using either filesize or line-count, on the assumption that the HTML version would in theory be the larger of the two. However, that didn't always appear to be the case. In over 1,400 duplicates, I would estimate around 20% had plain text emails with greater line-counts or filesize (I haven't figured out why yet)

Desired action:
It would be nice if, after running a de-dupe, you could set which email format you wanted to keep/delete (i.e. keep the HTML version of any duplicates)

Thanks

In content/removedupes.js, we use:

extensions.removedupes.allow_md5_id_substitute

but in defaults/preferences/removedupes.js, we set:

extensions.removedupes.comparison_criteria.allow_md5_id_substitute

There is no benefit (that I know of) in using .jar files within the extension's XPI - and it hinders the transition to TB 78 support. So, let's drop the use of JARs.