bepaald / signalbackup-tools Goto Github PK

I am busy too, so it's not a problem. Thanks a lot for your answer. Maybe there is a way to completely reverse all the messages in the friend's backup and then cropping the backup? I will open an issue here as soon as I have free time. Maybe, if reversing messages is possible in friend's backup, then there is no need to crop it, I'll just have to reverse messages and restore the whole backup. But the question is how I can do that.

Originally posted by @HiperFall in #9 (comment)

I used the --importthreads ALL option to merge two backups, more specifically I imported a 500MB Backup into a 35MB Backup and everything pretty much worked like a charm.
The only error messages I saw were some

SQL error after sqlite3_step(): UNIQUE constraint failed: remapped_recipients._id
SQL error after sqlite3_step(): UNIQUE constraint failed: remapped_recipients._id
SQL error after sqlite3_step(): UNIQUE constraint failed: remapped_recipients._id

for every thread that was imported.
This didn't seem to harm the output though, as the newly created database worked perfectly when restoring it on a new phone.

Thanks a lot for this useful utility!

When attempting to build on macOS (Darwin/Mach), despite my best efforts, the following occurs:

backupframe/../common_be.h:120:10: error: no matching function for call to 'bytesToHexString'

I haven't been able to figure this one out. It happens on every file that uses that function, which seems extensive. Any guidance is appreciated.

Thanks,
Michael

I haven't seen any user having the same problem, but me - when exporting xml or csv = the contacts area is empty.
In case of xml file - it says contact_name="null"
Tried both Signal app versions:
Stable 4.72.5
Beta 4.72.6
When i restore the backup file inside the app itself - everything works as it should

I was using latest Windows precompiled app (20200913)

On Debian to build I had to

apt install libsqlite3-dev libssl-dev g++

and to invoke BUILDSCRIPT.sh I had to use bash because Debian's /bin/sh is provided by dash and the script has some bashisms in it. Debian Buster is g++ 8 and clang 7, bullseye (still in dev) g++ 10 and clang 9.

I think it would be great if the HTML export in the future created a index.html which is styled like the Android conversation list, and more HTML files one for each conversation. The styling should be easy to do with some simple CSS.
Might be able to contribute in the future to write the CSS.

hello,

i want to simply extract the current profile image, unfortunately it is not included in the backup or just simply on filesystem and still can not any hint about where it is stored, does someone know how to get the picture?

thanks in advice & best wishes

Hey, i kinda have a bit of a problem here udnerstanding a few things.

I try to merge two chats of the same person who changed number and therefore has a new chat now.

I already got it as far to change the id and the thread id of all the old messages in sms and mms table, but i am unable to change that for attachements (images, videos, etc). I cant find anything in the database that seems to identify which chat a part? belongs to.

Would be great to know on how to change this so i have everything from both chats in one chat again.

I happen to have had a backup that missed the last 8k (due to the first 8k being duplicated during transmit). That basically means no END frame, causing the backup read to fail. However, I'd still like to keep whatever is there, so I made a tiny change in exporttodir.cc:

   std::cout << "Writing EndFrame..." << std::endl;
   if (!writeRawFrameDataToFile(directory + "/End.sbf", d_endframe))
-    return false;
+    ;

...which essentially doesn't exit if the end frame couldn't be written (due to not having been read before). That allowed me to copy the regular end frame (which does not contain any information...), and thus keep as much as possible from my backup.

I'm sure you know your code better to find a more suitable way to fix this, but this is what saved my day.

--dumpmeida can not be combined with --output, and it also hasn't at all been documented in readme

The readme is also missing the syntax for Dump unencrypted database to disk

After running sudo dnf install gcc-g++ openssl-devel sqlite-devel which && sh BUILDSCRIPT.sh, it builds, then links, then produces the following error

/usr/bin/ld: /tmp/ccbZb7wo.ltrans12.ltrans.o: in function `main':
:(.text.startup+0xe8f): undefined reference to `FileDecryptor::strugee(unsigned long)'
collect2: error: ld returned 1 exit status

Any idea why this might be?

For reference, this is on a fresh Fedora VM install (Fedora 34, Workstation Edition), and the software versions from dnf include g++ 11.1.1, OpenSSL 1.1.1k, and SQLite 3.34.1.

I have a broken backup and want to fix it but the program stops with some error.

As proposed I'll attach a full output of the program.
signalbackup-tools_log_20200503.log

At the risk of being an idiot, but is it possible to extract media with this tool asl well? I managed to export to csv en xml so I am perfectly happy.

I did manag to extract media with Signal-back tool but that one was giving me problems with the xml. So after using both I was pretty much everything I need.

But still wondering if your tool is doing media as well?

Ignore this first part. I manged to solve this on my own. See second post below.

I keep having issues trying to restore my backup in Signal, saying it's the wrong passphrase, but I don't think it's wrong. To be fair, my backup file is nearly 5GB, and there's a possibility of corruption. I'm trying to get the messages out to restore to my phone without all the media. I would like to get both, but messages are the priority. I get the following error when trying to read my backup file:

Reading backup file...
FRAME 85889 (099.9%)...  STOPPING BEFORE END OF ATTACHMENT!!! (EOF)
Failed to get attachment data for FrameWithAttachment... info:
Frame number: 85890
        Size: 15
        Type: ATTACHMENT
         - row id          : 7519 (8 bytes)
         - attachment id   : 1617254927478 (8 bytes)
         - length          : 5148917 (8 bytes)

done!
WARNING EndFrame was not read: backup is probably incomplete

I also ran --listframes after and it's been running for several minutes with stuff along the lines of the following:

Reading backup file...
FRAME 85890 (100.0%)... Failed to read 4 bytes from file to get next frame size... (-1 / 4294967295)
Failed to read next frame (0 bytes at filepos 4295633025)
Starting bruteforcing offset to next valid frame... starting after: 665729
Checking offset 270 bytes
GOT GOOD MAC AT OFFSET 277 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 27 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 580 bytes
GOT GOOD MAC AT OFFSET 589 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 58 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 860 bytes
GOT GOOD MAC AT OFFSET 861 BYTES!

The numbers for bytes and frames is different each time, but it's quite a lot. Could this mean my backup is very corrupted, or that the backup password might just actually be wrong?

I was in desperate need to merge two Signal backups and am thus extremely grateful for your tool! ❤️

Merging two backups with db versions 105 and 112 failed at first with no such column: thread_recipient_id. It was however obvious that this is a database issue. When I imported the first one into Signal and rewrote the backup with db 112, merging worked like a charm.

• searching through conversations works
• "all media" works
• I don't have any voice messages, so I can't test those

Hello, when compiling under Fedora Live 30 I'm getting the following error:

LINKING: /usr/bin/g++ -Wall -Wextra -Wl,-z,now -O3 -s -flto=4 "cryptbase/o/cryptbase.o" "cryptbase/o/getbackupkey.o" "cryptbase/o/getcipherandmac.o" "endframe/o/statics.o" "filedecryptor/o/getframe.o" "filedecryptor/o/initbackupframe.o" "filedecryptor/o/getattachment.o" "filedecryptor/o/filedecryptor.o" "arg/o/arg.o" "sharedprefframe/o/statics.o" "avatarframe/o/statics.o" "attachmentframe/o/statics.o" "fileencryptor/o/encryptattachment.o" "fileencryptor/o/encryptframe.o" "fileencryptor/o/fileencryptor.o" "sqlstatementframe/o/statics.o" "sqlstatementframe/o/buildstatement.o" "signalbackup/o/importthread.o" "signalbackup/o/buildsqlstatementframe.o" "signalbackup/o/croptothread.o" "signalbackup/o/exportbackup.o" "signalbackup/o/updatethreadsentries.o" "signalbackup/o/signalbackup.o" "o/main.o" "backupframe/o/init.o" "sqlitedb/o/sqlitedb.o" "databaseversionframe/o/statics.o" "headerframe/o/statics.o" "stickerframe/o/statics.o" -lcryptopp -lsqlite3 -o "signalbackup-tools"
lto-wrapper: schwerwiegender Fehler: execvp: No such file or directory
Kompilierung beendet.
/usr/bin/ld: error: lto-wrapper failed
collect2: Fehler: ld gab 1 als Ende-Status zurück

I did the following steps on a Fedora30 on hardware:

$ git clone https://github.com/bepaald/signalbackup-tools.git
$ sudo dnf install gcc-g++ cryptopp-devel sqlite-devel
$ cd signalbackup-tools && chmod +x BUILDSCRIPT.sh
$ sh BUILDSCRIPT.sh

Can anyone help? My linux skills are limited.

Hi, I used your tool (thank you!) to merge two Signal databases together. One was an old backup from Aug 2019 and had a database version of 11 and the other I made yesterday on database version 80. I wiped my Signal data and restored the old database into Signal to update it since you said that version difference would not merge well.

I initially tried to use my older/smaller database as the first_database and my newer/larger database as the source_database. This merge would not import into Signal and it took longer due to the iterative nature of the script since it went through the whole larger database for each thread in the database. When that failed I flipped it so the newer database was my first_database and my older database was my source_database. It took about half the time and the Signal app was able to import it.

I am not sure why the first merge would not import. I realized I didn't have a .backup extension on it and maybe Signal didn't like that. It didn't give me any errors or indication that it wasn't going to import the merged database it simply went to a new blank conversation window. I tried it a few times.

Thanks again! I appreciate it. You have any places to receive donations?

Exporting backup to 'signal-merged-2020-11-07-10-01.backup'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing SqlStatementFrame(s)...
  Dealing with table 'sms'... 55612/55612 entries...done
  Dealing with table 'mms'... 12434/12434 entries...done
  Dealing with table 'part'... 63/3350 entries...Warning: attachment data not found (rowid: 63, uniqueid: 1535921771541)
  Dealing with table 'part'... 590/3350 entries...Warning: attachment data not found (rowid: 594, uniqueid: 1552673824961)
  Dealing with table 'part'... 873/3350 entries...Warning: attachment data not found (rowid: 877, uniqueid: 1560209088426)
  Dealing with table 'part'... 877/3350 entries...Warning: attachment data not found (rowid: 881, uniqueid: 1560216250670)
  Dealing with table 'part'... 2184/3350 entries...Warning: attachment data not found (rowid: 2193, uniqueid: 1598130700405)
  Dealing with table 'part'... 2636/3350 entries...Warning: attachment data not found (rowid: 2649, uniqueid: 1604445062856)
  Dealing with table 'part'... 2929/3350 entries...Warning: attachment data not found (rowid: 2942, uniqueid: 1527010679847)
  Dealing with table 'part'... 3028/3350 entries...Warning: attachment data not found (rowid: 3041, uniqueid: 1531845121343)
  Dealing with table 'part'... 3039/3350 entries...Warning: attachment data not found (rowid: 3052, uniqueid: 1532717777308)
  Dealing with table 'part'... 3062/3350 entries...Warning: attachment data not found (rowid: 3075, uniqueid: 1533340237078)
  Dealing with table 'part'... 3350/3350 entries...done
  Dealing with table 'thread'... 116/116 entries...done
  Dealing with table 'identities'... 9/9 entries...done
  Dealing with table 'drafts'... 5/5 entries...done
  Dealing with table 'push'... 2/2 entries...done
  Dealing with table 'groups'... 36/36 entries...done
  Dealing with table 'group_receipts'... 4452/4452 entries...done
  Dealing with table 'job_spec'... 0/0 entries...
  Dealing with table 'constraint_spec'... 0/0 entries...
  Dealing with table 'dependency_spec'... 0/0 entries...
  Dealing with table 'sticker'... 53/53 entries...done
  Dealing with table 'recipient'... 370/370 entries...done
  Dealing with table 'storage_key'... 0/0 entries...
  Dealing with table 'key_value'... 0/0 entries...
  Dealing with table 'megaphone'... 7/7 entries...done
  Dealing with table 'remapped_recipients'... 0/0 entries...
  Dealing with table 'remapped_threads'... 0/0 entries...
  Dealing with table 'mention'... 0/0 entries...
Writing SharedPrefFrame(s)...
Writing Avatars...
Writing EndFrame...
Done!

Thanks for making this! I have messages in Signal going back to 2011, and wanted to make sure I had a safe non-signal backup. (I imported all my old SMS into Signal when I first started using it, so 11 years of messages, and yes, I am a packrat!)

I noticed that there's not a lot of documentation for the various command line options. Or am I looking in the wrong place? Is there a single reference?

If not, would you be able to answer some questions as I try to make one?

Does anyone know what the encoding for the reaction emojis is?
It seems to be a 32-length char string like
(censored)

I ran this today on my laptop running Windows 10 x64 version 2004 and it does read the entire backup. But when it says its exporting to the specified file it throws 4 identical errors (in my case anyway). This is what I get at the end:

Reading backup file...
FRAME 84066 (100.0%)... Read entire backup file...

done!

Exporting backup to '.\Signal-decrypt-sms.xml'
ERROR Failed to retrieve required field 'address'
ERROR Failed to retrieve required field 'address'
ERROR Failed to retrieve required field 'address'
ERROR Failed to retrieve required field 'address'

Any idea what this could be? It has me scratching my head as to what could cause it.

Upon inspection of the file properties, I see that theres something in the .xml however idk if there is anything missing.

Have a good night,
T. H.

Briefly a "skype" icon pops up on the task manager, goes away after a few seconds after clicking the .exe

Program doesn't appear to launch.

Using latest version of windows 10.

I am trying to migrate signal to a different phone, but backups created by signal are broken. Therefore i tried to recover the backupfile with this tool. It works, i can properly decrypt the file (If i inspect the sqlite file, everything seems reasonable complete)

However, if i try to rebuild the file with the same password, it strips basically every attachment (8mb instead of ~220mb)

That is fine, i still can archive the attachments since they are beeing decrypted. Sadly, the new file wont restore. The only difference is that now signal recognizes my 4000 messages but continues to fail after that (With reporting that i have used a wrong password)

Any suggestions why this fails?

The tool you wrote works wonders, i have now access to my data, but getting it back into signal would be amazing.

I can provide logs if you need them

Downloaded the Signalbackup-tools_win and attempted to run the .exe file to get the backup working. CMD flashes on the screen and then nothing happens. Please advise.

Hi,
my problem: I have one backup with the thread id 203 (database version 85) which contains ALL messages until 25th of december 2020. Then in a another backup with the same id (database version 89) I have every message since 26th. I want to merge both threads to one long one. Is this actually possible as I dont succeed at all not matter which one I choose as source or first database.
Thx a lot!
gerega

• Ran the code on bash and also on windows (using the latest static build given in the repository )

$ bash BUILDSCRIPT.sh
./signalbackup-tools --output signal_backup/ signal-YYYY-MM-DD-HH-MM-SS.backup <PASS>

• The program read the backup file successfully but when export started it printed this :

Exporting backup to 'signal_backup/'
Writing HeaderFrame...
Failed to export backup to 'signal_backup/'

Please do not delete tags; versioning and build repeatability are important aspects of a configuration management strategy afforded by tagging.

I ran the merge function as detailed in the documentation and it worked perfectly! For reference, the two database versions I merged were from versions 80 and 89 respectfully.

The source currently uses very, very recent C++ constructs, requiring really, really new compilers. I'm not sure -std=c++2a actually means a released C++ version. To allow other people to compile this, I'd reduce the requirements to C++17. This is provided by clang++-6.0 and g++-8 from Ubuntu 18. Patch attached, including simple Makefile.

Hi, this tool is really useful! Would it be possible for you to commit the tooling that generated the build script?

Hi,

I tried your tool today to see if I could fix a somewhat broken signal backup file. The original file is shown with a size of 3.2 G, the resulting file is 2.1 G big, but I'm still unable to import it. I'll attach the output of the run, it seems like "105/2018 entries...Warning: attachment data not found " problems were not fixed, as they are still show up if I rerun your tool against the fixed backup file.

COUNTER: 3341106540
Reading backup file...
FRAME 26197 (062.6%)...
WARNING: Bad MAC in attachmentdata: theirMac: (hex:) 9b ce 14 32 ef 51 6c c9 aa 84
                                      ourMac: (hex:) bd a9 d1 d0 8b 14 96 1f 38 09 bf 6e 4b 06 6d 2f ec 04 57 8d d6 b3 8d bb 27 40 ef 9e 1b b8 e9 84

WARNING: Bad MAC in frame, trying to print frame info:
Frame number: 26198
        Type: ATTACHMENT
         - row id          : 2037 (8 bytes)
         - attachment id   : 1582746100843 (8 bytes)
         - length          : 2295648 (8 bytes)
         - attachment      : (hex:) ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 43 00 ... (2295648 bytes total)
Frame is attachment, it belongs to entry in the 'part' table of the database:
 - _id : 2037
 - mid : 2826
 - seq : 0
 - ct : image/jpeg
 - name : (NULL)
 - chset : (NULL)
 - cd : O2WuMgC5+YpercFkV3M1gEXg2XzSIYr6Y0xWH892WMUXIPp7Bi3ALbXClhsAb6bkVrs4LpIE5vN2jZ65oZp1Eg==
 - fn : (NULL)
 - cid : (NULL)
 - cl : 292886834820839875
 - ctt_s : (NULL)
 - ctt_t : (NULL)
 - encrypted : (NULL)
 - pending_push : 0
 - _data : /data/user/0/org.thoughtcrime.securesms/app_parts/part7364302166359440396.mms
 - data_size : 2295648
 - file_name : (NULL)
 - thumbnail : /data/user/0/org.thoughtcrime.securesms/app_parts/part4545823281927312143.mms
 - aspect_ratio : 1
 - unique_id : 1582746100843
 - digest : (hex:) 4b 1f eb fc 85 21 4b f8 83 c3 70 6c df a4 c6 01 5b f1 5c 57 c3 33 56 25 3c 9b 5e 27 3c 0b 49 03
 - fast_preflight_id : 7620226491031352570
 - voice_note : 0
 - data_random : (hex:) 77 1e 69 fa 01 c1 d6 27 0a 47 b7 08 bc f5 0f 19 f8 84 f2 43 33 3c cd 86 42 fc 79 69 f6 37 7a 58
 - thumbnail_random : (hex:) c6 d3 ee 58 64 d4 34 8a 8b 28 00 8b ae 38 33 64 7a af ab e2 77 fd 2f ce cb 3d 86 ec cb 39 1a 0c
 - width : 4032
 - height : 3024
 - quote : 0
 - caption : (NULL)
 - sticker_pack_id : (NULL)
 - sticker_pack_key : (NULL)
 - sticker_id : -1
 - data_hash : xhwj9ntWiYTHa1kPydkXxIgdHlfYtIr/P4/AUHpA1v8=
 - blur_hash : LLI#My-pInR*%$EKj]RP~psDRPoe
 - transform_properties : {"skipTransform":true}
 - transfer_file : (NULL)
 - display_order : 6
 - upload_timestamp : 0
 - cdn_number : 0
 - borderless : 0

Which belongs to entry in 'mms' table:
 - _id : 2826
 - thread_id : 20
 - date : 2020-02-26 19:42:00 +0000 (1582746120192)
 - date_received : 2020-02-26 19:42:00 +0000 (1582746120197)
 - msg_box : 10485783
 - read : 1
 - m_id : (NULL)
 - sub : (NULL)
 - sub_cs : (NULL)
 - body :
 - part_count : 0
 - ct_t : (NULL)
 - ct_l : (NULL)
 - address : 240
 - address_device_id : (NULL)
 - exp : (NULL)
 - m_cls : (NULL)
 - m_type : 128
 - v : (NULL)
 - m_size : (NULL)
 - pri : (NULL)
 - rr : (NULL)
 - rpt_a : (NULL)
 - resp_st : (NULL)
 - st : (NULL)
 - tr_id : (NULL)
 - retr_st : (NULL)
 - retr_txt : (NULL)
 - retr_txt_cs : (NULL)
 - read_status : (NULL)
 - ct_cls : (NULL)
 - resp_txt : (NULL)
 - d_tm : (NULL)
 - delivery_receipt_count : 2
 - mismatched_identities : (NULL)
 - network_failures : (NULL)
 - d_rpt : (NULL)
 - subscription_id : -1
 - expires_in : 0
 - expire_started : 0
 - notified : 0
 - read_receipt_count : 2
 - quote_id : 0
 - quote_author : (NULL)
 - quote_body : (NULL)
 - quote_attachment : -1
 - shared_contacts : (NULL)
 - quote_missing : 0
 - unidentified : 0
 - previews : (NULL)
 - reveal_duration : 0
 - reveal_start_time : 0
 - reactions : (NULL)
 - reactions_unread : 0
 - reactions_last_seen : -1
 - date_server : -1
 - remote_deleted : 0
 - quote_mentions : (NULL)
 - mentions_self : 0
Trying to dump decoded attachment to file 'attachment_2826.bin'
FRAME 26198 (062.7%)... Failed to read next frame (334893159 bytes at filepos 2148656909)
Starting bruteforcing offset to next valid frame... starting after: 2148656909
Checking offset 70 bytes
GOT GOOD MAC AT OFFSET 76 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 7 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 350 bytes
GOT GOOD MAC AT OFFSET 354 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 35 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 670 bytes
GOT GOOD MAC AT OFFSET 679 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 67 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 950 bytes
GOT GOOD MAC AT OFFSET 955 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 95 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 1260 bytes
GOT GOOD MAC AT OFFSET 1260 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 126 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 1530 bytes
GOT GOOD MAC AT OFFSET 1533 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 153 frames... nope! :(
No valid frame found at maximum frameskip for this offset...
Checking offset 1810 bytes
GOT GOOD MAC AT OFFSET 1814 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 110 frames... YEAH! :(
Good frame: 26798 (AvatarFrame)

Got frame, breaking
FRAME 26798 (062.7%)... Failed to get valid frame from decoded data...
Data was verified ok, but does not represent a valid frame... Don't know what happened, but it's bad... Aborting :(
Attachment data with BAD MAC was encountered:
Short info on message to which attachment with bad mac belongs (1/1):
Date          : 2020-02-26 19:42:00 +0000 (1582746120192)
Date received : 2020-02-26 19:42:00 +0000 (1582746120197)
Sent to       :
Message body  :

done!
WARNING EndFrame was not read: backup is probably incomplete

Exporting backup to 'signal-2020-09-20-22-55-44-fixed.backup'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing SqlStatementFrame(s)...
  Dealing with table 'sms'... 17807/17807 entries...done
  Dealing with table 'mms'... 4303/4303 entries...done
  Dealing with table 'part'... 105/2018 entries...Warning: attachment data not found (rowid: 108, uniqueid: 1516771290884)
  Dealing with table 'part'... 106/2018 entries...Warning: attachment data not found (rowid: 109, uniqueid: 1516771292418)
  Dealing with table 'part'... 107/2018 entries...Warning: attachment data not found (rowid: 110, uniqueid: 1516771293336)
  Dealing with table 'part'... 108/2018 entries...Warning: attachment data not found (rowid: 111, uniqueid: 1516771295123)
  Dealing with table 'part'... 201/2018 entries...Warning: attachment data not found (rowid: 205, uniqueid: 1522489127686)
  Dealing with table 'part'... 350/2018 entries...Warning: attachment data not found (rowid: 366, uniqueid: 1530908240957)
  Dealing with table 'part'... 393/2018 entries...Warning: attachment data not found (rowid: 409, uniqueid: 1532770413682)
  Dealing with table 'part'... 482/2018 entries...Warning: attachment data not found (rowid: 500, uniqueid: 1537166862745)
  Dealing with table 'part'... 538/2018 entries...Warning: attachment data not found (rowid: 556, uniqueid: 1538849087089)
  Dealing with table 'part'... 713/2018 entries...Warning: attachment data not found (rowid: 731, uniqueid: 1547229649473)
  Dealing with table 'part'... 1339/2018 entries...Warning: attachment data not found (rowid: 1357, uniqueid: 1564526991077)
  Dealing with table 'part'... 1340/2018 entries...Warning: attachment data not found (rowid: 1358, uniqueid: 1564526991082)
  Dealing with table 'part'... 1341/2018 entries...Warning: attachment data not found (rowid: 1359, uniqueid: 1564526993343)
  Dealing with table 'part'... 1506/2018 entries...Warning: attachment data not found (rowid: 1525, uniqueid: 1568657470198)
  Dealing with table 'part'... 1664/2018 entries...Warning: attachment data not found (rowid: 1683, uniqueid: 1575396908093)
  Dealing with table 'part'... 1665/2018 entries...Warning: attachment data not found (rowid: 1684, uniqueid: 1575396908219)
  Dealing with table 'part'... 1666/2018 entries...Warning: attachment data not found (rowid: 1685, uniqueid: 1575397001172)
  Dealing with table 'part'... 1855/2018 entries...Warning: attachment data not found (rowid: 1874, uniqueid: 1579376535561)
  Dealing with table 'part'... 2018/2018 entries...done
  Dealing with table 'thread'... 0/0 entries...
  Dealing with table 'identities'... 0/0 entries...
  Dealing with table 'drafts'... 0/0 entries...
  Dealing with table 'push'... 0/0 entries...
  Dealing with table 'groups'... 0/0 entries...
  Dealing with table 'group_receipts'... 0/0 entries...
  Dealing with table 'job_spec'... 0/0 entries...
  Dealing with table 'constraint_spec'... 0/0 entries...
  Dealing with table 'dependency_spec'... 0/0 entries...
  Dealing with table 'sticker'... 0/0 entries...
  Dealing with table 'recipient'... 0/0 entries...
  Dealing with table 'storage_key'... 0/0 entries...
  Dealing with table 'key_value'... 0/0 entries...
  Dealing with table 'megaphone'... 0/0 entries...
  Dealing with table 'remapped_recipients'... 0/0 entries...
  Dealing with table 'remapped_threads'... 0/0 entries...
  Dealing with table 'mention'... 0/0 entries...
Writing SharedPrefFrame(s)...
Writing Avatars...
Writing EndFrame...
Done!

When I try to build the app using yay, I get a permission denied error.

I think I'm having the same problem as described in https://aur.archlinux.org/packages/signalbackup-tools-git/#comment-837222

Logs:

[user@PC user]$ yay -S signalbackup-tools-git
:: Checking for conflicts...
:: Checking for inner conflicts...
[Aur:1]  signalbackup-tools-git-20210128.r0.g563fe38-1

:: (1/1) Downloaded PKGBUILD: signalbackup-tools-git
  1 signalbackup-tools-git           (Build Files Exist)
==> Diffs to show?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==>
:: (1/1) Parsing SRCINFO: signalbackup-tools-git
==> Making package: signalbackup-tools-git 20210128.r0.g563fe38-1 (Sat 08 Jan 2022 03:42:20 PM CET)
==> Retrieving sources...
  -> Cloning signalbackup-tools git repo...
Cloning into bare repository '/home/user/.cache/yay/signalbackup-tools-git/signalbackup-tools'...
remote: Enumerating objects: 2495, done.
remote: Counting objects: 100% (1058/1058), done.
remote: Compressing objects: 100% (639/639), done.
remote: Total 2495 (delta 776), reused 663 (delta 402), pack-reused 1437
Receiving objects: 100% (2495/2495), 772.41 KiB | 645.00 KiB/s, done.
Resolving deltas: 100% (1868/1868), done.
==> Validating source files with md5sums...

    signalbackup-tools ... Skipped
==> Making package: signalbackup-tools-git 20210128.r0.g563fe38-1 (Sat 08 Jan 2022 03:42:23 PM CET)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Updating signalbackup-tools git repo...

Fetching origin
==> Validating source files with md5sums...
    signalbackup-tools ... Skipped
==> Removing existing $srcdir/ directory...
==> Extracting sources...
  -> Creating working copy of signalbackup-tools git repo...
Cloning into 'signalbackup-tools'...
done.
==> Starting pkgver()...
==> Updated version: signalbackup-tools-git 20220107.r0.g35b0f4c-1
==> Sources are ready.
==> Making package: signalbackup-tools-git 20220107.r0.g35b0f4c-1 (Sat 08 Jan 2022 03:42:25 PM CET)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Starting pkgver()...
==> Starting build()...
/home/user/.cache/yay/signalbackup-tools-git/PKGBUILD: line 24: ./BUILDSCRIPT.sh: Permission denied
==> ERROR: A failure occurred in build().
    Aborting...
 -> error making: signalbackup-tools-git

Hey I just successfully merged two backups of versions 64 and 89.

These were the errors I could find searching for error, warn and note (anything else I should search for?):

SQL error during sqlite3_prepare_v2(): no such table: mention
SQL error during sqlite3_prepare_v2(): no such table: remapped_recipients
SQL error during sqlite3_prepare_v2(): no such table: remapped_threads
  NOTE: Dropping column 'groups.avatar' from source : Column not present in target database
  NOTE: Dropping column 'mms.reveal_start_time' from source : Column not present in target database
  NOTE: Skipping table 'megaphone', as it is not present in target database. Data may be missing.
  Dealing with table 'part'... 121/946 entries...Warning: attachment data not found (rowid: 121, uniqueid: 1600536479253)

Let me know if you need more info or want me to run it again later.

Also, I have a group that changed id after being upgraded to a v2 group. Before I look into renaming it in the old backup, maybe you have some advice? It's such a small thing, but I will give it a try unless you know it's more complex than I hope :)

Thank you for a wonderful tool!

P.S. I couldn't find your email address.

Per the documentation, I just wanted to drop a note of thanks - I was successfully able to merge 2 signal databases - both database versions were 87.

I had to drop to an old version of Signal to import my SMS messages. I updated Signal before merging the databases, so I didn't rely on the tool for that.

The merged database successfully restored on the latest version of Signal 👍

Hi,

thanks for your work! I tried to compile your tool, but it didn't work. I did the following steps on a Fedora30 VM:

$ git clone https://github.com/bepaald/signalbackup-tools.git
$ sudo dnf install gcc-g++ cryptopp-devel sqlite-devel
$ cd signalbackup-tools && chmod +x BUILDSCRIPT.sh
$ sh BUILDSCRIPT.sh

and got the following error:

Assembler messages:
Fatal error: can't create cryptobase/o/cryptobase.o: No such file or directory

Do you know what I am doing wrong?

Thanks in advance
Chris

Hi, thanks for your software, I just used it to successfully merge an old backup with my current one (I sent you an email with feedback about it).

As I'm on Arch and noticed it wasn't yet packaged in the AUR, I went on and created one: signalbackup-tools-git.

As such I thought I'd do my due diligence and report the one warning I got from namcap:

signalbackup-tools-git W: Unused shared library '/usr/lib/libm.so.6' by file ('usr/bin/signalbackup-tools')

A cursory search revealed that people in that situation tended to report it upstream, so... Here I am 😀

So that's it, just to let you know. Thank you very much for your work and all the best!

Hello, thanks for great tool, but in Windows it ends in my progress always like following:
Frame number: 27945
Size: 10
Type: SHAREDPREFERENCEFRAME
- (file : "
There it hangs and hangs...please telle me, what to do. Thanks!

Original backup would crash Signal on Android after about 2900 messages. Used the signalbackup-tools to repair the database, dropping it from 2.2GB to 1.8GB. Signal doesn't crash when importing the repaired database (and reads ~3100 messages according to the progress), but after import no messages are displayed. I dumped the repaired backup to a folder. Browsing the database seems to show all of the important tables as being empty. Dumping to mms and sms to CSV produces 0-byte files. If I crack some of the .bin files in the raw export, there's legit data in there it seems (I can see EXIF headers and such). What's the next thing to try?

This is not an issue, I just wanted to quickly say thank you for making this tool!

Using this I was able to make a little script to convert my signal backups to nice HTML. This wouldn't have been feasible without all the work you've done to deal with the backup files, so thank you very much! :)

First of all, let me just say thank you so much for all of the work you've put into this! I ended up following this simple guide to installing/using the tool: https://gitlab.com/splatops/cntn-signalbackup-tools

Then I was able to merge two databases by running:

signalbackup-tools signal-2021-03-06-07-59-10.backup "PASSWORD" --importthreads ALL --source signal-2021-03-05-05-29-31.backup --sourcepassword "PASSWORD" --output signal_merged.backup

I also ended up exporting an XML so I could import old conversations to my default SMS app.

signalbackup-tools signal-2021-03-06-07-59-10.backup "PASSWORD" --importthreads ALL --source signal-2021-03-05-05-29-31.backup --sourcepassword "PASSWORD" --exportxml signal_merged.xml

Reference: Issue #27

I have a Signal backup that cannot be restored in the Signal app. I've built signalbackup-tools and tried to use it to repair the broken backup, but the tool seems to have hit a lot of corruption. In #17 (comment), this description:

It's from a phone that initially had not enough space to perform a backup but at some point it seemed that enough space was available and the backup finished correctly.

sounds exactly like me. So, my question is twofold: am I encountering the same problem as #17, where somehow part of the backup file is a copy of the earlier part, and if not, is there a way to restore my data? If the answer to the first question is yes, I'll be reporting that bug to Signal upstream as that really seems related to the disk space issue.

I'm on revision f3db274. I don't have access to the original device (I'm in this situation because I accidentally factory reset my phone while flashing Google factory images today), but I do have access to several years worth of Signal backups. At least some of these are known-good, although the known-good ones are probably pretty old. I also have access to a Signal Desktop installation that was originally linked to the installation that generated the backup. I did not test this backup in any way prior to today, a mistake which I will not be making in the future ;)

Examination of server backups show that the file has not been corrupted since being uploaded to my personal Nextcloud. I believe but am not sure that Nextcloud mobile verifies checksums when uploading, so likely the file was corrupted before it ever made it onto the network.

The VM I was running it in seems to have crashed just before the tool finished, but I already have 12 GB of logs... I'm wondering if that will be enough to determine whether the repeated data problem has occurred 😅 if so, I could email you a link. Or if you want to give me a script or something to just run on it locally I can make that happen too.

I have been having considerable problems compiling for Android. In particular, the error from #9 is familiar:

../src/fileencryptor/../cryptbase/../common_be.h:145:10: error: no matching function for call to 'bytesToHexString'

@bepaald concluded here:

Apparently, clang is the problem (on MacOS, it works fine on Linux).

Following a deeper investigation, I concluded that the problem is not actually clang itself, but rather libc++, which is LLVM's implementation of the C++ standard library. The problem is fairly uncommon on the Linux desktop, where GNU libstdc++ is still the most commonly used C++ standard library, even when using clang to compile. On Ubuntu 20.04 for example, builds with clang are by default still done by including headers and linking against libraries from GNU libstdc++.

The exact same issue was opened against the NDK itself. It is a longstanding issue (2.5 years) with no resolution, apparently in large part because the problem has not been fully addressed within upstream libc++. However, an astute comment notes that there is an open but still unmerged review for a fix.

To be clear, this is not a defect in signalbackup-tools. Rather, libc++ has a defect that substantially breaks its C++ 17 compatibility. However, given the current (and increasing) prevalence of the LLVM toolchain, I believe that compatibility with LLVM's libc++ remains a worthy goal, especially in the case of Android where Google has been steadily removing support for the GNU tools for years already. Unlike the workaround discussed in #9 for the macOS use case, the use of gcc and libstdc++ is not a practical option for Android, thus motivating me to open this new issue.

Further examination of the fix reveals that it is effectively a header-only change, so it can be easily deployed without either a standard library rebuild or even a system configuration change. In particular what I did was create a new subdirectory in my signalbackup-tools workspace ("sysinclude") into which I placed the new header. Then I prepended the subdirectory to the system include search path with a clang command line option: -isystem./sysinclude". The matter cannot be readily addressed with a source change since in some cases the included system headers include <memory> ahead of the explicit inclusions of the header directly by the signalbackup-tools source itself.

As an alternative approach which would likely lead to increased compatibility on older platforms, signalbackup-tools could be reworked such that it targets an older C++ standard (e.g., C++ 14) for which support is more mature across a larger range of environments.

It is also worth noting that in this environment, setlocale() is in the global namespace rather than the "std" namespace. I addressed this naively in sqlitedb/widestring.cc with a preprocessor definition:

#ifdef NDK_LOCALE_WORKAROUND
  setlocale(LC_ALL, "en_US.utf8");
#else
  std::setlocale(LC_ALL, "en_US.utf8");
#endif

The change may then be activated as needed as needed with a clang command line option: -DNDK_LOCALE_WORKAROUND.

Note that earlier NDK versions cause even more problems: of particular note, std::filesystem support needed by signalbackup-tools was not supported bu the bundled libc++ shipped with NDK versions prior to r22.

Of course, at the end of the day, the user attempting to build signalbackup-tools for his/her preferred platform should not need to be concerned with such details; administration of such particulars should be handled with Autoconf/Makefile or a similar technology; this has already been discussed in both #11 and #13.

I opened the signal bug for the corrupt backups and was able to get back up and running using your fix (thanks a ton!)

I was wondering if there was any progress on the backup merge capability? During the wait, I had to start fresh. I would like to merge these two backups :)

No worries if this isn't something that is a priority right now.

Also, do you have a link to donate? I super appreciate the tool you have developed (it has helped me understand how signal does backups)

Thanks!

Apologies if I'm just doing something wrong, but after downloading the latest Windows executable, which I believe was released yesterday, I just get an error:

C:\Users\ ... \Phone backups>signalbackup-tools_win.exe [signal-2021-08-10-00-22-45.backup] [54...password...74] --exportxml [dht.xml]
signalbackup-tools (signalbackup-tools_win.exe) source version 20210809.171715 (OpenSSL)
Failed to open file: '[signal-2021-08-10-00-22-45.backup]'
Failed to create filedecrypter
Failed to open backup

When merging an existing database (6 MB) with a large source database (380 MB) I quickly ran out of memory. RAM usage steadily increased with each thread processed, until I ran out both of RAM (4 GB) and cache (4 GB).

Would it be possible to free up memory after processing each thread?

I bypassed the issue by selecting only a small number of threads (by using --importthreads), but ideally I would like to import all of the threads from the source database.

Other than that, I didn't run into any other problems, so it seems that merging works fine. Good job!

I did get a lot of

Column not present in target database

and a

NOTE: Skipping table 'megaphone', as it is not present in target database. Data may be missing.

at each thread, but that's probably just database incompatibility (versions 80 and 89) as you have already mentioned they change rather frequently.

Observed that the Windows.exe is skipping attachments while decrypting export from database version 90 (latest signal on Android)

also tried the latest Windows release still same issue.

So I got a disappearing message today which expires in 6 days from today, so I created a backup of it today and tried decrypting the data, but the message does not appear in the database. Is this expected?

Does the signal export include disappearing messages in their android backup?

image.jpg
image.jpg(2)
image.jpg(2)(2)

this should have been

image.jpg
image(2).jpg
image(3).jpg

For most parameters forgetting to end the path with a slash will result in a single file, rather than recognizing the path as a directory. Ideally the tool recognizes any output path as a directory regardless of the final slash being omitted.

For the --dumpmedia parameter the slash has to be omitted otherwise the tool throws an error, that's inconsistent with other parameters.

Since there are specific requirements for this tool to be compiled adding Docker support will alleviate being able to run everywhere.

I've already done this so feel free to link to my repo:
https://gitlab.com/splatops/cntn-signalbackup-tools

Or if you'd like feel free to add this directly to your project. Thank you for the tool!

I like the feature of merging backups
It will be great if backup could be enhanced with history from other communicators SMS, WhatsApp, Telegram etc
My idea is simple just read CSV file with chatname,timestamp,number,text and merge it with matching conversation.
What do you think? I can help with implementation of I get some hints.