Comments (10)
emmercm
commented on June 9, 2024
1
I'm sorry @chainsawsalad, I had read this and had been thinking about it, and then never got back to you.
This is because a number of PC Engine CD discs share the same track 1 it looks like, with the SHA1 e1baaf01eee215def8bd6f6aa080e3a41ffc74f4. Internally, igir de-duplicates input files, because the thought was it doesn't particularly matter where a file of a certain hash comes from. But in the case of moving games with multiple files (e.g. discs), we do kind of care, and would prefer the files come from the same archive.
This is going to take quite a bit of internal refactoring to tackle, to break that input file assumption. I'll see what I can do!
On the logging, I agree with you, that's really aggressive. I'll drop the level down to DEBUG.
from igir.
lostb1t
commented on June 9, 2024
1
@emmercm just tried it as i ran in the same issue for some ps games. Works like a charm for those games.
from igir.
chainsawsalad
commented on June 9, 2024
1
@emmercm just re-ran my on my whole PC Engine CD collection--no errors!
from igir.
emmercm
commented on June 9, 2024
Hi @chainsawsalad, I've been hacking at this slowly over the past few weeks, and I think I might have the solution.
Could you help me by running the testing instructions at #467 (comment) against your local setup? I spent some time being able to run igir from a GitHub branch/PR.
from igir.
chainsawsalad
commented on June 9, 2024
@emmercm I re-ran my test example with the original two ROMs, and things look good, however I ran it against the rest of my library and I do still see examples of this happening with other ROMs.
Working example:
npm exec --yes -- "github:emmercm/igir#emmercm/same-input-file-for-everything" -vvv move test report --dat ../../DATs/Redump/NEC\ -\ PC\ Engine\ CD\ \&\ TurboGrafx\ CD\ -\ Datfile\ \(541\)\ \(2023-04-09\ 22-11-42\).dat --input ./input --output ./output
> ls -l ./output/
total 514436
-rwxrwxrwx 1 jeff users 368156053 May 20 13:28 'Fighting Street (USA).zip'
-rwxrwxrwx+ 1 jeff users 158625630 May 20 13:23 'Jack Nicklaus Turbo Golf (USA).zip'
Not working examples, extracted out of command output:
[21:20:12.938] WARN: CandidateFilter: Cannot move different files to: /output/Zero 4 Champ II (Japan).zip:
[21:20:12.938] WARN: CandidateFilter: /input/Zero 4 Champ II (Japan).zip
[21:20:12.938] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:21:27.716] WARN: CandidateFilter: Cannot move different files to: /output/Bonanza Bros. (Japan).zip:
[21:21:27.716] WARN: CandidateFilter: /input/Bonanza Bros. (Japan).zip
[21:21:27.716] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:21:30.518] WARN: CandidateFilter: Cannot move different files to: /output/Record of Lodoss War II (Japan) (Demo).zip:
[21:21:30.518] WARN: CandidateFilter: /input/Record of Lodoss War II (Japan) (Demo).zip
[21:21:30.518] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:21:32.957] WARN: CandidateFilter: Cannot move different files to: /output/Faceball (Japan) (Demo).zip:
[21:21:32.957] WARN: CandidateFilter: /input/Faceball (Japan) (Demo).zip
[21:21:32.957] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:22:13.799] WARN: CandidateFilter: Cannot move different files to: /output/Travel Eple (Japan).zip:
[21:22:13.799] WARN: CandidateFilter: /input/Travel Eple (Japan).zip
[21:22:13.799] WARN: CandidateFilter: /input/Gradius II - Gofer no Yabou (Japan).zip
[21:23:54.717] WARN: CandidateFilter: Cannot move different files to: /output/Martial Champion (Japan).zip:
[21:23:54.717] WARN: CandidateFilter: /input/Martial Champion (Japan).zip
[21:23:54.717] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:23:54.717] WARN: CandidateFilter: /input/Fighting Street (Japan) (Rev 5).zip
[21:24:37.618] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 2 - 1993 Spring (Japan).zip:
[21:24:37.618] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 2 - 1993 Spring (Japan).zip
[21:24:37.618] WARN: CandidateFilter: /input/CDI Karaoke 10 (Japan).zip
[21:24:37.618] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM (Japan).zip
[21:24:37.618] WARN: CandidateFilter: /input/Lords of Thunder (USA).zip
[21:24:37.618] WARN: CandidateFilter: /input/Chou Eiyuu Densetsu - Dynastic Hero (Japan).zip
[21:24:39.419] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 3 - 1993 Summer (Japan).zip:
[21:24:39.419] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 3 - 1993 Summer (Japan).zip
[21:24:39.419] WARN: CandidateFilter: /input/Doukyuusei (Japan) (Rev 3).zip
[21:24:39.419] WARN: CandidateFilter: /input/CDI Karaoke 10 (Japan).zip
[21:24:39.419] WARN: CandidateFilter: /input/Might and Magic III - Isles of Terra (Japan).zip
[21:24:39.419] WARN: CandidateFilter: /input/Blood Gear (Japan).zip
[21:24:46.020] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 4 - 1993 Winter (Japan).zip:
[21:24:46.020] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 4 - 1993 Winter (Japan).zip
[21:24:46.020] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:46.020] WARN: CandidateFilter: /input/Godzilla (USA).zip
[21:24:46.020] WARN: CandidateFilter: /input/Faceball (Japan) (Demo).zip
[21:24:46.020] WARN: CandidateFilter: /input/Faceball (Japan).zip
[21:24:46.020] WARN: CandidateFilter: /input/Buster Bros. (USA).zip
[21:24:46.186] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 5 - 1994 Spring (Japan).zip:
[21:24:46.186] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 5 - 1994 Spring (Japan).zip
[21:24:46.186] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:46.186] WARN: CandidateFilter: /input/Doukyuusei (Japan) (Rev 3).zip
[21:24:50.495] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc
A).zip:
[21:24:50.495] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc A).zip
[21:24:50.495] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:50.495] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc A).zip
[21:24:50.495] WARN: CandidateFilter: /input/Neo Nectaris (Japan).zip
[21:24:50.495] WARN: CandidateFilter: /input/Blood Gear (Japan).zip
[21:24:54.269] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc
B).zip:
[21:24:54.269] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc B).zip
[21:24:54.269] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:54.269] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc B).zip
[21:26:02.399] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc
A).zip:
[21:26:02.399] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc A).zip
[21:26:02.399] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:26:02.399] WARN: CandidateFilter: /input/Neo Nectaris (Japan).zip
[21:26:02.399] WARN: CandidateFilter: /input/Blood Gear (Japan).zip
[21:26:02.533] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc
B).zip:
[21:26:02.533] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc B).zip
[21:26:02.533] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:26:02.533] WARN: CandidateFilter: /input/Dungeon Master - Theron's Quest (USA).zip
[21:26:22.963] WARN: CandidateFilter: Cannot move different files to: /output/Godzilla (USA).zip:
[21:26:22.963] WARN: CandidateFilter: /input/Godzilla (USA).zip
[21:26:22.963] WARN: CandidateFilter: /input/Buster Bros. (USA).zip
[21:26:54.580] WARN: CandidateFilter: Cannot move different files to: /output/TV Show, The (Japan).zip:
[21:26:54.580] WARN: CandidateFilter: /input/TV Show, The (Japan).zip
[21:26:54.580] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:27:24.564] WARN: CandidateFilter: Cannot move different files to: /output/Gain Ground SX (Japan).zip:
[21:27:24.564] WARN: CandidateFilter: /input/Gain Ground SX (Japan).zip
[21:27:24.564] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:27:44.807] WARN: CandidateFilter: Cannot move different files to: /output/Garou Densetsu 2 - Aratanaru Tatakai (Japan) (Demo)
(SABS).zip:
[21:27:44.807] WARN: CandidateFilter: /input/Garou Densetsu 2 - Aratanaru Tatakai (Japan) (Demo) (SABS).zip
[21:27:44.807] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
from igir.
emmercm
commented on June 9, 2024
@chainsawsalad let's dig in on one of the examples and I can make a unit test around it.
For this one:
[21:20:12.938] WARN: CandidateFilter: Cannot move different files to: /output/Zero 4 Champ II (Japan).zip:
[21:20:12.938] WARN: CandidateFilter: /input/Zero 4 Champ II (Japan).zip
[21:20:12.938] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
I see a shared track 1 again (SHA1 3b472f5e80bbf8259e09f8e4f547567bddb02490).
Could you list the contents of those input zips? Something that includes name, size, and CRC32 such as:
unzip -vl <file.zip>
from igir.
chainsawsalad
commented on June 9, 2024
@emmercm does this get you what you need?
Archive: Zero 4 Champ II (Japan).zip
TORRENTZIPPED-0C146E99
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
7916832 Defl:X 4641426 41% 12-24-1996 23:32 22144d0f Zero 4 Champ II (Japan) (Track 01).bin
25542720 Defl:X 13182349 48% 12-24-1996 23:32 a3ecc4eb Zero 4 Champ II (Japan) (Track 02).bin
13041840 Defl:X 11439731 12% 12-24-1996 23:32 5f4717be Zero 4 Champ II (Japan) (Track 03).bin
11741184 Defl:X 10545042 10% 12-24-1996 23:32 a929f0e1 Zero 4 Champ II (Japan) (Track 04).bin
11475408 Defl:X 10249938 11% 12-24-1996 23:32 8ce00863 Zero 4 Champ II (Japan) (Track 05).bin
10151232 Defl:X 9223726 9% 12-24-1996 23:32 1b052c36 Zero 4 Champ II (Japan) (Track 06).bin
13135920 Defl:X 11843290 10% 12-24-1996 23:32 54d7853c Zero 4 Champ II (Japan) (Track 07).bin
24213840 Defl:X 21816717 10% 12-24-1996 23:32 aad2c61a Zero 4 Champ II (Japan) (Track 08).bin
11799984 Defl:X 10449265 11% 12-24-1996 23:32 10f0d49b Zero 4 Champ II (Japan) (Track 09).bin
16334640 Defl:X 14788157 10% 12-24-1996 23:32 9eb8f90f Zero 4 Champ II (Japan) (Track 10).bin
11338992 Defl:X 10188609 10% 12-24-1996 23:32 fa3d048a Zero 4 Champ II (Japan) (Track 11).bin
12084576 Defl:X 10836032 10% 12-24-1996 23:32 33167322 Zero 4 Champ II (Japan) (Track 12).bin
10513440 Defl:X 9508049 10% 12-24-1996 23:32 09fb1bd2 Zero 4 Champ II (Japan) (Track 13).bin
16325232 Defl:X 14875132 9% 12-24-1996 23:32 7404385c Zero 4 Champ II (Japan) (Track 14).bin
15266832 Defl:X 13763972 10% 12-24-1996 23:32 7f2b9388 Zero 4 Champ II (Japan) (Track 15).bin
11153184 Defl:X 9604136 14% 12-24-1996 23:32 0c360c9b Zero 4 Champ II (Japan) (Track 16).bin
11148480 Defl:X 10174000 9% 12-24-1996 23:32 2f712235 Zero 4 Champ II (Japan) (Track 17).bin
11357808 Defl:X 10320677 9% 12-24-1996 23:32 8a7b39b2 Zero 4 Champ II (Japan) (Track 18).bin
7067760 Defl:X 6152489 13% 12-24-1996 23:32 616c4702 Zero 4 Champ II (Japan) (Track 19).bin
13279392 Defl:X 11852326 11% 12-24-1996 23:32 27706096 Zero 4 Champ II (Japan) (Track 20).bin
7923888 Defl:X 6785898 14% 12-24-1996 23:32 fecb096f Zero 4 Champ II (Japan) (Track 21).bin
14888160 Defl:X 13536732 9% 12-24-1996 23:32 9528018f Zero 4 Champ II (Japan) (Track 22).bin
6385680 Defl:X 5609726 12% 12-24-1996 23:32 26c2432a Zero 4 Champ II (Japan) (Track 23).bin
16367568 Defl:X 14766983 10% 12-24-1996 23:32 ce330910 Zero 4 Champ II (Japan) (Track 24).bin
846720 Defl:X 717964 15% 12-24-1996 23:32 11bf5dbd Zero 4 Champ II (Japan) (Track 25).bin
3281040 Defl:X 2933325 11% 12-24-1996 23:32 f9188f3a Zero 4 Champ II (Japan) (Track 26).bin
846720 Defl:X 717964 15% 12-24-1996 23:32 11bf5dbd Zero 4 Champ II (Japan) (Track 27).bin
4887456 Defl:X 4323950 12% 12-24-1996 23:32 8d2dbe56 Zero 4 Champ II (Japan) (Track 28).bin
12945408 Defl:X 11156588 14% 12-24-1996 23:32 8d234a26 Zero 4 Champ II (Japan) (Track 29).bin
17611776 Defl:X 15340413 13% 12-24-1996 23:32 92ee3fb4 Zero 4 Champ II (Japan) (Track 30).bin
21073920 Defl:X 18202928 14% 12-24-1996 23:32 7b03db99 Zero 4 Champ II (Japan) (Track 31).bin
14556528 Defl:X 13229124 9% 12-24-1996 23:32 1f3057c2 Zero 4 Champ II (Japan) (Track 32).bin
10520496 Defl:X 9323887 11% 12-24-1996 23:32 887d8c2a Zero 4 Champ II (Japan) (Track 33).bin
11219040 Defl:X 9843611 12% 12-24-1996 23:32 05d3e3a4 Zero 4 Champ II (Japan) (Track 34).bin
12411504 Defl:X 10909543 12% 12-24-1996 23:32 b4cabb3b Zero 4 Champ II (Japan) (Track 35).bin
15285648 Defl:X 13532102 12% 12-24-1996 23:32 8f997117 Zero 4 Champ II (Japan) (Track 36).bin
11284896 Defl:X 9755615 14% 12-24-1996 23:32 798a830f Zero 4 Champ II (Japan) (Track 37).bin
24498432 Defl:X 21884726 11% 12-24-1996 23:32 ac1cbe07 Zero 4 Champ II (Japan) (Track 38).bin
19533360 Defl:X 17334422 11% 12-24-1996 23:32 36df83e6 Zero 4 Champ II (Japan) (Track 39).bin
21897120 Defl:X 19981346 9% 12-24-1996 23:32 137e2970 Zero 4 Champ II (Japan) (Track 40).bin
2629536 Defl:X 1992860 24% 12-24-1996 23:32 d02a90b7 Zero 4 Champ II (Japan) (Track 41).bin
14224896 Defl:X 12706215 11% 12-24-1996 23:32 aa345590 Zero 4 Champ II (Japan) (Track 42).bin
25192272 Defl:X 13171863 48% 12-24-1996 23:32 661f8f8e Zero 4 Champ II (Japan) (Track 43).bin
4187 Defl:X 359 91% 12-24-1996 23:32 a8c5c66e Zero 4 Champ II (Japan).cue
-------- ------- --- -------
555205547 473213207 15% 44 files
Archive: Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
TORRENTZIPPED-00CC60E5
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
7916832 Defl:X 4641426 41% 12-24-1996 23:32 22144d0f Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 1).bin
10536960 Defl:X 2477771 77% 12-24-1996 23:32 59cc5f3f Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 2).bin
7785120 Defl:X 6782271 13% 12-24-1996 23:32 877dde76 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 3).bin
18992400 Defl:X 17376419 9% 12-24-1996 23:32 3236adb9 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 4).bin
9252768 Defl:X 8439144 9% 12-24-1996 23:32 341cc45f Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 5).bin
27854736 Defl:X 25710412 8% 12-24-1996 23:32 e3646dc1 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 6).bin
5216736 Defl:X 4804265 8% 12-24-1996 23:32 e3792471 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 7).bin
984 Defl:X 202 80% 12-24-1996 23:32 54a01e7d Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).cue
-------- ------- --- -------
87556536 70231910 20% 8 files
from igir.
emmercm
commented on June 9, 2024
@chainsawsalad that's perfect, thank you! I was hoping there was something easy to spot in the output, but everything checks out.
I've been spending all evening staring at this, and the "how" is nitty gritty, but the "why" is because some of these games have duplicate tracks within them:
<rom name="Zero 4 Champ II (Japan) (Track 25).bin" size="846720" crc="11bf5dbd" md5="b0845f9d312f9686079470c88cb62596" sha1="9f5fd5f2788d144e0eb61e2edfbbad0deb090350"/>
<!-- ... -->
<rom name="Zero 4 Champ II (Japan) (Track 27).bin" size="846720" crc="11bf5dbd" md5="b0845f9d312f9686079470c88cb62596" sha1="9f5fd5f2788d144e0eb61e2edfbbad0deb090350"/>I'm still hacking away at it.
from igir.
emmercm
commented on June 9, 2024
The unit tests aren't passing, I need to clean them up, but feel free to try again for a sneak peek @chainsawsalad 👍
from igir.
github-actions
commented on June 9, 2024
🔒 Inactive issue lock
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
from igir.
Related Issues (20)
- Output path token for RomM folders HOT 1
- Add output token matching for "Famicom" and "Super Famicom" HOT 3
- MAME merged strategy fails to create files when identical files are found HOT 4
- Allow matching of files to DATs via non-CRC32 checksums HOT 9
- Documentation for which archive formats require temp files HOT 2
- igir creates a lot of empty files for certain libretro dat files HOT 6
- Add 3 letter or more suffix to directories instead of letter only when roms exceed letter directory limit HOT 9
- Igir fails on corrupted Zip files HOT 7
- spawnSync EBADF error since v2.1.2 HOT 16
- app crash at EMFILE: too many open files HOT 7
- Out of memory crash HOT 11
- Handle Disk Image Formats (CHD, CSO, RVZ, etc.) HOT 23
- Show Changes Performed During the Run HOT 5
- Symlinks Cause "Illegal Operation" Error HOT 2
- Clean Command Dry Run HOT 4
- Option to prevent sets with excess files when raw-copying HOT 7
- Output Hardlink Support HOT 5
- Duplicate roms reported as UNUSED HOT 4
- --prefer-revision-newer or --prefer-revision-older has no impact on ROM selection HOT 5
- failed to create zip: TypeError [ERR_INVALID_ARG_TYPE]: The "start" argument must be of type number. Received null HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from igir.