evalempire / gitpan Goto Github PK

Right now, gitpan spews a lot of information at the screen. It needs improved logging to identify:

• Problems vs information
• Which dists had problems
• What that problem was
• Enough info to solve it or repeat it

"Archive of Foo-Bar releases. NOT THE DEVELOPMENT REPOSITORY."

Blunt but effective.

Sometimes a module is renamed, and its distribution is renamed accordingly. It would be useful to be able to mark that a pair of distributions with different base names are in fact versions of the same project, and so a module's history is followed through all the names it had. Separately, but probably more difficult, is following when distributions have been split up or combined, in that sometimes a particular module is bundled with others and other times it is released on its own. Both of these cases have happened with my projects.

importing CAM-PDF-1.02_01
downloading file:///var/local/backpan/authors/id/C/CL/CLOTHO/CAM-PDF-1.02_01.tgz
extracting distribution
Use of uninitialized value $_ in stat at /usr/local/lib/perl5/site_perl/File/chmod.pm line 37.
Use of uninitialized value in bitwise and (&) at /usr/local/lib/perl5/site_perl/File/chmod.pm line 37.
Use of uninitialized value $_ in chmod at /usr/local/lib/perl5/site_perl/File/chmod.pm line 66.
invalid top directory at /usr/local/lib/perl5/5.10.1/File/Find.pm line 598.

Yanick pointed out that if you add gitpan as a remote to your dev repository and fetch it, gitpan's tags will silently overwrite your own. So tag 1.2.3 suddenly points at gitpan's release. This is Bad.

• Investigate if annotated tags solves this
• Prefix all tags with cpan/ by default.
• Provide an option to change the prefix.

I am willing to help with this.

http://github.com/schwern/gitpan/blob/master/README says:

See http://github.com/schwern/github/issues for a list of open problems.

But it should, obviously, be http://github.com/schwern/gitpan/issues as the former URL gives a 404.

If you tack a .diff or .patch on the end of a github commit URL it will produce a diff. Krishna was pointing out this would be handy for submitting a patch to an author (one who isn't on github). Document this in the FAQ and perhaps make the gitpan command line tool spit out the URLs.

Like Parse::BACKPAN::Packages and the CPANPLUS object.

Immediately following extraction, CPANPLUS has either $mod->package_name or $mod->package_version undef.

[MSG] Extracted 'perl-c++-api.0.0_2/test/loop-time.cc'
[MSG] Extracted 'perl-c++-api.0.0_2/test/example.pm'
[MSG] Extracted 'perl-c++-api.0.0_2/MANIFEST'
[MSG] Extracted 'perl-c++-api.0.0_2/Artistic'
Use of uninitialized value in concatenation (.) or string at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202.
at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202
CPANPLUS::Internals::Extract::_extract('CPANPLUS::Backend=HASH(0x11e63b4)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/4LMlmc3YWL', 'module', 'CPANPLUS::Module::Fake=HASH(0x100f8b38)') called at /usr/local/lib/perl5/5.10.1/CPANPLUS/Module.pm line 621
CPANPLUS::Module::extract('CPANPLUS::Module::Fake=HASH(0x100f8b38)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/4LMlmc3YWL') called at /usr/local/lib/perl5/site_perl/Git/CPAN/Import.pm line 83
Git::CPAN::Import::main('file:///var/local/backpan/authors/id/K/KE/KENFOX/perl-c++-api...', 'HASH(0x1bfa238)') called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 68
eval {...} called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 67
Use of uninitialized value in concatenation (.) or string at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202.
at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202
CPANPLUS::Internals::Extract::_extract('CPANPLUS::Backend=HASH(0x11e63b4)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/4LMlmc3YWL', 'module', 'CPANPLUS::Module::Fake=HASH(0x100f8b38)') called at /usr/local/lib/perl5/5.10.1/CPANPLUS/Module.pm line 621
CPANPLUS::Module::extract('CPANPLUS::Module::Fake=HASH(0x100f8b38)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/4LMlmc3YWL') called at /usr/local/lib/perl5/site_perl/Git/CPAN/Import.pm line 83
Git::CPAN::Import::main('file:///var/local/backpan/authors/id/K/KE/KENFOX/perl-c++-api...', 'HASH(0x1bfa238)') called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 68
eval {...} called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 67
[MSG] Extracted 'perl-c++-api.0.0_2.tar.gz' to '/private/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/4LMlmc3YWL/perl-c++-api.0.0_2'

importing ipad.tar.gz (file:///var/local/backpan/authors/id/T/TI/TIMB/ipad.tar.gz)
downloading ipad.tar.gz
[MSG] Trying to get 'file:///var/local/backpan/authors/id/T/TI/TIMB/ipad.tar.gz'
extracting distribution
[MSG] Extracted 'cmi2.ps'
[MSG] Extracted 'cost.txt'
[MSG] Extracted 'example1.tk'
[MSG] Extracted 'margin.csv'
[MSG] Extracted 'pad12.ppt'
[MSG] Extracted 'pngsetup.exe'
[MSG] Extracted 'pres0.png'
[MSG] Extracted 'x.pl'
Use of uninitialized value in concatenation (.) or string at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202.
at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202
CPANPLUS::Internals::Extract::_extract('CPANPLUS::Backend=HASH(0x11e62ac)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/TQNm6LwraS', 'module', 'CPANPLUS::Module::Fake=HASH(0x100d5870)') called at /usr/local/lib/perl5/5.10.1/CPANPLUS/Module.pm line 621
CPANPLUS::Module::extract('CPANPLUS::Module::Fake=HASH(0x100d5870)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/TQNm6LwraS') called at /usr/local/lib/perl5/site_perl/Git/CPAN/Import.pm line 83
Git::CPAN::Import::main('file:///var/local/backpan/authors/id/T/TI/TIMB/ipad.tar.gz', 'HASH(0x100606d0)') called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 68
eval {...} called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 67
Use of uninitialized value in concatenation (.) or string at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202.
at /usr/local/lib/perl5/5.10.1/CPANPLUS/Internals/Extract.pm line 202
CPANPLUS::Internals::Extract::_extract('CPANPLUS::Backend=HASH(0x11e62ac)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/TQNm6LwraS', 'module', 'CPANPLUS::Module::Fake=HASH(0x100d5870)') called at /usr/local/lib/perl5/5.10.1/CPANPLUS/Module.pm line 621
CPANPLUS::Module::extract('CPANPLUS::Module::Fake=HASH(0x100d5870)', 'extractdir', '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/TQNm6LwraS') called at /usr/local/lib/perl5/site_perl/Git/CPAN/Import.pm line 83
Git::CPAN::Import::main('file:///var/local/backpan/authors/id/T/TI/TIMB/ipad.tar.gz', 'HASH(0x100606d0)') called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 68
eval {...} called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 67
[MSG] Extracted 'ipad.tar.gz' to '/private/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/TQNm6LwraS'
[ERROR] Unable to find 'Build.PL' or 'Makefile.PL' for 'ipad.tar.gz'; Will default to 'CPANPLUS::Dist::MM' but might be unable to install!

add 'cmi2.ps'
add 'cost.txt'
add 'example1.tk'
add 'ipad.tar.gz'
add 'margin.csv'
add 'pad12.ppt'
add 'pngsetup.exe'
add 'pres0.png'
add 'temp_git_index.lock'
add 'x.pl'
opening backpan index
couldn't import Parse::BACKPAN::Packages::Release=HASH(0xbd8c748) at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 77.
at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 77
fatal: git checkout: updating paths is incompatible with switching branches.
Did you intend to checkout 'cpan/master' which can not be resolved as commit?
at /usr/local/lib/perl5/site_perl/Error.pm line 184
Error::throw('Git::Error::Command', 'checkout -t -b master cpan/master', 128) called at /usr/local/lib/perl5/site_perl/Git.pm line 467
Git::command_noisy('Git=HASH(0x1bdcc78)', 'checkout', '-t', '-b', 'master', 'cpan/master') called at /Users/schwern/devel/git-cpan-patch/scripts/git-backpan-init line 80

This is going to need somewhere between 5 and 10 gigs (based on the 14 gig size of BackPAN and the compression of ongoing imports). Importing a random selection of dists would give a more accurate estimate.

I've emailed support explaining the project and our space needs, as well as the number of dists we plan to import.

Will not unpack entirely on a non-case sensitive filesystem (ie. my Mac) because it contains multiple overlaping directories and files.

Solutions: Run the import on another machine. Make a temporary case-sensitive filesystem to import into.

Commits (not authors) are currently being done as schwern. It would be less confusing, and egotistical, if they went in as gitpan.

So 1.2.3 is tagged as v1.2.3. This makes versions stand out, follows the git semi-standard, and prevents versions like .01 from choking git.

v1.2.3 is left as v1.2.3.

It does a bunch of things inefficiently

importing Bundle-Slash-2.11
downloading file:///var/local/backpan/authors/id/C/CN/CNANDOR/Bundle-Slash-2.11.tar.gz
extracting distribution
Use of uninitialized value $dir in concatenation (.) or string at lib/Git/CPAN/Patch/Import.pm line 47.

Right now if something screws up I have to delete the local and github repositories manually. It would be nice to have an --overwrite flag to do that.

I tried going straight to the URL and i get the jedi-octocat 404 page.

I was surprised to see that the search/filter box is working, but it turned up very few results for plack
(I don't think I saw any Middleware).

I thought everything was here. Is there a manual process to start including a given module?
Or is the sheer size of gitpan breaking github?

At this point it will be a lot of extra work to merge back into Git::CPAN::Import. It will be less confusing to just fork completely.

Github seems to think some gitpan modules are actually Smalltalk:

http://github.com/languages/Smalltalk

I don't know how that can be rectified.

Cheers,
Ovid

D/DU/DURIST/Drupal-Admin-.01.tar.gz for example. .01 is not a valid tag name.

Solution 1: Normalize it as 0.1
Solution 2: Prefix version tags with v so v.1
#2 jives with git community version tagging style.

Hi,

Don't know if this is the proper place for this, but HTML-FormHandler have version 0.29002 in gitpan and 0.32003 on cpan and backpan. Is there a way to update this?

• Bjørn-Olav

Webservice-Google-Suggest vs WebService-Google-Suggest. Problem is, github is case insensitive.

There's also a rogue uri distribution.

CPAN has an ownership policy and not everything on BackPAN is actually part of a distribution. Knowing what's indexed comes from the PAUSE uploads database. gitpan should make use of this and only select stable tarballs which were in the index.
http://devel.cpantesters.org/

Trouble is, PAUSE goes by module. gitpan by distribution. You have releases which contain some authorized and some unauthorized modules. For example, this appears in the PAUSE uploads database. Its clearly the result of an incomplete ownership transfer, and valid.
http://search.cpan.org/~chorny/Apache-Session-1.88/

But this does not. However, for the purposes of gitPAN there's no problem making a repo for this. gitPAN has no global module index to worry about.
http://search.cpan.org/dist/lcwa/

There isn't a whole lot one can do about that. The data in the uploads database just lists tarballs. If we had the same info that search.cpan does we could maybe apply some heuristics and say that if the module matching the dist name is authorized (ie. Apache-Session-1.88.tar.gz is authorized for Apache::Session) then its ok.

Keep a tag of the latest stable version and alpha version. Seems handy.

A commit like this one has by name as "AEvar Arnfjord Bjarmason" instead of "Ævar Arnfjörð Bjarmason". It's converted by PAUSE (presumably).

Please convert the commits to use original author names if possible.

importing Eludia-07.08.30
downloading file:///var/local/backpan/authors/id/D/DM/DMOW/Eludia-07.08.30.tar.gz
extracting distribution
Error listing contents of archive '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/Zr2KPqhhtT/Eludia-07.08.30.tar.gz': tar: Unrecognized archive format: Inappropriate file type or format
tar: Error exit delayed from previous errors.
at lib/Git/CPAN/Patch/Import.pm line 107
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Invalid header block at offset unknown at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 809
Use of uninitialized value in -d at /usr/local/lib/perl5/5.10.1/Archive/Extract.pm line 860.
Unable to extract '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/Zr2KPqhhtT/Eludia-07.08.30.tar.gz': Invalid header block at offset unknown at lib/Git/CPAN/Patch/Import.pm line 107
Extract failed due to errors at lib/Git/CPAN/Patch/Import.pm line 107
Couldn't extract /var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/Zr2KPqhhtT/Eludia-07.08.30.tar.gz to /var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/Zr2KPqhhtT because Error listing contents of archive '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/Zr2KPqhhtT/Eludia-07.08.30.tar.gz': tar: Unrecognized archive format: Inappropriate file type or format
tar: Error exit delayed from previous errors.

Unable to extract '/var/folders/eJ/eJzTVP7oG7GVsKYHJtMprE+++TI/-Tmp-/Zr2KPqhhtT/Eludia-07.08.30.tar.gz': Invalid header block at offset unknown
Extract failed due to errors at lib/Git/CPAN/Patch/Import.pm line 107.

I/IL/ILIAL/DPGS.zip and W/WC/WCATLAN/Constants.zip are examples.

Basically just don't tag them.

Not particularly crucial for this distribution, since rjbs has his repo available, but presumably there is something wrong that might be causing hassles for other distros too.

https://github.com/gitpan/Email-Sender only goes to 0.093380 but http://search.cpan.org/dist/Email-Sender/ is at 0.102370 currently (and that was several months ago)

This is going to need somewhere between 5 and 10 gigs (based on the 14 gig size of BackPAN and the compression of ongoing imports). Importing a random selection of dists would give a more accurate estimate.

I've emailed support explaining the project and our space needs, as well as the number of dists we plan to import.

It might be handy to be able to ask "what commit goes with this tarball I have here?" without having to try and parse the version. That information is available in the commit log as git-cpan-file but it could also be a tag.

There's been a lot of unauthorized releases and things are a bit out of date.

On http://github.com/schwern/gitpan/blob/master/README at line 40 the word 'especially' is spelled especially badly. :-)

CPANPLUS doesn't do a very good job figuring out distribution information. Parse::BACKPAN already has the info, so use that instead.

Rather than hacking up our own github interface, use Net::GitHub::V2

Example: MARCEL/-0.01.tar.gz

Possibly make it an option to apply heuristics (default) or report it as is.

This can't live on my laptop forever. Ask has maybe offered some space.

Currently the biggest bottleneck is creating a github repo and pushing. This would lend itself very well to parallel processing, either with threads or whatever.

On all github repositories.

Business-Bof_0.01.tar.gz and Business-Bof-0.01.tar.gz. gitpan chokes on the repeated tag.

I see that a number of modules are showing up for me (thanks!), but I have no commit access. I suppose that's a feature since if I commit and gitpan pulls them from github again, would this confuse the history? Still, it would be nice to not have to fork my own modules :)

Ah, but here's the counter-example:

http://github.com/gitpan/Class-Sniff
http://github.com/Ovid/class--sniff

Cheers,
Ovid

So it can share resources with the gitpan script and Git::CPAN::Import

21k directories is a lot and makes shell globbing worthless.

Split the repo_dir into directories like AC/Acme-Pony TE/Test-Simple

If a SCHLAMEEL releases version 1, 2 and 5 and SCHLAMAZEL releases 3 and 4 you tag 4 with SCHLAMAZEL and 5 with SCHLAMEEL. This will let you find the last version released by X which might be useful. Can't hurt.

I can think of one use case, the current author who never got the past author's repo can use it to find the old history.

Currently they are just skipped

gitpan should ignore ~/.gitignore if it isn't already.

It would be neat to be able to git clone all of CPAN. This could be done by having a huge repo with lots of submodules or in one insanely big repo. The latter would make for a neat alternative to rsyncing all of CPAN for testing / greping.

Couldn't find upload date for Math-BigInt-1.88.tar.gz
Couldn't find author for Math-BigInt-1.88.tar.gz

Seemed to have that problem with most versions. This will be solved once CPANPLUS is removed from BackPAN resolution.