webysther / packagist-mirror Goto Github PK

Hi,

I've mirrored a packagist repo sucessfully

$php webysthe-packagist-mirror/bin/mirror create -vvv
Loading providers from https://packagist.hesse.im/
packages.json updated

My private NGINX server shows me on https://private.repo.lan/packages.json :
{ "packages": [], "notify-batch": "https:\/\/packagist.org\/downloads\/", "providers-url": "\/p\/%package%$%hash%.json", "search": "https:\/\/packagist.org\/search.json?q=%query%&type=%type%", "list": "https:\/\/packagist.org\/packages\/list.json", "providers-api": "https:\/\/packagist.org\/providers\/%package%.json", "warning": "You are using an outdated version of Composer. Composer 2 is now available and you should upgrade. See https:\/\/getcomposer.org\/2", "warning-versions": "<1.99", "provider-includes": { "p\/provider-2013$%hash%.json": { "sha256": "d0dc77fd03e5732f1bdb80037c61df42ed94764bddb5f24774761c59739791cc" }, "p\/provider-2014$%hash%.json": { "sha256": "ab43f689a426163c7bd97eee1359d66270cf969532a976b03d4d4e608395b25c" }, "p\/provider-2015$%hash%.json": { "sha256": "5dd99e47c377393321c91a2b7cede7c1dff20f23728c89dd551c20543750cf5e" }, "p\/provider-2016$%hash%.json": { "sha256": "7b8f1749c41d2c755e25992d763588fa527308b20f53fb15523c5a938d960bc4" }, "p\/provider-2017$%hash%.json": { "sha256": "62c94621ae4a052b1d7e00cb9398703ef09bc8f69fe8fce22e309a831b6011f9" }, "p\/provider-2018$%hash%.json": { "sha256": "df4e171e392f024639a924670d6c2a243a645277d43fb3b141cef9efe7b9df12" }, "p\/provider-2019$%hash%.json": { "sha256": "166db562306e9fd8967eb173a9dbf601c65ce4a2e67bf3d18c132730f1f1f6f9" }, "p\/provider-2020$%hash%.json": { "sha256": "ff7e5318ac974ec8ce64a1d0232f038f9fbac3d987a2146719ff31c6fc578d40" }, "p\/provider-2020-07$%hash%.json": { "sha256": "22517059027c992f913d337d6b79f67b1f3305c1d969e99bfd8ec859e54d9866" }, "p\/provider-2020-10$%hash%.json": { "sha256": "9d48effa58bf64e0764d894494902ffdee59924ad8f63fe1126b315601a2cb9b" }, "p\/provider-2021-01$%hash%.json": { "sha256": "0d37334fff077049a79655aa7f4ecd4ad7a4c01b66a4fe4b24c8aa6170ae5789" }, "p\/provider-2021-04$%hash%.json": { "sha256": "070fc16ec215d0422e7791c4fb0e1279d00704fecaac221863b56f0d68bb3514" }, "p\/provider-archived$%hash%.json": { "sha256": "60d5f896e444581ddd4baae5eaf7f04b63ec618d00f1d8ddf3af1fcf41264015" }, "p\/provider-latest$%hash%.json": { "sha256": "be8f9b5a0f6e46b14e976d6f56ae297509644bbd11916bd90d6894aea4946d27" } } }

But when requiring diffferent libs, I got each time an error ; example with drush

$ composer require drush/drush

Using version ^10.5 for drush/drush
./composer.json has been created
Running composer update drush/drush
Loading composer repositories with package information
Installation failed, deleting ./composer.json.
[Composer\Repository\RepositorySecurityException]
The contents of https://private.repo.lan/p/consolidation/site-process%24f510087614b0f459ed763ade5338c9bbade5e69b1b574df465e4e13105e11643.json do not match its signature. This could indicate a man-in-the-middle attack or e.g. antivirus software corrupting files. Try running composer again and report this if you think it is a mistake.
require [--dev] [--dry-run] [--prefer-source] [--prefer-dist] [--prefer-install PREFER-INSTALL] [--fixed] [--no-suggest] [--no-progress] [--no-update] [--no-install] [--no-scripts] [--update-no-dev] [-w|--update-with-dependencies] [-W|--update-with-all-dependencies] [--with-dependencies] [--with-all-dependencies] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [--sort-packages] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--] []...

What's wrong with my mirror ??
Thanks a lot :)

I just published https://github.com/composer/mirror which supports Composer 1 and Composer 2 style metadata mirroring. I would appreciate very much if you could update this repo to use this instead of however you are doing it now. I am fairly sure our implementation is correct, and especially that it will be maintained.

I'm happy to discuss how to achieve this etc.

Also note that generally speaking I hope Composer 2 will reduce the need for third party mirrors as it does parallelize requests to the repository, and data transfer is overall smaller, so latency should be much less problematic.

https://docs.fossa.com/docs/travisci

Hello,

Unless I missed something, but I can't build a mirror behind a proxy
In streamhandler.php there is a dns request with php_network_getaddress()

The proxy works well with env and curl export https_proxy=http://X.X.X.X:port

This feature could be very useful inside company without direct internet access

Thanks

After deploying this app to a DigitalOcean droplet I was able to connect my local composer instance and install dependencies.

After enabling HTTP on the server using LetsEncrypt, composer is no longer able to fetch package.json as it appears encrypted/compressed.

I have tried enabling/disabling gzip to no avail.

Update: the file is encrypted/encoded on the server when opened with sudo nano package.json.

An error "Impossible to create the root directory" occurred during the installation.

$ cd public/p
$ ls -1 | wc -l
32825
$

This is a limit subfolder on OS. Can do subfolder of the form public/p/0/..., public/p/1/.., ... , public/p/z/..

I don’t want to switch to ZFS. Knowledge of PHP is still not enough to do it yourself.

OS: FreeBSD 12 (i386 and amd64)
FileSystem: UFS

Hi there, I'm trying to use the brazilian mirror but I'm aways getting this when trying to run composer require:

...
"https://packagist.com.br/packages.json" does not contain valid JSON
Parse error on line 1:                                                                                                                                                                          
\\\\\\Qo                                                                                                                                                                                        
                                                                                                                                                                                                
^                                                                                                                                                                                               
Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['                                                                                                                          
https://packagist.com.br could not be fully loaded, package information was loaded from the local cache and may be out of date

I tried to fetch the URL directly from curl and the return was this:

danielcosta:~ danielcosta$ curl -I https://packagist.com.br/packages.json
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Thu, 06 Sep 2018 17:28:58 GMT
Content-Type: application/json
Content-Length: 683
Last-Modified: Thu, 06 Sep 2018 17:28:40 GMT
Connection: keep-alive
ETag: "5b9163c8-2ab"
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: DENY
Accept-Ranges: bytes

danielcosta:~ danielcosta$ curl https://packagist.com.br/packages.json
Qo
  )
cy_e)Q |nw/C0L>p''ힵ_}}onkOq+}y8xұ=s>OߏNoBrwp겹'k?~      ێ3^0
                                                            pP*|EH
##=עsfI5p4'
BD8f*yam??'M^i"&\P2A6f%,qXM=TK4e;v+JN!u6(6F]p[8X]U@ڵ1gBhJ\0.3{C(tHK4J3Vp.VsSDγ^*fEr.fˎ,)^cNFGX>L1$eX
                                                                                                    =6ZOsr1Jk
danielcosta:~ danielcosta$ 

Do you have any idea why is this happening?

I tried the commands from two different machines and connections, is this a general issue or is it only happening with me?

Today morning i setup a digitalocean server to mirror packages in my nearby country / place

but it stopped downloading files after 2015

and to download meta for (2013,2014,2015) it almost took around 4+ hrs and sometimes the process just kills it self. i need to do it manually :-(

From #4

[Seld\JsonLint\ParsingException]
"http://localhost:8881/packages.json" does not contain valid JSON
Parse error on line 1:
\\\??͎
??y
^
Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

I'm getting packagist.hostuj.to refused to connect.. Recommending to remove this mirror from the list.

Hi There How i can sync packages more frequent ?

Like very 20 or 30 seconds ?

Maybe I missed something, but I don't see where illuminate/support and nesbot/carbon are being used.

I found that packagist place the zip files of packages in github, if this mirror only sync json files without package files, that if is limit used, because download file from github is also slow.

Seek solutions.

As we know the mirror only download metadata with JSON file.
Passed the first step, then download zipball for code from dst field in metadata.
But the dst download url blocked too in China. eg. https://api.github.com/xx

Can your have some solutions for the case?

Warning: include(/packagist-mirror/vendor/league/flysystem/src/FileNotFoundException.php): failed to open stream: Too many open file

$ ulimit -n
$ ulimit -n 8192

php bin/mirror create --no-clean

To reduce IO for every update and keep for a while old packages metadata: usefull for high frequency mirror vs slow connection client.

This feature need #9 to be really useful.

Better alternative to fork-change files style (sample #117)

The LICENSE file has the MIT license, but composer.json says CC0. Could you clarify which one it's supposed to be?

Also as a more minor note, the source files say "For the full license information, please view the LICENSE.md", except that file doesn't exist since it's just named LICENSE.

When I tried to build a mirror with an internal DNS name, the composer request is broken because packages.json contains repo.packagist.org

I can use mydomain.int as mirror ?

@webysther Thanks for this project.

1. I'm still trying to work out the differences between this and satis. Is packagist-mirror "just" enabling a geographical close copy while satis has a larger scope?
2. In terms of architecture, would the result of packagist-mirror just be pushed to S3 and served via something like Cloudflare?

Get parameters in services are lost during the mirroring:

# https://packagist.org/packages.json
search: "https://packagist.org/search.json?q=%query%&type=%type%",

# mirrored version
search: "https://packagist.org/search.json"

I think the issue is in src/Provider.php
(not tested myself yet.)
https://github.com/Webysther/packagist-mirror/blob/9c6d9b4a735d59fd34616f346359228d222b84ad/src/Provider.php#L109-L119
It seems to me that during path_url only the path itself is kept, get parameters are lost.

Add support for ga for website.