Thanks again for sharing this repo and making it easy to host binaries repos.

I understand that currently in order to use the replicate script, user need to have an AWS account to push artefacts on their own bucket, and then copy the bucket content to their on-prem web server. Please correct me if I'm wrong. This implies 3 transfers:
1- source -> desktop
2- desktop -> aws
3- aws -> onprem web server

I was wondering how to make this more independent of AWS and faster. Is there a way to access the files on the local

Would usage of fog storage abstraction enable other object store providers, including local file systems while preserving support AWS S3 ?

On prem users could just rsync/scp their local.

Alternatively, an option to only save on local disk and ignore s3 upload could make it, then let users of other blobstore (e.g. rackspace or google storage) pick native tools to sync the local repo.

OpenJDK said its jdk but when I download it, its just JRE, is this intended behavior?

We recently experienced an outage when replicating with the new ssl code. #8

When an artifact fails to download it is deleted in the mirror.

It seems to me that the replication code should do what it can to preserve the mirror in the event of any failure. Otherwise if my org's internet or proxy goes down my repo would get deleted and non of my apps would be able to stage.

Thanks again for sharing the java-buildpack-dependency-builder, and its replicate script to be able to replicate artifacts within an offline private paas instance.

Following @nebhale 's suggestion to have users request most needed features in the java buildpack, it would be great if the java-buildpack-dependency-builder provided a way to trace back to the update sources and verify the integrity/authenticity of the artifact binaries.

Most artifacts repos make available checksums or digital signatures that enables their users to validate that binaries in downstream distributions properly match original distribution. This allows users to get protected against ~"man-in-the-middle" attacks that would inject malicious code into the replicated artifacts run by the java buildpack.

E.g. bintray provides a way to digitally sign artifacts using PGP keys, see also related binary manual.
As an example, the upload source for groovy is bintray, onto which the PGP signature is available as HTTPS download urls. Groovy ships and digital signature associated to each release (e.g. groovy-binary-2.2.2.zip.asc )

So the need is for java-buildpack users or paas operators using the replicate script to be able to verify that their locally replicated artifact match the checksum/hash or digital signatures of the corresponding update locations. Ideally this verification would be done automatically.

As an example of the value associated to this feature, some company policies require that binaries run in production can be tracked back to their origin, being either commercial vendors, distributions or opensource communities. This could prevent CloudFoundry java-buildpack adoption in those.

Some possible ways this could be provided:

• for users to semi-manually verify origin of replicated artifacts, provide a machine readable description of each artifacts with the following (could that fit into the current index.yml file) ?
  • the local file path (e.g. groovy/groovy-2.2.1.zip) of the artifact
  • the original update location (e.g. http://dl.bintray.com/groovy/maven/groovy-binary-2.4.0-beta-2.zip )
  • the traceability/authenticity verification method (eg 'pgp' or 'md5') and associated params (e.g. http://dl.bintray.com/groovy/maven/#groovy-binary-2.2.2.zip.asc and https://bintray.com/user/downloadSubjectPublicKey?username=groovy)
• users trusting the buildand replicate scripts to verify authenticity of the fetched/ replicated artifacts. The script would then automatically verify the authenticity of fetched/replicated artifacts against update locations, and fail on differences. Paas users or Paas operator would merely have to rate their confidence on the update source to be confident about code they use in their app.

It seems that the download url used doesn't have links matching the version regex. Maybe the page changed recently. The only page that has the "old" style named link is the archives page but that doesn't include the latest version.

The issue is that concourse can't find any versions for your-kit because of the above problem.

I replicated the latest from http://download.run.pivotal.io/ and it appears to not be replicating many newer versions of Tomcat.

Doing a fresh sync it only downloads the tomcats listed in the s3 bucket here http://download.run.pivotal.io/.

Although http://download.run.pivotal.io/tomcat/tomcat-7.0.54.tar.gz does exist tomcat-7.0.54.tar.gz doesn't appear to be getting replicated.

I'm running latest master.

We have a cron that executes replication. We'd like to set up our cron to email us the log if anything about the script fails. To do this we need the replicate script to return a non 0 value when anything goes wrong.

I think this would include the failure to download any of the artifacts. When the Proxy wasn't correctly handling ssl it would have been useful for us to know about the failure to download artifacts before our users.

It looks like all the build scripts under bin/ have been moved to some concourse pipeline setup, but there is no pointer to the pipeline definition.

I was hoping that there is still some mechanism to manually build dependencies locally for testing purposes, but can't figure out how to do it. I guess I could try playing with the latest versions of the build scripts, before they got deleted, but they may no longer work anyways, so this may be a waste of time.

While trying to create an offline-buildpack against the cflinuxfs3 stack, I noticed that the jvmkill asset being pulled in that should be ELF libraries, is just text. Can this be corrected?

https://java-buildpack.cloudfoundry.org/jvmkill/bionic/x86_64/jvmkill-1.13.0_RELEASE.so

Do you know if download.pivotal.io is committed to keep all the versions in the index.yml? I am worried that when using this script, new index might remove the paths to old versions.

Thanks in advance

Perhaps caused by 78c1dbe

Still investigating.

Hey! I'm a really big fan of this tool! I used the old version (using bundle) to make a copy to my server and automatically change all the index.yml.

Now, I've just tried the new version (using aws cli) and I've checked the first downloads to be sure it's working right but unfortunately it doesn't. I could not check if after downloading all the files it updates the content of the index because I ran out of space in disk. Does it changes when downloading process is finished?

Before creating a new VM I'd like to be sure that this tools stills enables changing the path to my server in the index automatically. Is it possible?

Thanks in advance

The VM needs to have a newer version.

WARNING: C++ compiler too old, need g++ 4.8 or clang++ 3.4 (CXX=g++)

We host our internal buildpack cache under a context "http://someurl.com/java-buildpack" .

The replicate job only accepts a "host-name". As a workaround I can set a host-name of --host-name someurl.com/java-buildpack. That said, I thought I'd present the idea of setting a base url instead in case that workaround stops working. :)

Mike