Giter VIP home page Giter VIP logo

rpm-adapter's Introduction

Join our Telegramm group

Javadoc License codecov Hits-of-Code Docker Pulls Docker Image Version (latest by date)

Artipie is a binary artifact management tool, similar to Artifactory, Nexus, Archiva, ProGet, and many others. The following set of features makes Artipie unique among all others:

Learn more about Artipie in our Wiki.

Publications about Artipie:

Quickstart

Artipie is distributed as Docker container and as fat jar. The jar file can be downloaded on the GitHub release page and here is a Wiki page describing how to start it. The fastest way to start Artipie is by using Docker container. First, make sure you have already installed Docker Engine. Then, open command line and instruct Docker Engine to run Artipie container:

docker run -it -p 8080:8080 -p 8086:8086 artipie/artipie:latest

It'll start a new Docker container with latest Artipie version, the command includes mapping of two ports: on port 8080 repositories are served and on port 8086 Artipie Rest API and Swagger documentation is provided. A new image generate default configuration, prints a list of running repositories, test credentials and a link to the Swagger documentation to console. To check existing repositories using Artipie Rest API:

  • go to Swagger documentation page http://localhost:8086/api/index.html, choose "Auth token" in "Select a definition" list,
  • generate and copy authentication token for user artipie/artipie,
  • switch to "Repositories" definition, press "Authorize" button and paste the token
  • then perform GET /api/v1/repository/list request. Response should be a json list with three default repositories:
[
  "my-bin",
  "my-docker",
  "my-maven"
]

Artipie server side (repositories) is served on 8080 port and is available on URI http://localhost:8080/{reponame}, where {reponame} is the name of the repository. Let's put some text data into binary repository:

curl -X PUT -d 'Hello world!' http://localhost:8080/my-bin/test.txt

With this request we added file test.txt containing text "Hello world!" into repository. Let's check it's really there:

curl -X GET http://localhost:8080/my-bin/test.txt

"Hello world!" should be printed in console.

To dive in deeper into Artipie configuration, features, explore repositories and storages settings, please, address our Wiki.

Default server configuration in Docker Container refers to /var/artipie/repo to look up for repository configurations. You may want to mount local configurations <your-local-config-dir> to /var/artipie/repo to check and edit it manually.

Important: for provided Artipie docker containers <your-local-config-dir> should have directory ownership set to 2021:2020. To change it correctly use sudo chown -R 2021:2020 <your-local-config-dir>.

If you have any question or suggestions, do not hesitate to create an issue or contact us in Telegram.
Artipie roadmap.

How to contribute

Fork repository, make changes, send us a pull request. We will review your changes and apply them to the master branch shortly, provided they don't violate our quality standards. To avoid frustration, before sending us your pull request please run full Maven build:

$ mvn clean install

To avoid build errors use Maven 3.2+ and please read contributing rules.

Thanks to FreePik for the logo.

How to release

Artipie service is released in several formats:

All these distributions are created by GitHub workflows. To publish release, push tag starting with v into this repository masted branch:

git tag v1.2.0
git push --tags origin

rpm-adapter's People

Contributors

acheshkov avatar ammaratef45 avatar andreoss avatar baudoliver7 avatar brastak avatar carlosmiranda avatar chgen avatar dependabot-preview[bot] avatar dependabot[bot] avatar dgarus avatar g4s8 avatar genryxy avatar guseyn avatar hdouss avatar marceloamadeu avatar milkysoft avatar olegmoz avatar olenagerasimova avatar paulodamaso avatar sammers21 avatar swizbiz avatar victornoel avatar yegor256 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

rpm-adapter's Issues

Update readme

Examples in readme should be updated as Rpm is deprecated in favour to RpmAbstraction.Base.

Also please correct phrase about Storage implementation:

Then, you implement com.yegor256.rpm.Storage interface.

Storage from artipie.asto should be implemented.

primary.xml is not generated entirely correct

We are not filling all XML elements, as they are filled by the createrepo. Try to create a sample repository with createrepo and compare with what we are generating in our XML files. You will see that we don't fill every required element. Many of them are simply omitted. This is definitely a bug.

CLI tool for RPM metadata generators

Let's create simple CLI tool to generate metadata for rpm packages. We need some class for this CLI with main() method (don't forget to define main method in pom.xml). It will be a wrapper for Rpm class. The syntax of this tool can be:

java -jar rpm-adapter.jar --naming-policy=simple --digest=SHA256 update ./package.rpm
# or
java -jar rpm-adapter.jar batchUpdate ./repo-dir/

So there are some options, one command, and one (or more) required positional arguments.

Options are:

  • naming-policy - (optional, default simple) configures NamingPolicy for Rpm
  • digest - (optional, default sha256) configures Digest instance for Rpm

Commands are:

  • update - update one or more .rpm packages. It's equivalent of Rpm#update call
  • batchUpdate - update specified directory as RPM repository, same as Rpm#batchUpdate call

Rpm.java:67-71: Add option to exclude `filelists.xml`...

The puzzle 69-a899429c from #69 has to be resolved:

* @todo #69:30min Add option to exclude `filelists.xml` metadata file from
* updates. Som RPM repositories contains too many files in RPM packages,
* so it may take too many time to update the filelists. Just add an option
* to `Rpm` constructor and exclude filelists output from `Repository`
* list of metadata files in `batchUpdate` method.

The puzzle was created by @g4s8 on 08-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

RpmTest.java:54-56: This test class needs refactoring....

The puzzle 32-75a1be06 from #32 has to be resolved:

* @todo #32:30min This test class needs refactoring. What is still need to be done
* is to get rid of multi-threading in matcher, to simplify the test,
* and to test only one thing in each unit test (split test to multiple if needed).

The puzzle was created by HDouss on 12-Mar-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Re-implement docker container start with testcontainers

Now integration test uses ProcessBuilder to execute docker. Testcontainers library provides higher abstraction level. It allows to create simple-in-use and reusable components. Also, it's designed for the unit testing purpose and provides lots of useful containers to create necessary environment for integration tests runs.

We need to rewrite existing code to start docker container with testcontainers library usage

StringIndexOutOfBoundsException in PrimaryProcessor

The file v8location_rootfs-1213-lpu-1.0.0-3.aarch64.rpm causes unexpected failure in RPM adapter (version 0.51.1), exception:

'String index out of range: 0'
java.lang.StringIndexOutOfBoundsException: String index out of range: 0
        at java.lang.String.charAt(String.java:658) ~[?:1.8.0_212]
        at com.artipie.rpm.PrimaryProcessor.writeFiles(PrimaryProcessor.java:298) ~[rpm-adapter-0.51.1.jar:?]
        at com.artipie.rpm.PrimaryProcessor.lambda$null$5(PrimaryProcessor.java:199) ~[rpm-adapter-0.51.1.jar:?]
        at io.reactivex.internal.operators.completable.CompletableFromAction.subscribeActual(CompletableFromAction.java:34)

Update interface needs to be redefined

We should redefine this interface to support different business scenarios.
For example

class RepoData {
       String filename.
       InputStream fileStream.
}
RepoData repoData updata(RepoData[] repodata,String[] fileList,String model)

Description:
1、repodata:Existing repodata data needs to be transferd
2、filelist:Batch files need to generate repodata.
3、model: Add or delete files
4、we hope you can return the file stream.

MetadataFile.java:45-46: Add a unit test for this class,...

The puzzle 69-b66a2919 from #69 has to be resolved:

* @todo #69:30min Add a unit test for this class, it should verify that
* it updates `repomd.xml` file correctly after save.

The puzzle was created by @g4s8 on 08-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Repomd.java:191-194: This method and saveGzip are very...

The puzzle 41-ab144851 from #41 has to be resolved:

* @todo #41:30min This method and saveGzip are very similar
* but implemented differently. Factor those together as
* the same method based on the implementation of the saveGzip
* that is closer to the rx paradigm than saveRepo.

The puzzle was created by Victor Noël on 31-Mar-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Repomd.java:142-146: Continue refactoring...

The puzzle 41-df9cd1a3 from #41 has to be resolved:

* @todo #41:30min Continue refactoring Repomd.performUpdate method
* and all other dependencies: it is very long and complex to read
* and understand without reading everything. The content of
* the zipWith below should be extracted in its own method. If possible
* make it a class that can be tested by itself.

The puzzle was created by Victor Noël on 02-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Repomd uses links to src files instead of gzipped

Metadata file repomd.xml uses wrong links. E.g. if a source file has a name 123abc-filelists.xml (where 123abc is a hashsum of the file), and gzipped file has a name 567def-filelists.xml.gz, then repomd.xml will contains a link:

<data type="filelists">
  <location href="123abc-filelists.xml.gz" />
  <!-- other data -->
</data>

So it just appends .gz suffix to source file name instead of using correct gzipped file name.

Batch update support for rpm repository

Now rpm-adapter can generate the repodata files for a single rpm package. That's not enough.
We should support directory-level repodata for all the rpm packages in one directory. As createrepo did.

WithFilelists.java:34-37: Implement WithFilelists...

The puzzle 17-f66e78ab from #17 has to be resolved:

* @todo #17:30min Implement WithFilelists decorator.
* WithFilelists decorator create filelists.xml and filelists.xml.gz files on
* upload. Implement this behavior and then enable the test in
* WithFilelistsTest.

The puzzle was created by paulodamaso on 24-Mar-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

More unit tests

Let's add more tests for Rpm metadata generators:

  • verify that it can update multiple files one by one
  • verify that the content is correct for all supported digests and NamingPolicy implementations
  • verify that it doesn't brake existing metadata in case of invalid package (e.g. generate metadata for first package, then try to generate for broken package)

com.artipie.rpm.RpmTest#addsSingleRpm is broken

The test com.artipie.rpm.RpmTest#addsSingleRpm is broken with error

java.io.FileNotFoundException: /etc/hosts (Too many open files)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:155)
	at io.netty.resolver.HostsFileParser.parse(HostsFileParser.java:132)
	at io.netty.resolver.HostsFileParser.parseSilently(HostsFileParser.java:89)
	at io.netty.resolver.HostsFileParser.parseSilently(HostsFileParser.java:76)
	at io.vertx.core.impl.resolver.DnsResolverProvider.<init>(DnsResolverProvider.java:123)
	at io.vertx.core.spi.resolver.ResolverProvider.factory(ResolverProvider.java:39)
	at io.vertx.core.impl.AddressResolver.<init>(AddressResolver.java:75)
	at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:169)
	at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:27)
	at io.vertx.core.Vertx.vertx(Vertx.java:75)
	at io.vertx.reactivex.core.Vertx.vertx(Vertx.java:118)
	at com.artipie.asto.fs.RxFile.<init>(RxFile.java:64)
	at com.artipie.rpm.Repomd.lambda$performUpdate$17(Repomd.java:197)

MacOS.

Don't generate xml files if not needed

Rpm class generates 7 files on update():

  • repomd.xml
  • primary.xml
  • primary.xml.gz
  • filelists.xml
  • filelists.xml.gz
  • other.xml
  • other.xml.gz

Most probably primary.xml, filelists.xml and other.xml files are not needed, since repomd.xml (the main metadata file) has a links only to .xml.gz versions of these files. Check if createrepo tool generates .xml versions of metadata files and don't upload it to storage in Repomd class if not needed.

HeaderTags.java:33-34: Write javadocs for all these...

The puzzle 69-5a24f18b from #69 has to be resolved:

* @todo #69:30min Write javadocs for all these method. When done, remove
* JavadocMethodCheck suppression annotation for PMD checker.

The puzzle was created by @g4s8 on 08-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

RpmTest.java:52-56: Refactor this test. Convert it to...

The puzzle 30-6f533735 from #30 has to be resolved:

* @todo #30:30min Refactor this test. Convert it to Junit5 and use `TempDir`
* extension, refactor same logic for extracting rpm file from resources,
* get rid of multi-threading in matcher, simplify the test,
* test only one thing in each unit test (split test to multiple if needed).
* Also, remove supressed PMD warning.

The puzzle was created by @g4s8 on 10-Mar-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

FilePackage.java:113-115: Create unit tests to verify...

The puzzle 69-3c6bbf6c from #69 has to be resolved:

* @todo #69:30min Create unit tests to verify that
* FilePackage headers can parse headers and header range correctly,
* can calculate file size and check-sums correctly.

The puzzle was created by @g4s8 on 08-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Cleanup files on delete

We should cleanup removed files on Rpm.batchUpdate, e.g. if repository has these files:

111.rpm
222.rpm
333.rpm

and user called Rpm.batchUpdate, these files will be added to metadata files. But if user removes any 222.rpm file:

111.rpm
333.rpm

and call Rpm.batchUpdate, then all metadata files will contains links to 222.rpm. We need to remove this file from metadata files too.

XmlOthers.java:53-55: Create a unit test to verify that...

The puzzle 69-0165dd8a from #69 has to be resolved:

* @todo #69:30min Create a unit test to verify that this class
* can write `others.xml` file correctly. The example of others.xml can
* be found at test resources.

The puzzle was created by @g4s8 on 08-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

The return value needs to be optimized

Once the updata processing is complete, you need to return to the production file stream list. It includes other file stream/ primary file stream /repomd.xml file stream /filelist file stream.

Rpm doesn't remove old versions of metadata file

When we use some custom naming policy for metadata files (e.g. hash prefixes), then Rpm can update these files but doesn't remove old versions, e.g. if we have files:

111-filelists.xml.gz
222-others.xml.gz
333-primary.xml.bz
package-one.rpm
repomd.xml

and add new package package-two.rpm, then filelists, others and primary will be updated and the name of them will change (because of hash based naming policy): 111-filelists -> aaa-filelists, 222-others -> bbb-others, 333-primary -> ccc-primary. repomd.xml will have correct links to new files, but old files will present in the storage:

111-filelists.xml.gz
222-others.xml.gz
333-primary.xml.bz
aaa-filelists.xml.gz
bbb-others.xml.gz
ccc-primary.xml.bz
package-one.rpm
repomd.xml

We need to remove old versions of files after such updates.

Mutil-file scenario repodata produced by rpm-adapter is wrong. please fix it.

When we save three rpm package information by blockingStorage, then we use the rpm.batchUpdate to to generate the repodata,the problem happens as below:

1.The rpm-adapter generated three set of filelist/primary/other files, this is ok, but we prefer to get a set of repodata data.
2. the newest set of filelist/other/primary files only contains one package basic information, this point is wrong.

I have sent the incorrect repodata.rar to kirill, please refer to them.

fix it. @g4s8 thanks.

Repomd.java:166-170: Refactor Repomd.performUpdate...

The puzzle 29-399d2455 from #29 has to be resolved:

* @todo #29:30min Refactor Repomd.performUpdate method.
* It was hot-fixed by adding additional required parameters,
* it's hard to read and understand without reading the whole code.
* Most probably this method can be split into multiple methods
* or some parts can be moved to another class.

The puzzle was created by @g4s8 on 13-Mar-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Remove incorrect usage of doOnSuccess

I noticed in the code some places where doOnSuccess was used, but incorrectly IMHO.

Sometimes the argument passed to doOnSuccess returns a Completable, but it will never be executed because doOnSuccess does nothing with it. Also sometimes it makes more sense to use map when the passed lambda use the builder pattern.

In this module, we have that is wrong

.doOnSuccess(
file -> this.metaFile(
new XMLDocument(repomd.toFile())
.registerNs("ns", "http://linux.duke.edu/metadata/repo").nodes(
String.format("/ns:repomd/data[type='%s']", type)
),
file
)
)

We also have that should use map imho:

return Single.just(new RepoXml())
.doOnSuccess(
rxml -> rxml.base(type, gziploc)
).zipWith(

There are no other place in this module, but I think it would make sense to fix it also in other modules? there are no other module with this problem

Update all tests to use junit 5

Currently we use junit 4 in several unit tests. Actual runner is junit 5, but these tests use junit 4 and it works together with junit-vintage-engine.
Need to upgrade all tests to junit 5 and remove redundant libraries.

Package.java:49-50: This class has too many methods,...

The puzzle 69-56010550 from #69 has to be resolved:

* @todo #69:30min This class has too many methods, consider refactoring it.
* Most probably headers methods can be moved to another Headers (or Tags) object.

The puzzle was created by @g4s8 on 08-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.