What is the plan how to implement the search module in the artipie engine?

• If we introduce a database to Artipie, can we use the search feature in that database?
• Do we need to introduce Elasticsearch to Artipie?
  We must also understand what kind of search for artifacts are currently supported by the existing commercial software. By doing a little bit search, here is the list from https://www.jfrog.com/confluence/display/RTF6X/Searching+for+Artifacts. However, we must look at the customers how they current use and prioritize these search features.
• Quick: Search by artifact file name.
• Package: Search for artifacts according to the criteria specific to the package format.
• Archive Entries: Search for files that reside within archives (e.g. within a jar file).
• Property: Search for artifacts based on names and values of properties assigned to them.
• Checksum: Search for artifacts based on their checksum value.

We will also implement an endpoint for all searching queries.

transfer the example config _config.yml to the example directory

Create benchmark tests and choose the environment. Investigate what benchmarks we may use, how to implement it and where to test.

Support for gem packages in artipie

The puzzle 12-94a9871a from #12 has to be resolved:

The puzzle was created by @g4s8 on 21-Feb-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Create simple Artipie server. It should:

• read configuration files and include adapters from these files
• delegate HTTP requests to enabled adapters
• implements Front interface from artipie/http module

Add documentation how to extend Artipie and create custom module or storage

The puzzle 12-c03f07e2 from #12 has to be resolved:

The puzzle was created by @g4s8 on 13-Feb-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

Let's update rultor image

Extend RepoConfig with method that will return repo-specifig configuration part.
Proposed naming: method custom() that returns YAML node custom. Example:

repo:
  type: npm-proxy
  path: npm-proxy
  storage:
    type: fs
    path: /var/artipie/npm-proxy
  custom:
    remote-url: https://registry.npmjs.org

Artipie just returns 500 error without any message if repository file was not found or directory doesn't exist. Let's return 404 message with text that repository %s was not found

When trying to build Docker image with docker build . I'm getting an error:

[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.460 s
[INFO] Finished at: 2020-03-23T08:03:39Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:go-offline (default-cli): Goal requires a project to execute but there is no POM in this directory (/). Please verify you invoked Maven from the correct directory. -> [Help 1]

These methods were added because YAML library we use doesn't check for presence of value by key and just return null. We can change library, improve this library or think of some other solution maybe.

When I run command latexmk -pdf wp as in the README I get this error:

! LaTeX Error: File `footmisc.sty' not found.

Inside class YamlSettings there are the mapping() and string() methods which are basically checking for null values in the YamlMapping.

In the library eo-yaml there is the decorator StrictYamlMapping which already performs those checks and complains about it.

I know because I wrote the library.

On every request received by artipie (when using the default Pie slice), we do the following operations:

• retrieve storage from settings, which reads the yaml content (does not read the file again, but the file content as read at the first time) for "meta -> storage" entry.
• get the value of this storage scalar.

This is corresponding to that code:

CompletableFuture.supplyAsync(
    () -> new Unchecked<>(this.settings::storage).value()
)

These two operations are redundant because they will always have the same result, there is no need to make it at every received request.
Ideally, if we won't support hot changing repos config files, we could implement a sort of cache that will return the whole slice (that is decorated by the AsyncSlice) based on the repo. Every repo is associated to a single slice, there is no need to recalculate that at every request. By "the whole slice that is decorated by the AsyncSlice", I meant this part of the code:

CompletableFuture.supplyAsync(
    () -> new Unchecked<>(this.settings::storage).value()
)
.thenComposeAsync(
    storage -> storage.value(new Key.From(String.format("%s.yaml", repo)))
)
.thenApply(content -> new RepoConfig(content))
.thenCompose(Pie::sliceForConfig)

Parse Artipie server settings from yaml file as in white-paper:
https://github.com/artipie/white-paper/releases/download/30d9ae/white-paper.pdf

It's stored as a file with format:

meta:
  storage:
    type: fs
    url: file:///artipie/settings

We need to construct Storage implementation from ASTO library using storage settings, where
type param may have value fs and url a link to root settings directory location.

I think it can be implemented as a class which implements Storage interface and encapsulates yaml configuration, e.g. Storage settings = new AstoSettings(yaml).

Artipie prints this warning on each incoming HTTP request:

WARNING: You're already on a Vert.x context, are you sure you want to create a new Vertx instance?

It seems we're creating new Vertx instance for storage on each request.

create example directory with exaples, simple config etc

Create Permissions (from artipie/http module) implementation based on repository yaml configuration files. Permissions structure includes list of users and list of permissions for each user.

We need to support wildcard * permissions and users. Permissions for * user means that any user has such permissions, * permission for user means that the user is allowed to do everything.

Example:

repo:
  type:
    maven
  storage:
    type: fs
    path: /var/artipie/maven
  permissions:
    # admin can do everything
    # john can deploy and delete 
    # jane can only deploy
    # any user can download
    - admin:
      - "*"
    - john:
      - deploy
      - delete
    - jane:
      - deploy
    - "*":
      - download

For now, properties are passed in the following way:

-Dartipie.storage=./config.yml -Dartipie.port=8080

That is not gonna work in case is compiled as a native image. We should move to command line args

Docker image should provide a way for config file to be persistent. Usually it is achieved with copying pre-defined config to the volume path on the first run

I tried to start docker image from latest master with commands:

docker build -t artipie:local .
docker run -v /tmp/artipie:/artipie/repositories artipie:local

And got an error:

APP main com.artipie.VertxMain: Using default port: 80
Exception in thread "main" java.lang.IllegalStateException: Storage is not configured
	at com.artipie.VertxMain.main(VertxMain.java:108)

Issue to accumulate code style improvements

Add RPM repository HTTP API.

We should provide opportunity to implement and use a custom auth method.

Which will be implemented in Artipie from the box:

• - Users log into Artipie via user-name and password;

• - GitHub tokens;

• - Users to authenticate using KeyCloack.

• - Custom auth method (create according abstractions);

• authenticate using SAML SSO providers like ADFS, OneLogin and Okta. (won't be implemented, since repository authentication mechanisms doesn't support these providers).

The minimum artipie configuration, for now, requires 2 files. Which makes it a little painful for a newcomer to get started with the project.

I would propose a single file configuration option:

meta:
  storage:
    type: in-place
repo:
  type: rpm
  storage:
    type: fs
    path: /var/artipie/centos

This option will definitely make Artipie more user-friendly.

The example given in the documentation of the repo settings for the NPM example gives a confusing permissions example. This is more important as it is the only permissions example in the documentation.
admin and the other usernames does not have the same indentation and its child element is 4 spaces away.
I expect the other usernames to be in the same indentation level as admin and that every child element will always be 2 space-indented.

It should be possible to install artipie on almost any platform and obtain as:

• a .rpm/.deb package available for major Linux distributions via dnf/apt/yum package managers
• a brew package for Mac OS X
• a docker container (#41)
• a fat.jar file

Now we have approach then requires multiple config files for every repository. I don't think that this approach is manageable. What's wrong with the idea - every repository defines everything: at this moment at least storage and users. But in the most cases these things should be globally defined. If we have 20 managed repositories, almost all of them will use the same storage (some part of the common storage). If administrator decides to change storage location he must change 20 configs. If administrator is asked to add new user that should have access to the repos, he must change 20 configs.

I think, we need to redesign this approach. One possible solution is to introduce new high-level objects like storages, roles, groups and users (or, maybe, user-provider and group-provider, I don't think that file lists should be the only possible approach) and to allow cross-references between files. But this approach works good only for when we don't need dynamic changes - all config files are read on startup and one global config is created.

I tried to start docker image from latest master with commands:

docker build -t artipie:local .
docker run -v /tmp/artipie:/artipie/repositories artipie:local

And got an error:

Error: Could not find or load main class com.artipie.Main
Caused by: java.lang.ClassNotFoundException: com.artipie.Main

Docker image should find main class correctly.

What is our strategy for the high availability of Artipie?
For example, in Apache Archiva project, it supports two different installations
Option 1: master-slave-strategy (not really a cluster). This option may not be good enough for us, as there will be downtime when upgrade happens.
Option 2: two Servers + httpd-balancer + shared file system. How can we keep the file system synced if necessary?
In our case, we might also need to support multiple regions installation.

Let's add Docker and Nuget adapters to Artipie, since we already have Slice implementation for them.

The puzzle 12-b6e20264 from #12 has to be resolved:

The puzzle was created by @g4s8 on 11-Feb-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

The puzzle 69-a0681291 from #69 has to be resolved:

The puzzle was created by olenagerasimova on 13-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

The puzzle 76-fe362791 from #76 has to be resolved:

The puzzle was created by @g4s8 on 22-Apr-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

I noticed the Dockerfile is extremely long to build, it looks like it is downloading the whole internet even though it's not needed when one just want to build the package.

I propose to:

• not call dependency:go-offline
  • it's useless since it is used for building the package 1 second later...
  • when calling mvn package, it will download only the needed dependencies.
• skip tests, qulice and stuffs like that, it's not needed to build the image, the CI is already protecting us from it

RepoConfig class has a path method. The implementation reads the yaml setting repo.path in the repo file setting.
First, it is not documented at all that the repo file settings could have an entry like repo -> path.
Second, I am not sure if that is redundant with repo -> storage -> path, in which case, the method should just disappear.
Finally, the only usage of this method is when building PhpComposer (in Pie class). Looking at the usage of this data in PhpComposer makes me wonder what this "base" is for, it is only used to check if the incoming request has that "base" as a prefix.
Whether there is some sort of a bug or there is a clear lack of documentation: at least what is a "base path" for PhpComposer slice, and how it is different from its storage path.

It returns 500 error, let's investigate and fix it.

I suppose this is a mistake, I noticed all other projects are depending on ppom 0.3 but artipie module depends on ppom 0.2.1 and my IDE sets it up with Java 11 instead of Java 8.

The README seems outdated:

• it's referring to old ways of configuring artipe using system properties in the "How to start" section
  • instead of using CLI options
• it is missing instructions to build the docker image
• it contains duplicate information between "Setup the server" and "Configuration" sections

Support NPM adapter as Artipie module

Suggested format:

meta:
  storage:
    type: s3
    bucket: my-bucket
    endpoint: https://obs.ap-southeast-3.myhuaweicloud.com
    region: ap-southeast-3
    credentials:
      type: basic
      accessKeyId: ***
      secretAccessKey: ***

Only required parameter would be bucket, all others may be not specified.

• Default uri is amazon URI which is embedded in S3 client
• Default region could be taken from environment, see https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html
• Default credentials are anonymous

When starting new docker image I'm getting an error:

Exception in thread "main" APP main com.artipie.VertxMain: Using default port: 80
java.lang.IllegalStateException: Storage is not configured
	at com.artipie.VertxMain.main(VertxMain.java:108)

Please create a data model (or link to an existing one) for the artipie project. Something like this
Project: ID, name, namespace
ProjectVersion: ID, Version, repsitoryID, etc
Artifact: ID, namespace, etc,
Repository: .....
.....

The puzzle 40-06b6b8fc from #40 has to be resolved:

The puzzle was created by @Sammers21 on 18-Mar-20.

Estimate: 30 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

• Create a docker file for artipie(#43)
• Deploy a container to Docker Hub via Github Actions(#44)
• Update readme in accordance with the change