Version change:

<aws-java-sdk.version>1.11.478</aws-java-sdk.version>

By:

<software.amazon.awssdk>2.10.24</software.amazon.awssdk>

Dependency change:

            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-kinesis</artifactId>
                <version>${aws-java-sdk.version}</version>
            </dependency>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-s3</artifactId>
                <version>${aws-java-sdk.version}</version>
            </dependency>

By:

            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>kinesis</artifactId>
                <version>${software.amazon.awssdk}</version>
            </dependency>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>s3</artifactId>
                <version>${software.amazon.awssdk}</version>
            </dependency>

Because AWS Kinesis can be used as a Transaction Store and AWS S3 as a Snapshot Store.

Because we need to know the storeName to do migrations that depend on the store.
The storeName does not need to be stored, just to be provided when the SerializableSnapshot is deserialized

The new StoreManager.maintenance() method is added.
It switch the database in read only (no Command can be processed).
Queries are still working.
Once maintenance mode is effective, a snapshot is autmatically triggered.

We may want to bootstrap a store from a snapshot at first startup, and so without the transactions related to it.

In such a case, because the transactions store is not aligned with the snapshot, the first transaction is ignored.

Store#query(Consumer<ReadOnlyTransaction>) don't let the possibility to return value without cumbersome way like using a container object declared outside the query call

The trick to offer support for store migration in Iron by the means of custom client-side deserializers performing migration operations should be refactored to use client-defined post-processor approach.
Iron should invoke client-side defined post processing step after deserializing store snapshot.
Version management should be also simplified in Iron to use simple variable at store level to maintain store version (instead of older VersionPO - way).

During a layout migration, only snapshots are migrated, not transactions.
In SPI File, the transaction counter is based on the transactionId of the last transaction file.
After a migration, transactions are numbered from 0 (zero) instead of last snapshotId.

Description

Iron SPI file get stuck under highly concurrent access :

How to reproduce

Reproduction Steps

• Apply patch HighlyConcurrentFileStoreTest.zip
• Run the test (Not Debug, because it does not get stuck in Debug)

Actual result

• After minutes, no more transactions are created (around 3 000 are created, but 10 000 expected) - see iron\iron\iron-spi-file-inttest\iron-store-XXX\tx
• Never terminates

Expected result

• Terminates after 10 000 transactions

Add check for consistency on "nextId" entries in the snapshots to avoid assuming values under the highest "id" in the instance set.

Reproduction Steps
Using an object definition with 2 @unique constraints (like SourcePO.class on uniqueName and secret fields)
Create a 1st entry (uniqueName = name1, secret = secret1)
Create a 2nd entry with a violation (uniqueName = name2, secret = secret1 or uniqueName = name1, secret = secret2)) -> should raise an expected error
Create a 3rd entry without violation (uniqueName = name2, secret = secret2)

Actual result
The 3rd entry creation is failing because of a supposed Unique Constraint Violation

Expected result
The 3rd entry is correctly stored

Instead of using the store name as the stream name, I want to add a prefix to the store name.
stream name = prefix + store name

That leads to a huge CPU consumption (125% with 1000 stores).

Support enumerations as field types, doing the translation between "JSON strings" and "Java enumerations" declared in the models

When many stores are created with a FileTransactionStore, the consumer loop keeps looking for next transaction with a Files.exist(). This IO being done every 10ms for each store, the process is quickly overloaded.

Example :
With:

MY_VAR_ENV=myVarEnvValue
-DmySysProperty=mySysValue

iron.properties:

myProperty=prefix/${SYS:mySysProperty}/sysToEnv/${ENV:MY_VAR_ENV}/envToProp/${otherProperty}/rest
otherProperty=myOtherPropertyValue"

Should be replaced by:

myProperty=prefix/mySysValue/sysToEnv/myVarEnvValue/envToProp/myOtherPropertyValue/rest

Kafka spi encounters this issue.

When generating javadocs of the model classes, annotations like @unique and @id are not marked as @documented. The result is that the annotations are taken in account by javadoc.

Inttests for AWS SPIs need the AWS services (Kinesis and S3) to be run. They are currently disabled.
AWS localstack (https://github.com/localstack/localstack) have to be deployed on Travis to be able to enable back theses tests.

In order to update a multiple relation between Iron entities (add, remove etc. value in the collection), retrieval of the entire collection is needed. This collection needs to be modified and the updated collection needs to be set back in the entity attribute.

Instead of this strategy, we would like to be able to directly affect the collection by adding or removing one ore more elements.

The configuration part is using sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl which is not part of the Java API.
http://www.oracle.com/technetwork/java/faq-sun-packages-142232.html

Just need to be replaced by the interface java.lang.reflect.ParameterizedType

• allocate resources at factory level and share them between stores
• snapshots should also be at factory level
• get rid of consumer loop

Instead of instantiating the different factories myself, I would like to be able to provide a Properties object containing all needed configuration and let Iron instantiate them itself.

Some Transaction storage have a "at least once" semantics (e.g. Kinesis), duplicate transactions have to be ignored.
As transaction ids are in strict ascending order, Iron can ignore transaction which id is less or equal to the greatest processed transaction id.

• IronMigration : miss some snapshots when tenants have not the same snapshot ids
• SPI File : snapshot does not occur after a single transaction

Because transactionId type long cannot support Kinesis sequence number (no limit on the digit number)

• manage the 1000 objects limitation returned by listObjects
• manage the situation of a snapshot is read during a snapshot written in progress

Currently, Iron does not offer any support regarding store migration. Iron clients need to deal with the natural need to have changes at store level applied.