apache / ignite Goto Github PK

Apache Ignite

License: Apache License 2.0

JavaScript 0.01% Shell 0.21% Java 78.25% C++ 7.55% PHP 0.02% HTML 0.01% Batchfile 0.07% C# 13.05% C 0.01% PowerShell 0.02% Makefile 0.01% Dockerfile 0.02% Python 0.61% CMake 0.09% Jinja 0.06% FreeMarker 0.03%

distributed-sql-database iot osgi network-client ignite data-management-platform big-data cloud database network-server

ignite's Introduction

Apache Ignite

What is Apache Ignite?

Apache Ignite is a distributed database for high-performance computing with in-memory speed.

Multi-Tier Storage

Apache Ignite is designed to work with memory, disk, and Intel Optane as active storage tiers. The memory tier allows using DRAM and Intel® Optane™ operating in the Memory Mode for data storage and processing needs. The disk tier is optional with the support of two options -- you can persist data in an external database or keep it in the Ignite native persistence. SSD, Flash, HDD, or Intel Optane operating in the AppDirect Mode can be used as a storage device.

Ignite Native Persistence

Even though Apache Ignite is broadly used as a caching layer on top of external databases, it comes with its native persistence - a distributed, ACID, and SQL-compliant disk-based store. The native persistence integrates into the Ignite multi-tier storage as a disk tier that can be turned on to let Ignite store more data on disk than it can cache in memory and to enable fast cluster restarts.

ACID Compliance

Data stored in Ignite is ACID-compliant both in memory and on disk, making Ignite a strongly consistent system. Ignite transactions work across the network and can span multiple servers.

ANSI SQL Support

Apache Ignite comes with a ANSI-99 compliant, horizontally scalable, and fault-tolerant SQL engine that allows you to interact with Ignite as with a regular SQL database using JDBC, ODBC drivers, or native SQL APIs available for Java, C#, C++, Python, and other programming languages. Ignite supports all DML commands, including SELECT, UPDATE, INSERT, and DELETE queries as well as a subset of DDL commands relevant for distributed systems.

High-Performance Computing

High-performance computing (HPC) is the ability to process data and perform complex calculations at high speeds. Using Apache Ignite as a high-performance compute cluster, you can turn a group of commodity machines or a cloud environment into a distributed supercomputer of interconnected Ignite nodes. Ignite enables speed and scale by processing records in memory and reducing network utilization with APIs for data and compute-intensive calculations. Those APIs implement the MapReduce paradigm and allow you to run arbitrary tasks across the cluster of nodes.

ignite's People

Contributors

Stargazers

Watchers

Forkers

sirinath bbejeck chenghao-intel amuise mindis gridgain siyuanhe ashwin-os sergeyeliseev areddy7021 narayana1208 sssankar8 cstroe abhishek-ch wandisco gilv applied-duality alphadyz evans-ye easycoder99 erenavsarogullari yaloo chhao01 willshen xiaoyuyao johnnykwon umeshvk hj5 surpass ilganeli louishust vimal-das agura heftysolutions ashutakgg raulk vsuslov hao707822882 dmagda arijitt sboikov vsisko akuznetsov-gridgain agoncharuk codegladiator ptupitsyn svladykin ksatya77 nightwolfzor vladisav rlxrlxrlx adeelmahmood blackbl thutom sylentprayer gargvish saha-shekhar89 cjhcms doulejian zzcclp hadoop835 forrestfinch imace wmz7year pecbali yubobo amirakhmedov dlnufox swathimystery samaitra murador memorygrids mzalcmanis kakamessi99 zhiqinghuang djsecilla teamofmy pkdevbox suyannone shroman hbdrawn ajrocker lijingang irudyak kjniemi a10936283 tomzhang juanavelez cdsalmons dmitriyff mayku testwangs shkimkor harshilambagade dhirenshumsher vladimirershov fysoft2006 reamy sunli1223 nereuschen

ignite's Issues

should be id0 not id, ClusterGroup.forNodeId(UUID id, UUID... ids) in ClusterGroupAdapter.class

ignite/modules/core/src/main/java/org/apache/ignite/internal/cluster/ClusterGroupAdapter.java

Line 461 in 3de1dc4

if (contains(id))

should be
if (contains(id0))

IndexQuery returns wrong results for IN Query combined with other clauses

the order of the indexed fields defines the order how the clauses apply and it looks like that only if all clauses from the start are IN clauses they work as expected.
once a different clause (eg EQ) is evaluated following IN clauses do no longer apply.

For example if combining GTE with IN and a second IN the second IN clause gets ignored:

>>> orgId >= 0, role[a,b], other[x]
>>>     Entry [key=PersonKey{id=1, orgId=1}, val=Person{role='a', other='x', salary=1}]
>>>     Entry [key=PersonKey{id=5, orgId=1}, val=Person{role='a', other='x', salary=2}]
>>>     Entry [key=PersonKey{id=2, orgId=1}, val=Person{role='a', other='y', salary=1}]
>>>     Entry [key=PersonKey{id=6, orgId=1}, val=Person{role='a', other='y', salary=2}]
>>>     Entry [key=PersonKey{id=4, orgId=1}, val=Person{role='b', other='x', salary=1}]
>>>     Entry [key=PersonKey{id=8, orgId=1}, val=Person{role='b', other='x', salary=2}]
>>>     Entry [key=PersonKey{id=3, orgId=1}, val=Person{role='b', other='y', salary=1}]
>>>     Entry [key=PersonKey{id=7, orgId=1}, val=Person{role='b', other='y', salary=2}]
>>>     Entry [key=PersonKey{id=9, orgId=2}, val=Person{role='a', other='x', salary=1}]
>>>     Entry [key=PersonKey{id=13, orgId=2}, val=Person{role='a', other='x', salary=2}]
>>>     Entry [key=PersonKey{id=10, orgId=2}, val=Person{role='a', other='y', salary=1}]
>>>     Entry [key=PersonKey{id=14, orgId=2}, val=Person{role='a', other='y', salary=2}]
>>>     Entry [key=PersonKey{id=12, orgId=2}, val=Person{role='b', other='x', salary=1}]
>>>     Entry [key=PersonKey{id=16, orgId=2}, val=Person{role='b', other='x', salary=2}]
>>>     Entry [key=PersonKey{id=11, orgId=2}, val=Person{role='b', other='y', salary=1}]
>>>     Entry [key=PersonKey{id=15, orgId=2}, val=Person{role='b', other='y', salary=2}]

I used CacheQueryExample to show the issue im facing: https://github.com/DemonicTutor/ignite/blob/a3d4cf14d41bbc869950f4a21aed73f276794d5e/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java

My UseCase / What i tried to achive: Basically a Specification-Pattern search over an Entity (8 properties i want to be indexed/searchable) with a CompositeKey (3 properties) of which i require one to define the SortOrder

MyKey {
  long: id; <<== Affinity and sort order for results therefore also indexed "my_idx"
  String id2;
  String id3;
}

MyValue {
  private String first; <<- indexed  "my_idx"
  private String second; <<-- indexed  "my_idx"
  private boolean third; <<-- indexed "my_idx"
}


public Observable<JsonObject> search(final Criteria criteria, final int limit, final int skip) {
    // Iterator returning search result ordered by MyKey.id to enable "limit" and "skip" consistently    
    final var result = outcomes.query(criteria.query()))
   ....
}

IndexQuery<OutcomeKey, Outcome> query() {
      final var criteria = new ArrayList<IndexQueryCriterion>(8);

      if (!optionalFirstContains.isEmpty()) {
        criteria.add(IndexQueryCriteriaBuilder.in("first", optionalFirstContains));
      }
      if (!optionalSecondContains.isEmpty()) {
        criteria.add(IndexQueryCriteriaBuilder.in("second", optionalSecondContains));
      }
     criteria.add(IndexQueryCriteriaBuilder.gt("id", -1)) // artificially added because ignite's index implementation demands it
      return new IndexQuery<OutcomeKey, Outcome>(Outcome.class).setCriteria(criteria);
    }

Can I use the binary client protocol to connect to the ignite cluster?

Setting TimeZone. setDefault for databases in idea does not work

I added TimeZone to the jdbc connection for ignite core SetDefault () was then packaged as a jar and loaded into databases using a custom driver, causing my query to select now (); The results are not displayed correctly, but the query was correct when I used jdbc to connect normally

Embedding Ignite 3

I am trying to upgrade our codebase from Ignite 2 to 3. However, I cannot find any guides on how to do it.

We use the Java API.

Our Kotlin code looks something like this , currently:

import org.apache.ignite.*

// .. some config setup

           val igniteDiscoverySpi: DiscoverySpi = createDiscoverySpi(clusterConfiguration, configuration, ports)

            val config = ApacheIgniteConfiguration().apply {
                igniteInstanceName = configuration.id()
                localHost = "0.0.0.0"
                consistentId = configuration.serverInformation.id
                clientConnectorConfiguration = clientConnectorConfig
                discoverySpi = igniteDiscoverySpi
                communicationSpi = igniteCommunicationSpi
                isClientMode = false
                shutdownPolicy = ShutdownPolicy.GRACEFUL
                isPeerClassLoadingEnabled = false
                metricsLogFrequency = if (configuration.enableCacheStatistics) 30_000 else 0
                gridLogger = IgniteLoggerFactory.createLogger(configuration)
                workDirectory = distributedCacheWorkingDirectory(configuration.id())
                if (clusterConfiguration != null) {
                    sslContextFactory = TlsContextFactory(clusterConfiguration)
                }
            }

            val ignite = try {
                Ignition.start(config)
            } catch (e: IgniteException) {
                // error handling
            }

I am aware that the discovery mechanism, in particular, has changed a lot... but I cannot even find the Ignite class on version 3 (or most of other classes needed)... could you post code that's similar to the above for Ignite 3, or is there code somewhere I can look?

The documentation I found is currently nearly empty: : https://ignite.apache.org/docs/3.0.0-beta/

We know it's a beta version, but we need to release a feature in a few months that will use Ignite, and we want to see if Ignite 3 will work for our use case (we have had many issues with Ignite 2, specially regarding cluster discovery, and are hoping Ignite 3 will be helpful).

Anything you can provide to help us would be appreciated! Thanks.

SqlFieldQueries with partitions on Calcite

I tried to switch my existing code to Calcite on Ignite 2.14.0.

We do use some SqlFieldQuery, and call .setPartitions(int...) on them to only return results from specific partitions. This works fine with the H2 engine.
However, on calcite engine it seem to return all results for a query whatever we pass to .setPartitions(int...).

Th javadoc for .setPartitions(int...) says: "The query will be executed only on nodes which are primary for specified partitions.", which makes me thinks that the correct behavior is calcite, and that H2 is incorrect.

In any case, it would be better to have the same behavior whatever the engine is. Additionally, is there a way to return results for a specific partition on calcite?

Here is a simple test case reproducing this issue: https://gist.github.com/bgaraude/873c40305ca0874f88f76f1ebe5dcf74

If we want to use Calcite,how should we choose between Ignite 2 and Ignite 3?

We are using Ignite 2.13.
Because of vulnerability scanning,H2 version of 1.4.197 needs to be upgraded.So we want to use Apache Calcite as SQL engine.we are confused about using Ignite 2 or Ignite 3 If we want to use calcite.
If we choose Ignite 2,we don't know the stability of Apache Calcite in Ignite 2；If we choose Ignite 3，we don't know release time of the stable version of Ignite 3.
Thanks~

IGNITE-20024 When H2 is used as the underlying storage, an exception is thrown "Can not set java.time.LocalDateTime field org.apache.ignite.examples.model.Person.createdTime to java.sql.Timestamp"

Question：

When I use IgniteCache.get(key) to get the corresponding entity, ignite throws the following error "Can not set java.time.LocalDateTime field org.apache.ignite.examples.model.Person.createdTime to java.sql .Timestamp"

Conditions and usage：

In ignite 2.14.0, I use H2 as the underlying storage;
In CacheJdbcPojoStoreFactory, I configured the mapping between the datetime type of the database and JDK8's LocalDateTime:
new JdbcTypeField(Types.TIMESTAMP, "created_time", LocalDateTime.class, "createdTime")

Problem analysis:

By default, CacheJdbcPojoStore uses org.apache.ignite.cache.store.jdbc.JdbcTypesTransformer#getColumnValue to parse jdbc's ResultSet.
For the LocalDateTime type of JDK8, java.sql.ResultSet#getObject(int) is used to obtain the value of the specified column (here it is read as java.sql.Timestamp). When the object is finally constructed, the type is inconsistent Causes sun.reflect.FieldAccessor#set assignment to fail.

Repair suggestion:

I want to parse the value of the specified column using java.sql.ResultSet#getObject(int, java.lang.Class<T>), what do you think?

test

Lets see if this works

Publish ignite-nodejs-thin-client at version 1.1.0 on npm

It looks like [email protected] supports Typescript but has never been published on npmjs.org, despite being 2 years old. Could you please do it so we can use Typescript support in Node?

How to clean up invalid BinarySchema

I created a cache like this IgniteCache<String, BinaryObject>. In order to do upsert, I used EntryProcessor as follows.

EntryProcessor

public class UpsertEntryProcessorV1
    implements EntryProcessor<String, BinaryObject, Void> {

    @IgniteInstanceResource
    private Ignite ignite;

    @SuppressWarnings("unchecked")
    @Override
    public Void process(MutableEntry<String, BinaryObject> entry,
        Object... arguments) throws EntryProcessorException {
        if (entry == null || entry.getKey() == null) {
            return null;
        }
        Map<String, Map<String, Object>> params =
            (Map<String, Map<String, Object>>) arguments[0];
        Map<String, Object> param = params.get(entry.getKey());
        if (param == null || param.isEmpty()) {
            return null;
        }

        BinaryObjectBuilder builder;
        BinaryObject entryVal = entry.getValue();
        if (entryVal != null) {
            builder = entryVal.toBuilder();
        } else {
            String valueType = (String) arguments[1];
            builder = ignite.binary().builder(valueType);
        }

        for (Map.Entry<String, Object> item : param.entrySet()) {
            if (item.getKey() == null) {
                continue;
            }
            builder.setField(item.getKey(), item.getValue());
        }

        entry.setValue(builder.build());
        return null;
    }
}

In this cache, I wrote 20,000 records, each record has 1 to 1,000 fields. I found that the ignite service has a lot of full gc, and even crashes directly. Through heap dump analysis, I found that BinarySchema occupies a lot of memory.

heap

Eventually I figured out that the problem was when I was randomly writing to some fields of a record every time.

when I write a record for the first time, a BinarySchema will be created.

The next time I update this record and write one more field, a new BinarySchema will be created and written to ./work/db/ binary_meta/, the old schema will not be cleared.

In the end, tens of thousands of BinarySchemas will be created, but there are only dozens of BinarySchemas that are actually valid in serialized storage.

The following are some places I found to store BinarySchema, is there a way to clean up these BinarySchema?

BinaryContext#descByCls#schemaReg
BinaryContext#schemas
CacheObjectBinaryProcessorImpl#metadataLocCache#metadata#schemas

I understand that ignite3 will unify the client protocol, and I would like to know how different the 3.x version of the protocol is from the 2.x version

Connection configuration for control.sh in Kubernetes

Up until Apache Ignite 2.14 it was possible to use Visor console to connect to a cluster using Spring XML configuration file as documented at https://github.com/apache/ignite/blob/2.14.0/modules/visor-console/src/main/scala/org/apache/ignite/visor/commands/VisorConsole.scala#L91 It was useful for Kubernetes deployments as we use TcpDiscoveryKubernetesIpFinder.

How do I do the same with control.sh script that is supposed to replace Apache Ignite Visor functionality going forward? I would like to be able to inspect caches stats.

Apache Ignite Table support Schemaless ？

Apache Ignite Table usage supports Schemaless ？. For example, columns are automatically created when writing. More common in IoT scenarios

ignite 2 nodes goes down when i start spark job.

i have six node ignite cluster and whenever i start the spark streaming job, 2 ignite goes down with below error.

[2023-02-21 14:09:13,968][WARN ][grid-timeout-worker-#214][PoolProcessor] >>> Possible starvation in striped pool.
Thread name: sys-stripe-13-#14
Queue: []
Deadlock: false
Completed: 104
Thread [name="sys-stripe-13-#14", id=30, state=WAITING, blockCnt=506, waitCnt=101]
Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5246d2a6, ownerName=exchange-worker-#283, ownerId=327]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

[2023-02-21 14:10:02,814][ERROR][tcp-disco-msg-worker-[a37cbeb9 192.168.2.7:47500]-#2-#277][root] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-31, igniteInstanceName=null, finished=false, heartbeatTs=1676968733815]]]

In ignite, I want to how to create a table with field id auto increment

Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT

I run Ignite as a local node within my java application process where I've observed a few Ignite crashes as mentioned below intermittently. In fact, I'm using Ignite as a SQL engine and around 100k records are persisted on an average and a continuous stream of data is getting updated for a given primary key in a record. Ignite is running in a persistent mode with a low memory footprint.

Error Example 1:
ERROR [tcp-disco-msg-worker] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, heartbeatTs=1671441824592]]]

Error Example 2:
ERROR [node-stopper] Stopping local node on Ignite failure: [failureCtx=FailureContext [type=SYSTEM_CRITICAL_OPERATION_TIMEOUT, err=class o.a.i.IgniteException: Checkpoint read lock acquisition has been timed out.]]

What could be the potential issue here? I've tried put in required config such as systemWorkerBlockedTimeout set to ~30sec and enabled page throtting for checkpointing process but that doesn't help.

IgniteCluster: Discovery in the cloud document for Azure Blob storage refers to google compute java code sample

https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud

In the above document the part for azure blob storage, the java code sample refers to google compute instead of a sample for azure example

Assertion error in GridSqlQuerySplitter.normalizeExpression

An auto-generated SQL query is causing Ignite 2.14 with default h2 based SQL parser to crash with an AssertionError. I've distilled the SQL to the simplest it can be to expose the problem:

select count(*) from (
    select "dates_" from (
        select "dates_" from "3a91b5ae9ccDM_FULAir_Passen_"
    ) 
    where "dates_" is not null
)

Result:

Feb 16, 2023 8:07:03 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: GridWorker [name=message-received-notify, igniteInstanceName=null, finished=false, heartbeatTs=1676534823448, hashCode=1652503014, interrupted=false, runner=client-connector-#263]
java.lang.AssertionError: 0
("dates_" IS NOT NULL)
	at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.normalizeExpression(GridSqlQuerySplitter.java:1513)
	at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.normalizeExpression(GridSqlQuerySplitter.java:1501)
	at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.normalizeQuery(GridSqlQuerySplitter.java:1394)
	at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split0(GridSqlQuerySplitter.java:280)
	at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:221)
	at org.apache.ignite.internal.processors.query.h2.QueryParser.parseH2(QueryParser.java:552)
	at org.apache.ignite.internal.processors.query.h2.QueryParser.parse0(QueryParser.java:229)
	at org.apache.ignite.internal.processors.query.h2.QueryParser.parse(QueryParser.java:142)
	at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:984)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor$2.applyx(GridQueryProcessor.java:3101)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor$2.applyx(GridQueryProcessor.java:3074)
	at org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:3751)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$querySqlFields$3(GridQueryProcessor.java:3118)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuerySafe(GridQueryProcessor.java:3190)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:3070)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:3024)
	at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.querySqlFields(JdbcRequestHandler.java:773)
	at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.executeQuery(JdbcRequestHandler.java:641)
	at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.doHandle(JdbcRequestHandler.java:311)
	at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.handle(JdbcRequestHandler.java:251)
	at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:204)
	at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:55)
	at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
	at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
	at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
	at org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Problematic code:

// Unique aliases must be generated for all the table filters already.
assert uniqueAlias != null : childIdx + "\n" + parent.getSQL();

Now clearly this SQL could be written as

select count(*) from (
    select "dates_" from (
        select "dates_" from "3a91b5ae9ccDM_FULAir_Passen_"
      where "dates_" is not null
    ) 
)

which solves the problem and stops the crash. But this is a simplification of auto-generated SQL; there are cases that need the triple nested querying to work this way.

I wonder whether the assertion error is identifying a bug, or is simply a poor way of communicating missing support for a particular SQL query structure.

I have not tried it with Calcite.

How to deserialize a class with final field, like Record class

I have a class

public record Coordinate(
        Integer id,
        Integer x,
        Integer y,
        Integer z
) {}

It will report an exception when using when use cache.get(key) , the cache type is org.apache.ignite.IgniteCache<String, Object> cache
exception is:

Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=com.me.boot.dataignite.boot.dto.Coordinate]
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=id]
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to set value for field: private final java.lang.Integer com.me.boot.dataignite.boot.dto.Coordinate.id
Caused by: java.lang.IllegalAccessException: Can not set final java.lang.Integer field com.me.boot.dataignite.boot.dto.Coordinate.id to java.lang.Integer

then when I debug, find org.apache.ignite.internal.binary.BinaryClassDescriptor#read, in switch (mode) case OBJECT::
The process of deserialization is to use res = newInstance() create res first, and then in info.read(res, reader) use field.set(obj, val) set field value, but when Field is final, an exception will be throws

For the Record class, I currently add a piece of code to solve the deserialization problem:

                case OBJECT:
                    if (Record.class.isAssignableFrom(cls)) {
                        // the record class default has All-args constructor
                        // read all field value to Object[] values
                        Object[] values = new Object[fields.length];
                        for (int i = 0; i < fields.length; i++) {
                            BinaryFieldAccessor fieldAccessor = fields[i];
                            Object val = reader.readField(fieldAccessor.id);
                            values[i] = val;
                        }
                        // invoke All-args constructor to create res
                        Constructor<?> constructor = cls.getDeclaredConstructors()[0];
                        res = constructor.newInstance(values);
                    } else {
                        res = newInstance();
                        reader.setHandle(res);
                        for (BinaryFieldAccessor info : fields)
                            info.read(res, reader);
                    }
                    break;

But besides this way, is there any other way?

like only @Override org.apache.ignite.internal.binary.BinaryClassDescriptor#read and use it, no need to modify dependent source code
like set BinaryConfiguration, but setSerializer(BinarySerializer serializer) doesn't seem to work

Ignite 2.14.0, .NET, failed to create page directory, JDK 17

Using 2.14.0, getting the given exception when calling Ignition.Start against OpenJDK 17:

JavaException: class org.apache.ignite.IgniteException: Failed to start processor: GridProcessorAdapter []
	at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1145)
	at org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:48)
	at org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:74)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter []
	at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1787)
	at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1757)
	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1679)
	at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1121)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:631)
	at org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:43)
	... 1 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to create page store work directory: C:\ignitework\db\0_0_0_0_0_0_0_1_127_0_0_1_172_20_240_1_172_20_48_1_172_23_224_1_172_23_48_1_192_168_175_81_2603_8080_7400_45d9_2c57_895a_1fc0_9b07_2603_8080_7400_45d9_350c_88b1_7dc7_29f8_2603_8080_7400_45d9_6ee0_c850_1080_e364_2603_8080_7400_45d9_7ddf_2fff_94ce_4d0f_2603_8080_7400_45d9_7dee_c31b_fa5c_4897_2603_8080_7400_45d9_9d14_a51d_7729_c0c9_2603_8080_7400_45d9_d96_17d8_4e99_1e77_2603_8080_7400_45d9_f88f_c2b5_7a72_f08b_44120
	at org.apache.ignite.internal.util.IgniteUtils.ensureDirectory(IgniteUtils.java:9971)
	at org.apache.ignite.internal.processors.cache.GridLocalConfigManager.<init>(GridLocalConfigManager.java:127)
	at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:618)
	at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1784)
	... 7 more

I had added custom JvmOptions consisting of all of the various options I found in jvmdefaults.sh. This advanced it a bit.

This directory name is kind of insane. What's it trying to do? And why?

CorruptedTreeException: B+Tree is corrupted

Java version 1.8
Ignite version 2.14

The error appears at the stage of writing data to the table using the following construction

insert into saa_sv_rawdbo_dm_transactions_month
select ... from ...

Use without template=onheapcache work currently

CREATE TABLE saa_sv_rawdbo_dm_transactions_month(
 pay_client_pin VARCHAR
 , pay_transaction_count int
 , pay_tarif_plan_set VARCHAR
 ...
 , PRIMARY KEY (pay_client_pin)
) WITH "affinityKey=pay_client_pin,DATA_REGION=persist_data_region,template=onheapcache";

CREATE INDEX idx_fin_client ON public.saa_sv_rawdbo_dm_transactions_month(pay_client_pin) INLINE_SIZE 6;

<property name="dataRegionConfigurations">
 <list>
  <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
   <property name="name" value="persist_data_region"/>
   <property name="initialSize" value="#{1L * 1024 * 1024 * 1024}"/>
   <property name="maxSize" value="#{100L * 1024 * 1024 * 1024}"/>
   <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
   <!-- persist options -->
   <property name="persistenceEnabled" value="true"/>
   <property name="warmUpConfiguration">
    <bean class="org.apache.ignite.configuration.LoadAllWarmUpConfiguration"/>
   </property>
  </bean>
 </list>
</property>

<property name="cacheConfiguration">
<list>
 <bean abstract="true" class="org.apache.ignite.configuration.CacheConfiguration" id="cache-template-bean">
  <property name="name" value="onheapcache*"/>
  <property name="sqlOnheapCacheEnabled" value="true"/>
 </bean>
</list>
</property>

log:

[12:38:31,424][SEVERE][build-idx-runner-#228][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=-288979615, pageIds=[844420635189922], cacheId=-288979615, cacheName=SQL_PUBLIC_SAA_SV_RAWDBO_DM_TRANSACTIONS_MONTH, indexName=IDX_FIN_CLIENT, msg=Runtime failure on row: Row@5c1a42b0[ key: A201IX, val: SQL_PUBLIC_SAA_SV_RAWDBO_DM_TRANSACTIONS_MONTH_177aaa96_e8b5_4ee1_baff_10353741c660 [idHash=1390257987, hash=-511005807, PAY_TRANSACTION_COUNT=2, ] ][ A201IX, A201IX ]]]]
class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=-288979615, pageIds=[844420635189922], cacheId=-288979615, cacheName=SQL_PUBLIC_SAA_SV_RAWDBO_DM_TRANSACTIONS_MONTH, indexName=IDX_FIN_CLIENT, msg=Runtime failure on row: Row@5c1a42b0[ key: A201IX, val: SQL_PUBLIC_SAA_SV_RAWDBO_DM_TRANSACTIONS_MONTH_177aaa96_e8b5_4ee1_baff_10353741c660 [idHash=1390257987, hash=-511005807, PAY_TRANSACTION_COUNT=2, PAY_TARIF_PLAN_SET=11] ][ A201IX, A201IX ]]

        at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.corruptedTreeException(InlineIndexTree.java:561)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2671)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.put(BPlusTree.java:2602)
        at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.putx(InlineIndexImpl.java:348)
        at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.onUpdate(InlineIndexImpl.java:325)
        at org.apache.ignite.internal.cache.query.index.IndexProcessor.lambda$createIndexDynamically$0(IndexProcessor.java:225)
        at org.apache.ignite.internal.processors.query.schema.SchemaIndexCachePartitionWorker$SchemaIndexCacheVisitorClosureWrapper.apply(SchemaIndexCachePartitionWorker.java:302)
        at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateIndex(GridCacheMapEntry.java:4193)
        at org.apache.ignite.internal.processors.query.schema.SchemaIndexCachePartitionWorker.processKey(SchemaIndexCachePartitionWorker.java:236)
        at org.apache.ignite.internal.processors.query.schema.SchemaIndexCachePartitionWorker.processPartition(SchemaIndexCachePartitionWorker.java:191)
        at org.apache.ignite.internal.processors.query.schema.SchemaIndexCachePartitionWorker.body(SchemaIndexCachePartitionWorker.java:130)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: class org.apache.ignite.IgniteException: Failed to store new index row.
        at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineInnerIO.storeByOffset(AbstractInlineInnerIO.java:106)
        at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineInnerIO.storeByOffset(AbstractInlineInnerIO.java:37)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.BPlusIO.store(BPlusIO.java:228)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.BPlusIO.insert(BPlusIO.java:317)
        at org.apache.ignite.internal.processors.cache.persistence.tree.io.BPlusInnerIO.insert(BPlusInnerIO.java:150)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insertSimple(BPlusTree.java:3962)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insert(BPlusTree.java:3944)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$1700(BPlusTree.java:3786)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:505)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:486)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:6216)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:6202)
        at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:381)
        at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:348)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11500(BPlusTree.java:162)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:4100)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.finishOrLockTail(BPlusTree.java:3824)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:3035)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:3022)
        at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2631)
        ... 13 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:92)
        at org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
        at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineInnerIO.storeByOffset(AbstractInlineInnerIO.java:96)
        ... 32 more
[12:38:31,426][SEVERE][build-idx-runner-#228][FailureProcessor] A critical problem with persistence data structures was detected. Please make backup of persistence storage and WAL files for further analysis. Persistence storage path: null WAL path: db/wal WAL archive path: db/wal/archive
[12:38:31,426][SEVERE][build-idx-runner-#228][FailureProcessor] No deadlocked threads detected.
[12:38:31,841][SEVERE][build-idx-runner-#228][FailureProcessor] Thread dump at 2023/01/23 12:38:31 MSK
Thread [name="stat-mgmt-#234", id=307, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e9d4a5b, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

Thread [name="checkpoint-runner-IO-#233", id=306, state=WAITING, blockCnt=1, waitCnt=9912]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22493384, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

Thread [name="checkpoint-runner-IO-#232", id=305, state=WAITING, blockCnt=3, waitCnt=9936]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22493384, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

...


[12:38:31,998][SEVERE][build-idx-runner-#228][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [groupId=-288979615, pageIds=[844420635189922], cacheId=-288979615, cacheName=SQL_PUBLIC_SAA_SV_RAWDBO_DM_TRANSACTIONS_MONTH, indexName=IDX_FIN_CLIENT, msg=Runtime failure on row: Row@5c1a42b0[ key: A201IX, val: SQL_PUBLIC_SAA_SV_RAWDBO_DM_TRANSACTIONS_MONTH_177aaa96_e8b5_4ee1_baff_10353741c660 [idHash=1390257987, hash=-511005807, PAY_TRANSACTION_COUNT=2, PAY_TARIF_PLAN_SET=11] ][ A201IX, A201IX ]]]]

ignite-cloud seems to be missing from the 2.14.0-release onward

At least it is missing from maven central. Also, there seems to be no release note mentioning where else the functionality went.

Ignite default consistentId generation is broken on IPv6 nodes using RFC8981 security mode.

In correctly configured networks, hosts may implement RFC8981 which creates a rotating list of temporary IPv6 addresses used for outbound IPv6 traffic. This prevents leaking internal host addresses to untrusted external services as IPv6 does not rely on NAT. In the referenced code, Ignite will create a default consistentId which is too long for use as a pathname for local persistence, for example.

ignite/modules/core/src/main/java/org/apache/ignite/spi/discovery/tcp/internal/TcpDiscoveryNode.java

Line 191 in 6c38868

 this.consistentId = consistentId != null ? consistentId : U.consistentId(sortedAddrs, discPort); 

ignite/modules/core/src/main/java/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.java

Line 1190 in 6c38868

consistentId = U.consistentId(sortedAddrs);

ignite/modules/zookeeper/src/main/java/org/apache/ignite/spi/discovery/zk/ZookeeperDiscoverySpi.java

Line 279 in da8a6bb

consistentId = U.consistentId(sortedAddrs);

The following error log (IPv6 addresses obfuscated to protect my network) shows this behavior. This may not be the only symptom.

SYNC modes due to message queues growth on sender and receiver sides.
Apr 13, 2023 5:53:10 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Exception during start processors, node will be stopped and close connections
class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter []
	at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1787)
	at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1757)
	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1679)
	at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1121)
	at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1015)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:840)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:710)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:679)
	at org.apache.ignite.Ignition.start(Ignition.java:353)
	at training.ServerStartup.main(ServerStartup.java:23)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to create page store work directory: /Volumes/workspaces/jeremy.mcmillan/ignite-essentials-developer-training.git/ignite/work/db/0_0_0_0_0_0_0_1_lo0_10_0_1_96_127_0_0_1_2100_f00f_f00f_f00f_f00f_f00f_f00f_f00f_en0_2100_f00f_f00f_f00f_f00f_f00f_f00f_f00f_en0_2100_f00f_f00f_f00f_f00f_f00f_f00f_f00f_en0_2100_f00f_f00f_f00f_f00f_f00f_f00f_f00f_en0_2100_f00f_f00f_f00f_f00f_f00f_f00f_f00f_en0_47500
	at org.apache.ignite.internal.util.IgniteUtils.ensureDirectory(IgniteUtils.java:9971)
	at org.apache.ignite.internal.processors.cache.GridLocalConfigManager.<init>(GridLocalConfigManager.java:127)
	at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:618)
	at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1784)
	... 11 more

Apache Ignite 2.15.0 startup failure using OpenJDK 17 version

Hi ,

Iam currently using Apache Ignite of version 2.15.0 . But Iam unable to Run Ignite using JDK version of 17 .

What version of JDK to be installed to run Apche Ignite 2.15.0 .....?

Could you please help me on this...?

JDBC driver doesn't work on JDK 21-ea

java.lang.ExceptionInInitializerError
	at org.apache.ignite.internal.util.IgniteUtils.<clinit>(IgniteUtils.java:893)
	at org.apache.ignite.lang.IgniteProductVersion.fromString(IgniteProductVersion.java:312)
	at org.apache.ignite.internal.IgniteVersionUtils.<clinit>(IgniteVersionUtils.java:94)
	at org.apache.ignite.IgniteJdbcThinDriver.<clinit>(IgniteJdbcThinDriver.java:140)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:392)
	at java.base/java.lang.Class.forName(Class.java:383)
	at org.testcontainers.containers.JdbcDatabaseContainer.getJdbcDriverInstance(JdbcDatabaseContainer.java:199)
	at org.testcontainers.containers.JdbcDatabaseContainer.createConnection(JdbcDatabaseContainer.java:237)
	at org.testcontainers.containers.JdbcDatabaseContainer.createConnection(JdbcDatabaseContainer.java:218)
	at org.testcontainers.containers.JdbcDatabaseContainer.waitUntilContainerStarted(JdbcDatabaseContainer.java:158)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:490)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at io.trino.plugin.ignite.TestingIgniteServer.<init>(TestingIgniteServer.java:46)
	at io.trino.testing.SharedResource.getInstanceLease(SharedResource.java:46)
	at io.trino.plugin.ignite.TestingIgniteServer.getInstance(TestingIgniteServer.java:39)
	at io.trino.plugin.ignite.TestIgniteConnectorTest.createQueryRunner(TestIgniteConnectorTest.java:52)
	at io.trino.testing.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:104)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at org.testng.TestRunner.privateRun(TestRunner.java:756)
	at org.testng.TestRunner.run(TestRunner.java:610)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
	at org.testng.TestNG.runSuites(TestNG.java:1133)
	at org.testng.TestNG.run(TestNG.java:1104)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
Caused by: java.lang.RuntimeException: Unable to set up byte buffer creation using reflections :java.nio.DirectByteBuffer.<init>(long,int)
	at org.apache.ignite.internal.util.GridUnsafe.createNewDirectBufferCtor(GridUnsafe.java:1706)
	at org.apache.ignite.internal.util.GridUnsafe.createAndTestNewDirectBufferCtor(GridUnsafe.java:1671)
	at org.apache.ignite.internal.util.GridUnsafe.<clinit>(GridUnsafe.java:171)
	... 43 more
	Suppressed: java.lang.RuntimeException: jdk.internal.misc.JavaNioAccess class is unavailable.
Please add the following parameters to JVM startup settings and restart the application: {parameters: --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
--add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
--add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--illegal-access=permit
}
See https://apacheignite.readme.io/docs/getting-started#section-running-ignite-with-java-9-10-11 for more information.
		at org.apache.ignite.internal.util.GridUnsafe.javaNioAccessObject(GridUnsafe.java:1627)
		at org.apache.ignite.internal.util.GridUnsafe.<clinit>(GridUnsafe.java:175)
		... 43 more
	Caused by: java.lang.ClassNotFoundException: jdk.internal.misc.SharedSecrets
		at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
		at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
		at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
		at java.base/java.lang.Class.forName0(Native Method)
		at java.base/java.lang.Class.forName(Class.java:392)
		at java.base/java.lang.Class.forName(Class.java:383)
		at org.apache.ignite.internal.util.GridUnsafe.javaNioAccessObject(GridUnsafe.java:1620)
		... 44 more
Caused by: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
	at java.base/java.lang.Class.getConstructor0(Class.java:3689)
	at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2858)
	at org.apache.ignite.internal.util.GridUnsafe.createNewDirectBufferCtor(GridUnsafe.java:1699)
	... 45 more

JVM flags used:

--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
--add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
--add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--illegal-access=permit
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
--add-opens=java.base/jdk.unsupported=ALL-UNNAMED

ignite-core: 2.15.0

For reference: https://github.com/trinodb/trino/actions/runs/4996810421/jobs/8951203565?pr=17520

When will ignite have the go sdk

Apache Ignite 2.15 startup failure with JDK 17 on windows server

Hi ,
In our project , ignite is started using "Topshelf" command to run as windows service by passing JAVA_HOME as one of the Argument.

when i tried to run Apche Ignite 2.15.0 with OpenJDK 17, Its not working.
Can you please provide information on this.....?

Note : I am running Ignite on Windows Server

Getting the following error when i tried to start Ignite 2.15 with jdk 17
2023-06-26 14:56:49.514 [ERR] Exception while starting ignite service.
Apache.Ignite.Core.Common.IgniteException: Java exception occurred [class=java.lang.ExceptionInInitializerError, message=] ---> Apache.Ignite.Core.Common.JavaException: java.lang.ExceptionInInitializerError
at org.apache.ignite.internal.processors.platform.memory.PlatformAbstractMemory.(PlatformAbstractMemory.java:27)
at org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.init(PlatformAbstractBootstrap.java:54)
at org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:65)
Caused by: java.lang.RuntimeException: java.nio.DirectByteBuffer.address field is unavailable.
at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1570)
at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1557)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at org.apache.ignite.internal.util.GridUnsafe.bufferAddressOffset(GridUnsafe.java:1557)
at org.apache.ignite.internal.util.GridUnsafe.(GridUnsafe.java:109)
... 3 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @4b85612c
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1562)
... 7 more

at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.ExceptionCheck()
at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedUtils.IgnitionStart(Env env, String cfgPath, String gridName, Boolean clientMode, Boolean userLogger, Int64 igniteId, Boolean redirectConsole)
at Apache.Ignite.Core.Ignition.Start(IgniteConfiguration cfg)
--- End of inner exception stack trace ---
at Apache.Ignite.Core.Ignition.Start(IgniteConfiguration cfg)

Using Continuous Query with thin client causing OutOfMemoryError

Using continuous query with thin clients(Java ignite version 2.14.0), resulted in OutOfMemoryError when one of the thin clients becomes slow.

Tried to set SlowClientQueueLimit to 100 and MessageQueueLimit to 200, but seems it is not taking effect. The queue in GridSelectorNioSessionImpl continues to grow until run out of memory

OpenJDK 17 Ignite.Start, custom JvmDllPath

Not sure what's up with this. I'm trying to launch Ignite from within a .NET program, running Apache.Ignite 2.14.0, against OpenJDK 17.

Upon Ignition.Start, I'm receiving:

JavaException: java.lang.ExceptionInInitializerError
	at org.apache.ignite.internal.processors.platform.memory.PlatformAbstractMemory.<clinit>(PlatformAbstractMemory.java:27)
	at org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.init(PlatformAbstractBootstrap.java:54)
	at org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:65)
Caused by: java.lang.RuntimeException: java.nio.DirectByteBuffer.address field is unavailable.
	at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1570)
	at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1557)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.ignite.internal.util.GridUnsafe.bufferAddressOffset(GridUnsafe.java:1557)
	at org.apache.ignite.internal.util.GridUnsafe.<clinit>(GridUnsafe.java:109)
	... 3 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @30946e09
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Unknown Source)
	at java.base/java.lang.reflect.Field.setAccessible(Unknown Source)
	at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1562)
	... 7 more

My guess is it's probably some of these missing VM options that are supposed to be passed, and mentioned in other sources. However, I'd expect the .NET library to set all that up automatically. Maybe it's not. Will let ya'll know.

Inconsistent reads using Index API and OPTIMISTIC SERIALIZABLE transactions

Using ignite 2.14.0 you can get an inconsistent read when using OPTIMISTIC SERIALIZABLE transactions and the index API. Please see this example code. We do PUTs to a cache on one thread and at the same time we query the cache using the index API on another thread. We put 100 entries in total and the query will see a partial set of items, e.g. it can see 13 items, then 32, then 73 etc. The read is inconsistent. I assumed this inconsistent read would fail when we commit the transaction surrounding the query but it doesn't fail, the OPTIMISTIC SERIALIZABLE transactions aren't protecting from the inconsistent read.

I suspect the issue is that Ignite isn't acquiring a range lock when it does the index query, quoting from Wikipedia https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable :

Also range-locks must be acquired when a SELECT query uses a ranged WHERE clause, especially to avoid the phantom reads phenomenon

Here's the code:


import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.mutable.MutableBoolean;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteCluster;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.query.IndexQuery;
import org.apache.ignite.cache.query.IndexQueryCriteriaBuilder;
import org.apache.ignite.cache.query.QueryCursor;
import org.apache.ignite.cache.query.annotations.QuerySqlField;
import org.apache.ignite.cluster.ClusterState;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.transactions.Transaction;
import org.apache.ignite.transactions.TransactionConcurrency;
import org.apache.ignite.transactions.TransactionIsolation;

import javax.cache.Cache.Entry;
import java.io.File;
import java.util.HashSet;

public class DebugReadConsistencyIssue1 {
    private static final int NB_RECS = 100;

    /**
     * Small class which stores a foreign key with an index that we query against
     */
    public static class MyValue {

        @QuerySqlField(name = "fkey", orderedGroups = {
                @QuerySqlField.Group(name = "fkeyindex", order = 0)
        })
        private String foreignKey;

        public String getForeignKey() {
            return foreignKey;
        }

        public void setForeignKey(String foreignKey) {
            this.foreignKey = foreignKey;
        }
    }

    public static void main(String[] args) throws Exception {
        // initialise ignite with data persistence and longer transaction
        // timeouts
        File dir = new File("c:\\temp\\ignite-debug");
        File work = new File(dir, "work");
        FileUtils.deleteDirectory(dir);
        work.mkdirs();
        IgniteConfiguration cfg = new IgniteConfiguration();
        cfg.setIgniteHome(dir.getAbsolutePath());
        cfg.setWorkDirectory(work.getAbsolutePath());
        cfg.setDataStorageConfiguration(new DataStorageConfiguration());
        cfg.getDataStorageConfiguration().getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
        cfg.getTransactionConfiguration().setDefaultTxTimeout(60000);
        cfg.getTransactionConfiguration().setDeadlockTimeout(60000);

        try (Ignite ignite = Ignition.start(cfg)) {

            // ensure cluster is active
            IgniteCluster cluster = ignite.cluster();
            cluster.state(ClusterState.ACTIVE);

            // get or create the cache, setting it transactional, then clear it if it already existed
            CacheConfiguration<String, MyValue> config = new CacheConfiguration<>("MyValue");
            config.setIndexedTypes(String.class, MyValue.class);
            config.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
            config.setCacheMode(CacheMode.PARTITIONED);
            IgniteCache<String, MyValue> cache = ignite.getOrCreateCache(config);
            cache.clear();

            // put some data values on one thread
            MutableBoolean putFailed = new MutableBoolean(false);
            Thread putThread = new Thread(() -> {
                if (!runPut(ignite, cache)) {
                    putFailed.setValue(true);
                }
            });
            putThread.start();

            // at the same time use the index API to get all entries within the foreign key
            Thread queryThread = new Thread(() -> {
                while (runGetQuery(ignite, cache) != NB_RECS && !putFailed.booleanValue()) {
                }
            });
            queryThread.start();

            // wait until both threads completed
            putThread.join();
            queryThread.join();
        } finally {

        }
    }

    private static boolean runPut(Ignite ignite, IgniteCache<String, MyValue> cache) {
        // Put multiple objects, doing it slowly with sleeps in-between PUTs
        // so we can see that the query on the other thread sees a partial
        // number of objects written
        try (Transaction tx = ignite.transactions().txStart(TransactionConcurrency.OPTIMISTIC,
                TransactionIsolation.SERIALIZABLE)) {
            for (int j = 0; j < 100; j++) {
                MyValue value = new MyValue();
                value.foreignKey = "fk1";
                cache.put("key" + j, value);

                try {
                    Thread.sleep(20);
                } catch (Exception e) {
                }
                System.out.print(".");
            }
            tx.commit();
            System.out.println("Committed PUT transaction");
        } catch (Exception e) {
            System.err.println("Put failed " + e.toString());
            return false;
        }
        return true;
    }

    private static int runGetQuery(Ignite ignite, IgniteCache<String, MyValue> cache) {
        // Run query against the foreign key using the index API and report if we
        // see an odd number of objects after the transaction has finished
        HashSet<String> queryResult = new HashSet<>();
        try (Transaction tx = ignite.transactions().txStart(TransactionConcurrency.OPTIMISTIC,
                TransactionIsolation.SERIALIZABLE)) {

            IndexQuery<String, MyValue> query =
                    new IndexQuery<String, MyValue>(MyValue.class, "fkeyindex").setCriteria(IndexQueryCriteriaBuilder.eq("fkey", "fk1"));
            queryResult.clear();
            try (QueryCursor<Entry<String, MyValue>> cursor = cache.query(query)) {
                cursor.getAll().forEach(entry -> {
                    queryResult.add(entry.getKey());
                });
            }

            tx.commit();
        } catch (Exception e) {
            System.err.println();
            System.err.println("Query transaction failed: " + e.toString());
            return 0;
        }

        // Transaction finished without failing
        if (queryResult.size() != 0 && queryResult.size() != NB_RECS) {
            System.err.println();
            System.err.println("Found " + queryResult.size() + " entries after committed query transaction. We saw a " +
                    "partially written dataset!!!");
        }
        return queryResult.size();
    }
}

Typical output:

_....................................................................................................
Found 1 entries after committed query transaction. We saw a partially written dataset

Found 13 entries after committed query transaction. We saw a partially written dataset

Found 14 entries after committed query transaction. We saw a partially written dataset

Found 14 entries after committed query transaction. We saw a partially written dataset_

Previous Version Request : Need ignite 0.2.1 version

Previous Version Request : Please provideignite 0.2.1 version (or let me know the branch to clone) for an old project build. Greatly appreciated!

Cluster data went back two months after node rejoin.

Hello,

I have a three-node Apache Ignite cluster using version 2.14.0 on both persistent and in-memory mode, we had to bring down a node for maintenance two months ago and unfortunately, the issues on that server took about 2 months to be resolved.

Yesterday we were able to bring up the node and we did that by just turning on the server and starting the Apache Ignite systemd service. Issuing the control.sh --baseline command on the first node indicated that the second node (which was down for two months) is now online, problem is the data on the cluster appears to be reverted back to two months ago as well.

Since the topology version had changed I figure the rebalancing procedure had taken effect and data on the other two nodes would overwrite the data on the second node but after `./control.sh --cache idle_verify' I got something like the below error:

Consistency checked failed, found conflicts [ CounterConflicts=0, hashConflicts=512]

I checked the output and it seemed the partition hash of the second node was different than the others so before doing anything else I decided to remove the second node from the cluster by ./control.sh --baseline remove Ignite02 which worked and ./control.sh --cache idle_verify didn't show any conflicts anymore, however, the data is still two months old.

I highly doubt that the cluster had overwritten the old data on its own, there is probably something that I've done wrong.

Q: Is there any way to restore the correct data?
Q2: If there is something wrong with what I've done, please tell to avoid future mistakes.

Any help is appreciated.

JDBC Driver add support for OpenJDK 17

I have tested the JDBC driver under OpenJDK17, and does not work. I tested ignite-core-2.14.0.jar.
Same driver works fine with OpenJDK15.

Is there a function to obtain hotkeys implemented

I want to obtain get hotkeys, do you have this feature?

Question on loading data onto ignite

I tried searching for where to ask questions but didn't found anything useful.Hence the issue.

What are various ways though which I can load data from Postgres table apart from using spark?

I've tried reading data directly through Psycopg2 to Pyignite(thin client) but it seems slow and have parsing errors.
I read data streamers are best for high speed transfer but is it possible to have streamer from Postgres(RDBMS) to ignite just for loading whole table using python? Its around 100k rows.

Ignite semaphore release vs. close

Hi,
I have a question about race condition on closing semaphores. As I understand semaphores are not removed automatically and must therfore be closed manually. Lets say two threads want to change some object and try to aquire semaphore for it:

IgniteSemaphore semaphore = ignite.semaphore(objectId, 1, true, true);
if (semaphore.tryAcquire(5, TimeUnit.SECONDS)) {
    try {
        changeObject(objectId);
    } finally {
        semaphore.release(); // First thread releases semaphore
        // Second thread aquires semaphore
        semaphore.close(); // First thread closes semaphore
    }
}

What happens if first thread tries to close semaphore while second thread has aquired it?
But my main quiestion is - can I close the semaphote without calling release before it? So second thread cannot aquire it.

Ignite cluster (both client and server) breaking intermittently with 401 http code in AKS

I am able to deploy ignite cluster in AKS successfully

Note: 1. service account, role and role binding are created already and have also mentioned service account in deployment.yaml
2. service name and namespace are also mentioned in ignite configuration
3. 3. We have configured thin clients in cluster

Client and server are able to join and communicate initially but after below intermittent exception, both are unable to reconnect to cluster and unfortunately, we have to redeploy entire helm chart,

org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to clean IP finder up
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.
at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:81)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:2057)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:2279)
Caused by: class org.apache.ignite.IgniteException: Failed to retrieve Ignite pods IP addresses.
at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:123)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/ignite/endpoints/cache-service
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2000)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:111)

A sudden increase of epollWait() CPU utilization after a particular commit.

Greetings! I am a master's student conducting research on performance diagnosis. My current focus is on studying the evolution of distributed systems and the occurrence of performance regression during their development. As part of my research, I am examining Apache Ignite as a case study.

To conduct my study, I used the YCSB benchmark (https://github.com/brianfrankcooper/YCSB/tree/master/ignite), configuring it to perform 1,000,000 update operations and initialize with 100,000 records. I conducted the testing with a single thread and 3 nodes on a single machine using 3 different ports. Upon analyzing the results, I observed that the average update latency increased by approximately 10% from version 2.7.6 to version 2.14.0 (compiled by jdk8 locally). I am currently attempting to understand the cause behind this increase. For your reference, I have included the detailed configuration of one of the nodes below.

<?xml version="1.0" encoding="UTF-8"?>
  
    <!--
      Copyright (c) 2018 YCSB contributors. All rights reserved.

      Licensed to the Apache Software Foundation (ASF) under one or more
      contributor license agreements.  See the NOTICE file distributed with
      this work for additional information regarding copyright ownership.
      The ASF licenses this file to You under the Apache License, Version 2.0
      (the "License"); you may not use this file except in compliance with
      the License.  You may obtain a copy of the License at
           http://www.apache.org/licenses/LICENSE-2.0
      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.
    -->

    <!--
        Ignite Spring configuration file to startup Ignite cache.
        This file demonstrates how to configure cache using Spring. Provided cache
        will be created on node startup.
        Use this configuration file when running HTTP REST examples (see 'examples/rest' folder).
        When starting a standalone node, you need to execute the following command:
        {IGNITE_HOME}/bin/ignite.{bat|sh} examples/config/example-cache.xml
        When starting Ignite from Java IDE, pass path to this file to Ignition:
        Ignition.start("examples/config/example-cache.xml");
    -->

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="walMode" value="LOG_ONLY"/>
                <property name="storagePath" value="/data/dbignite2.10.0-SNAPSHOT"/>
                <property name="walPath" value="/data/walignite2.10.0-SNAPSHOT"/>
                <property name="walArchivePath" value="/data/walarchignite2.10.0-SNAPSHOT"/>
                <property name="walHistorySize" value="1"/>
                <property name="metricsEnabled" value="true"/>

                <property name="defaultDataRegionConfiguration">
                    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="name" value="default_data_region"/>
                        <property name="persistenceEnabled" value="false"/>
                        <!-- Setting the max size of the default region to 10GB. -->
                        <property name="maxSize" value="#{10L * 1024 * 1024 * 1024}"/>
                        <!-- Setting the initial size of the default region to 10GB. -->
                        <property name="initialSize" value="#{10L * 1024 * 1024 * 1024}"/>
                        <property name="checkpointPageBufferSize" value="#{1L * 1024 * 1024 * 1024}"/>
                        <property name="metricsEnabled" value="true"/>
                    </bean>
                </property>
            </bean>
        </property>


        <property name="cacheConfiguration">
            <list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="usertable"/>
                    <property name="atomicityMode" value="ATOMIC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="backups" value="1"/>
                    <property name="writeSynchronizationMode" value="FULL_SYNC"/>
                </bean>
            </list>
        </property>


    <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
    <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="localPort" value="47500" />
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--The list of hosts includes client host. -->
                            <!--<value><hostname_or_IP>:47500..47509</value>-->
                            <!--<value><hostname_or_IP>:47500..47509</value>-->
                            <value>10.1.0.16:47500</value>
                            <value>10.1.0.16:47501</value>
                            <value>10.1.0.16:47502</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>
</bean>
</beans>

Next, I utilized JFR to profile the runtime and discovered a significant increase in the profiling samples of sun.nio.ch.EPollArrayWrapper.epollWait(). Specifically, the overall method profiling samples increased by approximately 2000 (equivalent to around 10% of v2.7.6's overall method profiling samples, which matches the increase ratio of latency), while the increase in samples of epollWait() was around 5500, and java.net.PlainSocketImpl.socketAccept() decreased by approximately 3900 samples. To gather this data, I conducted ten rounds of testing using the benchmark mentioned above on an Ubuntu 18.04.1 LTS system with x86_64 arch, 16 cores, and 132GB memory.

Based on my findings, I concluded that epollWait() is the primary cause of the latency increase. I have since attempted to narrow down the issue and locate the particular commit(s) responsible. I discovered that commit 1094fff had an increase in samples of approximately 3400 compared to its parent. This increase accounts for around 61.82% of the total increase in epollWait() samples from v2.7.6 to v2.14.0. However, the latency did not significantly increase from its parent, as socketAccept() also decreased by approximately 3400 samples.

At this point, I would like to understand why this particular commit caused such a significant increase in epollWait() samples. The commit appears to only disable JMX monitoring, but since I lack context, I am seeking suggestions on where to focus my further study. Can you please provide any advice or recommendations?

disable or enable partition awareness from XML file for client node or thin client node.

We are using java thin client in our spark streaming job and want to disable partition awareness in clinetConfigurtion class.

There is no option to pass that variable from XML file.

Please help

Apache ignite doc is not good!

Who wrote it ? have you tested it ? Does your sample code work ? Can you actually write documentation ?

StopNodeOrHaltFailureHandler

StopNodeOrHaltFailureHandler.java

when we use StopNodeOrHaltFailureHandler it takes 10 to 15 minutes for JVM to kick in and kill the process due to critical error or system worker blocked.

How can we make sure jvm should kill it asap.

.NET library for integrating with NHibernate

Just wanted to check if there is any initiative to add support for integrating with NHibernate

If I want to create a table, do I have to set a primary key, I don't want a primary key

Issue with Running client behind NAT - Apache Ignite

Hi Team,

We have an Ignite cluster(2.14) running in a private network, and we are trying to load data into the cluster from outside.

Using Thick client:

We are trying run spark data load job from OracleDataCloud Dataflow(k8s architecture)
But client is not able to join the cluster even with

Using Thin client:

When we try to load data, we see memory spikes in one/few nodes only, with partition awareness enabled
Data is not getting distributed. PFA

Could you please suggest a solution.

Nested queries are throwing exception

I am using the JDBC thin driver for accessing my Ignite DB but I am having issues with nested queries:

This nested query works on a given OrderDetails table:

SELECT ORDERNUMBER AS xx__rs_col_0 FROM ( 
      SELECT * FROM OrderDetails
) colQry

but adding one more level does not work:

SELECT xx__rs_col_0 FROM ( 
      SELECT ORDERNUMBER AS xx__rs_col_0 FROM ( 
            SELECT * FROM OrderDetails
      ) colQry
) aliasQry

as I get this error:
Exception in thread "main" java.sql.SQLException: class org.apache.ignite.IgniteCheckedException: null at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:1010) at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:234) at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.executeQuery(JdbcThinStatement.java:142) at infofabrik.Main.main(Main.java:25)

Any ideas what may be going wrong here? is this supported ? Using ignite-core 2.14.0

Apache ignite failing on statup

I am trying mapp a field but getting this error on startup

Caused by: class org.apache.ignite.IgniteCheckedException: Failed to create page store work directory: /home/user/apache-ignite/work/db/0_0_0_0_0_0_0_1_lo_10_0_0_100_10_0_0_203_10_99_0_5_127_0_0_1_198_18_0_1_198_18_0_129_198_18_0_65_198_18_2_1_2604_3d09_679_2b00_0_0_0_1dc7_wlp0s20f3_2604_3d09_679_2b00_8332_15b9_3cd0_484_wlp0s20f3_2604_3d09_679_2b00_ae91_a1ff_fe7e_63ba_enxac91a17e63ba_2604_3d09_679_2b00_d986_6638_3c47_cd7c_wlp0s20f3_2604_3d09_679_2b00_e8bf_ac63_b39_630e_enxac91a17e63ba_47501 at org.apache.ignite.internal.util.IgniteUtils.ensureDirectory(IgniteUtils.java:9971) at org.apache.ignite.internal.processors.cache.GridLocalConfigManager.<init>(GridLocalConfigManager.java:127) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:618) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1784) ... 11 more Failed to start grid: Failed to start processor: GridProcessorAdapter []

in https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/GridDiagnostic.java i see

long initBytes = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getInit();

I understand its warning but to get rid it what settings what changes i should make in my shell scripts ?

I m using binary install.

apache / ignite Goto Github PK

ignite's Introduction

Apache Ignite

What is Apache Ignite?

Multi-Tier Storage

Ignite Native Persistence

ACID Compliance

ANSI SQL Support

High-Performance Computing

ignite's People

Contributors

Stargazers

Watchers

Forkers

ignite's Issues

Question：

Conditions and usage：

Problem analysis:

Repair suggestion:

I tried searching for where to ask questions but didn't found anything useful.Hence the issue.

What are various ways though which I can load data from Postgres table apart from using spark?

Recommend Projects

Recommend Topics

Recommend Org