jpos / jpos-ee Goto Github PK

Hi, i'm trying to use the elasticsearch module as it appears in the documentation and it does not working like we expect.

In this commit 63b619a we can see two main configuration files:

• modules/elasticsearch/src/main/resources/META-INF/q2/installs/deploy/00_logger_elastic.xml
• modules/elasticsearch/src/main/resources/META-INF/q2/installs/deploy/01_logger_elastic_daemon.xml

I configured both, and I don't see logs in kibana. So I then proceeded to set up a Remote JVM Debug application to debug both clases (ElasticSearchLogListener.java and ElasticSearchService.java)

When I run q2-outgoing (we set the module there), the ElasticSearchLogListener breakpoints stop the threads fine, and debugging I saw all works like we need.
But later when the message goes to the queue, the breakpoints into ElasticSearchService class never stop the threads and we don't see anything in elastic/kibana.. it's like this code has never been excecuted.

Then I proceed to change the 01_logger_elastic_daemon.xml configuration as the attachment file (pulling out the logger tag).
01_logger_elastic_daemon.xml.txt

Restart q2-outgoing and then the breakpoints into ElasticSearchService class stop the threads fine, and we can see all logs into Kibana correctly.

Unfortunately, the code works synchronously and if elastic service is down, the transaction never go back to q2-incoming.

I'm missing something, right?

Thanks!

Jpos server hangs on HttpResponse 204

When sending a request to an API which returns a 204 on success.

If you need to have the responseBodyOnError = true
and the API sends back a 204 on success the Jpos server just hangs. The issue is around the following code within the HttpQuery class

boolean includeResponse= (sc == HttpStatus.SC_CREATED) || (sc == HttpStatus.SC_OK) || responseOnError; if (includeResponse) { try { ctx.put (responseName, EntityUtils.toString(result.getEntity())); } catch (IOException e) { ctx.log (e); } }

The actual problem is in this line
ctx.put (responseName, EntityUtils.toString(result.getEntity()));

It is because the EntityUtils is attempting to convert the result.getEntity into an entity to assign as the value for the response name in the context map, in this case the result is actually null so instead of the EntityUtils throwing an exception, it just causes the server to hang.

This causes a hang because the response does not have a body given the 204.

There should be null safety checks here and if the result.getEntity is null then assign the status value (sc) to the context instead.

After the recent fixups submitted around balance calculations I think it's worth using something like embedded-postgres in the tests to ensure all the implementations produce the same results.

Change

and

from

<!DOCTYPE hibernate-mapping
  SYSTEM "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">

to

<!DOCTYPE hibernate-mapping 
  SYSTEM "http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">

TooLongFrameException: HTTP header is larger than 8192 bytes

I propose implementing the TriggerListener for the QuartzAdaptor. In the methods we can log info like when the trigger will fire again on trigger fire

e.g.

@Override
    public void triggerFired(Trigger trigger, JobExecutionContext context) {

        LogEvent evt = getLog().createInfo();
        evt.addMessage(getName() + " Trigger Fired and it will fire again at " + trigger.getNextFireTime() + "  "
                + context.getJobDetail());
        Logger.log(evt);

    }

    @Override
    public boolean vetoJobExecution(Trigger trigger, JobExecutionContext context) {
        return false;
    }

    @Override
    public void triggerMisfired(Trigger trigger) {
        LogEvent evt = getLog().createInfo();
        evt.addMessage(getName() + " Trigger MIS-Fired and will fire again at " + trigger.getNextFireTime());
        Logger.log(evt);

    }

I also propose adding a a misfire instruction when the trigger is created. At times the trigger does not fire and we have no clue what happened. The above trigger listener will help with logging the misfire.
Based on the default config of "org.quartz.jobStore.misfireThreshold", "60000", if the trigger is not fired in one minutes time it will be treated as a misfire and the instruction withMisfireHandlingInstructionFireAndProceed it will fire the trigger.

This may not suit everyone but can be made configurable to select what to do on a misfire.
e.g.

                trigger = TriggerBuilder.newTrigger()
                                        .withIdentity(e.getAttributeValue("id"), getName())
                                        .withSchedule(CronScheduleBuilder.cronSchedule(e.getAttributeValue("when"))
                                                                         .withMisfireHandlingInstructionFireAndProceed())
                                        .build();

Reported in jPOS Users:

DB db = new DB();    // uses the db.properties and used by the (Qi app)

DB secondDB = new DB("secondDB.cfg.xml");  // contains all the connection properties and the mapping, used by (QRest app)

DB thirdDB = new DB("thirdDB.cfg.xml");  // contains all the connection properties and the mapping, used by (QServer app)


These connections successfully created but when calling db "The first one" after secondDB is created it uses the secondDB tables.
and the same if thirdDB created it use it's tables, the scondDB also will use the the thirdDB tables.

As of now, jPOS-EE references Jetty 9.4.x, for which End of Community Support has been announced as of June 1st, 2022.

While EOL is unlikely to happen before 2025, it might be good to start making plans to migrate over to 10.x or 11.x.

I see the changelog https://github.com/jpos/jPOS-EE/wiki/ChangeLog#jpos-ee-229----released-2023-12-23
has logback upgraded to 1.4.2 230766b4

This wont work if java 8 is being used for building (I had run into this issue when I was upgrading libs due to CVE)

This returns the same instance of the scheduler if the adapter is used in multiple deploys.
Which means un-deploying one file destroys the scheduler and the jobs in other deploy files don't execute. To keep jobs independent of other jobs deployed in multiple files there is a need to get a new instance of the scheduler. Multiple jobs in one file can also be a problem if you have set to execute a job at recurring short interval as you may miss the window of its trigerring during the undeploy->edit->redploy.
It can be fixed by mimicking the properties that are used by the default scheduler and setting its instance name to the bean. This wont break all jobs in one file behavior and just adds flexibility to having multiple schedulers.

        Properties p = new Properties();

        p.setProperty("org.quartz.scheduler.instanceName", getName());
        p.setProperty("org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread", "true");
        p.setProperty("org.quartz.scheduler.rmi.proxy", "false");
        p.setProperty("org.quartz.scheduler.rmi.export", "false");
        p.setProperty("org.quartz.jobStore.misfireThreshold", "60000");
        p.setProperty("org.quartz.threadPool.threadCount", "10");
        p.setProperty("org.quartz.scheduler.instanceName", getName());
        p.setProperty("org.quartz.scheduler.wrapJobExecutionInUserTransaction", "false");
        p.setProperty("org.quartz.threadPool.class", "org.quartz.simpl.SimpleThreadPool");
        p.setProperty("org.quartz.jobStore.class", "org.quartz.simpl.RAMJobStore");
        p.setProperty("org.quartz.threadPool.threadPriority", "5");

        SchedulerFactory sf = new StdSchedulerFactory(p);

        scheduler = sf.getScheduler();

The above properties can be found in the quartz jar's \lib\quartz-2.2.3.jar\org\quartz\quartz.properties , this is the default file that is used.
It could be 2.2.3 or an older version it doesn't matter.

Currently, the timeout is hard coded to 60000L, can we make the timeout configurable?

Since the routes can use different Txn Mgrs, a timeout per route would be good to have.

In heartbeat the session is kept open for longer than required.
The sleep should probably go after the session close?

https://github.com/jpos/jPOS-EE/blob/master/modules/status/src/main/java/org/jpos/ee/status/ChannelMonitor.java#L31-L32

Please, can we change these 2 to protected?
I extend the channel monitor (different package name) and need access to the variables to allow additional commands like signon/signoff/key exchange etc, and I don't have my package structure like the ChannelMonitor's.

It would be nice that, in addition to be able to load DB properties from a file, it would be possible to load it from a resource in the class path.

I'm proposing that, similarly to the packages, use the jar: prefix for that, I'm writing the code for the PR, but if anyone else sees a reason not to do this or to do it differently I can adapt. Personally, I would prefer classpath: (or cp) or something else like that as a prefix, but for consistency I will go with jar:. Or I could implement all of them if agreed.

I'm talking about the DB_PROPERTIES system property in:

running gradlew build, hangs on the following.
<=======------> 58% EXECUTING [6m 18s]

:modules:qi-core:vaadinCompile

environment windows 7

Buen dia Senor Alejandro, descargue jpos-ee con github para maven pero la verdad no he podido creae un proyecto. Gracias

In investigating a performance regression in Java 9 and newer it was discovered that on Java 8 does not properly flush the buffers to disk. So we should probably see if we can fix this on Java 8 and find a workaround for the performance issue on Java 9 and newer on OSX. See #152 and this for details.

As I migrate to the current jpos/jposee, I run sonarqube scanner and it picked this up.

If it makes sense, change it to the following so that the socket is always closed, or add a finally block.

  try (Socket socket = new Socket(host, port)) {
            socket.setSoLinger (true, 0);
            rc = true;
        } 

Per Sumeet's use case described in https://groups.google.com/forum/#!msg/jpos-users/JGaFKF1modQ/0WoIuD5Yo-oJ we need the ability to slow down the delivery of SAF entries for the first time.

When you add multiple databases (slave), there is a convention of using colon ":" in database configuration file names. But while working on windows operating system, it seems impossible to create a file name having colon in its name.
For example:
In cfg directory, db.properties is fine. But following file names are problematic as these cannot be created.
slave1:db.properties
slave2:db.properties

If there is a workaround, please assist. And if it's really an issue, would appreciate to provide its support.

After the configuration of the project is done and I try to initialize ./modules/testbed/build/install/testbed/bin/q2 -cli it fails with the following error:

[0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:log/gc.log instead.
Unrecognized VM option 'ExplicitGCInvokesConcurrentAndUnloadsClasses'
Did you mean '(+/-)ExplicitGCInvokesConcurrent'? Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Where can I change the initialization vars of the JVM?

status column does not exist in SysLog entity. So accessing the events property creates an invalid SQL sentence. Maybe it should point to source?

return validParam(ctx, entry, value);

should be changed to the following to return false only if the method returns false, else continue in the for loop

if (!validParam(ctx, entry, value);)
  return false;

This was reported (#265)

Hi,

I was running some tests and found balance cache mechanism looks a bit broken to me.

Scenario is my transentry and transacc table are empty and I am running my very first GL transaction.

The method getSafeMaxGLEntryId returns zero till the safe window difference is reached MAX(maxGLEntryId - SAFE_WINDOW, 0)
https://github.com/jpos/jPOS-EE/blame/master/modules/minigl/src/main/java/org/jpos/gl/GLSession.java#L2026

The method createFinalbalanceCache doesn't create the cache because the following condition is not met maxId != c.getRef()
https://github.com/jpos/jPOS-EE/blame/master/modules/minigl/src/main/java/org/jpos/gl/GLSession.java#L1547

Not sure if this is the intended purpose of balance cache, but I think it should be created starting from the very first GL entry?

If we initialize the ref with -1 in BalanceCache class constructor. So if a balance cache is not found and a new one is going to be created the following condition should allow it maxId != c.getRef()

I know we can override the safe window, but doing so will increase the chances of race condition and kill the purpose of the safe window.

For example the balance cache is never used by FinalBalance min/max rules as it sends in a max id of 0
I think the balance cache should be allowed to use when maxId is sent in as 0?

See this line: https://github.com/jpos/jPOS-EE/blob/master/modules/minigl/src/main/java/org/jpos/gl/GLSession.java#L1072

Have been running some tests to see what it would take to migrate from my old hibernate 3.6 to the current 5 something.

When my hibernate cfg references a mapping that does have its getters setters set correctly an exception gets thrown at this line which is not caught and user has no idea what happened. One will see newSessionFactory being invoked everytime a db is used meaning that there is a problem somewhere and sessionFactory creation is failing. But root cause is not identified as org.hibernate.MappingException does not get caught and logged (at least in my test case).

I wrapped the above referenced line in a try catch block with a printstacktrace only then do I know from the stack trace towards the end "Could not locate getter method for property [a.b.c.WebUserRoles#userId]"

Suggestion is to somehow make this visible to track misconfiguration or bad code.

org.hibernate.MappingException: Could not get constructor for org.hibernate.persister.entity.SingleTableEntityPersister
	at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:123)
	at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:77)
	at org.hibernate.metamodel.internal.MetamodelImpl.initialize(MetamodelImpl.java:181)
	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:301)
	at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:469)
	at org.hibernate.boot.internal.MetadataImpl.buildSessionFactory(MetadataImpl.java:195)
	at org.jpos.ee.DB.newSessionFactory(DB.java:191)
	at org.jpos.ee.DB.getSessionFactory(DB.java:160)
	at org.jpos.ee.DB.<init>(DB.java:123)
	at org.jpos.ee.DB.<init>(DB.java:95)
	at Test.main(Test.java:31)
Caused by: org.hibernate.HibernateException: Unable to instantiate default tuplizer [org.hibernate.tuple.entity.PojoEntityTuplizer]
	at org.hibernate.tuple.entity.EntityTuplizerFactory.constructTuplizer(EntityTuplizerFactory.java:91)
	at org.hibernate.tuple.entity.EntityTuplizerFactory.constructDefaultTuplizer(EntityTuplizerFactory.java:116)
	at org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:413)
	at org.hibernate.persister.entity.AbstractEntityPersister.<init>(AbstractEntityPersister.java:604)
	at org.hibernate.persister.entity.SingleTableEntityPersister.<init>(SingleTableEntityPersister.java:125)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:96)
	... 10 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.hibernate.tuple.entity.EntityTuplizerFactory.constructTuplizer(EntityTuplizerFactory.java:88)
	... 19 more
Caused by: org.hibernate.PropertyNotFoundException: Could not locate getter method for property [a.b.c.WebUserRoles#userId]
	at org.hibernate.internal.util.ReflectHelper.findGetterMethod(ReflectHelper.java:465)
	at org.hibernate.property.access.internal.PropertyAccessBasicImpl.<init>(PropertyAccessBasicImpl.java:41)
	at org.hibernate.property.access.internal.PropertyAccessStrategyBasicImpl.buildPropertyAccess(PropertyAccessStrategyBasicImpl.java:27)
	at org.hibernate.mapping.Property.getGetter(Property.java:311)
	at org.hibernate.tuple.entity.PojoEntityTuplizer.buildPropertyGetter(PojoEntityTuplizer.java:193)
	at org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:145)
	at org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:53)
	... 24 more

As I migrate to the current jpos/jposee, I run sonarqube scanner and it picked this up.

If it makes sense, change it to the following so that the file is always closed, or add a finally block to close it.

        if (f.exists()) {
            try (FileReader fr = new FileReader(f)) {
                props.load(fr);
            }
        }

https://github.com/jpos/jPOS-EE/blob/master/modules/minigl/src/test/java/org/jpos/gl/RulesTest.java#L132

This test doesn't pass if you use postgres. Instead of throwing an exception, it continues until the fail() call. The test passes successfully when using the test database - H2. Since it's using H2, GLSession will use getBalancesORM to look up the balance. When using postgres, it'll create a postgres dialect specific query instead.

Since there was a difference between H2/postgres I tweaked getBalances to always use getBalancesORM, to see if the tests agreed again, and they did.

It appears when using postgres, the wrong balance can be returned in that test (unsure why), but on H2 at least, the assertions based on the 48001 amount appear to hold. I'm not sure which is wrong (getBalancesORM / getBalances + postgres) because the transaction amount 48001 was probably chosen just because it triggers the error in tests.. which always used H2.

To test with postgres, I created a hibernate.cfg.xml file in modules/minigl/src/main/test/resources, and in the TestBase and Import classes, made sure to have db = new DB("hibernate.cfg.xml") instead of new DB(). Also - you need to add testImplementation project(':modules:db-postgresql') to the minigl build.gradle

Since Debug.java extends the TxnSupport's prepare(...) method instead of doPrepare(...) the profiler does not create the checkpoint and no timing info is available for the debug participant.

Similarly abort can be changed to doPrepareForAbort(...)

This methods for creating a reversal of a gl transaction

• jPOS-EE/modules/minigl/src/main/java/org/jpos/gl/GLTransaction.java

  Line 293 in 59b3ea4

  public GLTransaction createReverse() {

• jPOS-EE/modules/minigl/src/main/java/org/jpos/gl/GLTransaction.java

  Line 317 in 59b3ea4

  public GLTransaction createReverse(Tags withTags, Tags withoutTags) {

Loose the tags of the original transactions, which in some use cases may be needed to propagate to the reversal entries.

Create versions of those methods with an added boolean keepEntryTags parameter that specifies if the reversal entries should maintain the originals tags.

https://github.com/jpos/jPOS-EE/blob/master/modules/qrest/src/main/java/org/jpos/qrest/SendResponse.java#L49-L50

I have a participant that takes a lot of time. LongRunningParticpant

This LongRunningParticpant is placed after the SendResponse participant in the txn mgr.

But since the response is only sent in the commit method

https://github.com/jpos/jPOS-EE/blob/master/modules/qrest/src/main/java/org/jpos/qrest/SendResponse.java#L54-L60

It gets sent after the  'LongRunningParticpant' completes.

Snippet of profiler

    <profiler>
     prepare: o.j.q.p.Router [0.6/0.6]
     prepare: c.o.a.PrepareContext [1.7/2.4]
     prepare: c.o.a.InsertRulesInDB [21.8/24.2]
     prepare: o.j.q.SendResponse [0.1/24.4]
     prepare: c.o.a.TriggerMerchantBlockCacheReload [60241.7/60266.1]
      commit: c.o.a.PrepareContext [0.1/60266.2]
      commit: c.o.a.InsertRulesInDB [0.0/60266.3]
      commit: o.j.q.SendResponse [0.8/60267.1]
      commit: c.o.a.TriggerMerchantBlockCacheReload [0.0/60267.1]
     end [1.2/60268.3]
   </profiler>

So the question is why does the prepare not do anything in the SendResponse participant?

Use Case:
Want to use the client sim to load a million transactions in the DB through the switch.

Used the count attribute in the test case to run a transaction a million times.

https://github.com/jpos/jPOS-EE/blob/master/modules/client-simulator/src/main/java/org/jpos/simulator/TestRunner.java#L161

Quickly ran into an out of memory due to the the way the suite is created.
The suite is basically a list of testcases in the file and it repeats the addition of test cases based on the count, which I feel is not the efficient way of doing it, why repeat a test case in the list when its practically the same test.

So instead of adding the same test case a million (count) times in the list, just add it once, add a new property for the testcase called repetition count.

Add a getter and setter on count in the TestCase class.
So at https://github.com/jpos/jPOS-EE/blob/master/modules/client-simulator/src/main/java/org/jpos/simulator/TestRunner.java#L163 insert

tc.setCount(count);

Now in the TestRunner.runsuite get the count and loop there.

https://github.com/jpos/jPOS-EE/blob/master/modules/client-simulator/src/main/java/org/jpos/simulator/TestRunner.java#L96 insert

for (long repetition = 0; repetition < tc.getCount(); repetition++) {

https://github.com/jpos/jPOS-EE/blob/master/modules/client-simulator/src/main/java/org/jpos/simulator/TestRunner.java#L126

Close the for loop.

Sorry, my file and the one in the repo are formatted differenty and versions are different, hence resorting to this.

CryptoService creates a SecureRandom instance to create random IVs, but it's never re-created or re-seeded. Is that intended?

Bear in mind that a JPOS application could be running for months and performing thousands of encryptions, and articles like this one https://www.synopsys.com/blogs/software-security/proper-use-of-javas-securerandom/ suggest that a SecureRandom instance shouldn't be used for a long time with the same seed. Maybe a simple solution like creating a new instance of SecureRandom at the same time a new AES key is created would be an improvement.

Thank you, and please let me know if there is a better place to ask questions like this.