elasticsearch without Xpack about log4j2-elasticsearch HOT 11 CLOSED

<Elasticsearch name="ELASTICSEARCHASYNCBATCH" >
  <IndexName indexName="log4j2" />
  <AsyncBatchDelivery>
    <ElasticsearchBulkProcessor  serverUris="http://myurl">
      <XPackAuth>
        <BasicCredentials username="elasticsearch" password="elasticsearch123" />
      </XPackAuth>
    </ElasticsearchBulkProcessor>
  </AsyncBatchDelivery>
</Elasticsearch>

from log4j2-elasticsearch.

You're right, but I didn't write XPackAuth for this scenario.

Could you test it with your own build and modified org.appenders.log4j2.elasticsearch.bulkprocessor.XPackAuth (remove @required and null check)? If not, I'll test it out with more relaxed rules and nginx in between and publish a SNAPSHOT for you before I include it in 1.3.1.

from log4j2-elasticsearch.

I did your test with a local build and got
2019-01-16 10:10:44,825 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.AsyncBatchDelivery for element AsyncBatchDelivery: java.lang.NullPointerException java.lang.NullPointerException
at org.appenders.log4j2.elasticsearch.bulkprocessor.XPackAuth.configure(XPackAuth.java:51)

@Override
public void configure(Settings.Builder settings) {
    credentials.applyTo(settings);
    ==> certInfo.applyTo(settings);
}

from log4j2-elasticsearch.

Null check should do the trick here, but have in mind that BulkProcessorObjectFactory.getClientProvider() returns PreBuiltXPackTransportClient - it may have more limitations. If you have issues with log4j2-elasticsearch6-bulkprocessor, log4j2-elasticsearch-jest will give you what you need.

I've started to work on more relaxed rules in all modules yesterday. It should be finished within the next few days.

from log4j2-elasticsearch.

Ok, I have to mention a couple of things here:

• I've relaxed the requirement for certInfo in log4j2-elasticsearch-jest; you should be able to use JestHttp with clusters secured only with credentials (wil be released in 1.3.1 probably this week)
• I've noticed that the error message you provided contains org.appenders.log4j2.elasticsearch.jest.XPackAuth from -jest module, but log4j2.xml config uses ElasticsearchBulkProcessor from -bulkprocessor module; this will not work - you should use only one of the client modules: -jest (for all versions) or one of -bulkprocessor (specific to your cluster version)
• -bulkprocessor module uses TCP client with custom protocol - it cannot be secured in a way that you expect; curl works because you execute it against HTTP, not TCP port; if you're not using XPack, you have to switch to -jest module (HTTP client) to get it working.

Please have a look at the commit mentioned above and see if it meets your expectations.

from log4j2-elasticsearch.

pulled your changes and build locally:

log4j config

<Elasticsearch name="elasticsearchAsyncBatch">
  <IndexName indexName="log4j2" />
  <AsyncBatchDelivery batchSize="1000" deliveryInterval="5000" >
    <JestHttp serverUris="????" >
      <XPackAuth>
        <BasicCredentials username="????" password="????" />
      </XPackAuth>
    </JestHttp>
  </AsyncBatchDelivery>
</Elasticsearch>

maven …

<dependency>
  <groupId>org.appenders.log4j</groupId>
  <artifactId>log4j2-elasticsearch-jest</artifactId>
  <version>1.3.1-SNAPSHOT</version>
</dependency>

running springboot …
at startup
2019-01-22 16:57:45,280 main INFO BatchEmitterFactory class found org.appenders.log4j2.elasticsearch.jest.BulkEmitterFactory
2019-01-22 16:57:45,285 main INFO Using org.appenders.log4j2.elasticsearch.jest.BulkEmitterFactory@1d76aeea as BatchEmitterFactoryProvider
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.(SSLConnectionSocketFactory.java:146)
at io.searchbox.client.config.HttpClientConfig$Builder.build(HttpClientConfig.java:297)
at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory$JestClientProvider.createClient(JestHttpObjectFactory.java:271)
at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory$JestClientProvider.createClient(JestHttpObjectFactory.java:260)
at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory.createClient(JestHttpObjectFactory.java:102)
at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory$1.(JestHttpObjectFactory.java:112)
at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory.createBatchListener(JestHttpObjectFactory.java:109)
at org.appenders.log4j2.elasticsearch.jest.BulkEmitterFactory.createInstance(BulkEmitterFactory.java:40)
at org.appenders.log4j2.elasticsearch.jest.BulkEmitterFactory.createInstance(BulkEmitterFactory.java:30)
at org.appenders.log4j2.elasticsearch.spi.BatchEmitterServiceProvider.createInstance(BatchEmitterServiceProvider.java:70)
at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery.(AsyncBatchDelivery.java:50)
at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery$Builder.build(AsyncBatchDelivery.java:122)
at org.appenders.log4j2.elasticsearch.AsyncBatchDelivery$Builder.build(AsyncBatchDelivery.java:84)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
at org.apache.logging.log4j.core.config.AbstractConfiguration. at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:619)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:636)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.commons.logging.LogAdapter$Log4jLog.(LogAdapter.java:135)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:102)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:79)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
at org.springframework.boot.SpringApplication.(SpringApplication.java:194)
at pxm.rosy.ocs.jconfig.Application.main(Application.java:16)
Caused by: java.lang.NullPointerException
at org.apache.commons.logging.LogAdapter$Log4jLog.(LogAdapter.java:140)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:102)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:79)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
at org.apache.http.conn.ssl.AbstractVerifier.(AbstractVerifier.java:61)
at org.apache.http.conn.ssl.AllowAllHostnameVerifier.(AllowAllHostnameVerifier.java:44)
at org.apache.http.conn.ssl.AllowAllHostnameVerifier.(AllowAllHostnameVerifier.java:46)
... 35 more

from log4j2-elasticsearch.

Error comes from io.searchbox:jest:HttpClientConfig:297 (so the XPackAuth patch worked correctly)
if (this.httpsIOSessionStrategy == null) {
this.httpsIOSessionStrategy = SSLIOSessionStrategy.getSystemDefaultStrategy();
}
org.apache.http.conn.ssl.AbstractVerifier tries to obtain the Logger before Log4j2 config is ready and it leads to a chicken-egg problem.

Could you send me a zip with minimal project that reproduces it?

from log4j2-elasticsearch.

It's a bug in AsyncBatchDelivery. IndexTemplate is sent to cluster during initialization (basically, early Log4j2 configuration phase) when LoggerContext is not ready yet. It causes premature HTTP client initialization when loggers requested by underlying classes are not available.

I'll introduce basic life cycle to patch it in 1.3.1 and extend the life cycle in 1.4 (maybe earlier).

WORKAROUND Add
private static final Logger LOG = LogManager.getRootLogger() (or any other Logger)
to your Application class to initialize LoggerContext before Spring Boot starts . It's very nasty, because the whole plugin will be initialized more than once, but at least it will let you start the application.

from log4j2-elasticsearch.

workaround is OK, thanks for your effort

from log4j2-elasticsearch.

Glad to help. Thanks for the feedback! 👍
I'll commit a fix over the weekend and release it next week. Could you retest it then and close the issue if everything goes well, please?

from log4j2-elasticsearch.

I did a retest with 1.3.0 and it is ok

from log4j2-elasticsearch.