How to add custom fields

[mobby38]

Hi,

Would it be possible to add custom field value only in the Json message to elastic?
For example, it would be good to add the name or ip of the host, so that you can recognize where comes the log.

Thanks.

[rfoltyns]

@mobby38 You can use ThreadContext as decribed here

[mobby38]

Yes, but how you can do that without modifying the existing code.

The proof of concept I'm trying to achieve is to replace filebeat with just a log4j2 appender, but it misses those data, and I can't modify the code of the application.

What I could do is to provide Environment properties and give it to Log4j2 via the configuration.
Is there a way to do so?

[rfoltyns]

You can do that using configuration. Just switch to Log4j2 JsonLayout and do that with KeyValuePairs and env variables. You'll sacrifice the ability to use object pooling and customisable output though. Plus, you'll need to specify a new date column in your mappings as default set of mixins in JsonLayout serialize LogEvent a bit differently.
I'll add support for KeyValuePairs in JacksonJsonLayout in 1.4.

[mobby38]

Thanks :)

[mobby38]

In my opinion, you should also remove jsonIgnore in LogEventJacksonJsonMixIn and avoid renaming threadName by thread.
Just let the trick on timeMillis.

It tooks me some times to understand why not all attribute (including context) was hidden in the Json message.

[rfoltyns]

Good point on the contextData. There's a lot of confusion over this Log4j2 feature and I think I've gone too far while making the LogEvent 'thinner'.

Thankfully, you can augment any part of the output quite easily with JacksonMixIn.

[ricsil]

Is 1.4 coming? Just thinking whether to wait for VirtualProperty or try to use KeyValuePair

[rfoltyns]

Yes. 1.4 should be out soon. It got heavily delayed due to FileBackedRetryFailoverPolicy work - sorry for that..

[ricsil]

Could you add an example of JsonLayout to docs? Simply replacing JacksonJsonLayout with JsonLayout doesn't error out but doesn't push anything to Elastic either, was wondering whether I missed something.

[rfoltyns]

@ricsil Sure, will do. Could you post your updated configuration in the meantime?

[ricsil]

So, this works fine (mix-in is there for timestamp override)

    <Elasticsearch name="elasticsearchAsyncBatch">
      <RollingIndexName indexName="test-log-app" pattern="yyyy-MM-dd" />
      <JacksonJsonLayout>
        <JacksonMixIn mixInClass="path.to.utils.CustomLogEventMixIn"
          targetClass="org.apache.logging.log4j.core.LogEvent"/>
      </JacksonJsonLayout>
      <AsyncBatchDelivery>
        <JestHttp serverUris="http://elastic-server:9200" />
      </AsyncBatchDelivery>
    </Elasticsearch>

Replacing JacksonJson with Json doesn't produce anything

    <Elasticsearch name="elasticsearchAsyncBatch">
      <RollingIndexName indexName="test-log-app" pattern="yyyy-MM-dd" />
      <JsonLayout />
      <AsyncBatchDelivery>
        <JestHttp serverUris="http://elastic-server:9200" />
      </AsyncBatchDelivery>
    </Elasticsearch>

On the other hand, replacing JacksonJsonLayout with EcsLayout (not forgetting to add dependency) works fine:

    <Elasticsearch name="elasticsearchAsyncBatch">
      <RollingIndexName indexName="test-log-app" pattern="yyyy-MM-dd" />
      <EcsLayout />
      <AsyncBatchDelivery>
        <JestHttp serverUris="http://elastic-server:9200" />
      </AsyncBatchDelivery>
    </Elasticsearch>

https://github.com/elastic/ecs-logging-java/tree/master/log4j2-ecs-layout

Also, EcsLayout supports KeyValuePair in case someone needs it.

[rfoltyns]

@ricsil you need
<JsonLayout compact="true"/> cause default one prints pretty JSON. Have a look and updated docs

[rfoltyns]

VirtualProperty support released in 1.4.