Fluentd Conditional Matches Based on Environment Variables

You can enable/disable FluentD Matches with environment variables in following way.

Below is fluent.conf.

<source>
  @type dummy
  dummy {"hello":"world"}
  @label @DUMMY
  tag dummy
</source>

<label @DUMMY>
    <match dummy>
        @type copy
        copy_mode deep
        <store>
            @type relabel
            @label @OPENSEARCH 
        </store>
        <store>
            @type relabel
            @label @ELASTICSEARCH 
        </store>
    </match>
</label>

<label @OPENSEARCH>
   @include "#{ENV['FLUENTD_OPENSEARCH']}"
</label>

<label @ELASTICSEARCH>
   @include "#{ENV['FLUENTD_ELASTICSEARCH']}"
</label>

Following is the content for fluent-elasticsearch.conf. For testing purpose, we will send events to stdout.

<match>
  @type stdout
</match>

Following is the content for fluent-opensearch.conf. For testing purpose, we will send events to stdout.

<match>
  @type stdout
</match>

And lastly, below is the content for fluent-null.conf which discards the events.

<match>
  @type null
</match>

With environment variables FLUENTD_OPENSEARCH & FLUENTD_ELASTICSEARCH you can provide either actual path to match conf or provide null match conf.

Below are the sample docker commands to test this approach.

Build

docker build -t localfluentd .

Run with both labels activated.

docker run --rm -e FLUENTD_ELASTICSEARCH="fluent-elasticsearch.conf" -e FLUENTD_OPENSEARCH="fluent-opensearch.conf" localfluentd

Disable @ELASTICSEARCH label where FLUENTD_ELASTICSEARCH will point to fluent-null.conf which discards the events.

docker run --rm -e FLUENTD_ELASTICSEARCH="fluent-null.conf" -e FLUENTD_OPENSEARCH="fluent-opensearch.conf" localfluentd

This repo has example configuration to test this approach.