Environment variable in Logstash not being parsed correctly - elasticsearch

Question

I come here after reading How to reference environment variables in logstash configuration file?.
Unfortunately it did not work for me.
I am running:
bin/logstash -f my_filters.conf --debug
And my config file is:
input {
file {
path => "/tmp/${RUN_ID}/*.txt"
start_position => beginning
sincedb_path => "/dev/null"
ignore_older => 0
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "${RUN_ID}"
}
}
And no new index is being created.
This is after setting:
export RUN_ID=500
For example.
If I change my config to have a hard-coded value (e.g. 500), then the index is created without a problem.
I have read the documentation and it mentions exactly what I'm doing right now...
What am I doing wrong, how can I get the environment variable working?

Answer 1

Logstash 2.4 requires a command line argument of --allow-env to do environment substitutions.
Without the flag it doesn't complain (but doesn't work)
bin/logstash -f test.conf
Settings: Default pipeline workers: 8
Pipeline main started
With the flag, it will complain if you don't set it:
bin/logstash --allow-env -f test.conf
fetched an invalid config {:config=>"input {\n file {\n path => \"/tmp/${RUN_ID}/*.txt\"\n start_position => beginning\n sincedb_path => \"/dev/null\"\n ignore_older => 0\n }\n}\n\noutput {\nstdout { codec=>rubydebug}\n elasticsearch {\n hosts => [ \"localhost:9200\" ]\n index => \"${RUN_ID}\"\n }\n}\n\n\n", :reason=>"Cannot evaluate `${RUN_ID}`. Environment variable `RUN_ID` is not set and there is no default value given.", :level=>:error}
And of course with the argument and flag, everything works right:
export RUN_ID=10
bin/logstash --allow-env -f test.conf
Pipeline main started
{
"message" => "asdfasdf",
"#version" => "1",
"#timestamp" => "2016-11-01T21:10:15.964Z",
"path" => "/tmp/10/test.txt",
"host" => "XXXXXXXXX.local"
}