elasticsearch can't resolve environment variables in elasticsearch.yml - elasticsearch

Question

I have the following two settings in my elasticsearch.yml file. They are the only ones that pull from environment variables.
cloud.aws.access_key: ${AWS_ACCESS_KEY_ID}
cloud.aws.secret_key: ${AWS_SECRET_KEY}
When I restart elasticsearch to load these from the environment, I get an error that it can't resolve them. I've tested it and it will not resolve either, so this error applies to both (it just fails on the bottom one first)
- IllegalArgumentException[Could not resolve placeholder 'AWS_SECRET_KEY']
java.lang.IllegalArgumentException: Could not resolve placeholder 'AWS_SECRET_KEY'
at org.elasticsearch.common.property.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:124)
at org.elasticsearch.common.property.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:81)
at org.elasticsearch.common.settings.ImmutableSettings$Builder.replacePropertyPlaceholders(ImmutableSettings.java:1060)
at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareSettings(InternalSettingsPreparer.java:101)
at org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:106)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:177)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
I did some investigating through elasticsearch's code repository on github and discovered this bit of code that pulls from the environment variables.
ImmutableSettings.java#resolvePlaceholder from elasticsearch#github
Namely. the lines inside that function that should be pulling from the environment variables are these one:
Code from resolvePlaceholder that pulls out environment variables
However, after resolvePlaceholder is run from inside function PropertyPlaceholder#parseStringValue, the System.getenv call must be returning null as that is the only way for that error to be thrown.
I wrote a simple test program that is essentially a copy of ImmutableSettings.java#resolvePlaceholder to test that System.getenv was pulling out the environment variables correctly on my system. This in fact returns the values I expect.
public class Cool {
public static void main(String[] args) {
System.out.println(resolvePlaceholder(args[0]));
}
public static String resolvePlaceholder(String placeholderName) {
if (placeholderName.startsWith("env.")) {
// explicit env var prefix
System.out.println("1: placeholderName.startsWith(\"env.\")");
return System.getenv(placeholderName.substring("env.".length()));
}
String value = System.getProperty(placeholderName);
if (value != null) {
System.out.println("2: System.getProperty");
return value;
}
value = System.getenv(placeholderName);
if (value != null) {
System.out.println("3: System.getenv");
return value;
}
return "Map should've had it";
}
}
When run, this is the output, showing we are getting the set environment variables (keys hidden for obvious reasons):
[ec2-user#ip-172-31-34-195 ~]$ java Cool AWS_SECRET_KEY
3: System.getenv
XXXXXXXXXXXXXXXXXX
[ec2-user#ip-172-31-34-195 ~]$ java Cool AWS_ACCESS_KEY_ID
3: System.getenv
XXXXXXXXXXXXXXXXXX
What is it about elasticsearch that isn't able to parse my environment variables from elasticsearch.yml? I've done quite a bit of digging at this point but I'm sure there is a simple solution around the corner. Any help would be very much appreciated.

Answer 1

I figured out the issue.
As I am running elasticsearch as a linux service, rather than a shell application, it has access to no environment variables except for a very select few.
I added the following line to the end of /etc/sysconfig/elasticsearch to load the environment variables I wanted available to the program:
. /path/to/environment/variables