AWS Signed requests when indexing documents with Spring Data Elasticsearch - elasticsearch

Question

I'm unable to index a document in the AWS-hosted Elasticsearch cluster using signed requests.
Infrastructure setup
Elasticsearch version: 7.4
Access policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:<RESOURCE>/*"
}
]
}
Code
The following code loads the client libraries using version 7.6. I have also downgraded them to match the cluster version but with no effect.
build.gradle
// ...
implementation("org.springframework.data:spring-data-elasticsearch")
implementation("org.elasticsearch:elasticsearch")
implementation("org.elasticsearch.client:elasticsearch-rest-high-level-client")
// ...
The client configuration definition. The environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_PROFILE are filled.
#Configuration
public class ElasticsearchClientConfig extends AbstractElasticsearchConfiguration {
#Value("${elasticsearch.host}")
private String elasticsearchHost;
#Value("${elasticsearch.port}")
private int elasticsearchPort;
#Override
#Bean
public RestHighLevelClient elasticsearchClient() {
var SERVICE_NAME = "es";
var REGION = "us-east-1";
var defaultCP = new DefaultAWSCredentialsProviderChain();
AWS4Signer signer = new AWS4Signer();
signer.setServiceName(SERVICE_NAME);
signer.setRegionName(REGION);
HttpRequestInterceptor interceptor = new AWSRequestSigningApacheInterceptor
(SERVICE_NAME, signer, defaultCP);
RestClientBuilder restClientBuilder = RestClient
.builder(HttpHost.create(elasticsearchHost))
.setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(interceptor));
return new RestHighLevelClient(restClientBuilder);
}
}
Where the AWSRequestSigningApacheInterceptor is taken from here.
So far so good. When the application loads it's accessing the cluster and manages to create relevant indices correctly.
Problem
There problem is when performing save() operation from Spring Data repository. There are two requests made to ES
#Override
public <S extends T> S save(S entity) {
Assert.notNull(entity, "Cannot save 'null' entity.");
operations.save(entity, getIndexCoordinates());
operations.indexOps(entity.getClass()).refresh();
return entity;
}
Looking at the logs the first one succeeds. The following error ends the second call
org.elasticsearch.client.ResponseException: method [POST], host [HOST], URI [/asset/_refresh?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true], status line [HTTP/1.1 403 Forbidden]
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
Looking at more detailed logs for both operations
Call for saving (ends with 200 status code):
com.amazonaws.auth.AWS4Signer : AWS4 Canonical Request: '"PUT
/asset/_doc/2
timeout=1m
content-length:128
content-type:application/json
host:<HOST>
user-agent:Apache-HttpAsyncClient/4.1.4 (Java/11.0.2)
x-amz-date:20200715T110349Z
content-length;content-type;host;user-agent;x-amz-date
55c1faf282ca0da145667bf7632f667349dbe30ed1edc64439cec2e8d463e176"
2020-07-15 13:03:49.240 DEBUG 3942 --- [nio-8080-exec-1] com.amazonaws.auth.AWS4Signer : AWS4 String to Sign: '"AWS4-HMAC-SHA256
20200715T110349Z
20200715/us-east-1/es/aws4_request
76b6547ad98145ef7ad514baac4ce67fa885bd56073e9855757ade19e28f6fec"
Call for refreshing (ends with 403 status code):
com.amazonaws.auth.AWS4Signer : AWS4 Canonical Request: '"POST
/asset/_refresh
host:<HOST>
user-agent:Apache-HttpAsyncClient/4.1.4 (Java/11.0.2)
x-amz-date:20200715T110349Z
host;user-agent;x-amz-date
bbe4763d6a0252c6e955bcc4884e15035479910b02395548dbb16bcbad1ddf95"
2020-07-15 13:03:49.446 DEBUG 3942 --- [nio-8080-exec-1] com.amazonaws.auth.AWS4Signer : AWS4 String to Sign: '"AWS4-HMAC-SHA256
20200715T110349Z
20200715/us-east-1/es/aws4_request
189b39cf0475734e29c7f9cd5fd845fc95f73c95151a3b6f6d430b95f6bee47e"
When indexing documents directly using lower-level clients everything works fine. I suspect that signature calculation behaves incorrectly for subsequent API calls.

Answer 1

I had a same issue, in my case i'm using AWSRequestSigningApacheInterceptor, and i had old version. after upgrade to latest version, it's fixed.