I'm using reactive spring and trying to upload a ".csv" file into AWS S3 using a presigned URL.
I spring webclient for this and my code snippet as below
Webclient bean
#Bean
public WebClient webClient(WebClient.Builder webClientBuilder) {
ConnectionProvider provider = ConnectionProvider
.builder("customFixedConnectionPool").maxConnections(maxConnections).pendingAcquireTimeout(
Duration.ofMillis(acquireTimeout)).build();
HttpClient httpClient = HttpClient.create(provider).resolver(DefaultAddressResolverGroup.INSTANCE);
return webClientBuilder
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build(); }
FileUploadClient.java
Note: in below, BodyInserters.fromResource(resource) returns BodyInserter<Resource, ReactiveHttpOutputMessage>
#Component
public class FileUploadClient implements FileUploadClientInterface {
public FileUploadClient(WebClient webClient) {
this.webClient = webClient.mutate()
.build();
}
private final WebClient webClient;
#Override
public Mono<FileUploadResponse> uploadFIle(String preSignedUrl, String filePath) {
Resource resource = new PathResource(filePath);
return webClient.put()
.uri(preSignedUrl)
.body(BodyInserters.fromResource(resource))
.exchangeToMono(response -> {
if (response.statusCode().equals(HttpStatus.NO_CONTENT)) {
fileUploadResponse.setCode(HttpStatus.NO_CONTENT.value());
fileUploadResponse.setMessage("FILE_UPLOAD_SUCCESS");
return Mono.just(fileUploadResponse);
} else {
return response.bodyToMono(String.class)
.flatMap(responseBodyString -> {
fileUploadResponse.setCode(response.statusCode().value());
fileUploadResponse.setMessage(responseBodyString);
return Mono.just(fileUploadResponse);
});
}
}).onErrorResume(throwable -> {
fileUploadResponse.setCode(HttpStatus.INTERNAL_SERVER_ERROR.value());
fileUploadResponse.setMessage(throwable.getMessage());
System.out.println("uploadLog Error: "+ throwable.getMessage());
return Mono.just(fileUploadResponse);
});
}
}
`
When executing, I receive below error:
<?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; the Credential is mal-formed; expecting "<YOUR-AKID>/YYYYMMDD/REGION/SERVICE/aws4_request".</Message><RequestId>xxxxxx</RequestId><HostId>xxxxxx</HostId></Error>
Presigned URL i use has below like format:
https://xxx-xx-xxx-xx.s3.ap-southeast-1.amazonaws.com/PG00009912/PENDING/REQ_PG00009912_11112022.csv?X-Amz-Security-Token=dzEIz%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDLAplr5Vvi%2F0mZqIMSLABMetINeSCoOWcMxBPkBQN44hB1gyNBFfSPoAOLe0FY8lhBgsBis%2F7oRp3WU8H6qDUpI%2BsxCZEIFMWTkoP%2FIZP%2FAUG5ib8qAIx3gCuVtFLLx77kxdclqm8PhC6OlkYZ5fBrlUl7hHvY9w8hdFAWvRByaqsNKGWiiwwhaMvWZNI65O%2BoWZ84ESwBQCZxbp3e4TdWvUEuB%2FnVpyoMbUfugGxVK2rG58rgJpUKK5E9rv8ZTzIkUqjH7Ql5ZL%2FCt0%2FTa%2Fbj98617MKMk2bPu7dcLZpurNJ75J4AIH2a5%2Bql5tk7FB2f8uwIad3ZskiV5rtz%2FmyoqxFxc5eaaC6YbXDC06eHcMirg81U15XXqFgM2Qi%2FFjx3yutm0E59aPHSEz1K0ucfe2%2FUYLYbB57V%2FgFD8MO8pwHKmLDC0hXVFhXny8nnpPq5PBkqzZomk%2BBOiwBnkP5H98tRXRshwhqnmIMh80xyEy6%2FfCLBRSmJEMrzTpQqDpi%2B1%2Fg7wnOK8Gpv2iyLIU%2FBTD1%2FwgQ716qVsDjH8eTyPga513G6uZ94Kp4pKr1M1E1wsh2IbuvfMlPQ%2FOnL7zA07jZtNxWe%2B538Gs5%2FMwtz9fM%2FoEkeNolx96k3cgDCyQCiD8HU%2FKJk8UYUttSv8CBwkI3PwAW1Sdty0O2L0dGGyFFwwtu6wql8%2BJYxkAZOdZCxDDg2YoaeayVrjYq9mp9yZ%2BxJR1OsVuwmfRkQZcgkUPj7MnCGdgKN1KJAiNk6uVs%2FpSRxEWtSeJlhU806%2FACCiznbObBjIrEwg1WuV%2BkReFAoD86rDec%2BE%2BKmhjb2ahFqHqtY9V4YAYF82RtEzUQld6MQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221110T101307Z&X-Amz-SignedHeaders=host&X-Amz-Expires=26212&X-Amz-Credential=H2DP6FQ%2F20221110%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Signature=787c80356d749c9e0e1231ceb75d64df77f1cf40830146dfa
I'm using Java 1.8 on Windows platform and I don't set any headers in my webclient .
Update (Solution)
Issue was due to webclient has further encoded my presigned url which is unnecessary. Fixed by using .uri(URI.create(preSignedUrl)) instead of just .uri(preSignedUrl).
Related
In one of our service I tried to configure AWS signing in Spring data Reactive Elasticsearch configuration.
Spring provides the configuring the webclient through webclientClientConfigurer
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo("localhost:9200")
.usingSsl()
.withWebClientConfigurer(
webClient -> {
return webClient.mutate().filter(new AwsSigningInterceptor()).build();
})
. // ... other options to configure if required
.build();
through which we can configure to sign the requests but however AWS signing it requires url, queryparams, headers and request body(in case of POST,POST) to generate the signed headers.
Using this I created a simple exchange filter function to sign the request but in this function I was not able to access the request body and use it.
Below is the Filter function i was trying to use
#Component
public class AwsSigningInterceptor implements ExchangeFilterFunction
{
private final AwsHeaderSigner awsHeaderSigner;
public AwsSigningInterceptor(AwsHeaderSigner awsHeaderSigner)
{
this.awsHeaderSigner = awsHeaderSigner;
}
#Override
public Mono<ClientResponse> filter(ClientRequest request, ExchangeFunction next)
{
Map<String, List<String>> signingHeaders = awsHeaderSigner.createSigningHeaders(request, new byte[]{}, "es", "us-west-2"); // should pass request body bytes in place of new byte[]{}
ClientRequest.Builder requestBuilder = ClientRequest.from(request);
signingHeaders.forEach((key, value) -> requestBuilder.header(key, value.toArray(new String[0])));
return next.exchange(requestBuilder.build());
}
}
I also tried to access the request body inside ExchangeFilterFunction using below approach but once i get the request body using below approach.
ClientRequest.from(newRequest.build())
.body(
(outputMessage, context) -> {
ClientHttpRequestDecorator loggingOutputMessage =
new ClientHttpRequestDecorator(outputMessage) {
#Override
public Mono<Void> writeWith(Publisher<? extends DataBuffer> body) {
log.info("Inside write with method");
body =
DataBufferUtils.join(body)
.map(
content -> {
// Log request body using
// 'content.toString(StandardCharsets.UTF_8)'
String requestBody =
content.toString(StandardCharsets.UTF_8);
Map<String, Object> signedHeaders =
awsSigner.getSignedHeaders(
request.url().getPath(),
request.method().name(),
multimap,
requestHeadersMap,
Optional.of(
requestBody.getBytes(StandardCharsets.UTF_8)));
log.info("Signed Headers generated:{}", signedHeaders);
signedHeaders.forEach(
(key, value) -> {
newRequest.header(key, value.toString());
});
return content;
});
log.info("Before returning the body");
return super.writeWith(body);
}
#Override
public Mono<Void>
setComplete() { // This is for requests with no body (e.g. GET).
Map<String, Object> signedHeaders =
awsSigner.getSignedHeaders(
request.url().getPath(),
request.method().name(),
multimap,
requestHeadersMap,
Optional.of("".getBytes(StandardCharsets.UTF_8)));
log.info("Signed Headers generated:{}", signedHeaders);
signedHeaders.forEach(
(key, value) -> {
newRequest.header(key, value.toString());
});
return super.setComplete();
}
};
return originalBodyInserter.insert(loggingOutputMessage, context);
})
.build();
But with above approach I was not able to change the request headers as adding headers throws UnsupportedOperationException inside writewith method.
Has anyone used the spring data reactive elastic search and configured to sign with AWS signed headers?
Any help would be highly appreciated.
I have a spring boot application that uses Retrofit to make requests to a secured server.
My endpoints:
public interface ServiceAPI {
#GET("/v1/isrcResource/{isrc}/summary")
Call<ResourceSummary> getResourceSummaryByIsrc(#Path("isrc") String isrc);
}
public interface TokenServiceAPI {
#FormUrlEncoded
#POST("/bbcb6b2f-8c7c-4e24-86e4-6c36fed00b78/oauth2/v2.0/token")
Call<Token> obtainToken(#Field("client_id") String clientId,
#Field("scope") String scope,
#Field("client_secret") String clientSecret,
#Field("grant_type") String grantType);
}
Configuration class:
#Bean
Retrofit tokenAPIFactory(#Value("${some.token.url}") String tokenUrl) {
Retrofit.Builder builder = new Retrofit.Builder()
.baseUrl(tokenUrl)
.addConverterFactory(JacksonConverterFactory.create());
return builder.build();
}
#Bean
Retrofit serviceAPIFactory(#Value("${some.service.url}") String serviceUrl, TokenServiceAPI tokenAPI) {
OkHttpClient okHttpClient = new OkHttpClient.Builder()
.addInterceptor(new ServiceInterceptor(clientId, scope, clientSecret, grantType, apiKey, tokenAPI))
.build();
Retrofit.Builder builder = new Retrofit.Builder()
.baseUrl(repertoireUrl)
.client(okHttpClient)
.addConverterFactory(JacksonConverterFactory.create());
return builder.build();
}
Interceptor to add the Authorization header to every request
public class ServiceInterceptor implements Interceptor {
public ServiceInterceptor(String clientId,
String scope,
String clientSecret,
String grantType,
String apiKey,
TokenServiceAPI tokenAPI) {
this.clientId = clientId;
this.scope = scope;
this.clientSecret = clientSecret;
this.grantType = grantType;
this.apiKey = apiKey;
this.tokenAPI = tokenAPI;
}
#Override
public Response intercept(Chain chain) throws IOException {
Request newRequest = chain.request().newBuilder()
.addHeader(AUTHORIZATION_HEADER, getToken())
.addHeader(API_KEY_HEADER, this.apiKey)
.build();
return chain.proceed(newRequest);
}
private String getToken() throws IOException {
retrofit2.Response<Token> tokenResponse = repertoireTokenAPI.obtainToken(clientId, scope, clientSecret, grantType).execute();
String accessToken = "Bearer " + tokenAPI.body().getAccessToken();
return accessToken;
}
}
This is working as expected, the problem is that the token is being requested for every request rather than using the existing valid one. How can one store the token somewhere and re-use it? I was wondering if Retrofit had a built-in solution.
a possible option with caching:
add caffeiene
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
</dependency>
add #Cacheable("your-token-cache-name") on the method returning the token, looks like getToken above
add max cache size and expiration configuration in application.yml
e.g. 500 entries and 10 minutes for configuration below
spring.cache.cache-names=your-token-cache-name
spring.cache.caffeine.spec=maximumSize=500,expireAfterAccess=600s
example from: https://www.javadevjournal.com/spring-boot/spring-boot-with-caffeine-cache/
I have a method that fetches a PDF from another web service, but we have to fetch a cross reference ID before we can make the call:
PdfService.groovy
#Service
class PdfService {
#Autowired
WebClient webClient
#Autowired
CrossRefService crossRefService
InputStreamResource getPdf(String userId, String pdfId) {
def pdf = return crossRefService
.getCrossRefId(userId)
.flatMapMany(crossRefResponse -> {
return webClient
.get()
.uri("https://some-url/${pdfId}.pdf", {
it.queryParam("crossRefId", crossRefResponse.id)
it.build(pdfId)
})
.accept(MediaType.APPLICATION_PDF)
.retrieve()
.bodyToFlux(DataBuffer)
})
convertDataBufferToInputStreamResource(pdf)
}
// https://manhtai.github.io/posts/flux-databuffer-to-inputstream/
InputStreamResource getInputStreamFromFluxDataBuffer(Flux<DataBuffer> data) throws IOException {
PipedOutputStream osPipe = new PipedOutputStream();
PipedInputStream isPipe = new PipedInputStream(osPipe);
DataBufferUtils.write(data, osPipe)
.subscribeOn(Schedulers.elastic())
.doOnComplete(() -> {
try {
osPipe.close();
} catch (IOException ignored) {
}
})
.subscribe(DataBufferUtils.releaseConsumer());
new InputStreamResource(isPipe);
}
}
PdfController.groovy
#RestController
#RequestMapping("/pdf/{pdfId}.pdf")
class PdfController {
#Autowired
PdfService service
#GetMapping
Mono<ResponseEntity<Resource>> getPdf(#AuthenticationPrincipal Jwt jwt, #PathVariable String pdfId) {
def pdf = service.getPdf(jwt.claims.userId, pdfId)
Mono.just(ResponseEntity.ok().body(pdf))
}
}
When I run my service class as an integration test, everything work fine, and the InputStreamResource has a content length of 240,000. However, when I try to make this same call from the controller, it seems as if the internal cross reference call is never made.
What is the correct way to place an InputStreamResource into a Publisher? Or is it even needed?
So im trying to figure out how to write consumer contracts for the following class. I have written junit tests fine using mockwebserver.
However for pact testing im struggling and cant seem to see how you get the weblient to use the response from server, all the examples tend to be for resttemplate.
public class OrdersGateway {
public static final String PATH = "/orders";
private final WebClient webClient;
#Autowired
public OrdersGateway(String baseURL) {
this.webClient = WebClient.builder()
.baseUrl(baseURL)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.ALL_VALUE)
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.build();
}
#Override
public Orderresponse findOrders() {
return this.webClient
.post()
.uri(PATH)
.httpRequest(httpRequest -> {
HttpClientRequest reactorRequest = httpRequest.getNativeRequest();
reactorRequest.responseTimeout(Duration.ofSeconds(4));
})
.exchangeToMono(response())
.block();
}
private Function<ClientResponse, Mono<OrderResponse>> response() {
return result -> {
if (result.statusCode().equals(HttpStatus.OK)) {
return result.bodyToMono(OrderResponse.class);
} else {
String exception = String.format("error", result.statusCode());
return Mono.error(new IllegalStateException(exception));
}
};
}
}
Its the #test method for verification, im not sure how to create that. I cant see how the pact-mock-server can intercept the webcleint call.
There might be an assumption that Pact automatically intercepts requests - this is not the case.
So when you write a Pact unit test, you need to explicitly configure your API client to communicate to the Pact mock service, not the real thing.
Using this example as a basis, your test might look like this:
#ExtendWith(PactConsumerTestExt.class)
#PactTestFor(providerName = "orders-gateway")
public class OrdersPactTest {
#Pact(consumer="orders-provider")
public RequestResponsePact findOrders(PactDslWithProvider builder) {
PactDslJsonBody body = PactDslJsonArray.arrayEachLike()
.uuid("id", "5cc989d0-d800-434c-b4bb-b1268499e850")
.stringType("status", "STATUS")
.decimalType("amount", 100.0)
.closeObject();
return builder
.given("orders exist")
.uponReceiving("a request to find orders")
.path("/orders")
.method("GET")
.willRespondWith()
.status(200)
.body(body)
.toPact();
}
#PactTestFor(pactMethod = "findOrders")
#Test
public void findOrders(MockServer mockServer) throws IOException {
OrdersGateway orders = new OrdersGateway(mockServer.getUrl()).findOrders();
// do some assertions
}
}
I'm trying to figure out how to log exceptions from the webclient, whatever the error status code that is returned from the api that gets called.
I've seen the following implementation:
.onStatus(status -> status.value() != HttpStatus.OK.value(),
rs -> rs.bodyToMono(String.class).map(body -> new IOException(String.format(
"Response HTTP code is different from 200: %s, body: '%s'", rs.statusCode(), body))))
Another example I've seen uses a filter. I guess this filter could be used to log errors as well, aside from requests like in this example:
public MyClient(WebClient.Builder webClientBuilder) {
webClient = webClientBuilder // you can also just use WebClient.builder()
.baseUrl("https://httpbin.org")
.filter(logRequest()) // here is the magic
.build();
}
But are we serious that there is no dedicated exception handler to this thing?
Found it.
bodyToMono throws a WebClientException if the status code is 4xx (client error) or 5xx (Server error).
Full implementation of the service:
#Service
public class FacebookService {
private static final Logger LOG = LoggerFactory.getLogger(FacebookService.class);
private static final String URL_DEBUG = "https://graph.facebook.com/debug_token";
private WebClient webClient;
public FacebookService() {
webClient = WebClient.builder()
.filter(logRequest())
.build();
}
public Mono<DebugTokenResponse> verifyFbAccessToken(String fbAccessToken, String fbAppToken) {
LOG.info("verifyFacebookToken for " + String.format("fbAccessToken: %s and fbAppToken: %s", fbAccessToken, fbAppToken));
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(URL_DEBUG)
.queryParam("input_token", fbAccessToken)
.queryParam("access_token", fbAppToken);
return this.webClient.get()
.uri(builder.toUriString())
.retrieve()
.bodyToMono(DebugTokenResponse.class);
}
private static ExchangeFilterFunction logRequest() {
return ExchangeFilterFunction.ofRequestProcessor(clientRequest -> {
LOG.info("Request: {} {}", clientRequest.method(), clientRequest.url());
clientRequest.headers().forEach((name, values) -> values.forEach(value -> LOG.info("{}={}", name, value)));
return Mono.just(clientRequest);
});
}
#ExceptionHandler(WebClientResponseException.class)
public ResponseEntity<String> handleWebClientResponseException(WebClientResponseException ex) {
LOG.error("Error from WebClient - Status {}, Body {}", ex.getRawStatusCode(), ex.getResponseBodyAsString(), ex);
return ResponseEntity.status(ex.getRawStatusCode()).body(ex.getResponseBodyAsString());
}
}