Java S3 upload using Spring RestTemplate - spring-boot

Question

I want to make this call using SpringBoot RestTemplate to upload a file to a S3 bucket: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
PUT /my-image.jpg HTTP/1.1
Host: myBucket.s3.<Region>.amazonaws.com
Date: Wed, 12 Oct 2009 17:50:00 GMT
Authorization: authorization string
Content-Type: text/plain
Content-Length: 11434
x-amz-meta-author: Janet
Expect: 100-continue
[11434 bytes of object data]
and
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.rootUri("")
.additionalInterceptors((request, body, execution) -> {
request.getHeaders().add("Authorization",
"Bearer a0d78d7922f333ee22d75bea53d01hhkjk83f5ac03f11ccd87787");
return execution.execute(request, body);
}).build();
}
I've tried
Resource resource = new ClassPathResource("logback.xml");
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.TEXT_PLAIN);
HttpEntity<byte[]> requestEntity
= new HttpEntity<>(StreamUtils.copyToByteArray(resource.getInputStream()), headers);
Map<String, Object> parameters = new HashMap<>(4);
parameters.put("cors_enabled", true);
parameters.put("acl", "private");
parameters.put("key", "my-key");
parameters.put("Bucket", "parameters.put("Bucket", "https://cloud.linode.com/object-storage/buckets/eu-central-1/my-bucket-2020");");
restTemplate.put("https://api.linode.com/v4/object-storage/buckets", requestEntity, parameters);
but I got
org.springframework.web.client.HttpClientErrorException$MethodNotAllowed: 405 METHOD NOT ALLOWED: [{"errors": [{"reason": "Method Not Allowed"}]}]
also when Getting I have a problem:
MultiValueMap<String, Object> body
= new LinkedMultiValueMap<>();
UriComponentsBuilder builder =
UriComponentsBuilder.fromHttpUrl("https://api.linode.com/v4/object-storage/buckets/eu-central-1/my-bucket-2020/object-url");
builder.queryParam("method", "GET");
builder.queryParam("name", "43f959d9-a11a-4f2cec88fd7e.JPG");
body.add("method", "GET");
body.add("name", "43f959d9-a11a-4f2cec88fd7e.JPG");
HttpHeaders headers = new HttpHeaders();
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
restTemplate.postForEntity(builder.build().encode().toUri(),
requestEntity, LinodeResponse.class);
and the response:
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 BAD REQUEST: [{"errors": [{"reason": "name is required", "field": "name"}, {"reason": "method is required", "field": "method"}]}]
ans when accessing with AWS-SDK I have this error:
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records.

Answer 1

Linode seems to offer an API to generate presigned urls for interact with objects in S3.
To use the API, first, you can create two POJO that represent the request and response we will send and receive from the API so we can use to serialize an deserialize JSON information.
For the request object:
public class LinodeGeneratePresignedUrlRequest {
private String method;
private String name;
#JsonProperty("content_type")
private String contentType;
#JsonProperty("expires_in")
private int expiresIn;
// Getters and setters
}
And for the response:
pubic class LinodeGeneratePresignedUrlResponse {
private String url;
// Getters and setters
}
These objects match the information required by the endpoint.
If you want to create an object in your bucket with the Linode API, you first need to request a presigned URL. Once obtained, you will use this URL to perform the actual operation over the bucket object. The operation is defined by the method parameter passed to the API. Consider the following example:
// Obtain a reference to the RestTemplate instance.
// It should support the interchange of JSON information
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
// Set content type to the one required by the Linode API application/json
headers.setContentType(MediaType.APPLICATION_JSON);
// Set the appropriate credentials for the Linode API
String token = "your token";
headers.set(HttpHeaders.AUTHORIZATION, "Bearer" + token);
// Create the presigned url request
LinodeGeneratePresignedUrlRequest linodeGeneratePresignedUrlRequest =
new LinodeGeneratePresignedUrlRequest();
// Operation to perform when you interact with AWS later
// In this case, PUT because you need to create a new object
linodeGeneratePresignedUrlRequest.setMethod("PUT");
// The object name: can match or not the actual file you want to upload
linodeGeneratePresignedUrlRequest.setName("my-object-name.pdf");
// As you are performing an upload (PUT, POST), indicate the content type of
// the information you are uploading to AWS. It should match the provided later
// when you interact with AWS. For instance, consider that you are uploading a PDF file
linodeGeneratePresignedUrlRequest.setContentType("application/pdf");
// Optionally, you can set the expiration time of the generated presigned url
// By default, an hour (3600 seconds)
// Perform the actual Linode API invocation
HttpEntity<LinodeGeneratePresignedUrlRequest> requestEntity =
new HttpEntity<LinodeGeneratePresignedUrlRequest>(linodeGeneratePresignedUrlRequest, headers);
// The Linode API URL for your cluster and bucket
String linodeApiUrl = "https://api.linode.com/v4/object-storage/buckets/eu-central-1/my-bucket-2020/object-url";
HttpEntity<LinodeGeneratePresignedUrlResponse> responseEntity = restTemplate.exchange(linodeApiUrl, HttpMethod.POST, requestEntity, LinodeGeneratePresignedUrlResponse.class);
// Linde wil provide a response with a property named 'url' corresponding
// to the presigned url that we can use to interact with AWS S3
LinodeGeneratePresignedUrlResponse linodeGeneratePresignedUrlResponse = responseEntity.getBody();
String signedUrl = linodeGeneratePresignedUrlResponse.getUrl();
// Now, send the actual file.
// I am following the example provided in the AWS documentation:
// https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObjectJavaSDK.html adapt for RestTemplate
HttpHeaders headersForS3 = new HttpHeaders();
// You should provide the same content type you indicated previously
headersForS3.set("Content-Type", "application/pdf");
Resource resource = new FileSystemResource("my-object-name.pdf");
HttpEntity<byte[]> requestEntityForS3 =
new HttpEntity<>(
StreamUtils.copyToByteArray(resource.getInputStream()), headersForS3);
// You should use the same HTTP verb as indicated in
// the 'method' parameter before
restTemplate.exchange(signedUrl, HttpMethod.PUT, requestEntityForS3, Void.class);
The process for retrieving the object created is very similar:
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
String token = "your token";
headers.set(HttpHeaders.AUTHORIZATION, "Bearer" + token);
LinodeGeneratePresignedUrlRequest linodeGeneratePresignedUrlRequest =
new LinodeGeneratePresignedUrlRequest();
// Instead of PUT, indicate that you want to retrieve the object
linodeGeneratePresignedUrlRequest.setMethod("GET");
// your object name
linodeGeneratePresignedUrlRequest.setName("my-object-name.pdf");
HttpEntity<LinodeGeneratePresignedUrlRequest> requestEntity =
new HttpEntity<LinodeGeneratePresignedUrlRequest>(linodeGeneratePresignedUrlRequest, headers);
String linodeApiUrl = "https://api.linode.com/v4/object-storage/buckets/eu-central-1/my-bucket-2020/object-url";
HttpEntity<LinodeGeneratePresignedUrlResponse> responseEntity = restTemplate.exchange(linodeApiUrl, HttpMethod.POST, requestEntity, LinodeGeneratePresignedUrlResponse.class);
LinodeGeneratePresignedUrlResponse linodeGeneratePresignedUrlResponse = responseEntity.getBody();
String signedUrl = linodeGeneratePresignedUrlResponse.getUrl();
// Read the object from your bucket
byte[] objectBytes = restTemplate.getForObject(signedUrl, byte[].class);
// And use the information as you need
Files.write(Paths.get("my-object-name.pdf"), objectBytes);
Of course, if Linode provides you the appropriate credentials, you can also use the AWS SDK to interact with S3 directly.

Answer 2

Spring equivalent of the cURL command you've provided can be:
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
String token = "";
headers.set(HttpHeaders.AUTHORIZATION, token);
JSONObject data = new JSONObject();
data.put("cors_enabled", true);
data.put("acl", "private");
HttpEntity<String> requestEntity = new HttpEntity<String>(data.toString(), headers);
String url = "https://api.linode.com/v4/object-storage/buckets/eu-central-1/bonansa15122020/access";
HttpEntity<String> responseEntity = restTemplate.exchange(url, HttpMethod.PUT, requestEntity, String.class);
In your first example, you've haven't provided Authorization header, so you're getting a 401 response. You're aren't using the RestTemplate you'd created using RestTemplateBuilder here.
In your second example, it seems the request body isn't a JSON (you're reading the logback file, so highly unlikely there's a JSON inside it). It seems the Linode API expects a JSON body.

Answer 3

Update:
I believe you can use PUT request as part of POST request to endpoint https://api.linode.com/v4/object-storage/buckets/{clusterId}/{bucket}/object-url
More details here - https://developers-linode.netlify.app/api/v4/object-storage-buckets-cluster-id-bucket-object-url#post
I won't be able to test because I don't have account with linode.
I think another viable solution would be to use aws sdk for s3 to upload files to the linode endpoint.
Here is the simple example -
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/java/example_code/s3/src/main/java/aws/example/s3/PutObject.java
Original:
Based on the linode api documentation the method should be post.
https://www.linode.com/docs/api/object-storage/#object-storage-bucket-create
restTemplate.post("https://api.linode.com/v4/object-storage/buckets", requestEntity, parameters)
Also please review the request body to conform to documentation.