The below code is trying downloaded a single file from within a folder on S3. It is using the message handler below.
NOTE: The code below works for a file that is placed within the root directory of the S3 bucket. It does not work for a file placed in a folder.
#Bean
#ServiceActivator(inputChannel = "s3Channel")
public MessageHandler s3Handler() {
ExpressionParser PARSER = new SpelExpressionParser();
S3MessageHandler s3MessageHandler = new S3MessageHandler(amazonS3(), "test.queue", true);
s3MessageHandler.setOutputChannel(this.channel);
s3MessageHandler.setCommand(S3MessageHandler.Command.DOWNLOAD);
s3MessageHandler.setKeyExpression(PARSER.parseExpression("payload instanceof T(java.io.File) ? payload.name : headers.key"));
s3MessageHandler.setObjectAclExpression(new ValueExpression<>(CannedAccessControlList.PublicReadWrite));
return s3MessageHandler;
}
Using the payload:
Message<?> payload = MessageBuilder.withPayload(new File("test/s3testfile.xml")).build();
this.s3ProcessChannel.send(payload);
The following stack trace occurs downloading a file from with a folder.
org.springframework.messaging.MessageHandlingException: error occurred in message handler [s3MessageHandler]; nested exception is com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found;)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:139) ~[spring-integration-core-4.2.5.RELEASE.jar:na]
at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:135) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.messaging.support.ExecutorSubscribableChannel.sendInternal(ExecutorSubscribableChannel.java:91) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:117) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:104) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at com.rapid7.live.assessment.aws.inbound.channels.SqsAssessmentMessageChannel.send(SqsAssessmentMessageChannel.java:44) ~[classes/:na]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:105) ~[spring-integration-core-4.2.5.RELEASE.jar:na]
at org.springframework.integration.aws.inbound.SqsMessageDrivenChannelAdapter.access$400(SqsMessageDrivenChannelAdapter.java:53) ~[spring-integration-aws-1.0.0.M1.jar:na]
at org.springframework.integration.aws.inbound.SqsMessageDrivenChannelAdapter$IntegrationQueueMessageHandler.handleMessageInternal(SqsMessageDrivenChannelAdapter.java:154) ~[spring-integration-aws-1.0.0.M1.jar:na]
at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMessage(AbstractMethodMessageHandler.java:389) ~[spring-messaging-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.executeMessage(SimpleMessageListenerContainer.java:181) ~[spring-cloud-aws-messaging-1.1.0.RC1.jar:1.1.0.RC1]
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$MessageExecutor.run(SimpleMessageListenerContainer.java:314) ~[spring-cloud-aws-messaging-1.1.0.RC1.jar:1.1.0.RC1]
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$SignalExecutingRunnable.run(SimpleMessageListenerContainer.java:368) [spring-cloud-aws-messaging-1.1.0.RC1.jar:1.1.0.RC1]
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:251) [spring-core-4.2.5.RELEASE.jar:4.2.5.RELEASE]
The file does exist in the correct bucket.
Well, the problem is here that File.getName() returns only file name, without any parent dirs.
Consider change your keyExpression to the :
"payload instanceof T(java.io.File) ? payload.path : headers.key"
Or just don't reply on the File payload but always specify the proper key in the headers.
Related
I have a simple API function to upload a file similar to:
#PostMapping(value = "/documents",
consumes = {MediaType.MULTIPART_FORM_DATA_VALUE})
public Mono<ResponseEntity<String>> uploadDocument(#RequestPart Mono<FilePart> file){
return storeDocumentService
.upload(file)
.map(fileLocation->ResponseEntity.ok(fileLocation))
}
The code works ok and uploads the file. The problem comes when I want to make the response a bit better by returning the link to the uploaded file. For this I want to use HATEOAS 'org.springframework.boot:spring-boot-starter-hateoas'. As soon as I add the dependency 'org.springframework.boot:spring-boot-starter-hateoas' to my 'build.gradle' the endpoint stops working and I get a response:
{
"timestamp": "2023-02-20T04:28:10.620+00:00",
"status": 415,
"error": "Unsupported Media Type",
"path": "/documents"
}
and also I get in the logs:
2023-02-20T05:28:10.618+01:00 WARN 2993 --- [nio-8080-exec-4] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpMediaTypeNotSupportedException: Content-Type 'application/pdf' is not supported]
It is important to point out that I upload a ".pdf" file with a header "Content-Type:multipart/form-data". And most important the only change in the working code and not working code is that i just add the dependency for HATEOAS 'org.springframework.boot:spring-boot-starter-hateoas'
For Uploading File We can easily use the type MultiPartFile , This will handles all the types of files and we can easily retrive the fileInputStream(data) from it.
The following code may helps you!..
#PostMapping("uploadExcelData")
public ResponseEntity<?> uploadExcelData(#RequestParam MultipartFile file) throws IOException {
List<...> dataList = fileHandling.convertFileAsJson(file);
if (!dataList.isEmpty()) {
return ....
} else {
return ResponseEntity.ok("No Records found !!");
}
}
I hope the above code will helps you to handle the File in the Endpoint.
I'm trying to upload a file using slackbot and then posting it as a url_private link in my channel. Then when I click on the url_private link the browser throws the error:
The page isn’t redirecting properly An error occurred during a connection to xxx.slack.com. This problem can sometimes be caused by disabling or refusing to accept cookies.
My code:
client = WebClient(token=os.environ.get("SLACK_BOT_TOKEN"))
logger = logging.getLogger(__name__)
try:
result = client.files_upload(
initial_comment="Here's my file :smile:",
file=file_name,
)
logger.info(result)
os.environ['PRIVATE_URL'] = result["file"]["url_private"]
except SlackApiError as e:
logger.error("Error uploading file: {}".format(e))
def post_message_to_slack(blocks = None):
return requests.post('https://slack.com/api/chat.postMessage', {
'token': os.environ['USER_TOKEN'],
'channel': "#test",
'as_user': 'username',
'blocks': json.dumps(blocks)
}).json()
print(post_message_to_slack(blocks))
This is how I receive result:
The SLACK_BOT_TOKEN and USER_TOKEN has files:write, files:read permissions.
I don't know why I'm not able to access the file after I login to my slack account.
I was able to successfully integrate the elastic search sink in my .net app, now I am trying to setup the FailureCallback option but every time there is an error the exception in LogEvent is null, I can see the actual error in the console but I want to be able to capture this exception in my failure callback function, here is my current configuration:
myLogger = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(elasticSearchUrl))
{
AutoRegisterTemplate = true,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6,
IndexDecider = (#event, dateTimeOffset) =>
{
//some logic here
return $"custom-index";
},
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | EmitEventFailureHandling.RaiseCallback,
FailureCallback = HandleElasticError
})
.Enrich.WithProperty("Environment", Config.Environment)
.Destructure.ByTransforming<ExpandoObject>(JsonConvert.SerializeObject)
.CreateLogger();
Here is my HandleElasticError function:
private void HandleElasticError(LogEvent e)
{
FileLogger.Error(e.Exception, e.MessageTemplate.Text, e.Properties);
}
When I attach the debugger and inspect the LogEvent, exception is null, however in the output window in Visual Studio I can see the actual error:
2020-11-30T20:55:07.7820674Z Caught exception while preforming bulk operation to Elasticsearch: Elasticsearch.Net.ElasticsearchClientException: The underlying connection was closed: An unexpected error occurred on a send.. Call: Status code unknown from: POST /_bulk ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: The handshake failed due to an unexpected packet format.
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
I am not concerned about the error, I know what the issue is, I just want to be able to capture this exception in my failure callback function.
Any ideas?
LogEvent.Exception is the Exception shipped with your call to (for example) logger.Error(myException) - not the exception thrown by failure to write to Elastic Search.
The exception you're trying to catch doesn't look to be exposed by the sink as it just passes the LogEvent to FailureCallback and not exception that caused the failure.
Here's what it could look like were the exception exposed.
you can use:
StreamWriter writer = File.CreateText(...);
Serilog.Debugging.SelfLog.Enable(writer);
I have a simple downstream service for file upload. Sample code
#RestController
#RequestMapping("/file")
public class FileController {
#PostMapping("/upload")
public ResponseEntity<?> uploadFile(#RequestParam("file") MultipartFile file,
#RequestParam(value = "delay", required = false, defaultValue = "0") int delay) throws Exception {
System.out.println(String.join(System.getProperty("line.separator"),
"File Name => " + file.getOriginalFilename(),
"File Size => " + file.getSize() + "bytes",
"File Content Type => " + file.getContentType()));
TimeUnit.MILLISECONDS.sleep(delay);
return ResponseEntity.ok(file.getName() + " uploaded");
}
}
and a CustomExceptionHandler that returns BAD_REQUEST if there is a MultipartException:
#Configuration
#ControllerAdvice
public class CustomExceptionHandler {
#ExceptionHandler(MultipartException.class)
public ResponseEntity<String> handleMultipartException(MultipartException ex) {
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(ex.getMessage());
}
}
The size limit is 10MB in application.yml:
spring:
servlet:
multipart:
max-file-size: 10MB
max-request-size: 10MB
If I upload a large file, it gives me a a 400 status as expected
When I try to hit the same via spring cloud gateway I get the following result:
and the logs shows following:
2019-11-08 00:36:10.797 ERROR 21904 --- [ctor-http-nio-2] a.w.r.e.AbstractErrorWebExceptionHandler : [86e57f7e] 500 Server Error for HTTP POST "/product-service/file/upload"
reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body
Note that the gateway is configured to take in large file size with RequestSize filter set globally to take way more than 10MB.
How can I get the same response code as given by the downstream service?
Also, I check with traditional Zuul, and i get a 500 error too.
For the gateway, for this particular case I know we can use the RequestSize filter and now the gateway will return the error code, but then we have to identify all the routes that expect this beforehand.
Also, other validation in the API, like authorization, etc will have the same the same issue. The response code produced because of these validations will not propagate up.
Sample code spring-cloud-gateway/product-service/eureka - https://github.com/dhananjay12/spring-cloud/tree/master/spring-routing
can you try to go through a non limitation of the volume of the file directly to without going through the getway? try the value -1 for the properties :
properties file of the MS where you want to upload the file
spring.servlet.multipart.max-file-size =-1
spring.servlet.multipart.max-request-size =-1
if it good, it may give a problem with the zuul proxy's ribbon socket size, there are properties informed for this type of situation, the following:
Properties file of the getway :
ribbon.eager-load.enabled=true
hystrix.command.default.execution.timeout.enabled=false
hystrix.command.default.execution.isolation.strategy=THREAD
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=3999996
ribbon.ConnectTimeout=999999
ribbon.ReadTimeout=999999
ribbon.SocketTimeout=999999
zuul.host.socket-timeout-millis=999999
zuul.host.connect-timeout-millis=999999
zuul.sensitiveHeaders=Cookie,Set-Cookie
I have setup an application server using springboot which allows upload of multiple files.
I have configured the application.properties using the following:
spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=20MB
When running a request using postman,
I am uploading 3 files under the key called "files"
File-1 is 11MB
File-2 is 1MB
File-3 is 1MB
The request fails with the following error:
"message": "Maximum upload size exceeded; nested exception is java.lang.IllegalStateException: org.apache.tomcat.util.http.fileupload.FileUploadBase$FileSizeLimitExceededException: The field files exceeds its maximum permitted size of 10485760 bytes."
I understand this is from providing size limits for the server.
But Is there a way to handle the exception such that only 1 file is excluded from upload, and the others are allowed.
Tried with exception handler to fetch request.
But this handler is applied to the overall request causing all the files to be skipped.
#PostMapping("/file-upload-service/file-uploader")
public List<UploadFileResponse> uploadMultipleFiles(#RequestParam String destination, #RequestParam("files") MultipartFile[] files) throws FileStorageException {
logger.info("Uploading multiple files...");
Date uploadDate = new Date();
logger.info("Upload operation started at {}", uploadDate);
return Arrays.asList(files)
.stream()
.map(file -> uploadFile(destination, file))
.collect(Collectors.toList());
}
}