Some extra characters are added at the end of the file after uploading the file into storage account. And there is no issue with 1.33gb file, observed the size difference for 2.22gb file. Below is the code snippet and pom.xml details.
how to resolve it? let me know any details are needed.
Code:
private boolean upload(final MultipartFile file) throws IOException {
BlobClientBuilder blobClientBuilder = new BlobClientBuilder();
blobClientBuilder.endpoint(STORAGE_URL).connectionString(storageConnectionString); blobClientBuilder.containerName(CONTAINER_NAME);
BlobClient blobClient = blobClientBuilder.blobName(file.getOriginalFilename()).buildClient();
blobClient.upload(file.getInputStream(), file.getSize());
boolean uploadStatus = blobClient.exists();
pom.xml:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.2</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>1.18.0</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-blob</artifactId>
<version>12.12.0</version>
<exclusions>
<exclusion>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/io.projectreactor/reactor-core -->
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>3.4.8</version>
<!--$NO-MVN-MAN-VER$ -->
<!-- Please don't remove/degrade the version, possible for compatibility
issues -->
</dependency>
<!-- https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty -->
<dependency>
<groupId>io.projectreactor.netty</groupId>
<artifactId>reactor-netty</artifactId>
<version>1.0.9</version>
<!--$NO-MVN-MAN-VER$ -->
<!-- Please don't remove/degrade the version, possible for compatibility
issues -->
</dependency>
1.33gb file uploaded correctly but 2.22gb shows some extra characters which leads to increase the size of the file in bytes
Instead of uploading large file directly upload them in zip file or chucks
Try with this code
public static void uploadFilesByChunk() {
String connString = "<conn str>";
String containerName = "<container name>";
String blobName = "UploadOne.zip";
String filePath = "D:/temp/" + blobName;
BlobServiceClient client = new BlobServiceClientBuilder().connectionString(connString).buildClient();
BlobClient blobClient = client.getBlobContainerClient(containerName).getBlobClient(blobName);
long blockSize = 2 * 1024 * 1024; //2MB
ParallelTransferOptions parallelTransferOptions = new ParallelTransferOptions()
.setBlockSizeLong(blockSize).setMaxConcurrency(2)
.setProgressReceiver(new ProgressReceiver() {
#Override
public void reportProgress(long bytesTransferred) {
System.out.println("uploaded:" + bytesTransferred);
}
});
BlobHttpHeaders headers = new BlobHttpHeaders().setContentLanguage("en-US").setContentType("binary");
blobClient.uploadFromFile(filePath, parallelTransferOptions, headers, null, AccessTier.HOT,
new BlobRequestConditions(), Duration.ofMinutes(30));
}
For more details refer this SO Thread
Thanks #ShrutiJoshi-MT for your code snippet.
I am not sure why it is working with 'uploadFromFile' method and having issue with 'upload' method of BlobClient. Below is the final code I am using, it is working for different file extensions. Anyone finds bug or having suggestions for below code please let me know, it helps me a lot.
First copying Multipartfile to local file and then providing the path.
public boolean uploadWithFile(final MultipartFile multipartFile) throws Exception {
logger.info("uploadWithFile started");
File file = null;
try {
String fileName = multipartFile.getOriginalFilename();
file = new File(fileName);
logger.info("uploadWithFile fileName: {}", fileName);
Path path = Paths.get(fileName);
logger.debug("Copying from MultipartFile to file");
try (InputStream inputStream = multipartFile.getInputStream()) {
Files.copy(inputStream, path, StandardCopyOption.REPLACE_EXISTING);
}
logger.debug("Copied from MultipartFile to file");
String filePath = file.getPath();
logger.debug("Copied file name: {}", file.getName());
logger.debug("Copied file Path: {}", filePath);
logger.debug("Copied file length: {}", file.length());
String containerName = "temp";
String storageConnectionString = "<primarykey> or <secondarykey>";
BlobClientBuilder blobClientBuilder = new BlobClientBuilder();
blobClientBuilder.endpoint(STORAGE_URL).connectionString(storageConnectionString);
blobClientBuilder.containerName(containerName);
BlobClient blobClient = blobClientBuilder.blobName(fileName).buildClient();
logger.debug("uploading to storage account");
blobClient.uploadFromFile(filePath);
logger.debug("uploaded to storage account");
boolean uploadStatus = blobClient.exists();
logger.debug("uploaded status : {}", uploadStatus);
logger.info("uploadWithFile ended");
return uploadStatus;
} catch (Exception exception) {
logger.error("uploadWithFile upload failed: {}", exception);
throw exception;
} finally {
if (Objects.nonNull(file) && file.exists()) {
logger.debug("delete file: {}", file.getName());
file.delete();
logger.debug("deleted file: {}", file.getName());
}
}
}```
Related
Our Front End using PKCE flow and fetches a access Token. As per the old implementation (microsoft-graph#2.8.1 version) this below snippet gets a Graph Client using an existing access token. Now I cannot get the same working in the newer MS Graph Java SDK.
IGraphServiceClient client = GraphServiceClient.builder()
.authenticationProvider( request -> request.addHeader("Authorization", "Bearer " + tokenAuthentication.getToken().getTokenValue()) )
.buildClient();
Dependencies I have added to my project
<dependency>
<!-- Include the sdk as a dependency -->
<groupId>com.microsoft.graph</groupId>
<artifactId>microsoft-graph</artifactId>
<version>5.13.0</version>
</dependency>
<dependency>
<!-- This dependency is only needed if you are using the TokenCrendentialAuthProvider -->
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.2.5</version>
</dependency>
finally got it working.. see below snippet..
IAuthenticationProvider authProvider = new IAuthenticationProvider() {
#Override
public CompletableFuture<String> getAuthorizationTokenAsync(URL requestUrl) {
CompletableFuture<String> future = new CompletableFuture<>();
future.complete(yourToken);
return future;
}
};
GraphServiceClient<Request> graphClient = GraphServiceClient
.builder()
.authenticationProvider(authProvider)
.buildClient();
return graphClient.me().buildRequest().get();
I am try to upload the file using my spring boot API. The function is working fine when I am using small file (less than 1 MB), but when I upload large file it gives me an exception. I am using embedded Tomcat server.
Maximum upload size exceeded;
nested exception is java.lang.IllegalStateException: org.apache.tomcat.util.http.fileupload.impl.FileSizeLimitExceededException: The field file exceeds its maximum permitted size of 1048576 bytes.
I have tried the following code in my files but every time I am getting the error
1. application.property
server.tomcat.max-swallow-size=100MB
server.tomcat.max-http-post-size=100MB
spring.servlet.multipart.enabled=true
spring.servlet.multipart.fileSizeThreshold=100MB
spring.servlet.multipart.max-file-size=100MB
spring.servlet.multipart.max-request-size=100MB
I have also tried
spring.servlet.multipart.maxFileSize=100MB
spring.servlet.multipart.maxRequestSize=100MB
2. The belove is my file uploading code
public RestDTO uploadFile(MultipartFile file, String subPath) {
if (file.isEmpty()) {
return new RestFailure("Failed to store empty file");
}
try {
String fileName = new Date().getTime() + "_" + file.getOriginalFilename();
String filePath = uploadPath + subPath + fileName;
if (Objects.equals(file.getOriginalFilename(), "blob")) {
filePath += ".png";
fileName += ".png";
}
File uploadDir = new File(uploadPath + subPath);
if (!uploadDir.exists()) {
uploadDir.mkdirs();
}
FileOutputStream output = new FileOutputStream(filePath);
output.write(file.getBytes());
LOGGER.info("File path : " + filePath);
MediaInfoDTO mediaInfoDTO = getThumbnailFromVideo(subPath, fileName);
String convertedFileName = convertVideoToMP4(subPath, fileName);
System.out.println("---------------->" + convertedFileName);
return new RestData<>(new MediaDetailDTO(mediaInfoDTO.getMediaPath(), convertedFileName,
mediaInfoDTO.getMediaType(), mediaInfoDTO.getMediaCodec(), mediaInfoDTO.getWidth(),
mediaInfoDTO.getHeight(), mediaInfoDTO.getDuration()));
} catch (IOException e) {
LOGGER.info("Can't upload file: " + e.getMessage());
return new RestFailure("Failed to store empty file");
}
}
but every time I got the same exception.
Apart from comment might I suggest creating a #Bean for Factory MultipartConfigurationElement
This basically should override other restrictions if you have any from TomCat side.
#Bean
public MultipartConfigElement multipartConfigElement() {
MultipartConfigFactory factory = new MultipartConfigFactory();
factory.setMaxFileSize(DataSize.ofBytes(100000000L));
factory.setMaxRequestSize(DataSize.ofBytes(100000000L));
return factory.createMultipartConfig();
}
Here DataSize is of type org.springframework.util.unit.DataSize
Reference https://github.com/spring-projects/spring-boot/issues/11284
Another issue I suspect could be from TomCat maxSwallowSize see Baeldung's point #5 if above does not work.
https://www.baeldung.com/spring-maxuploadsizeexceeded
After reviewing many examples and after several tests with no results. I have managed to solve the problem with the following configuration:
Add to pom the follows dependencies:
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
Remove from yml:
sprint:
servlet:
multipart:
enabled: true
file-size-threshold: 2KB
max-file-size: 10MB
max-request-size: 10MB
Add to yml:
server:
tomcat:
max-swallow-size: -1
max-http-form-post-size: -1
And last but not least:
#Bean
public MultipartResolver multipartResolver() {
CommonsMultipartResolver resolver
= new CommonsMultipartResolver();
resolver.setDefaultEncoding(StandardCharsets.UTF_8.displayName());
resolver.setMaxUploadSize(52428800L); //50MB
resolver.setMaxUploadSizePerFile(52428800L); //50MB
return resolver;
}
#ExceptionHandler(MaxUploadSizeExceededException.class)
public ResponseEntity<Object> handleFileUploadError(MaxUploadSizeExceededException ex) {
return ResponseEntity.status(EXPECTATION_FAILED).body(
CustomResponse.builder()
.status(Status.ERROR)
.message(ex.getMessage())
.build());
}
// Where CustomResponse class is in my case:
/**
* The UploadResponse class
* <p>
* Contain the response body
*/
#Getter
#Builder(toBuilder = true)
#AllArgsConstructor
#JsonInclude(JsonInclude.Include.NON_NULL)
public class CustomResponse {
/**
* The status
*/
private final Status status;
/**
* The message
*/
private final String message;
/**
* The errors
*/
private final Set<String> errors;
}
I wrote some code to create a REST api to handle customer's infomation and a file from them,I want to save the file they upload to my local disk on server.My code is as below
#RequestMapping(value = "/alg", method = RequestMethod.POST)
public String postUsersNewAlg(#RequestParam(value = "file", required = true) CommonsMultipartFile jarfile,
#RequestParam(value = "username", required = true) String userName,
#RequestParam(value = "output", required = true) String output) {
//other code handles string infos etc.
try {
//get directory path that to save file
String filePath = PathUtil.getOffAlgJarFilePathRoot();
//get file path
String filePathWhole = null;
if (!filePath.endsWith(SeparatorUtils.getFileSeparator())) {
filePathWhole = filePath + SeparatorUtils.getFileSeparator() + algImpl.getOriginalFilename();
}
else {
filePathWhole = filePath + algImpl.getOriginalFilename();
}
FileUtil.copyFile(jarfile, filePath,filePathWhole);
}
catch(Exception e){
e.printStackTrace();
return e.getMessage();
}
}
Problem is a java.lang.NoClassDefFoundError:org/apache/commons/fileupload/FileUploadException] with root cause.So how can I do this?Thanks in advance.
Have you tried this one on pom.xml or add commons-fileupload.jar:
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.2.1</version>
</dependency>
The static method in my class has to read a property file located in another module.
public class Util
{
private static void readProp()
{
Properties prop = new Properties();
String fileName = "/appconfig.properties"; //File in another module
InputStream inputStream = null;
try
{
inputStream = ClassLoader.getSystemResourceAsStream(propFileName);
if (inputStream != null)
{
prop.load(inputStream);
}
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
The Util method is in module1 while the appconfig.properties is in module2.
The absolute paths for the two is
Util.java :
/Users/user1/IdeaProjects/myProject/module1/src/main/java/com/microsoft/e3/cx/service/windows/search/util/Util.java
appconfig.properties:
/Users/user1/IdeaProjects/myProject/module2/appconfig/base/appconfig.properties
The pom of module2 has module1 as a dependency
<dependency>
<groupId>microsoft.module1</groupId>
<artifactId>module1</artifactId>
<version>${project.version}</version>
</dependency>
My inputStream always comes out to be null. I suppose it is because it is unable to locate this file. Any suggestions on how to fix this?
Try,
Util.getClass().getResourceAsStream("appconfig.properties");
I'm creating a controller to register an user that has logged in using oauth2 but whenever I try to get the connection using ProviderSignInUtils.getConnection(request) it says the function does not exist.
This is my controller:
import org.springframework.social.connect.web.ProviderSignInUtils;
#RequestMapping(value = "/register", method = RequestMethod.GET)
public String showRegistrationForm(WebRequest request, Model model) {
Connection<?> connection = ProviderSignInUtils.getConnection(request);
RegistrationForm registration = createRegistrationDTO(connection);
model.addAttribute("user", registration);
return "user/registrationForm";
}
Those are the maven dependencies:
<dependency>
<groupId>org.springframework.social</groupId>
<artifactId>spring-social-config</artifactId>
<version>1.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.social</groupId>
<artifactId>spring-social-core</artifactId>
<version>1.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.social</groupId>
<artifactId>spring-social-security</artifactId>
<version>1.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.social</groupId>
<artifactId>spring-social-web</artifactId>
<version>1.1.2.RELEASE</version>
</dependency>
ProviderSignInUtils.getConnection was removed in Spring Social 1.1.2, however the documentation wasn't updated to reflect this. The example code at github shows this instead
#Inject
public SignupController(AccountRepository accountRepository,
ConnectionFactoryLocator connectionFactoryLocator,
UsersConnectionRepository connectionRepository) {
this.accountRepository = accountRepository;
this.providerSignInUtils = new ProviderSignInUtils(connectionFactoryLocator, connectionRepository);
}
#RequestMapping(value="/signup", method=RequestMethod.GET)
public SignupForm signupForm(WebRequest request) {
Connection<?> connection = providerSignInUtils.getConnectionFromSession(request);
if (connection != null) {
request.setAttribute("message", new Message(MessageType.INFO, "Your " + StringUtils.capitalize(connection.getKey().getProviderId()) + " account is not associated with a Spring Social Showcase account. If you're new, please sign up."), WebRequest.SCOPE_REQUEST);
return SignupForm.fromProviderUser(connection.fetchUserProfile());
} else {
return new SignupForm();
}
}
You need to create your own local providerSignInUtils so it has access to the connectionFactoryLocator and connectionRepository.