I am tring to get .csv file from a url, when i loop through the links using the "a" tag, the href that i want doesn't exist. how can i get it's href.
` String URL = internalConstant.getMlcuURL();
String ffiListFile = null;
logger.info("Page URl: " + URL);
Document doc;
try {
doc = Jsoup.connect(URL).get();
Elements links = doc.select("a[href]");
for (Element link : links) {
logger.info("Elements aaaa"+ link);
String absHref = link.attr("abs:href");
if (absHref.endsWith(".csv")) {
logger.info(absHref);
ffiListFile = absHref;
break;
}
}
if (ffiListFile.isEmpty()) {
logger.error("file not found");
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "MLCU File not found in website");
}
this.fileDownloadFFIListService.downloadExcelFile(ffiListFile);
} catch (IOException e) {
e.printStackTrace();
logger.error("download error " + e.getMessage());
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Failed to download file");
`
Related
This is re-worded from a previous question (which was probably a bit unclear).
I want to download a text file via FTP from a remote server, read the contents of the text file into a string and then discard the file. I don't need to actually save the file.
I am using the Apache Commons library so I have:
import org.apache.commons.net.ftp.FTPClient;
Can anyone help please, without simply redirecting me to a page with lots of possible answers on?
Not going to do the work for you, but once you have your connection established, you can call retrieveFile and pass it an OutputStream. You can google around and find the rest...
FTPClient ftp = new FTPClient();
...
ByteArrayOutputStream myVar = new ByteArrayOutputStream();
ftp.retrieveFile("remoteFileName.txt", myVar);
ByteArrayOutputStream
retrieveFile
Normally I'd leave a comment asking 'What have you tried?'. But now I'm feeling more generous :-)
Here you go:
private void ftpDownload() {
FTPClient ftp = null;
try {
ftp = new FTPClient();
ftp.connect(mServer);
try {
int reply = ftp.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
throw new Exception("Connect failed: " + ftp.getReplyString());
}
if (!ftp.login(mUser, mPassword)) {
throw new Exception("Login failed: " + ftp.getReplyString());
}
try {
ftp.enterLocalPassiveMode();
if (!ftp.setFileType(FTP.BINARY_FILE_TYPE)) {
Log.e(TAG, "Setting binary file type failed.");
}
transferFile(ftp);
} catch(Exception e) {
handleThrowable(e);
} finally {
if (!ftp.logout()) {
Log.e(TAG, "Logout failed.");
}
}
} catch(Exception e) {
handleThrowable(e);
} finally {
ftp.disconnect();
}
} catch(Exception e) {
handleThrowable(e);
}
}
private void transferFile(FTPClient ftp) throws Exception {
long fileSize = getFileSize(ftp, mFilePath);
InputStream is = retrieveFileStream(ftp, mFilePath);
downloadFile(is, buffer, fileSize);
is.close();
if (!ftp.completePendingCommand()) {
throw new Exception("Pending command failed: " + ftp.getReplyString());
}
}
private InputStream retrieveFileStream(FTPClient ftp, String filePath)
throws Exception {
InputStream is = ftp.retrieveFileStream(filePath);
int reply = ftp.getReplyCode();
if (is == null
|| (!FTPReply.isPositivePreliminary(reply)
&& !FTPReply.isPositiveCompletion(reply))) {
throw new Exception(ftp.getReplyString());
}
return is;
}
private byte[] downloadFile(InputStream is, long fileSize)
throws Exception {
byte[] buffer = new byte[fileSize];
if (is.read(buffer, 0, buffer.length)) == -1) {
return null;
}
return buffer; // <-- Here is your file's contents !!!
}
private long getFileSize(FTPClient ftp, String filePath) throws Exception {
long fileSize = 0;
FTPFile[] files = ftp.listFiles(filePath);
if (files.length == 1 && files[0].isFile()) {
fileSize = files[0].getSize();
}
Log.i(TAG, "File size = " + fileSize);
return fileSize;
}
You can just skip the download to local filesystem part and do:
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(server, port);
ftpClient.login(user, pass);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
InputStream inputStream = ftpClient.retrieveFileStream("/folder/file.dat");
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "Cp1252"));
while(reader.ready()) {
System.out.println(reader.readLine()); // Or whatever
}
inputStream.close();
} catch (IOException ex) {
ex.printStackTrace();
} finally {
try {
if (ftpClient.isConnected()) {
ftpClient.logout();
ftpClient.disconnect();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
Trying to use code in:
http://javasampleapproach.com/spring-framework/spring-data/springdata-mongodb-gridfstemplate-save-retrieve-delete-binary-files-image-text-files
with springboot 1.5.13.
In this code to retrieve an image use GridFSDBFile like this:
GridFSDBFile imageFile = gridOperations.findOne(new Query(Criteria.where("_id").is(imageFileId)));
and later to save in filesystem use this:
imageFile.writeTo
Baeldung has the same code:
http://www.baeldung.com/spring-data-mongodb-gridfs
I have my project in springboot 2.0.2 and this line:
GridFSDBFile imageFile = gridOperations.findOne(new Query(Criteria.where("_id").is(imageFileId)));
throws an error:
Type mismatch: cannot convert from GridFSFile to GridFSDBFile
so, once I have changed to:
GridFSFile imageFile = gridOperations.findOne(new Query(Criteria.where("_id").is(imageFileId)));
no errors, but now I have not available the method:
writeTo
to write to disk.
UPDATE 1:
imageFileId = "5b0bff8b7cc45b32f43b47f4";
GridFSFile imageFile = gridOperations.findOne(new Query(Criteria.where("_id").is(imageFileId)));
try {
File file = new File("c:/JSA/retrieve/" + imageFile.getFilename());
FileOutputStream streamToDownloadTo = new FileOutputStream(file);
//This line doesn't works
gridFSBucket.downloadToStream(imageFile.getId(), streamToDownloadTo);
streamToDownloadTo.close();
} catch (IOException e) {
// handle exception
System.out.println("error: " + e.getMessage());
} catch (Exception e1) {
System.out.println("error1: " + e1.getMessage());
}
UPDATE 2:
try {
File file = new File("c:/JSA/retrieve/" + imageFile.getFilename());
FileOutputStream streamToDownloadTo = new FileOutputStream(file);
System.out.println("imageFile.getId(): " + imageFile.getId());
System.out.println("streamToDownloadTo: " + streamToDownloadTo.toString());
gridFSBucket.downloadToStream(imageFile.getId(), streamToDownloadTo);
streamToDownloadTo.close();
} catch (IOException e) {
// handle exception
System.out.println("error: " + e.getMessage());
} catch (Exception e1) {
System.out.println("error1: " + e1.getMessage());
}
Console
imageFile.getId(): BsonObjectId{value=5b0bff8b7cc45b32f43b47f4}
streamToDownloadTo: java.io.FileOutputStream#3b20c8e2
This line thrown an exception:
gridFSBucket.downloadToStream(imageFile.getId(), streamToDownloadTo);
and return null
Solved
Inject:
#Autowired
MongoGridFsTemplate mongoGridFsTemplate;
GridFSBucket gridFSBucket = GridFSBuckets.create(mongoGridFsTemplate.mongoDbFactory().getDb());
imageFileId = "5b0bff8b7cc45b32f43b47f4";
GridFSFile imageFile = gridOperations.findOne(new Query(Criteria.where("_id").is(new ObjectId(imageFileId))));
try {
File file = new File("c:/JSA/retrieve/" + imageFile.getFilename());
FileOutputStream streamToDownloadTo = new FileOutputStream(file);
gridFSBucket.downloadToStream(imageFile.getId(), streamToDownloadTo);
streamToDownloadTo.close();
} catch (IOException e) {
// handle exception
System.out.println("error: " + e.getMessage());
} catch (Exception e1) {
e1.printStackTrace();
}
I am trying to upload an audio file using the following code on server.Right now, it works perfectly well for image files but not for audios.I think MultipartFile should work with audio files as well.Can any one tell me what is wrong here?
I am getting "The server refused this request because the request entity is in a format not supported by the requested resource for the requested method." error.
Does MultipartFile not work with audio files?If no what is an alternative?
#Transactional
public BaseVO uploadImage(MultipartFile file, long userId){
Map<String, Object> alertParams = new HashMap<String, Object>();
try{
if (!file.isEmpty()) {
Image profileImage = imageRepository.findByUserId(userId);
if(profileImage == null){
profileImage = new Image();
profileImage.setUserId(userId);
}
File dir = null;
String realPath;
if(liveBuild == "true"){
realPath = liveImageUploadRepository;
dir = new File(realPath);
}else{
HttpServletRequest request = ((ServletRequestAttributes)RequestContextHolder.getRequestAttributes()).getRequest();
realPath = request.getSession().getServletContext().getRealPath("/") + devImageUploadRepository;
dir = new File(realPath);
}
if (!dir.exists()) {
dir.mkdirs();
}
String finalFileStorePath = realPath + "/" + userId + ".jpg";
File path = new File(finalFileStorePath);
file.transferTo(path);
profileImage.setImagePath(realPath);
//profileImage.setImageName(userId + ".jpg");
profileImage.setImageName(userId+"");
profileImage.setImageType(".jpg");
imageRepository.save(profileImage);
alertParams.put("id", profileImage.getId());
} else {
return new BaseVO(alertParams, Constants.STATUS_OK, Constants.STATUS_OK_MSG);
}
}catch(HibernateException e){
return new BaseVO(alertParams, Constants.STATUS_ERROR, Constants.STATUS_ERROR_MSG + " " + e.getMessage());
} catch (IllegalStateException e) {
e.printStackTrace();
return new BaseVO(alertParams, Constants.STATUS_ERROR, Constants.STATUS_ERROR_MSG);
} catch (IOException e) {
e.printStackTrace();
return new BaseVO(alertParams, Constants.STATUS_ERROR, Constants.STATUS_ERROR_MSG);
}
return new BaseVO(alertParams, Constants.STATUS_OK, Constants.STATUS_OK_MSG);
}
I am trying to use the SkyDrive API to upload a file. I tried using the below code.GEtAccountInformaiton and GetQuotaInformaiton methods are successfully executed But it always sets this error "The request has already been submitted.” at the end (in UploadISOFileToSkyDriveAsync() method for the field lblMessageBar.Text ).
private async void GetAccountInformations()
{
try
{
LiveOperationResult operationResult = await App.liveConnectClient.GetAsync("me");
var jsonResult = operationResult.Result as dynamic;
string firstName = jsonResult.first_name ?? string.Empty;
string lastName = jsonResult.last_name ?? string.Empty;
lblMessageBar.Text = "Welcome " + firstName + " " + lastName;
GetQuotaInformations();
}
catch (Exception e)
{
lblMessageBar.Text = e.ToString();
}
}
private async void GetQuotaInformations()
{
try
{
LiveOperationResult operationResult = await App.liveConnectClient.GetAsync("me/skydrive/quota");
var jsonResult = operationResult.Result as dynamic;
quota = jsonResult.quota ?? string.Empty;
available = jsonResult.available ?? string.Empty;
lblMessageBar.Text = "Available space in bytes: " + ConvertBytesToGigabytes(available).ToString("#.####") + "GB " + "out of bytes " + ConvertBytesToGigabytes(quota).ToString("#.####") + "GB";
UploadISOFileToSkyDriveAsync();
}
catch (Exception e)
{
lblMessageBar.Text = e.ToString();
}
}
public async void UploadISOFileToSkyDriveAsync()
{
try
{
//http://developer.nokia.com/Community/Wiki/SkyDrive_-_How_to_upload_content_on_Windows_Phone
IsolatedStorageFile fileStorage = IsolatedStorageFile.GetUserStoreForApplication();
StreamWriter Writer = new StreamWriter(new IsolatedStorageFileStream("/shared/transfers/" + Constants.SkyDriveSavedLocationsFileName, FileMode.Append, fileStorage));
//get the data from local database and write to the isolated file and then use the path of this file to saved it to skydrive..
ObservableCollection<SavedLocationsTableEntity> SavedLocations = SavedLocationsTableEntity.GetSavedLocations();
foreach (SavedLocationsTableEntity item in SavedLocations)
{
Writer.WriteLine(UtilityLib.GetGoogleURL(new System.Device.Location.GeoCoordinate(item.SavedLocationLatitude, item.SavedLocationLongitude, item.SavedLocationAltitude)));
}
Writer.Close();
using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
{
fileStream = store.OpenFile("/shared/transfers/" + Constants.SkyDriveSavedLocationsFileName, FileMode.OpenOrCreate, FileAccess.Read);
//strEncryptedFileStream = Encoding.Unicode.GetBytes(fileStream.ToString()).ToString();
if (fileStream.Length == 0)
{
lblMessageBar.Text = "No data to upload to SkyDrive..";
return;
}
fileStream.Close();
}
//remove previous calls
var reqList = BackgroundTransferService.Requests.ToList();
foreach (var req in reqList)
{
if (req.UploadLocation.Equals(new Uri(MyFilePathInIsoStore, UriKind.Relative)))
BackgroundTransferService.Remove(BackgroundTransferService.Find(req.RequestId));
}
//Make a new call to upload
LiveOperationResult res = await App.liveConnectClient.BackgroundUploadAsync("me/skydrive", new Uri("/shared/transfers/" + Constants.SkyDriveSavedLocationsFileName, UriKind.Relative), OverwriteOption.Overwrite);
lblMessageBar.Text = "File " + Constants.SkyDriveSavedLocationsFileName + " uploaded.";
return;
}
catch (Exception ex)
{
lblMessageBar.Text = "Cannot upload to SkyDrive.. " + ex.Message;
return;
}
}
It looks like MyFilePathInIsoStore here:
if (req.UploadLocation.Equals(new Uri(MyFilePathInIsoStore
is not equals "/shared/transfers/" + Constants.SkyDriveSavedLocationsFileName here:
new Uri("/shared/transfers/" + Constants.SkyDriveSavedLocationsFileName, UriKind.Relative)
I'm creating an Android app that should do the following;
Use a form on a https (SSL!) page to login and receive a cookie
Issue httpGET actions to get html
parse that html and show it in a view, list or something.
I've been fooling around with Jsoup, httpUnit and HTMLUnit for quite some time now, but I'm running in to several problems;
A. Login is fine, works.. (I get the website's welcome page) but then, when I issue a GET statement (and include the cookie), I am redirected to the login form. So the response html is not what I expected. (might have something to do with a keepalivestrategy?)
B. InputBuffers are too small to receive entire HTML pages and set them up for parsing.
NB : I do not have control over the webserver
I'm totally new at this, so a tutorial or code snippets would be helpful.
For instance, this is what I use to login to the website :
public int checkLogin() throws Exception {
ArrayList<NameValuePair> data = new ArrayList<NameValuePair>();
data.add(new BasicNameValuePair("userid", getUsername()));
data.add(new BasicNameValuePair("password", getPassword()));
data.add(new BasicNameValuePair("submit_login", "Logmein"));
Log.d(TAG, "Cookie name : " + getCookieName());
Log.d(TAG, "Cookie cont : " + getCookie());
HttpPost request = new HttpPost(BASE_URL);
request.getParams().setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, false);
request.getParams().setParameter("http.protocol.handle-redirects",false);
request.setEntity(new UrlEncodedFormEntity(data, "UTF-8"));
HttpResponse response;
httpsclient.getCookieStore().clear();
List<Cookie> cookies = httpsclient.getCookieStore().getCookies();
Log.d(TAG, "Number of Cookies pre-login : " + cookies.size());
response = httpsclient.execute(request);
cookies = httpsclient.getCookieStore().getCookies();
Log.d(TAG, "Number of Cookies post-login : " + cookies.size());
String html = "";
// Problem : buffer is too small!
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line);
}
in.close();
html = str.toString();
Document doc = Jsoup.parse(html);
Log.v(TAG, "Ik heb nu dit : " + doc.toString());
if (cookies.size() > 0){
storeCookie(cookies.get(0).getName(), cookies.get(0).getValue());
return MensaMobileActivity.REQUEST_SUCCESS;
} else {
return MensaMobileActivity.REQUEST_ERROR;
}
}
You don't handle the SSL certificate at all, that's at least a part of the problem. I struggled starting to learn this recently as well. This block of code will grab the SSL cert from the webpage you're accessing.
try {
URL url = new URL(YOUR_WEBPAGE_HERE);
HttpsURLConnection connect = (HttpsURLConnection)url.openConnection();
connect.connect();
Certificate[] certs = connect.getServerCertificates();
if (certs.length > 0) {
cert = new File("YOUR_PATH_TO_THE_FILE");
//write the certificate obtained to the cert file.
OutputStream os = new FileOutputStream(cert);
os.write(certs[0].getEncoded());
return true;
}
}
catch (SSLPeerUnverifiedException e) {
e.printStackTrace();
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
catch (CertificateEncodingException e) {
e.printStackTrace();
}