I'm trying to download a .zip file using htmlunit-2.13.
From the web the file can be download at:
- http://www.bmfbovespa.com.br/consulta-isin/BuscaCodigosIsin.aspx?Idioma=pt-br
- Click in the link 'Download de Arquivos'
- And then link 'Banco de Dados Completo'
The following java coding is saving an html file instead of the .zip file
public class Teste {
public static void main(String args[])
{
try
{
LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.ALL);
HtmlPage page = null;
String url = "http://www.bmfbovespa.com.br/consulta-isin/BuscaCodigosIsin.aspx?Idioma=pt-br";
#SuppressWarnings("deprecation")
WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_10);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setJavaScriptEnabled(true);
WebRequest webRequest = new WebRequest(new URL(url));
webRequest.setCharset("UTF-8");
page = webClient.getPage( webRequest );
HtmlAnchor anchor1 = (HtmlAnchor) page.getElementById("ctl00_contentPlaceHolderConteudo_hplCompleto");
HtmlPage page2 = anchor1.click();
InputStream is = page2.getWebResponse().getContentAsStream();
FileOutputStream output = new FileOutputStream("/tmp/isin.zip");
IOUtils.copy(is, output);
output.close();
System.out.println("New file created!");
}
catch ( FailingHttpStatusCodeException e1 )
{
System.out.println( "FailingHttpStatusCodeException thrown:" + e1.getMessage() );
e1.printStackTrace();
}
catch ( MalformedURLException e1 )
{
System.out.println( "MalformedURLException thrown:" + e1.getMessage() );
e1.printStackTrace();
}
catch ( IOException e1 )
{
System.out.println( "IOException thrown:" + e1.getMessage() );
e1.printStackTrace();
}
catch( Exception e )
{
System.out.println( "General exception thrown:" + e.getMessage() );
e.printStackTrace();
}
}
}
I was having similar problems trying to get a zip from a page like this. This was my solution:
WebClient wc = new WebClient(BrowserVersion.CHROME);
HtmlPage p = wc.getPage(url)
// I believe clicking the element returns an UnexpectedPage
// because the content type is 'application/zip'
UnexpectedPage up = p.getElementById(buttonId).click();
InputStream in = up.getInputStream();
...
Related
Followed Professor Manning's suggestion to use the ProtobufAnnotationSerializer and did something wrong.
used serializer.writeCoreDocument on the correctly working document; Later read written file with pair = serializer.read; then used pair.second InputStream p2 = pair.second; p2 was empty resulting in a null pointer when running Pair pair3 = serializer.read(p2);
public void writeDoc(CoreDocument document, String filename ) {
AnnotationSerializer serializer = new
ProtobufAnnotationSerializer();
FileOutputStream fos = null;
try {
OutputStream ks = new FileOutputStream(filename);
ks = serializer.writeCoreDocument(document, ks);
ks.flush();
ks.close();
}catch(IOException ioex) {
logger.error("IOException "+ioex);
}
}
public void ReadSavedDoc(String filename) {
// Read
byte[]kb = null;
try {
File initialFile = new File(filename);
InputStream ks = new FileInputStream(initialFile);
ProtobufAnnotationSerializer serializer = new
ProtobufAnnotationSerializer();
InputStream kis = new
ByteArrayInputStream(ks.readAllBytes());
ks.close();
Pair<Annotation, InputStream> pair = serializer.read(kis);
InputStream p2 = pair.second;
int nump2 = p2.available();
logger.info(nump2);
byte[] ba = p2.readAllBytes();
Annotation readAnnotation = pair.first;
Pair<Annotation, InputStream> pair3 = serializer.read(p2);
kis.close();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (ClassCastException e) {
e.printStackTrace();
} catch(Exception ex) {
logger.error("Exception: "+ex);
ex.printStackTrace();
}
}
This line is unnecessary and should be deleted:
Pair<Annotation, InputStream> pair3 = serializer.read(p2);
If you have set up readAnnotation correctly that's the end of the read/write process. p2 is empty because you have read all its contents already.
There is a clear example of how to use serialization here:
https://github.com/stanfordnlp/CoreNLP/blob/master/itest/src/edu/stanford/nlp/pipeline/ProtobufSerializationSanityITest.java
You will have to also build a CoreDocument from an Annotation.
CoreDocument readDocument = new CoreDocument(readAnnotation);
I use spring mvc I want to uplaod image to jsp form so I add enctype="multipart/form-data" to the form tag but when i add this, modelAttribute values equals null in the controller
This is my form in jsp page:
<form:form action="saveContact" method="post" modelAttribute="Contacting" id="container" enctype="multipart/form-data">
This is the header of the function in controller:
#RequestMapping(value = "/saveContact", method = RequestMethod.POST)
public ModelAndView saveContact(#ModelAttribute ("Contacting") Contacting Contacting,ModelAndView modelndView,HttpServletRequest request ,HttpServletResponse response
) throws Exception {............}
#ModelAttribute ("Contacting") Contacting Contacting all values are null. and When I erease the enctype="multipart/form-data" from form tag its work well but I cant upload the image
this is the uplaud function:
public void uplaodImages(String url,HttpServletRequest request) {
// configures upload settings
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setSizeThreshold(THRESHOLD_SIZE);
ServletFileUpload upload = new ServletFileUpload(factory);
upload.setFileSizeMax(MAX_FILE_SIZE);
upload.setSizeMax(MAX_REQUEST_SIZE);
String uuidValue = "";
FileItem itemFile = null;
try {
// parses the request's content to extract file data
List formItems = upload.parseRequest(request);
Iterator iter = formItems.iterator();
// iterates over form's fields to get UUID Value
while (iter.hasNext()) {
FileItem item = (FileItem) iter.next();
if (item.isFormField()) {
if (item.getFieldName().equalsIgnoreCase(UUID_STRING)) {
uuidValue = item.getString();
}
}
// processes only fields that are not form fields
if (!item.isFormField()) {
itemFile = item;
}
}
if (itemFile != null) {
// get item inputstream to upload file into s3 aws
BasicAWSCredentials awsCredentials = new BasicAWSCredentials(AMAZON_ACCESS_KEY, AMAZON_SECRET_KEY);
AmazonS3 s3client = new AmazonS3Client(awsCredentials);
try {
ObjectMetadata om = new ObjectMetadata();
om.setContentLength(itemFile.getSize());
om.setContentType("image/png");
String ext = FilenameUtils.getExtension(itemFile.getName());
String keyName = uuidValue + '.' + ext;
// s3client.putObject(new PutObjectRequest(S3_BUCKET_NAME,"99/after/img", itemFile,st om));
// s3client.setObjectAcl(S3_BUCKET_NAME, "99/after/img", CannedAccessControlList.PublicRead);
TransferManager tm = new TransferManager(new ProfileCredentialsProvider());
System.out.println("Hello");
// TransferManager processes all transfers asynchronously,
// so this call will return immediately.
Upload upload1 = tm.upload(
S3_BUCKET_NAME, url, itemFile.getInputStream(),om);
System.out.println("Hello2");
try {
// Or you can block and wait for the upload to finish
upload1.waitForCompletion();
System.out.println("Upload complete.");
} catch (AmazonClientException amazonClientException) {
System.out.println("Unable to upload file, upload was aborted.");
amazonClientException.printStackTrace();
}
} catch (AmazonServiceException ase) {
// LOGGER.error(uuidValue + ":error:" + ase.getMessage());
} catch (AmazonClientException ace) {
//LOGGER.error(uuidValue + ":error:" + ace.getMessage());
}
} else {
//LOGGER.error(uuidValue + ":error:" + "No Upload file");
System.out.println("No Upload file");
}
} catch (Exception ex) {
//LOGGER.error(uuidValue + ":" + ":error: " + ex.getMessage());
System.out.println(ex.getMessage());
}
//LOGGER.info(uuidValue + ":Upload done");
System.out.println("Upload done");
}
#RequestMapping(value = "/form.html", method = RequestMethod.POST)
public String handleFormUpload(#RequestParam("name") String name,
#RequestParam("file") MultipartFile file) throws Exception {
}
I am trying to upload an audio file using the following code on server.Right now, it works perfectly well for image files but not for audios.I think MultipartFile should work with audio files as well.Can any one tell me what is wrong here?
I am getting "The server refused this request because the request entity is in a format not supported by the requested resource for the requested method." error.
Does MultipartFile not work with audio files?If no what is an alternative?
#Transactional
public BaseVO uploadImage(MultipartFile file, long userId){
Map<String, Object> alertParams = new HashMap<String, Object>();
try{
if (!file.isEmpty()) {
Image profileImage = imageRepository.findByUserId(userId);
if(profileImage == null){
profileImage = new Image();
profileImage.setUserId(userId);
}
File dir = null;
String realPath;
if(liveBuild == "true"){
realPath = liveImageUploadRepository;
dir = new File(realPath);
}else{
HttpServletRequest request = ((ServletRequestAttributes)RequestContextHolder.getRequestAttributes()).getRequest();
realPath = request.getSession().getServletContext().getRealPath("/") + devImageUploadRepository;
dir = new File(realPath);
}
if (!dir.exists()) {
dir.mkdirs();
}
String finalFileStorePath = realPath + "/" + userId + ".jpg";
File path = new File(finalFileStorePath);
file.transferTo(path);
profileImage.setImagePath(realPath);
//profileImage.setImageName(userId + ".jpg");
profileImage.setImageName(userId+"");
profileImage.setImageType(".jpg");
imageRepository.save(profileImage);
alertParams.put("id", profileImage.getId());
} else {
return new BaseVO(alertParams, Constants.STATUS_OK, Constants.STATUS_OK_MSG);
}
}catch(HibernateException e){
return new BaseVO(alertParams, Constants.STATUS_ERROR, Constants.STATUS_ERROR_MSG + " " + e.getMessage());
} catch (IllegalStateException e) {
e.printStackTrace();
return new BaseVO(alertParams, Constants.STATUS_ERROR, Constants.STATUS_ERROR_MSG);
} catch (IOException e) {
e.printStackTrace();
return new BaseVO(alertParams, Constants.STATUS_ERROR, Constants.STATUS_ERROR_MSG);
}
return new BaseVO(alertParams, Constants.STATUS_OK, Constants.STATUS_OK_MSG);
}
I want to upload a file to my project folder. My code is as follows:
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
File savedFile;
String destination;
List<FileItem> items = null;
try {
items = new ServletFileUpload(new DiskFileItemFactory()).parseRequest(request);
} catch (FileUploadException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
for (FileItem item : items) {
if (item.isFormField()) {
// Process regular form field (input type="text|radio|checkbox|etc", select, etc).
} else {
// Process form file field (input type="file").
String fieldName = item.getFieldName();
String fileName = FilenameUtils.getName(item.getName());
InputStream fileContent = item.getInputStream();
String userName = (String) session.getAttribute("newUser");
destination = getServletConfig().getServletContext().getContextPath() + "\\" + userName + ".jpeg";
savedFile = new File(destination);
//Check if file exists
if(!savedFile.exists())
savedFile.createNewFile();
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(savedFile));
byte[] buffer = new byte[1024];
int len;
//Read from file and write to new file destination
while((len = fileContent.read(buffer)) >= 0) {
bos.write(buffer, 0, len);
}
//Closing the streams
fileContent.close();
bos.close();
}
}
}
When I run the jsp file and browse and select the required image and submit the form, the servlet runs but it throws IOException. The exception is throws by the line where I create a new path using savedFile.createNewFile(). Before I used that code, it threw another FileNotFoundException. I am not sure if the path that I have provided is correct.
Try to use getRealPath() method.
String fileName="/" + userName + ".jpeg";
destination = getServletContext().getRealPath(fileName);
savedFile = new File(destination);
I am trying to read a text file from IsolatedStorage and check it contains a string. If not, the string is added to the end of the file. But When I am trying to write the string into file I got an error: "Operation not permitted on IsolatedStorageFileStream.". My code shown below. How can I overcome this problem?
public void AddToDownloadList()
{
IsolatedStorageFile downloadFile=IsolatedStorageFile.GetUserStoreForApplication();
try
{
string downloads = string.Empty;
if (!downloadFile.DirectoryExists("DownloadedFiles"))
downloadFile.CreateDirectory( "DownloadedFiles" );
if(downloadFile.FileExists("DownloadedFiles\\DownloadList.txt"))
{
IsolatedStorageFileStream downloadStream = downloadFile.OpenFile("DownloadedFiles\\DownloadList.txt",FileMode.Open, FileAccess.Read );
using ( StreamReader reader = new StreamReader( downloadStream ) )
{
downloads = reader.ReadToEnd();
reader.Close();
}
downloadFile.DeleteFile( "DownloadedFiles\\DownloadList.txt" );
}
downloadFile.CreateFile( "DownloadedFiles\\DownloadList.txt" );
string currentFile = FileName;
if ( !downloads.Contains( currentFile ) )
{
downloads += currentFile;
using ( StreamWriter writeFile = new StreamWriter( new IsolatedStorageFileStream( "DownloadedFiles\\DownloadList.txt", FileMode.Create, FileAccess.Write, downloadFile ) ) )
{
writeFile.Write( currentFile + "," );
writeFile.Close();
}
}
}
catch ( Exception ex )
{
string message = ex.Message;
}
}
I think the problem you were having has to do with the line where you create the StreamWriter by newing up the IsolatedStorageFileStream - when you already should have the right one from the return of the downloadFile.CreateFile() call.
Try this code, I think it does what you want to do:
public static void AddToDownloadList()
{
try
{
AddToDownloadList("DownloadedFiles", "this file name", "DownloadedFiles\\DownloadList.txt");
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine("Exception: " + ex.Message);
}
}
public static void AddToDownloadList(string directory, string fileName, string filePath)
{
string downloads = string.Empty;
using (IsolatedStorageFile downloadFile = IsolatedStorageFile.GetUserStoreForApplication())
{
if (!downloadFile.DirectoryExists(directory))
downloadFile.CreateDirectory(directory);
if (downloadFile.FileExists(filePath))
{
IsolatedStorageFileStream downloadStream = downloadFile.OpenFile(filePath, FileMode.Open, FileAccess.Read);
using (StreamReader reader = new StreamReader(downloadStream))
{
downloads = reader.ReadToEnd();
reader.Close();
}
}
string currentFile = fileName;
if (!downloads.Contains(currentFile))
{
downloadFile.DeleteFile(filePath);
using (IsolatedStorageFileStream stream = downloadFile.CreateFile(filePath))
{
downloads += currentFile;
using (StreamWriter writeFile = new StreamWriter(stream))
{
writeFile.Write(currentFile + ",");
writeFile.Close();
}
}
}
}
}