I'm trying to download the XLS file from this page: http://www.nordpoolspot.com/Market-data1/Elspot/Area-Prices/ALL1/Hourly/ (click on "Export to XLS" link).
However doing:
page.getAnchorByText("Export to XLS").click().getWebResponse().getContentAsStream();
returns the html of the web page, instead of the expected file.
Do you have any suggestion?
I already tried the 3 points here http://htmlunit.sourceforge.net/faq.html#AJAXDoesNotWork without success.
The following:
webClient = new WebClient(BrowserVersion.FIREFOX_3_6);
fixed my issue.
Related
Link: http://content.time.com/time/covers/0,16641,19230303,00.html [new DOM link]
Cover Page Html tag
How to get that SCR in Jason and download images
Next Button Tag
I want to scrap this 2 links using Scrapy
Any Help !!
I need to write a method to download images and click on next page, run them in for loop till final image get the download(Final Page).
how to download rest of part ill figure it out.
I follow this tutorial https://www.pyimagesearch.com/2015/10/12/scraping-images-with-python-and-scrapy/
[DOM is already outdated ]
I've already set all files and Pipelines for project
For Record, I tried different Different method XPath css response
https://github.com/Dhawal1306/Scrapy
Everything is done solution is on Github 4700 somewhere images we have and along with JSON also.
for a tutorial, any question you just have to ask !!
I know this is not scrapy but I found easier using BS4. so you have to "pip install beautifulsoup4". Here is a sample :
import requests
from bs4 import BeautifulSoup
import os
r = requests.get("https://mouradcloud.westeurope.cloudapp.azure.com/blog/blog/category/food/")
data = r.text
soup = BeautifulSoup(data, "lxml")
for link in soup.find_all('img'):
image_url = link.get("src")
print(image_url)
It worked like a charm
I am using laravel-snappy to generate pdfs and show them in the browser. I am getting a blank page in return.The code I am using is this.
return PDF::loadView('emails.flightInvoice', $data)->inline();
I used the below code to save the PDF and it is working.
PDF::loadView('emails.flightInvoice', $data')->save('invoice.pdf');
can anyone please help me with this?
The inline() method requires a filename
Try
return PDF::loadView('pdf.invoice', $data)->inline('filename.pdf');
I have this following code for downloading files :-
#Controller
public class FileController {
#RequestMapping(value = "/files/{file_name:.+}", method = RequestMethod.GET)
#ResponseBody
public FileSystemResource getFile(#PathVariable("file_name") String fileName) {
return new FileSystemResource("C:/Users/sourav/fileServer/"+fileName);
}
}
When I go to the link for the first time nothing is displayed .When I reload only a text file with name f.txt is downloaded instead of the pdf file. I want the pdf file to be displayed in the browser. How to solve this problem ?
I think you need to set the response headers. Otherwise there is no way for the browser to intuit the file format. Something like response.setContentType("application/pdf");.
your code is ok. I think if you try with pdf file it will work as you expected, it will be displayed in browser. I tested it and worked fine in Chrome and Firefox. May be your testing file is corrupted.
If you are using Spring Boot, you can add the MIME types of the files you want to download into spring.mvc.mediaTypes properties in the configuration file. For example:
spring.mvc.mediaTypes.yml=text/yaml
Source: https://github.com/spring-projects/spring-boot/issues/4220
I am looking for a method to download automatically a file from a website.
Currently the process is really manual and heavy.
I go on a webpage, I enter my pass and login.
It opens a pop up, where I have to click a download button to save a .zip file.
Do you have any advice on how I could automate this task ?
I am on windows 7, and I can use mainly MS dos batch, or python. But I am open to other ideas.
You can use selenium web driver to automate the downloading. You can use below snippet for browser download preferences in java.
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("browser.download.folderList", 2);
profile.setPreference("browser.download.manager.showWhenStarting", false);
profile.setPreference("browser.download.dir", "C:\\downloads");
profile.setPreference("browser.helperApps.neverAsk.openFile","text/csv,application/x-msexcel,application/excel,application/x-excel,application/vnd.ms-excel,text/html,text/plain,application/msword,application/xml");
To handle the popup using this class when popup comes.
Robot robot = new Robot();
robot.keyPress(KeyEvent.VK_DOWN);
robot.keyRelease(KeyEvent.VK_DOWN);
robot.keyPress(KeyEvent.VK_ENTER);
robot.keyRelease(KeyEvent.VK_ENTER);
You'll want to take a look at requests (to fetch the html and the file), Beautifulsoup (to parse the html and find the links)
requests has built in auth: http://docs.python-requests.org/en/latest/
Beautifulsoup is quite easy to use: http://www.crummy.com/software/BeautifulSoup/bs4/doc/
Pseudocode: use request to download the sites html and auth. Go through the links by parsing. If a link meets the criteria -> save in a list, else continue. When all the links have been scrapped, go through them and download the file using requests (req = requests.get('url_to_file_here', auth={'username','password'}), if req.status_code in [200], file = req.text
If you can post the link of the site you want to download from, maybe we can do more.
I'm new to this Ajax thing. I wanted to try this
http://labs.adobe.com/technologies/spry/samples/data_region/SuggestSample.html
neat little Autosuggest form.
The form doesn't work when i save it locally.
Below there is a list of what i've done and used so far :
Firefox -> save pages as ..(index.html)
new folder ( test23 )
also saved the products.xml
opened index.html
change this line : var dsProducts = new Spry.Data.XMLDataSet("../../demos/products/products.xml", "/products/product", { sortOnLoad: "name" })
into : var dsProducts = new Spry.Data.XMLDataSet("products.xml", "/products/product", { sortOnLoad: "name" })
test failed :(
Can anyone help me out ?
AJAX requests cannot access the local file system, so requests like that will fail. You will need to have the page up on a webserver. If you want a local one, install XAMPP or something similar.
I just tried for like three minutes and got it to work at the first try (without images). you have to remember to get all the scripts and actually point to them in the main html file.
Don't forget the script tags on lines 41 through 43.
Kris
-- additions:
I tested on my Mac's local filesystem without any server using Safari as my browser. I have since deleted the files but could easily do it again and put the files up for download.