When debugging a website (e.g. developed using Wicket), what client-tool one can use to see the webserver's exact response (e.g. 301 or 302)?
use apt-get or similar to install wget and then do
wget -S http://localhost:8090
Where "localhost:8090" is the url path to the site
To see all headers from the GET request. wget also has options that support POST requests
Alternatively use a browser with firebug to see the response codes
Related
i am currently stuck at a problem, that i think if i use a proxy, wget tries to connect to the server via http instead of ftp. Since i cannot post the full settings i will post an example of the behaviour I see:
Accessing the ftp server without a proxy works using the command:
wget -r --user=username --password=mypassord ftp://ftp.myadress.com/
works as expected.
Accessing the ftp server with
wget -r --ftp-user=username --ftp-password=mypassord ftp://ftp.myadress.com/
works too.
Accessing the server with proxy and
wget -r --ftp-user=username --ftp-password=mypassord ftp://ftp.myadress.com/
leads to a
401 Unauthorized
error.
Using
wget -r --user=username --password=mypassord ftp://ftp.myadress.com/
leads to an index.html to be created. Unfortunately, the ftp server does not have any index.html files in the folders. Accessing a file with its full path through the proxy via
wget --user=username --password=mypassord ftp://ftp.myadress.com/test/test.txt
downloads the file as expected.
Accessing a different ftp server through the same proxy, which has an index.html in every folder with the command:
wget -r --user=username2 --password=mypassord2 ftp://ftp.myadress2.com/
works fine.
So how can i force wget to use the ftp protocol through the proxy?
Thanks in advance
Have you tried to put the proxy settings into a .wgetrc file instead?
It should contain something similar:
use_proxy=yes
http_proxy=127.0.0.1:8080
From time to time I have to download a specific file from a website with wget. The URL is very long, so I created a free .tk-domain that forwards to the file. If I use my new .tk-URL in my browser, it downloads the file as I want it but on my VPS on Ubuntu, it only downloads the index.html file if I use wget. I've two forwarding options on Dot.TK
Frame (Cloaking)
Redirect (HTTP 301 Forwarding)
Which option should I use and is there a way to get the file instead of the index.html?
If you use a 301, wget should be able to download the file. You can also use curl -LO <URL> with the 301.
The title says it all. I realize that a similar question has been asked at https://askubuntu.com/questions/307566/wget-and-curl-somehow-modifying-bencode-file-when-downloading/310507#310507 but I don't think the same solution works, because I have tried to unzip the file using 7zip, and also Gzip for windows (http://gnuwin32.sourceforge.net/packages/gzip.htm). Both claim the file to be of wrong format. Renaming it's extension to .gz or .zip doesn't help either. The --compressed attribute is no help as well. So, my guess is something's changed on the torcache site. I've tried using the user-agent as well, to no avail.
In a related issue, I guess, when I try downloading from the https site, I recieve "curl: (52) Empty reply from server". Only http works, and that gives me invalid bencoding. When I enter the URL on my browser, the torrent file downloads all by itself.
The command I'm entering is as follows:
curl -O http://torcache.net/torrent/006DDC8C407ACCDAF810BCFF41E77299A373296A.torrent
I am trying to download all images off this website path http://www.samsung.com/sg/consumer/mobile-devices/smartphones/ using the below code
wget -e robots=off -nd -nc -np --recursive -r -p --level=5 --accept jpg,jpeg,png,gif --convert-links -N --limit-rate=200k --wait 1.0 -U 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1' -P testing_folder www.samsung.com/sg/consumer/mobile-devices/smartphones
I would expect to see the images of the phones downloaded to my testing_folder.But all I see is some global images like logo etc. I dont seem to be able to get the phone images downloaded. The code above seems to work on some other websites through.
I have gone through all the wget questions on this forum but this particular issue doesnt seem to have an answer. Can someone help, I am sure there is a easy out. What am I doing wrong ?
UPDATE:
It looks like it is an issue with possible javascript pages and hence seems like end of the road, since apparently wget cant handle javascript pages well. If anyone can still help, will be delighted.
Steps:
configure a proxy server, for example Apache httpd with mod_proxy and mod_http_proxy
visit the page with a web browser that supports JavaScript and is configured to use your proxy server
harvest the URLs from the proxy server log file and put them in a file
Or:
Start Firefox and open web page
F10 - Tools - Page Info - Media - right click - select all - right click - copy
Paste into file with your favourite editor
Then:
optionally, (if you don't want to find out how to get wget read a list of URLs from a file), add minimal html tags (html, body and img) to the file
use wget to download the image specifying the file created in step 3 or 4 as the starting point
I have been quite successfully using CMake to perform builds using the ExternalProject_Add function, but my company recently put in a proxy server... Which has broken the aforementioned build scripts.
The download step fails during the extract phase because the tarball that was downloaded is only the redirect request from the proxy server (at least I think this is what is contained in the tiny tarball it acquires).
I found this post on the CMake mailing-list. I thought maybe if it worked for the file() command it might work for the ExternalProject_Add() command. I set both http_proxy and HTTP_PROXY environment variables, but still received the same error. I have thought about overriding the DOWNLOAD_COMMAND argument with a wget call since this command seems to behave with the proxy settings. However, I wanted to know if there was a better way.
UPDATE 1: I checked the contents of the small tarball, and it does contain HTML; however, it is a notification that Authentication is required. I'm not sure why it is requiring authentication because I haven't had to enter any login information for wget. wget shows the following output:
Resolving webproxy... 10.0.1.50
Connecting to webproxy|10.0.1.50|:80... connected.
Proxy request sent, awaiting response... 200 OK
Download begins here...
UPDATE 2: I have also noticed that both apt-get and svn fail with this new proxy setup, but git does not... svn complains about "Server sent unexpected return value (307 Proxy Redirect)..." Very confusing...
Thanks!
What version of CMake are you using? The file(DOWNLOAD command started using the follow redirect flag in version 2.8.2, introduced by the following commit:
http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=ef491f78218e255339278656bf6dc26073fef264
Using a custom DOWNLOAD_COMMAND is certainly a reasonable workaround.