I am using following command in my bash script to trigger jenkins build:
wget --no-check-certificate "http://<jenkins_url>/view/some_view/job/some_prj/buildWithParameters?token=xxx"
Output:
HTTP request sent, awaiting response... 201 Created
Length: 0
Saving to: “buildWithParameters?token=xxx”
[ <=> ] 0 --.-K/s in 0s
2015-02-20 10:10:46 (0.00 B/s) - “buildWithParameters?token=xxx” saved [0/0]
And then it's creates empty file: “buildWithParameters?token=xxx”
My question is: why wget creates this file and how to turn that functionality off?
Most simply:
wget --no-check-certificate -O /dev/null http://foo
this will make wget save the file to /dev/null, effectively discarding it.
Related
I'm starting with page:
https://mysite/a"
I'd like to spider the page getting the full urls of any nested urls below this that begin with the same stem (like https://mysite/a/b ).
I've tried:
$ wget -r --spider --accept-regex "https://...*" 'https://.../' 2>test.txt
which produces a large amount of output inclusing what appear to be the urls I'm after like:
--2018-04-21 15:04:48-- https:/mysite/a/
Reusing existing connection to mysite:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'a/index.html.tmp.tmp'
How do I just print out a list of the urls?
Edit:
changed it to
$ wget -r --spider 'https://mysite/a/' |grep 'https://mysite/a*' 2>test.txt
as a test . No output is being saved in test.txt. The file is empty.
I have the following command
sudo wget --output-document=/dev/null http://speedtest.pixelwolf.ch which outputs
--2016-03-27 17:15:47-- http://speedtest.pixelwolf.ch/
Resolving speedtest.pixelwolf.ch (speedtest.pixelwolf.ch)... 178.63.18.88, 2a02:418:3102::6
Connecting to speedtest.pixelwolf.ch (speedtest.pixelwolf.ch) | 178.63.18.88|:80... connected.
HTTP Request sent, awaiting response... 200 OK
Length: 85 [text/html]
Saving to: `/dev/null`
100%[======================>]85 --.-K/s in 0s
2016-03-27 17:15:47 (8.79 MB/s) - `dev/null` saved [85/85]
I'd like to be able to parse the (8.79 MB/s) from the last line and store this in a file (or any other way I can get this into a local PHP file easily), I tried to store the full output by changing my command to --output-document=/dev/speedtest however this just saved "Could not reach website" in the file and not the terminal output of the command.
Not quite sure where to start with this, so any help would be awesome.
Not sure if it helps, but my intention is for this stored value (8.79) in this instance to be read by a PHP file and handled there, every 30 seconds which I'll achieve by: while true; do (run speed test and save speed variable to a file cmd); php handleSpeedTest.php; sleep 5; done where handleSpeedTest.php will be able to read that stored value and handle it accordingly.
I changed the URL to one that works. Redirected stderr onto stdout. Used grep --only-matching (-o) and a regex.
sudo wget -O /dev/null http://www.google.com 2>&1 | grep -o '\([0-9.]\+ [KM]B/s\)'
How can I download OracleXE using wget and avoid the login?
I tried applying logic from this question for Oracle Java but I couldn't get it to work.
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn/linux/oracle11g/xe/oracle-xe-11.2.0-1 .0.x86_64.rpm.zip
I get:
--2015-10-13 04:51:03-- http://download.oracle.com/otn/linux/oracle11g/xe/oracle-xe-11.2.0-1.0.x86_64.rpm.zip
Resolving download.oracle.com (download.oracle.com)... 206.248.168.160, 206.248.168.139, 206.248.168.160, ...
Connecting to download.oracle.com (download.oracle.com)|206.248.168.160|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://edelivery.oracle.com/akam/otn/linux/oracle11g/xe/oracle-xe-11.2.0-1.0.x86_64.rpm.zip [following]
--2015-10-13 04:51:03-- https://edelivery.oracle.com/akam/otn/linux/oracle11g/xe/oracle-xe-11.2.0-1.0.x86_64.rpm.zip
Resolving edelivery.oracle.com (edelivery.oracle.com)... 23.9.117.183, 23.9.117.183
Connecting to edelivery.oracle.com (edelivery.oracle.com)|23.9.117.183|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://login.oracle.com/pls/orasso/orasso.wwsso_app_admin.ls_login?Site2pstoreToken=v1.2~CA55CD32~7E777A421E00059BE8321AEAF3C29C59D68A2F46E15A49137CE5AAF6D6B46A0C599A4560AD622CF26FFFCF23FF8FC274F021B7E57B08CEF2076FADB1A57BBFB02B991E320BB3A417DDF966B4406E225736912745DE8F5E660631675765D519A5E7FF61481F567ED9C582AEAAEEC6E2A6C59D046AD82EA1C7AA08E9A1EDAFC44D97F22C470FE530A0F58872A00CAFD27012DF4851AD4964085264393C7220CF07817E14ED0B2130ECF06758DB538644A119246C4B65963CD1C825650BE3B3C86C1670EC8F754E943853BE4C58F0A4FD89B1CE14E7110087134765A9EBAA170769C75645798E1D978B944D2D896A564E49CD42478328D8661794E3DC377DBEF9F7C27184E0DFF7EAAB [following]
--2015-10-13 04:51:03-- https://login.oracle.com/pls/orasso/orasso.wwsso_app_admin.ls_login?Site2pstoreToken=v1.2~CA55CD32~7E777A421E00059BE8321AEAF3C29C59D68A2F46E15A49137CE5AAF6D6B46A0C599A4560AD622CF26FFFCF23FF8FC274F021B7E57B08CEF2076FADB1A57BBFB02B991E320BB3A417DDF966B4406E225736912745DE8F5E660631675765D519A5E7FF61481F567ED9C582AEAAEEC6E2A6C59D046AD82EA1C7AA08E9A1EDAFC44D97F22C470FE530A0F58872A00CAFD27012DF4851AD4964085264393C7220CF07817E14ED0B2130ECF06758DB538644A119246C4B65963CD1C825650BE3B3C86C1670EC8F754E943853BE4C58F0A4FD89B1CE14E7110087134765A9EBAA170769C75645798E1D978B944D2D896A564E49CD42478328D8661794E3DC377DBEF9F7C27184E0DFF7EAAB
Resolving login.oracle.com (login.oracle.com)... 209.17.4.8, 209.17.4.8
Connecting to login.oracle.com (login.oracle.com)|209.17.4.8|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2051 (2.0K) [text/html]
Saving to: ‘oracle-xe-11.2.0-1.0.x86_64.rpm.zip’
100%[======================================================================================================================================================>] 2,051 --.-K/s in 0s
2015-10-13 04:51:03 (142 MB/s) - ‘oracle-xe-11.2.0-1.0.x86_64.rpm.zip’ saved [2051/2051]
For download oracle for linux zips from url to directly server you have to:
1 - login Oracle.com with credentials.. (https://login.oracle.com/mysso/signon.jsp)
2 - export cookie.txt with browser
3 - copy this file to your Server
scp cookies.txt root#url:/path/
4 - go to path where your cookies.txt and copy install link and paste as to this to your server terminal
wget --load-cookies=cookies.txt http://download.oracle.com/otn/linux/oracle12c/121020/linuxamd64_12102_database_1of2.zip
wget --load-cookies=cookies.txt http://download.oracle.com/otn/linux/oracle12c/121020/linuxamd64_12102_database_2of2.zip
check file size with ls -lah
Note, that wget --header "Cookie: oraclelicense=accept-securebackup-cookie" breaks all other cookies, including authorization ones.
Instead You can use custom cookies.txt file and --user/--password (tested on Oracle Archive and OracleXE)
echo .oracle.com TRUE / FALSE 0 oraclelicense accept-securebackup-cookie >cookies.txt
wget -c --load-cookies cookies.txt --trust-server-names --user=SSO_USERNAME --password=SSO_PASSWORD URL
UPD: Attention! cookies.txt is tab-separated! To be sure of tabs please use `echo -e .oracle.com\tTRUE\t/\tFALSE\t0\toraclelicense\taccept-securebackup-cookie >cookies.txt
I'm trying to execute a long list of repetitive commands on Terminal.
The commands look like this:
wget 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-180,52,9/1280x1280.png' -O '/Volumes/Alaya/XXXXXXXXX/Downloads/MapTiles/Tile (52.-180) 0.png' \
wget 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-177,52,9/1280x1280.png' -O '/Volumes/Alaya/XXXXXXXXX/Downloads/MapTiles/Tile (52.-177) 1.png' \
If I copy the entire list into Terminal, it executes them all but seems to do it in such a rush that some only get partially downloadeed, and some missed out entirely. It doesn't seem to take them one by one and wait until each is finished before attempting the next.
I tried putting them entire list into a shell script and running it, but then for some reason it seems to download everything, but only produces one file, and looking at the output, it seems to be trying to save each file under the same filename:
2014-03-29 09:56:31 (4.15 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [28319/28319]
--2014-03-29 09:56:31-- http://%20%0Dwget/
Resolving \rwget... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ` \rwget'
--2014-03-29 09:56:31-- http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/171,52,9/1280x1280.png
Reusing existing connection to api.tiles.mapbox.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 33530 (33K) [image/jpeg]
Saving to: `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png'
100%[======================================>] 33,530 --.-K/s in 0.008s
2014-03-29 09:56:31 (3.90 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [33530/33530]
--2014-03-29 09:56:31-- http://%20%0Dwget/
Resolving \rwget... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ` \rwget'
--2014-03-29 09:56:31-- http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/174,52,9/1280x1280.png
Reusing existing connection to api.tiles.mapbox.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 48906 (48K) [image/jpeg]
Saving to: `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png'
100%[======================================>] 48,906 --.-K/s in 0.01s
2014-03-29 09:56:31 (4.88 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [48906/48906]
--2014-03-29 09:56:31-- http://%20%0Dwget/
Resolving \rwget... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ` \rwget'
--2014-03-29 09:56:31-- http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/177,52,9/1280x1280.png
Reusing existing connection to api.tiles.mapbox.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 45644 (45K) [image/jpeg]
Saving to: `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png'
100%[======================================>] 45,644 --.-K/s in 0.01s
2014-03-29 09:56:31 (4.36 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [45644/45644]
So it's saving every file to this name: Tile (52.180) 120.png
Note that it doesn't do this if I put in each command separately...so I don't understand why it's doing that.
Can someone tell me how to execute this list of commands so that it does each one properly?
Thanks!
Your file should look like this:
#!/bin/bash
wget -q 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-180,52,9/1280x1280.png' -O 'a.png'
wget -q 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-177,52,9/1280x1280.png' -O 'b.png'
BUT... you have a backslash at the end of each wget line, which is a continuation character for long lines and which you don't need. Remove it.
Essentially you are asking wget to get a file and then another file called wget and then another file and then another file. Your script only does a single wget - the first one. All the other wget commands are seen as parameters to the first wget because of the continuation character.
You are doing this:
wget URL file wget URL file wget URL file
Quoting from the log you've posted:
http://%20%0Dwget/
This suggests that your script contains CR+LF line endings. Remove those before executing the script:
sed $'s/\r//' scriptname
or
tr -d '\r' < scriptname
I have s shell script, the content is as below
I hope the screen output could be redirected to templog,
the screen output not the html content but is like
--2012-10-30 15:53:14-- http://www.youtube.com/results?search_query=pig
Resolving www.youtube.com... 173.194.34.5, 173.194.34.6, 173.194.34.7, ...
Connecting to www.youtube.com|173.194.34.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “search_result”
[ <=> ] 108,503 --.-K/s in 0.07s
2012-10-30 15:53:15 (1.40 MB/s) - “search_result” saved [108503]
but it can't
I tried 2>&1|cat > templog
still not OK
you can copy the content and make a wget.sh file and then run it
you will notice that the content can not be redirected to templog,
how to deal with this to achieve my target?
thanks
keyword=pig
page_nr=3
wget -O search_result http://www.youtube.com/results?search_query=${keyword}&page=${page_nr} > templog
You just need to put quotes around your url. wget then is using the stderr to print on screen, so you also have to the stderr instead of the stdout (using 2> instead of >):
keyword=pig
page_nr=3
wget -O search_result "http://www.youtube.com/results?search_query=${keyword}&page=${page_nr}" 2> templog