I have the following command
sudo wget --output-document=/dev/null http://speedtest.pixelwolf.ch which outputs
--2016-03-27 17:15:47-- http://speedtest.pixelwolf.ch/
Resolving speedtest.pixelwolf.ch (speedtest.pixelwolf.ch)... 178.63.18.88, 2a02:418:3102::6
Connecting to speedtest.pixelwolf.ch (speedtest.pixelwolf.ch) | 178.63.18.88|:80... connected.
HTTP Request sent, awaiting response... 200 OK
Length: 85 [text/html]
Saving to: `/dev/null`
100%[======================>]85 --.-K/s in 0s
2016-03-27 17:15:47 (8.79 MB/s) - `dev/null` saved [85/85]
I'd like to be able to parse the (8.79 MB/s) from the last line and store this in a file (or any other way I can get this into a local PHP file easily), I tried to store the full output by changing my command to --output-document=/dev/speedtest however this just saved "Could not reach website" in the file and not the terminal output of the command.
Not quite sure where to start with this, so any help would be awesome.
Not sure if it helps, but my intention is for this stored value (8.79) in this instance to be read by a PHP file and handled there, every 30 seconds which I'll achieve by: while true; do (run speed test and save speed variable to a file cmd); php handleSpeedTest.php; sleep 5; done where handleSpeedTest.php will be able to read that stored value and handle it accordingly.
I changed the URL to one that works. Redirected stderr onto stdout. Used grep --only-matching (-o) and a regex.
sudo wget -O /dev/null http://www.google.com 2>&1 | grep -o '\([0-9.]\+ [KM]B/s\)'
Related
i am not root user,so i install aria2-1.34.0 with ./configure --prefix=/home/xxx/.local, everything works fine but on-download-complete
i set on-download-complete=/home/xxx/aria2/a.sh
#!/bin/bash
echo 123 > 1.txt
when a task download completed, the log shows everything is ok
2019-10-28 19:04:11.295587 [NOTICE] [RequestGroup.cc:1216] Download complete: /home/xxx/aria2/data/0.png
2019-10-28 19:04:11.295598 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file /home/xxx/aria2/data/0.png.aria2 does not exist.
2019-10-28 19:04:11.295612 [INFO] [util.cc:2239] Executing user command: /home/xxx/aria2/a.sh b031d9399fb9d93f 1 /home/xxx/aria2/data/0.png
but actually nothing happened, it didn't work!!!
aria2c path:
/home/xxx/.local/aria2c
aria2.conf
enable-rpc=true
rpc-allow-origin-all=true
rpc-listen-all=false
max-concurrent-downloads=1
continue=true
max-connection-per-server=5
min-split-size=5M
split=5
max-overall-download-limit=5M
max-download-limit=2M
max-overall-upload-limit=0
max-upload-limit=0
dir=/home/xxx/aria2/data
file-allocation=prealloc
on-download-complete=/home/xxx/aria2/a.sh
log=/home/xxx/aria2/aria2.log
log-level=info
i don't know how to deal with it, could u give me any suggestions, thanks.
got a solution for me
put --on-download-complete in cmd instead of aria2.conf
just run aria2 like:
aria2c --conf-path=/xxx/xxx/arai2.conf --on-download-complete=/xxx/xxx/xxx.sh -D
and it works
but i have no idea why it didn't work if set --on-download-complete in conf file
Accroding to the official document
-D, --daemon [true|false]
Run as daemon. The current working directory will be changed to / and standard input, standard output and standard error will be redirected to /dev/null. Default: false
You should modify /home/xxx/aria2/a.sh to
#!/bin/bash
echo 123 > $(dirname $0)/1.txt
to write the output to the right place.
Recently I needed to check for a huge list of filenames if they exist on a server. I did this by running a for loop which tried to wget each of those files. That was efficient enough, but took about 30 minutes in this case. I wonder if there is a faster way to check whether a file exists or not (since wget is for downloading files and not performing thousands of requests).
I don't know if that information is relevant, but it's an Apache server.
Curl would be the best option in a for loop and here is a straight forward simple way, run this in your forloop
curl -I --silent http://www.yoururl/linktodetect | grep -m 1 -c 404
What this simply does is check the http response header for a 404 returned on the link and if its detected as a missing file/link throwing a 404 then the command line output will display you a number 1; otherwise, if the file/link is valid and does not return a 404 then the command line output will display you a number 0.
I am using following command in my bash script to trigger jenkins build:
wget --no-check-certificate "http://<jenkins_url>/view/some_view/job/some_prj/buildWithParameters?token=xxx"
Output:
HTTP request sent, awaiting response... 201 Created
Length: 0
Saving to: “buildWithParameters?token=xxx”
[ <=> ] 0 --.-K/s in 0s
2015-02-20 10:10:46 (0.00 B/s) - “buildWithParameters?token=xxx” saved [0/0]
And then it's creates empty file: “buildWithParameters?token=xxx”
My question is: why wget creates this file and how to turn that functionality off?
Most simply:
wget --no-check-certificate -O /dev/null http://foo
this will make wget save the file to /dev/null, effectively discarding it.
I'm trying to execute a long list of repetitive commands on Terminal.
The commands look like this:
wget 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-180,52,9/1280x1280.png' -O '/Volumes/Alaya/XXXXXXXXX/Downloads/MapTiles/Tile (52.-180) 0.png' \
wget 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-177,52,9/1280x1280.png' -O '/Volumes/Alaya/XXXXXXXXX/Downloads/MapTiles/Tile (52.-177) 1.png' \
If I copy the entire list into Terminal, it executes them all but seems to do it in such a rush that some only get partially downloadeed, and some missed out entirely. It doesn't seem to take them one by one and wait until each is finished before attempting the next.
I tried putting them entire list into a shell script and running it, but then for some reason it seems to download everything, but only produces one file, and looking at the output, it seems to be trying to save each file under the same filename:
2014-03-29 09:56:31 (4.15 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [28319/28319]
--2014-03-29 09:56:31-- http://%20%0Dwget/
Resolving \rwget... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ` \rwget'
--2014-03-29 09:56:31-- http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/171,52,9/1280x1280.png
Reusing existing connection to api.tiles.mapbox.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 33530 (33K) [image/jpeg]
Saving to: `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png'
100%[======================================>] 33,530 --.-K/s in 0.008s
2014-03-29 09:56:31 (3.90 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [33530/33530]
--2014-03-29 09:56:31-- http://%20%0Dwget/
Resolving \rwget... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ` \rwget'
--2014-03-29 09:56:31-- http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/174,52,9/1280x1280.png
Reusing existing connection to api.tiles.mapbox.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 48906 (48K) [image/jpeg]
Saving to: `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png'
100%[======================================>] 48,906 --.-K/s in 0.01s
2014-03-29 09:56:31 (4.88 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [48906/48906]
--2014-03-29 09:56:31-- http://%20%0Dwget/
Resolving \rwget... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ` \rwget'
--2014-03-29 09:56:31-- http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/177,52,9/1280x1280.png
Reusing existing connection to api.tiles.mapbox.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 45644 (45K) [image/jpeg]
Saving to: `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png'
100%[======================================>] 45,644 --.-K/s in 0.01s
2014-03-29 09:56:31 (4.36 MB/s) - `/Volumes/Alaya/XXXXXXXX/Downloads/MapTiles/Tile (52.180) 120.png' saved [45644/45644]
So it's saving every file to this name: Tile (52.180) 120.png
Note that it doesn't do this if I put in each command separately...so I don't understand why it's doing that.
Can someone tell me how to execute this list of commands so that it does each one properly?
Thanks!
Your file should look like this:
#!/bin/bash
wget -q 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-180,52,9/1280x1280.png' -O 'a.png'
wget -q 'http://api.tiles.mapbox.com/v3/localstarlight.hl2o31b8/-177,52,9/1280x1280.png' -O 'b.png'
BUT... you have a backslash at the end of each wget line, which is a continuation character for long lines and which you don't need. Remove it.
Essentially you are asking wget to get a file and then another file called wget and then another file and then another file. Your script only does a single wget - the first one. All the other wget commands are seen as parameters to the first wget because of the continuation character.
You are doing this:
wget URL file wget URL file wget URL file
Quoting from the log you've posted:
http://%20%0Dwget/
This suggests that your script contains CR+LF line endings. Remove those before executing the script:
sed $'s/\r//' scriptname
or
tr -d '\r' < scriptname
I have s shell script, the content is as below
I hope the screen output could be redirected to templog,
the screen output not the html content but is like
--2012-10-30 15:53:14-- http://www.youtube.com/results?search_query=pig
Resolving www.youtube.com... 173.194.34.5, 173.194.34.6, 173.194.34.7, ...
Connecting to www.youtube.com|173.194.34.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “search_result”
[ <=> ] 108,503 --.-K/s in 0.07s
2012-10-30 15:53:15 (1.40 MB/s) - “search_result” saved [108503]
but it can't
I tried 2>&1|cat > templog
still not OK
you can copy the content and make a wget.sh file and then run it
you will notice that the content can not be redirected to templog,
how to deal with this to achieve my target?
thanks
keyword=pig
page_nr=3
wget -O search_result http://www.youtube.com/results?search_query=${keyword}&page=${page_nr} > templog
You just need to put quotes around your url. wget then is using the stderr to print on screen, so you also have to the stderr instead of the stdout (using 2> instead of >):
keyword=pig
page_nr=3
wget -O search_result "http://www.youtube.com/results?search_query=${keyword}&page=${page_nr}" 2> templog