I have a cronjob getting a list of prices from a website in JSON format and copying it into the right folder and it looks like this:
curl 'https://somesite.web/api/list.json' > ~/list.json.tmp && cp ~/list.json.tmp /srv/www/list.json > /dev/null
The problem is that a couple of times the website was down while the cron was trying to get the list and got an empty JSON file. To prevent this in the future, is there a way to make the cron only copy the file if it's not empty (no cp option to do this)? or should I create a script to do that and call the script after getting the list?
Maybe curl --fail will accomplish what you want? From the man page:
-f, --fail
(HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.
This would cause curl to exit with a failure code, and thus the && in your statement would not execute the copy.
curl ... && [ -s ~/list/json.tmp ] && cp ~/list/json.tmp /srv/www/list.json
The -s test is true if the named file exists and is not empty.
(Incidentally, the > /dev/null redirection is not necessary. The cp command might print error messages to stderr, but it shouldn't print anything to stdout, which is what you're redirecting.)
Related
Some help with Shell Script and cURL/wGet ?
I need to call multiple URL´s from a file.
While testing I´m using just 3 rows in that file, meaming that I have 3 URL´s.
Fist step I create the file and then I read it row by row calling via cURL or wGet using the block bellow:
cat ${vMyFile} | while read vMyRow #I cat the file and read it on a "While" structure
do
echo curl -f ${vMyRow} >>${vMyLogFile} #I use cURL to call the rows with the URL`s
The script works just fine, and the log file (vMyLogFile) shows me that everything worked.
The "curl" called all 3 URL´s with no erros BUT... I can only get the last row(URL) to work.
When I call these URL´s, a service is activated and only the server at line 3 (the last one) get activated. :(
If I get the cURL line 1 and 2 from the log file and run it thru terminal, the service is activated successfully.
I already tried to use SLEEP and WAIT between the cURL´s and still can´t get it to work properly.
Is there a "CLOSE" or "END" that I must user before calling a cURL request again in the same process?
I need to call multiple URL´s from a file.
wget has option for such use case, namely -i. If not other options are specified it should be followed by name of file which holds one URL per line, so if this file is named for example urls.txt simplest usage is
wget -i urls.txt
If you want to know more about -i behavior consult wget's man page.
I've got a bash script I use to create a ngrok tunnel, then I use dweet.io to post the tunnel address & port.
If that's meaningless to you, don't worry, essentially I'm using wget --post-data to post a string to an address.
This bash script is auto-started with a cron job.
while true
do
#Gets the internal IP
IP="$(hostname -I)"
#Gets the external IP
EXTERNALIP="$(curl -s https://canihazip.com/s )"
echo "Dweeting IP... "
TUNNEL="$(curl -s http://localhost:4040/api/tunnels)"
echo "${TUNNEL}" > tunnel_info.json
#Gets the tunnel's address and port
TUNNEL_TCP=$(grep -Po 'tcp:\/\/[^"]+' ./tunnel_info.json )
#Pushes all this information to dweet.io
wget -q --post-data="tunnel=${TUNNEL_TCP}&internal_ip=${IP}&external_ip=${EXTERNALIP}" http://dweet.io/dweet/for/${dweet_id_tunnel}
sleep $tunnel_delay
done
This works, however, the directory I start the script from gets spammed with files named
dweet_id_tunnel.1,
dweet_id_tunnel.2,
dweet_id_tunnel.3,
...
These contain the HTTP response from the wget --post-data from dweet.io.
As this script runs regularly, it's rather annoying to have a folder filled with thousands of these responses. I'm not sure why they're even made because I added the -q argument to wget, which should suppress responses.
Any idea what I need to change to stop these files being created?
wget fetches the response and saves it to a file; that's what it does. If you don't want that, add -O /dev/null, or switch to curl which seems to be more familiar to you anyway, as well as more versatile.
The -q option turns off reporting, not downloading (i.e. progress messages etc, similar to curl -s).
Hi I have the following simple script to read some URLs from the text and post it to another text file with response.
#!/bin/bash
while read url
do
urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "$url")
echo "$url"
echo "$url $urlstatus" >> urlstatus.txt
done < $1
As an example I am trying following link:
www.proddigia.com/inmueble/pisos/en-venta/el-putget-i-el-farro/sarria-sant-gervasi/barcelona/6761
However I get 0 as response. When I check with google I get 200. Am I missing something in script?
Zero is not a valid HTTP response code.
If curl is unable to establish a HTTP connection to the server, or if the server (somehow) fails to deliver a well-formed HTTP response message, there will be no "http code" to return in that variable. Zero is what you would probably see in that scenario.
It could also be that the value of $url that you are using is invalid. For example, if the URL is enclosed in < and > characters, it curl won't understand it. I would expect a zero in that case too.
The problem is that --silent is telling curl to throw away all of the error messages, so it can't tell you what the problem is.
I suggest that you see what you get by running the following command:
curl -o /dev/null --head "$url"
with the identical url string to the one you are currently using.
I just figured out that if you use txt file created in windows OS it does not work as expcted in ubuntu. That was the reason that I got 0. You need to create the txt file in Ubuntu and copy the links over there. Thanks for the answers anyway.
My problem: I'm trying to run a database generation script at the command line via a batch file as part of a TFS build process to enable nightly testing on a known dataset.
The scripts we run are outputting Notices, Warnings and some Errors on the command line. I would like to suppress at least the Notices and Warnings, and if possible the Errors as they don't seem to have an impact on the overall success of the scripts. This output seems to be affecting the success or failure of the process as far as the TFS build process is concerned. It's highlighting every line of output from the scripts as errors and failing the build.
As our systems are running on Windows, most of the potential solutions I've found online don't work as they seem to target Linux.
I've changed the client_min_messages to error in the postgresql.conf file, but when looking at the same configuration from pgAdmin (tools > server configuration) it shows the value as Error but the current value as Notice.
All of the lines in the batch file that call psql use the -q flag as well but that only seems to prevent the basics such as CREATE TABLE and ALTER TABLE etc.
An example line from the batch file is:
psql -d database -q < C:\Database\scripts\script.sql
Example output line from this command:
WARNING: column "identity" has type "unknown"
DETAIL: Proceeding with relation creation anyway.
Specifying the file with the -f flag makes no difference.
I can manually run the batch file on my development machine and it produces the expected database regardless of what errors or messages show on the command prompt.
So ultimately I need all psql commands in my batch files to run silently.
psql COMMAND &> output.txt
Or, using your example command:
psql -d database -q < C:\Database\scripts\script.sql &> output.txt
use psql -o flag to send the command output to the filename you wish or /dev/null if you don't care about it.
The -q option will not prevent the query output.
-q, --quiet run quietly (no messages, only query output)
to avoid the output you have to send the query result to a file
psql -U username -d db_name -pXXXX -c "SELECT * FROM table_name LIMIT 5;" > C:\test.csv
use 1 > : create new file each time
use 2 >> : will create and keep adding
I've created a small Bash script that does a MySQL data dump. Because the dump can be fairly large I put the process in the background and then wait for either an error or the log to show up in the file system. I have the following code:
mysqldump main_db > /loc/to/dmp/file.sql 2>/loc/to/error/log/file.log &
The problem is that I get a '/loc/to/error/log/file.log' file the size of 0 (which I presume means no real error) sometimes when this command is run, which kills the process, even though there is no error.
I'm not sure why the STDERR would write a file when there was no data to write. Is this because of the & background process?
The redirected files are set up before your script is executed by the executing shell.
I.e. after having parsed your command which includes redirected stdout/stderr, the shell forks, opens(creates the files if they don't exists). attach the opened filedescriptors to filedescriptor 1 and 2 (stdout/err respectivly) and then executes the actual command.
The redirection file is created whether or not any data is ever written to it. Whichever process is watching the error log should check for non-zero filesize, not existence.
A possible easy workaround:
mysqldump main_db > file.sql 2> errors.log ; [ -s errors.log ] || rm -f errors.log
OR
(short readable easily adjustable script with timestamp)
OUTPUT="/loc/to/dmp/`date +%F.%T`.sql"
ERRORS="$OUTPUT.ERRORS"
mysqldump main_db > $OUTPUT 2> $ERRORS
[ -s $ERRORS ] || rm -f $ERRORS