I am after a bash script which I can use to trigger a delta import of XML files via CRON. After a bit of digging and modification I have this:
#!/bin/bash
# Bash to initiate Solr Delta Import Handler
# Setup Variables
urlCmd='http://localhost:8080/solr/dataimport?command=delta-import&clean=false'
statusCmd='http://localhost:8080/solr/dataimport?command=status'
outputDir=.
# Operations
wget -O $outputDir/check_status_update_index.txt ${statusCmd}
2>/dev/null
status=`fgrep idle $outputDir/check_status_update_index.txt`
if [[ ${status} == *idle* ]]
then
wget -O $outputDir/status_update_index.txt ${urlCmd}
2>/dev/null
fi
Can I get any feedback on this? Is there a better way of doing it? Any optimisations or improvements would be most welcome.
This certainly looks usable. Just to confirm, you intend to run this ever X minutes from your crontab? That seems reasonsable.
The only major quibble (IMHO) is discarding STDERR information with 2>/dev/null. Of course it depends on what are your expectations for this system. If this is for a paying customer or employer, do you want to have to explain to the boss, "gosh, I didn't know I was getting error message 'Cant connect to host X' for the last 3 months because we redirect STDERR to /dev/null"! If this is for your own project, and your monitoring the work via other channels, then not so terrible, but why not capture STDERR to file, and if check that there are no errors. as a general idea ....
myStdErrLog=/tmp/myProject/myProg.stderr.$(/bin/date +%Y%m%d.%H%M)
wget -O $outputDir/check_status_update_index.txt ${statusCmd} 2> ${myStdErrLog}
if [[ ! -s ${myStdErrLog} ]] ; then
mail -s "error on myProg" me#myself.org < ${myStdErrLog}
fi
rm ${myStdErrLog}
Depending on what curl includes in its STDERR output, you may need filter what is in the StdErrLog to see if there are "real" error messages that you need to have sent to you.
A medium quibble is your use backticks for command substitution, if you're using dbl-sqr-brackets for evaluations, then why not embrace complete ksh93/bash semantics. The only reason to use backticks is if you think you need to be ultra-backwards compatible and that you'll be running this script under the bourne shell (or possibly one of the stripped down shells like dash).Backticks have been deprecated in ksh since at least 1993. Try
status=$(fgrep idle $outputDir/check_status_update_index.txt)
The $( ... ) form of command substitution makes it very easy to nest multiple cmd-subtitutions, i.e. echo $(echo one $(echo two ) ). (Bad example, as the need to nest cmd-sub is pretty rare, I can't think of a better example right now).
Depending on your situation, but in a large production environement, where new software is installed to version numbered directories, you might want to construct your paths from variables, i.e.
hostName=localhost
portNum=8080
SOLRPATH=/solr
SOLRCMD='delta-import&clean=false"
urlCmd='http://${hostName}:${portNum}${SOLRPATH}/dataimport?command=${SOLRCMD}"
The final, minor quibble ;-). Are you sure ${status} == *idle* does what you want?
Try using something like
case "${status}" in
*idle* ) .... ;;
* ) echo "unknown status = ${status} or similar" 1>&2 ;;
esac
Yes, your if ... fi certainly works, but if you want to start doing more refined processing of infomation that you put in your ${status} variable, then case ... esac is the way to go.
EDIT
I agree with #alinsoar that 2>/dev/null on a line by itself will be a no-op. I assumed that it was a formatting issue, but looking in edit mode at your code I see that it appears to be on its own line. If you really want to discard STDERR messages, then you need cmd ... 2>/dev/null all on one line OR as alinsoar advocates, the shell will accept redirections at the front of the line, but again, all on one line ;-!.
IHTH
Related
I would like prevent other terminals from running a certain script whenever another terminal is running it however in bash but I'm not quite sure on how I would be able to go about in doing it. Any help or tip could be greatly appreciated!
In example:
When that script is being run on another terminal, all other terminals would be unable to run that certain script as well. And display a message "UNDER MAINTENANCE".
You can use the concept of a "lockfile." For example:
if [ -f ~/.mylock ]; then
echo "UNDER MAINTENANCE"
exit 1
fi
touch ~/.mylock
# ... the rest of your code
rm ~/.mylock
To get fancier/safer, you can "trap" the EXIT signal to remove it automatically at the end:
trap 'rm ~/.mylock' EXIT
Use flock and put this on top of your script:
if ! flock -xn /path/to/lockfile ; then
echo "script is already running."
echo "Aborting."
exit 1
fi
Note: path/to/lockfile could be the path to your script. Doing so would avoid to create an extra file.
To avoid race conditions, you could use flock(1) along with a
lock file. There is one flock(1) implementation
which claims to work on Linux, BSD, and OS X. I haven't seen one
explicitly for Unix.
There is some interesting discussion here.
UPDATE:
I found a really clever way from Randal L. Schwartz here. I really like this one. It relies on having flock(1) and bash, and it uses the script itself as its own lockfile. Check this out:
/usr/local/bin/onlyOne is a script to obtain the lock
#!/bin/bash
exec 200< $0
if ! flock -n 200; then
echo "there can be only one"
exit 1
fi
Then myscript uses onlyOne to obtain the lock (or not):
#!/bin/bash
source /usr/local/bin/onlyOne
# The real work goes here.
echo "${BASHPID} working"
sleep 120
I would like to setup inotifywait so that it monitors a folder and when something is copied to this folder (lsyncd which uses rsync) I would like inotifywait to sit tight and wait until rsync is done before calling a script to process the new folder.
I have been researching online to see if someone is doing this but I am not finding much.
I'm not the most well versed with bash scripting though I understand some basics.
Here is a little script I found that pauses for a second but it still triggers a dozen events per transfer:
EVENTS="CLOSE_WRITE,MOVED_TO"
if [ -z "$1" ]; then
echo "Usage: $0 cmd ..."
exit -1;
fi
inotifywait -e "$EVENTS" -m -r --format '%:e %f' . | (
WAITING="";
while true; do
LINE="";
read -t 1 LINE;
if test -z "$LINE"; then
if test ! -z "$WAITING"; then
echo "CHANGE";
WAITING="";
fi;
else
WAITING=1;
fi;
done) | (
while true; do
read TMP;
echo $#
$#
done
)
I would be happy to provide more details or information.
Thank you.
Depending on what action you want to take, you may want to take a look at the tools provided by Watchman.
There are two that might be most useful to you:
If you want to initiate some action after the files are synced up, you may want to try using watchman-make. This is most appropriate if the action is to run a tool like make where the tool itself will look over the tree and produce its output (in other words: where you don't need to pass the precise list of changed files directly to your tool). You can have it run some other tool instead of make. There is a --settle option that you can use to have it wait a few moments after the latest file change notification before executing your tool.
watchman-make --make='process-folder.sh' -p '**/*.*'
watchman-wait is more closely related to inotifywait. It also waits for changes to settle before reporting files as changed, but because this tool doesn't coalesce multiple different file changes into a single event, the settle period is configured as a property of the tree being watched rather than as a command line parameter
Disclaimer: I'm the creator of Watchman
Question,
I want to have a bash script that will have a global variable that can be incremented from other bash scripts.
Example:
I have a script like the following:
#! /bin/bash
export Counter=0
for SCRIPT in /Users/<user>/Desktop/*sh
do
$SCRIPT
done
echo $Counter
That script will call all the other bash scripts in a folder and those scripts will have something like the following:
if [ "$Output" = "$Check" ]
then
echo "OK"
((Counter++))
I want it to then increment the $Counter variable if it does equal "OK" and then pass that value back to the initial batch script so I can keep that counter number and have a total at the end.
Any idea on how to go about doing that?
Environment variables propagate in one direction only -- from parent to child. Thus, a child process cannot change the value of an environment variable set in their parent.
What you can do is use the filesystem:
export counter_file=$(mktemp "$HOME/.counter.XXXXXX")
for script in ~user/Desktop/*sh; do "$script"; done
...and, in the individual script:
counter_curr=$(< "$counter_file" )
(( ++counter_curr ))
printf '%s\n' "$counter_curr" >"$counter_file"
This isn't currently concurrency-safe, but your parent script as currently written will never call more than one child at a time.
An even easier approach, assuming that the value you're tracking remains relatively small, is to use the file's size as a proxy for the counter's value. To do this, incrementing the counter is as simple as this:
printf '\n' >>"$counter_file"
...and checking its value in O(1) time -- without needing to open the file and read its content -- is as simple as checking the file's size; with GNU stat:
counter=$(stat -f %z "$counter_file")
Note that locking may be required for this to be concurrency-safe if using a filesystem such as NFS which does not correctly implement O_APPEND; see Norman Gray's answer (to which this owes inspiration) for a working implementation.
You could source the other scripts, which means they're not running in a sub-process but "inline" in the calling script like this:
#! /bin/bash
export counter=0
for script in /Users/<user>/Desktop/*sh
do
source "$script"
done
echo $counter
But as pointed out in the comments i'd only advise to use this approach if you control the called scripts yourself. If they for example exit or have variables clashing with each other, bad things could happen.
As described, you can't do this, since there isn't anything which corresponds to a ‘global variable’ for shell scripts.
As the comment suggests, you'll have to use the filesystem to communicate between scripts.
One simple/crude way of doing what you describe would be to simply have each cooperating script append a line to a file, and the ‘global count’ is the size of this file:
#! /bin/sh -
echo ping >>/tmp/scriptcountfile
then wc -l /tmp/scriptcountfile is the number of times that's happened. Of course, there's a potential race condition there, so something like the following would sequence those accesses:
#! /bin/sh -
(
flock -n 9
echo 'do stuff...'
echo ping >>/tmp/stampfile
) 9>/tmp/lockfile
(the flock command is available on Linux, but isn't portable).
Of course, then you can start to do fancier things by having scripts send stuff through pipes and sockets, but that's going somewhat over the top.
I want to log stdout from some programs to loggly. There are a few simple utilities out there that I've found (e.g. https://github.com/meatballhat/loggly-pipe and https://github.com/segmentio/loggly-cat), but they seem like they're overkill.
Could I get away with doing something this simple:
log.sh:
#!/bin/bash
while read line
do
echo "$line"
curl -H "content-type:text/plain" -d "$line" https://logs-01.loggly.com/inputs/<my-token>/tag/tag1,tag2/ >/dev/null 2>&1
done < /dev/stdin
Then I run my program and pipe it to my loggly logging script:
./my_script.sh | ./log.sh
This seems to work okay, but I wonder if all the complicated-ness of the other solutions out there is necessary for some reason?
Could anything go wrong here?
Thanks!
Think about what your script does. It runs curl once per line of input.
Think about what that means.
If you log 10K lines, you'll then spawn 10K processes. Which will initiate 10K TCP connections. This is a massive waste of computing resources.
Also, you don't handle errors at all (in fact, you actively hide them by sending curl's stderr to /dev/null!). This means the script is not only inefficient, it is unreliable.
I recommend starting all Bash scripts with set -eu to exit on unhandled errors, but that's just one step, not a complete fix for the above.
I've been looking for a way to log some more detailed information about the history of commands. My main purpose is to have a rough log of commands that were issued in order to build rough server timelines when debugging issues with our application. It is not for highly detailed auditing purposes. I came across this post which suggested an excellent way to modify PROMPT_COMMAND to augment the history log with additional information about each command. It suggests adding the following to the ~/.bashrc file:
export PROMPT_COMMAND='hpwd=$(history 1); hpwd="${hpwd# *[0-9]* }"; if [[ ${hpwd%% *} == "cd" ]]; then cwd=$OLDPWD; else cwd=$PWD; fi; hpwd="${hpwd% ### *} ### $cwd"; history -s "$hpwd"'
This works awesome, except that it only happens when the PS1 prompt is issued. Is there a way to enhance this to work with non-interactive shells (I think that's the correct term)?
For example, I would like:
ssh host "ls | grep home"
To create an entry for ls | grep home on host as well, but since this isn't done through a PS1 prompt the linked solution falls short.
I have looked into auditd a little. This is a great utility, but the level of detail was way more than I need. I could have parsed the logs pretty easily, but pipes, redirects, loops become a nightmare to rebuild sanely into something pretty like what history already reports.
A simple wrapper around ssh would seem like a straightforward way to achieve this.
shout () {
local host
host=$1
shift
ssh "$host" <<____HERE
echo "$#" >>\$HOME/.shout-history
bash -c "$#"
____HERE
}
Or if you want the wrapper to run locally,
shout () {
local host
host=$1
shift
echo "$#" >>$HOME/.shout-history
ssh "$host" "$#"
}
I called this shout in opposition to ssh which ought to be, you know, quiet. See also this. Of course, if you are admin, you could simply move /usr/bin/ssh to someplace obscure and force your users to run a /usr/local/bin/ssh with contents similar to the above. It's easy enough to bypass by a knowledgeable user, but if you're really draconian, there are ways to make it harder.
If you are the admin of the remote host, you could force all users to run /usr/local/bin/shout as their shell, for example, and populate it with something more or less similar.
#!/bin/bash
echo "$#" >>/home/root/im.in.ur.sh.reading.ur.seekrit.cmds.lol
exec /bin/bash -c "$#"
Just make sure the transcript file is world writable but not world readable.