I have about a ton of scripts using the following function:
# Copies files over using FTP.
# Configurations set at the beggining of the script
# #param $1 = FTP Host
# $2 = FTP User
# $3 = FTP User password
# $4 = Source file name
# $5 = destination directory
# $6 = local directory
doftp() {
log_message_file "INFO" "Starting FTP"
ftp_hst=$1
ftp_usr=$2
ftp_pwd=$3
sourcefile=$4
destdir=$5
locdir=$6
ftp -nv $FTPH << EOF 2> ftp.err.$$
quote USER $ftp_usr
quote PASS $ftp_pwd
cd $destdir
lcd $locdir
bin
put $sourcefile
bye
EOF
if [ "$(wc ftp.err.$$|cut -d" " -f8)" != 0 ] ; then
log_message_file "ERROR" "Problem uploading files: $(cat ftp.err.$$)"
else
log_message_file "INFO" "FTP finished"
fi
rm ftp.err.$$
}
It works, does it's job, unless ftp fails. Fortunately for me, the scripts are quite precise and the FTP almost never fails. But this is one of these rare moments when one has the chance (time) to go back and review code marked in the TODO list. Only problem is that I'm not too sure on how to improve it... I'd take you guys recommendations on what to change there.
One obvious problem is the error parsing from the ftp, which is a totally lame. But I'll also take thoughts in other parts of the function :)
Worth mentioning this is run in a AIX server? Oh, and no, I cannot use SFTP :(
Thanks for any input!
ps.: log_message_file is just a basic logging... has no effect in the function.
good documentation
good variable names
good indentation, you might want to read about <<-EOF (the '-' being the item I intend to highlight with this comment. Using that feature allows you to indent (must use tab chars) the whole here-document to match indenting of rest of script.
good use of tmp filename with $$ (depending on how much this function gets used, you might want to append the parent script name as part of the tmp name, to further disambiguate, but low priority)
good use of $(cat ftp.err.$$) i.e. actually showing the error message, rather than just a message like 'error occured' (I see this all the time, what error? what was the msg?!)
you could extend the ftp service to use mput, but then you have to understand the vagaries of your particular ftp client AND make a note to your self that any time there is a change of ftp client you'll need to check if your mput ${fileNames} variable still works as you expect.
possibly the one place to think about improvement would be to use a case statement to parse the STDERR output, but again, the added benefit may not be worth the maintenance cost in the future.
errMsgs="$(cat ftp.err.$$)"
case "${errMsgs}" in
*warningStrings* ) print "warning found, msg was ${errMsg} ;;
*errorStrings* ) print "error found, msg was ${errMsg} ;;
*fatalStrings* ) pring "fatal error found, can't continue, msg was ${errMsg} ;;
esac
I hope this helps.
Related
i have config file like this for example:
# Blah blah, this is sample config file blah blah
# Something more blah blah
value1=YES
value2=something
#value3=boom
# Blah blah
valueN=4145
And i want to make script to read and edit config files like this. I thinking about make a menu with groups of config options, then after write an option console output will be like this:
Group of funny options (pick option to change value):
1. value1=YES
2. value2=something
3. [disabled]value3=boom
After picking 1 for exaple i can change value1 from YES to NO or disable and activate other (hash unhash) plus adding new variables to the end of file. Then in the end save all changes in this config file. Any tips what i need to use? Actually trying with read line + awk to skip # lines (with space), but still i have problem to get all this variables and making changes in config file. I will be grateful for your help.
Edit.
while read line
do
echo $line | awk '$1' != "#" && / / { print $1 $3 }'
done < config.conf
Thinking about this for now to read informations what i want. Plus i'm gonna use something like this to change values:
sed -c -i "s/("one" *= *).*/\1$two/" config.conf
I have completly no idea how i can get this variables to my script and use it like i write before. Actually i search for any tips, not someone who write this script for me. I'm beginner at linux scripting :V
I would recommend to abstain from such an, seemingly generic configuration program, because the comments might contain important informations about the current value and will be outdated, if the values change, while the comments don't.
Second problem is, that I would expect, if activating an entry is possible, deactivating it should be possible too. So now you have 2 options what to do with each value.
Third problem: In most cases, guessing a type by the value might work. YES seems to be a boolean, 47 an int, foobar a name - or is it a file? - but often a wider type is possible too, so YES can be just a string or a file, 47.3 might be valid where 47 is or might be not and so on.
However, for experimenting and trying things out, select and grep might be a start:
select line in $(grep "=" sample.conf) "write" "abort"
do
case $line in
"write") echo write; break ;;
"abort") echo abort; break ;;
'#'*=*) echo activate $line;;
*=[0-9]*) echo int value $line;;
*=YES|NO) echo boolean value $line;;
*) echo text value $line ;;
esac
done
Instead of 'echo intvalue $line' you would probably call a function "intconfigure" where only int values are accepted. For "write", you would write back to the file, but I omitted, conserving the comments without assignment and sorting them in again at the right position has to be done, which isn't trivial, given the opportunity to activate or deactivate comments.
But read up on the select command in shell and try it out and see how far you come.
If you think you have reached a usable solution, use this for all your configuration files privately and see, whether you prefer it over using a simple editor or not.
Old stuff:
Background:
- Ultimate goal is to put a script in my .bash_profile that warns me by changing text color if I'm typing a commit message and it gets too
long (yes I'm aware vim has something like this).
Progress:
- I found the read -n option which led me to write this:
while true; do
# This hits at the 53rd character
read -rn53 input
# I have commit aliased to gc so the if is just checking if I'm writing a commit
if [ "${input:0:2}" = "gc" ]; then
printf "\nMessage getting long"
fi
done
Question:
- However, running this takes the user out of the bash prompt. I need a way to do something like this while at a normal prompt. I can't find
information on anything like this. Does that mean it's not possible?
Or am I just going about it the wrong way?
New progress:
I found the bind -x option which led me to write this:
check_commit() {
if [ "${READLINE_LINE:0:13}" == 'git commit -m' ] && [ ${#READLINE_LINE} -gt 87 ]; then
echo "Commit is $((${#READLINE_LINE} - 87)) characters too long!"
fi
READLINE_LINE="$READLINE_LINE$1"
READLINE_POINT=$(($READLINE_POINT+1))
}
bind -x '"\"": check_commit "\""'
It listens for a double quote and if I'm writing a long commit message tells me how many characters I am over the limit. Also puts the character I typed into the current line since it is eaten by the bind.
New question:
Now I just need a way to put in a regex, character list or at least a variable instead of \" so I can listen on more keys (Yes, I'm aware bind -x probably wasn't intended to be used this way. I can check performance/footprint/stability myself). I tried "$char", "${char}", "$(char)" and a few other things, but none seem to work. What is the correct approach here ?
AFAIK, not possible in a sane way if you want this to happen during your normal prompt (when PROMPT_COMMAND and PS1 are evaluated). That would involved binding a custom compiled readline function for every insert-self and alike.
If you want this to happen in a script using prompt builtin, this is crudely possible with a loop of
read -e -i $(munge_buf $buf) -n $(buf_warn_len $buf) -p $(buf_warning $buf) buf
like commands. This will allow you to create munge_buf() to alter the currently typed text if needed, buf_warn_len() to calculate a new len to warn at (which may be very large if warning was already displayed), and buf_warn_msg() to derive a warning message based upon the buffer.
I have a bash script that calls a program which generates a humongous amount of output. A lot of this data is coming from a Python package that I have not created and whose output I can't really control, nor interests me.
I tried to filter the output generated by that external Python package and redirect the "cleaned" output to a log file. If I used regular pipes and grep expressions, I lost many chunks of information. I read that is something that can actually happen with the redirections (1 and 2).
In order to fix that, I made the redirections like this:
#!/bin/bash
regexTxnFilterer="\[txn\.-[[:digit:]]+\]"
regexThreadPoolFilterer="\[paste\.httpserver\.ThreadPool\]"
bin/paster serve --reload --pid-file="/var/run/myServer//server.pid" parts/etc/debug.ini 2>&1 < "/dev/null" | while IFS='' read -r thingy ; do
if [[ ! "$thingy" =~ $regexTxnFilterer ]] && [[ ! "$thingy" =~ $regexThreadPoolFilterer ]]; then
echo "$thingy" >> "/var/log/myOutput.log"
fi
done
Which doesn't lose any information (at least not that I could tell) and filters the strings I don't need (using the two regular expressions above).
The issue is that it has rendered the application (the bin/paster thing I'm executing) unbearably slow. Is there any way to achieve the same effect but with a better performance?
Thank you in advance!
Update #2012-04-13: As shellter pointed out in one of the comments to this question, it may be useful to provide examples of the outputs I want to filter. Here's a bunch of them:
2012-04-13 19:30:37,996 DEBUG [txn.-1220917568] new transaction
2012-04-13 19:30:37,997 DEBUG [txn.-1220917568] commit <zope.sqlalchemy.datamanager.SessionDataManager object at 0xbf4062c>
2012-04-13 19:30:37,997 DEBUG [txn.-1220917568] commit
Starting server in PID 18262.
2012-04-13 19:30:38,292 DEBUG [paste.httpserver.ThreadPool] Started new worker -1269716112: Initial worker pool
2012-04-13 19:33:08,158 DEBUG [txn.-1244144784] new transaction
2012-04-13 19:33:08,158 DEBUG [txn.-1244144784] commit
2012-04-13 19:32:06,980 DEBUG [paste.httpserver.ThreadPool] Added task (0 tasks queued)
2012-04-13 19:32:06,980 INFO [paste.httpserver.ThreadPool] kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
There's a few more different messages involving the ThreadPool though, but I couldn't catch any.
For one thing -- you're reopening the log file every time you want to append a line. That's silly.
Instead of this:
while ...; do
echo "foo" >>filename
done
Do this (which opens the output file on a new, non-stdout file handle, such that you still have a clear line to stdout should you wish to write to it):
exec 4>>filename
while ...; do
echo "foo" >&4
done
It's also possible to redirect stdout for the whole loop:
while ...; do
echo "foo"
done >filename
...notably, this will impact more than just the "echo" line, and thus have slightly different semantics from the original.
Or, better yet -- Configure the Python logging module to filter output to only what you care about, and don't bother with shell-script postprocessing at all.
If the version of Paste you're using is sufficiently similar to modern Pyramid, you can put this in your ini file (currently parts/etc/debug.ini):
[logger_paste.httpserver.ThreadPool]
level = INFO
[logger_txn]
level = INFO
...and anything below INFO level (including the DEBUG messages) will be excluded.
It may be faster to use a grep-based solution to this
#!/bin/bash
regexTxnFilterer="\[txn\.-[[:digit:]]+\]"
regexThreadPoolFilterer="\[paste\.httpserver\.ThreadPool\]"
bin/paster serve --reload --pid-file="/var/run/myServer//server.pid" parts/etc/debug.ini 2>&1 < "/dev/null" | grep -vf <(echo "$regexTxnFilterer"; echo "$regexThreadPoolFilterer") >> "/var/log/myOutput.log"
Your loop may be slow because the echo "$thingy" >> "/var/log/myOutput.log" line is opening and closing the log file every time it executes. I wouldn't expect there to be a big performance difference between grep's regex matching and bash's, but if there was it wouldn't surprise me.
Late Edit
There's a far simpler way to fix the performance issue caused by opening/closing the output once per line. Why this didn't occur to me before, I have no idea. Just move the >> to outside your loop
#!/bin/bash
regexTxnFilterer="\[txn\.-[[:digit:]]+\]"
regexThreadPoolFilterer="\[paste\.httpserver\.ThreadPool\]"
bin/paster serve --reload --pid-file="/var/run/myServer//server.pid" parts/etc/debug.ini 2>&1 < "/dev/null" | while IFS='' read -r thingy ; do
if [[ ! "$thingy" =~ $regexTxnFilterer ]] && [[ ! "$thingy" =~ $regexThreadPoolFilterer ]]; then
echo "$thingy"
fi
done >> "/var/log/myOutput.log"
I can't see any compelling reason why this would be either faster or slower than the grep solution, but it's a lot closer to the original code and a little less cryptic.
During the process of writing a script, I will use the command's output in varying ways, and to different degrees - in order to troubleshoot the task at hand.. For example, in this snippet, which reads an Application's icon resource and returns whether or not it has the typical .icns extension...
icns=`defaults read /$application/Contents/Info CFBundleIconFile`
if ! [[ $icns =~ ^(.*)(.icns)$ ]]; then
echo -e $icns "is NOT OK YOU IDIOT! **** You need to add .icns to "$icns"."
else
echo -e $icns "\t Homey, it's cool. That shits got its .icns, proper."
fi
Inevitably, as each bug is squashed, and the stdout starts relating more to the actual function vs. the debugging process, this feedback is usually either commented out, silenced, or deleted - for obvious reasons.
However, if one wanted to provide a simple option - either hardcoded, or passed as a parameter, to optionally show some, all, or none of "this kind" of message at runtime - what is the best way to provide that simple functionality? I am looking to basically duplicate the functionality of set -x but instead of a line-by rundown, it would only print the notifications that I had architected specificically.
It seems excessive to replace each and every echo with an if that checks for a debug=1|0, yet I've been unable to find a concise explanation of how to implement a getopts/getopt scheme (never can remember which one is the built-in), etc. in my own scripts. This little expression seemed promising, but there is very little documentation re: 2>$1 out there (although I'm sure this is key to this puzzle)
[ $DBG ] && DEBUG="" || DEBUG='</dev/null'
check_errs() {
# Parameter 1 is the return code Para. 2 is text to display on failure.
if [ "${1}" -ne "0" ]; then
echo "ERROR # ${1} : ${2}"
else
echo "SUCESSS "
fi }
Any concise and reusable tricks to this trade would be welcomed, and if I'm totally missing the boat, or if it was a snake, and it would be biting me - I apologize.
One easy trick is to simply replace your "logging" echo comamnd by a variable, i.e.
TRACE=:
if test "$1" = "-v"; then
TRACE=echo
shift
fi
$TRACE "You passed the -v option"
You can have any number of these for different types of messages if you wish so.
you may check a common open source trace library with support for bash.
http://sourceforge.net/projects/utalm/
https://github.com/ArnoCan/utalm
WKR
Arno-Can Uestuensoez
I'm writing shell scripts where quite regularly some stuff is written
to a file, after which an application is executed that reads that file. I find that through our company the network latency differs vastly, so a simple sleep 2 for example will not be robust enough.
I tried to write a (configurable) timeout loop like this:
waitLoop()
{
local timeout=$1
local test="$2"
if ! $test
then
local counter=0
while ! $test && [ $counter -lt $timeout ]
do
sleep 1
((counter++))
done
if ! $test
then
exit 1
fi
fi
}
This works for test="[ -e $somefilename ]". However, testing existence is not enough, I sometimes need to test whether a certain string was written to the file. I tried
test="grep -sq \"^sometext$\" $somefilename", but this did not work. Can someone tell me why?
Are there other, less verbose options to perform such a test?
You can set your test variable this way:
test=$(grep -sq "^sometext$" $somefilename)
The reason your grep isn't working is that quotes are really hard to pass in arguments. You'll need to use eval:
if ! eval $test
I'd say the way to check for a string in a text file is grep.
What's your exact problem with it?
Also you might adjust your NFS mount parameters, to get rid of the root problem. A sync might also help. See NFS docs.
If you're wanting to use waitLoop in an "if", you might want to change the "exit" to a "return", so the rest of the script can handle the error situation (there's not even a message to the user about what failed before the script dies otherwise).
The other issue is using "$test" to hold a command means you don't get shell expansion when actually executing, just evaluating. So if you say test="grep \"foo\" \"bar baz\"", rather than looking for the three letter string foo in the file with the seven character name bar baz, it'll look for the five char string "foo" in the nine char file "bar baz".
So you can either decide you don't need the shell magic, and set test='grep -sq ^sometext$ somefilename', or you can get the shell to handle the quoting explicitly with something like:
if /bin/sh -c "$test"
then
...
Try using the file modification time to detect when it is written without opening it. Something like
old_mtime=`stat --format="%Z" file`
# Write to file.
new_mtime=$old_mtime
while [[ "$old_mtime" -eq "$new_mtime" ]]; do
sleep 2;
new_mtime=`stat --format="%Z" file`
done
This won't work, however, if multiple processes try to access the file at the same time.
I just had the exact same problem. I used a similar approach to the timeout wait that you include in your OP; however, I also included a file-size check. I reset my timeout timer if the file had increased in size since last it was checked. The files I'm writing can be a few gig, so they take a while to write across NFS.
This may be overkill for your particular case, but I also had my writing process calculate a hash of the file after it was done writing. I used md5, but something like crc32 would work, too. This hash was broadcast from the writer to the (multiple) readers, and the reader waits until a) the file size stops increasing and b) the (freshly computed) hash of the file matches the hash sent by the writer.
We have a similar issue, but for different reasons. We are reading s file, which is sent to an SFTP server. The machine running the script is not the SFTP server.
What I have done is set it up in cron (although a loop with a sleep would work too) to do a cksum of the file. When the old cksum matches the current cksum (the file has not changed for the determined amount of time) we know that the writes are complete, and transfer the file.
Just to be extra safe, we never overwrite a local file before making a backup, and only transfer at all when the remote file has two cksums in a row that match, and that cksum does not match the local file.
If you need code examples, I am sure I can dig them up.
The shell was splitting your predicate into words. Grab it all with $# as in the code below:
#! /bin/bash
waitFor()
{
local tries=$1
shift
local predicate="$#"
while [ $tries -ge 1 ]; do
(( tries-- ))
if $predicate >/dev/null 2>&1; then
return
else
[ $tries -gt 0 ] && sleep 1
fi
done
exit 1
}
pred='[ -e /etc/passwd ]'
waitFor 5 $pred
echo "$pred satisfied"
rm -f /tmp/baz
(sleep 2; echo blahblah >>/tmp/baz) &
(sleep 4; echo hasfoo >>/tmp/baz) &
pred='grep ^hasfoo /tmp/baz'
waitFor 5 $pred
echo "$pred satisfied"
Output:
$ ./waitngo
[ -e /etc/passwd ] satisfied
grep ^hasfoo /tmp/baz satisfied
Too bad the typescript isn't as interesting as watching it in real time.
Ok...this is a bit whacky...
If you have control over the file: you might be able to create a 'named pipe' here.
So (depending on how the writing program works) you can monitor the file in an synchronized fashion.
At its simplest:
Create the named pipe:
mkfifo file.txt
Set up the sync'd receiver:
while :
do
process.sh < file.txt
end
Create a test sender:
echo "Hello There" > file.txt
The 'process.sh' is where your logic goes : this will block until the sender has written its output. In theory the writer program won't need modifiying....
WARNING: if the receiver is not running for some reason, you may end up blocking the sender!
Not sure it fits your requirement here, but might be worth looking into.
Or to avoid synchronized, try 'lsof' ?
http://en.wikipedia.org/wiki/Lsof
Assuming that you only want to read from the file when nothing else is writing to it (ie, the writing process has finished) - you could check whether nothing else has file handle to it ?