My requirement is to replace file only when it is not being accessed. I have following snippet:
if [ -f file ]
then
while true
do
if [ -n "$(fuser "file")" ]
then
echo "file is in use..."
else
echo "file is free..."
break
fi
done
fi
{
flock -x 3
mv newfile file
} 3>file
But I have a doubt that I am not handling concurrency properly. Please give some insights and possible way to achieve this.
Thanks.
My requirement is to replace file only when it is not being accessed.
Getting requirements right can be hard. In case your actual requirement is the following, you can boil down the whole script to just one command.
My guess on the actual requirement (not as strict as the original):
Replace file without disturbing any programs reading/writing file.
If this is the case, you can use a very neat behavior: In Unix-like systems file descriptors always point to the file (not path) for which they where opened. You can move or even delete the corresponding path. See also How do the UNIX commands mv and rm work with open files?.
Example:
Open a terminal and enter
i=1; while true; do echo $((i++)); sleep 1; done > file &
tail -f file
The first command writes output to file and runs in the background. The second command reads the file and continues to print its changing content.
Open another terminal and move or delete file, for instance with
mv file file2
echo overwritten > otherFile
mv otherFile file2
rm file2
echo overwritten > file
echo overwritten > file2
While executing these commands have a look at the output of tail -f in the first terminal – it won't be affected by any of these commands. You will never see overwritten.
Solution For New Requirement:
Because of this behavior you can replace the whole script with just one mv command:
mv newfile file
Consider lsof.
mvWhenClear() {
while [[ -f "$1" ]] && lsof "$1"
do sleep $delay
done
mv "$1" "$2" # still allows race condition
}
I'd like to write a shell script which checks if a certain file, archived_sensor_data.json, exists, and if so, deletes it. Following http://www.cyberciti.biz/tips/find-out-if-file-exists-with-conditional-expressions.html, I've tried the following:
[-e archived_sensor_data.json] && rm archived_sensor_data.json
However, this throws an error
[-e: command not found
when I try to run the resulting test_controller script using the ./test_controller command. What is wrong with the code?
You're missing a required space between the bracket and -e:
#!/bin/bash
if [ -e x.txt ]
then
echo "ok"
else
echo "nok"
fi
Here is an alternative method using ls:
(ls x.txt && echo yes) || echo no
If you want to hide any output from ls so you only see yes or no, redirect stdout and stderr to /dev/null:
(ls x.txt >> /dev/null 2>&1 && echo yes) || echo no
The backdrop to my solution recommendation is the story of a friend who, well into the second week of
his first job, wiped half a build-server clean. So the basic task is to figure out if a file exists,
and if so, let's delete it. But there are a few treacherous rapids on this river:
Everything is a file.
Scripts have real power only if they solve general tasks
To be general, we use variables
We often use -f force in scripts to avoid manual intervention
And also love -r recursive to make sure we create, copy and destroy in a timely fashion.
Consider the following scenario:
We have the file we want to delete: filesexists.json
This filename is stored in a variable
<host>:~/Documents/thisfolderexists filevariable="filesexists.json"
We also hava a path variable to make things really flexible
<host>:~/Documents/thisfolderexists pathtofile=".."
<host>:~/Documents/thisfolderexists ls $pathtofile
filesexists.json history20170728 SE-Data-API.pem thisfolderexists
So let's see if -e does what it is supposed to. Does the files exist?
<host>:~/Documents/thisfolderexists [ -e $pathtofile/$filevariable ]; echo $?
0
It does. Magic.
However, what would happen, if the file variable got accidentally be evaluated to nuffin'
<host>:~/Documents/thisfolderexists filevariable=""
<host>:~/Documents/thisfolderexists [ -e $pathtofile/$filevariable ]; echo $?
0
What? It is supposed to return with an error... And this is the beginning of the story how that entire
folder got deleted by accident
An alternative could be to test specifically for what we understand to be a 'file'
<host>:~/Documents/thisfolderexists filevariable="filesexists.json"
<host>:~/Documents/thisfolderexists test -f $pathtofile/$filevariable; echo $?
0
So the file exists...
<host>:~/Documents/thisfolderexists filevariable=""
<host>:~/Documents/thisfolderexists test -f $pathtofile/$filevariable; echo $?
1
So this is not a file and maybe, we do not want to delete that entire directory
man test has the following to say:
-b FILE
FILE exists and is block special
-c FILE
FILE exists and is character special
-d FILE
FILE exists and is a directory
-e FILE
FILE exists
-f FILE
FILE exists and is a regular file
...
-h FILE
FILE exists and is a symbolic link (same as -L)
Internally, the rm command must test for file existence anyway,
so why add another test? Just issue
rm filename
and it will be gone after that, whether it was there or not.
Use rm -f is you don't want any messages about non-existent files.
If you need to take some action if the file does NOT exist, then you must test for that yourself. Based on your example code, this is not the case in this instance.
If you're using a NFS, "test" is a better solution, because you can add a timeout to it, in case your NFS is down:
time timeout 3 test -f
/nfs/my_nfs_is_currently_down
real 0m3.004s <<== timeout is taken into account
user 0m0.001s
sys 0m0.004s
echo $?
124 <= 124 means the timeout has been reached
A "[ -e my_file ]" construct will freeze until the NFS is functional again:
if [ -e /nfs/my_nfs_is_currently_down ]; then echo "ok" else echo "ko" ; fi
<no answer from the system, my session is "frozen">
You could also uses stat :
stat /
File: /
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd01h/64769d Inode: 2 Links: 26
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2009-01-01 02:00:00.000000000 +0200
Modify: 2009-01-01 02:00:00.000000000 +0200
Change: 2009-01-01 02:00:00.000000000 +0200
Birth: -
On a path that doesn't exist, you will get:
stat /aaa
stat: cannot stat '/aaa': No such file or directory
I am working on some stuff where I am storing data in a file.
But each time I run the script it gets appended to the previous file.
I want help on how I can remove the file if it already exists.
Don't bother checking if the file exists, just try to remove it.
rm -f /p/a/t/h
# or
rm /p/a/t/h 2> /dev/null
Note that the second command will fail (return a non-zero exit status) if the file did not exist, but the first will succeed owing to the -f (short for --force) option. Depending on the situation, this may be an important detail.
But more likely, if you are appending to the file it is because your script is using >> to redirect something into the file. Just replace >> with >. It's hard to say since you've provided no code.
Note that you can do something like test -f /p/a/t/h && rm /p/a/t/h, but doing so is completely pointless. It is quite possible that the test will return true but the /p/a/t/h will fail to exist before you try to remove it, or worse the test will fail and the /p/a/t/h will be created before you execute the next command which expects it to not exist. Attempting this is a classic race condition. Don't do it.
Another one line command I used is:
[ -e file ] && rm file
You can use this:
#!/bin/bash
file="file_you_want_to_delete"
if [ -f "$file" ] ; then
rm "$file"
fi
If you want to ignore the step to check if file exists or not, then you can use a fairly easy command, which will delete the file if exists and does not throw an error if it is non-existing.
rm -f xyz.csv
A one liner shell script to remove a file if it already exist (based on Jindra Helcl's answer):
[ -f file ] && rm file
or with a variable:
#!/bin/bash
file="/path/to/file.ext"
[ -f $file ] && rm $file
Something like this would work
#!/bin/sh
if [ -fe FILE ]
then
rm FILE
fi
-f checks if it's a regular file
-e checks if the file exist
Introduction to if for more information
EDIT : -e used with -f is redundant, fo using -f alone should work too
if [ $( ls <file> ) ]; then rm <file>; fi
Also, if you redirect your output with > instead of >> it will overwrite the previous file
So in my case I wanted to remove a FIFO file before I create it again, so this worked for me:
#!/bin/bash
file="/tmp/test"
rm -rf $file | true
mkfifo $file
| true will continue the script even if file is not found.
I'm trying to call diff on two files in a bash script, one being given using a variable, and the other being the newest file in a directory. Here's the command I'm using, where $FILE is a variable that points to the file in the current directory, and $DIR is a variable that points to another directory where I'm searching for the newest file:
diff $FILE $DIR/$(ls -tr | tail -n 1) > diff.txt
I'm pretty sure the problem lies in trying to call the "search for newest file" command in the directory $DIR. Does this not return the newest file in the directory given by $DIR?
As you have written it the ls command is run in the current directory. You can fix this simply by explicitly telling ls which directory to list:
diff "$FILE" "$DIR/$(ls -tr $DIR | tail -n 1)" > diff.txt
The diff command has a return code of 0 if no difference is found and a return code of 1 if a difference is found. The return code of the previous command is stored in the variable $? and can be used in a following if statement:
if (( $? == 0 ))
then
echo "match..."
else
echo "no match..."
fi
Be careful when doing this as $? is overwritten by any subsequent commands. This can even include commands within the same pipeline.
how about this?
diff $FILE $DIR/$(ls -tr $DIR| tail -n 1) > diff.txt
Make sure to run ls on $DIR only:
diff "$FILE" "$DIR/$(ls -tr "$DIR"| tail -n 1)" > diff.txt
Also note extra quoting in command is highly recommended.
So I found that this was the only thing that worked right:
diff "$FILE" "$DIR/$(ls -tr "$DIR"| tail -n 1)" > diff.txt
Thanks guys, your input helped a lot.
One more quick question. Is there a way to tell if diff found that the two files were different? Could I add the entire line of code above into an "if" statement somehow?
You can try this way,
if ! DIFFSTATEMENT ; then
echo "they are different"
fi
I have a problem to manage long paths. How can I get quickly to paths like
/Users/User/.../.../.../.../.../Dev/C/card.c
I tried an alias
alias cd C='cd /Users/User/.../.../.../.../.../Dev/C'
but I am unable to do aliases for two separate words. I have long lists of Bash aliases and paths in CDPATH, so I am hesitating to make them more. How can manage long paths?
[Ideas for Replies]
The user litb's reply revealed some of my problems in the management. Things, such as "CTRL+R", "!-3:1:2:4:x" and "incremental search", are hard for me. They probably help in navigating long directories and, in the sense, management.
Using symlinks is probably the best idea; but you can do it even easier than dumping them all into your home directory.
As you mentioned, BASH has a feature called CDPATH which comes in really handy here.
Just make a hidden folder in your homedir (so it doesn't clutter your homedir too much):
$ mkdir ~/.paths
$ cd ~/.paths
$ ln -s /my/very/long/path/name/to/my/project project
$ ln -s /some/other/very/long/path/to/my/backups backups
$ echo 'CDPATH=~/.paths' >> ~/.bashrc
$ source ~/.bashrc
This creates a directory in your homedir called ".paths" which contains symlinks to all your long directory locations which you regularly use, then sets the CDPATH bash variable to that directory (in your .bashrc) and re-reads the .bashrc file.
Now, you can go to any of those paths from anywhere:
$ cd project
$ cd backups
Leaving you with a short CDPATH, no cluttering aliasses, and more importantly: A really easy way to navigate to those long paths from other applications, such as UI applications, by just going into ~/.paths or adding that directory into your UI application's sidebar or so.
Probably the easiest all-round solution you can have.
Consider using symbolic links. I have a ~/work/ directory where I place symlinks to all my current projects.
You may also use shell variables:
c='/Users/User/.../.../.../.../.../Dev/C'
Then:
cd "$c"
Create symlinks in your home directory (or somewhere else of your choosing)
ln -s longDirectoryPath ~/MySymLinkName
See man ln for more details.
Probably the easiest solution is to use:
alias cdc='cd /Users/User/.../.../.../.../.../Dev/C'
alias cdbin='cd /Users/User/.../.../.../.../.../Dev/bin'
alias cdtst='cd /Users/User/.../.../.../.../.../Dev/tst'
if you're only really working on one project at a time. If you work on multiple projects, you could have another alias which changed the directories within those aliases above.
So, you'd use something like:
proj game17
cdc
make
proj roman_numerals
cdbin
rm -f *
proj game17 ; cdc
Since this is a useful thing to have, I decided to put together a series of scripts that can be used. They're all based aroung a configuration file that you place in your home directory, along with aliases to source scripts. The file "~/.cdx_data" is of the form:
scrabble:top=~/dev/scrabble
scrabble:src=~/dev/scrabble/src
scrabble:bin=~/dev/scrabble/bin
sudoku:top=~/dev/scrabble
sudoku:src=~/dev/scrabble/src
sudoku:bin=~/dev/scrabble/bin
sudoku:data=~/dev/scrabble/data
and lists all the relevant projects (scrabble and sodoku in this case) and their directories (which may be different for each project, but have top, bin, src and data in this example).
The first action is to initialize stuff, so put:
. ~/.cdx_init
at the end of your .bash_profile and create the "~/.cdx_init" file as:
alias cdxl='. ~/.cdx_list'
alias projl='. ~/.cdx_projlist'
alias cdx='. ~/.cdx_goto'
alias proj='. ~/.cdx_proj'
This sets up the four aliases to source the files which I'll include below. Usage is:
cdxl - List all directories in current project.
projl - List all projects.
proj - Show current project.
proj <p> - Set current project to <p> (if allowed).
cdx - Show current project/directory and expected/actual real
directory, since they can get out of sync if you mix cd and cdx.
cdx . - Set actual real directory to expected directory (in other words,
get them back into sync).
cdx <d> - Set directory to <d> (if allowed).
The actual script follow. First, ".cdx_list" which just lists the allowed directories in the current project (pipelines are broken into multiple lines for readability but they should all be on one line).
echo "Possible directories are:"
cat ~/.cdx_data
| grep "^${CDX_PROJ}:"
| sed -e 's/^.*://' -e 's/=.*$//'
| sort -u
| sed 's/^/ /'
Similarly, ".cdx_projlist" shows all the possible projects:
echo "Possible projects are:"
cat ~/.cdx_data
| grep ':'
| sed 's/:.*$//'
| sort -u
| sed 's/^/ /'
In the meaty scripts, ".cdx_proj" sets and/or shows the current project:
if [[ "$1" != "" ]] ; then
grep "^$1:" ~/.cdx_data >/dev/null 2>&1
if [[ $? != 0 ]] ; then
echo "No project name '$1'."
projl
else
export CDX_PROJ="$1"
fi
fi
echo "Current project is: [${CDX_PROJ}]"
and ".cdx_goto" is the same for directories within the project:
if [[ "$1" == "." ]] ; then
CDX_TMP="${CDX_DIR}"
else
CDX_TMP="$1"
fi
if [[ "${CDX_TMP}" != "" ]] ; then
grep "^${CDX_PROJ}:${CDX_TMP}=" ~/.cdx_data >/dev/null 2>&1
if [[ $? != 0 ]] ; then
echo "No directory name '${CDX_TMP}' for project '${CDX_PROJ}'."
cdxl
else
export CDX_DIR="${CDX_TMP}"
cd $(grep "^${CDX_PROJ}:${CDX_DIR}=" ~/.cdx_data
| sed 's/^.*=//'
| head -1
| sed "s:^~:$HOME:")
fi
fi
CDX_TMP=$(grep "^${CDX_PROJ}:${CDX_DIR}=" ~/.cdx_data
| sed 's/^.*=//'
| head -1
| sed "s:^~:$HOME:")
echo "Current project is: [${CDX_PROJ}]"
echo "Current directory is: [${CDX_DIR}]"
echo " [${CDX_TMP}]"
echo "Actual directory is: [${PWD}]"
unset CDX_TMP
It uses three environment variables which are reserved for its own use: "CDX_PROJ", "CDX_DIR" and "CDX_TMP". Other than those and the afore-mentioned files and aliases, there are no other resources used. It's the simplest, yet most adaptable solution I could come up with. Best of luck.
Revisiting. Today I received this link from a social bookmarking site, then I immediately remembered this question:
Navigation with bm
We keep a simple, plain text bookmarks
file and use a tool called bm to do
the look-ups. The tool can also be
used to edit the bookmark index
dynamically as shown below where we
add the directories from the previous
example to the index.
Once i cd'ed into such a long directory, i have that in the history. Then i just type Ctrl-R for the "(reverse-i-search)" prompt and type in a few characters, like Dev/C that appear somewhere in the path, and it shows me the command what i issued back then and i can easily jump to it again.
That works pretty well in practice. Because it won't find an entry if you haven't typed that path for quite some time, which would mean doing work to make things easier probably wouldn't be worth the time. But it definitely will find it if you used it recently. Which is exactly what i need.
In some way, it's a self-organizing cache for long commands & path-names :)
You might want to consider using a script like this in your .bashrc. I've used it on a daily basis ever since I read that post. Pretty bloody useful.
The user jhs suggested Pushd and Popd-commands. I share here some of my Bash-scripts that I found in Unix Power Tools -book. They are very cool when your directories get a way too long :)
#Moving fast between directories
alias pd=pushd
alias pd2='pushd +2'
alias pd3='pushd +3'
alias pd4='pushd +4'
The command 'pushd +n' "rotates" the stack. The reverse command 'popd +n' deletes the n entry of the stack. If your stack gets too long, use 'repeat n popd'. For examle, your stack is 12 directories long:
repeat 11 popd
When you want to see your stack, write 'pushd'. For further reading, I recommend the book on pages 625-626.
In your .bashrc find PS1='${debian_chroot:+($debian_chroot)}[\033[01;32m]\u#\h[\033[00m]:[\033[01;34m]
\W[\033[00m]\$ '
and replace the \w with \W.I already have it changed here. This will only give you the main directory where you are working. You can get the full directory by typing pwd
There are fundamental well-known ideas, like creating aliases:
alias cdfoo="cd /long/path/to/foo"
and also "dropping pebbles"
export foo=/long/path/to/foo
and also making the above "project-based". I use 'ticket based' directories.
topdir=ticket_12345
alias cdfoo="cd home/me/sandbox/$topdir/long/path/to/foo"
export foo="/home/me/sandbox/$topdir/long/path/to/foo"
but beyond all this, sometimes it's just handy to jump back and forth to where you've been recently, using command-line menus. (pushd and popd are cumbersome, IMHO).
I use acd_func.sh (listed below). Once defined, you can do
cd --
to see a list of recent directories, with a numerical menu
cd -2
to go to the second-most recent directory.
Very easy to use, very handy.
Here's the code:
# Insert into .profile, .bash_profile or wherever
# acd_func 1.0.5, 10-nov-2004
# petar marinov, http:/geocities.com/h2428, this is public domain
cd_func ()
{
local x2 the_new_dir adir index
local -i cnt
if [[ $1 == "--" ]]; then
dirs -v
return 0
fi
the_new_dir=$1
[[ -z $1 ]] && the_new_dir=$HOME
if [[ ${the_new_dir:0:1} == '-' ]]; then
#
# Extract dir N from dirs
index=${the_new_dir:1}
[[ -z $index ]] && index=1
adir=$(dirs +$index)
[[ -z $adir ]] && return 1
the_new_dir=$adir
fi
#
# '~' has to be substituted by ${HOME}
[[ ${the_new_dir:0:1} == '~' ]] && the_new_dir="${HOME}${the_new_dir:1}"
#
# Now change to the new dir and add to the top of the stack
pushd "${the_new_dir}" > /dev/null
[[ $? -ne 0 ]] && return 1
the_new_dir=$(pwd)
#
# Trim down everything beyond 11th entry
popd -n +11 2>/dev/null 1>/dev/null
#
# Remove any other occurence of this dir, skipping the top of the stack
for ((cnt=1; cnt <= 10; cnt++)); do
x2=$(dirs +${cnt} 2>/dev/null)
[[ $? -ne 0 ]] && return 0
[[ ${x2:0:1} == '~' ]] && x2="${HOME}${x2:1}"
if [[ "${x2}" == "${the_new_dir}" ]]; then
popd -n +$cnt 2>/dev/null 1>/dev/null
cnt=cnt-1
fi
done
return 0
}
alias cd=cd_func
if [[ $BASH_VERSION > "2.05a" ]]; then
# ctrl+w shows the menu
bind -x "\"\C-w\":cd_func -- ;"
fi
This might also be a useful function to put in your .bashrc; it moves up either a number of directories, or to a named directory, i.e. if you're in /a/b/c/d/ you can do up 3 or up a to end up in a.
I have no idea where I found this; if you know, please comment or add the attribution.
function up()
{
dir=""
if [ -z "$1" ]; then
dir=..
elif [[ $1 =~ ^[0-9]+$ ]]; then
x=0
while [ $x -lt ${1:-1} ]; do
dir=${dir}../
x=$(($x+1))
done
else
dir=${PWD%/$1/*}/$1
fi
cd "$dir";
}
If you want to switch to zsh, this is very easy-- just use "alias -g" (global alias, i.e. an alias that works anywhere in the command, not just the first word).
# alias -g c=/my/super/long/dir/name
# cd c
# pwd
/my/super/long/dir/name
In bash, I think the closest thing you'll get to 'aliasing' style is to write a function:
function ccd {
case "$1" in
c) cd /blah/blah/blah/long/path/number/one ;;
foo) cd /blah/blah/totally/different path ;;
"multiword phrase") cd /tmp ;;
esac
}
This means using something other than "cd" as the command when you want a shortcut, but other than that, it's flexible; you can also add an "ls" to the function so that it always reminds you what's in the directory after you cd, etc.
(Note that to use a multiword argument as above, you need to quote it on the command line, like this:
ccd "multiword phrase"
so it's not really all that convenient. But it'll work if you need to.)
Based on Andrew Medico's suggestion, check out J
Look into pushd, which allows you to maintain a stack of directories which you can push onto, pop off of, or rearrange.
Check out autojmp or dirmarks
Management requires both fast creation and removal of directories. Create many directiories:
mkdir -p user/new_dir/new/_dir/.../new_dir
Remove recursively many directories (be very careful when you are in lower directories!):
rm -r dir/.../new_dir/
For further reading, the cheat sheet may help you:
http://www.scribd.com/doc/2082838/Bash-Command-Line-History-Cheat-Sheet
It contains some nuggets, but I find it rather hard to read. I cannot get commands, like Meta+>, working. They probably help you in navigating long directories.
I realize the question is pretty old, but none of the scripts out there satisfied me, so I wrote a new one.
Here's the requirements I had in mind:
1) Use only bash commands -- I intend to use this on many different unices -- Linux, cygwin, HP-UX, AIX, and a couple others, so I couldn't depend on grep being consistent. Luckily I do have bash everywhere I work.
2) Short code -- I wanted to be able to bind this to a key in GNU screen, and just hit that key to paste the script into the current bash shell I'm using, so that I don't have to setup bash profiles on every system I use. Anything super long would be annoying and take too much time to paste.
3) No file usage -- Don't want to be littering shared logons with random files.
4) Act just like "cd" in the normal case. Don't want to have to think about which command to use before I start typing.
5) Provide "up" usage like this answer: How to manage Long Paths in Bash?
6) Keep a list of recently used directories, and switch to the most recent.
Here's the script:
#Jump History - Isaiah Damron
function jfind() {
lp=${JNHIST//==${PWD}==/==}
lp=${lp%%${lp#==*$1*==}}
lp=${lp##${lp%==*$1*==*}}
lp=${lp//==/}
[[ -d "$lp" ]] && echo $lp && return 0
return 1;
}
function jadd() {
[[ -z "$JNHIST" ]] && export JNHIST='=='
[[ 3000 -lt ${#JNHIST} ]] && export JNHIST=${JNHIST:0:3000} && export JNHIST="${JNHIST%==*}=="
export JNHIST="==$PWD${JNHIST//==${PWD}==/==}"
}
function j() {
{ cd $* 2> /dev/null && jadd; } \
|| { cd ${PWD/$1*/}$1 2> /dev/null && jadd; } \
|| { jfind $1 \
&& { cd $( jfind $1 ) 2> /dev/null && jadd; } ; } \
|| cd $*
}
function jh() {
[[ -z "$1" ]] && echo -e ${JNHIST//==/\\n}
[[ -n "$1" ]] && jfind $1 && cd $(jfind $1) && jadd
}
Usage:
jh [parameters]
If called on its own, without any parameters, it outputs the current history list. If it has a parameter, then it searches through the history for the most recently used directory that contains the string $1, and cd's to it.
j {parameters}
Does cd parameters. If that fails, it checks if any of the parent directories of $PWD match $1, and cd's to it. If that fails, then it calls jh $1. If that fails, then it outputs the result of cd parameters
Note: I used '==' as an internal separator. Hopefully you don't have any directories that contain a '==', but if you do you'll have to change around the script. Just :%s/==/whatever/g