I'm trying to recapture some of the simplicity of the c-shell and tcsh. I had a simple alias that allowed me to list directories (alias lsdd 'ls | grep /'). I found a post with several solutions, none of which were particularly satisfying. For instance,
ls -d */
works well unless there are no sub-directories, in which case you get an error message--not exactly elegant.
echo */
doesn't give that error, but the list is not as easily readable as a single column.
So, I have been routing around in /etc to find where bash defines its ls command so that it uses color, and so that it strips the / following the directory name. That seems to be a great place to do some bud nipping. In what startup file does bash strip slashes from directory names in an ls command?
If you want to list only directories without the trailing slash and in a single column ls is not necessarily the best utility. And aliases are somehow obsolete; functions are recommended nowadays.
lsdd() {
local -a list=( */ )
printf '%s\n' "${list[#]%/}"
}
If there are no sub-directories by default a single * is printed. To get rid of it we can temporarily set the nullglob option. In the following we record the option's initial state and restore it afterwards:
lsdd() {
local tmp=$(shopt -p nullglob)
shopt -s nullglob
local -a list=( */ )
printf '%s\n' "${list[#]%/}"
eval "$tmp"
}
Related
I'm trying to rename multiple files that match a pattern in one directory.
Files:
stack_overflow_one.xml
stack_overflow_two.xml
stack_overflow_one.html
I would like to rename stack_overflow to heap_graph
heap_graph_one.xml
heap_graph_two.xml
heap_graph_one.html
I have tried the following:
Using rename:
rename stack_overflow heap_graph stack_overflow* # returns 'The syntax of the command is incorrect.'
Using for loop in Bash
# how can I write this in one line? I've tried wrapping in one line, but also does not work
for i in stack_overflow* do
mv "$i" "${i/stack_overflow/heap_graph}"
done
However, none of these are working.
What you have is a trivial syntax error in the for loop. The rest of your script should work fine without any problem.
for i in stack_overflow*; do
# ^^^ missing semi-colon
# The below condition to handle graceful loop termination when no files are found
[ -f "$i" ] || continue
mv "$i" "${i/stack_overflow/heap_graph}"
done
As noted by ghoti below if you are in the bourne again shell bash and not the POSIX bourne shell (sh) for which the solution above is portable, you can use special globbing options to avoid the condition of having to deal with case when no files are returned by the glob.
shopt -s nullglob
for i in stack_overflow*; do
mv "$i" "${i/stack_overflow/heap_graph}"
done
shopt -u nullglob
The -s option sets it and -u unsets it. More on shopt built-in from the GNU bash page
The semi-common utility mmv
(article)
is useful for the "multi-move" case you've described.
$ mmv -n 'stack_overflow_*' 'heap_graph__#' # remove -n after testing
mv -- stack_overflow_one.html heap_graph_one.html
mv -- stack_overflow_one.xml heap_graph_one.xml
mv -- stack_overflow_two.xml heap_graph_two.xml
As you can see, it's just calling down to mv multiple times for the
pattern(s) matched.
The * is a wildcard like Bash uses, but it's quote-escaped to be
passed to mmv. The _# is the reference to the match, also escaped
(though the docs suggest #1 would work instead).
This family of commands is also handy for copying (cp) and linking
(ln).
If you happen to have Zsh (a common Bash upgrade/replacement), you've
already got zmv, which would similarly do the job with:
% zmv -n 'stack_overflow_(*)' 'heap_graph_$1'
When I enter in terminal
files=(/var/db/*); printf '%s\n' "${files[#]}"
and run it, I get a list of files in that folder, but going into restricted TokenCache dir does not give anything:
files=(/var/db/TokenCache/*); printf '%s\n' "${files[#]}"
This command gives me back /var/db/TokenCache/* and not files/folders inside. Is there any way to make it work inside restricted folders as sudo ls and even sudo rm work inside? For example:
sudo ls -la /var/db/TokenCache
shows its content, namely two folders config and tokens.
The answer could be something like this:
files=($(sudo ls "/var/db/TokenCache"))
printf '%s\n' "${files[#]}"
But this is only safe under the assumption, that the specified folder (TokenCache in this case) only contains elements without any spaces in their name.
If you want to get the full path out of the array for each file I would suggest something like this:
directory="/var/db/TokenCache"
files=($(sudo ls "${directory}"))
printf "${directory}/%s\n" "${files[#]}"
Notice that the ' changed to " in the format specifier of the printf call. Thats necessary to have variable expansion by the shell.
The glob expansion happens in the shell, so you need a shell instance that is running as a user with the correct permissions.
sudo bash -c 'files=(/var/db/*); printf "%s\n" "${files[#]}"'
I have a question on how to approach a problem I've been trying to tackle at multiple points over the past month. The scenario is like so:
I have a a base directory with multiple sub-directories all following the same sub-directory format:
A/{B1,B2,B3} where all B* have a pipeline/results/ directory structure under them.
All of these results directories have multiple *.xyz files in them. These *.xyz files have a certain hierarchy based on their naming prefixes. The naming prefixes in turn depend on how far they've been processed. They could be, for example, select.xyz, select.copy.xyz, and select.copy.paste.xyz, where the operations are select, copy and paste. What I wish to do is write a ls | grep or a find that picks these files based on their processing levels.
EDIT:
The processing pipeline goes select -> copy -> paste. The "most processed" file would be the one with the most of those stages as prefixes in its filename. i.e. select.copy.paste.xyz is more processed than select.copy, which in turn is more processed than select.xyz
For example, let's say
B1/pipeline/results/ has select.xyz and select.copy.xyz,
B2/pipeline/results/ has select.xyz
B3/pipeline/results/ has select.xyz, select.copy.xyz, and select.copy.paste.xyz
How can I write a ls | grep/find that picks the most processed file from each subdirectory? This should give me B1/pipeline/results/select.copy.xyz, B2/pipeline/results/select.xyz and B3/pipeline/results/select.copy.paste.xyz.
Any pointer on how I can think about an approach would help. Thank you!
For this answer, we will ignore the upper part A/B{1,2,3} of the directory structure. All files in some .../pipeline/results/ directory will be considered, even if the directory is A/B1/doNotIncludeMe/forbidden/pipeline/results. We assume that the file extension xyz is constant.
A simple solution would be to loop over the directories and check whether the files exist from back to front. That is, check if select.copy.paste.xyz exists first. In case the file does not exist, check if select.copy.xyz exists and so on. A script for this could look like the following:
#! /bin/bash
# print paths of the most processed files
shopt -s globstar nullglob
for d in **/pipeline/result; do
if [ -f "$d/select.copy.paste.xyz" ]; then
echo "$d/select.copy.paste.xyz"
elif [ -f "$d/select.copy.xyz" ]; then
echo "$d/select.copy.xyz"
elif [ -f "$d/select.xyz" ]; then
echo "$d/select.xyz"
else
# there is no file at all
fi
done
It does the job, but is not very nice. We can do better!
#! /bin/bash
# print paths of the most processed files
shopt -s globstar nullglob
for dir in **/pipeline/result; do
for file in "$dir"/select{.copy{.paste,},}.xyz; do
[ -f "$file" ] && echo "$file" && break
done
done
The second script does exactly the same thing as the first one, but is easier to maintain, adapt, and so on. Both scripts work with file and directory names that contain spaces or even newlines.
In case you don't have whitespace in your paths, the following (hacky, but loop-free) script can also be used.
#! /bin/bash
# print paths of the most processed files
shopt -s globstar nullglob
files=(**/pipeline/result/select{.copy{.paste,},}.xyz)
printf '%s\n' "${files[#]}" | sed -r 's#(.*/)#\1 #' | sort -usk1,1 | tr -d ' '
I've been following this tutorial (the idea can also be found in other posts of SO)
http://www.cyberciti.biz/faq/bash-loop-over-file/
This is my test script:
function getAllTests {
allfiles=$TEST_SCRIPTS/*
# Getting all stests in the
if [[ $1 == "s" ]]; then
for f in $allfiles
do
echo $f
done
fi
}
The idea is to print all files (one per line) in the directory found in TEST_SCRIPTS.
Instead of that this is what I get as an output:
/path/to/dir/*
(The actual path obviously, but this is to convey the idea).
I have tried the followign experiment on bash. Doing this
a=(./*)
And this read me all files in the current directory into a as an array. However if anything other than ./ is used then it does not work.
How can I use this procedure with a directory other than ./?
When there are no matches, the wildcard is not expanded.
I speculate that TESTSCRIPTS contains a path which does not exist; but without access to your code, there is obviously no way to diagnose this properly.
Common solutions include shopt -s nullglob which causes the shell to replace the wildcard with nothing when there are no matches; and explicitly checking for the expanded value being equal to the wildcard (in theory, this could misfire if there is a single file named literally * so this is not completely bulletproof!)
By the by, the allfiles variable appears to be superfluous, and you should generally be much more meticulous about quoting. See When to wrap quotes around a shell variable? for details.
function getAllTests {
local nullglob
shopt -q nullglob || nullglob=reset
shopt -s nullglob
# Getting all stests in the # fix sentence fragment?
if [[ $1 == "s" ]]; then
for f in "$TEST_SCRIPTS"/*; do # notice quotes
echo "$f" # ditto
done
fi
# Unset if it wasn't set originally
case $nullglob in 'reset') shopt -u nullglob;; esac
}
Setting and unsetting nullglob inside a single function is probably excessive; most commonly, you would set it once at the beginning of your script, and then write the script accordingly.
I am a total newbie to Linux and bash scripting and am currently stumped with this problem!
I have a directory containing many images from which I need to copy the unique images to a new location. I know there are numerous options for how to go about doing this but have very limited knowledge at the moment so appreciate I may be going about this the wrong way.
I used find and cat to create this list and have attempted to copy the files across with the intention of comparing them (using md5 and checking file names) when they are there.
However, the text file has 30 files on it but only 18 have been copied over. Can anyone advise?
My code to find files is -
find $1 -name "IMG_****.JPG" | cat > list.txt
and my code to copy from the list is
for image in $(cat list.txt);
do
cp $image $2
done
You're doing this much too complicated. Do not pipe find output to cat to pipe it into a list. This is an unnecessary use of cat. If you must, you can redirect the output of every program directly:
find "$1" -name "IMG_*.JPG" > list.txt
Also, do not use for to read lines from a file. Better use while with read:
while read -r filename; do
cp "$filename" "$2"
done < list.txt
But it's even easier. You can just work with the files directly from find:
find "$1" -name "IMG_*.JPG" -exec cp {} "$2" \;
Here, {} will be replaced by each filename that find finds. Don't forget to quote your variables, so that spaces in file paths are no problem.
Another much simpler method with Bash options:
shopt -s nullglob globstar
cp -t "$2" -- "$1"/**/IMG_*.JPG
Here, globstar enables recursive matching of directories through **. The -t option to cp specifies the target of the copy operation.* The command will be expanded to cp -t target -- source1/IMG_foo.JPG source2/IMG_bar.JPG et cetera.
Now, as to your original issue, it could have been that some images have a space in their name. This would have broken your original script. If your image files contained a newline in their name, it also wouldn't have worked with while read … – but you would have gotten an error in that case of a file not being found.
Also note that cp overwrites files with the same name. Without asking for confirmation. So if in your subdirectories there are images with the same filename, you'd only get one result, with the latest overwriting the existing one.
* The -- isn't strictly necessary, but it's a good habit to include it to tell the command when the options arguments are over.