This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 2 years ago.
I'm trying to compare sizes of files inside two directories.
My problem is that when I store the sizes of the files inside of a "for" loop, my variable takes all the sizes at once instead of taking them one by one.
Here is the part of my code that is problematic :
for dir1Files in dir1/*
do
sizeFile1=`stat -c%s $dir1Files`
for dir2Files in dir2/*
do
sizeFile2=`stat -c%s $dir2Files`
diffSize=$((sizeFile1-sizeFile2))
echo "$diffSize"
done
done
I realised, thanks to set -x, that my variables sizeFile1 and sizeFile2 are not integers. Instead, they are a few lines long and contain the sizes of my files in directories, with "one line = one integer", if that makes sense.
For example, with three files in dir1, my variable sizeFile1 is :
12500
14534
23000
What I would like is for my variable to vary from 12500 to 14534 to 23000. How should I do that ? I'm guessing I need to change my "for" into something else ?
Thanks in advance.
Nothing in this is broken 100% of the time, but it certainly can be broken if run with unusual filenames present. To make this code more robust:
Use quotes whenever you expand a variable. This prevents a file named dir1/ * (with the space in its name) from being replaced with a list of all files in the current directory when generating a stat command line.
Use shopt -s nullglob to make the loops not run at all when no glob matches exist, instead of running with the glob expression as a filename itself.
shopt -s nullglob # prevent dir1/* from ever evaluating to itself
for dir1File in dir1/*; do
sizeFile1=$(stat -c%s "$dir1File")
for dir2File in dir2/*; do
sizeFile2=$(stat -c%s "$dir2File")
diffSize=$((sizeFile1-sizeFile2))
echo "$diffSize"
done
done
Related
I have 40 csv files that I need to edit. 20 have matching format and the names only differ by one character, e.g., docA.csv, docB.csv, etc. The other 20 also match and are named pair_docA.csv, pair_docB.csv, etc.
I have the code written to edit and combine docA.csv and pair_docA.csv, but I'm struggling writing a loop that calls both the above files, edits them, and combines them under the name combinedA.csv, then goes on the the next pair.
Can anyone help my rudimentary bash scripting? Here's what I have thus far. I've tried in a single for loop, and now I'm trying in 2 (probably 3) for loops. I'd prefer to keep it in a single loop.
set -x
DIR=/path/to/file/location
for file in `ls $DIR/doc?.csv`
do
#code to edit the doc*.csv files ie $file
done
for pairdoc in `ls $DIR/pair_doc?.csv`
do
#code to edit the piar_doc*.csv files ie $pairdoc
done
#still need to combine the files. I have the join written for a single iteration,
#but how do I loop the code to save each join as a different file corresponding
#to combined*.csv
Something along these lines:
#!/bin/bash
dir=/path/to/file/location
cd "$dir" || exit
for file in doc?.csv; do
pair=pair_$file
# "${file#doc}" deletes the prefix "doc"
combined=combined_${file#doc}
cat "$file" "$pair" >> "$combined"
done
ls, on principle, shouldn't be used in a shell script in order to iterate over the files. It is intended to be used interactively and nearly never needed within a script. Also, all-capitalized variable names shouldn't be used as ordinary variables, since they may collide with internal shell variables or environment variables.
Below is a version without changing the directory.
#!/bin/bash
dir=/path/to/file/location
for file in "$dir/"doc?.csv; do
basename=${file#"$dir/"}
pair=$dir/pair_$basename
combined=$dir/combined_${basename#doc}
cat "$file" "$pair" >> "$combined"
done
This might work for you (GNU parallel):
parallel cat {1} {2} \> join_{1}_{2} ::: doc{A..T}.csv :::+ pair_doc{A..T}.csv
Change the cat commands to your chosen commands where {1} represents the docX.csv files and {2} represents the pair_docX.csv file.
N.B. X represents the letters A thru T
I have a question concerning why this doesn't work. Probably, it's a simple answer, but I just can't seem to figure it out.
I want to move a couple of files I have. They all have the same filename (let's say file1) but they are all in different directories (lets say /tmp/dir1,dir2 and dir3). If I were to move these individually I could do something along the lines of:
mv /tmp/dir1/file1 /tmp
That works. However, I have multiple directories and they're all going to end up in the same spot....AND I don't want to overwrite. So, I tried something like this:
mv /tmp/{dir1,dir2,dir3}/file1 /tmp/file1.{a,b,c}
When I try this I get:
/tmp/file1.c is not a directory
Just to clarify...this also works:
mv /tmp/dir1/file1 /tmp/file1.c
Pretty sure this has to do with brace expansion but not certain why.
Thanks
Just do echo to understand how the shell expands:
$ echo mv /tmp/{dir1,dir2,dir3}/file1 /tmp/file1.{a,b,c}
mv /tmp/dir1/file1 /tmp/dir2/file1 /tmp/dir3/file1 /tmp/file1.a /tmp/file1.b /tmp/file1.c
Now you can see that your command is not what you want, because in a mv command, the destination (directory or file) is the last argument.
That's unfortunately now how the shell expansion works.
You'll have to probably use an associative array.
!/bin/bash
declare -A MAP=( [dir1]=a [dir2]=b [dir3]=c )
for ext in "${!MAP[#]}"; do
echo mv "/tmp/$ext/file1" "/tmp/file1.${MAP[$ext]}"
done
You get the following output when you run it:
mv /tmp/dir2/file1 /tmp/file1.b
mv /tmp/dir3/file1 /tmp/file1.c
mv /tmp/dir1/file1 /tmp/file1.a
Like with many other languages key ordering is not guaranteed.
${!MAP[#]} returns an array of all the keys, while ${MAP[#]} returns the an array of all the values.
Your syntax of /tmp/{dir1,dir2,dir3}/file1 expands to /tmp/dir1/file /tmp/dir2/file /tmp/dir3/file. This is similar to the way the * expansion works. The shell does not execute your command with each possible combination, it simply executes the command but expands your one value to as many as are required.
Perhaps instead of a/b/c you could differentiate them with the actual number of the dir they came from?
$: for d in 1 2 3
do echo mv /tmp/dir$d/file1 /tmp/file1.$d
done
mv /tmp/dir1/file1 /tmp/file1.1
mv /tmp/dir2/file1 /tmp/file1.2
mv /tmp/dir3/file1 /tmp/file1.3
When happy with it, take out the echo.
A relevant point - brace expansion is not a wildcard. It has nothing to do with what's on disk. It just creates strings.
So, if you create a bunch of files named with single letters or digits, echo ? will wildcard and list them all, but only the ones actually present. If there are files for vowels but not consonants, only the vowels will show. But -
if you say echo {foo,bar,nope} it will output foo bar nope regardless of whether or not any or all of those exist as files or directories, etc.
This question already has answers here:
Forcing bash to expand variables in a string loaded from a file
(13 answers)
Closed 7 years ago.
Let's say I have a file called path.txt containing the text $HOME/filedump/ on a single line. How can I then read the contents of path.txt into a variable, while having Bash parse said content?
Here's an example of what I'm trying to do:
#!/bin/bash
targetfile="path.txt"
target=$( [[ -f $targetfile ]] && echo $( < $targetfile ) || echo "Not set" )
echo $target
Desired output: /home/joe/filedump/
Actual output: $HOME/filedump/
I've tried using cat in place of <, wrapping it in quotes, and more. Nothing seems to get me anywhere.
I'm sure I'm missing something obvious, and there's probably a simple builtin command. All I can find on Google is pages about reading variables from ini/config files or splitting one string into multiple variables.
If you want to evaluate the contents of path.txt and assign that to target, then use:
target=$(eval echo $(<path.txt))
for example:
$ target=$(eval echo $(<path.txt)); echo "$target"
/home/david/filedump/
This might not necessarily suit your needs (depending on the context of the code you provided), but the following worked for me:
targetfile="path.txt"
target=$(cat $targetfile)
echo $target
Here's a safer alternative than eval. In general, you should not be using configuration files that require bash to evaluate their contents; that just opens a security risk in your script. Instead, detect if there is something that requires evaluation, and handle it explicitly. For example,
IFS= read -r path < path.txt
if [[ $path =~ '$HOME' ]]; then
target=$HOME/${path#\$HOME}
# more generally, target=${path/\$HOME/$HOME}, but
# when does $HOME ever appear in the *middle* of a path?
else
target=$path
fi
This requires you to know ahead of time what variables might appear in path.txt, but that's a good thing. You should not be evaluating unknown code.
Note that you can use any placeholder instead of a variable in this case; %h/filedump can be detected and processed just as easily as $HOME/filedump, without the presumption that the contents can or should be evaluated as shell code.
This question already has answers here:
Looping on empty directory content in Bash [duplicate]
(2 answers)
Closed 7 years ago.
Consider the following bash code:
for f in /tmp/*.dat; do echo ${f}; done
when I run this and there is no *.dat file in /tmp the output is:
/tmp/*.dat
which is clearly not what I want. However, when there is such a file, it will print out the correct one
/tmp/foo.dat
How can I force the for loop to return 'nothing' when there is no such file in the directory. The find-command is not an option, sorry for that :/ I would like to have also a solution without testing, if *.dat is a file or not. Any solutions so far?
This should work:
shopt -s nullglob
...
From Bash Manual
nullglob
If set, Bash allows filename patterns which match no files to expand
to a null string, rather than themselves.
This question already has answers here:
How to skip the for loop when there are no matching files?
(2 answers)
Closed 3 years ago.
I want to process a set of files (*.ui) in the current directory. The following script works as expected if some *.ui files are found. But if no .ui file exist the current directory, the for loop is entered all the same. Why is that ?
for f in *.ui
do
echo "Processing $f..."
done
It prints :
Processing *.ui...
Use:
shopt -s nullglob
From man bash:
nullglob
If set, bash allows patterns which match no files (see Pathname Expansion
above) to expand to a null string, rather than themselves.
You already have the how, the 'why' is that bash will first try to match *.ui to files, but if that doesn't work (it gets no results) it will assume you meant the string "*.ui".
for f in "*.ui"
do
echo "Processing $f..."
done
wil indeed print "Processing *.ui".