Bash: regarding loops, arrays and variables [duplicate] - bash

This question already has an answer here:
Variables set in a bash 'while read' loop are unset after it [duplicate]
(1 answer)
Closed 8 years ago.
I hope I am asking this in the right place.
I have a fairly simple bash script that I am working on, however I just started scripting bash a few days ago, and I have hit a bit of a stump.
I have seen familiar questions, but before saying read this, or duplicate, or google, understand I have tried many approaches and I am just stuck with this one roadblock.
I have the following code:
find ~/bin/ -type f ! \( -iname '*~' -o -iname '.*' \) -exec basename {} \; | sort | while read line; do
let i=i+1
command_array[$i]="$line"
echo "$i : ${command_array[$i]}"
done
echo $i
echo "array check : ${command_array[*]}"
I would like to get those variables from the loop. As I've read, it seems that somehow I need to make those variables before the pipes ? I am not sure how to accomplish this with my current code. It's a combination of: I am not sure how to restructure the code to find files to accommodate this. It looks like the foolowing is what I should be trying to acheive:
n=0
printf %s "$foo" |
while IFS= read -r line; do
n=$(($n + 1))
done
echo $n
And while I partially understand what's going on here, I am still lost as to how to start to switch what I have to something like this to get variables.
I'm not looking for someone to re-code it for me, but some tips, or hints in getting this going in the right direction would be great.

The typical way to avoid the subshell when using a while read loop to consume command output is to rewrite it to redirect from process substition.
Instead of:
foo | bar | while read baz; do (( i++ )); done
echo "$i" # Unset
use
while read baz; do (( i++ )); done < <(foo | bar)
echo "$i" # Works
PS: ShellCheck automatically warns about this, and links to a page with documentation on how to restructure a while loop like this.

Related

Terminate a while loop with multiple conditions in a bash script file [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 12 months ago.
Improve this question
I tried everything, but I still can't get the while loop to stop based on the following condition:
#!/bin/sh
clear
counter=1
breakCond=$(tail -n 1 /home/a/Desktop/Triple_Graphs/11_nodes/Res/Com_Output.txt|sed -r 's/^([^.]+).*$/\1/; s/^[^0-9]*([0-9]+).*$/\1/')
cd /home/a/Desktop/cliquer-1.21
rm /home/a/Desktop/Triple_Graphs/11_nodes/Res/Com_Output.txt
clear
Mu=$(head -1 /home/a/Desktop/Triple_Graphs/11_nodes/Mu.txt)
while [[ $counter -le 97 && $breakCond -ne $Mu ]]
do
#echo $counter| tee -a outErr.txt
./cl -u /home/a/Desktop/Triple_Graphs/11_nodes/G_$counter.txt &> /home/a/Desktop/Triple_Graphs/11_nodes/Res/calculated_$counter.txt
#echo -e "calculated_$counter.txt \n"
output=$(tail -1 /home/a/Desktop/Triple_Graphs/11_nodes/Res/calculated_$counter.txt)
echo $output>>/home/a/Desktop/Triple_Graphs/11_nodes/Res/Com_Output.txt
((counter++))
breakCond=$(tail -n 1 /home/a/Desktop/Triple_Graphs/11_nodes/Res/Com_Output.txt|sed -r 's/^([^.]+).*$/\1/; s/^[^0-9]*([0-9]+).*$/\1/')
done
The second condition in my while loop does not work. I'm reading two values from txt files ("breakCond" and "Mu") and attempting to compare them.
If you want to use Bash features, you need to make sure the script is executed by Bash. The precise location of Bash can vary, so I'll use env for portability here; but probably, you might want to hardcode e.g. #!/bin/bash to avoid this indirection.
Bash has built-in facilities for looping a specified number of times.
As a general design, I would remove the hard-coded absolute paths, and perhaps allow the user to override a built-in default. I also took out the apparently completely superfluous cd and the user-hostile clear. The current redesign simply assumes that you are running in /home/a/Desktop/Triple_Graphs/11_nodes so that you can run it in a different directory with sample data to test it.
#!/usr/bin/env bash
break_maybe () {
# sed $!d replaces tail -n 1
sed -r '$!d;s/^([^.]+).*$/\1/; s/^[^0-9]*([0-9]+).*$/\1/' "$1"
}
rm -f Res/Com_Output.txt
Mu=$(head -1 Mu.txt)
# {1..97} is Bash-specific
for counter in {1..97}; do
if [[ $(break_maybe Res/Com_Output.txt) == "$Mu" ]]; then
break
fi
/home/a/Desktop/cliquer-1.21/cl -u "G_$counter.txt" > "Res/calculated_$counter.txt" 2>&1
tail -n 1 "Res/calculated_$counter.txt"
done >>Res/Com_Output.txt
I obviously have no way to test this, but it should at least suggest some ways to refactor this.

bash loop same variable from multiple configs

I'm trying to loop same variables from multiple files in bash. Here's my file structure and their contents;
script.sh
first.conf
second.conf
Inside of first.conf;
var=Hello1
Inside of second.conf;
var=Hello2
Inside of script.sh;
#!/bin/bash
_a=`find ~/ -name "*.conf"`
source ${_a}
for x in ${_a}
do
echo "$var"
done
This might look really dumb tho, I'm really new to programming.
What I'm trying to do is to loop and echo these $vars from 2 different configs.
How can I do that?
Consider:
while IFS= read -r -d '' conf; do
(source "$conf" && echo "$var")
done < <(find ~ -name '*.conf' -print0)
Breaking down how this works:
The while read syntax is discussed in BashFAQ #1. The variant with -d '' expects input separated by NULs rather than newlines -- more on that later.
Putting (source "$conf" && echo "$var") in parens prevents side effects on the rest of your script -- while this has a performance cost, it ensures that variables added by source are only present for the echo. Using the && prevents the echo from running if the source command fails.
<(...) is process substitution syntax; it's replaced with a filename that can be read to retrieve the output of the command therein (in this case find). Using this syntax rather than piping into the loop avoids the bugs discussed in BashFAQ #24.
The -print0 action in find prints the name of the file found, followed by a NUL character -- a zero byte. What's useful about NUL bytes is that, unlike any other ASCII character, they can't exist in UNIX paths; using them thus prevents your code from being subject to trickery (think about someone running d=$'./tmp/\n/etc/passwd\n' && mkdir -p -- "$d" && touch "$d/hi.conf" -- traditional find output would have /etc/passwd showing up on its own line, but with find -print0, the newlines in the name aren't mistaken for a separator between files.
This is a shorter and simpler way.
#!/bin/bash
for f in *.conf
do
source "$f"; echo "$f : $var"
done

Extracting end of filename in bash script

Within my backup script, I'd like to only keep 7 days worth of backups (tried using logrotate for this and it worked perfectly, but I ran into issues with the timing of cron.daily and how it affected the "dateext"). I'm running into problems using parameter expansion to extract the date from the filenames.
Here are some examples of some of the files
foo.bar.tar.gz-20120904
bar.baz.tar.gz-20120904
...
Here is my bash script:
#!/bin/bash
path="/foo/"
today=$(date +%Y%m%d)
keepDays=7
keepSeconds=$(date -d "-$keepDays day" +%s)
for f in $path"*"; do
fileSeconds=$(date -d ${f##*-} +%s)
if [ $fileSeconds -lt $keepSeconds ]
then
rm $f
fi
done
Here is the error I'm getting:
date: extra operand `/foo/foo.bar.tar.gz-20120904'
Remove the quotes around the *, that prevents globbing:
for f in ${path}*; do
(the { } are not strictly required here, but make it easier to read)
Not part of the question, but the Bourne shell syntax [ $fileSeconds -lt $keepSeconds ] could be written as (( $fileSeconds < $keepSeconds )) which is possibly safer.
As cdarke says, remove the quotes around the * in the for loop:
for f in ${path}/*; do
What happens is that the shell executing date gets '/foo/*' and expands that into a list of file names (more than one) and then uses ${f##*-} on part of the list, and date is called with multiple names, and objects.
You'd see this if you ran with bash -x your-script.sh, for instance. When something mysterious goes on, the first step is to make sure you know what the shell is doing. Adding echo "$f" or echo $f in the loop would help you understand — though you'd get two different answers.

Handle special characters in bash for...in loop

Suppose I've got a list of files
file1
"file 1"
file2
a for...in loop breaks it up between whitespace, not newlines:
for x in $( ls ); do
echo $x
done
results:
file
1
file1
file2
I want to execute a command on each file. "file" and "1" above are not actual files. How can I do that if the filenames contains things like spaces or commas?
It's a little trickier than I think find -print0 | xargs -0 could handle, because I actually want the command to be something like "convert input/file1.jpg .... output/file1.jpg" so I need to permutate the filename in the process.
Actually, Mark's suggestion works fine without even doing anything to the internal field separator. The problem is running ls in a subshell, whether by backticks or $( ) causes the for loop to be unable to distinguish between spaces in names. Simply using
for f in *
instead of the ls solves the problem.
#!/bin/bash
for f in *
do
echo "$f"
done
UPDATE BY OP: this answer sucks and shouldn't be on top ... #Jordan's post below should be the accepted answer.
one possible way:
ls -1 | while read x; do
echo $x
done
I know this one is LONG past "answered", and with all due respect to eduffy, I came up with a better way and I thought I'd share it.
What's "wrong" with eduffy's answer isn't that it's wrong, but that it imposes what for me is a painful limitation: there's an implied creation of a subshell when the output of the ls is piped and this means that variables set inside the loop are lost after the loop exits. Thus, if you want to write some more sophisticated code, you have a pain in the buttocks to deal with.
My solution was to take the "readline" function and write a program out of it in which you can specify any specific line number that you may want that results from any given function call. ... As a simple example, starting with eduffy's:
ls_output=$(ls -1)
# The cut at the end of the following line removes any trailing new line character
declare -i line_count=$(echo "$ls_output" | wc -l | cut -d ' ' -f 1)
declare -i cur_line=1
while [ $cur_line -le $line_count ] ;
do
# NONE of the values in the variables inside this do loop are trapped here.
filename=$(echo "$ls_output" | readline -n $cur_line)
# Now line contains a filename from the preceeding ls command
cur_line=cur_line+1
done
Now you have wrapped up all the subshell activity into neat little contained packages and can go about your shell coding without having to worry about the scope of your variable values getting trapped in subshells.
I wrote my version of readline in gnuc if anyone wants a copy, it's a little big to post here, but maybe we can find a way...
Hope this helps,
RT

Renaming part of a filename [duplicate]

This question already has answers here:
Rename multiple files based on pattern in Unix
(24 answers)
Closed 5 years ago.
I have loads of files which look like this:
DET01-ABC-5_50-001.dat
...
DET01-ABC-5_50-0025.dat
and I want them to look like this:
DET01-XYZ-5_50-001.dat
...
DET01-XYZ-5_50-0025.dat
How can I do this?
There are a couple of variants of a rename command, in your case, it may be as simple as
rename ABC XYZ *.dat
You may have a version which takes a Perl regex;
rename 's/ABC/XYZ/' *.dat
for file in *.dat ; do mv $file ${file//ABC/XYZ} ; done
No rename or sed needed. Just bash parameter expansion.
Something like this will do it. The for loop may need to be modified depending on which filenames you wish to capture.
for fspec1 in DET01-ABC-5_50-*.dat ; do
fspec2=$(echo ${fspec1} | sed 's/-ABC-/-XYZ-/')
mv ${fspec1} ${fspec2}
done
You should always test these scripts on copies of your data, by the way, and in totally different directories.
You'll need to learn how to use sed http://unixhelp.ed.ac.uk/CGI/man-cgi?sed
And also to use for so you can loop through your file entries http://www.cyberciti.biz/faq/bash-for-loop/
Your command will look something like this, I don't have a term beside me so I can't check
for i in `dir` do mv $i `echo $i | sed '/orig/new/g'`
I like to do this with sed. In you case:
for x in DET01-*.dat; do
echo $x | sed -r 's/DET01-ABC-(.+)\.dat/mv -v "\0" "DET01-XYZ-\1.dat"/'
done | sh -e
It is best to omit the "sh -e" part first to see what will be executed.
All of these answers are simple and good. However, I always like to add an interactive mode to these scripts so that I can find false positives.
if [[ -n $inInteractiveMode ]]
then
echo -e -n "$oldFileName => $newFileName\nDo you want to do this change? [Y/n]: "
read run
[[ -z $run || "$run" == "y" || "$run" == "Y" ]] && mv "$oldFileName" "$newFileName"
fi
Or make interactive mode the default and add a force flag (-f | --force) for automated scripts or if you're feeling daring. And this doesn't slow you down too much: the default response is "yes, I do want to rename" so you can just hit the enter key at each prompt (because of the -z $run test.

Resources