bash - loop through file contents and append to string [duplicate] - bash

This question already has answers here:
A variable modified inside a while loop is not remembered
(8 answers)
Closed 8 months ago.
In my bash script, I'd like to read the contents of a file and append each line of the file to an empty string via a loop. This seems like it would be easy to do and I thought I had implemented it correctly based on some other posts on SO (like Concatenate inputs in string while in loop), but the end result still seems to be an empty string. I'm clearly doing something wrong, but I'm not very experienced with bash scripting so I could use a quick hand.
My bash script:
#!/bin/bash
SOME_STRING=""
cat .env | while read line
do
SOME_STRING+="$line"
done
echo "$SOME_STRING"
and the .env file it references:
FOO=bar
BAZ=bim
I'd expect the output to be FOO=barBAZ=bim, but it just writes an empty string. If I toss an echo "$SOME_STRING" inside of the loop, I do see the string building up as expected, though.
I'm going to assume that this has something to do with the way that I'm reading the file contents and/or looping through it - for example, I tried a for/in loop through a space-separated string instead of a while loop through the file contents, and that worked fine.
Thanks much!

By putting the read loop in a pipe, you are building the string in a subprocess. Instead, do something like:
#!/bin/bash
some_string=""
while read line; do
some_string+="$line"
done < .env
echo "$some_string"
But, really, don't do any of that. Instead, do:
some_string=$(tr -d \\n < .env)
It's worth noting that sometimes you want to keep the subprocess, but you need to be aware that the variables will lose their values at the end of the process. But it is sometimes very convenient to do things like:
#!/bin/bash
some_string=""
cmd | {
while read line; do
some_string+="$line"
done
echo "in pipe, some_string=$some_string"
}
echo "after pipe, some_string=$some_string"

Related

Echoing an environment variable, keeping newlines intact? [duplicate]

This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 7 years ago.
I want to create some scripts for filling some templates and inserting them into my project folder. I want to use a shell script for this, and the templates are very small so I want to embed them in the shell script. The problem is that echo seems to ignore the line breaks in my string. Either that, or the string doesn't contain line breaks to begin with. Here is an example:
MY_STRING="
Hello, world! This
Is
A
Multi lined
String."
echo -e $MY_STRING
This outputs:
Hello, world! This Is A Multi lined String.
I'm assuming echo is the culprit here. How can I get it to acknowledge the line breaks?
You need double quotes around the variable interpolation.
echo -e "$MY_STRING"
This is an all-too common error. You should get into the habit of always quoting strings, unless you specifically need to split into whitespace-separated tokens or have wildcards expanded.
So to be explicit, the shell will normalize whitespace when it parses your command line. You can see this if you write a simple C program which prints out its argv array.
argv[0]='Hello,'
argv[1]='world!'
argv[2]='This'
argv[3]='Is'
argv[4]='A'
argv[5]='Multi'
argv[6]='lined'
argv[7]='String.'
By contrast, with quoting, the whole string is in argv[0], newlines and all.
For what it's worth, also consider here documents (with cat, not echo):
cat <<"HERE"
foo
Bar
HERE
You can also interpolate a variable in a here document.
cat <<HERE
$MY_STRING
HERE
... although in this particular case, it's hardly what you want.
echo is so nineties. The new (POSIX) kid on the block is printf.
printf '%s\n' "$MY_STRING"
No -e or SYSV vs BSD echo madness and full control over what gets printed where and how wide, escape sequences like in C. Everybody please start using printf now and never look back.
Try this :
echo "$MY_STRING"

Bash difference between pipeline and parameters

I need to write a script which gets a file from stdin and run over the lines of it.
My question is can I do something like that :
TheFile= /dev/stdin
while read line; do
{
....
}
done<"$(TheFile)"
or can I write --done<"$1"
or in that case the minute I send a parameter to the function which is a file it will be sent to the while function ?
Where to start... Are you sure're up for this?
What are you trying to do with the lines of the file? You might be better off not iterating like your example, just using sed, awk, or grep on it like this example:
sed -e 's/apple/banana/' $TheFile
That will output the contents of $TheFile, replacing all occurrences of "apple" with "banana". That's a trivial example, but you could do much more.
If you really want to loop, then remove the $() from your example. Also, you cannot have a space after = in your code.

Printf splits a string at spaces using Bash [duplicate]

This question already has answers here:
Why a variable assignment replaces tabs with spaces
(2 answers)
Closed 7 years ago.
I'm having some troubles with the printf function in bash.
I wrote a little script on which I pass a name and two letters (such as "sh", "py", "ht") and it creates a file in the current working directory named "name.extension".
For instance, if I execute seed test py a file named test.py is created in the current working dir with the shebang #!/usr/bin/python3.
So far, so good, nothing fancy: I'm learning shell scripting and I thought this could be a simple exercise to test the knowledge gained so far.
The problem is when I want to create an HTML file. This is the function that I use:
creaHtml(){
head='<!--DOCTYPE html-->\n<html>\n\t<head>\n\t\t<meta charset=\"UTF-8\">\n\t</head>\n\t<body>\n\t</body>\n</html>'
percorso=$CARTELLA_CORRENTE/$NOME_FILE.html
printf $head>>$percorso
chmod 755 $percorso
}
If I run, for instance, seed test ht the correct function (creaHtml) is called, test.html is created but if I try to look into it I only see:
<!--DOCTYPE
And nothing else.
This is the trace for that function:
[sviluppo:~/bin]$ seed test ht
+ creaHtml
+ head='<!--DOCTYPE html-->\n<html>\n\t<head>\n\t\t<meta charset=\"UTF-8\">\n\t</head>\n\t<body>\n\t</body>\n</html>'
+ percorso=/home/sviluppo/bin/test.html
+ printf '<!--DOCTYPE' 'html-->\n<html>\n\t<head>\n\t\t<meta' 'charset=\"UTF-8\">\n\t</head>\n\t<body>\n\t</body>\n</html>'
+ chmod 755 /home/sviluppo/bin/test.html
+ set +x
However, if I try to run printf '<!--DOCTYPE html-->\n<html>\n\t<head>\n\t\t<meta charset=\"UTF-8\">\n\t</head>\n\t<body>\n\t</body>\n</html>' from the terminal, I see the correct output: the "skeleton" of an HTML file neatly displayed with indentation and everything. What am I missing here?
Try echo -e instead of printf. printf is for printing formatted strings. Since you didn't protect $head with quotes, bash splits the string to form the command. The first word (before first white space) forms the format string. The rest are just arguments for things you didn't specify to print.
echo -e "$head" > "$percorso"
The -e evaluates your \n into newlines. I changed your >> to > since it looks like you want this to be the whole file, rather than append to any existing file you might have.
You have to be careful with quotes in bash. One thing can become many things. This actually makes it more powerful, but it can be confusing for people learning. Notice that I also put the file name "$percorso" in double quotes too. This evaluates the variable and makes sure that it ends up as one thing. If you use single quotes, it will be one word, but not evaluated. Unlike Python, there is a big difference between single and double quotes.
If you want to use printf for compatibility as #chepner pointed out, just be sure to quote it:
printf "$head" > "$percorso"
Actually that is much simpler anyway.

Simple map for pipeline in shell script

I'm dealing with a pipeline of predominantly shell and Perl files, all of which pass parameters (paths) to the next. I decided it would be better to use a single file to store all the paths and just call that for every file. The issue is I am using awk to grab the files at the beginning of each file, and it's turning out to be a lot of repetition.
My question is: I do not know if there is a way to store key-value pairs in a file so shell can natively do something with the key and return the value? It needs to access an external file, because the pipeline uses many scripts and a map in a specific file would result in parameters being passed everywhere. Is there some little quirk I do not know of that performs a map function on an external file?
You can make a file of env var assignments and source that file as need, ie.
$ cat myEnvFile
path1=/x/y/z
path2=/w/xy
path3=/r/s/t
otherOpt1="-x"
Inside your script you can source with either . myEnvFile or the more versbose version of the same feature sourc myEnvFile (assuming bash shell) , i.e.
$cat myScript
#!/bin/bash
. /path/to/myEnvFile
# main logic below
....
# references to defined var
if [[ -d $path2 ]] ; then
cd $path2
else
echo "no pa4h2=$path2 found, can't continue" 1>&1
exit 1
fi
Based on how you've described your problem this should work well, and provide a-one-stop-shop for all of your variable settings.
IHTH
In bash, there's mapfile, but that reads the lines of a file into a numerically-indexed array. To read a whitespace-separated file into an associative array, I would
declare -A map
while read key value; do
map[$key]=$value
done < filename
However this sounds like an XY problem. Can you give us an example (in code) of what you're actually doing? When I see long piplines of grep|awk|sed, there's usually a way to simplify. For example, is passing data by parameters better than passing via stdout|stdin?
In other words, I'm questioning your statement "I decided it would be better..."

Bash script execute shell command with Bash variable as argument

I have one loop that creates a group of variables like DISK1, DISK2... where the number at the end of the variable name gets created by the loop and then loaded with a path to a device name. Now I want to use those variables in another loop to execute a shell command, but the variable doesn't give its contents to the shell command.
for (( counter=1 ; counter<=devcount ; counter++))
do
TEMP="\$DISK$counter"
# $TEMP should hold the variable name of the disk, which holds the device name
# TEMP was only for testing, but still has same problem as $DISK$counter
eval echo $TEMP #This echos correctly
STATD$counter=$(eval "smartctl -H -l error \$DISK$counter" | grep -v "5.41" | grep -v "Joe")
eval echo \$STATD$counter
done
Don't use eval ever, except maybe if there is no other way AND you really know what you are doing.
The STATD$counter=$(...) should give an error. That's not a valid assignment because the string "STATD$counter" is not a valid variable name. What will happen is (using a concrete example, if counter happened to be 3 and your pipeline in the $( ) output "output", bash will only expand that line as far as "STATD3=output" so it will try to find a command named "STATD3=output" and run it. Odds are this is not what you intended.
It sounds like everything you want to do can be accomplished with arrays instead. If you are not familiar with bash arrays take a look at Greg's Wiki, in particular this page or the bash man page to find out how to use them.
For example, in the loop you didn't post in your question: make disk (not DISK: don't use all upper case variable names) an array like so
disk+=( "new value" )
or even
disk[counter]="new value"
Then in the loop in your question, you can make statd an array as well and assign it with values from disk by
statd[counter]="... ${disk[counter]} ..."
It's worth saying again: avoid using eval.

Resources