How to read environmental variable in bash that contains spaces and possibly other characters that need to be escaped?
I have a file server.env
PUBLIC_KEY=ssh-rsa whatever+whatever+8whatever/whatever+p+whatever user#alans-MacBook-Pro.local
I am trying to read this file into bash script as an environmental variable:
export $(cat server.env | xargs)
I am getting an error:
-bash: export: `user#alans-MacBook-Pro.local': not a valid identifier
OK, trying to quote the value in server.env:
PUBLIC_KEY='ssh-rsa whatever+whatever+8whatever/whatever+p+whatever user#alans-MacBook-Pro.local'
the same error
Double quote:
PUBLIC_KEY="ssh-rsa whatever+whatever+8whatever/whatever+p+whatever user#alans-MacBook-Pro.local"
also errors
What am I missing here?
Bash provides parameter expansion with substring removal that will allow you to separate the NAME=value pairs read from the server.env file and then export the NAME=value pairs.
A simple read loop is all you need:
#!/bin/bash
while read -r line; do ## read each line of server.env
val="${line#*=}" ## trim to 1st =, save in val
export ${line%=$val}="$val" ## remove =$val leaving name, export val with name
done < server.env
printf "%s\n" "$PUBLIC_KEY" ## confirm
Example Use/Output
$ bash test.sh
ssh-rsa whatever+whatever+8whatever/whatever+p+whatever user#alans-MacBook-Pro.local
Where the basic Parameter Expansions are:
${var#pattern} Strip shortest match of pattern from front of $var
${var##pattern} Strip longest match of pattern from front of $var
${var%pattern} Strip shortest match of pattern from back of $var
${var%%pattern} Strip longest match of pattern from back of $var
(note: pattern can contain normal shell globs like '*' and '?')
There are literally dozens of useful parameter expansions you can use for string manipulations. Just check man bash under the "Parameter Expansion" heading.
PARTIAL ANSWER (does not export, so the variable is not available in subsequent runs of bash, only in the current session):
source server.env
The problem here is that field splitting still occurs on the whitespace.
You should do this:
export "$(cat server.env)"
(i.e. add double quotes).
For multiple variables, loop over the lines:
while read -r line
do
export "$line"
done < server.env
If this is in a script called export.sh, then you will also want to source it for the variables to be set in the current process:
source export.sh
Related
I have the following array:
SPECIFIC_FILES=('resources/logo.png' 'resources/splash.png' 'www/img/logo.png' 'www/manifest.json')
And the following variable:
CUSTOMER=default
How can I loop through my array and generate strings that would look like
resources/logo_default.png
depending on the variable.
The below uses parameter expansion to extract the relevant substrings, as also described in BashFAQ #100:
specific_files=('resources/logo.png' 'resources/splash.png' 'www/img/logo.png' 'www/manifest.json')
customer=default
for file in "${specific_files[#]}"; do
[[ $file = *.* ]] || continue # skip files without extensions
prefix=${file%.*} # trim everything including and after last "."
suffix=${file##*.} # trim everything up to and including last "."
printf '%s\n' "${prefix}_$customer.$suffix" # concatenate results of those operations
done
Lower-case variable names are used here in keeping with POSIX-specified conventions (all-caps names are used for variables meaningful to the operating system or shell, whereas variables with at least one lower-case character are reserved for application use; setting a regular shell variable overwrites any like-named environment variable, so the conventions apply to both classes).
Here's a solution with sed:
for f in "${SPECIFIC_FILES[#]}"; do
echo "$f" | sed "s/\(.*\)\.\([^.]*\)/\1_${CUSTOMER}.\2/p"
done
If you know that there is only one period per filename, you can use expansion on each element directly:
$ printf '%s\n' "${SPECIFIC_FILES[#]/./_"$CUSTOMER".}"
resources/logo_default.png
resources/splash_default.png
www/img/logo_default.png
www/manifest_default.json
If you don't, Charles' answer is the robust one covering all cases.
I have been following the answers given in these questions
Shellscript Looping Through All Files in a Folder
How to iterate over files in a directory with Bash?
to write a bash script which goes over files inside a folder and processes them. So, here is the code I have:
#!/bin/bash
YEAR="2002/"
INFOLDER="/local/data/datasets/Convergence/"
for f in "$INFOLDER$YEAR*.mdb";
do
echo $f
absname=$INFOLDER$YEAR$(basename $f)
# ... the rest of the script ...
done
I am receiving this error: basename: extra operand.
I added echo $f and I realized that f contains all the filenames separated by space. But I expected to get one at a time. What could be the problem here?
You're running into problems with quoting. In the shell, double-quotes prevent word splitting and wildcard expansion; generally, you don't want these things to happen to variable's values, so you should double-quote variable references. But when you have something that should be word-split or wildcard-expanded, it cannot be double-quoted. In your for statement, you have the entire file pattern in double-quotes:
for f in "$INFOLDER$YEAR*.mdb";
...which prevents word-splitting and wildcard expansion on the variables' values (good) but also prevents it on the * which you need expanded (that's the point of the loop). So you need to quote selectively, with the variables inside quotes and the wildcard outside them:
for f in "$INFOLDER$YEAR"*.mdb;
And then inside the loop, you should double-quote the references to $f in case any filenames contain whitespace or wildcards (which are completely legal in filenames):
echo "$f"
absname="$INFOLDER$YEAR$(basename "$f")"
(Note: the double-quotes around the assignment to absname aren't actually needed -- the right side of an assignment is one of the few places in the shell where it's safe to skip them -- but IMO it's easier and safer to just double-quote all variable references and $( ) expressions than to try to keep track of where it's safe and where it's not.)
Just quote your shell variables if they are supposed to contain strings with spaces in between.
basename "$f"
Not doing so will lead to splitting of the string into separate characters (see WordSplitting in bash), thereby messing up the basename command which expects one string argument rather than multiple.
Also it would be a wise to include the * outside the double-quotes as shell globbing wouldn't work inside them (single or double-quote).
#!/bin/bash
# good practice to lower-case variable names to distinguish them from
# shell environment variables
year="2002/"
in_folder="/local/data/datasets/Convergence/"
for file in "${in_folder}${year}"*.mdb; do
# break the loop gracefully if no files are found
[ -e "$file" ] || continue
echo "$file"
# Worth noting here, the $file returns the name of the file
# with absolute path just as below. You don't need to
# construct in manually
absname=${in_folder}${year}$(basename "$file")
done
just remove "" from this line
for f in "$INFOLDER$YEAR*.mdb";
so it looks like this
#!/bin/bash
YEAR="2002/"
INFOLDER="/local/data/datasets/Convergence/"
for f in $INFOLDER$YEAR*.mdb;
do
echo $f
absname=$INFOLDER$YEAR$(basename $f)
# ... the rest of the script ...
done
This is a simple question but i am unable to find it in tutorials. Could anybody please explain what this statement does when executed in a bash shell within a folder containing .sh scripts. I know -i does in place editing, i understand that it will run sed on all scripts within the current directory. And i know that it does some sort of substitution. But what does this \(.*\) mean?
sed -i 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
Thanks in advance.
You have an expression like:
sed -i 's/XXX=\(YYY\)/XXX=ZZZ/' file
This looks for a string XXX= in a file and captures what goes after. Then, it replaces this captured content with ZZZ. Since there is a captured group, it is accessed with \1. Finally, using the -i flag in sed makes the edition to be in-place.
For the replacement, it uses the following syntax described in Shell parameter expansion:
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.
Example:
$ d=5
$ echo ${d-3}
5
$ echo ${a-3}
3
So with ${MY_BASE_DIR-SOMETHING-\1} you are saying: print $MY_BAS_DIR. And if this variable is unset or null, print what is stored in \1.
All together, this is resetting MY_BASE_DIR to the value in the variable $MY_BASE_DIR unless this is not set; in such case, the value remains the same.
Note though that the variable won't be expanded unless you use double quotes.
Test:
$ d=5
$ cat a
d=23
blabla
$ sed "s/d=\(.*\)/d=${d-\1}/" a # double quotes -> value is replaced
d=5
blabla
$ sed 's/d=\(.*\)/d=${d-\1}/' a # single quotes -> variable is not expanded
d=${d-23}
blabla
Andd see how the value remains the same if $d is not set:
$ unset d
$ sed "s/d=\(.*\)/d=${d-\1}/" a
d=23
The scripts contain lines like this:
MY_BASE_DIR=/usr/local
The sed expression changes them to:
MY_BASE_DIR=${MY_BASE_DIR-/usr/local}
The effect is that /usr/local is not used as a fixed value, but only as the default value. You can override it by setting the environment variable MY_BASE_DIR.
For future reference, I would take a look at the ExplainShell website:
http://explainshell
that will give you a breakdown of the command structure etc. In this instance, let step through the details...Let's start with a simple example, let's assume that we were going to make the simple change - commenting out all lines by adding a "#" before each line. We can do this for all *.sh files in a directory with the ".sh" extension in the current directory:
sed 's/^/\#/' *.sh
i.e. Substitute beginning of line ^, with a # ...
Caveat: You did not specify the OS you are using. You may get different results with different versions of sed and OS...
ok, now we can drill into the substitution in the script. An example is probably easier to explain:
File: t.sh
MY_BASE_DIR="/important data/data/bin"
the command 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
will search for "MY_BASE_DIR" in each .sh file in the directory.
When it encounters the string "MY_BASE_DIR=.*", in the file, it expands it to be MY_BASE_DIR="/important data/data/bin", this is now replaced on the right side of the expression /MY_BASE_DIR=${MY_BASE_DIR-\1}/ which becomes
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
essentially what happens is that the substitute operation takes
MY_BASE_DIR="/important data/data/bin"
and inserts
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
now if we run the script with the variable MY_BASE_DIR set
export MY_BASE_DIR="/new/import/dir"
the scripts modified by the sed script referenced will now substitute /important data/data/bin with /new/import/dir...
I have a bash script and I want to use that for replacing some lines with a string and add a date to the end of the line:
#! /bin/bash
today=`date '+%Y_%m_%d__%H_%M_%S'`;
sed -i '3s/.*/CONFIG_LOCALVERSION=/' ~/Desktop/file1 file2 ...
Also, can I do this for a range of files that start with a string like "file"?
To use variable expansion in bash, the variables must be non-quoted or double-quoted. Single quotes will prevent the expansion. On the other hand, you'd want to avoid expansion of * in 3s/.*/ in case you have a directory 3s containing files starting with ..
Fortunately, you can just chain strings together, so you can do
#!/bin/bash
today=$(date '+%Y_%m_%d__%H_%M_%S');
sed -i '3s/.*/CONFIG_LOCALVERSION='"$today"'/' ~/Desktop/file{1,2,Foo}
and can i do this for a range of file that start with a string like "file" ?
The glob ~/Desktop/file{1,2,Foo} will expand to ~/Desktop/file1 ~/Desktop/file2 ~/Desktop/fileFoo. If instead you want to match all files on your Desktop with a name starting with 'file', use ~/Desktop/file* instead.
I have a text file with a few basic words:
-banana
-mango
-sleep
When I run my script:
#!/bin/sh
WORD_FILE="testwoord.txt"
WORDS_ARRAY=cat $WORD_FILE
The output is like:
/home/username/bin/testword.txt: 1 /home/username/bin/restword.txt: banana: not found
/home/username/bin/testword.txt: 1 /home/username/bin/restword.txt: mango: not found
/home/username/bin/testword.txt: 1 /home/username/bin/restword.txt: sleep: not found
Why is it doing this? What I actually want is a script that reads words from a .txt file and puts it in an array.
To explain why this doesn't work:
WORDS_ARRAY=cat $WORD_FILE
runs the command generated by expanding, string-splitting, and glob-expanding $WORD_FILE with the variable WORDS_ARRAY exported in the environment with the value cat.
Instead, consider:
#!/bin/bash
# ^^ -- important that this is bash, not sh: POSIX sh doesn't have arrays!
WORD_FILE=testword.txt
readarray -t WORDS_ARRAY <"$WORD_FILE"
printf 'Read a word: %q\n' "${WORDS_ARRAY[#]}"
...which will create an actual array, not a string variable containing whitespace (as WORDS_ARRAY=$(cat $WORD_FILE) would).
By the way, using all-upper-case variable names is bad form here. To quote the POSIX spec:
Environment variable names used by the utilities in the Shell and Utilities volume of POSIX.1-2008 consist solely of uppercase letters, digits, and the ( '_' ) from the characters defined in Portable Character Set and do not begin with a digit. Other characters may be permitted by an implementation; applications shall tolerate the presence of such names. Uppercase and lowercase letters shall retain their unique identities and shall not be folded together. The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities.
To complement Charles Duffy's helpful answer:
Note that the variable names were changed to lowercase, as per Charles' recommendation.
Here a bash 3.x (and above) version of the command for reading lines into a bash array (readarray requires bash 4.x):
IFS=$'\n' read -d '' -ra words_array < "$word_file"
If you want to store individual words (across lines), use:
read -d '' -ra words_array < "$word_file"
To print the resulting array:
for ((i=0; i<"${#words_array[#]}"; i++)); do echo "word #$i: "${words_array[i]}""; done