This question already has answers here:
How to pass the value of a variable to the standard input of a command?
(9 answers)
Closed 1 year ago.
I have a shell script:
TOPDIR=`pwd`
FOLDER=$($TOPDIR | sed 's/\//\_/g')
if [[ condition ]];then
source ~/[$FOLDER]-build/build-env.sh
fi
the TOPDIR here is /home/uname/project, so the variable FOLDER is supposed to be _home_uname_project because sed is called to replace / with _.
But it goes wrong when executing, terminal tells that /home/uname/[]-build/build-env.sh: No such file or directory which, I guess, means that FOLDER is unexpected empty in the if-then statement. Can anybody help me with figuring this out?
If you look at the output of just
$TOPDIR | sed 's/\//\_/g'
you'll realize that it's empty; it's trying to execute a command equal to the contents of $TOPDIR and pipe the output of that into sed, but there is no output in the first place.
You could do
pwd | sed 's\//_/g'
instead (no need to escape _), which would work.
Or, instead of using an external tool, you could use parameter expansion
topdir="$(pwd)"
topdir="${topdir//\//_}"
with the same result.
Notice that uppercase variable names are discouraged, as they're more likely to clash with existing, reserved names.
Related
This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 6 years ago.
I have a variable LINE and want to use it with awk to pull out the contents of the line numbered LINE from table.txt and make that a new variable called NAME which is then used to grep another file.
NAME=`awk 'FNR==$LINE' < table.txt`
echo "this is $NAME"
Seems to be close, but not quite the syntax.
If I use:
NAME=`awk 'FNR==1' < table.txt`
echo "this is $NAME"
Then echo gives me the first line of table.txt, if I use 2 I get the 2nd line, 3 the 3rd line, then I stopped variations.
Thanks for any advice.
EDITed first post formatting faux pas.
You're looking for:
NAME=`awk -v line="$LINE" 'FNR==line' < table.txt`
but the backticks notation is obsolete so this is better:
NAME=$(awk -v line="$LINE" 'FNR==line' < table.txt)
and you should never use all-upper-case for variable names unless they are exported (in shell) to avoid clashing with builtin names so really it should be:
name=$(awk -v line="$line" 'FNR==line' < table.txt)
but whatever you're doing is almost certainly the wrong approach and should be done entirely within awk. Make sure you fully understand everything discussed in why-is-using-a-shell-loop-to-process-text-considered-bad-practice if you're considering using shell to manipulate text.
To complement Ed Morton's helpful awk-based answer:
If you only need to extract a single line by index, sed allows for a more concise solution that is also easier to optimize (note that I've changed the variable names to avoid all-uppercase variable names):
name=$(sed -n "$line {p;q;}")
-n tells sed not to print (possibly modified) input lines by default
$line, assuming this shell variable expands it to a positive integer (see caveat below), only matches the input line with that (1-based) index.
{p;q;}, prints (p) the matching line, then exits the overall script immediately (q) as an optimization (no need to read the remaining lines).
Note:
For more complex sed scripts it is NOT a good idea to use a double-quoted shell string with shell-variable expansion as the sed script, because understanding what is interpreted by the shell up front vs. what sed ends up seeing as a result can become confusing.
Heed Ed's point that you're likely better off solving your problem with awk alone, given that awk can do its own regex matching (probably no need for grep).
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 3 years ago.
I wanted to know how vulnerable is bash by code injection. So I wrote a script as simple as this:
#!/bin/bash
grep $1 $2
and saved it as greptest.sh. The quotes around the variables were dropped intentionally for vulnerability test, so grep "$1" "$2" is the preferred way.
Then I created test.txt:
sadhuer
sadjfh Hello
cusad
Hello
fgdfg
First was to show its proper use.
$ ./greptest.sh 'Hello' 'test.txt'
Output as expected:
sadjfh Hello
Hello
Then the first attack:
$ ./greptest.sh 'Hello test.txt'
Outputs the same as above. So, obviously, it does something due to missing quotes within the script - altough $2 is empty! Next try with $2 not empty for proving my assumption that $2 will be interpreted as a further input file:
$ ./greptest.sh 'Hello test.txt' 'nonexistingfile.txt'
outputs:
test.txt:sadjfh Hello
test.txt:Hello
grep: nonexistingfile.txt: No such file or directory
Then the harder attack: Trying to execute an arbitrary command:
$ ./greptest.sh 'Hello test.txt' '; ls'
outputs:
test.txt:sadjfh Hello
test.txt:Hello
grep: ;: No such file or directory
grep: ls: No such file or directory
I did not expect this. I thought the variables were subsitituted to yield
grep Hello test.txt ; ls
which should result in listing the current directory. So, is missing these quotes just ugly and error prone or a serious security concern I should care about (given the values of these parameters come from an untrusted source)?
You don't need to worry about the parameters passed to your greptest.sh bash script, as you are passing the parameters directly as parameters to the called program (grep in this case).
You need to worry about whatever calls greptest.sh injecting parameters onto that command line.
See this post for information.
One security issue of unquoted variables (though not code injection) is glob expansion DoS – a variable without quotes not only undergoes word splitting, but also globbing!
If the variable contains a pattern like */*/../../ repeated several times, like */*/../../*/*/../../, */*/../../*/*/../../*/*/../../ and so on, then the number of files that expands to increases exponentially with the length of the pattern.
Not related to quoting, but if the attacker can influence $PATH, $LD_PRELOAD, etc, then he can make grep in your script mean anything.
This question already has answers here:
Replace a string in shell script using a variable
(12 answers)
Closed 7 years ago.
I want to use sed command in a loop passing a variable say a such that it searches for a and in the line it gets a it replaces "true" to "false".
I have a text file containing 3000 different names and another xml file containing 15000 lines. in the lines in which these 3000 entries are there i need to make changes.
I have written a code snippet but that is not giving expected output. Can anyone help. Thanks in advance.
for i in {1..3000}; do
a=`awk NR==$i'{print $1}' names.txt`
# echo $a
sed -e '/$\a/ s/true/false/' abc.xml > abc_new.xml
done
You have to replace single-quotes(') around sed's parameters with double-quotes("). In bash, single-quote won't allow variable expansion. Also, you might want to use sed's in-place edit (pass -i option) in your for loop.
So the one liner script will look like:
for a in `cat names.txt`; do sed -i.bak -e "/$a/s/true/false/" abc.xml ; done
This question already has answers here:
Stop shell wildcard character expansion?
(4 answers)
Closed 7 years ago.
I have 4 parameters adding to excetute file:
./project01.sh /* */ /^ ^/
and i want to make each parameters as variable
but before that i want to modify it because it will be unconvenient for futher operation. So i would like to make it like "/*" so put it in "" and before each character give \ because some of then are special characters.
I tried use like that:
beg1=echo $1 | sed(some change with $1)
but it change immediately $1 which is /* to direction /bin /boot itd. what can i do in that case?
First, when calling your script, there's nothing you can do to avoid quoting. You must call it as:
./project01.sh '/*' '*/' '/^' '^/'
...if you want to prevent any potential for shell manipulation. (^ is safe with some shells but not all).
This is because in the case of ./project01.sh /* (without quotes), expansion happens before the script is even started, so once your script is running, it's too late to make changes.
Second, use more quotes within your script:
echo "$1" | sed ...
...or, better (to fix cases where $1 contains -E, -n, or a similar value):
printf '%s\n' "$1" | sed ...
...or, better yet, if your shell is bash rather than /bin/sh...
sed ... <<<"$1"
However, if your goal in using sed is to add syntax quoting to your arguments, this will never work: The arguments are already expanded before the script is even run, so $1 is already /bin.
This question already has answers here:
Bash and filenames with spaces
(6 answers)
Closed 8 years ago.
I'm writing a script to do variable substitution into a Java properties file, of the format name=value. I have a source file, source.env like this:
TEST_ENV_1=test environment variable one
TEST_ENV_2=http://test.environment.com/one
#this is a comment with an equal sign=blah
TEST_ENV_3=/var/log/test/env/2.log
My script will replace every occurence of TEST_ENV_1 in the file dest.env with "test environment variable one", and so on.
I'm trying to process a line at a time, and having problems because looping on output from a command like sed or grep tokenizes on white space rather than the entire line:
$ for i in `sed '/^ *#/d;s/#.*//' source.env`; do
echo $i
done
TEST_ENV_1=test
environment
variable
one
TEST_ENV_2=http://test.environment.com/one
TEST_ENV_3=/var/log/test/env/2.log
How do I treat them as lines? What I want to be able to do is split each line apart on the "=" sign and make a sed script with a bunch of substitution regex's based on the source.env file.
sed '/^ *#/d;s/#.*//' source.env | while read LINE; do
echo "$LINE"
done
An alternative is to change $IFS as per #Jim's answer. It's better to avoid backticks in this case as they'll cause the entire file to be read in at once, whereas piping the output of sed to while above will allow the file to be processed line by line without reading the whole thing in to memory.