Works:
repquota $HOME | awk "{if(\$3 > $MIN && \$3 < $MAX )print}"
But if i try insert this to variable it isn't working:
VARIABLE=`repquota $FULL_HOME | awk "{if(\$3 > $MIN && \$3 < $MAX )print}"`
awk: {if( > 1572864 && < 302118056)print}
awk: ^ syntax error
Your bash syntax is way off. You're not quoting variables, wrongly quoting an awk script, and using deprecated backticks. What you seem to be trying to do would be:
VARIABLE=$(repquota "$FULL_HOME" | awk -v min="$MIN" -v max="$MAX" '($3>min) && ($3<max)')
but since you didn't provide any sample input and expected output it's an untested guess and it's always hard to tell what you DO want from reading a script that doesn't do what you want.
Use the new command substitution syntax $(command):
VARIABLE=$(repquota $FULL_HOME | awk "{if(\$3 > $MIN && \$3 < $MAX )print}")
Explanation
From man bash:
When the old-style backquote form of substitution is used, backslash
retains its literal meaning except when followed by $, `, or \. The
first backquote not preceded by a backslash terminates the command sub‐
stitution. When using the $(command) form, all characters between the
parentheses make up the command; none are treated specially.
When using backslashes, a \$var inside a double-quoted string is not scaped, resulting in the value of $var being substituted, so awk does not see $3, as you expected.
You can see it with these commands:
var="I am a test string"
echo `echo "\$var"` # output: I am a test string
echo $(echo "\$var") # output: $var
Edit: As Ed Morton comments, you should not pass awk variables from shell that way, instead use the -v switch of awk:
VARIABLE=$(repquota $FULL_HOME | awk -v min="$MIN" -v max="$MAX" '{if($3 > min && $3 < max )print}')
Related
Say I have strings that look like this:
$ a='/o\\'
$ echo $a
/o\
$ b='\//\\\\/'
$ echo $b
\//\\/
I'd like a shell script (ideally a one-liner) to replace / occurrences by \ and vice-versa.
Suppose the command is called invert, it would yield (in a shell prompt):
$ invert $a
\o/
$ invert $b
/\\//\
For example using sed, it seems unavoidable to use a temporary character, which is not great, like so:
$ echo $a | sed 's#/#%#g' | sed 's#\\#/#g' | sed 's#%#\\#g'
\o/
$ echo $b | sed 's#/#%#g' | sed 's#\\#/#g' | sed 's#%#\\#g'
/\\//\
For some context, this is useful for proper printing of git log --graph --all | tac (I like to see newer commits at the bottom).
tr is your friend:
% echo 'abc' | tr ab ba
bac
% echo '/o\' | tr '\\/' '/\\'
\o/
(escaping the backslashes in the output might require a separate step)
I think this can be done with (g)awk:
$ echo a/\\b\\/c | gawk -F "/" 'BEGIN{ OFS="\\" } { for(i=1;i<=NF;i++) gsub(/\\/,"/",$i); print $0; }'
a\/b/\c
$ echo a\\/b/\\c | gawk -F "/" 'BEGIN{ OFS="\\" } { for(i=1;i<=NF;i++) gsub(/\\/,"/",$i); print $0; }'
a/\b\/c
$
-F "/" This defines the separator, The input will be split in "/", and should no longer contain a "/" character.
for(i=1;i<=NF;i++) gsub(/\\/,"/",$i);. This will replace, in all items in the input, the backslash (\) for a slash (/).
If you want to replace every instance of / with \, you can uses the y command of sed, which is quite similar to what tr does:
$ a='/o\'
$ echo "$a"
/o\
$ echo "$a" | sed 'y|/\\|\\/|'
\o/
$ b='\//\\/'
$ echo "$b"
\//\\/
$ echo "$b" | sed 'y|/\\|\\/|'
/\\//\
If you are strictly limited to GNU AWK you might get desired result following way, let file.txt content be
\//\\\\/
then
awk 'BEGIN{FPAT=".";OFS="";arr["/"]="\\";arr["\\"]="/"}{for(i=1;i<=NF;i+=1){if($i in arr){$i=arr[$i]}};print}' file.txt
gives output
/\\////\
Explanation: I inform GNU AWK that field is any single character using FPAT built-in variable and that output field separator (OFS) is empty string and create array where key-value pair represent charactertobereplace-replacement, \ needs to be escaped hence \\ denote literal \. Then for each line I iterate overall all fields using for loop and if given field hold character present in array arr keys I do exchange it for corresponding value, after loop I print line.
(tested in gawk 4.2.1)
I don't quite know the term(s) for this part of the bash shell. In one specific, yet critical, case my script falls afoul of this effect:
Objective
Perform this command chain within a script where the awk expression (-e) comes form a variable. This example works when it is a script argument.
echo "test string" | awk -e { print $0; }
Problem example
On the command line I am seeking to produce output of: "test string", viz.:
$ optE="-e "
$ argE="{ print \$0; }"
$ set -x; echo "test string" | awk $optE$argE ; set +x
+ awk -e '{' print '$0;' '}'
+ echo 'test string'
awk: cmd. line:1: {
awk: cmd. line:1: ^ unexpected newline or end of string
+ set +x
In a way I can see what's happened. Is there a good/best way to not have the $argE variable tokenised after it is expanded?
Typing that same command on the command line works as you know:
$ echo "test string" | awk -e '{ print $0; }'
test string
Because the expression is enclosed in single quote. I haven't found a way to make that happen using a variable...
$ optE="-e "
$ argE="'{ print \$0; }'"
$ set -x; echo "test string" | awk $optE$argE ; set +x
+ echo 'test string'
+ awk -e ''\''{' print '$0;' '}'\'''
awk: cmd. line:1: '{
awk: cmd. line:1: ^ invalid char ''' in expression
+ set +x
Needless to say, I'm on stackoverflow because the things I've tried and read in ohter questions, etc. Don't give desirable result.
Solutions welcome
Word splitting expansion is applied after the unquoted variable expansion (${var} or $var) is replaced by its expansion. Word splitting splits the result on (white-)spaces, no matter the quotes. No matter what you put inside the string, if the variable expansion is unquoted, the result will be word split. To affect word splitting, you have to change the way you expand the variable, not it's content (i.e. change $var to "$var").
Is there a good/best way to not have the $argE variable tokenised after it is expanded?
Yes, quote the expansion. Rule of a thumb: never $var always "$var". Also, check your scripts with shellcheck.
In your case, it's simple, just assign the variable the content you want to execute, and quote the expansions:
optE="-e"
argE='{ print $0; }'
echo "test string" | awk "$optE" "$argE"
^^^^^^^ - variable expansion inside quotes
For more cases, use bash arrays arr=(-e '{ print $0; }'), and properly awk "${arr[#]}" expand them.
Research: https://www.gnu.org/software/bash/manual/html_node/Shell-Operation.html , bash manual shell expansions , https://mywiki.wooledge.org/BashFAQ/050 https://mywiki.wooledge.org/Quotes https://mywiki.wooledge.org/BashPitfalls , When should I wrap quotes around a shell variable? https://github.com/koalaman/shellcheck/wiki/Sc2086 .
Need 2 arguments in awk command for one column.
Script, name todo.
#!/bin/bash
folder_main="$( cd $( dirname "${BASH_SOURCE[0]}" ) >/dev/null 2>&1 && pwd )"
if [ $1 = 'e' ]; then
mcedit $folder_main/kb/todo.kb
else
awk -F ',' '$1=="'$1'"' $folder_main/kb/todo.kb
fi
Expectation is when I write todo i, it will grep me lines with i OR c by the first column divided by ,.
I tried this.
awk -F ',' '$1=="{c|'$1'}"' $folder_main/kb/todo.kb
But nothing.
Thanks.
You should pass your shell variable to awk using -v and fix your awk syntax:
awk -F, -v a="$1" '$1 == "c" || $1 == a' "$folder_main/kb/todo.kb"
This sets the awk variable a to the value of the shell positional parameter $1, and prints the line if the first column is either "c" or whatever you passed as the first argument to the script.
You could also shorten the line slightly by using a regular expression match instead of two ==:
awk -F, -v a="$1" '$1 ~ "^(c|"a")$"' "$folder_main/kb/todo.kb"
Although I think that the first option is easier to read, personally. It is also safer to use, as a character with special meaning inside a regular expression (such as *, [, ( or {) could cause the script to either fail or behave in an unexpected way.
You can't use shell variables directly in awk like this. Instead you pass them into your awk script using the -v flag:
awk -F ',' -v searchterm=$1 '$1==searchterm' $folder_main/kb/todo.kb
I have a file VALIDATION_CONFIG_FILE.cfg which contains the records below:
ES_VDF_1|1
DE_VDF_1|2
ES_VDF_1|7
When I am using the grep command below by using variable then the command is returning ES_VDF_1 output. As per my understanding, command should not give any results. When I use the same command without using variables (use values directly) then command is returning no results, which is as expected. So what is the problem with variables which I am using?
FEED_ID_1_7="HU_VDF_1"
FEED_ID_2_7="ES_VDF_1"
FEED_ID_3_7="PT_VDF_2"
awk -F'|' '{ if($2=="7") print $1; }' VALIDATION_CONFIG_FILE.cfg |
grep -E -v '${FEED_ID_1_7}|${FEED_ID_2_7}|${FEED_ID_3_7}'
Output: ES_VDF_1
awk -F'|' '{ if($2=="7") print $1; }' VALIDATION_CONFIG_FILE.cfg |
grep -E -v 'ES_VDF_1|HU_VDF_1|PT_VDF_2'
Output: nothing
The problem you are seeing is that single quotes in Bash do not interpolate variables, whereas double quotes do.
For example with a variable imaginatively called "VARIABLE":
alex#yuzu:~$ export VARIABLE="foo"
If you echo it with double quotes, it is interpolated and the value of the variable is used:
alex#yuzu:~$ echo "$VARIABLE"
foo
But if you use single quotes the literal string '$VARIABLE' is used instead:
alex#yuzu:~$ echo '$VARIABLE'
$VARIABLE
The same goes for your grep.
grep -E -v '${FEED_ID_1_7}|${FEED_ID_2_7}|${FEED_ID_3_7}'
Should be:
grep -E -v "${FEED_ID_1_7}\|${FEED_ID_2_7}\|${FEED_ID_3_7}"
For example:
alex#yuzu:~$ echo "foo" | grep -E "$VARIABLE|$HOME|$USER"
foo
alex#yuzu:~$ echo "foo" | grep -E '$VARIABLE|$HOME|$USER'
[ no output ]
This is happening due to quotes.
Single quotes won't interpolate anything, but double quotes will do. Replace single quotes to double quotes with variables like below :
awk -F'|' '{ if($2=="7") print $1; }' VALIDATION_CONFIG_FILE.cfg |
grep -E -v "${FEED_ID_1_7}|${FEED_ID_2_7}|${FEED_ID_3_7}"
Refer bash manual for more details
Adding to Kaoru/Nishu Tayal's answer, you can make it safer further by using normal text search with fgrep and multiple -e:
fgrep -v -e "${FEED_ID_1_7}" -e "${FEED_ID_2_7}" -e "${FEED_ID_3_7}"
This would help prevent misinterpretations just in case special characters would be added to the values of variables.
If you don't have fgrep try grep -F.
Trying to pass a variable into awk from user input:
Have tried variations of awk -v with errors stating 'awk: invalid -v option', even though the option is listed in man files.
#! /bin/bash
read -p "Enter ClassID:" CLASS
read -p "Enter FacultyName:" FACULTY
awk '/FacultyName/ {print}' data-new.csv > $FACULTY.csv
awk -vclass=${CLASS} '/class/ {print}' data-new.csv >> $FACULTY.csv
echo Class is $CLASS
echo Faculty member is $FACULTY
Some versions of awk require a space between the -v and the variable assignment. Also, you should put the bash variable in double-quotes to prevent unwanted parsing by the shell (e.g. word splitting, wildcard expansion) before it's passed to awk. Finally, in awk /.../ is a constant regular expression (i.e. /class/ will search for the string "class", not the value of the variable "class"). With all of this corrected, here's the awk command that I think will do what you want:
awk -v class="${CLASS}" '$0 ~ class {print}' data-new.csv >> $FACULTY.csv
Now, is there any reason you're using this instead of:
grep "$CLASS" data-new.csv >> $FACULTY.csv
Your script is not clear to me, but these all work:
CLASS=ec123
echo | awk -vclass=$CLASS '{print class}'
echo | awk -vclass=${CLASS} '{print class}'