I'm trying to make an awk command which stores an entire config file as variables.
The config file is in the following form (keys never have spaces, but values may):
key=value
key2=value two
And my awk command is:
$(awk -F= '{printf "declare %s=\"%s\"\n", $1, $2}' $file)
Running this without the outer subshell $(...) results in the exact commands that I want being printed, so my question is less about awk, and more about how I can run the output of awk as commands.
The command evaluates to:
declare 'key="value"'
which is somewhat of a problem, since then the double quotes are stored with the value. Even worse is when a space is introduced, which results in:
declare 'key2="value' two"
Of course, I cannot remove the quotes or the multi-word values cause problems.
I've tried most every solution I could find, such as set -f, eval, and system().
You don't need to use Awk for this but the do this with built-ins available. Read the config file properly using input redirection
#!/bin/bash
while IFS== read -r k v; do
declare "$k"="$v"
done < config_file
and source the file as
$ source script.sh
$ echo "$key"
value
$ echo "$key2"
value two
If source is not available explicitly, POSIX-ly way of doing it would be to do just
. ./script.sh
Related
I have a file with below commands
cat /some/dir/with/files/file1_name.tsv|awk -F "\\t" '{print $21$19$23$15}'
cat /some/dir/with/files/file2_name.tsv|awk -F "\\t" '{print $2$13$3$15}'
cat /some/dir/with/files/file3_name.tsv|awk -F "\\t" '{print $22$19$3$15}'
When i loop through the file to run the command, i get below error
cat file | while read line; do $line; done
cat: invalid option -- 'F'
Try `cat --help' for more information.
You are not executing the command properly as you intended it. Since you are reading line by line on the file (for unknown reason) you could call the interpreter directly as below
#!/bin/bash
# ^^^^ for running under 'bash' shell
while IFS= read -r line
do
printf "%s" "$line" | bash
done <file
But this has an overhead of creating a forking a new process for each line of the file. If your commands present under file are harmless and is safe to be run in one shot, you can just as
bash file
and be done with it.
Also for using awk, just do as below for each of the lines to avoid useless cat
awk -F "\\t" '{print $21$19$23$15}' file1_name.tsv
You are expecting the pipe (|) symbol to act as you are accustomed to, but it doesn't. To help you understand, try this :
A="ls / | grep e" # Loads a variable with a command with pipe
$A # Does not work
eval "$A" # Works
When expanding a variable without using eval, expansion and word splitting occurs after the shell interprets redirections and pipes, so your pipe symbol is seen just as a literal character.
Some options you have :
A) Avoid piping, by passing the file name as an argument
awk -F "\\t" '{print $21$19$23$15}' /some/dir/with/files/file1_name.tsv
B) Use eval as shown below, the potential security implications of which I would suggest you to research.
C) Put arguments in file and parse it, avoiding the use of eval, something like :
# Assumes arguments separated by spaces
IFS=" " read -r -a arguments;
awk "${arguments[#]-}"
D) Implement the parsing of your data files in Bash instead of awk, and use your configuration file to specify output without the need for expanding anything (e.g. by specifying fields to print separated by spaces).
The first three approaches involve some form of interpretation of outside data as code, and that comes with risks if the file used as input cannot be guaranteed safe. Approach C might be considered a bit better in that regard, but since the command you are calling is awk, an actual program is passed to awk, so whatever awk can do, an attacker (or careless user) with write access to your file can cause your script to do anything awk can do.
I have 3 text files. I would like to read them and store them in different variables and later concatenate them using paste and print them in console.
I tried the following code but it threw an error saying
File not found
Here is my code
#!/bin/sh
value_1=`cat file_1.txt`
value_2=`cat file_2.txt`
value_3 = paste $value_1 $value_2
echo "$value_3"
paste expects its arguments to be the names of files, not the content of files. With bash, ksh, or zsh, there is a way around this. Replace:
paste $value_1 $value_2
with:
paste <(echo "$value_1") <(echo "$value_2")
<(...) is called process substitution. It makes the output from the command inside the parens look like a file.
Improvement
If we don't know the first character in the output, then printf is more reliable than echo:
paste <(printf "%s" "$value_1") <(printf "%s" "$value_2")
Example
Let's use these two test files:
$ cat file1
1
2
$ cat file2
a
b
Now, let's read those files into variables and apply paste to those variables:
$ value_1=$(cat file1); value_2=$(cat file2)
$ paste <(printf "%s" "$value_1") <(printf "%s" "$value_2")
1 a
2 b
Or, saving the output in a variable:
$ value_3=$(paste <(printf "%s" "$value_1") <(printf "%s" "$value_2"))
$ echo "$value_3"
1 a
2 b
The third line should read
value_3=`paste file_1.txt file_2.txt`
You need the backticks, no space after value_3 and don't use the variables as arguments, use the file names.
The reason it is saying "file not found" is because the value_3 with a space after it is being interpreted as a command to be run.
#!/bin/sh
value_1=$(cat file_1.txt
value_2=$(cat file_2.txt)
value_3=$(echo $value_1 $value_2 | paste) # or value_3="$value1 $value2"
echo "$value_3"
Note :
no space allowed around = in shell
The backquote ` is used in the old-style command
substitution, e.g.
foo=`command`
The foo=$(command) syntax is recommended instead. Backslash handling inside $() is less surprising, and $() is easier to nest.
Check http://mywiki.wooledge.org/BashFAQ/082
I am trying to create a file using the following script (see below). While the script runs without errors (at least according to shellcheck), I cannot get the resulting file to have the correct name.
#!/bin/bash
# Set some variables
export site_path=~/Documents/Blog
drafts_path=~/Documents/Blog/_drafts
title="$title"
# Create the filename
title=$("$title" | "awk {print tolower($0)}")
filename="$title.markdown"
file_path="$drafts_path/$filename"
echo "File path: $file_path"
# Create the file, Add metadata fields
cat >"$file_path" <<EOL
---
title: \"$title\"
layout:
tags:
---
EOL
# Open the file in BBEdit
bbedit "$file_path"
exit 0
Very new to bash, so I'm not quite sure what I'm doing wrong...
The most glaring error is this:
title=$("$title" | "awk {print tolower($0)}")
It's wrong for several reasons:
This pipeline runs "$title" as a command -- meaning that it looks for a command named with the title of your blog post to run -- and pipes the output of that command (a command that presumably won't exist) to awk.
Using double-quotes around the entire awk command means you're looking for a command named something like /usr/bin/awk {print tolower(bash-)} (if $0 evaluates to bash-, which it will in an interactive interpreter; behavior will differ elsewhere).
Using double-quotes rather than single-quotes to protect your awk script means that the $0 gets evaluated to the shell rather than by awk.
A better alternative might look like:
title=$(awk '{print tolower($0)}' <<<"$title")
...or, to use simpler tools:
title=$(tr '[:upper:]' '[:lower:]' <<<"$title")
...or, to use bash 4.x built-in functionality:
title=${title,,}
Of course, all that assumes that title is set to start with. If you aren't passing it through your environment, you might want something like title=$1 rather than title="$title" earlier in your script.
I'm using the ls command to list files to be used as input. For each file found, I need to
Perform a system command (importdb) and write to a log file.
Write to an error log file if the first character of column 2, line 6 of the log file created in step 1 is not "0".
rename the file processed so it won't get re-processed on the next run.
My script:
#!/bin/sh
ls APCVENMAST_[0-9][0-9][0-9][0-9]_[0-9][0-9] |
while read LINE
do
importdb -a test901 APCVENMAST ${LINE} > importdb${LINE}.log
awk "{if (NR==6 && substr($2,1,1) != "0")
print "ERROR processing ", ${LINE} > importdb${LINE}err.log
}" < importdb${LINE}.log
mv ${LINE} ${LINE}.PROCESSED
done
This is very preliminary code, and I'm new to this, but I can't get passed parsing errors as the one below.
The error context is:
{if (NR==6 && >>> substr(, <<< awk The statement cannot be correctly parsed.
Issues:
Never double quote an awk script.
Always quote literal strings.
Pass in shell variables correctly either by using -v if you need to access the value in the BEGIN block or after the scripts i.e. awk -v awkvar="$shellvar" 'condition{code}' file or by awk condition{code}' awkvar="$shellvar"
Always quote shell variables.
Conditional should be outside block.
There is ambiguity with redirection and concatenation precedence so use parenthesis.
So the corrected (syntactical) script:
awk 'NR==6 && substr($2,1,1) != 0 {
print "ERROR processing ", line > ("importdb" line "err.log")
}' line="${LINE}" "importdb${LINE}.log"
You have many more issues but as I don't know what you are trying to achieve it's difficult to suggest the correct approach...
You shouldn't parse the output of ls
Awk reads files you don't need to loop using shell constructs
Is it possible to have an awk command within a bash script return values to a bash variable, i.e.,if my awk script does some arithmetic operations, can I store the answers in variables so, they can be accessed in the bash script. If possible, how to distinguish between multiple return variables. Thanks.
No. You can use exit to return an error code, but in general you can't modify the shell environment from a subprocess.
You can also, of course, print the desired content in awk and put it into variables in bash by using read:
read a b c <<< $(echo "foo" | awk '{ print $1; print $1; print $1 }')
Now $a, $b and $c are all 'foo'. Note that you have to use the <<<$() syntax to get read to work. If you use a pipeline of any sort a subprocess is created too and the environment read creates the variables in is lost when the pipeline is done executing.
var=$(awk '{ print $1}')
This should set var to the output of awk. Then you can use string functions or whatever from there to differentiate within the value or have awk print only the part you want.
I know this question is old, but there's another way to do this that's worked really well for me, and that's using an unused file descriptor. We all know stdin (&0), stdout (&1), and stderr (&2), but as long as you redirect it (aka: use it), there's no reason you can't use fd3 (&3).
The advantage to this method over other answers is that your awk script can still write to stdout like normal, but you also get the result variables in bash.
In your awk script, at the end, do something like this:
END {
# print the state to fd3
printf "SUM=%s;COUNT=%s\n", tot, cnt | "cat 1>&3"
}
Then, in your bash script, you can do something like this:
awk -f myscript.awk <mydata.txt 3>myresult.sh
source myresult.sh
echo "SUM=${SUM} COUNT=${COUNT}"