I have a Python script that outputs two numbers like so: 1.0 2.0 (that's a space in between the numbers, but it can be a \t, or whatever. I want a bash variable to save the 1.0, and another variable to save the 2.0. Is this possible?
In the past, I've only "piped" one value into a variable like so:
var=`python file.py` ;
but now, I'm interested in saving two values from the python file. Conceptually, similar to:
var1,var2=`python file.py` ;
Any advice / help?
Thanks!
You can use something like this:
read var1 var2 < <(python file.py)
The funky <( ) syntax is called process substitution.
The one-liner I use for splitting fields is
... | awk '{print $1}' | ... # or $2, $3, etc.
so you could do
var = `foo`
var1 = `echo "$var" | awk '{print $1}'`
var2 = `echo "$var" | awk '{print $2}'`
edit: added quotes around $var
I guess the most efficient and elegant thing here would be to use readarray in order to read the value into an array. That's if you're okay with using arrays, of course. You should be, but you never know. This would require the delimiter to be a newline, though. Anyhow :
readarray -t values < <(python file.py)
Will get you an array of one element for each line output by the python file.py with the trailing newline removed. Check out man bash for other options for this very cool builtin.
Related
I want to parse the output of fio, I have format them so it has a nice delimiter.
182.07 MB/s|182.55 MB/s|364.62 MB/s|45.5k|45.6k|91.2k#682.65 MB/s|686.24 MB/s|1.36 GB/s|10.7k|10.7k|21.4k#665.21 MB/s|700.56 MB/s|1.36 GB/s|1.3k|1.4k|2.7k#751.97 MB/s|802.05 MB/s|1.55 GB/s|0.7k|0.8k|1.5k
I want to process each string separated with # sign, currently this is what I do.
Convert # to \n (newline)
fio_result=$(printf %s "$fio_result" | tr '#' '\n')
This will output the string like so.
182.07 MB/s|182.55 MB/s|364.62 MB/s|45.5k|45.6k|91.2k
682.65 MB/s|686.24 MB/s|1.36 GB/s|10.7k|10.7k|21.4k
665.21 MB/s|700.56 MB/s|1.36 GB/s|1.3k|1.4k|2.7k
751.97 MB/s|802.05 MB/s|1.55 GB/s|0.7k|0.8k|1.5k
Only after that, loop through the variable fio_result.
echo "$fio_result" | while IFS='|' read -r bla bla...
Does anyone have better idea how to achieve what I want ?
With bash you can do:
#!/bin/bash
fio_result='182.07 MB/s|182.55 MB/s|364.62 MB/s|45.5k|45.6k|91.2k#682.65 MB/s|686.24 MB/s|1.36 GB/s|10.7k|10.7k|21.4k#665.21 MB/s|700.56 MB/s|1.36 GB/s|1.3k|1.4k|2.7k#751.97 MB/s|802.05 MB/s|1.55 GB/s|0.7k|0.8k|1.5k'
while IFS='|' read -d '#' -ra arr
do
declare -p arr #=> shows what's inside 'arr'
done < <(
printf '%s' "$fio_result"
)
But, if your need is to format/extract/compute something from fio output then you should switch to an other tool more fit for the job than bash.
Example with awk: calculate the average of the first two columns:
printf '%s' "$fio_result" |
awk -F'|' -v RS='#' '{print ($1+$2)/2}'
I have a script (I don't have edit access) which output whether the server is up or not. Essentially outputting either true or false.
How can I convert that output from lets say true to a key value pair like server_up=true or server_up=false. I tried using awk '{print $0} but didn't get very far. Not really sure how I can prepend string before it.
Thanks for any insights team!
If the output is only true or false, for one server, you can assign it directly to a variable.
server_up=$( prog.sh some arg list )
If you need that as a k=v pair,
printf "server_up=%s\n" $( prog.sh some arg list )
Obviously you have to run the program itself to get the output, but there's no need for an awk.
If you just want to use awk,
prog.sh your args | awk '{ print "server_up="$0 }'
or sed,
prog.sh your args | sed 's/^/server_up=/'
if there might be spaces to clean up, then
prog.sh your args | sed 's/^ */server_up=/'
or
prog.sh your args | sed -E 's/^\s*(\S+)\s*$/server_up=\1/'
Just output the variable with = in front of it then.
printf server_up=
prog.sh your args
Using the builtin read command with ProcSub
#!/usr/bin/env bash
read -r value < <(script_that_generates_the_output)
server_up=$value
To answer this:
How can I convert that output from lets say true to a key value pair like server_up=true or server_up=false.
#!/usr/bin/env bash
read -r value < <(script_that_generates_the_output)
case "$value" in
true) server_up=true;;
false) server_up=false;;
esac
The case statement is most probably superfluous though.
I have a file called info.log which contains the line:
/home/jax/Main_X_1_A
X, 1 and A are meaningful and they can change. However "Main" and the underscores remain the same.
Is it possible to use a utility to assign a shell variable a value based on the information in info.log?
E.g.
MY_VERSION="?_?_?";
Where the question marks represent the single characters that are found in those locations.
For example if info.log contained this line:
/home/jax/Main_1_2_3
And we used that data to initialise a shell variable:
MY_VERSION=...
echo $MY_VERSION
The output would be:
1_2_3
Updating question with better example:
Info.log
MODULE=TEST
QUICK_BUILD_DIR=/usr/apps/Main_1_2_3
ANT_FILE=build.xml
FANCE=/usr/apps/test/Main_1_2_3
I want to be able to take these three numbers(1, 2 and 3):
QUICK_BUILD_DIR=/usr/apps/Main_1_2_3
And assign them to variables.
Note: 1, 2 and 3 are just example numbers and they can change.
Can you try this?
var="MY_VERSION=1_3_2"
version=$(echo $var | sed 's/.*MAIN_\(.*\)/\1/') #version will be 1_3_2
This uses bash and sed.
A GNU Awk Solution
$ MY_VERSION=$(awk -F/ '/Main_/ { sub(/Main_/, "", $NF); print $NF }' info.log)
$ echo "$MY_VERSION"
X_1_A
You can use this awk command:
cat file
/home/jill/Main_1_2_4
/home/jax/Main_1_2_3
/home/john/Main_X_1_A
awk -v u=jax -F '/' '$3==u{sub(/^Main_/, "", $4); print $4}' file
1_2_3
Here you can pass any username in u variable to awk (as jax is being passed here) and version will be picked from that particular line.
No need for external utilities. Bash can do the string manipulation for you:
$ cat info.log
/home/jax/Main_X_1_A
$ read -r a < info.log
$ b="${a#*_}"
$ echo "$b"
X_1_A
I have a file currently in the form
location1 attr attr ... attr
location2 attr attr ... attr
...
locationn attr atrr ... attr
What I want to do is go through each line, grab the location (first field) then iterate through the attributes. So far I know how to grab the first field, but not iterate through the attributes. There are also a different number of attributes for each line.
TEMP_LIST=$DIR/temp.list
while read LINE
do
x=`echo $LINE | awk '{print $1}'`
echo $x
done<$TEMP_LIST
Can someone tell me how to iterate through the attributes?
I want to get the effect like
while read LINE
do
location=`echo $LINES |awk '{print $1}'`
for attribute in attributes
do something involving the $location for the line and each individual $attribute
done<$TEMP_LIST
I am currently working in ksh shell, but any other unix shell is fine, I will find out how to translate. I am really grateful if someone could help as it would save me alot of time.
Thank you.
Similar to DreadPirateShawn's solution, but a bit simpler:
while read -r location all_attrs; do
read -ra attrs <<< "$all_attrs"
for attr in "${attrs[#]}"; do
: # do something with $location and $attr
done
done < inputfile
The second read line makes use of bash's herestring feature.
This might work in other shells too, but here's an approach that works in Bash:
#!/bin/bash
TEMP_LIST=temp.list
while read LINE
do
# Split line into array using space as delimiter.
IFS=' ' read -a array <<< $LINE
# Use first element of array as location.
location=${array[0]}
echo "First param: $location"
# Remove first element from array.
unset array[0]
# Loop through remaining array elements.
for i in "${array[#]}"
do
echo " Value: $i"
done
done < $TEMP_LIST
As you're already using awk in your posted code, why not learn how to use awk, as it is designed for this sort of problem.
while read LINE
do
location=`echo $LINES |awk '{print $1}'`
for attribute in attributes
do something involving the $location for the line and each individual $attribute
done<$TEMP_LIST
is written in awk as
#!/bin/bash
tempList="MyTempList.txt"
awk '{ # implied while loop for input records by default
location=$1
print "location=" location # location as a "header"
for (i=2;i<NF;i++) {
printf("attr%d=%s\t", i, $i) # print each attr with its number
}
printf("\n") # add new-line char to end of each line of attributes
}' ${tempList}
If you want to save your output, use awk '{.....}' ${tempList}> ${tempList}.new
Awk has numerous vars that it sets as it reads your files. NF mean NumberOfFields for the current line. So the for loop, starts at field 2, and prints all remaining fields on that line in the format provided (change to suit your needs). The i<=NF drives the ability to print all elems on a line.
Sometimes you'll want the 3rd to last elem on line, so you can perform math on the value stored in NF, like thirdFromLast=$(NF-3). For all variables that are numbers, you can "dereference" it as a value, and ask awk to print the value stored of the $N(th) field. i.e. try
print "thirdFromLast="(NF-3)
print "thirdFromLast="$(NF-3)
... to see the difference that the $ makes on a variable that holds a number.
(For large amounts of data, 1 awk process will be considerably more efficient that using subprocesses to gather parts of files.)
Also work your way through this tutorial grymoire's awk tutorial
IHTH
This is a very common script:
#!/bin/bash
teststr="col1 col2"
var1=`echo ${teststr} | awk '{print $1}'`
var2=`echo ${teststr} | awk '{print $2}'`
echo var1=${var1}
echo var2=${var2}
However I dont like this, especially when there are more fields to parse.
I guess there should be a better way like:
(var1,var2)=`echo ${teststr} | awk '{print $1 $2}'
(in my imagination)
Is that so?
Thanks for help to improve effeciency and save some CPU power.
This might work for you:
var=(col0 col1 col2)
echo "${var[1]}"
col1
Yes, you can, but the best practice is to use the awk way to pass variables to awk.
Example using shell script variables
awk -v awkVar1="$scriptVar1" -v awkVar2="$scriptVar2" '<your awk code>'
Example using environmental variables
awk -v awkVar1=ENVIRON["ENV_VAR1"] -v awkVar2=ENVIRON["ENV_VAR2"] '<your awk code>'
It's possible to use script and environmental variables at the same time
awk -v awkVar1=ENVIRON["ENV_VAR1"] -v awkVar2="$scriptVar2" '<your awk code>'
You may find bash tricks to circumvent the awk way to do it, but it's not safe.
Explanation and more examples
Awk works this way, because it's a programming language by itself and has it's own way to use variables 'inside' awk statements.
By 'inside' i mean the part between the single quotes.
Let's see an example, where we turn off DHCP in a config file, all done using variables in a shell script. I'm going to explain the last line of code.
The script isn't optimal, it's main purpose is to use script variables. Explaining how the script does its job is out of scope of this answer, the focus is on explaining the use of variables.
#!/bin/bash
# set some variables
# set path to the config file to edit
CONFIG_FILE=/etc/netplan/01-netcfg.yaml
# find the line number of the line to change using awk and assign it to a variable
DHCP_LINE=$(awk '/dhcp4: yes/{print FNR}' $CONFIG_FILE)
# get the number of spaces used for identation using awk and assign it to a variable
SPACES=$(awk -v awkDHCP_LINE="$DHCP_LINE" 'FNR==awkDHCP_LINE {print match($0,/[^ ]|$/)-1}' $CONFIG_FILE)
# find DHCP setting and turn it off if needed
awk -v awkDHCP_LINE="$DHCP_LINE" -v awkSPACES="$SPACES" 'FNR==awkDHCP_LINE {sub("dhcp4: yes", "dhcp4: no")}' $CONFIG_FILE
Let's break this last line up to pieces for explanation.
awk -v awkDHCP_LINE="$DHCP_LINE" -v awkSPACES="$SPACES"
This part above assigns the value of DHCP_LINE script variable to the awkDHCP_LINE awk variable and the the value of SPACES script variable to the awkSPACESawk variable.
Please note, that the SPACES variable is passed to awk for the sake of showing how to pass multiple variables only; the awk command doesn't process it.
'FNR==awkDHCP_LINE {sub("dhcp4: yes", "dhcp4: no")}'
This one above is the 'inside' part of awk where the variable(s) passed to awk can be used.
$CONFIG_FILE
This part is outside awk, a generic script variable is used to specify the file that should be processed.
I hope this clears things a bit :)
Note: if you have lots of variables to pass, the solution provided by #potong may prove a better approach depending on your use case.
Bash has Array Support, We just need to supply values dynamically :)
function test_set_array_from_awk(){
# Note : -a is required as declaring array
let -a myArr;
# Hard Coded Valeus
# myArr=( "Foo" "Bar" "other" );
# echo "${myArr[1]}" # Print Bar
# Dynamic values
myArr=( $(echo "" | awk '{print "Foo"; print "Bar"; print "Fooo-And-Bar"; }') );
# Value #index 0
echo "${myArr[0]}" # Print Foo
# Value #index 1
echo "${myArr[1]}" # Print Bar
# Array Length
echo ${#myArr[#]} # Print 3 as array length
# Safe Reading with Default value
echo "${myArr[10]-"Some-Default-Value"}" # Print Some-Default-Value
echo "${myArr[10]-0}" # Print 0
echo "${myArr[10]-''}" # Print ''
echo "${myArr[10]-}" # Print nothing
# With Dynamic Index
local n=2
echo "${myArr["${n}"]-}" # Print Fooo-And-Bar
}
# calling test function
test_set_array_from_awk
Bash Array Documentation : http://tldp.org/LDP/abs/html/arrays.html
You can also use shell set builtin to place whitespace seperated (or more accurately, IFS seperated) into the variables $1, $2 and so on:
#!/bin/bash
teststr="col1 col2"
set -- $teststr
echo "$1" # col1
echo "$2" # col2