Output the value/word after one pattern has been found in string in variable (grep, awk, sed, pearl etc) - bash

Question

I have a program that prints data into the console like so (separated by space):
variable1 value1
variable2 value2
variable3 value3
varialbe4 value4
EDIT: Actually the output can look like this:
data[variable1]: value1
pre[variable2] value2
variable3: value3
flag[variable4] value4
In the end I want to search for a part of the name e.g. for variable2 or variable3 but only get value2 or value3 as output.
EDIT: This single value should then be stored in a variable for further processing within the bash script.
I first tried to put all the console output into a file and process it from there with e.g.
# value3_var="$(grep "variable3" file.log | cut -d " " -f2)"
This works fine but is too slow. I need to process ~20 of these variables per run and this takes ~1-2 seconds on my system. Also I need to do this for ~500 runs. EDIT: I actually do not need to automatically process all of the ~20 'searches' automatically with one call of e.g. awk. If there is a way to do it automaticaly, it's fine, but ~20 calls in the bash script are fine here too.
Therefore I thought about putting the console output directly into a variable to remove the slow file access. But this will then eliminate the newline characters which then again makes it more complicated to process:
# console_output=$(./programm_call)
# echo $console_output
variable1 value1 variable2 value2 variable3 value3 varialbe4 value4
EDIT: IT actually looks like this:
# console_output=$(./programm_call)
# echo $console_output
data[variable1]: value1 pre[variable2] value2 variable3: value3 flag[variable4] value4
I found a solution for this kind of string arangement, but these seem only to work with a text file. At least I was not able to use the string stored in $console_output with these examples
How to print the next word after a found pattern with grep,sed and awk?
So, how can I output the next word after a found pattern, when providing a (long) string as variable?
PS: grep on my system does not know the parameter -P...

Answer 1

I'd suggest to use awk:
$ cat ip.txt
data[variable1]: value1
pre[variable2] value2
variable3: value3
flag[variable4] value4
$ cat var_list
variable1
variable3
$ awk 'NR==FNR{a[$1]; next}
{for(k in a) if(index($1, k)) print $2}' var_list ip.txt
value1
value3
To use output of another command as input file, use ./programm_call | awk '...' var_list - where - will indicate stdin as input.
This single value should then be stored in a variable for further processing within the bash script.
If you are doing further text processing, you could do it within awk and thus avoid a possible slower bash loop. See Why is using a shell loop to process text considered bad practice? for details.
Speed up suggestions:
Use LC_ALL=C awk '..' if input is ASCII (Note that as pointed out in comments, this doesn't apply for all cases, so you'll have to test it for your use case)
Use mawk if available, that is usually faster. GNU awk may still be faster for some cases, so again, you'll have to test it for your use case
Use ripgrep, which is usually faster than other grep programs.
$ ./programm_call | rg -No -m1 'variable1\S*\s+(\S+)' -r '$1'
value1
$ ./programm_call | rg -No -m1 'variable3\S*\s+(\S+)' -r '$1'
value3
Here, -o option is used to get only the matched portion. -r is used to get only the required text by replacing the matched portion with the value from the capture group. -m1 option is used to stop searching input once the first match is found. -N is used to disable line number prefix.

Answer 2

Exit after the first grep match, like so:
value3_var="$(grep -m1 "variable3" file.log | cut -d " " -f2)"
Or use Perl, also exiting after the first match. This eliminates the need for a pipe to another process:
value3_var="$(perl -le 'print $1, last if /^variable3\s+(.*)/' file.log)"

Answer 3

If I'm understanding your requirements correctly, how about feeding
the output of programm_call directly to the awk script instead of
assinging a shell variable.
./programm_call | awk '
# the following block is invoked line by line of the input
{
a[$1] = $2
}
# the following block is executed after all lines are read
END {
# please modify the print statement depending on your required output format
print "variable1 = " a["variable1"]
print "variable3 = " a["variable3"]
}'
Output:
variable1 = value1
variable3 = value3
As you see, the script can process all (~20) variables at once.
[UPDATE]
Assumptions including the provided information:
The ./program_call prints approx. 50 pairs of "variable value"
variable and value are delimited by blank character(s)
variable may be enclosed with [ and ]
variable may be followed by :
We have interest with up to 20 variables out of the ~50 pairs
We use just one of the 20 variables at once
We don't want to invoke ./program_call whenever accessing just one variable
We want to access the variable values from within bash script
We may use an associative array to fetch the value via the variable name
Then it will be convenient to read the variable-value pairs directly within
bash script:
#!/bin/bash
declare -A hash # declare an associative array
while read -r key val; do # read key (variable name) and value
key=${key#*[} # remove leading "[" and the characters before it
key=${key%:} # remove trailing ":"
key=${key%]} # remove trailing "]"
hash["$key"]="$val" # store the key and value pair
done < <(./program_call) # feed the output of "./program_call" to the loop
# then you can access the values via the variable name here
foo="${hash["variable2"]}" # the variable "foo" is assigned to "value2"
# do something here
bar="${hash["variable3"]}" # the variable "bar" is assigned to "value3"
# do something here
Some people criticize that bash is too slow to process text lines,
but we process just about 50 lines in this case. I tested a simulation by
generating 50 lines, processing the output with the script above,
repeating the whole process 1,000 times. It completed within a few seconds. (Meaning one batch ends within a few milliseconds.)

Answer 4

This is how to do the job efficiently AND robustly (your approach and all other current answers will result in false matches from some input and some values of the variables you want to search for):
$ cat tst.sh
#!/usr/bin/env bash
vars='variable2 variable3'
awk -v vars="$vars" '
BEGIN {
split(vars,tmp)
for (i in tmp) {
tags[tmp[i]":"]
tags["["tmp[i]"]"]
tags["["tmp[i]"]:"]
}
}
$1 in tags || ( (s=index($1,"[")) && (substr($1,s) in tags) ) {
print $2
}
' "${#:--}"
$ ./tst.sh file
value2
value3
$ cat file | ./tst.sh
value2
value3
Note that the only loop is in the BEGIN section where it populates a hash table (tags[]) with the strings from the input that could match your variable list so that while processing the input it doesn't have to loop, it just does a hash lookup of the current $1 which will be very efficient as well as robust (e.g. will not fail on partial matches or even regexp metachars).
As shown, it'll work whether the input is coming from a file or a pipe. If that's not all you need then edit your question to clarify your requirements and improve your example to show a case where this does not do what you want.