I store information in a varible HOST in ansible via register command. Then I am trying to print the contents(INFORMATION-1 to 3) of variable HOST line after line.
INFORMATION-1
INFORMATION-2
INFORMATION-3
Instead I am getting this result when printed.
[[u'INFORMATION-1'], [u'INFORMATION-2'], [u'INFORMATION-3']]
Any ideas how I eliminate those unwanted characters like brackets [ ], u and apostrophe(') and print the result in my desired format?
Any ideas how I eliminate those unwanted characters like brackets [ ], u and apostrophe(') and print the result in my desired format?
It's because whatever thing you are printing is actually a python list containing 3 python lists which themselves contain a unicode str
If you want them to be line delimited, then the join() filter will do that for you, and it should be safe to use a join for the inner lists, too, in case you ever end up with more than one value in the inner list:
# assuming your values are in a variable named "list_list_str"
- debug: var=the_output
vars:
the_output: '{{ list_list_str | map("join", "\n") | join("\n") }}'
Related
I have a program that prints data into the console like so (separated by space):
variable1 value1
variable2 value2
variable3 value3
varialbe4 value4
EDIT: Actually the output can look like this:
data[variable1]: value1
pre[variable2] value2
variable3: value3
flag[variable4] value4
In the end I want to search for a part of the name e.g. for variable2 or variable3 but only get value2 or value3 as output.
EDIT: This single value should then be stored in a variable for further processing within the bash script.
I first tried to put all the console output into a file and process it from there with e.g.
# value3_var="$(grep "variable3" file.log | cut -d " " -f2)"
This works fine but is too slow. I need to process ~20 of these variables per run and this takes ~1-2 seconds on my system. Also I need to do this for ~500 runs. EDIT: I actually do not need to automatically process all of the ~20 'searches' automatically with one call of e.g. awk. If there is a way to do it automaticaly, it's fine, but ~20 calls in the bash script are fine here too.
Therefore I thought about putting the console output directly into a variable to remove the slow file access. But this will then eliminate the newline characters which then again makes it more complicated to process:
# console_output=$(./programm_call)
# echo $console_output
variable1 value1 variable2 value2 variable3 value3 varialbe4 value4
EDIT: IT actually looks like this:
# console_output=$(./programm_call)
# echo $console_output
data[variable1]: value1 pre[variable2] value2 variable3: value3 flag[variable4] value4
I found a solution for this kind of string arangement, but these seem only to work with a text file. At least I was not able to use the string stored in $console_output with these examples
How to print the next word after a found pattern with grep,sed and awk?
So, how can I output the next word after a found pattern, when providing a (long) string as variable?
PS: grep on my system does not know the parameter -P...
I'd suggest to use awk:
$ cat ip.txt
data[variable1]: value1
pre[variable2] value2
variable3: value3
flag[variable4] value4
$ cat var_list
variable1
variable3
$ awk 'NR==FNR{a[$1]; next}
{for(k in a) if(index($1, k)) print $2}' var_list ip.txt
value1
value3
To use output of another command as input file, use ./programm_call | awk '...' var_list - where - will indicate stdin as input.
This single value should then be stored in a variable for further processing within the bash script.
If you are doing further text processing, you could do it within awk and thus avoid a possible slower bash loop. See Why is using a shell loop to process text considered bad practice? for details.
Speed up suggestions:
Use LC_ALL=C awk '..' if input is ASCII (Note that as pointed out in comments, this doesn't apply for all cases, so you'll have to test it for your use case)
Use mawk if available, that is usually faster. GNU awk may still be faster for some cases, so again, you'll have to test it for your use case
Use ripgrep, which is usually faster than other grep programs.
$ ./programm_call | rg -No -m1 'variable1\S*\s+(\S+)' -r '$1'
value1
$ ./programm_call | rg -No -m1 'variable3\S*\s+(\S+)' -r '$1'
value3
Here, -o option is used to get only the matched portion. -r is used to get only the required text by replacing the matched portion with the value from the capture group. -m1 option is used to stop searching input once the first match is found. -N is used to disable line number prefix.
Exit after the first grep match, like so:
value3_var="$(grep -m1 "variable3" file.log | cut -d " " -f2)"
Or use Perl, also exiting after the first match. This eliminates the need for a pipe to another process:
value3_var="$(perl -le 'print $1, last if /^variable3\s+(.*)/' file.log)"
If I'm understanding your requirements correctly, how about feeding
the output of programm_call directly to the awk script instead of
assinging a shell variable.
./programm_call | awk '
# the following block is invoked line by line of the input
{
a[$1] = $2
}
# the following block is executed after all lines are read
END {
# please modify the print statement depending on your required output format
print "variable1 = " a["variable1"]
print "variable3 = " a["variable3"]
}'
Output:
variable1 = value1
variable3 = value3
As you see, the script can process all (~20) variables at once.
[UPDATE]
Assumptions including the provided information:
The ./program_call prints approx. 50 pairs of "variable value"
variable and value are delimited by blank character(s)
variable may be enclosed with [ and ]
variable may be followed by :
We have interest with up to 20 variables out of the ~50 pairs
We use just one of the 20 variables at once
We don't want to invoke ./program_call whenever accessing just one variable
We want to access the variable values from within bash script
We may use an associative array to fetch the value via the variable name
Then it will be convenient to read the variable-value pairs directly within
bash script:
#!/bin/bash
declare -A hash # declare an associative array
while read -r key val; do # read key (variable name) and value
key=${key#*[} # remove leading "[" and the characters before it
key=${key%:} # remove trailing ":"
key=${key%]} # remove trailing "]"
hash["$key"]="$val" # store the key and value pair
done < <(./program_call) # feed the output of "./program_call" to the loop
# then you can access the values via the variable name here
foo="${hash["variable2"]}" # the variable "foo" is assigned to "value2"
# do something here
bar="${hash["variable3"]}" # the variable "bar" is assigned to "value3"
# do something here
Some people criticize that bash is too slow to process text lines,
but we process just about 50 lines in this case. I tested a simulation by
generating 50 lines, processing the output with the script above,
repeating the whole process 1,000 times. It completed within a few seconds. (Meaning one batch ends within a few milliseconds.)
This is how to do the job efficiently AND robustly (your approach and all other current answers will result in false matches from some input and some values of the variables you want to search for):
$ cat tst.sh
#!/usr/bin/env bash
vars='variable2 variable3'
awk -v vars="$vars" '
BEGIN {
split(vars,tmp)
for (i in tmp) {
tags[tmp[i]":"]
tags["["tmp[i]"]"]
tags["["tmp[i]"]:"]
}
}
$1 in tags || ( (s=index($1,"[")) && (substr($1,s) in tags) ) {
print $2
}
' "${#:--}"
$ ./tst.sh file
value2
value3
$ cat file | ./tst.sh
value2
value3
Note that the only loop is in the BEGIN section where it populates a hash table (tags[]) with the strings from the input that could match your variable list so that while processing the input it doesn't have to loop, it just does a hash lookup of the current $1 which will be very efficient as well as robust (e.g. will not fail on partial matches or even regexp metachars).
As shown, it'll work whether the input is coming from a file or a pipe. If that's not all you need then edit your question to clarify your requirements and improve your example to show a case where this does not do what you want.
I'm trying to get back an array from perl to bash.
My perl scrip has an array and then I use return(#arr)
from my bash script I use
VAR = `perl....
when I echo VAR
I get the aray as 1 long string with all the array vars connected with no spaces.
Thanks
In the shell (and in Perl), backticks (``) capture the output of a command. However, Perl's return is normally for returning variables from subroutines - it does not produce output, so you probably want print instead. Also, in bash, array variables are declared with parentheses. So this works for me:
$ ARRAY=(`perl -wMstrict -le 'my #array = qw/foo bar baz/; print "#array"'`); \
echo "<${ARRAY[*]}> 0=${ARRAY[0]} 1=${ARRAY[1]} 2=${ARRAY[2]}"
<foo bar baz> 0=foo 1=bar 2=baz
In Perl, interpolating an array into a string (like "#array") will join the array with the special variable $" in between elements; that variable defaults to a single space. If you simply print #array, then the array elements will be joined by the variable $,, which is undef by default, meaning no space between the elements. This probably explains the behavior you mentioned ("the array vars connected with no spaces").
Note that the above will not work the way you expect if the elements of the array contain whitespace, because bash will split them into separate array elements. If your array does contain whitespace, then please provide an MCVE with sample data so we can perhaps make an alternative suggestion of how to return that back to bash. For example:
( # subshell so IFS is only affected locally
IFS=$'\n'
ARRAY=(`perl -wMstrict -e 'my #array = ("foo","bar","quz baz"); print join "\n", #array'`)
echo "0=<${ARRAY[0]}> 1=<${ARRAY[1]}> 2=<${ARRAY[2]}>"
)
Outputs: 0=<foo> 1=<bar> 2=<quz baz>
Here is one way using Bash word splitting, it will split the string on white space into the new array array:
array_str=$(perl -E '#a = 1..5; say "#a"')
array=( $array_str )
for item in ${array[#]} ; do
echo ": $item"
done
Output:
: 1
: 2
: 3
: 4
: 5
I have the snippet below. Basically, for an included task I would like to provide a variable whose contents look like the below string:
--date='something'
or it should be empty if the original variable is an empty string. The thing is, I need the string to be in the form above, including the single quotes around the value.
If I wouldn't need the single quotes, everything works perfectly! However, as I need them, I am trying to escape them using the below snippet. Unfortunately, what I have doesn't seem to work, as \' doesn't apply as expected. How can I properly escape ' so that get them in my string?
tasks:
- include_tasks: ../tasks/get_current.yml
- include_tasks: ../tasks/failed_jobs_stats.yml
vars:
date_param: "{{ date_start != '' | ternary('--date=\''+date_start+'\'', '') }}"
This is not a quoting problem, this is an operator precedence problem.
In your example, you:
apply the ternary filter to the empty string ''
compare the above result to date_start
What you need to do is to enclose the condition in parens:
date_param: "{{ (date_start != '') | ternary('--date=\''+date_start+'\'', '') }}"
I have a sample file like this:
r 2.1238 /NodeList/1/DeviceList/1/ ( type=0x806, source=00:00, destination=ff:ff) ns3::IPv4Header (source ipv4: 10.1.2.1 dest ipv4: 10.1.2.4)
+ 2.0076 /NodeList/0/DeviceList/1/ ( type=0x806, source=00:00, destination=ff:ff) ns3::ArpHeader (source ipv4: 10.1.2.1 dest ipv4: 10.1.2.4)
- 2.0077 /NodeList/1/DeviceList/1/ ( type=0x806, source=00:00, destination=ff:ff) ns3::IPv4Header (source ipv4: 10.1.2.1 dest ipv4: 10.1.2.4)
d 4.0042 /NodeList/2/DeviceList/1/ ( type=0x806, source=00:00, destination=ff:ff) ns3::IPv4Header (source ipv4: 10.1.2.1 dest ipv4: 10.1.2.4)
and so on.
The bold parts are the important fields. The first field can be '-', '+', 'r' or 'd'. The second field has a time stamp, in bold. Third field is to be the Node number, again in bold. The node number refers to the file to which the previous two fields is to be printed.
The restriction is that I want data from only those lines containing IPv4Header, eg. 1st, 3rd and 4th line in above sample.
So, I want my output to be like:
Node0.txt:
+ 2.0076
Node1.txt:
r 2.1238
- 2.0077
As you can see Node0 in line 2 is rejected as it does not contain IPv4Header. As we can see the number of output files being generated is equal to the number of nodes having IPv4Header.
Assume the number of lines and nodes to be multiple and variable for different files. Can someone please provide me with the necessary grep code?
awk to the rescue!
$ awk -F/ '/IPv4Header/{print $1 > "Node" $3 ".txt" }' file
should create two files with contents
Node1.txt: r 2.1238
- 2.0077
Node2.txt: d 4.0042
note that Node0 won't be created as per your definition.
What you are actually looking for is sed. For example, using sed you could do the following:
$ sed -n '/IPv4Header/s/^\([+-rd]\)[ ]*\([0-9.]*\)[^0-9]*\([0-9]\).*$/Node\3.txt \1 \2/p' file
Node1.txt r 2.1238
Node1.txt - 2.0077
Node2.txt d 4.0042
Where
sed -n suppresses normal outoput,
/IPv4Header/ locate only lines containing IPv4Header
What following is the standard sed substitute s/match/replace/ where match uses capture groups \(stuff\) to save stuff for use in replace using backreferences where \1, \2... is replaced with stuff1, stuff2, etc..
^\([+-rd]\) save the first character if it is one of +-rd in capture group 1,
[ ]* skip any number of spaces,
\([0-9.]*\) save sequence of 0-9 and . in capture group 2,
[^0-9]* skip anything not a 0-9,
\([0-9]\) capture the next number in capture group 3,
.*$ skip remaining chars
/Node\3 \1 \2/ replace with the backreferences in the order shown
p print it.
Give it a try and let me know if it is doing what you need.
I have a CSV file that I am piping through a set of awk/sed commands.
Some lines in the CSV file look like this:
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,"[1.1 - 3.0]","[0.384 - 0.768]"
where the 8th and 9th columns are a string representing a numeric range.
How can I use awk or sed to replace those fields with a numeric value? Either the beginning of the range, or the end of the range?
So this line would end up as
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,1.1,0.384
or
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,3.0,0.768
I got as far as removing the brackets but past that I'm stuck. I considered splitting on the " - ", but many lines in my file have a regular numeric value, not a range, in those last two columns, and that makes things messy (I don't want to end up with some lines having a different number of columns).
Here is a sed command that will take each range and break it up into two fields. It looks for strings like "[A - B]" and converts them to A,B. It can easily be modified to just use one of the values if needed by changing the \1,\2 portion. The regular expression assumes that all numbers have at least one digit on either side of a required decimal place. So, 1, .5, and 3. would not be valid. If you need that, the regex can be made to be more accommodating.
$ cat file
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,"[1.1 - 3.0]","[0.384 - 0.768]"
$ sed -Ee 's|"\[([0-9]+\.[0-9]+) - ([0-9]+\.[0-9]+)\]"|\1,\2|g' file
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,1.1,3.0,0.384,0.768
Since your data is field-based, awk is the logical choice.
Note that while awk generally isn't aware of double-quoted fields, that is not a problem here, because the double-quoted fields do not have embedded , instances.
#!/usr/bin/env bash
useStart1=1 # set to `0` to use the *end* of the *penultimate* fields' range instead.
useStart2=1 # set to `0` to use the *end* of the *last* field's range instead.
awk -v useStart1=$useStart1 -v useStart2=$useStart2 '
BEGIN { FS=OFS="," }
{
split($(NF-1), tokens1, /[][" -]+/)
split($NF, tokens2, /[][" -]+/)
$(NF-1) = useStart1 ? tokens1[2] : tokens1[3]
$NF = useStart2 ? tokens2[2] : tokens2[3]
print
}
' <<'EOF'
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,"[1.1 - 3.0]","[0.384 - 0.768]"
EOF
The code above yields:
10368,"Verizon DSL",DSL,NY,NORTHEAST,-5,-4,1.1,0.384
Modifying the values of $useStart1 and $useStart2 yields the appropriate variations.