Here is a sample bash script:
#!/bin/bash
array[0]="google.com"
array[1]="yahoo.com"
array[2]="bing.com"
pasteCommand="/usr/bin/paste -d'|'"
for val in "${array[#]}"; do
pasteCommand="${pasteCommand} <(echo \$(/usr/bin/dig -t A +short $val)) "
done
output=`$pasteCommand`
echo "$output"
Somehow it shows an error:
/usr/bin/paste: invalid option -- 't'
Try '/usr/bin/paste --help' for more information.
How can I fix it so that it works fine?
//EDIT:
Expected output is to get result from the 3 dig executions in a string delimited with | character. Mainly I am using paste that way because it allows to run the 3 dig commands in parallel and I can separate output using a delimiter so then I can easily parse it and still know the dig output to which domain (e.g google.com for first result) is assigned.
First, you should read BashFAQ/050 to understand why your approach failed. In short, do not put complex commands inside variables.
A simple bash script to give intended output could be something like that:
#!/bin/bash
sites=(google.com yahoo.com bing.com)
iplist=
for site in "${sites[#]}"; do
# Capture command's output into ips variable
ips=$(/usr/bin/dig -t A +short "$site")
# Prepend a '|' character, replace each newline character in ips variable
# with a space character and append the resulting string to the iplist variable
iplist+=\|${ips//$'\n'/' '}
done
iplist=${iplist:1} # Remove the leading '|' character
echo "$iplist"
outputs
172.217.18.14|98.137.246.7 72.30.35.9 98.138.219.231 98.137.246.8 72.30.35.10 98.138.219.232|13.107.21.200 204.79.197.200
It's easier to ask a question when you specify input and desired output in your question, then specify your try and why doesn't it work.
What i want is https://i.postimg.cc/13dsXvg7/required.png
$ array=("google.com" "yahoo.com" "bing.com")
$ printf "%s\n" "${array[#]}" | xargs -n1 sh -c '/usr/bin/dig -t A +short "$1" | paste -sd" "' _ | paste -sd '|'
172.217.16.14|72.30.35.9 98.138.219.231 98.137.246.7 98.137.246.8 72.30.35.10 98.138.219.232|204.79.197.200 13.107.21.200
I might try a recursive function like the following instead.
array=(google.com yahoo.com bing.com)
paster () {
dn=$1
shift
if [ "$#" -eq 0 ]; then
dig -t A +short "$dn"
else
paster "$#" | paste -d "|" <(dig -t A +short "$dn") -
fi
}
output=$(paster "${array[#]}")
echo "$output"
Now finally clear with expected output:
domains_arr=("google.com" "yahoo.com" "bing.com")
out_arr=()
for domain in "${domains_arr[#]}"
do
mapfile -t ips < <(dig -tA +short "$domain")
IFS=' '
# Join the ips array into a string with space as delimiter
# and add it to the out_arr
out_arr+=("${ips[*]}")
done
IFS='|'
# Join the out_arr array into a string with | as delimiter
echo "${out_arr[*]}"
If the array is big (and not just 3 sites) you may benefit from parallelization:
array=("google.com" "yahoo.com" "bing.com")
parallel -k 'echo $(/usr/bin/dig -t A +short {})' ::: "${array[#]}" |
paste -sd '|'
Related
I'm relatively new to bash scripting and I would like someone to explain this properly, thank you. Here is my code:
#! /bin/bash
echo "first arg: $1"
echo "first arg: $2"
var="$( grep -rnw $1 -e $2 | cut -d ":" -f1 )"
var2=$( grep -rnw $1 -e $2 | cut -d ":" -f1 | awk '{print substr($0,length,1)}')
echo "$var"
echo "$var2"
The problem I have is with the output, the script I'm trying to write is a c++ function searcher, so upon launching my script I have 2 arguments, one for the directory and the second one as the function name. This is how my output looks like:
first arg: Projekt
first arg: iseven
Projekt/AX/include/ax.h
Projekt/AX/src/ax.cpp
h
p
Now my question is: how do can I save the line by line output as a variable, so that later on I can use var as a path, or to use var2 as a character to compare. My plan was to use IF() statements to determine the type, idea: IF(last_char == p){echo:"something"}What I've tried was this question: Capturing multiple line output into a Bash variable and then giving it an array. So my code looked like: "${var[0]}". Please explain how can I use my line output later on, as variables.
I'd use readarray to populate an array variable just in case there's spaces in your command's output that shouldn't be used as field separators that would end up messing up foo=( ... ). And you can use shell parameter expansion substring syntax to get the last character of a variable; no need for that awk bit in your var2:
#!/usr/bin/env bash
readarray -t lines < <(printf "%s\n" "Projekt/AX/include/ax.h" "Projekt/AX/src/ax.cpp")
for line in "${lines[#]}"; do
printf "%s\n%s\n" "$line" "${line: -1}" # Note the space before the -1
done
will display
Projekt/AX/include/ax.h
h
Projekt/AX/src/ax.cpp
p
My intent is to write a shell script to extract a pattern ,using regular expressions, from a file and fill an array with all the ocurrences of the pattern in order to foreach it.
What is the best way to achieve this?
I am trying to do it using sed. And a problem I am facing is that the patterns can have newlines and these newlines must be considered, eg:
File content:
"My name
is XXX"
"My name is YYY"
"Today
is
the "
When I extract all patterns between double quotes, including the double quotes, the output of the first ocurrence must be:
"My name
is XXX"
fill an array with all the ocurrences of the pattern
First convert your file to have meaningful delimiter, ex. null byte, with ex. GNU sed with -z switch:
sed -z 's/"\([^"]*\)"[^"]*/\1\00/g'
I've added the [^"]* on the end, so that characters not between " are removed.
After it it becomes more trivial to parse it.
You can get the first element with:
head -z -n1
Or sort and count the occurrences:
sort -z | uniq -z -c
Or load to an array with bash's maparray:
maparray -d '' -t arr < <(<input sed -z 's/"\([^"]*\)"[^"]*/\1\00/'g))
Alternatively you can use ex. $'\01' as the separator, as long as it's unique, it becomes simple to parse such data in bash.
Handling such streams is a bit hard in bash. You can't set variable value in shell with embedded null byte. Also expect sometimes warnings on command substitutions. Usually when handling data with arbitrary bytes, I convert it with xxd -p to plain ascii and back with xxd -r -p. With that, it becomes easier.
The following script:
cat <<'EOF' >input
"My name
is XXX"
"My name is YYY"
"Today
is
the "
EOF
sed -z 's/"\([^"]*\)"[^"]*/\1\x00/g' input > input_parsed
echo "##First element is:"
printf '"'
<input_parsed head -z -n1
printf '"\n'
echo "##Elemets count are:"
<input_parsed sort -z | uniq -z -c
echo
echo "##The array is:"
mapfile -d '' -t arr <input_parsed
declare -p arr
will output (the formatting is a bit off, because of the non-newline delimetered output from uniq):
##First element is:
"My name
is XXX"
##Elemets count are:
1 My name
is XXX 1 My name is YYY 1 Today
is
the
##The array is:
declare -a arr=([0]=$'My name\nis XXX' [1]="My name is YYY" [2]=$'Today\nis\nthe ')
Tested on repl.it.
This may be what you're looking for, depending on the answers to the questions I posted in a comment:
$ readarray -d '' -t arr < <(grep -zo '"[^"]*"' file)
$ printf '%s\n' "${arr[0]}"
"My name
is XXX"
$ declare -p arr
declare -a arr=([0]=$'"My name \nis XXX"' [1]="\"My name is YYY\"" [2]=$'"Today\nis\nthe "')
It uses GNU grep for -z.
Sed can extract your desired pattern with or without newlines.
But if you want to store the multiple results into a bash array,
it may be easier to make use of bash regex.
Then please try the following:
lines=$(< "file") # slurp all lines
re='"[^"]+"' # regex to match substring between double quotes
while [[ $lines =~ ($re)(.*) ]]; do
array+=("${BASH_REMATCH[1]}") # push the matched pattern to the array
lines=${BASH_REMATCH[2]} # update $lines with the remaining part
done
# report the result
for (( i=0; i<${#array[#]}; i++ )); do
echo "$i: ${array[$i]}"
done
Output:
0: "My name
is XXX"
1: "My name is YYY"
2: "Today
is
the "
I wanted to modify dig command to automatically apply a reverse lookup to any A record I receive in output.
Therefore, I've created the following function:
dt ()
{
remove=$(echo $# | sed 's^https*^^' | sed 's^[,/*:]^^g' );
dig any +trace +nocl +nodnssec $remove | sed "s/\($remove.*A\s*\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\)/dig -x \2 +short | xargs echo \"\1 ->\"/e"
}
With this code I have the following output (only A record part is shown so to avoid the question getting bigger):
domain.com. 1200 A 198.54.115.174 -> server224-3.web-hosting.com.
However, now I also need to make a whois lookup using the IP I receive from dig output, but only in case dig -x \2 +short doesn't give any result (stackoverflow.com can be a good example of a domain with A records that do not have PTR).
I tried something like this to check exit code of regular host command (since dig implies that output is successful even if it's empty) and execute proper command depending on the result:
dig any +trace +nocl +nodnssec "$remove" | sed -e "s/\($remove.*A\s*\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\)/echo -n \"\1 -> \"; host \2 | grep -q ''; if [ ${PIPESTATUS[0]} -eq 0 ]; then dig -x \2 +short; else whois \2 | grep 'network:Network-Name:';fi; /e"
But it seems that sed somehow affects the value of ${PIPESTATUS[0]} array.
I wanted to do these modifications in sed because I needed something that will print lines on the go. If I use variables and modify the output from them, the function will work slower, at least visually.
Maybe awk can be of use here, but I am not sure how I should write the code using this command.
Is there a way around this problem? Can sed be used for this purpose or should I use some other tool? Thanks.
The good old' bash gives you variety of tools grep, awk, xargs, while read and so on. Using sed with e command with inside checking PIPESTATUS and executing xargs.... is just unreadable and very long too read. It's not clear to me what do you want to do, as too much happens on one line.
The dig command results zero status on success - you can check it's output if it has zero length when it "fails".
Consider the following script:
dt () {
remove=$(
<<<$1 \
sed 's#^http[s]?://\([^/]*\).*#\1#'
)
dig any +trace +nocl +nodnssec "$remove" |
tr -s '\t' |
awk -v "remove=${remove}." '{
if ($1 == remove && $3 == "A") {
print
}
}' |
while IFS= read -r line; do
IFS=$'\t' read -r _ _ _ ip _ <<<"$line"
if ! dig_result=$(dig -x "$ip" +short); then
echo "Dig error" >&2
exit 1
fi
if [ -z "$dig_result" ]; then
printf "%s -> no-result-from-dig\n" "$line"
else
printf "%s -> %s\n" "$line" "$dig_result"
fi
done
}
First dig is run. Notice the "$remove" is quoted.
Just a precaution - I squeze the tabulators in the output.
Then I filter - the first column should have "$remove". with a dot and the third column should be an "A"
Then for each line (as there are meny)
I get the ip address (ok, maybe ip=$(<<<"$line" cut -f4) would be cleaner and simpler).
I get the result from the dig -x using the ip address
If the result is empty I print "no-result-from-dig" if it not, I print the result.
Running:
dt stackoverflow.com
dt google.pl
Outputs:
stackoverflow.com. 300 A 151.101.65.69 -> no-result-from-dig
stackoverflow.com. 300 A 151.101.193.69 -> no-result-from-dig
stackoverflow.com. 300 A 151.101.129.69 -> no-result-from-dig
stackoverflow.com. 300 A 151.101.1.69 -> no-result-from-dig
google.pl. 300 A 216.58.215.67 -> waw02s16-in-f3.1e100.net.
The reason your command did not work is that $PIPESTATUS was quoted inside " quotes thus expanded before running the shell. You should escape the $ so that's \$PIPESTATUS or pass it inside nonexpading single quotes " ... "'$PIPESTATUS'" ... ".
Suppose we have the following command and its related output :
gsettings list-recursively org.gnome.Terminal.ProfilesList | head -n 1 | grep -oP '(?<=\[).*?(?=\])'
Output :
'b1dcc9dd-5262-4d8d-a863-c897e6d979b9', 'ca4b733c-53f2-4a7e-8a47-dce8de182546', '802e8bb8-1b78-4e1b-b97a-538d7e2f9c63', '892cd84f-9718-46ef-be06-eeda0a0550b1', '6a7d836f-b2e8-4a1e-87c9-e64e9692c8a8', '2b9e8848-0b4a-44c7-98c7-3a7e880e9b45', 'b23a4a62-3e25-40ae-844f-00fb1fc244d9'
I need to use gsettings command in a script and create filenames regarding to output ot gessetings command. For example a file name should be
b1dcc9dd-5262-4d8d-a863-c897e6d979b9
the next one :
ca4b733c-53f2-4a7e-8a47-dce8de182546
and so on.
How I can do this?
Another solution... just pipe the output of your command to:
your_command | sed "s/[ ']//g" | xargs -d, touch
You can use process substitution to read your gsettings output and store it in an array :
IFS=', ' read -r -a array < <(gsettings)
for f in "${array[#]}"
do
file=$(echo $f |tr -d "'" ) # removes leading and trailing quotes
touch "$file"
done
I have a shell script and a common configuration file where all the generic path, username and other values are stored. I want to get the value from this configuration file while I am running the sh script.
example:
sample.conf
pt_user_name=>xxxx
pt_passwd=>Junly#2014
jrnl_source_folder=>x/y/v
pt_source_folder=>/x/y/r/g
css_source_folder=>/home/d/g/h
Now i want get some thing like this in my sh script.
cd $css_source_folder
this command inside the shell script should take me to the location d/g/h while the script is running.
Is there any way to achieve this other than with grep and awk??
Thanks
Rinu
If you want to read from conf file everytime then grep and cut might help you,
suppose you need value for css_source_folder property
prop1="css_source_folder" (I am assuming you know property name whose value you want)
value_of_prop1=`grep $prop1 sample.conf| cut -f2 -d "=" | cut -f2 -d ">"`
like,
[db2inst2#pegdb2 ~]$ vi con.conf
[db2inst2#pegdb2 ~]$ grep css_source_folder con.conf
css_source_folder=>/home/d/g/h
[db2inst2#pegdb2 ~]$ value=`grep css_source_folder con.conf | cut -f2 -d "="`
[db2inst2#pegdb2 ~]$ echo $value
>/home/d/g/h
[db2inst2#pegdb2 ~]$ value=`grep css_source_folder con.conf | cut -f2 -d "=" | cut -f2 -d ">"`
[db2inst2#pegdb2 ~]$ echo $value
/home/d/g/h
If you want to read all properties at once, then apply loop and this will solve the purpose
Yes, you can get the configuration names and values relatively simple and associate them through array indexes. Reading your config can be done like this:
#!/bin/bash
test -r "$1" || { echo "error: unable to read conf file [$1]\n"; exit 1; }
declare -a tag
declare -a data
let index=0
while read line || test -n "$line"; do
tag[index]="${line%%\=*}"
data[index]="${line##*\>}"
((index++))
done < "$1"
for ((i=0; i<${#tag[#]}; i++)); do
printf " %18s %s\n" "${tag[$i]}" "${data[$i]}"
done
After reading the config file, you then have the config name tags and config values stored in the arrays tag and value, respectively:
pt_user_name xxxx
pt_passwd Junly#2014
jrnl_source_folder x/y/v
pt_source_folder /x/y/r/g
css_source_folder /home/d/g/h
At that point, it is a matter of determining how you will use them, whether as a password or as a directory. You may have to write a couple of functions, but the basic function of given a tag, get the correct data can be done like this:
function getvalue {
test -n "$1" || { echo "error in getvalue, no data supplied"; return 1; }
for ((i=0; i<${#tag[#]}; i++)); do
if test "$1" = "${tag[$i]}"; then
echo " eval cmd ${data[$i]}"
return $i
fi
done
return -1
}
echo -e "\nget value for 'jrnl_source_folder'\n"
getvalue "jrnl_source_folder"
The function will return the index of the data value and can execute any command needed. You seem to have directory paths and passwords, so you may need a function for each. To illustrate, the output of the example is:
get value for jrnl_source_folder
eval cmd x/y/v
You can also use an associative array in later versions of BASH to store the tag and data in a single associative array. You may also be able to use indirect references on the tag and data values to process them. I simply took the straight forward approach in the example.
Try this eval $(awk -F'=>' '{print $1"=\""$2"\";"}' sample.conf):
EX:
eval $(awk -F'=>' '{print $1"=\""$2"\";"}' sample.conf); echo $pt_user_name
xxxx
Using sed :
eval $(sed -re 's/=>/="/g' -e 's/$/";/g' sample.conf); echo $pt_passwd
Junly#2014
Using perl :
eval $(perl -F'=>' -alne 'print "$F[0]=\"$F[1]\";"' sample.conf); echo $pt_source_folder
/x/y/r/g
Using tr :
eval $(tr -d '>' <sample.conf); echo "$css_source_folder"
/home/d/g/h
PS. Using tr blindly to remove > may cause undesirable results depending on the content of sample.conf, but for the one provided works fine.