How do I pass subshell results (array) to an SSH command? - bash

Trying it this way:
#!/bin/bash
myvals=`psql -d mydb -c "select id from table1 where 't'"`
ssh user1#host1.domain.tld "for i in $myvals; do echo \$i >> values; done"
As long as psql returns just one value, it works fine. But if its several values, I receive this response:
bash: -c: line 1: syntax error near unexpected token `2'
bash: -c: line 1: `2'
Also, I tried to:
myvals='1 2 3'
And then it works fine: the values 1 2 3 are appended to the "values" file on the remote host; no error mesages.
If I try another subshell command, such as myvals=ls /bin, errors reappear.
It's clear that $myvals is evaluated on the local host already but what makes the subshell results so different?

If It's Not Really An Array...
Iterating over a string as if it were an array is innately buggy. Don't do it. That said, to generate a safely-escaped (eval-safe) version of your value, use printf %q.
#!/bin/bash
myvals=`psql -d mydb -c "select id from table1 where 't'"`
printf -v myvals_q %q "$myvals"
ssh user1#host1.domain.tld \
"myvals=$myvals_q;"' for i in $myvals; do echo "$i"; done >>values'
If You Actually Had An Array
#!/bin/bash
readarray -t myvals < <(psql -d mydb -c "select id from table1 where 't'")
printf -v myvals_q '%q ' "${myvals[#]}"
ssh user1#host1.domain.tld \
"myvals=( $myvals_q );"' for i in "${myvals[#]}"; do echo "$i"; done >>values'
If You Don't Need To Store The Value Locally In The First Place
#!/bin/bash
ssh user1#host1.domain.tld \
'while read -r i; do echo "$i"; done >>values' \
< <(psql -d mydb -c "select id from table1 where 't'")
General Notes
Running echo "$i" >>values over and over in a loop is inefficient: Every time the line is run, it re-opens the values file. Instead, run the redirection >values over the whole loop; this truncates the file exactly once, at the loop's start, and appends all values generated therein.
Unquoted expansions are generally dangerous. For example, if foo='*', then $foo will be replaced with a list of files in the current directory, but "$foo" will emit the exact contents -- *. Similarly, tabs, whitespace runs, and various other contents can be unintentionally damaged by unquoted expansion, even when passing directly to echo.
You can switch quoting types in the same string -- thus, "$foo"'$foo' is one string, the first part of which is replaced with the value of the variable named foo, and the second component of which is the exact string $foo.

You can send the output as a file:
#!/bin/bash
psql -d mydb -c "select id from table1 where 't'" > /tmp/values
scp values user1#host1.domain.tld:/tmp/
or pipe it to the remote host:
psql -d mydb -c "select id from table1 where 't'" | \
ssh user1#host1.domain.tld 'while read line; do echo $line; done'

Related

Git Bash script echo command output inverted after parameter

i'm using git bash for windows (git version 2.18.0.windows.1) to write a bash script like this one:
file=$1
if [[ ! -s "$file" ]] ; then #empty
echo -e "${RED}Invalid argument. Pass the file with all GCIDs as INPUT!!!${NOCOLOR}"
else
number=$(cat $file | wc -l )
number=$(($number+1))
echo -e "** ${number} GCID detected **"
echo ""
while read -r gcidRead
do
gcid=${gcidRead}
echo -e "select distinct operation from audit_trail.audit_trail where gcid='$gcid';" >> query.txt
value=$(psql "host=XXXX port=62013 dbname=prodemeagcdm user=XXXX password=XXXX" <<-EOF
select distinct operation from audit_trail.audit_trail where gcid='$gcid';
\q
EOF
)
echo -e "${value}" >> output.txt
if grep -q delete_bupa output.txt ; then
echo -e "${gcid}" >> gcidDeleted.txt
fi
done < $file
fi
I created just to debug the query.txt file in which the output is:
';lect distinct operation from audit_trail.audit_trail where gcid='XXX
instead of
select distinct operation from audit_trail.audit_trail where gcid='XXX'
In short, every string after $gcid parameter will be written at the beginning of the entire string.
If I use a unix terminal the echo output is ok.
Why in git bash the "echo" command has the wrong output mentioned?
Thanks in advance
I think you see the output on terminal of a string which contains a CR (13 in dec or 0D in hex) : the last '; cahracters are written from the beginning of the line, thus overwriting the first two characters of the string.
The string actually consists of select dist... gcid='XXX\r';, and is just printed awkwardly (to a human) on the terminal.
There are many ways to drop CR from the input, here are two of them :
# remove all CR chars from the input :
cat $file | tr -d '\r' | while read -r gcdiRead; do
...
done
# remove all CR chars only at end of lines (e.g : when followed by LF) :
cat $file | sed -e 's/\r$//' | while read -r gcdiRead; do
...
done

Bash script with long command as a concatenated string

Here is a sample bash script:
#!/bin/bash
array[0]="google.com"
array[1]="yahoo.com"
array[2]="bing.com"
pasteCommand="/usr/bin/paste -d'|'"
for val in "${array[#]}"; do
pasteCommand="${pasteCommand} <(echo \$(/usr/bin/dig -t A +short $val)) "
done
output=`$pasteCommand`
echo "$output"
Somehow it shows an error:
/usr/bin/paste: invalid option -- 't'
Try '/usr/bin/paste --help' for more information.
How can I fix it so that it works fine?
//EDIT:
Expected output is to get result from the 3 dig executions in a string delimited with | character. Mainly I am using paste that way because it allows to run the 3 dig commands in parallel and I can separate output using a delimiter so then I can easily parse it and still know the dig output to which domain (e.g google.com for first result) is assigned.
First, you should read BashFAQ/050 to understand why your approach failed. In short, do not put complex commands inside variables.
A simple bash script to give intended output could be something like that:
#!/bin/bash
sites=(google.com yahoo.com bing.com)
iplist=
for site in "${sites[#]}"; do
# Capture command's output into ips variable
ips=$(/usr/bin/dig -t A +short "$site")
# Prepend a '|' character, replace each newline character in ips variable
# with a space character and append the resulting string to the iplist variable
iplist+=\|${ips//$'\n'/' '}
done
iplist=${iplist:1} # Remove the leading '|' character
echo "$iplist"
outputs
172.217.18.14|98.137.246.7 72.30.35.9 98.138.219.231 98.137.246.8 72.30.35.10 98.138.219.232|13.107.21.200 204.79.197.200
It's easier to ask a question when you specify input and desired output in your question, then specify your try and why doesn't it work.
What i want is https://i.postimg.cc/13dsXvg7/required.png
$ array=("google.com" "yahoo.com" "bing.com")
$ printf "%s\n" "${array[#]}" | xargs -n1 sh -c '/usr/bin/dig -t A +short "$1" | paste -sd" "' _ | paste -sd '|'
172.217.16.14|72.30.35.9 98.138.219.231 98.137.246.7 98.137.246.8 72.30.35.10 98.138.219.232|204.79.197.200 13.107.21.200
I might try a recursive function like the following instead.
array=(google.com yahoo.com bing.com)
paster () {
dn=$1
shift
if [ "$#" -eq 0 ]; then
dig -t A +short "$dn"
else
paster "$#" | paste -d "|" <(dig -t A +short "$dn") -
fi
}
output=$(paster "${array[#]}")
echo "$output"
Now finally clear with expected output:
domains_arr=("google.com" "yahoo.com" "bing.com")
out_arr=()
for domain in "${domains_arr[#]}"
do
mapfile -t ips < <(dig -tA +short "$domain")
IFS=' '
# Join the ips array into a string with space as delimiter
# and add it to the out_arr
out_arr+=("${ips[*]}")
done
IFS='|'
# Join the out_arr array into a string with | as delimiter
echo "${out_arr[*]}"
If the array is big (and not just 3 sites) you may benefit from parallelization:
array=("google.com" "yahoo.com" "bing.com")
parallel -k 'echo $(/usr/bin/dig -t A +short {})' ::: "${array[#]}" |
paste -sd '|'

How to get a bash variable from inside postgre's?

I'm kind of new in bash script and postgresql.
I saw in another question a way to run a bash script as psql user here.
I tried making a bash function as follow,
postgres_create_db(){
sudo su postgres <<- EOF
if psql -lqt | cut -d \| -f 1 | grep -qw nokia_aaa_poc_db; then
psql -c '\dt'
else
psql -c 'CREATE DATABASE nokia_AAA_poc_db;'
fi
EOF
exit
}
where this function will be called further in code, but I wonder if I can add a RETURN to the function that's actualy returning a varible that was first declared inside postgres bash (in between the EOF's). Like bellow:
postgres_create_db(){
sudo su postgres <<- EOF
if psql -lqt | cut -d \| -f 1 | grep -qw nokia_aaa_poc_db; then
psql -c '\dt'
exists=1 #where thats a variable that I want to access outside the postgres bash.
else
psql -c 'CREATE DATABASE nokia_AAA_poc_db;'
fi
EOF
exit
return exists
}
but it gives an error on shellcheck
return exists
^-- SC2152: Can only return 0-255. Other data should be written to stdout.
Functions in bash can only return values from 0 to 255 where 0 is success. Reference: Return value in a Bash function
So you can echo the variable like this instead:
#!/usr/bin/env bash
postgres_test() {
psql -c '\dt' &> /dev/null
declare exists=1
echo $exists
}
printf "%s\n" "$(postgres_test)"
This prints "1".
You'll also notice that I redirected the output of the Postgres command to /dev/null. This is because it would be combined in the function's output otherwise.
You might wish to redirect that output to a file instead.

How do you split a string from shell-redirect or `read`?

I'm trying to split key value pairs (around an = sign) which I then use to edit a config file, using bash. But I need an alternative to the <<< syntax for IFS.
The below works on my host system, but when i log in to my ubuntu virtual machine through ssh I have the wrong bash version. Whatever I try, <<< fails. (I am definitely calling the right version of bash at the top of the file, using #!/bin/bash (and I've tried #!/bin/sh etc too)).
I know I can use IFS as follows on my host mac os x system:
var="word=hello"
IFS='=' read -a array <<<"$var"
echo ${array[0]} ${array[1]]}
#alternative -for calling through e.g. sh file.sh param=value
for var in "$#"
do
IFS='=' read -a array <<<"$var"
echo ${array[0]} ${array[1]]}
done
#alternative
IFS='=' read -ra array <<< "a=b"
declare -p array
echo ${array[0]} ${array[1]}
But this doesn't work on my vm.
I also know that I can should be able to switch the <<< syntax through backticks, $() or echo "$var" | ... but I can't get it to work - as follows:
#Fails
IFS='=' read -ra myarray -d '' <"$var"
echo ${array[0]} ${array[1]]}
#Fails
echo "$var" | IFS='=' read -a array
echo ${array[0]} ${array[1]]}
#fails
echo "a=b" | IFS='=' read -a array
declare -p array
echo ${array[0]} ${array[1]}
Grateful for any pointers as I'm really new to bash.
Your first failed attempt is because < and <<< are different operators. < opens the named file.
The second fails because read only sets the value of array in the subshell started by the pipe; that shell exits after the completion of the pipe, and array disappears with it.
The third fails for the same reason as the second; the declare that follows doesn't make any difference.
Your attempts have been confounded because you have to use the variable in the same sub-shell as read.
$ echo 'foo=bar' | { IFS='=' read -a array; echo ${array[0]}; }
foo
And if you want your variable durable (ie, outside the sub-shell scope):
$ var=$(echo 'foo=bar' | { IFS='=' read -a array; echo ${array[0]}; })
$ echo $var
foo
Clearly, it isn't pretty.
Update: If -a is missing, that suggests you're out of the land of arrays. You can try parameter substitution:
str='foo=bar'
var=${str%=*}
val=${str#*=}
And if that doesn't work, fall back to good ole cut:
str='foo=bar'
var=$(echo $str | cut -f 1 -d =)
val=$(echo $str | cut -f 2 -d =)

psql --(record|field)-separator NUL

Is there some way to make psql separate the fields and records by \0, aka NUL? It's the only way to be able to pass arbitrary data to Bash scripts.
Based on Matthew Wood's answer, I would expect this to print more that 1 on a newly initialized database:
declare -i count=0
echo "\pset recordsep '\000'
\f '\000'
select typname from pg_type" | \
sudo -iu postgres psql --no-align --quiet --tuples-only -d dbname -U username | while IFS= read -r -d ''
do
#echo "$REPLY"
let count++
done
if [ -n "$REPLY" ]
then
#echo "$REPLY"
let count++
fi
echo $count
Workaround: Iff the SELECT results are unique, you can use this workaround to handle one at a time:
next_record() {
psql --no-align --quiet --tuples-only -d dbname -U username <<SQL
SELECT colname
FROM tablename
WHERE colname > '${1}'
ORDER BY colname
LIMIT 1
SQL
}
last_col=
while true
do
colx="$(next_record "$last_col"; printf x)"
if [ "$colx" = x ]
then
exit
fi
col="${colx%$'\nx'}" # The extra \n character is from psql
# Do your thing here
col_escaped="${col//"'"/''}" # Double single quotes
col_escaped="${col_escaped//\\/\\\\}" # Double backslashes
last_col="$col_escaped"
done
This is not supported. psql uses C print functions to print out the result tables, and printing a zero byte just doesn't work there.
Update: This is now supported in PostgreSQL 9.2-to-be (git).
Try this:
psql --field-separator '\000' --no-align -c '<your query>'
Edit: Maybe not. However, it appear to work in psql using these commands:
\f '\000'
\a
Newer versions of psql support the --field-separator-zero flag.

Resources