Bash scripts return a substring not whole value in column for psql - bash

I want to get the value of message in db, which for example is "the error is missing index " but when I run the code belong, it only return me 'the' for error_type. How could this happen that only return me the first word but not the whole value of the query result? How can I get the query results with the whole value?
declare -a ROW=($(psql \
-X \
-U $DB_USER \
-h $DB_HOST \
-d $DB_NAME \
-p $DB_PORT \
--single-transaction \
--set AUTOCOMMIT=off \
--set ON_ERROR_STOP=on \
--no-align \
-t \
--field-separator ' ' \
--quiet \
-c "SELECT message
FROM error_message
WHERE created_at > '$t1' and created_at < '$t'")
)
error_type=${ROW[0]}
echo $error_type

Don't use an array, use an ordinary string variable to contain the whole result.
error_type=$(psql \
-X \
-U $DB_USER \
-h $DB_HOST \
-d $DB_NAME \
-p $DB_PORT \
--single-transaction \
--set AUTOCOMMIT=off \
--set ON_ERROR_STOP=on \
--no-align \
-t \
--field-separator ' ' \
--quiet \
-c "SELECT message
FROM error_message
WHERE created_at > '$t1' and created_at < '$t'")
echo "$error_type"
Remember to quote variables unless you need word splitting and wildcard expansion to be done.

Related

Addition of two variables in slurm script

I am having slurm scirpt processing fmri data and the maximum value I can give in an array is 999, but the name of my subject ist over 1000.
So I need to to an addition in my slurm script. I tried:
a=${SLURM_ARRAY_TASK_ID} sum=$(($a + 1200))
#!/bin/sh
#
#SBATCH --job-name psy-stephan_fmriprep_gsp
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8GB
#SBATCH --output /projects/core-psy/logs/nako/stephan/slurm-%j.log
#SBATCH --error /projects/core-psy/logs/nako/stephan/slurm-%j.err
a=${SLURM_ARRAY_TASK_ID}
sum=$(($a + 1200))
singularity run \
--home /projects/core-psy/tmp3:/home/fmriprep \
--cleanenv \
-B /projects/core-psy/data/nako//swunderl/GSP_new/:/input \
-B /projects/core-psy/data/nako/swunderl/GSP_new/derivatives:/output \
-B //projects/core-psy/data/nako/swunderl/GSP_new_workdir/:/workdir \
-B /projects/core-psy/data/nako/swunderl/license.txt:/license \
/projects/core-psy/images/fmriprep-20.2.2.simg /input/sub-$sum /output participant \
--fs-license-file /license \
--skip-bids-validation \
--use-aroma \
--fs-no-reconall \
-w /workdir/ \
#--output-layout bids \
# sbatch --account=core-psy sbatch-multiple-job.slurm
So i can pass as a command SLURM_ARRAY_TASK_ID as 1.
But the addition keeps giving me sub-0+1200 and not the actual sum of both numbers.
Since you do not need to perform math on a, you can perform variable expansion on the string to make a 4 digit subject label for your fmriprep command:
sum="1${SLURM_ARRAY_TASK_ID}"
This way sbatch -a 200 ./your_job_script.sh will run for sub-1200. If you have labels like 1001, you will need to add to the variable expansion since 001 becomes a SLURM_ARRAY_TASK_ID of 1.
Here's an example adapted from my own - albeit not the most succint - code for sbatch scripts
if [ ${#SLURM_ARRAY_TASK_ID} == 1 ];
then
inputNo="100${SLURM_ARRAY_TASK_ID}"
singularity run \
--home /projects/core-psy/tmp3:/home/fmriprep \
--cleanenv \
-B /projects/core-psy/data/nako//swunderl/GSP_new/:/input \
-B /projects/core-psy/data/nako/swunderl/GSP_new/derivatives:/output \
-B //projects/core-psy/data/nako/swunderl/GSP_new_workdir/:/workdir \
-B /projects/core-psy/data/nako/swunderl/license.txt:/license \
/projects/core-psy/images/fmriprep-20.2.2.simg /input/sub-${inputNo} /output participant \
--fs-license-file /license \
--skip-bids-validation \
--use-aroma \
--fs-no-reconall \
-w /workdir/
elif [ ${#SLURM_ARRAY_TASK_ID} == 2 ];
then
inputNo="10${SLURM_ARRAY_TASK_ID}"
singularity run \
--home /projects/core-psy/tmp3:/home/fmriprep \
--cleanenv \
-B /projects/core-psy/data/nako//swunderl/GSP_new/:/input \
-B /projects/core-psy/data/nako/swunderl/GSP_new/derivatives:/output \
-B //projects/core-psy/data/nako/swunderl/GSP_new_workdir/:/workdir \
-B /projects/core-psy/data/nako/swunderl/license.txt:/license \
/projects/core-psy/images/fmriprep-20.2.2.simg /input/sub-${inputNo} /output participant \
--fs-license-file /license \
--skip-bids-validation \
--use-aroma \
--fs-no-reconall \
-w /workdir/
elif [ ${#SLURM_ARRAY_TASK_ID} == 3 ];
then
inputNo="1${SLURM_ARRAY_TASK_ID}"
singularity run \
--home /projects/core-psy/tmp3:/home/fmriprep \
--cleanenv \
-B /projects/core-psy/data/nako//swunderl/GSP_new/:/input \
-B /projects/core-psy/data/nako/swunderl/GSP_new/derivatives:/output \
-B //projects/core-psy/data/nako/swunderl/GSP_new_workdir/:/workdir \
-B /projects/core-psy/data/nako/swunderl/license.txt:/license \
/projects/core-psy/images/fmriprep-20.2.2.simg /input/sub-${inputNo} /output participant \
--fs-license-file /license \
--skip-bids-validation \
--use-aroma \
--fs-no-reconall \
-w /workdir/
fi

query return null in bash for psql

I am tring to get the count for the records in a certain table in bash scripts but the $num is null when returned (should be a number). And the query is correct when I directly run in pgadmin that I can get the number of rows. Any one know what is wrong?
declare -a ROW=($(psql \
-X \
-U $DB_USER \
-h $DB_HOST \
-d $DB_NAME \
-p $DB_PORT \
--set ON_ERROR_STOP=on \
--no-align \
-t \
--field-separator ' ' \
--quiet \
-c "SELECT count(*) as num
FROM table_test)")
)
echo "num_error: $num here"
if [[ $num == 0 ]]; then
echo "no error occur within the past 1 hour"
elif [[ $num == '' ]]; then
echo "return nothing"
else echo "$num"
fi
SQL aliases don't become shell variables, so using AS num in the query will not set $num in the shell.
The output of the query is being put in the ROW array, so you can get the value you want from ${ROW[0]}. There's also no need to use an array if the query just returns a single value. So you could do:
num=$(psql \
-X \
-U $DB_USER \
-h $DB_HOST \
-d $DB_NAME \
-p $DB_PORT \
--set ON_ERROR_STOP=on \
--no-align \
-t \
--field-separator ' ' \
--quiet \
-c "SELECT count(*)
FROM table_test)")
)

How to set Content-Type in s-nail?

I am using s-nail to send emails via SMTP by a bash script. Here is an example:
cat data.html | s-nail -v \
-s "Subject Text" \
-r "reply#abc.de" \
-S smtp=smtp.abc.de:587 \
-S smtp-use-starttls \
-S smtp-auth=login \
-S smtp-auth-user="admin#abc.de" \
-S smtp-auth-password="password1234" \
-a foo.img \
foo#bar.com
By default the content type is being set to Text.
How to set the Content-Type to html?
A little late but here you go
-M "text/html"

Submit multiple json payloads with curl

I am working with the confluent kafka, zookeeper in docker. I successfully submit a json file to kafka topic then consume as follow
curl -X POST \
-H "Content-Type: application/json" \
--data '{"name": "quickstart-file-source", "config {"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector", "tasks.max":"1", "topic":"quickstart-data", "file": "/tmp/quickstart/input.json"}}' \
http://localhost:28081/connectors
Above curl command has only one json file which executes successfully but I need to post multiple json files. Is there any way to do it?
Here is my kafka connect
docker run -d \
--name=kafka-connect-avro \
--net=host \
-e CONNECT_BOOTSTRAP_SERVERS=localhost:29091 \
-e CONNECT_REST_PORT=28081 \
-e CONNECT_GROUP_ID="quickstart-avro" \
-e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-avro-config" \
-e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-avro-offsets" \
-e CONNECT_STATUS_STORAGE_TOPIC="quickstart-avro-status" \
-e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_REST_ADVERTISED_HOST_NAME="localhost" \
-e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
-v /tmp/quickstart/file:/tmp/quickstart \
confluentinc/cp-kafka-connect:latest
Reference link
You could make individual JSON files in the current directory and post them separately in a loop
e.g.
$ ls *.json # list your connectors
payload1.json
payload2.json
And then loop over them
for f in `ls *.json`; do
curl -X POST -H "Content-Type: application/json" \
--data#${f} http://localhost:28081/connectors
done
Or simpler to use cat

Add a conditional SORT variable to mongoexport

I'm trying to set up a conditional --sort option in mongoexport but I'm having trouble with the string interpretation of my variable.
Here is the code I'm trying to run :
#!/bin/bash
if [[ $IS_PROD == "true" ]]
then
SORT='--sort "{_id : -1}"'
else
SORT=""
fi
$MONGODB_HOME/bin/mongoexport \
--host=$HOST \
--port=$PORT \
--username=$USER \
--password=$PWD \
--db=$DB \
--limit=$LIMIT \
$SORT \
--collection=my_collection | \
sed 's,\\,\\\\,g' \
> $TMP_FILE
While running this I get the following error error parsing command line options: invalid argument for flag '--sort' (expected string): invalid syntax
I've tried several quotes configuration and still couldn't make it work. Could someone please help me on this one?
Thanks
using bash array
#!/bin/bash
if [[ $IS_PROD == "true" ]]
then
SORT=(--sort "{_id : -1}")
else
SORT=()
fi
$MONGODB_HOME/bin/mongoexport \
--host=$HOST \
--port=$PORT \
--username=$USER \
--password=$PWD \
--db=$DB \
--limit=$LIMIT \
"${SORT[#]}" \
--collection=my_collection | \
sed 's,\\,\\\\,g' \
> $TMP_FILE
Explanation: using single quotes prevent shell expansions and double quotes are literal, but after variable expansion the double quotes are still litteral and expanded string is split by spaces.
Otherwise to work around unbound variable bug
#!/bin/bash
options=(--host=$HOST \
--port=$PORT \
--username=$USER \
--password=$PWD \
--db=$DB \
--limit=$LIMIT)
if [[ $IS_PROD == "true" ]]
then
options+=(--sort "{_id : -1}")
fi
$MONGODB_HOME/bin/mongoexport \
"${options[#]}" \
--collection=my_collection | \
sed 's,\\,\\\\,g' \
> $TMP_FILE

Resources