mongodb export and remove docs - bash

I would like to run a unix cron every day that does:
export all docs that are over 3 month old to a document
remove from the same docs from the collections.
For the export part I use:
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" --out ./test2.json
and the "./test2.sh" file contains:
#!/bin/bash
d=`date --date="-3 month" -u +"%Y-%m-%dT%H:%M:%SZ"`
echo '{ "timeCreated": { "$lte": { "$date": "'$d'" } } }'
For the remove part I can do:
mongo mydb /home/dev/removeDocs.js
removeDocs.js:
var d = new Date();
d.setMonth(d.getMonth()-3);
db.GameHistory.remove({ timeCreated: { $lte: d} });
How can I synchronize the 2 commands? I want to run the remove commend after the export finished. can I merge the 2 to the same cron?

Yes, you can.
The easiest way is to merge both commands into single one-liner:
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" --out ./test2.json && mongo mydb /home/dev/removeDocs.js
But I would recommend you to create shell script to archive your database:
#!/bin/bash
set -e # stop on first exception
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" --out ./test2.json
mongo mydb /home/dev/removeDocs.js
If you want to append each new chunk of exported data, you should replace --out with standard unix stdio redirection:
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" >> ./test2.json

Related

How to use sql query with Curl command

I have written a bash script using Curl command. I'm using an SQL query that gives a count of value 5. I want to assign that count value to T_4. Not sure how to do that in bash script using Curl command.
#!/bin/sh
result=$(
curl --netrc-file ~/.netrc -d '[
{
"T_1": "Test1",
"T_2": "Test2",
"T_3": "Test3",
"T_4": "1"
}
]' -X POST -H "Content-Type: application/json" https://www.testurl.com -d "SELECT count(5) FROM DUAL")
echo "Response from server"
echo $result
exit
Also, when I run the above script in putty, I'm getting an error that says -
"errorCode":"SIP-10322","errorMessage":"SIP-10322: rows updated is not 1:0"
Need your input on this.
The output of the SQL query which is a metric value, I have to use it in Rest call(Post API). Can anyone guide me on this?
Use command substitution to execute the query and assign the output to a shell variable. For instance, if the DB is MySQL, you would use:
t_4=$(mysql -e "SELECT 5 FROM DUAL")
Then use the variable inside the JSON parameter.
json='[
{
"T_1": "Test1",
"T_2": "Test2",
"T_3": "Test3",
"T_4": "'$t_4'"
}
]'
result=$(curl --netrc-file ~/.netrc -d "$json" -X POST -H "Content-Type: application/json" https://www.testurl.com)

How to run aws bash commands consecutively?

How can I execute the following bash commands consecutively?
aws logs create-export-task --task-name "cloudwatch-log-group-export1" \
--log-group-name "/my/log/group1" \
--from 1488708419000 --to 1614938819000 \
--destination "my-s3-bucket" \
--destination-prefix "my-log-group1"
aws logs create-export-task --task-name "cloudwatch-log-group-export" \
--log-group-name "/my/log/group2" \
--from 1488708419000 --to 1614938819000 \
--destination "my-s3-bucket" \
--destination-prefix "my-log-group2"
The problem I have with the above commands is that after the first command completes execution, the script will stuck at the following state, making the second command not reachable.
{
"taskId": "0e3cdd4e-1e95-4b98-bd8b-3291ee69f9ae"
}
It seems that I should find a way to wait for cloudwatch-log-group-export1 task to complete.
You could have to crate a waiter function which uses describe-export-tasks to get current status of an export job.
Example of such function:
wait_for_export() {
local sleep_time=${2:-10}
while true; do
job_status=$(aws logs describe-export-tasks \
--task-id ${1} \
--query "exportTasks[0].status.code" \
--output text)
echo ${job_status}
[[ $job_status == "COMPLETED" ]] && break
sleep ${sleep_time}
done
}
Then you use it:
task_id1=$(aws logs create-export-task \
--task-name "cloudwatch-log-group-export1" \
--log-group-name "/my/log/group1" \
--from 1488708419000 --to 1614938819000 \
--destination "my-s3-bucket" \
--destination-prefix "my-log-group1" \
--query 'taskId' --output text)
wait_for_export ${task_id1}
# second export
aws-cli auto access to vim edit mode by default.
You can avoid it by setting AWS_PAGER environment variable is "" before execute aws command.
export AWS_PAGER=""
aws logs create-export-task...
Or, you can fix it in to aws's config file (~/.aws/config):
[default]
cli_pager=

“was unexpected at this time.”

I'm trying to run the following code but is encountering the "was unexpected at this time" error.
(echo COPY (SELECT ta.colA as name, ta.colB as user_e, ta.colC as user_n, ta.activation_dt, ta.creation_dt, MAX(tb.update_dt) as updated_at, MAX(tb.login_dt) as lastest_login, tc.colD as roleFROM tblA ta, tblB tb, tblC tc WHERE ta.id = tb.tb_id AND ta.tc_id = tc.id AND tc.colD <> 'Guest' GROUP BY ta.colA, ta.colB, ta.colC, ta.activation_dt, ta.creation_dt, tc.colD ORDER BY ta.colA, tc.colD^^^) TO 'E:\Details.csv' CSV DELIMITER ',' HEADER;) | psql -h localhost -p 8060 -U uname -d dbase
Looking for some insights please. Thank you.
Screenshot of error encountered
Try adding some quotes around the SQL, and lose the brackets:
echo "COPY ..." | psql -h localhost -p 8060 -U uname -d dbase
or use -c option:
psql -h localhost -p 8060 -U uname -d dbase -c "COPY ..."
I prefer the -c because it works on all OS

Find if a PostgreSQL database exists with bash

I have a bash function where I check if a PostgreSQL database already exists.
I capture the output. If database exist PostgreSQL returns the database name as response.
function is_database() {
local database=$1
local output=$(sudo -u postgres psql -c "SELECT datname FROM pg_catalog.pg_database WHERE datname=\"$database\";")
if [[ $output = *"${1}"* ]]
then
return 0
else
return 1
fi
}
is_database test
I get the following error:
column "test" does not exist
I am not searching for a table, but a database.
Use single quotes for string literals:
sudo -u postgres psql \
-c "SELECT datname FROM pg_catalog.pg_database WHERE datname='$database'"
Your code as it is won't work for database names like has spaces or has'quotes.

Store value from postgresql in a bash variable [duplicate]

This question already has answers here:
store postgresql result in bash variable
(2 answers)
Closed 2 years ago.
I'm making an script in bash in what i run a psql query with this:
VAR="$(psql -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = 'query'")"
The problem is the content on $VAR is like this:
value ----------------- result (1 row)
I just need result in my $VAR to use it in the rest of the script.
VAR=`psql -t -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = 'query'"`
or
VAR=$(psql -t -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = 'query'")
The -t returns only the tuple (data).
See psql documentation about available options.
EDIT
I've been able to use a subsheel as suggested here : https://stackoverflow.com/a/21193276/14673
psql -t -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = '`echo $VAR`'"

Resources