I would like to run a unix cron every day that does:
export all docs that are over 3 month old to a document
remove from the same docs from the collections.
For the export part I use:
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" --out ./test2.json
and the "./test2.sh" file contains:
#!/bin/bash
d=`date --date="-3 month" -u +"%Y-%m-%dT%H:%M:%SZ"`
echo '{ "timeCreated": { "$lte": { "$date": "'$d'" } } }'
For the remove part I can do:
mongo mydb /home/dev/removeDocs.js
removeDocs.js:
var d = new Date();
d.setMonth(d.getMonth()-3);
db.GameHistory.remove({ timeCreated: { $lte: d} });
How can I synchronize the 2 commands? I want to run the remove commend after the export finished. can I merge the 2 to the same cron?
Yes, you can.
The easiest way is to merge both commands into single one-liner:
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" --out ./test2.json && mongo mydb /home/dev/removeDocs.js
But I would recommend you to create shell script to archive your database:
#!/bin/bash
set -e # stop on first exception
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" --out ./test2.json
mongo mydb /home/dev/removeDocs.js
If you want to append each new chunk of exported data, you should replace --out with standard unix stdio redirection:
mongoexport --db mydb --collection mycollection\ --query "`./test2.sh`" >> ./test2.json
Related
I have written a bash script using Curl command. I'm using an SQL query that gives a count of value 5. I want to assign that count value to T_4. Not sure how to do that in bash script using Curl command.
#!/bin/sh
result=$(
curl --netrc-file ~/.netrc -d '[
{
"T_1": "Test1",
"T_2": "Test2",
"T_3": "Test3",
"T_4": "1"
}
]' -X POST -H "Content-Type: application/json" https://www.testurl.com -d "SELECT count(5) FROM DUAL")
echo "Response from server"
echo $result
exit
Also, when I run the above script in putty, I'm getting an error that says -
"errorCode":"SIP-10322","errorMessage":"SIP-10322: rows updated is not 1:0"
Need your input on this.
The output of the SQL query which is a metric value, I have to use it in Rest call(Post API). Can anyone guide me on this?
Use command substitution to execute the query and assign the output to a shell variable. For instance, if the DB is MySQL, you would use:
t_4=$(mysql -e "SELECT 5 FROM DUAL")
Then use the variable inside the JSON parameter.
json='[
{
"T_1": "Test1",
"T_2": "Test2",
"T_3": "Test3",
"T_4": "'$t_4'"
}
]'
result=$(curl --netrc-file ~/.netrc -d "$json" -X POST -H "Content-Type: application/json" https://www.testurl.com)
How can I execute the following bash commands consecutively?
aws logs create-export-task --task-name "cloudwatch-log-group-export1" \
--log-group-name "/my/log/group1" \
--from 1488708419000 --to 1614938819000 \
--destination "my-s3-bucket" \
--destination-prefix "my-log-group1"
aws logs create-export-task --task-name "cloudwatch-log-group-export" \
--log-group-name "/my/log/group2" \
--from 1488708419000 --to 1614938819000 \
--destination "my-s3-bucket" \
--destination-prefix "my-log-group2"
The problem I have with the above commands is that after the first command completes execution, the script will stuck at the following state, making the second command not reachable.
{
"taskId": "0e3cdd4e-1e95-4b98-bd8b-3291ee69f9ae"
}
It seems that I should find a way to wait for cloudwatch-log-group-export1 task to complete.
You could have to crate a waiter function which uses describe-export-tasks to get current status of an export job.
Example of such function:
wait_for_export() {
local sleep_time=${2:-10}
while true; do
job_status=$(aws logs describe-export-tasks \
--task-id ${1} \
--query "exportTasks[0].status.code" \
--output text)
echo ${job_status}
[[ $job_status == "COMPLETED" ]] && break
sleep ${sleep_time}
done
}
Then you use it:
task_id1=$(aws logs create-export-task \
--task-name "cloudwatch-log-group-export1" \
--log-group-name "/my/log/group1" \
--from 1488708419000 --to 1614938819000 \
--destination "my-s3-bucket" \
--destination-prefix "my-log-group1" \
--query 'taskId' --output text)
wait_for_export ${task_id1}
# second export
aws-cli auto access to vim edit mode by default.
You can avoid it by setting AWS_PAGER environment variable is "" before execute aws command.
export AWS_PAGER=""
aws logs create-export-task...
Or, you can fix it in to aws's config file (~/.aws/config):
[default]
cli_pager=
I'm trying to run the following code but is encountering the "was unexpected at this time" error.
(echo COPY (SELECT ta.colA as name, ta.colB as user_e, ta.colC as user_n, ta.activation_dt, ta.creation_dt, MAX(tb.update_dt) as updated_at, MAX(tb.login_dt) as lastest_login, tc.colD as roleFROM tblA ta, tblB tb, tblC tc WHERE ta.id = tb.tb_id AND ta.tc_id = tc.id AND tc.colD <> 'Guest' GROUP BY ta.colA, ta.colB, ta.colC, ta.activation_dt, ta.creation_dt, tc.colD ORDER BY ta.colA, tc.colD^^^) TO 'E:\Details.csv' CSV DELIMITER ',' HEADER;) | psql -h localhost -p 8060 -U uname -d dbase
Looking for some insights please. Thank you.
Screenshot of error encountered
Try adding some quotes around the SQL, and lose the brackets:
echo "COPY ..." | psql -h localhost -p 8060 -U uname -d dbase
or use -c option:
psql -h localhost -p 8060 -U uname -d dbase -c "COPY ..."
I prefer the -c because it works on all OS
I have a bash function where I check if a PostgreSQL database already exists.
I capture the output. If database exist PostgreSQL returns the database name as response.
function is_database() {
local database=$1
local output=$(sudo -u postgres psql -c "SELECT datname FROM pg_catalog.pg_database WHERE datname=\"$database\";")
if [[ $output = *"${1}"* ]]
then
return 0
else
return 1
fi
}
is_database test
I get the following error:
column "test" does not exist
I am not searching for a table, but a database.
Use single quotes for string literals:
sudo -u postgres psql \
-c "SELECT datname FROM pg_catalog.pg_database WHERE datname='$database'"
Your code as it is won't work for database names like has spaces or has'quotes.
This question already has answers here:
store postgresql result in bash variable
(2 answers)
Closed 2 years ago.
I'm making an script in bash in what i run a psql query with this:
VAR="$(psql -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = 'query'")"
The problem is the content on $VAR is like this:
value ----------------- result (1 row)
I just need result in my $VAR to use it in the rest of the script.
VAR=`psql -t -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = 'query'"`
or
VAR=$(psql -t -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = 'query'")
The -t returns only the tuple (data).
See psql documentation about available options.
EDIT
I've been able to use a subsheel as suggested here : https://stackoverflow.com/a/21193276/14673
psql -t -h prov-db-cl -p 5446 -d prov -U prov -c "SELECT value FROM table where query = '`echo $VAR`'"