I need to run the following command in a bash script or any script for that matter:
$ aws rds download-db-log-file-portion --db-instance-identifier $LIST1 --log-file-name $LIST2 --region us-west-2
I have a text file with all the hostnames $LIST1 and another file with all the log files names $LIST2.
Basically I would like to know the best way to take each entry from $LIST1 and download all the logs for that entry from $LIST2.
$LIST1 sample:
host1
host2
host3
$LIST2 sample:
error/mysql-error.log
error/mysql-error.log.0
error/mysql-error.log.1
Example of a regular run of the command:
$ aws rds download-db-log-file-portion --db-instance-identifier host1 --log-file-name error/mysql-error.log--region us-west-2
The problem is I have 100+ hosts and each host has about 90 logs.
Here is a bash solution:
while read -r host
do
while read -r logfile
do
aws rds download-db-log-file-portion --db-instance-identifier $host --log-file-name $logfile --region us-west-2
done < logfiles.txt
done < hosts.txt
Related
I'm creating ec2 instances and would like to get the user_data for each instance using the command:
aws ec2 describe-instance-attribute --instance-id i-xxxx --attribute userData --output text --query "UserData.Value" | base64 --decode > file.txt
When running this directly via terminal it works, I'm able to get the userData printed into file.txt. However, I need this to run in a shell script I have, which gets the instance id as a parameter.
The lines in test.sh are the following:
#!/bin/bash
echo "$(aws ec2 describe-instance-attribute --instance-id $1 --attribute userData --output text --query "UserData.Value" | base64 --decode)" > file.txt
Where $1 is the instance-id. When running:
./test.sh i-xxxxxxx
It creates an empty file.txt. I have changed the line in the script to:
echo "$(aws ec2 describe-instance-attribute --instance-id $1 --attribute userData --output text --query "UserData.Value" | base64 --decode)"
and it prints the userData to stdout. So why it is not working for file redirection?
Thank you,
This question already has answers here:
How can I loop over the output of a shell command?
(4 answers)
Closed 1 year ago.
I am trying to execute this script as bash test.sh
#!/bin/bash
for region in "aws ec2 describe-regions --output text | cut -f4";
do
echo -e "\nListing Instances in region:'$region'...";
aws ec2 describe-instances --region $region;
done
But the only output that I get for $region is the print of the command. so it says
"Listing Instances aws ec2 describe-regions .. "
How can I fix this ?
You aren't executing a command; you are iterating over a sequence containing exactly one element, the strings aws ec2 ...
You need a command substitution to actually execute the command:
for region in $(aws ec2 describe-regions --output text | cut -f4); do
echo -e ...
aws ec2 describe-instances --region "$region"
done
I am trying to create a file from a shell script variable (.sh file ext)
#!/bin/bash
LOGNAME=`date +"error/postgresql.log.%Y-%m-%d-%H00.csv"`
echo Log File Name is: ${LOGNAME}
echo > ${LOGNAME}
aws rds download-db-log-file-portion --db-instance-identifier randomDBname --log-file-name ${LOGNAME} --starting-token 0 --output -n text > ${LOGNAME} --profile aws
sleep 20
I receive this error:
error/postgresql.log.%Y-%m-%d-%H00.csv: No such file or directory
I have tried the following and all have the same error:
echo >> ${LOGNAME}
echo -n >> ${LOGNAME}
echo "Starting Log" > ${LOGNAME}
The echo log file name works fine and prints the log file name without issue. Any ideas what is causing this?
Resolution:
#!/bin/bash
LOGNAME=`date +"error/postgresql.log.%Y-%m-%d-%H00.csv"`
FILENAME=`date +"error-postgresql.log.%Y-%m-%d-%H00.csv"`
echo Log File Name is: ${LOGNAME}
aws rds download-db-log-file-portion --db-instance-identifier randomDBname --log-file-name ${LOGNAME} --starting-token 0 --output csv > ${FILENAME} --profile aws
Found the issue, it was the / in error/postgresql.
we have to check for the status of instance and iam trying to capture if any error to logfile. logfile has instance inforamtion but the error is not being writen to logfile below is code let me know what needs to be corrected
function wait-for-status {
instance=$1
target_status=$2
status=unknown
while [[ "$status" != "$target_status" ]]; do
status=`aws rds describe-db-instances \
--db-instance-identifier $instance | head -n 1 \
| awk -F \ '{print $10}'` >> ${log_file} 2>&1
sleep 30
echo $status >> ${log_file}
done
Rather than using all that head/awk stuff, if you want a value out of the CLI, you should use the --query parameter. For example:
aws rds describe-db-instances --db-instance-identifier xxx --query 'DBInstances[*].DBInstanceStatus'
See: Controlling Command Output from the AWS Command Line Interface - AWS Command Line Interface
Also, if your goal is to wait until an Amazon RDS instance is available, then you should use db-instance-available — AWS CLI Command Reference:
aws rds wait db-instance-available --db-instance-identifier xxx
I wrote a simple bash script to download my RDS postgres files.
But the kicker is that is all works fine tine terminal, but when I try the same thing in the script I get an error:
An error occurred (DBLogFileNotFoundFault) when calling the DownloadDBLogFilePortion operation: DBLog File: "error/postgresql.log.2017-11-05-23", is not found on the DB instance
The command in question is this:
aws rds download-db-log-file-portion --db-instance-identifier foobar --starting-token 0 --output text --log-file error/postgresql.log.2017-11-05-23 >> test.log
It all works fine, but when I put the exact same line in the bash script I get the error message that there are no db log files - which is nonsense, they are there.
This is the bash script:
download_generate_report() {
for filename in $( aws rds describe-db-log-files --db-instance-identifier $1 | awk {'print $2'} | grep $2 )
do
echo $filename
echo $1
aws rds download-db-log-file-portion --db-instance-identifier $1 --starting-token 0 --output text --log-file $filename >> /home/ubuntu/pgbadger_script/postgres_logs/postgres_$1.log.$2
done
}
Tnx,
Tom
I re-wrote your script a little and it seems to work for me. It barked about grep. This uses jq.
for filename in $( aws rds describe-db-log-files --db-instance-identifier $1 | jq -r '.DescribeDBLogFiles[] | .LogFileName' )
do
aws rds download-db-log-file-portion --db-instance-identifier $1 --output text --no-paginate --log-file $filename >> /tmp/postgres_$1.log.$2
done
Thanks you Ian, I have an issue, with aws cli 2.4, because the log files download truncated.
to solve this I changed --no-paginate with --starting-token 0 more info in the RDS Reference
finally in bash:
#/bin/bash
set -x
for filename in $( aws rds describe-db-log-files --db-instance-identifier $1 | jq -r '.DescribeDBLogFiles[] | .LogFileName' )
do
aws rds download-db-log-file-portion --db-instance-identifier $1 --output text --starting-token 0 --log-file $filename >> $filename
done