How can I use Expect to automate an AWS job? [closed] - bash

Question

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed last year.
Improve this question
I have a very large copy to my local machine from an S3 bucket, which iterates through a file of identifiers and copies matching files. It all works well… except that the copy is so large that it keeps timing out and asking me for my password (every 2 - 3 hours). All told, it took a fortnight to run last time I ran it - but would have been much faster if I'd entered the password immediately that I was requested for it (which was never going to happen because of meals / sleep / other).
There are two scripts. One does the actual work (the core script) and one does the reading of the file of identifiers and calls the core script.
I'd like to automate this task, but I'm having trouble working out how to exactly.
This is the core of the working (but slow, because it asks for the password) script…
#!/bin/bash
function errorexit() {
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
exit 1
}
​
FILEID=$(echo $1)
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
aws-adfs login --profile=default --adfs-host=myhost.com --role-arn=myroledeets --region=eu-west-1 && export AWS_PROFILE=default
if [ $? -ne 0 ]; then
exit 1
fi
​
ASSUMED_ROLE=$(aws sts assume-role --role-arn myroledeets --role-session-name seshname)
export AWS_SECRET_ACCESS_KEY=$(echo $ASSUMED_ROLE | jq -r .Credentials.SecretAccessKey)
export AWS_ACCESS_KEY_ID=$(echo $ASSUMED_ROLE | jq -r .Credentials.AccessKeyId)
export AWS_SESSION_TOKEN=$(echo $ASSUMED_ROLE | jq -r .Credentials.SessionToken)
​
if [ ${#AWS_ACCESS_KEY_ID} -lt 16 ] || [ ${#AWS_ACCESS_KEY_ID} -gt 128 ]; then
echo "Access Key Failure" >&2
errorexit
fi
​
if [ ${#AWS_SECRET_ACCESS_KEY} -lt 16 ] || [ ${#AWS_SECRET_ACCESS_KEY} -gt 128 ]; then
echo "Secret Access Key Failure" >&2
errorexit
fi
​
if [ ${#AWS_SESSION_TOKEN} -lt 128 ]; then
echo "Session Token Failure" >&2
errorexit
fi
​
AWSPATH="s3://mypath/$FILEID"
aws s3 ls $AWSPATH | cut -c 32- | while IFS= read -r line; do
COPYPATH="s3://mypath/$FILEID/$line"
echo $COPYPATH
aws s3 cp $COPYPATH ./$FILEID/$line --recursive
done
if [ $? -ne 0 ]; then
errorexit
fi
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
exit 0
I thought it would be a good idea to use Expect to sort this issue. So I did the following (this doesn't work!):
#!/bin/bash
function errorexit() {
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
exit 1
}
​
ACCESSPW=$(echo $1)
FILEID=$(echo $2)
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
expect <<EOD
spawn aws-adfs login --profile=default --adfs-host=myhost.com --role-arn=myroledeets --region=eu-west-1
set timeout 5
expect -r {
"Password: $" {
send "$ACCESSPW\r"
}
}
EOD
​
if [ $? -ne 0 ]; then
exit 1
fi
​
export AWS_PROFILE=default
ASSUMED_ROLE=$(aws sts assume-role --role-arn myroledeets --role-session-name seshname)
export AWS_SECRET_ACCESS_KEY=$(echo $ASSUMED_ROLE | jq -r .Credentials.SecretAccessKey)
export AWS_ACCESS_KEY_ID=$(echo $ASSUMED_ROLE | jq -r .Credentials.AccessKeyId)
export AWS_SESSION_TOKEN=$(echo $ASSUMED_ROLE | jq -r .Credentials.SessionToken)
​
if [ ${#AWS_ACCESS_KEY_ID} -lt 16 ] || [ ${#AWS_ACCESS_KEY_ID} -gt 128 ]; then
echo "Access Key Failure" >&2
errorexit
fi
​
if [ ${#AWS_SECRET_ACCESS_KEY} -lt 16 ] || [ ${#AWS_SECRET_ACCESS_KEY} -gt 128 ]; then
echo "Secret Access Key Failure" >&2
errorexit
fi
​
if [ ${#AWS_SESSION_TOKEN} -lt 128 ]; then
echo "Session Token Failure" >&2
errorexit
fi
​
AWSPATH="s3://mypath/$FILEID"
aws s3 ls $AWSPATH | cut -c 32- | while IFS= read -r line; do
COPYPATH="s3://mypath/$FILEID/$line"
echo $COPYPATH
aws s3 cp $COPYPATH ./$FILEID/$line --recursive
done
if [ $? -ne 0 ]; then
errorexit
fi
​
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
​
exit 0
So what happens? Well, when I try to run this Expect automated version I get…
./awscopy.sh <password> <file>
spawn aws-adfs login --profile=default --adfs-host=myhost.com --role-arn=myroledeets --region=eu-west-1
2022-02-14 11:09:29,372 [authenticator authenticator.py:authenticate] [43954-MainProcess] [6536770074-MainThread] - ERROR: Cannot extract saml assertion. Re-authentication needed?
Password:
An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired
Access Key Failure
As you may have guessed, this is the first time I've tried to use Expect. What have I done wrong?