EC2 setup - execute a UserData script or Clone a Repository? - amazon-ec2

Question

We recently built and deployed an ingestion pipeline for iot data and centered our architecture around aws lambda. Now that we are reaching scale some of these lambdas start timing out, and I want to use temporary EC2 instances to process these longer running tasks.
I have a lambda setup that invokes spins up an ec2 instance and runs a UserData script. This is the relevant code involved:
import { EC2 } from 'aws-sdk'
const region = 'eu-central-1'
const ImageId = 'ami-0a02ee601d742e89f'
const InstanceType = 't2.micro'
const ec2 = new EC2({ apiVersion: '2016-11-15', region })
const ec2Scheduler = async () => {
const initScript = `#!/bin/bash
shutdown -h +5`
const UserData = new Buffer(initScript).toString('base64')
console.log(`Running EC2 instance with script: ${UserData}`)
const instance = await ec2
.runInstances({
ImageId,
InstanceType,
UserData,
MinCount: 1,
MaxCount: 1,
InstanceInitiatedShutdownBehavior: 'terminate',
})
.promise()
const instanceId = instance?.Instances?.[0]?.InstanceId
console.log(
`Ec2 instance with id ${instanceId} created. will auto shutdown in 5 minutes`
)
}
All fine and dandy. Except I'm stuck on how to get my javascript executable transferred over to my temporary EC2 instance.
What is the way to go here? I'm currently considering either:
a. Storing bitbucket credentials in secretsmanager. Then using my userdata script to install node/git and clone the repository from there.
b. Update my deployment pipelines to store the javascript executable in s3, then using the aws cli from my userdata script to fetch the executable and run it.
Both options seem a bit unwieldy. Is there a more direct/straightforward/lazy approach?
EDIT=====================
I think I need to adjust my mental model in a way that is not quite clear to me yet. My problem is not how do i get the code out of s3 from whitin my ec2 instance, but how do i use cloudformation to specify an executable to be made available in s3. From working with cloudformation/lambdas i am used to writing things like:
Scheduler:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs12.x
CodeUri: ../dist/task-scheduler
As a result of the package command the code referenced in CodeUri is then bundled and uploaded to an s3 deployment bucket and made available to the lambdas. I imagine a similar solution here. A resource type that i can provide with a codeUri, which my ec2 instance can then fetch from s3, and execute.
Thanks for the insightful comments so far!

Answer 1

A common practice is to download scripts from an S3 bucket. The UserData is limited to 16KB so any large scripts you need to download and execute anyway (unless you bake the AMI with the files on there).
It will be easier to keep it on all in AWS for PROD and you want PROD to be easy!
For dev machines I clone repo's (after copying ssh keys) and set them up that way. For production machines its all Cloud Formation with UserData and S3 (or Artifactory), any credentials stored in SSM Parameter store and all permissions locked down with an IAM role assigned to the EC2 with specific access to the S3 Bucket.
The key point is there's some control before it goes to PROD, we don't clone off repo's directly to the PROD machine, there's a build, test and deployment phase. Where as Dev go for it - clone from a Branch if you want!
Sorry I don't have a version in Javascript/NodeJS, here's a x-platform PowerShell example you can use or at least to follow the steps I use to configure either:
DEV:
$region = 'us-west-1'
Write-Host 'Install NuGet, Git, SSH Keys directory and set region'
choco install git -y
$newPath = "$($env:PATH)C:\Program Files\Git\cmd;"
[Environment]::SetEnvironmentVariable( 'PATH', $newPath, "Machine")
$env:PATH = $newPath
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
install-module posh-git -force
import-module awspowershell
New-Item -ItemType Directory -Force -Path 'C:\Users\Administrator\.ssh' | out-null
$destinDir = "C:\Users\$($env:username)\.ssh"
$prefix = "tempcreds/$user/"
set-defaultawsregion -region $region
if ($user) {
Write-Host 'Download GitHub Keys: Copy-S3Object -bucketname ' + $bucket + ' -key ' + $($prefix) + 'YOURKEYNAME -localfile ' + $destinDir + '\YOURKEYNAME'
Copy-S3Object -bucketname $bucket -key "$($prefix)YOURKEYNAME" -localfile $destinDir\YOURKEYNAME -region $region
Copy-S3Object -bucketname $bucket -key "$($prefix)YOURKEYNAME.pub" -localfile $destinDir\YOURKEYNAME.pub -region $region
Write-Host 'Remove GitHub Keys: -key ' + $($prefix) + 'YOURKEYNAME'
Remove-S3Object -bucketname $bucket -key "$($prefix)YOURKEYNAME" -force -region $region
Remove-S3Object -bucketname $bucket -key "$($prefix)YOURKEYNAME.pub" -force -region $region
Write-Host 'Save the GitHub Known_Hosts file'
add-content -path "$destinDir\known_hosts" `
-value $githubKnownHosts
git config --global user.email $email
git config --global user.name $user
}
Write-Host 'cd to dev directory'
$devDir = 'C:\DEV'
new-item -itemtype directory -force -path $devDir | out-null
cd $devDir
Write-Host 'Execute git clone <git.com/YOUREPO>'
git clone <git.com/YOUREPO>.git
PROD:
#using the AWS API with S3 fetch the powershell install script and execute it
$S3BucketName = "unique-bootstrap-bucketname"
$bootstrap = "install-YOURREPO.ps1"
$script = ($path + $bootstrap)
Set-DefaultAWSRegion -Region $region
Copy-S3Object -BucketName $S3BucketName -key $bootstrap -LocalFile ($path + $bootstrap)
& $script -S3Name $S3BucketName