Baking AMIs in Spinnaker Pipeline using Chef-Solo - amazon-ec2

I'm trying to build a test pipeline in Spinnaker to bake an AMI and then update a CloudFormation template to deploy the AMI to EC2 instances in an auto scaling group.
I have a small test Chef cookbook developed which works great when running packer locally. I am running berks local on my laptop to vendor my cookbooks and pull them from our internal Chef supermarket. Packer is configured with the chef-solo provisioner as shown in the sample packer template below, and will transfer the cookbooks to the packer builder EC2 instance and run Chef. Right now, we're testing with Linux but want to support both Linux and Windows AMIs.
Is it possible to use chef-solo with a custom packer template with Spinnaker? If so, when and where should berks run to vendor the cookbooks before packer executes?
{
"variables": {
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
"aws_region": "{{env `AWS_REGION`}}",
"ssh_private_key_file": "{{env `SSH_PRIVATE_KEY_FILE`}}",
"subnet_id": "{{env `AWS_SUBNET_ID`}}",
"vpc_id": "{{env `AWS_DEFAULT_VPC_ID`}}"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "amzn2-ami-hvm-*-x86_64-gp2",
"root-device-type": "ebs"
},
"most_recent": true,
"owners": [
"amazon"
]
},
"ami_name": "test-ami-{{timestamp}}",
"ami_description": "Test Linux AMI",
"communicator": "ssh",
"instance_type": "m4.large",
"subnet_id": "{{user `subnet_id`}}",
"tags": {
"Name": "Test Linux AMI"
},
"ssh_username": "ec2-user",
"ssh_keypair_name": "TestKeypair",
"ssh_private_key_file": "{{user `ssh_private_key_file`}}",
"vpc_id": "{{user `vpc_id`}}"
}
],
"provisioners": [
{
"type": "shell-local",
"command": "berks vendor --delete -b ./Berksfile ./cookbooks"
},
{
"type": "chef-solo",
"cookbook_paths": [
"./cookbooks"
],
"run_list": [
"recipe[test_cookbook]"
]
}
}

Related

Packer provisioners don't save installed packages

I have encountered an issue during provisioning with HashiCorp Packer for virtualbox-iso on Alpine Linux v3.16.
Provisioning script runs OK, and it logs that build has finished, however when I open the outputted ovf file in VirtualBox moved files and docker are not present.
I would be grateful for any advice.
I run packer build packer-virtualbox-alpine-governator.json
packer-virtualbox-alpine-governator.json file:
{
"variables": {
"password": "packer"
},
"builders": [
{
"type": "virtualbox-iso",
"memory": 8192,
"guest_os_type": "Other_64",
"iso_url": "https://dl-cdn.alpinelinux.org/alpine/v3.16/releases/x86_64/alpine-standard-3.16.0-x86_64.iso",
"iso_checksum": "file:https://dl-cdn.alpinelinux.org/alpine/v3.16/releases/x86_64/alpine-standard-3.16.0-x86_64.iso.sha256",
"ssh_username": "root",
"ssh_password": "{{user `password`}}",
"shutdown_command": "poweroff",
"hard_drive_interface": "sata",
"boot_command": [
"root<enter><wait>",
"setup-alpine<enter><wait>us<enter><wait>us<enter><wait><enter><wait><enter><wait><enter><wait><enter><wait5>{{user `password`}}<enter><wait>{{user `password`}}<enter><wait><enter><wait><enter><wait><enter><wait15><enter><wait>openssh<enter><wait>openssh-full<enter><wait5>test123<enter><wait5>test123<enter><wait><enter><wait><enter><wait>sda<enter><wait>sys<enter><wait>y<enter><wait30>",
"echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config<enter><wait>",
"/etc/init.d/sshd restart<enter><wait5>"
]
}
],
"provisioners": [
{
"type": "shell",
"inline": ["mkdir -p /opt/site/governator"]
},
{
"type": "file",
"source": "files/docker-compose.yaml",
"destination": "/opt/site/"
},
{
"type": "file",
"source": "files/governator.conf",
"destination": "/opt/site/governator/"
},
{
"type": "shell",
"scripts": [
"scripts/alpine/install-docker-on-alpine.sh"
]
}
]
}
./scritps/alpine/install-docker-on-alpine.sh
#! /bin/ash
cat > /etc/apk/repositories << EOF; $(echo)
https://dl-cdn.alpinelinux.org/alpine/v$(cut -d'.' -f1,2 /etc/alpine-release)/main/
https://dl-cdn.alpinelinux.org/alpine/v$(cut -d'.' -f1,2 /etc/alpine-release)/community/
https://dl-cdn.alpinelinux.org/alpine/edge/testing/
EOF
apk update
apk add docker
addgroup $USER docker
rc-update add docker boot
service docker start
apk add docker-compose
sync

AWS | Laravel | ECS(G/B) | CodeDeploy unfinish

I'm using Laravel.
I want to deploy it to ECS (B/G) to see how it works.
In the development environment, Laravel is running.
I was able to launch my Laravel project on EC2 using docker.
I want to use Fargate for the first time and deploy to ECS!
Also, CodeBuild has completed successfully.
appspec.yml
version: 0.0 Resources:
- TargetService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: "<TASK_DEFINITION>"
LoadBalancerInfo:
ContainerName: "nginx"
ContainerPort: "80"
taskdef.json
{
"taskRoleArn": "arn:aws:iam::**********:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::**********:role/ecsTaskExecutionRole",
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/****-system",
"awslogs-region": "******",
"awslogs-stream-prefix": "ecs"
}
},
"entryPoint": [
"sh",
"-c"
],
"command": [
"php artisan config:cache && php artisan migrate && chmod -R 777 storage/ && chmod -R 777 bootstrap/cache/"
],
"cpu": 0,
"environment": [
{
"name": "APP_ENV",
"value": "staging"
}
],
"workingDirectory": "/var/www/html",
"image": "<IMAGE1_NAME>",
"name": "php"
},
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/****-system",
"awslogs-region": "****",
"awslogs-stream-prefix": "ecs"
}
},
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 80
}
],
"environment": [
{
"name": "APP_ENV",
"value": "staging"
}
],
"workingDirectory": "/var/www/html",
"image": "**********.dkr.ecr.**********.amazonaws.com/**********-nginx:latest",
"name": "nginx"
}
],
"placementConstraints": [],
"memory": "2048",
"family": "*****-system",
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "1024",
"volumes": []
}
CodeDeploy stopped at INSTALL, and there are no errors.
As you can see in the capture, we can confirm that "<TASK_DEFINITION>" has been replaced.
I'd like to know if there's any information I'm missing.
I'm not sure how to set environment variables such as ".env", so I'm thinking this might be the cause.
CodeDeploy Failed
Revision
Task Definitions
ECR
ECR nginx
ECR src(laravel)
If you want to change .env file to set env variable, you may use ssh connection to your webserver and run nano .env command at root folder, to write the file.
You can also modify the file using ftp connection.

Why does my SystemD service not restart on reboots?

I am building an AMI using Packer, which has a custom SystemD unit configured. The AMI is then deployed to an EC2. The issue is that if the EC2 reboots, then the unit is not restarted.
Here is my unit:
[Unit]
Description=My service
After=network.target
StartLimitIntervalSec=0
StartLimitAction=reboot
[Service]
Type=simple
Restart=always
RestartSec=30
User=ubuntu
ExecStart=/opt/app/app
[Install]
WantedBy=multi-user.target
And here is my Packer configuration:
{
"variables": {
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
"region": "{{env `AWS_REGION`}}"
},
"builders": [
{
"access_key": "{{user `aws_access_key`}}",
"ami_name": "my-app-{{timestamp}}",
"instance_type": "t2.micro",
"region": "{{user `region`}}",
"secret_key": "{{user `aws_secret_key`}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*",
"root-device-type": "ebs"
},
"owners": [ "099720109477" ],
"most_recent": true
},
"ssh_username": "ubuntu",
"type": "amazon-ebs"
}
],
"provisioners": [
{
"type": "shell",
"script": "{{template_dir}}/provision.sh"
},
{
"type": "file",
"source": "{{template_dir}}/files/app.service",
"destination": "/tmp/upload/etc/systemd/system/app.service"
},
{
"type": "file",
"source": "{{template_dir}}/../bin/Release/netcoreapp3.1/",
"destination": "/tmp/upload/opt/app"
},
{
"type": "shell",
"inline": [
"sudo rsync -a /tmp/upload/ /",
"cd /opt/app",
"sudo systemctl daemon-reload",
"sudo systemctl enable app.service"
]
}
]
}
Strangely, if I SSH into the running EC2 and enable the service, then it does restart after a reboot.
sudo systemctl enable app.service
sudo reboot
This makes me think I am not creating the AMI correctly, but in my Packer configuration I do enable the service!
Why does my AMI not have my SystemD unit enabled?

File provisioner in Packer for Windows Server

Issue with File Provisioner for Windows Azure VM
We tried implementing the Packer template referring the Microsoft Docs-
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/build-image-with-packer#create-azure-resource-group
Also, we replaced the script with the provisioner block
Below is the JSON script used to deploy the File Provisioner
{
builders: [{
type: "azure-arm",
"client_id": "",
"client_secret": "",
"tenant_id": "",
"subscription_id": "",
"managed_image_resource_group_name": "Packer-RG",
"managed_image_name": "myPackerImage",
"os_type": "Windows",
"image_publisher": "MicrosoftWindowsServer",
"image_offer": "WindowsServer",
"image_sku": "2016-Datacenter",
"communicator": "winrm",
"winrm_use_ssl": true,
"winrm_insecure": true,
"winrm_timeout": "5m",
"winrm_username": "packer",
"azure_tags": {
dept: "Engineering",
"task": "Image deployment"
},
"location": "East US",
"vm_size": "Standard_DS2_v2"
}],
"provisioners": [{
"type": "file",
"source": "C:/Test-1",
"destination": "C:/Test-1"
}]
}
After implementing the steps, the builds were finished but no artifacts were created.

Vagrant Private Boxfile generated via Packer/Atlas is 404 when accessed as logged in user

I'm generating an AMI image, and passing that through to a vmware_fusion vagrant.box post-processor. This completes successfully, and the vagrant box page claims that the box is accessible and available. Using the instructions provided on the box file page to init a new project with the box result in...
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.
The requested URL returned error: 404 Not Found
When copy/pasting the 404'd URL into a browser, I also get the Atlas 404 page.
I have verified that I am logged in via vagrant login at the console and I am logged in to the Atlas site, so the 404 is not a result of the box being private and myself not being logged in.
I have run other box builds, and they did successfully download at this stage. It kind of seems like Packer/Atlas is bugged right now, but I have no way to verify that.
Here's what my Packer config looks like:
{
"variables": {
"aws_access_key": "{{env `AWS_ACCESS`}}",
"aws_secret_key": "{{env `AWS_SECRET`}}"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"ami_name": "ami_name_here {{timestamp}}",
"instance_type": "t2.medium",
"region": "us-east-1",
"source_ami": "ami-df38e6b4",
"user_data_file": "ec2-setup.sh"
}
],
"provisioners": [
{
"type": "shell",
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
"script": "packer_scripts/setup.sh"
},
{
"type": "shell",
"inline": [
"sleep 30",
"cd /tmp && sudo wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb",
"sudo dpkg -i /tmp/puppetlabs-release-pc1-trusty.deb",
"sudo apt-get update && sudo apt-get upgrade -y",
"sudo apt-get install puppet -y"
]
},
{
"type": "puppet-masterless",
"manifest_file": "manifests/default.pp",
"module_paths": [
"modules/"
]
}
],
"post-processors": [
[
{
"type": "atlas",
"artifact": "my/artifact",
"artifact_type": "amazon.ami",
"metadata": {
"created_at": "{{timestamp}}"
}
},
{
"type": "atlas",
"artifact": "my/artifact",
"artifact_type": "vagrant.box",
"metadata": {
"created_at": "{{timestamp}}",
"provider": "vmware_fusion"
}
}
]
],
"push": {
"name": "my/artifact",
"vcs": true
}
}
After more digging I found more complete documentation about how the Packer/Atlas process works. It would seem that Atlas cannot accept an AMI image built from a Packer Builder and convert that into a VM image for other platforms (VMware, Virtualbox). Which is unfortunate, since my builds complete much more quickly on my own EC2 instance.
If I'm incorrect here I'd love to know how it can be done. If I find a way, I'll be back to update.

Resources