Cloud Init Fails to Install Packages on CentOS 7 in EC2 - amazon-ec2

Question

I have a cloud config file defined on my EC2 instance in the user data. Many of the parts/actions are run by the server properly, but package installation seems to fail invariably:
#cloud-config
package-update: true
package-upgrade: true
# ...
packages:
- puppet3
- lvm2
- btrfs-progs
# ...
I see the following in the logs:
May 20 20:23:39 cloud-init[1252]: util.py[DEBUG]: Package update failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 74, in handle
cloud.distro.update_package_sources()
File "/usr/lib/python2.6/site-packages/cloudinit/distros/rhel.py", line 278, in update_package_sources
["makecache"], freq=PER_INSTANCE)
File "/usr/lib/python2.6/site-packages/cloudinit/helpers.py", line 197, in run
results = functor(*args)
File "/usr/lib/python2.6/site-packages/cloudinit/distros/rhel.py", line 274, in package_command
util.subp(cmd, capture=False, pipe_cat=True, close_stdin=True)
File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1529, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['yum', '-t', '-y', 'makecache']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
May 20 20:23:39 cloud-init[1252]: amazon.py[DEBUG]: Upgrade level: security
May 20 20:23:39 cloud-init[1252]: util.py[DEBUG]: Running command ['yum', '-t', '-y', '--exclude=kernel', '--exclude=nvidia*', '--exclude=cudatoolkit', '--security', '--sec-severity=critical', '--sec-severity=important', 'upgrade'] with allowed return codes [0] (shell=False, capture=False)
May 20 20:23:52 cloud-init[1252]: util.py[WARNING]: Package upgrade failed
May 20 20:23:52 cloud-init[1252]: util.py[DEBUG]: Package upgrade failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 81, in handle
cloud.distro.upgrade_packages(upgrade_level, upgrade_exclude)
File "/usr/lib/python2.6/site-packages/cloudinit/distros/amazon.py", line 50, in upgrade_packages
return self.package_command('upgrade', args=args)
File "/usr/lib/python2.6/site-packages/cloudinit/distros/rhel.py", line 274, in package_command
util.subp(cmd, capture=False, pipe_cat=True, close_stdin=True)
File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1529, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['yum', '-t', '-y', '--exclude=kernel', '--exclude=nvidia*', '--exclude=cudatoolkit', '--security', '--sec-severity=critical', '--sec-severity=important', 'upgrade']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
May 20 20:23:52 cloud-init[1252]: util.py[DEBUG]: Running command ['yum', '-t', '-y', 'install', 'puppet3', 'lvm2', 'btrfs-progs'] with allowed return codes [0] (shell=False, capture=False)
May 20 20:24:03 cloud-init[1252]: util.py[WARNING]: Failed to install packages: ['puppet3', 'lvm2', 'btrfs-progs']
May 20 20:24:03 cloud-init[1252]: util.py[DEBUG]: Failed to install packages: ['puppet3', 'lvm2', 'btrfs-progs']
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 88, in handle
cloud.distro.install_packages(pkglist)
File "/usr/lib/python2.6/site-packages/cloudinit/distros/rhel.py", line 70, in install_packages
self.package_command('install', pkgs=pkglist)
File "/usr/lib/python2.6/site-packages/cloudinit/distros/rhel.py", line 274, in package_command
util.subp(cmd, capture=False, pipe_cat=True, close_stdin=True)
File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1529, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['yum', '-t', '-y', 'install', 'puppet3', 'lvm2', 'btrfs-progs']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
May 20 20:24:03 cloud-init[1252]: cc_package_update_upgrade_install.py[WARNING]: 3 failed with exceptions, re-raising the last one
May 20 20:24:03 cloud-init[1252]: util.py[WARNING]: Running package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.pyc'>) failed
May 20 20:24:03 cloud-init[1252]: util.py[DEBUG]: Running package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.pyc'>) failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cloudinit/stages.py", line 553, in _run_modules
cc.run(run_name, mod.handle, func_args, freq=freq)
File "/usr/lib/python2.6/site-packages/cloudinit/cloud.py", line 63, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python2.6/site-packages/cloudinit/helpers.py", line 197, in run
results = functor(*args)
File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 111, in handle
raise errors[-1]
ProcessExecutionError: Unexpected error while running command.
Command: ['yum', '-t', '-y', 'install', 'puppet3', 'lvm2', 'btrfs-progs']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
When I actually run the command yum -t -y install puppet3 lvm2 btrfs-progs as root, it just runs fine, but Cloud Init is failing to run it on its own.
Is there something I'm doing wrong here?

Answer 1

Evidently there was a bug fixed with later versions of the image. If this is happening to you, it may be a legitimate bug in Cloud-Init or your servers implementation of it. A package upgrade may fix the problem, also an update to the image will fix it globally.