AWS EC2 User Data Not Decoding Correctly - bash

I am experimenting with creating an EC2 instance to host a Perforce server. My instance is configured with the following user data:
#!/bin/bash
# Add a newline to the ec2-user prompt string
echo PS1=\"\\n\$PS1\" >> /home/ec2-user/.bashrc
# Update all packages
yum update –y
# Install Perforce packages
# The RHEL/7 part of the baseurl should be replaced with
# the latest RHEL version that both Amazon and Perforce support
rpm –import https://package.perforce.com/perforce.pubkey
cd /etc/yum.repos.d/
echo [perforce] > perforce.repo
echo name=Perforce >> perforce.repo
echo baseurl=http://package.perforce.com/yum/rhel/7/x86_64 >> perforce.repo
echo enabled=1 >> perforce.repo
echo gpgcheck=1 >> perforce.repo
yum install –y helix-p4d
# Make directories for the server, owned by new “perforce” user
cd /opt/perforce/servers/
mkdir danware
cd danware
mkdir danware-db danware-chkpts journal
chown –R perforce:perforce danware
I have tested each of the above commands, and know that they work when executed manually in this order. However, some aspect of Amazon's base64 encode/decode system seems to be getting in the way. When I go to "Actions > Instance Settings > View/Change User Data" from the EC2 Console after launching (and passing all system checks), I see the following user data. Note how almost every hyphen "-" has been replaced with some strange "a" character.
However, I'm not sure that this is the issue, because the log file at /var/log/cloud-init-output.log gives me the following output (I replaced some repetitive text with [...] to save space). Note the line that says Failed running /var/lib/cloud/instance/scripts/part-001 I have verified that this part-001 file actually does have the correctly displayed hyphen characters.
[...]
Cloud-init v. 0.7.6 running 'modules:final' at Fri, 09 Sep 2016 06:23:39 +0000. Up 86.66 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper
No Match for argument: –y
No packages marked for update
RPM version 4.11.2
Copyright (C) 1998-2002 - Red Hat, Inc.
This program may be freely redistributed under the terms of the GNU GPL
Usage: rpm [-aKfgpqVcdLilsiv?] [-a|--all] [-f|--file] [-g|--group] [...]
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> [...]
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
helix-p4d x86_64 2016.1-1429894 perforce 24 k
Installing for dependencies:
helix-cli x86_64 2016.1-1429894 perforce 8.8 k
helix-cli-base x86_64 2016.1-1429894 perforce 1.4 M
helix-p4d-base x86_64 2016.1-1429894 perforce 3.1 k
helix-p4d-base-16.1 x86_64 2016.1-1429894 perforce 2.4 M
helix-p4dctl x86_64 2016.1-1429894 perforce 1.2 M
Transaction Summary
================================================================================
Install 1 Package (+5 Dependent packages)
Total download size: 5.0 M
Installed size: 13 M
Is this ok [y/d/N]: Exiting on user command
Your transaction was saved, rerun it with:
yum load-transaction /tmp/yum_save_tx.2016-09-09.06-23.dRP_r2.yumtx
/var/lib/cloud/instance/scripts/part-001: line 22: cd: /opt/perforce/servers/: No such file or directory
chown: invalid user: ‘–R’
Sep 09 06:23:41 cloud-init[2517]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Sep 09 06:23:41 cloud-init[2517]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Sep 09 06:23:41 cloud-init[2517]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.6 finished at Fri, 09 Sep 2016 06:23:41 +0000. Datasource DataSourceEc2. Up 88.53 seconds
Even more annoying, I assumed that the early No Match for argument: –y line from the log file was referring to the yum update -y line from my user data. Sure enough, just running the example user data script from the EC2 documentation page, which also uses yum update -y, gives me this same error/warning! Amazon's own example script doesn't work!? So can anyone answer why A) AWS is not displaying the user data code correctly, and B) why my user data is yielding the errors shown above? The help is much appreciated!

For lines such as
yum update –y
The character you are using is a "EN DASH U+2013"
The usual character for a hyphen is "HYPHEN-MINUS U+002D"
Fix your user data source to use "hyphen minus" and have another go
I checked the character codes by cut n pasting into this online site http://www.fileformat.info/info/unicode/char/search.htm?q=-&preview=entity
Don't know if you can see the difference but this is your hyphen
yum update –y
and this is a "hyphen minus"
yum update -y

Related

yum and dnf install/update takes 5 minutes to start on RHEL8

If I run yum or dnf as root on most recent updated RHEL8 it works smoothly. As soon as I try running it with sudo users (added to wheel group) it takes up to 5 minutes.
Doing a yum clean did not help eiher.
I usually run updates using Ansible from a remote host but Ansible disconnects after several minutes of trying to run the yum module.
I set SELinux to disabled, just in case and checked proxy settings in dnf.conf - It all looks fine. Any input is very much appreciated.
This is the dnf.log
2020-02-20T15:33:13Z INFO Updating Subscription Management repositories.
2020-02-20T15:37:36Z DEBUG DNF version: 4.2.7
2020-02-20T15:37:36Z DDEBUG Command: yum update
2020-02-20T15:37:36Z DDEBUG Installroot: /
2020-02-20T15:37:36Z DDEBUG Releasever: 8
2020-02-20T15:37:36Z DEBUG cachedir: /var/cache/dnf
2020-02-20T15:37:36Z DDEBUG Base command: update
2020-02-20T15:37:36Z DDEBUG Extra commands: ['update']
2020-02-20T15:37:37Z DEBUG repo: using cache for: epel-modular
2020-02-20T15:37:37Z DEBUG epel-modular: using metadata from Sat Feb 15 03:19:39 2020.
2020-02-20T15:37:37Z DEBUG repo: using cache for: epel
2020-02-20T15:37:37Z DEBUG epel: using metadata from Thu Feb 20 06:38:22 2020.
2020-02-20T15:37:38Z DEBUG reviving: 'rhel-8-for-x86_64-baseos-rpms' can be revived - repomd matches.
2020-02-20T15:37:38Z DEBUG rhel-8-for-x86_64-baseos-rpms: using metadata from Thu Feb 13 10:30:57 2020.
2020-02-20T15:37:38Z DEBUG reviving: 'rhel-8-for-x86_64-appstream-rpms' can be revived - repomd matches.
2020-02-20T15:37:39Z DEBUG rhel-8-for-x86_64-appstream-rpms: using metadata from Thu Feb 20 11:06:31 2020.
2020-02-20T15:37:39Z DDEBUG timer: sack setup: 3550 ms
2020-02-20T15:37:39Z DEBUG Completion plugin: Generating completion cache...
2020-02-20T15:37:40Z DEBUG --> Starting dependency resolution
2020-02-20T15:37:40Z DEBUG --> Finished dependency resolution
2020-02-20T15:37:40Z DDEBUG timer: depsolve: 277 ms
2020-02-20T15:37:40Z INFO Dependencies resolved.
2020-02-20T15:37:40Z INFO Nothing to do.
2020-02-20T15:37:40Z INFO Complete!
2020-02-20T15:37:40Z DDEBUG Cleaning up.
I resolved the issue. The proxy was not correctly set in /etc/rhsm/rhsm.conf
I don't understand why running as root would not cause the same problem.

Git dependency errors when using Composer with private VCS repository on IBM i

More details below, but I'm running into the following error when I issue a composer install or composer update command on a project which uses a custom VCS repository for an included project. This is on an IBM i v7r3 system via SSH session. I've run out of ideas on how to troubleshoot further, so here I am. Has anyone ever seen this error or have any ideas on how to troubleshoot further?
Error:
bash-4.4$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 113 installs, 0 updates, 0 removals
- Installing sshUser/myPackage (v1.0.26): Cloning a2db0666b1
[RuntimeException]
Failed to clone gitUser:/opt/git/myPackage.git, git was not found, check that
it is installed and in your PATH env.
Could not load program git:
Dependent module /usr/local/zendphp7/lib/libz.so.1(shr_64.o) could not be loaded.
File /usr/local/zendphp7/lib/libz.so.1 is not an
archive or the file could not be read properly.
System error: Exec format error
I have a remote IBM i server which has two users. The first user "gitUser" is used for my private Git repositories (headless). I then have another user "sshUser" which is used to SSH in with and do my typical work. I have SSH keys configured to simplify connections for both users and the "gitUser" ssh key doesn't require a password, so I can use it for automated scripting.
I have a private VCS repository included in my composer.json file for a project. So long as I have the entry in the "repository" section I'm getting the above error. This occurs even if I'm not requireing a package from the repository.
Repository Entry:
"repositories": [
{
"type": "vcs",
"url": "gitUser:/opt/git/myPackage.git"
}
]
Composer on the IBM i is up to date at 1.7.2.
It's clear the issue lies with git and the inability to find a dependency in libz.so.1.
However, if I follow the chain of dependencies for the version of Git in my path, all the appropriate files exist with adequate permissions:
bash-4.4$ which git
/QOpenSys/pkgs/bin/git
bash-4.4$ dump -X64 -H /QOpenSys/pkgs/bin/git
/QOpenSys/pkgs/bin/git:
***Loader Section***
Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x000000df 0x0000433f 0x0000009d
#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000006 0x00044910 0x00000a18 0x000449ad
***Import File Strings***
INDEX PATH BASE MEMBER
0 /QOpenSys/pkgs/lib:/QOpenSys/usr/lib
1 libz.so.1 shr_64.o
2 libiconv.so.2 shr_64.o
3 libcrypto.so.1.0.0 shr_64.o
4 libpthread.a shr_xpg5_64.o
5 libc.a shr_64.o
bash-4.4$
I can confirm there are no permission failures through out all of this. So this doesn't seem to be the result of inadequate permissions. Also, I do not think the issue lies within the user profile gitUser as if I issue a composer install or composer update from another machine (with appropriate keys), everything works flawlessly. So it seems to be specific to the IBM i and the sshUser profile.
I'm starting to think Composer is using a different git executable and the dependencies of that executable do not exist. Though I assume Composer would just use my path.
Note*: I tried specifying my LIBPATH in my ./.profile with the following. But it doesn't seem to make any difference:
LIBPATH=/usr/local/zendphp7/lib
LIBPATH=$LIBPATH:/QOpenSys/pkgs/lib:/QOpenSys/usr/lib
export LIBPATH
When echoing:
bash-4.4$ echo $LIBPATH
/usr/local/zendphp7/lib:/QOpenSys/pkgs/lib:/QOpenSys/usr/lib
bash-4.4$
If I don't add the above to my ./.profile, then echo $LIBPATH prints a blank line to screen.
Solution:
I just changed the symlink in the ZENDPHP7 directory structure to the repo version. Alternatively you can manipulate your library path, but I couldn't figure out the proper way to do it on IBM i.
bash-4.4$ pwd
/usr/local/zendphp7/lib
bash-4.4$ ln -s /QOpenSys/pkgs/lib/libz.so.1 ./libz.so.1
bash-4.4$ ls -la /usr/local/zendphp7/lib/libz.so.1
lrwxrwxrwx 1 dl 0 56 Oct 11 14:21 /usr/local/zendphp7/lib/libz.so.1 -> /QOpenSys/pkgs/lib/libz.so.1
bash-4.4$
I'm guessing something is off with your path when you do the install. Is it maybe looking at the Zend PHP libraries rather than the expected ones in /usr/lib?
Notice that the errors shows a path of /usr/local/zendphp7/lib/ libz.so.1(shr_64.o) while your git dump instead shows /QOpenSys/pkgs/lib:/QOpenSys/usr/lib.

deluge: recurring corruption of torrent state forces me to re-check them every time

I'm using Deluge and get these errors for the same torrents all the time:
/usr/lib/python2.7/dist-packages/deluge/_libtorrent.py:59: RuntimeWarning: to-Python converter for boost::shared_ptr<libtorrent::alert> already registered; second conversion method ignored.
import libtorrent as lt
[ERROR ] 14:14:13 torrentmanager:1147 on_alert_fastresume_rejected: fast resume rejected. (*omitted*/torrent/Fargo Season 3 Complete 720p HDTV x264 [i_c]/Fargo S03E06 The Lord of No Mercy.mkv): mismatching file size
[ERROR ] 14:14:14 torrentmanager:1147 on_alert_fastresume_rejected: fast resume rejected. stat(*omitted*/torrent/Keith.Jarrett-Collection.1967-2010.Flac-Tracks.Lossless/1973 - Solo Concerts Bremen & Lausanne ( 2CD )/CD 1/Keith Jarrett - Solo Concerts Bremen Lausanne Disc 1.log): mismatching file timestamp
[ERROR ] 14:14:14 torrentmanager:1147 on_alert_fastresume_rejected: fast resume rejected. stat(*omitted*/torrent/B.B.King - Discography 1957-2008 Mp3 320 Kbps/2 Live albums/1965 Live At The Regal (1995) #320/09 You Done Lost Your Good Thing Now.mp3): mismatching file timestamp
[ERROR ] 14:14:27 torrent:468 Restart deluge to clear this torrent error
[ERROR ] 14:14:27 torrent:468 Restart deluge to clear this torrent error
Unhandled error in Deferred:
Then I have to right click those torrents and force a re-check. Next time I launch Deluge I get the same errors...
Additional info:
$ deluge --version
deluge: 1.3.15
/usr/lib/python2.7/dist-packages/deluge/_libtorrent.py:59: RuntimeWarning: to-Python converter for boost::shared_ptr<libtorrent::alert> already registered; second conversion method ignored.
import libtorrent as lt
libtorrent: 1.1.1.0
$ uname -a
Linux lubuntu 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
No LSB modules are available.
Are there any workarounds? I've googled it but found no solution.

CentOS automatic installation fails with "Installation Destination" error

I'm trying to install the centos netboot install on ppc64le platform but automation installation(kickstart) fails with the following error:
Starting installer, one moment...
find_file: stat /proc/device-tree/chosen/bootpath, No such file or directory
anaconda 21.48.22.56-1 for CentOS AltArch 7 started.
* installation log files are stored in /tmp during the installation
* shell is available on TTY2
* when reporting a bug add logs from /tmp as separate text/plain attachments
Starting automated install..........
Generating updated storage configuration
storage configuration failed: failed to find a suitable stage1 device
================================================================================
================================================================================
Installation
1) [x] Language settings 2) [x] Timezone settings
(English (United States)) (UTC timezone)
3) [x] Installation source 4) [x] Software selection
(http://mirror.centos.org/altar (Custom software selected)
ch/7/os/ppc64le/) 6) [x] Kdump
5) [!] Installation Destination (Kdump is enabled)b | Help: F1
(Error checking storage configu
ration)
7) [x] Network configuration
(Wired (enp0s1) connected)
** (anaconda:1253): WARNING **: Could not open X display
The installation was stopped due to incomplete spokes detected while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
The exact error message is:
The following mandatory spokes are not completed:
Installation Destination.
The installer will now terminate.
[root#llmtul01b qemu]#
And kickstart file looks like:
url --url="http://mirror.centos.org/altarch/7/os/ppc64le/"
install
keyboard us
rootpw --lock --iscrypted locked
timezone --isUtc --nontp UTC
selinux --enforcing
firewall --disabled
network --bootproto=dhcp --device=link --activate --onboot=on
reboot
bootloader --disable
lang en_US
# Repositories to use
repo --name="CentOS" --baseurl=http://mirror.centos.org/altarch/7/os/ppc64le/ --cost=100
## Uncomment for rolling builds
repo --name="Updates" --baseurl=http://mirror.centos.org/altarch/7/updates/ppc64le/ --cost=100
# Disk setup
zerombr
clearpart --all --initlabel
part / --fstype ext4 --size=3000
Looking for the option to select the installation destination through kickstart file.
This problem solved by adding one more partition which is specific for ppc64le pltform.
Update and working kickstart file looks like :
url --url="http://mirror.centos.org/altarch/7/os/ppc64le/"
install
keyboard us
rootpw --lock --iscrypted locked
timezone --isUtc --nontp UTC
selinux --enforcing
firewall --disabled
network --bootproto=dhcp --device=link --activate --onboot=on
reboot
bootloader --disable
lang en_US
# Repositories to use
repo --name="CentOS" --baseurl=http://mirror.centos.org/altarch/7/os/ppc64le/ --cost=100
## Uncomment for rolling builds
repo --name="Updates" --baseurl=http://mirror.centos.org/altarch/7/updates/ppc64le/ --cost=100
# Disk setup
zerombr
clearpart --all --initlabel
part / --fstype ext4 --size=3000
part prepboot --fstype "PPC PReP Boot" --size=10

How do I use the sed command to remove all lines between 2 phrases (including the phrases themselves)

I am generating a log from which I want to remove X startup output which looks like this:
X.Org X Server 1.7.6
Release Date: 2010-03-17
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.31-607-imx51 armv7l Ubuntu
Current Operating System: Linux nvidia 2.6.33.2 #1 SMP PREEMPT Mon May
31 21:38:29 PDT 2010 armv7l
Kernel command line: mem=448M#0M nvmem=64M#448M mem=512M#512M
chipuid=097c81c6425f70d7 vmalloc=320M video=tegrafb
console=ttyS0,57600n8 usbcore.old_scheme_first=1 tegraboot=nand
root=/dev/nfs ip=:::::usb0:on rw tegra_ehci_probe_delay=5000 smp dvfs
tegrapart=recovery:1b80:a00:800,boot:2680:1000:800,environment:3780:40:800,system:38c0:2bc00:800,cache:2f5c0:4000:800,userdata:336c0:c840:800
envsector=3080
Build Date: 23 April 2010 05:19:26PM
xorg-server 2:1.7.6-2ubuntu7 (Bryce Harrington <bryce#ubuntu.com>)
Current version of pixman: 0.16.4
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Wed Jun 16 19:52:00 2010
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using config directory: "/usr/lib/X11/xorg.conf.d"
Is there any way to do this without manually checking pattern for each line?
Possibly:
sed '1,/^(==) Using config directory: "/d'
Or, possibly:
sed '/^X.Org X Server /,/^(==) Using config directory: "/d'

Resources