bad character showing up in bash script execution - bash

I have a bash script that is getting an accented character appended to some strings that is causing it to fail, and I can't find where or how these characters are getting in there.
Here is some example output:
mv: cannot move â/tmp/myapp.zipâ to â/opt/myserver/myapp/deploys/myapp.1.2.21.zipâ: No such file or directory
ln: failed to create symbolic link â/opt/myserver/myapp/deploys/myapp_beta.zipâ: No such file or directory
cp: cannot stat â/opt/myserver/myapp/deploys/myapp_beta.zipâ: No such file or directory
the invalid character is the â.
The script is below:
#!/bin/bash
BRANCH=$1
SVN_LOC="https://svn/svn/myserver/"
MYAPP_REPO="myapp.git"
COREJS_REPO="core-js.git"
SPARTAN_REPO="core-spartan.git"
MYAPP_LOCATION="myapp/"
COREJS_LOCATION="corejs/"
SPARTAN_LOCATION="spartan/"
DEPLOY_LOCATION="/tmp/deploy/"
CLEANUP="${DEPLOY_LOCATION}*"
DEPLOY_STORE="/opt/myserver/myapp/deploys/"
DEPLOY_TIME=$(date +%s)
failed ()
{
rm -rf $CLEANUP
exit 1
}
mkdir -p $DEPLOY_LOCATION
echo "Retrieving Code from Git Branch ${BRANCH}"
echo "Retrieving myapp code"
mkdir -p "${DEPLOY_LOCATION}${MYAPP_LOCATION}"
pushd /opt/myserver/myapp/myapp
git archive $BRANCH | tar -x -C "${DEPLOY_LOCATION}${MYAPP_LOCATION}"
if [ $? -ne 0 ]
then
echo "Failed retrieving code from git ${MYAPP_REPO} repo";
failed
fi
popd
echo "Checking version numbers"
VERSION=$(php "${DEPLOY_LOCATION}${MYAPP_LOCATION}version.php" output)
DEPLOY_PACKAGE="${DEPLOY_STORE}myapp.${VERSION}.zip"
if [ -f $DEPLOY_PACKAGE ]
then
echo "A deploy with the same version number (${VERSION}) already exists! Please increment version number or manually deal with existing ${DEPLOY_PACKAGE}";
failed
fi
echo "Retrieving corejs code"
mkdir -p "${DEPLOY_LOCATION}${COREJS_LOCATION}"
pushd /opt/myserver/myapp/core-js
git archive $BRANCH | tar -x -C "${DEPLOY_LOCATION}${COREJS_LOCATION}"
if [ $? -ne 0 ]
then
echo "Failed retrieving code from git ${COREJS_REPO} repo";
failed
fi
popd
echo "Retrieving spartan code"
mkdir -p "${DEPLOY_LOCATION}${SPARTAN_LOCATION}"
pushd /opt/myserver/myapp/spartan
git archive $BRANCH | tar -x -C "${DEPLOY_LOCATION}${SPARTAN_LOCATION}"
if [ $? -ne 0 ]
then
echo "Failed retrieving code from git ${SPARTAN_REPO} repo";
failed
fi
popd
echo "Minifying js and css"
pushd "${DEPLOY_LOCATION}${MYAPP_LOCATION}Server/Deploy/"
php MinifyLyroke.php --deploytime $DEPLOY_TIME
popd
ASSETS_DEPLOY_PACKAGE="${DEPLOY_STORE}myappassets.${VERSION}.zip"
TEMP_ASSETS_ZIP_LOC="/tmp/myappassets.zip"
DEPLOY_ASSETS="${DEPLOY_LOCATION}myapp/Assets/"
ASSETS_DEPLOY_LOCATION="/tmp/assetsdeploy/"
DEPLOYED_ASSETS="${ASSETS_DEPLOY_LOCATION}myappassets_${DEPLOY_TIME}"
mkdir -p $ASSETS_DEPLOY_LOCATION
echo "Packaging assets deploy to ${ASSETS_DEPLOY_PACKAGE}"
mv $DEPLOY_ASSETS $DEPLOYED_ASSETS
pushd $ASSETS_DEPLOY_LOCATION
zip -r ${TEMP_ASSETS_ZIP_LOC} *
popd
mv ${TEMP_ASSETS_ZIP_LOC} ${ASSETS_DEPLOY_PACKAGE}
ln -sfn ${ASSETS_DEPLOY_PACKAGE} "${DEPLOY_STORE}myappassets_beta.zip"
cp "${DEPLOY_STORE}myappassets_beta.zip" "/opt/myserver/myapp/myapp/Server/Deploy/"
rm -rf $DEPLOYED_ASSETS
rm -rf $ASSETS_DEPLOY_LOCATION
echo "Packaging deploy to ${DEPLOY_PACKAGE}"
TEMP_ZIP_LOC="/tmp/myapp.zip"
pushd ${DEPLOY_LOCATION}
zip -r ${TEMP_ZIP_LOC} *
popd
mv "${TEMP_ZIP_LOC}" "${DEPLOY_PACKAGE}"
ln -sfn "${DEPLOY_PACKAGE}" "${DEPLOY_STORE}myapp_beta.zip"
cp "${DEPLOY_STORE}myapp_beta.zip" "/opt/myserver/myapp/myapp/Server/Deploy"
echo "Cleaning up"
rm -rf $CLEANUP
can anyone possibly see the issue or suggest a way I can go about finding where the issue is?

Those â characters are just mangled smart quotes printed from your shell. Your shell is probably outputting UTF-8, but your terminal is reading ISO-8859-1. Note that â is the rendering of a UTF-8 encoded smart quote ‘ in ISO-8859-1, with two nonprintable characters following the â. Most modern terminal emulators come with an option to enable UTF-8; see if you can enable that (it will make your life easier).
The problem is in your script, not the funny characters.

Try opening the script in another text editor like Notepad++ and see if there are any special characters present.

From the command line, type both of these commands. One or more of the files/directories you are expecting to exist, does not exist.
ls /tmp/myapp.zip
ls /opt/myserver/myapp/deploys

The accepted answer explains the problem, thanks #nneonneo. This is what you can do for a quick fix:
A) check your locale settings with:
locale
B) before calling your script or in the top of your bash-script try:
export LANG=en_US.UTF-8
export LC_ALL=C

Related

How to "cd" to a directory which is created using "mkdir $(date '+%d-%b-%Y')"

mkdir $(date '+%d-%b-%Y')
then cd to the dynamically created directory
How to "cd" to a directory which is created using "mkdir $(date '+%d-%b-%Y')" and do the operations by moving into the created directory in bash script
Simple way would be, you store the directory name in a variable
dirname=$(date '+%d-%b-%Y')
if [ -n "$dirname" ]; then
mkdir "$dirname"
if [ -d "$dirname" ]; then
cd "$dirname"
fi
fi
Added some error handling and also if your file is written in Windows and being run in an unix environment or vice-versa, I would recommend using dos2unix which will handle the new line character conversions (this is for the ? characters OP is seeing in ls).
Can you show me your case?
In most cases, you should not cd in to the directory. Use absolute path instead:
Good practice:
mkdir /tmp/mydir/
cp -R /usr/local/example/ /tmp/mydir/
sed 's/foo/bar/g' /tmp/mydir/afile
Bad practice:
mkdir /tmp/mydir/
cd /tmp/mydir/
cp -R /usr/local/example/ .
sed 's/foor/bar/g' afile
P.S.
Subj:
$ mkdir $(date '+%d-%b-%Y')
$ cd $(date '+%d-%b-%Y')
$ pwd
/Users/user/18-Feb-2019
In Bash, $_ expands to the last argument to the previous command. So you could do:
mkdir $(date '+%d-%b-%Y')
cd $_
In a real Bash program you would want to quote the expansions (use Shellcheck on your code to check for missing quotes), and check for errors on both mkdir and cd.

Symlink dotfiles with a script

As many others have done, I want to create a repo to store my dotfile customizations. Instead of doing ln -s manually, I am using the following script to set things up.
#!/bin/bash
set -e
DIR="$HOME/Documents/Dotfiles"
OLDDIR="$HOME/Documents/Other\ Files/Dotfiles_old"
FILES=($HOME/.bash_profile)
echo "Creating $OLDDIR for backup of any existing dotfiles in ~"
mkdir -p "$OLDDIR"
echo "…done"
echo "Changing to the $DIR directory"
cd "$DIR"
echo "…done"
for FILE in "${FILES[#]}"; do
echo "Backup dotfile $FILE from ~/ to $OLDDIR"
cp -L "$HOME/$FILE" "$OLDDIR"
done
for FILE in "${FILES[#]}"; do
echo "copy $FILE from ~ to $DIR."
cp -L "$HOME/$FILE $DIR/"
echo "Creating symlink to $FILE from ~ to $DIR."
ln -sfn "$DIR/$FILE" "$HOME/$FILE";
done
shellcheck source "$HOME/.bash_profile"
When I run this, cp fails because it thinks that .bash_profile isn't there, which obviously isn't the case:
I think my path to the files may be incorrect, although shellcheck reports nothing. What am I forgetting here?
UPDATE: Made another run at this - minus the cp. The one thing I am still unsure of is the use of exit, in particular since I'm already using -e to check for errors.
Shellcheck and bash -n return 0.
#!/bin/bash
set -e
function makeFiles() {
touch .bash_profile \
touch .gitconfig \
touch .gitignore_global
}
function makeLinks() {
ln -sfn ~/Documents/Dotfiles/.bash_profile ~/.bash_profile \
ln -sfn ~/Documents/Dotfiles/.gitconfig ~/.gitconfig \
ln -sfn ~/Documents/Dotfiles/.gitignore_global ~/.gitignore_global \
source ~/.bash_profile
}
read -rp "This may overwrite existing files. Are you sure? (y/n) " -n 1;
echo "";
if [[ $REPLY =~ ^[Yy]$ ]]; then
makeFiles && makeLinks
fi;
Sigh, ln decides that .bash_profile needs to be a directory for some crazy reason.
You're building the path of the dotfile incorrectly - $FILE already contains the full path of the dotfile, and there's no need to prepend $HOME again. Try with this cp command:
cp -L "$FILE $DIR/"

Getting test: too many arguments error while checking existence of directory in shell script

I am trying to write a shell script for extraction of .tar.gz file. I need to check first that if same directory is not present then extract the zip file. Otherwise do some thing else. Below is my shell script.
#!/bin/bash
INSTALL_DIR=$HOME/Test/LogShipper
LOGSTASH_PATH=logstash-2.3.2
LOGSTASH_FOLDER=$HOME/Test/LogShipper/logstash
LOGSTASH_BINARY=$LOGSTASH_PATH.tar.gz
ES_PATH=elasticsearch-2.3.2
ES_BINARY=$ES_PATH.tar.gz
KIBANA_VERSION=kibana-4.5.0
KIBANA_OS=darwin-x64
KIBANA_BINARY=$KIBANA_VERSION-$KIBANA_OS
echo Installing ELK stack into $INSTALL_DIR
mkdir -p $INSTALL_DIR
cd $INSTALL_DIR
if test [! -d "$LOGSTASH_FOLDER" ];
then
if test -s $LOGSTASH_BINARY
then
echo Logstash Zip Exists
echo Now installing...
echo Unpacking logstash...
tar zxf $LOGSTASH_BINARY $LOGSTASH_FOLDER
echo Unpacking Completed.
else
echo Downloading logstash 2.3.2
curl -O https://download.elasticsearch.org/logstash/logstash/$LOGSTASH_BINARY
fi
else
echo Logstash already installed.
fi
I am getting error test: too many arguments at line if test [! -d "$LOGSTASH_FOLDER" ];
You probably don't need to use test when using [] as it implicitly invokes the same during evaluation. Also include a space after open brace [.
if [ ! -d "$LOGSTASH_FOLDER" ];
You can explicitly use test command the following way, try
if ! test -d "$LOGSTASH_FOLDER"
If "$HOME" has any spaces in it, and were unquoted (as it is in the given code), that would cause the error in question. Suggest changing:
INSTALL_DIR=$HOME/Test/LogShipper
...
LOGSTASH_FOLDER=$HOME/Test/LogShipper/logstash
...
mkdir -p $INSTALL_DIR
...
cd $INSTALL_DIR
To:
INSTALL_DIR="$HOME"/Test/LogShipper
...
LOGSTASH_FOLDER="$HOME"/Test/LogShipper/logstash
...
mkdir -p "$INSTALL_DIR"
...
cd "$INSTALL_DIR"

Run wget and other commands in shell script

I'm trying to create a shell script that I will download the latest Atomic gotroot rules to my server, unpack them, copy them to the correct folder, etc.,
I've been reading shell tutorials and forum posts for most of the day and the syntax escapes me for some of these. I have run all these commands and I know they work if I manually run them.
I know I need to develop some error checking, but I'm just trying to get the commands to run correctly. The main problem at the moment is the syntax of the wget commands, i've got errors about missing semi-colons, divide by zero, unsupported schemes - I've tried various quoting (single and double) and escaping - / " characters in various combinations.
Thanks for any help.
The raw wget command is
wget --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/VERSION"
#!/bin/sh
update_modsec_rules(){
wget=/usr/bin/wget
tar=/bin/tar
apachectl=/usr/bin/apache2ctl
TXT="Script Run Finished"
WORKING_DIR="/var/asl/updates"
TARGET_DIR="/usr/local/apache/conf/modsec_rules/"
EXISTING_FILES="/var/asl/updates/modsec/*"
EXISTING_ARCH="/var/asl/updates/modsec-*"
WGET_OPTS='--user=jim --password=xxx-yyy-zzz'
URL_BASE="http://updates.atomicorp.com/channels/rules/subscription"
# change to working directory and cleanup any downloaded files and extracted rules in modsec/ directory
cd $WORKING_DIR
rm -f $EXISTING_ARCH
rm -f $EXISTING_FILES
rm -f VERSION*
# wget to download VERSION file
$wget ${WGET_OPTS} "${URL_BASE}/VERSION"
# get current MODSEC_VERSION from VERSION file and save as variable
source VERSION
TARGET_DATE=$MODSEC_VERSION
echo $TARGET_DATE
# wget to download current archive
$wget ${WGET_OPTS} "${URL_BASE}/modsec-${TARGET_DATE}.tar.gz"
# extract archive
echo "extracting files . . . "
tar zxvf $WORKING_DIR/modsec-${TARGET_DATE}.tar.gz
echo "copying files . . . "
cp -uv $EXISTING_FILES $TARGET_DIR
echo $TXT
}
update_modsec_rules $# 2>&1 | tee -a /var/asl/modsec_update.log
RESTART_APACHE="/usr/local/cpanel/scripts/restartsrv httpd"
$RESTART_APACHE
Here are some guidelines to use when writing shell scripts.
Always quote variables when you use them. This helps avoid the possibility of misinterpretation. (What if a filename contains a space?)
Don't trust fileglobbing on commands like rm. Use for loops instead. (What if a filename starts with a hyphen?)
Avoid subshells when possible. Your lines with backquotes make me itchy.
Don't exec if you can help it. And especially don't expect any parts of your script after your exec to actually get run.
I should point out that while your shell may be bash, you've specified /bin/sh for execution of this script, so it is NOT a bash script.
Here's a rewrite with some error checking. Add salt to taste.
#!/bin/sh
# Linux
wget=/usr/bin/wget
tar=/bin/tar
apachectl=/usr/sbin/apache2ctl
# FreeBSD
#wget=/usr/local/bin/wget
#tar=/usr/bin/tar
#apachectl=/usr/local/sbin/apachectl
TXT="GOT TO THE END, YEAH"
WORKING_DIR="/var/asl/updates"
TARGET_DIR="/usr/local/apache/conf/modsec_rules/"
EXISTING_FILES_DIR="/var/asl/updates/modsec/"
EXISTING_ARCH="/var/asl/updates/"
URL_BASE="http://updates.atomicorp.com/channels/rules/subscription"
WGET_OPTS='--user="jim" --password="xxx-yyy-zzz"'
if [ ! -x "$wget" ]; then
echo "ERROR: No wget." >&2
exit 1
elif [ ! -x "$apachectl" ]; then
echo "ERROR: No apachectl." >&2
exit 1
elif [ ! -x "$tar" ]; then
echo "ERROR: Not in Kansas anymore, Toto." >&2
exit 1
fi
# change to working directory and cleanup any downloaded files
# and extracted rules in modsec/ directory
if ! cd "$WORKING_DIR"; then
echo "ERROR: can't access working directory ($WORKING_DIR)" >&2
exit 1
fi
# Delete each file in a loop.
for file in "$EXISTING_FILES_DIR"/* "$EXISTING_ARCH_DIR"/modsec-*; do
rm -f "$file"
done
# Move old VERSION out of the way.
mv VERSION VERSION-$$
# wget1 to download VERSION file (replaces WGET1)
if ! $wget $WGET_OPTS $URL_BASE}/VERSION; then
echo "ERROR: can't get VERSION" >&2
mv VERSION-$$ VERSION
exit 1
fi
# get current MODSEC_VERSION from VERSION file and save as variable,
# but DON'T blindly trust and run scripts from an external source.
if grep -q '^MODSEC_VERSION=' VERSION; then
TARGET_DATE="`sed -ne '/^MODSEC_VERSION=/{s/^[^=]*=//p;q;}' VERSION`"
echo "Target date: $TARGET_DATE"
fi
# Download current archive (replaces WGET2)
if ! $wget ${WGET_OPTS} "${URL_BASE}/modsec-$TARGET_DATE.tar.gz"; then
echo "ERROR: can't get archive" >&2
mv VERSION-$$ VERSION # Do this, don't do this, I don't know your needs.
exit 1
fi
# extract archive
if [ ! -f "$WORKING_DIR/modsec-${TARGET_DATE}.tar.gz" ]; then
echo "ERROR: I'm confused, where's my archive?" >&2
mv VERSION-$$ VERSION # Do this, don't do this, I don't know your needs.
exit 1
fi
tar zxvf "$WORKING_DIR/modsec-${TARGET_DATE}.tar.gz"
for file in "$EXISTING_FILES_DIR"/*; do
cp "$file" "$TARGET_DIR/"
done
# So far so good, so let's restart apache.
if $apachectl configtest; then
if $apachectl restart; then
# Success!
rm -f VERSION-$$
echo "$TXT"
else
echo "ERROR: PANIC! Apache didn't restart. Notify the authorities!" >&2
exit 3
fi
else
echo "ERROR: Apache configs are broken. We're still running, but you'd better fix this ASAP." >&2
exit 2
fi
Note that while I've rewritten this to be more sensible, there is certainly still a lot of room for improvement.
You have two options:
1- changing this to
WGET1=' --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/VERSION"'
then run
wget $WGET1 same to WGET2
Or
2- encapsulating $WGET1 with backquotes ``.
e.g.:
`$WGET`
This applies to any command your executing out of a variable.
Suggested changes:
#!/bin/sh
TXT="GOT TO THE END, YEAH"
WORKING_DIR="/var/asl/updates"
TARGET_DIR="/usr/local/apache/conf/modsec_rules/"
EXISTING_FILES="/var/asl/updates/modsec/*"
EXISTING_ARCH="/var/asl/updates/modsec-*"
WGET1='wget --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/VERSION"'
WGET2='wget --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/modsec-$TARGET_DATE.tar.gz"'
## change to working directory and cleanup any downloaded files and extracted rules in modsec/ directory
cd $WORKING_DIR
rm -f $EXISTING_ARCH
rm -f $EXISTING_FILES
## wget1 to download VERSION file
`$WGET1`
## get current MODSEC_VERSION from VERSION file and save as variable
source VERSION
TARGET_DATE=`echo $MODSEC_VERSION`
## WGET2 command to download current archive
`$WGET2`
## extract archive
tar zxvf $WORKING_DIR/modsec-$TARGET_DATE.tar.gz
cp $EXISTING_FILES $TARGET_DIR
## restart server
exec '/usr/local/cpanel/scripts/restartsrv_httpd' $*;
Pro Tip: If you need string substitution, using ${VAR} is much better to eliminate ambiguity, e.g.:
tar zxvf $WORKING_DIR/modsec-${TARGET_DATE}.tar.gz

LOCAL_DIR variable prepends the scripts current directory (totally not what I expect)

Consider the following simple rsync script I am tryint to slap up:
#!/bin/bash
PROJECT="$1"
USER=stef
LOCAL_DIR="~/drupal-files/"
REMOTE_HOST="hostname.com"
REMOTE_PROJECTS_PATH=""
# Should not have anything to change below
PROJECT_LIST="proj1 proj2 proj3 quit"
echo "/nSelect project you wish to rsync\n\n"
select PROJECT in $PROJECT_LIST
do
if [ "$PROJECT" = "quit" ]; then
echo
echo "Quitting $0"
echo
exit
fi
echo "Rsynching $PROJECT from $REMOTE_HOST into" $LOCAL_DIR$PROJECT
rsync -avzrvP $USER#$REMOTE_HOST:/var/projects/$PROJECT/ $LOCAL_DIR$PROJECT
done
echo "Rsync complete."
exit;
The variable $LOCALDIR$PROJECT set in the rsync command always includes the scripts path, :
OUTPUT:
Rsynching casa from hostname.com.com into ~/drupal-files/casa
opening connection using: ssh -l stef hostname.com rsync --server --sender -vvlogDtprz e.iLsf . /var/groupe_tva/casa/
receiving incremental file list
rsync: mkdir "/home/stef/bin/~/drupal-files/proj1" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(605) [Receiver=3.0.9]
The line with mkdir should not have /home/stef/bin, why is bash adding the script's running dir on the variable?
Thanks
LOCAL_DIR="~/drupal-files/"
The string is in quotes so there's pathname expansion, and the variable will contain the literal string.
Remove the quotes.
$ x="~/test"; echo $x
~/test
$ x=~/test; echo $x
/home/user/test

Resources