This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 3 years ago.
So I have a relatively simple bash script:
#!/bin/bash
declare ALL="";
while IFS="" read -r line || [ -n "$line" ]
do
if [[ $line == 'ENV ms'* ]]; then
words=( $line )
if [[ ${#ALL} > 0 ]]; then
ALL="$ALL;${words[1]}=${words[2]}"
else
ALL="${words[1]}=${words[2]}"
fi
if [[ ${#ALL} > 0 ]]; then
printf '%s\n' "$ALL"
echo "${#ALL}"
fi
fi
done < Dockerfile
echo "$ALL"
echo "${#ALL}"
parsing a Dockerfile that looks like this:
#
# Configuration Stage
#
FROM maven:3.6.1-jdk-12 AS build
ENV HOME=/usr/local/ms-cards
RUN mkdir -p $HOME
WORKDIR $HOME
COPY maven-settings.xml /root/.m2/settings.xml
COPY pom.xml $HOME
RUN mvn -Dmaven.test.skip=true clean verify --fail-never
COPY . $HOME
RUN mvn -Dmaven.test.skip=true clean package
#
# Package stage
#
FROM openjdk:12
COPY --from=build /usr/local/ms-cards/target/ms-cards-1.0-exec.jar /usr/local/lib/ms-cards.jar
ENV ms_oauth_ip ms-oauth
ENV ms_oauth_port 48001
ENV ms_cards_client_id clientapp
ENV ms_cards_client_secret 123456
ENV ms_cards_port 48002
ENV ms_connection_port 48003
ENV ms_connection_ip ms-connection
EXPOSE 48002
ENTRYPOINT ["java","-jar","/usr/local/lib/ms-cards.jar"]
and it gives me this output:
ms_oauth_ip=ms-oauth
21
;ms_oauth_port=48001
42
;ms_cards_client_id=clientapp
72
;ms_cards_client_secret=123456
103
;ms_cards_port=48002ret=123456
124
;ms_connection_port=4800323456
150
;ms_connection_ip=ms-connection
182
;ms_connection_ip=ms-connection
182
So I can see that my ALL variable is growing in length...but when printf'ing it never seems to be growing...Anyone know what I'm doing wrong?
The Dockerfile has Windows \r\n line endings. Each \r that is printed causes the cursor to jump back to the beginning of the line and overwrite the previous setting.
Debugging tip: Use declare -p var to see exactly what's in a variable.
Related
I need to split a comma-separated string into array and run k6 for each of the array values parallely. Since shell script doesn't support arrays, I m using bash command in the script. I am not able to run it as bash using Dockerfile in TeamCity.
Dockerfile:
FROM loadimpact/k6:0.34.1
COPY ./src/lib /lib
COPY ./src/scenarios /scenarios
COPY ./src/k6-run-all.sh /k6-run-all.sh
WORKDIR /
ENTRYPOINT []
RUN bash -c "./k6-run-all.sh"
Shell script:
#!/bin/bash
K6_RUN_OPTIONS=${K6_RUN_OPTIONS}
ENV_NAME=${ENV_NAME:-qa}
IS_TEST_RUN=${IS_TEST_RUN:-true}
SCENARIO_NAME=${SCENARIO_NAME:-"full-card-visa"}
GWC_PC_ID=${GWC_PC_ID}
IFS=',' read -r -a PCIds <<< "$GWC_PC_ID"
echo "Number of PC ids provided in environment variables=" ${#PCIds[#]}
if [[ ${#PCIds[#]} > 0 ]]; then
for pcId in "$#"
do
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT GWC_PC_ID=$pcId k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js'' &
done
fi
existCode=$?
if [ $existCode -ne 0 ]; then
echo "Scenario $SCENARIO_NAME completed with the error"
exit $existCode
fi
Error:
#9 [5/6] RUN bash -c "./k6-run-all.sh"
17:02:02 #9 0.356 /bin/sh: bash: not found
17:02:02 #9 ERROR: executor failed running [/bin/sh -c bash -c "./k6-run-all.sh"]: exit code: 127
17:02:02 ------
17:02:02 > [5/6] RUN bash -c "./k6-run-all.sh":
17:02:02 #9 0.356 /bin/sh: bash: not found
17:02:02 ------
17:02:02 failed to solve: executor failed running [/bin/sh -c bash -c "./k6-run-all.sh"]: exit code: 127
How to modify Dockerfile or shell script to run this shell script as bash?
Previously to run it as bash script the last line of Dockerfile used to be:
CMD ["sh", "-c", "./k6-run-all.sh"]
******* Edit: **********
Updated full script after knittl's answer (current issue is after adding & for parallel runs it is not working, it is not running anything inside the for loop and it is not giving any extra error or information in logs, it is like it is skipping it):
K6_RUN_OPTIONS=${K6_RUN_OPTIONS}
ENV_NAME=${ENV_NAME:-qa}
IS_TEST_RUN=${IS_TEST_RUN:-true}
SCENARIO_NAME=${SCENARIO_NAME:-"full-card-visa"}
GWC_PC_ID=${GWC_PC_ID}
OPTIONS_VARIANT=""
if $IS_TEST_RUN; then
OPTIONS_VARIANT="-test"
fi
SCENARIO_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
SCENARIO_PATH="${SCENARIO_DIR}/scenarios"
SCENARIO="${SCENARIO_PATH}/${SCENARIO_NAME}"
echo "Executing scenario path $SCENARIO"
SCENARIO_NAME=${SCENARIO:${#SCENARIO_PATH}+1:${#SCENARIO}}
echo "Scenario Name: $SCENARIO_NAME"
echo "Run option: $SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT"
echo "pc ids provided in environment variable=" $GWC_PC_ID
if [ -z "$GWC_PC_ID" ]
then
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js''
else
for pcId in $(printf '%s' "$GWC_PC_ID" | tr , ' ');
do
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT GWC_PC_ID=$pcId k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js'' &
done
fi
existCode=$?
if [ $existCode -ne 0 ]; then
echo "Scenario $SCENARIO_NAME completed with the error"
exit $existCode
fi
k6 Docker containers do not come with bash preinstalled, but with busybox. I see two options:
Create your own Docker image based off grafana/k6 and manually install bash in your image.
Rewrite your script to not rely on bashisms. Should be fairly easy: split your list of tests to run into one path per line and while read -r path; do …; done them.
Or if support for whitespace in filenames is not required, then for path in $(printf '%s' "$GWC_PC_ID" | tr , ' '); do …; done
Note that your current script will return with the exit code of your last k6 process, meaning that if any other test failed but the last one was successfull, that will mask the error.
PS. Time to upgrade your base Docker image too. loadimpact/k6:0.34.1 is really old (exactly 1 year). It's better to switch to grafana/k6:0.40.0, which was released a week ago.
I have the following Dockerfile:
# => Build container
FROM node:14-alpine AS builder
WORKDIR /app
# split the dependecies from our source code so docker builds can cache this step
# (unless we actually change dependencies)
COPY package.json yarn.lock ./
RUN yarn install
COPY . ./
RUN yarn build
# => Run container
FROM nginx:alpine
# Nginx config
WORKDIR /usr/share/nginx/html
RUN rm -rf ./*
COPY ./nginx/nginx.conf /etc/nginx/conf.d/default.conf
# Default port exposure
EXPOSE 8080
# Copy .env file and shell script to container
COPY --from=builder /app/dist .
COPY ./env.sh .
COPY .env .
USER root
# Add bash
RUN apk update && apk add --no-cache bash
# Make our shell script executable
RUN chmod +x env.sh
# Start Nginx server
CMD ["/bin/bash", "-c", "/usr/share/nginx/html/env.sh && nginx -g \"daemon off;\""]
But when I try to run it I get the following errors when trying to add bash:
------
> [stage-1 8/9] RUN apk update && apk add --no-cache bash:
#19 0.294 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#19 0.358 140186175183688:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
#19 0.360 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#19 0.360 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: Permission denied
#19 0.360 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#19 0.405 140186175183688:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
#19 0.407 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: Permission denied
#19 0.407 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#19 0.408 2 errors; 42 distinct packages available
------
executor failed running [/bin/sh -c apk update && apk add --no-cache bash]: exit code: 2
Which I need to add to execute this script:
#!/bin/bash
# Recreate config file
rm -rf ./env-config.js
touch ./env-config.js
# Add assignment
echo "window._env_ = {" >> ./env-config.js
# Read each line in .env file
# Each line represents key=value pairs
while read -r line || [[ -n "$line" ]];
do
# Split env variables by character `=`
if printf '%s\n' "$line" | grep -q -e '='; then
varname=$(printf '%s\n' "$line" | sed -e 's/=.*//')
varvalue=$(printf '%s\n' "$line" | sed -e 's/^[^=]*=//')
fi
# Read value of current variable if exists as Environment variable
value=$(printf '%s\n' "${!varname}")
# Otherwise use value from .env file
[[ -z $value ]] && value=${varvalue}
# Append configuration property to JS file
echo " $varname: \"$value\"," >> ./env-config.js
done < .env
echo "}" >> ./env-config.js
From what I have read adding USER root should fix this but it does not in this case.
Any ideas on how to fix?
You should be able to use /bin/sh as a standard Bourne shell; also, you should be able to avoid the sh -c wrapper in the CMD line.
First, rewrite the script using POSIX shell syntax. Scanning over the script, it seems like it is almost okay as-is; change the first line to #!/bin/sh, and correct the non-standard ${!varname} expansion (also see Dynamic variable names in Bash)
# Read value of current variable if exists as Environment variable
value=$(sh -c "echo \$$varname")
# Otherwise use value from .env file
[[ -z $value ]] && value=${varvalue}
You can try testing it using an alpine or busybox image with a more restricted shell, or setting the POSIXLY_CORRECT environment variable with bash.
Secondly, there's a reasonably standard pattern of using ENTRYPOINT and CMD together. The CMD gets passed as arguments to the ENTRYPOINT, so if the ENTRYPOINT ends with exec "$#", it will replace itself with that command.
#!/bin/sh
# ^^ not bash
# Recreate config file
...
echo "window._env_ = {" >> ./env-config.js
...
echo "}" >> ./env-config.js
# Run the main container CMD
exec "$#"
Now, in your Dockerfile, you don't need to install GNU bash, because you're not using it, but you do need to correctly split out the ENTRYPOINT and CMD.
ENTRYPOINT ["/usr/share/nginx/html/env.sh"] # must be JSON-array syntax
CMD ["nginx", "-g", "daemon off;"] # can be either syntax
As an aside, the cmd1 && cmd2 syntax is basic shell syntax and not a bash extension, so you could write CMD ["/bin/sh", "-c", "cmd1 && cmd2"]; but, if you write a command without the JSON array syntax, Docker will insert the sh -c wrapper for you. You should almost never need to write out sh -c in a Dockerfile.
Trying to read values from a file inside gitlab-ci.yml file:
#!/usr/bin/bash
- IN_FILE="base_dir/apps_namespaces.txt"
- echo "IN_FILE = " > $IN_FILE
- while IFS= read -r line || [ -n "$line" ]; do
- echo "Name read from file - " > $line
- done < $IN_FILE
But this script fails with error - "/usr/bin/bash: line 131: read: line: invalid number"
Whereas same logic works in a plain script file, having following code:
#!/bin/bash
while IFS= read -r line || [[ -n "$line" ]];
do
echo "Text read from file: $line"
done < "$1"
when ran this script using command : ./scrpt apps_namespaces.txt
it rans and print each line.
What can be done if needs a long list to be read inside a gitlab-ci yml ?
Check if you can emulate this gitlab-ci.yml loop example, which:
does not include the shebang (#!/bin/bash)
does not have a - in front of each line
script:
[...]
- while read line; do
echo $line;
./sireg-linux exec --loader-sitemap-sitemap \"$line\" >> ./output/${line##*/}_out.txt;
done < sitemap-index
Of course, the alternative is to put the logic in a script, called from the pipeline, as documented in gitlab-org/gitlab-runner issue 2672
This is basic Unix, you just need your script to be executable in your git repo
chmod a+x scripts/test.sh
git add scripts/test.sh
git commit "fixing permissions"
git push
don't put /usr/bin/env for sh nor bash: both are 100% guaranteed to be present as #! /bin/sh or /bin/bash if bash is installed, or at least a much guaranteed to be there than /usr/bin/env to exists (and using env sh is not very good security)
A better approach which I followed:
put the logic to read file contents inside a script file. And refer this script inside gitlab-ci.yml worked.
It is cleaner and manageable.
Currently, I'm setting terminal title within screen command, but the bash script gives me:
Cannot exec 'source /etc/profile && title.set root#test': No such file or directory
And I can run above command successful directly from the command line, here are my scripts:
/usr/local/bin/s
#!/bin/bash
if [ $1 ]
then
screen -D -R $1 -m "source /etc/profile && title.set `whoami`#$1"
else
screen -R
fi
/etc/profile
...
# Source global bash config
if test "$PS1" && test "$BASH" && test -z ${POSIXLY_CORRECT+x} && test -r /etc/bash.bashrc; then
. /etc/bash.bashrc
fi
function title.set() {
if [[ -z "$ORIG" ]]; then
ORIG=$PS1
fi
TITLE="\[\e]2;$*\a\]"
PS1=${ORIG}${TITLE}
}
# Termcap is outdated, old, and crusty, kill it.
unset TERMCAP
# Man is much better than us at figuring this out
unset MANPATH
...
So What's going wrong here?
The keyword source is a bash built-in command, i.e., something for which there is not necessarily an actual file to exec (another built-in command). You can only exec something that is a file — like bash, e.g., something like this:
screen -D -R $1 -m bash -c "source /etc/profile && title.set `whoami`#$1"
I have a script called autoinstall:
#!/usr/bin/sh
echo "Installasi membutuhkan free space minimal 2MB, pastikan ada punya cukup space di router anda"
read -p "Anda yakin ingin melanjutkan installasi?(y/n) " -n 1 -r
echo ""
if [[ $REPLY = ^[Yy]$ ]]
then
cd /
cd /tmp/
tar -xvf OpenWrt_Angel_Beats_Edition_v1.3.3.tar -C /
chmod -R 744 /root/crt
chmod 744 /www/wget/wget_download.sh
chmod 744 /usr/bin/gsm
chmod 744 /usr/bin/profile
opkg update && opkg install elinks
cp /etc/rc.local /etc/rc.local.backup
cat > /etc/rc.local << END
#!bin/sh
# /etc/rc.local: Local system initialization script.
#
# Put any local startup commands in here. Also, if you have
# anything that needs to be run at shutdown time you can
# make an /etc/rc.d/rc.local_shutdown script and put those
# commands in there.
sh /www/wget/wget_download.sh > /dev/null 2>&1 &
exit 0
END
killall sh /www/wget/wget_download.sh
sh /www/wget/wget_download.sh > /dev/null 2>&1 &
echo "File backup /etc/rc.local.backup telah dibuat, gunakan file ini untuk mengembalikan konfigurasi rc.local anda yang dulu jika diperlukan"
echo "Installasi selesai. Jangan lupa di akun openvpn yang digunakan (/root/crt/xxx.ovpn) tambahkan baris ini:
script-security 2
up client-connect.sh"
else
echo ""
echo "Installasi dibatalkan"
fi
Every command that I put in the first line always gets the error above (line 1:xxx not found) and I'm sure I've typed in the correct command, even echo gives the error like that, how do I solve this?
There can be two problems here:
The file doesn't exist. Usually, for sh, the path is /bin/sh, so it should be #!/bin/sh
You're editing the file on Windows. Windows uses CR+LF as line ending. Unix (and Linux) uses just LF. So for Linux, the command reads "execute /bin/sh<CR> and sh<CR> doesn't exist.
Solution: When editing the file, make sure you use Unix line endings.
The file might have been edited with an editor that insert a Unicode BOM (Byte Order Mark).
Have a look to the first line contents with:
od -c autoinstall | head -1
or
hd -n 16 autoinstall
If you see unexpected characters before #!/usr/bin/sh, you might try one of the methods described here Using awk to remove the Byte-order mark to remove the BOM.