How to rewrite grep in #!/bin/sh? - shell

I have the following shell script, that does not compile:
#!/bin/sh
.....
.....
clean_up() {
echo "Clean up"
docker stop $KC_TEST_SVC
docker stop $KC_NAME
docker stop $POSTGRES_NAME
docker network rm $KC_NETWORK
}
# Check network if exists, if not then create
docker network inspect $KC_NETWORK
if [ $? -eq 1 ]; then
docker network create $KC_NETWORK
if [ $? -eq 1 ]; then
exit 1
fi
fi
docker run -d --rm --name $POSTGRES_NAME \
-e POSTGRES_DB=$POSTGRES_DB \
-e POSTGRES_USER=$POSTGRES_USER \
-e POSTGRES_PASSWORD=$POSTGRES_PW \
--network=$KC_NETWORK \
postgres:12.3
if [ $? -eq 1 ]; then
exit 1
fi
docker build --build-arg STAGE=int --tag $KC_TAG .
if [ $? -eq 1 ]; then
exit 1
fi
docker run -d --rm --name $KC_NAME \
-e DB_VENDOR=POSTGRES \
-e DB_ADDR=$POSTGRES_NAME \
-e DB_DATABASE=$POSTGRES_DB \
-e DB_USER=$POSTGRES_USER \
-e DB_PASSWORD=$POSTGRES_PW \
-e KEYCLOAK_USER=$KC_USER \
-e KEYCLOAK_PASSWORD=$KC_PW \
-e KEYCLOAK_LOGLEVEL=DEBUG \
--network=$KC_NETWORK \
$KC_TAG "-Dkeycloak.migration.action=import -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=/opt/jboss/keycloak/import-dir -Dkeycloak.migration.strategy=OVERWRITE_EXISTING"
if [ $? -eq 1 ]; then
exit 1
fi
# Wait until Keycloak get started
sleep 30
# Run test jetty service that is protected with Keycloak
docker run -d --rm --name $KC_TEST_SVC \
-v $(pwd)/app:/app \
--network $KC_NETWORK \
hub.databaker.io/devops/jetty-keycloak:0.1.6
if [ $? -eq 1 ]; then
exit 1
fi
# Request Tokens for credentials
KC_URL=http://$KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/token
echo "Keycloak URL => $KC_URL"
echo "Test the client connection to Keycloak with user $KC_TEST_USER"
KC_RESPONSE=$(
docker run --rm --network=$KC_NETWORK curlimages/curl:7.71.1 -X POST \
-d "username=$KC_TEST_USER" \
-d "password=$KC_TEST_PW" \
-d "grant_type=password" \
-d "client_id=$KC_TEST_CLIENT" \
$KC_URL | docker run --rm -i stedolan/jq .
)
echo "Response from Keycloak $KC_RESPONSE"
if grep -q "error" <<< "$KC_RESPONSE"; then
echo "++++++++Error+++++++++"
exit 1
fi
KC_ACCESS_TOKEN=$(echo "$KC_RESPONSE" | docker run --rm -i stedolan/jq -r .access_token)
echo "Access token from KC => $KC_ACCESS_TOKEN"
echo "Make request to protected service"
SVC_URL=http://$KC_TEST_SVC:8080/api/health
SVC_RES=$(docker run --rm --network=$KC_NETWORK curlimages/curl:7.71.1 -v -k \
-H "Authorization: Bearer $KC_ACCESS_TOKEN" \
-H "Accept: application/json" \
$SVC_URL | docker run --rm -i stedolan/jq .status)
clean_up
echo "$SVC_RES"
if [ "$SVC_RES" != "I am healthy" ]; then
echo "Test failed."
exit 1
else
echo "Test was successful. The service response $SVC_RES."
fi
it complains:
Syntax error: redirection unexpected
due to of the following line:
if grep -q "error" <<< "$KC_RESPONSE"; then
echo "++++++++Error+++++++++"
exit 1
fi
#!/bin/sh does not support grep. How to rewrite it?

<<< is a here string and it's a bash extension. In posix shell, just pipe the data.
if printf "%s\n" "$KC_RESPONSE" | grep -q "error"; then
Do not do if [ $? -eq 1 ]; then, it's error prone. Do if ! command; then. (You might want to research set -eu).

Change #!/bin/sh to #!/bin/bash (ev. replace bash location, which bash helps to find location of bash, and to see if bash is installed), or you may need to rewrite your script in a portable way. In this case, create a temporary file, put the string in such file, redirect such file to grep (with single <), and then delete the temporary file.
With such long description, you see why there is an extension in bash, on the other hand, it is nearly a syntactic sugar: you can easily do the same.

Related

How to check if commands inside the deployment.yaml are working?

I'm trying to write some bash commands inside my deployment.yaml file. I want to execute this commands at the postStart.
How can I check if the commands are working well? And if possible, how to get the node IP address?
lifecycle:
postStart:
exec:
command:
- "sh"
- "-c"
- |
GATEWAY_HTTPS_NAME="${GATEWAY_SERVICE_NAME}_SERVICE_PORT_HTTPS"
GATEWAY_HTTPS_PORT=$(eval "echo \$$GATEWAY_HTTPS_NAME")
cat /app/gateway/ip
BOOL=true
while [ $BOOL ]; do
if [ "$GATEWAY_HTTPS_PORT" != '' ]; then
nohup sh -c "sleep 30; cat /app/gateway/ip | xargs -I[] curl -H 'Content-Type: application/json' -d '{}' -k https://[]/actuator/refresh" &
ROUTES=$(nohup sh -c "cat /app/gateway/ip | xargs -I[] curl -H 'Content-Type: application/json' -d '{}' -k https://[]/actuator/gateway/routes")
else
nohup sh -c "sleep 30; cat /app/gateway/ip | xargs -I[] curl -H 'Content-Type: application/json' -d '{}' -k http://[]/actuator/refresh" &
ROUTES=$(nohup sh -c "cat /app/gateway/ip | xargs -I[] curl -H 'Content-Type: application/json' -d '{}' -k http://[]/actuator/gateway/routes")
fi
NODE_IP=???
echo $ROUTES
if [ $ROUTES[*] =~ $NODE_IP ]; then
BOOL=$false
fi
BOOL=$false
sleep 10
done
You can validate the command using exit status, I usually do that by adding a double pipe which will output the exit status if something went wrong, here is an example:
ls /unexistant/dir &>/dev/null && echo Success, Code=$? || echo Something went wrong, Code=$?
This will obviously throw an error and run the commands after double pipe since the status code not equals 0 will return Something went wrong, Code=2
Instead, if we run this:
ls /existant/dir &>/dev/null && echo Success, Code=$? || echo Something went wrong, Code=$?
This will run the first command with success and exit status will be 0 which means success, this is the output: Success, Code=0
You can of course play around with this and read more about double pipe

How to use bash variables in Jenkins multi-line shell script

I have unsuccessfully tried to use bash variables in Jenkins pipeline.
My first attempt
sh """#!/bin/bash
for file in *.map; do
filename=`basename $file .map`
echo "##### uploading ${$filename}"
curl -X POST ${SERVER_URL}/assets/v1/sourcemaps \
-F service_name="${SERVICE_NAME}" \
-F service_version="${revision}" \
-F bundle_filepath="${main_url}${filename}" \
-F sourcemap="#${filename}.map" &
done
wait
"""
Resulted in exception: MissingPropertyException: No such property: file
The second attempt, after seeing this answer https://stackoverflow.com/a/35047530/9590251
sh """#!/bin/bash
for file in *.map; do
filename=`basename \$file .map`
echo "##### uploading \$filename"
curl -X POST ${SERVER_URL}/assets/v1/sourcemaps \
-F service_name="${SERVICE_NAME}" \
-F service_version="${revision}" \
-F bundle_filepath="${main_url}\$filename" \
-F sourcemap="#\$filename.map" &
done
wait
"""
Simply omitted bash variables. So $filename was empty.
How do I need to property encode bash variables in this scenario?
Try this:
sh """#!/bin/bash
set -x
for file in *.map; do
filename="\$(basename "\$file" .map)"
echo "Uploading \$filename"
curl -X POST "${SERVER_URL}/assets/v1/sourcemaps" \
-F service_name="${SERVICE_NAME}" \
-F service_version="${revision}" \
-F bundle_filepath="${main_url}\$filename" \
-F sourcemap="#\${filename}.map" &
done
wait
"""

Environment variables not defined in SSH AuthorizedKeysCommand (Docker)

I'm trying to make the private key SSH connection with LDAP.
/etc/ssh/sshd_config
AuthorizedKeysCommand /etc/ldap_ssh_authorized_keys.sh
AuthorizedKeysCommandUser nobody
Script to get public keys from LDAP server
/etc/ldap_ssh_authorized_keys.sh
#!/bin/bash
USERSLIST=$( ldapsearch -x -D "${LDAP_USER}" -w "${LDAP_PASSWORD}" -H $LDAP_URI -b "${LDAP_BASEDN}" -s sub '(objectClass=posixAccount)' -u 'uid' \
grep '^uid:' | sed -n '/^ /{H;d};/uid:/x;$g;s/\n *//g;s/uid: //gp' \
)
while IFS= read -r line; do
exists=$(ldapsearch -x -D "${LDAP_USER}" -w "${LDAP_PASSWORD}" -H $LDAP_URI -b "${LDAP_BASEDN}" \
-s sub "(&(objectClass=posixGroup)(cn=sysadmin)(memberUid=${line}))" | grep "^# numEntries:")
if [[ ! -z $exists ]]
then
ldapsearch -x -D "${LDAP_USER}" -w "${LDAP_PASSWORD}" -H $LDAP_URI -b "${LDAP_BASEDN}" \
-s sub "(&(objectClass=posixAccount)(uid=${line}))" \
-u 'sshPublicKey' \
| sed -n '/^ /{H;d};/sshPublicKey:/x;$g;s/\n *//g;s/sshPublicKey: //gp'
echo -e "";
fi;
done <<< "$USERSLIST"
When I'm running script with /bin/bash it's working well and return my public keys.
All environment variables defined normally.
LDAP_URI
LDAP_BASEDN
LDAP_USER
LDAP_PASSWORD
The script also running normally when trying to make an SSH connection. But environment variables not available.
I'm trying also with AuthorizedKeysCommandUser as root. But nothing changed.
I solved this problem by getting the environment variables from /proc/1/environ.
Reference

Curl Works in Command Line, but Not Shell Script

I'm trying to automate a post-build process to get closer to a one-click release. However, I'm having problems with a script that.
Expected: The script to work exactly like the command line.
Results: Everything works, including grabbing the versionCodes except I am getting a curl: (26) Failed to open/read local data from file/application
The script I wanted to imitate with a more automated approach.
react-native bundle \
--platform android \
--dev false \
--entry-file index.js \
--bundle-output android-release.bundle \
--sourcemap-output android-release.bundle.map &&
curl https://upload.bugsnag.com/react-native-source-map \
-F apiKey=API-KEY \
-F appVersion=10.6.7 \
-F appVersionCode=4515 \
-F dev=false \
-F platform=android \
-F sourceMap=#android-release.bundle.map \
-F bundle=#android-release.bundle
react-native bundle \
--platform ios \
--dev false \
--entry-file index.js \
--bundle-output ios-release.bundle \
--sourcemap-output ios-release.bundle.map &&
curl https://upload.bugsnag.com/react-native-source-map \
-F apiKey=API-KEY \
-F appVersion=10.6.7 \
-F appBundleVersion=4515 \
-F dev=false \
-F platform=ios \
-F sourceMap=#ios-release.bundle.map \
-F bundle=#ios-release.bundle
So I wrote a script to fix this, which is
#!/bin/sh
# Ensure this is run on the commit used to generate the published app!
# Comment this line out to enable bundling and publishing
# debug=echo
bundle () {
platform=$1
$debug yarn react-native bundle \
--platform "$platform" \
--dev false \
--entry-file index.js \
--bundle-output "$platform-release.bundle" \
--sourcemap-output "$platform-release.bundle.map"
}
upload () {
platform=$1
version=$2
version_code=$3
# See https://docs.bugsnag.com/api/rn-source-map-upload/#uploading-source-maps
curl --http1.1 https://upload.bugsnag.com/react-native-source-map \
-F apiKey="$BUGSNAG_API_KEY" \
-F appVersion="$version" \
-F appVersionCode="$version_code" \
-F dev=false \
-F platform="$platform" \
-F sourceMap=#"$platform-release-bundle.map" \
-F bundle=#"$platform-release.bundle" \
}
read_version () {
local tag=$1
local file=$2
echo "$(grep -m 1 $tag $file | grep -o '[0-9]\+\(\.[0-9]\+\)*')"
}
BUGSNAG_API_KEY=$(plutil -p ios/RVLife/Info.plist | grep BugsnagAPIKey | awk '{ print $3 }' | sed 's/"//g')
if [[ -z $BUGSNAG_API_KEY ]]; then
echo "Couldn't find bugsnag API key"
exit 1
fi
IOS_VERSION=$(read_version MARKETING_VERSION ios/RVLife.xcodeproj/project.pbxproj)
echo "iOS app version: $IOS_VERSION"
IOS_VERSION_CODE=$(read_version CURRENT_PROJECT_VERSION ios/RVLife.xcodeproj/project.pbxproj)
echo "iOS version code: $IOS_VERSION_CODE"
if [[ -z $IOS_VERSION || -z $IOS_VERSION_CODE ]]; then
echo "Couldn't get iOS app versions"
exit 1
fi
bundle ios
upload ios "$IOS_VERSION" "$IOS_VERSION_CODE"
echo ""
ANDROID_VERSION=$(read_version versionName android/app/build.gradle)
echo "Android app version: $ANDROID_VERSION"
ANDROID_VERSION_CODE=$(read_version versionCode android/app/build.gradle)
echo "Android version code: $ANDROID_VERSION_CODE"
if [[ -z $ANDROID_VERSION || -z $ANDROID_VERSION_CODE ]]; then
echo "Couldn't get Android app versions"
exit 1
fi
bundle android
upload android "$ANDROID_VERSION" "$ANDROID_VERSION_CODE"
I'm assuming I messed up some syntax or permissions or something somewhere, but can't seem to find the area I did. Any help is appreciated.
Difference between bash -x myScript and set -x on command line
Command Line
omz_termsupport_preexec:1> [[ '' == true ]]
+omz_termsupport_preexec:3> emulate -L zsh
+omz_termsupport_preexec:4> setopt extended_glob
+omz_termsupport_preexec:7> local -a cmdargs
+omz_termsupport_preexec:8> cmdargs=( curl https://upload.bugsnag.com/react-native-source-map -F -F -F -F -F - )
+omz_termsupport_preexec:10> [[ curl = fg ]]
+omz_termsupport_preexec:44> local CMD=curl
+omz_termsupport_preexec:45> local LINE='curl https://upload.bugsnag.com/react-native-source-map -F -F -F -F -F -'
+omz_termsupport_preexec:47> title '$CMD' '%100>...>$LINE%<<'
+title:1> emulate -L zsh
+title:2> setopt prompt_subst
+title:4> [[ '' == *term* ]]
+title:8> : '%100>...>$LINE%<<'
+title:10> case xterm-256color (cygwin | xterm*)
+title:12> print -Pn '\e]2;%100\>...\>\$LINE%\<\<\a'
+title:13> print -Pn '\e]1;\$CMD\a'
+-zsh:75> curl https://upload.bugsnag.com/react-native-source-map -F 'apiKey=API-KEY' -F 'appVersion=10.6.7’ -F 'appBundleVersion=4515' -F 'dev=false' -F 'platform=ios' -F 'sourceMap=#ios-release.bundle.map' -F 'bundle=#ios-release.bundle'
OK%
SH Script
+ upload ios 10.6.7 4515
+ platform=ios
+ version=10.6.7
+ version_code=4515
+ curl --http1.1 https://upload.bugsnag.com/react-native-source-map -F apiKey=API-KEY -F appVersion=10.6.7 -F appVersionCode=4515 -F dev=false -F platform=ios -F sourceMap=#ios-release-bundle.map -F bundle=#ios-release.bundle
curl: (26) Failed to open/read local data from file/application
+ echo ''

Makefile (running outside scripts that have makefile macros)

I have some .PHONY targets such as 'clean', 'backup', and 'help'
the rule for some of these targets is very large.
For example:
.PHONY: backup
backup:
#$(GREEN)
#mkdir -p backup/include #make an backup include folder if it doesn't already exist
#mkdir -p backup/src #make a backup src folder if it doesn't already exist
#mkdir -p backup/docs #make a backup docs folder if it doesn't already exist
#total=0; headerCount=0; sourceCount=0; documentCount=0; \
for file in $(HEADER_PATH)*; do \
if ls $$file[~] >/dev/null 2>&1; then \
mv -fu $$file[~] backup/$$file; \
let "headerCount+=1"; \
echo $(DATE)[Backed Up] $$file~ >> $(LOG); \
fi; \
done; \
for file in $(SOURCE_PATH)*; do \
if ls $$file[~] >/dev/null 2>&1; then \
mv -fu $$file[~] backup/$$file; \
let "sourceCount+=1"; \
echo $(DATE)[Backed Up] $$file~ >> $(LOG); \
fi; \
done; \
for file in $(DOC_PATH)*; do \
if ls $$file[~] >/dev/null 2>&1; then \
mv -fu $$file[~] backup/$$file; \
let "documentCount+=1"; \
echo $(DATE)[Backed Up] $$file~ >> $(LOG); \
fi; \
done; \
let "total= headerCount + sourceCount + documentCount"; \
echo -n $(OUTPUT_PROMPT)" "; \
if test $$total -eq 0; then \
echo Nothing To Back up; \
else \
if test $$headerCount -eq $$total; then \
echo -n $$total" "; \
echo -n "Header"; \
if test $$total -ge 2; then \
echo -n "s"; \
fi; \
echo " Backed Up"; \
elif test $$sourceCount -eq $$total; then \
echo -n $$total" "; \
echo -n "Source"; \
if test $$total -ge 2; then \
echo -n "s"; \
fi; \
echo " Backed Up"; \
elif test $$documentCount -eq $$total; then \
echo -n $$total" "; \
echo -n "Document"; \
if test $$total -ge 2; then \
echo -n "s"; \
fi; \
echo " Backed Up"; \
else \
$(UNDERLINE); echo $$total " Files Backed Up"; $(UNUNDERLINE); \
if test $$headerCount -eq 1; then \
echo $(OUTPUT_PROMPT)" "$$headerCount header; \
elif test $$headerCount -ge 2; then \
echo $(OUTPUT_PROMPT)" "$$headerCount headers; \
fi; \
if test $$sourceCount -eq 1; then \
echo $(OUTPUT_PROMPT)" "$$sourceCount source; \
elif test $$sourceCount -ge 2; then \
echo $(OUTPUT_PROMPT)" "$$sourceCount sources; \
fi; \
if test $$documentCount -eq 1; then \
echo $(OUTPUT_PROMPT)" "$$documentCount document; \
elif test $$documentCount -ge 2; then \
echo $(OUTPUT_PROMPT)" "$$documentCount documents; \
fi; \
fi; \
fi;
#$(DEFAULT_TEXT)
what the code does is not important, but I wanted to illustrate that it has macros in which 'make' must expand, and that it also performs shell code (bash), and that some indication on what the script did, is displayed in the terminal.
I want to put this script outside of 'make' in another directory, and turn that code into something like this:
.PHONY: backup
backup:
#run scripts/backup.scr
#or something similar to that
How can I put the rule of my target (which is makefile/bash code) into a separate file, and have make practically paste it in so that it runs how I had it originally?
I thought I might be able to use the "include" command inside 'make'.
It looks like it is used to include other makefiles though..
maybe I should just paste the entire target/rule into another makefile, and then include that makefile into my main one?
Would that be the best way?
In your case, you have quite few output variables. It might be worth the hassle to separate the generation and execution, like:
clean : clean-script
sh clean-script
rm -f clean-script
clean-script : clean-script.in
sed -e 's:[#]HEADER_PATH[#]:$(HEADER_PATH):g' $<.in > $#
And write clean-script.in as a clean sh script with a few substitutions.
If you use GNU make, you can of course build a list of output varables like:
clean-script : clean-script.in
sed $(foreach var,$(SUBSTVARS),-e 's:[#]$(var)[#]:$($(var)):g') $<.in > $#
I don't know if it can help you, but you can run make inside some Makefile usually with a command (inside a rule) e.g.
$(MAKE) subtarget
See the section Recursive use of Make in the GNU make documentation.
I tend to dislike using make for complex projects (but unfortunately, I have to). If you are free to chose some other tool, you might consider omake and many others (cmake, scons, bake, ...)

Resources