I am trying to finalyze a script in Gitlab CI but struggling with some syntax error
script: |
echo "running jenkins job from user $EMAIL using following settings - $BRANCH / $TAGS in $ENV enviroment"
lastbuildNumber=$(curl -s --user ${EMAIL}:${TOKEN} "$JENKINS_URL/$ENV-SmokeTests-UIOnly/lastBuild/api/json" | jq ".number")
echo "last build was number ${lastbuildNumber}"
currentBuild=$((lastbuildNumber + 1 ))
echo "current build is ${currentBuild}"
echo "view cucumber report here"
baseurl="$JENKINS_URL/${ENV}-SmokeTests-UIOnly"
echo $baseurl
curl -s --user $EMAIL:$TOKEN $JENKINS_URL/$ENV-SmokeTests-UIOnly/ --output cucumber.txt
cucumber_endpoint=$(cat cucumber.txt | grep -o -m 1 "[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*")
full_cucumber=$baseurl$cucumber_endpoint
echo $full_cucumber
The script works fine on my local terminal, but fails in the CI when running
cucumber_endpoint=$(cat cucumber.txt | grep -o -m 1 "[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*")
is for sure something related to quotes but cannot work out what the issue is.
update:
I changed to:
after_script:
- |
echo "view cucumber report here"
baseurl="$JENKINS_URL/job/${ENV}-SmokeTests-UIOnly"
curl -s --user "$EMAIL":"$TOKEN" $JENKINS_URL/"$ENV"-SmokeTests-UIOnly/ --output cucumber.txt
cat cucumber.txt
cucumber_endpoint=$(cat cucumber.txt | grep -o -m 1 '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*')
full_cucumber="${baseurl}${cucumber_endpoint}"
echo "${full_cucumber}"
and I have run the script through 'shellcheck.net'
it's the grep that is not working but is not returning anyerror now.
the result of the cat command are correct, as on my local machine.
proving is not an issue with set -e
#!/bin/bash
set -e
echo "view cucumber report here"
baseurl="https://example"
cucumber_endpoint=$(curl -s --user "$EMAIL":"$TOKEN" ${JENKINS_URL}/"$ENV"-SmokeTests-UIOnly/ | grep -o -m 1 '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*')
# cat cucumber.txt
# cucumber_endpoint=$(cucumber.txt | grep -o -m 1 '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*')
full_cucumber="${cucumber_endpoint}"
echo "${baseurl}${full_cucumber}"
which gets what I want:
➜ ./cucumber.sh [16/02/23|11:39:59|]
view cucumber report here
https://example/cucumber-html-reports_fb3a3468-c298-3fb5-ad9a-dacbc0323763/overview-features.html
apparently gitlab ci did not like the -m 1 option in the grep call
so changed to
cucumber_endpoint=$(curl -s --user "$EMAIL":"$TOKEN" ${JENKINS_URL}/"$ENV"-SmokeTests-UIOnly/ | grep -o '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*'| sort -u)
Related
Introduction
Currently, I'm writing a customized GitHub workflow inside it I use a curl and grep command.
GitHub repo
action.yml
- name: get new tag
id: get_new_tag
shell: bash
run: |
temp_result=$(curl -s https://api.github.com/repos/${{inputs.github_repository}}/tags | grep -h "name" | grep -h "${{inputs.selector}}" | head -1 | grep -ho "${{inputs.regex}}")
echo "new-tag=${temp_result}" >> $GITHUB_OUTPUT
Full code here
test for action.yml
uses: ./
with:
files: tests/test1.txt tests/test2.txt
github_repository: MathieuSoysal/file-updater-for-release
prefix: MathieuSoysal/file-updater-for-release#
Full code are is here
Problem
I don't understand why but my GitHub Actions don't work with MacOS.
The given error:
temp_result=$(curl -s https://api.github.com/repos/MathieuSoysal/file-updater-for-release/tags | grep -h "name" | grep -h "" | head -1 | grep -ho "v\?[0-9.]*")
echo "new-tag=${temp_result}" >> $GITHUB_OUTPUT
shell: /bin/bash --noprofile --norc -e -o pipefail {0}
Error: Process completed with exit code 1.
Full log error is here
Question
Does someone know how we can fix this issue?
Alternatively, you can simply use the jq command line utility for this which is already installed and available on the GitHub runners. See the preinstalled software.
$ export URL='https://api.github.com/repos/MathieuSoysal/file-updater-for-release/tags'
$ curl -s $URL | jq -r .[0].name
v1.0.3
The issue is from this command grep -h "", this command is not supported on MacOS.
The empty string is not supported, the solution is to add something inside it.
I have the following shell script:
for var in $(curl -u "abc_oOQxMv:tgMn6FYCKJcd6ujuDLEK" -X GET "https://api-cloud.browserstack.com/app-automate/recent_apps" | jq '.[] | select(.custom_id =="android")' | jq -r '.app_id')
do
echo "Deleting $var"
curl -u "abc_oOQxMv:tgMn6FYCKJcd6ujuDLEK" -X DELETE "https://api-cloud.browserstack.com/app-automate/app/delete/$var"
done
How do I write it into a YML file (i.e. config.yml in CircleCI) in Github Action?
You can run multiline commands in the workflow yaml file using run : | (with pipe) in a step.
For example:
script-job:
runs-on: ubuntu-latest
steps:
- name: Run my script
shell: bash
run: |
for var in $(curl -u "abc_oOQxMv:tgMn6FYCKJcd6ujuDLEK" -X GET
"https://api-cloud.browserstack.com/app-automate/recent_apps" | jq '.[] | select(.custom_id =="android")' | jq -r '.app_id')
do
echo "Deleting $var"
curl -u "abc_oOQxMv:tgMn6FYCKJcd6ujuDLEK" -X DELETE "https://api-cloud.browserstack.com/app-automate/app/delete/$var"
done
i am trying to download postgresql database dump to my local machine by using the below shell script
#!/bin/bash -x
namespace="staging"
current_date=`date +%d_%b_%Y`
file_name="${namespace}_${current_date}.dump"
kubectl config use-context <context_name> -n $namespace
pod_name=`kubectl get pods -n ${namespace} -l app=backend | grep backend | awk '{print $1}'`
kubectl exec -it $pod_name --container <cname> -n $namespace -- sh -c PGPASSWORD=$DATABASE_PASSWORD pg_dump --no-owner -x -Fc $DATABASE_NAME -h $DATABASE_HOST -U $DATABASE_USERNAME > $file_name
but this does not work and resultant file is blank without any error.
$DATABASE_PASSWORD, $DATABASE_NAME, $DATABASE_HOST & $DATABASE_USERNAME
are all environment variables.
Any help on how to get this working would be really helpful.
Thanks.
I am trying to use wget in a bash and display a custom download percentage per file so that the user knows the process is running and what has been downloaded. The below seems to download the files but there is no percentage or filename being downloaded displayed. I am not sure why and cannot seem to figure it out. Thank you :).
list
xxxx://www.xxx.com/xxx/xxxx/xxx/FilterDuplicates.html
xxxx://www.xxx.com/xxx/xxxx/xxx/file1.bam
xxxx://www.xxx.com/xxx/xxxx/xxx/file2.bam
xxxx://www.xxx.com/xxx/xxxx/xxx/file1.vcf.gz
xxxx://www.xxx.com/xxx/xxxx/xxx/file2.vcf.gz
bash that uses list to download all fiiles
# download all from list
download() {
local url=$1
echo -n " "
wget --progress=dot $url 2>&1 | grep --line-buffered "%" | sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
echo -ne "\b\b\b\b"
echo " starting download"
}
cd "/home/user/Desktop/folder/subfolder"
wget -i /home/cmccabe/list --user=xxx--password=xxx --xxx \
xxxx://www.xxx.com/xxx/xxxx/xxx/ 2>&1 -o wget.log | download
I need an output for multiple grep commands.
patterns: ([^"#]+)
wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"'
wget -q -O - http://www.site2.com | grep -o -E -m 1 'site2content"([^"#]+)"
.........
Output file:
http://www.site1.com***pattern
http://www.site2.com***pattern
Just redirect the output of your commands to a file.
wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"' > output.txt
wget -q -O - http://www.site2.com | grep -o -E -m 1 'site2content"([^"#]+)"' >> output.txt
> overwrites old content and >> appends to the end of the file.
Edit:
Not pretty but a quick and dirty solution might be
echo 'http://www.site1.com***'`wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"'` > output.txt
(untested)
As is the output you got from the above commends consists only of the pattern found because of the -o parameter:
http://explainshell.com/explain?cmd=grep+-o
I suggest using the above site for an explanation.