I am trying to finalyze a script in Gitlab CI but struggling with some syntax error
script: |
echo "running jenkins job from user $EMAIL using following settings - $BRANCH / $TAGS in $ENV enviroment"
lastbuildNumber=$(curl -s --user ${EMAIL}:${TOKEN} "$JENKINS_URL/$ENV-SmokeTests-UIOnly/lastBuild/api/json" | jq ".number")
echo "last build was number ${lastbuildNumber}"
currentBuild=$((lastbuildNumber + 1 ))
echo "current build is ${currentBuild}"
echo "view cucumber report here"
baseurl="$JENKINS_URL/${ENV}-SmokeTests-UIOnly"
echo $baseurl
curl -s --user $EMAIL:$TOKEN $JENKINS_URL/$ENV-SmokeTests-UIOnly/ --output cucumber.txt
cucumber_endpoint=$(cat cucumber.txt | grep -o -m 1 "[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*")
full_cucumber=$baseurl$cucumber_endpoint
echo $full_cucumber
The script works fine on my local terminal, but fails in the CI when running
cucumber_endpoint=$(cat cucumber.txt | grep -o -m 1 "[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*")
is for sure something related to quotes but cannot work out what the issue is.
update:
I changed to:
after_script:
- |
echo "view cucumber report here"
baseurl="$JENKINS_URL/job/${ENV}-SmokeTests-UIOnly"
curl -s --user "$EMAIL":"$TOKEN" $JENKINS_URL/"$ENV"-SmokeTests-UIOnly/ --output cucumber.txt
cat cucumber.txt
cucumber_endpoint=$(cat cucumber.txt | grep -o -m 1 '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*')
full_cucumber="${baseurl}${cucumber_endpoint}"
echo "${full_cucumber}"
and I have run the script through 'shellcheck.net'
it's the grep that is not working but is not returning anyerror now.
the result of the cat command are correct, as on my local machine.
proving is not an issue with set -e
#!/bin/bash
set -e
echo "view cucumber report here"
baseurl="https://example"
cucumber_endpoint=$(curl -s --user "$EMAIL":"$TOKEN" ${JENKINS_URL}/"$ENV"-SmokeTests-UIOnly/ | grep -o -m 1 '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*')
# cat cucumber.txt
# cucumber_endpoint=$(cucumber.txt | grep -o -m 1 '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*')
full_cucumber="${cucumber_endpoint}"
echo "${baseurl}${full_cucumber}"
which gets what I want:
➜ ./cucumber.sh [16/02/23|11:39:59|]
view cucumber report here
https://example/cucumber-html-reports_fb3a3468-c298-3fb5-ad9a-dacbc0323763/overview-features.html
apparently gitlab ci did not like the -m 1 option in the grep call
so changed to
cucumber_endpoint=$(curl -s --user "$EMAIL":"$TOKEN" ${JENKINS_URL}/"$ENV"-SmokeTests-UIOnly/ | grep -o '[a-zA-Z.-]*/cucumber-html-reports_[a-zA-Z0-9.-]*/[a-zA-Z0-9.-]*'| sort -u)
I have a bash script that is executing a dataflow job like so
/deploy.sh
python3 main.py \
--runner DataflowRunner \
--region us-west1 \
--job_name name \
--project project \
--autoscaling_algorithm THROUGHPUT_BASED \
--max_num_workers 10 \
--environment prod \
--staging_location gs staging loc \
--temp_location temp loc \
--setup_file ./setup.py \
--subnetwork subnetwork \
--experiments use_network_tags=internal-ssh-server
So I use gitlab ci to run this
./gitlab-ci.yml
Deploy Prod:
stage: Deploy Prod
environment: production
script:
- *setup-project
- pip3 install --upgrade pip;
- pip install -r requirements.txt;
- chmod +x deploy.sh
- ./deploy.sh
only:
- master
So now my code runs and logs in the gitlab pipeline AND in the logs viewer in dataflow. What I want to be able to do is that once gitlab sees JOB_STATE_RUNNING, it marks the pipeline as passed and stops outputting logs to gitlab. Maybe there's a way to do this in the bash script? Or can it be done in gitlab ci?
GitLab doesn't have this capability as a feature, so your only is to script the solution.
Something like this should work to monitor the output and exit once that text is encountered.
python3 myscript.py > logfile.log &
( tail -f -n0 logfile.log & ) | grep -q "JOB_STATE_RUNNING"
echo "Job is running. We're done here"
exit 0
reference: https://superuser.com/a/900134/654263
You'd need to worry about some other things in the script, but that's the basic idea. GitLab unfortunately has no success condition based on trace output.
Little late, hope this helps someone, The way i solved it was I created two scripts
build.sh
#!/bin/bash
# This helps submit the beam dataflow job using nohup and parse the results for job submission
nohup stdbuf -oL bash ~/submit_beam_job.sh &> ~/output.log &
# Wait for log file to appear before checking on it
sleep 5
# Count to make sure the while loop is not stuck
cnt=1
while ! grep -q "JOB_STATE_RUNNING" ~/output.log; do
echo "Job has not started yet, waiting for it to start"
cnt=$((cnt+1))
if [[ $cnt -gt 5 ]]; then
echo "Job submission taking too long please check!"
exit 1
fi
# For other error on the log file, checking the keyword
if grep -q "Errno" ~/output.log || grep -q "Error" ~/output.log; then
echo "Error submitting Dataflow job, please check!!"
exit 1
fi
sleep 30
done
The submit beam job is like this
#!/bin/bash
# This submits individual beam jobs for each topics
export PYTHONIOENCODING=utf8
# Beam pipeline for carlistingprimaryimage table
python3 ~/dataflow.py \
--input_subscription=projects/data-warehouse/subscriptions/cars \
--window_size=2 \
--num_shards=2 \
--runner=DataflowRunner \
--temp_location=gs://bucket/beam-python/temp/ \
--staging_location=gs://bucket/beam-python/binaries/ \
--max_num_workers=5 \
--project=data-warehouse \
--region=us-central1 \
--gcs_project=data-warehouse \
--setup_file=~/setup.py
I've got a crontab running a shell script for an automated build of an ios app.
Running the shell script from the command line compiles and builds the .ipa with no issue. But when the crontab runs the shell script, the app gets compiled but does not fully finish with an .app or .ipa file. Why doesn't it run the same way?
CONFIGURATION="Adhoc"
OUTPUT_DIR="$PWD/Build/Products/Adhoc-iphoneos"
HOCKEY_TOKEN="token"
DEVELOPER_NAME="name"
########
#
# Cleanup
#
if [ -f "$OUTPUT_DIR/AppName-App.app.dSYM.zip" ]; then
rm -f "$OUTPUT_DIR/AppName-App.app.dSYM.zip"
fi
########
#
# Build The App
#
xcodebuild \
-workspace "AppName.xcworkspace" \
-scheme "AppName-App" \
-configuration "$CONFIGURATION" \
ONLY_ACTIVE_ARCH=NO clean build
########
#
# Code Sign
#
xcrun \
-sdk iphoneos PackageApplication \
-v "$OUTPUT_DIR/AppName-App.app" \
-o "$OUTPUT_DIR/AppName-App.ipa" \
--sign "$DEVELOPER_NAME"\
--embed AppName/BuildResources/AppName.mobileprovision
########
#
# Zip the dSYM
#
zip -r "$OUTPUT_DIR/AppName-App.app.dSYM.zip" "$OUTPUT_DIR/AppName-App.app.dSYM"
########
#
# Upload to HockeyApp
#
curl \
-F "status=2" \
-F "notify=0" \
-F "notes=automated build" \
-F "notes_type=1" \
-F "ipa=#$OUTPUT_DIR/AppName-App.ipa" \
-F "dsym=#$OUTPUT_DIR/AppName-App.app.dSYM.zip" \
-H "X-HockeyAppToken: $HOCKEY_TOKEN" \
https://rink.hockeyapp.net/api/2/apps/upload
When there are differences between a script executed in a logged in shell and via cron, it's almost always environment. Do you know what the value of $PWD is when run via cron? Looks like the provisioning profile you're embedding is using a relative path and might not be found.
I would try making all the paths absolute so you know exactly what the environment is when run in cron.
Also, you aren't showing the actual error messages. Those would obviously help. If you aren't redirecting to a file, cron will mail them to the owner of the crontab.
I want to Install Xcode Project in to iOS Device thru Terminal.I know how to Build Application using xcodebuild clean install this command.I want to launch app in device.Please can any body help me in this task?
I have tried some commands which executes successfully as well but now i am getting the following error when i tried to execute the following command.
xcodebuild -target "GoldenGate" -sdk "iOS 6.1" -configuration Release
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer/
PROJECT="GoldenGate"
SIGNING_IDENTITY="iPhone Developer: Deepak Shukla"
PROVISIONING_PROFILE="${WORKSPACE}/E6FD2816-7827-41AA-AC7E-2DC4833E637C.mobileprovision"
ARCHIVE="$(ls -dt ~/Library/Developer/Xcode/Archives//${PROJECT}.xcarchive|head -1)"
IPA_DIR="${WORKSPACE}"
DSYM="${ARCHIVE}/dSYMs/${PROJECT}.app.dSYM"
APP="${ARCHIVE}/Products/Applications/${PROJECT}.app"
/bin/rm -f "${IPA_DIR}/${PROJECT}.ipa"
/usr/bin/xcrun -sdk iphoneos PackageApplication \
-o "${IPA_DIR}/${PROJECT}.ipa" \
-verbose "${APP}" \
-sign "${SIGNING_IDENTITY}" \
--embed "${PROVISIONING_PROFILE}"
Embedding '/E6FD2816-7827-41AA-AC7E-2DC4833E637C.mobileprovision'
/bin/rm -rf /var/folders/wv/kv98qhfj6v36b2h0fkf_l66w0000gn/T/ouA93u702I/Payload/GoldenGate.app/embedded.mobileprovision
Program /bin/rm returned 0 : []
/bin/cp -rp /E6FD2816-7827-41AA-AC7E-2DC4833E637C.mobileprovision /var/folders/wv/kv98qhfj6v36b2h0fkf_l66w0000gn/T/ouA93u702I/Payload/GoldenGate.app/embedded.mobileprovision
Program /bin/cp returned 1 : [cp: /E6FD2816-7827-41AA-AC7E-2DC4833E637C.mobileprovision: No such file or directory
]
error: Unable to copy '/E6FD2816-7827-41AA-AC7E-2DC4833E637C.mobileprovision' to '/var/folders/wv/kv98qhfj6v36b2h0fkf_l66w0000gn/T/ouA93u702I/Payload/GoldenGate.app/embedded.mobileprovision'
Please help me in this task.
#!/bin/sh
# build.sh
#
# Created by iOSRider on 27/01/2014.
APPLICATION_NAME=MyApp
PROJDIR=/Users/iOSRider/Desktop/MyApp
PROJECT_NAME= MyApp
TARGET_SDK="iphoneos"
PROJECT_BUILDDIR="${PROJDIR}/build/Release-iphoneos"
TARGET_TEST_NAME="MyApp"
BUILD_HISTORY_DIR="/Users/iOSRider/Desktop/MyApp"
DEVELOPPER_NAME="iPhone Distribution: iOSRider India Limited (R8UAKS2M7L)"
PROVISONNING_PROFILE="/Users/iOSRider/Desktop/MyApp/iOS.mobileprovision"
# compile project echo Building Project cd "${PROJDIR}" xcodebuild -target "${PROJECT_NAME}" -sdk "${TARGET_SDK}" -configuration Release
#Check if build succeeded
if [ $? != 0 ] then exit 1 fi
/usr/bin/xcrun -sdk iphoneos PackageApplication -v "${PROJECT_BUILDDIR}/${APPLICATION_NAME}.app" -o "${BUILD_HISTORY_DIR}/${APPLICATION_NAME}.ipa" --sign "${DEVELOPPER_NAME}" --embed "${PROVISONNING_PROFILE}"
Running xcodebuild from the console will bring you very verbose output and I wasn't able to locate any options for limit its output in order to display only warnings and errors.
I'm looking for a way to capture the xcodebuild output and filter it. It would prefer a Python solution that will work with pipes but I'm open to other approaches as long they are command line based solutions.
Are any tools that are already able to do this?
Use xcodebuild -quiet.
According to the xcodebuild man page:
-quiet : Do not print any output except for warnings and errors.
Bonus: No other tools necessary! (Although I also like xcodebuild | xcpretty)
I build with Travis CI, which complains after 4 MB of logs. This argument solved the problem.
There’s a Ruby gem called xcpretty.
It filters the output of xcodebuild and also provides different formatters and coloring.
UPDATE: As Mike Hardy correctly states in the comments to this answer, xcpretty is no longer maintained.
This isn't sufficient for me. Piping to /dev/null will just show you that a build failed, but you don't see the reason(s) why. Ideally we could see just the errors and/or warnings without all of the successful compiler commands.
This basically does the job:
xcodebuild | grep -A 5 error:
To only see the error output messages, redirect the standard output to /dev/null (a special file that works as a black hole) like this:
xcodebuild > /dev/null
If you want to capture the error output into a file, you can do:
xcodebuild 2> ./build_errors.log
-quiet is the best way to do it at now.
There’s a Swift command line tool https://github.com/thii/xcbeautify that can also format xcodebuild output.
I love xcpretty for looking at as a human, but I had a need to find build errors in an automated setting for propagation elsewhere, and I wanted to be sure I captured just the relevant information. The following sed command serves that purpose:
xcodebuild | sed -nE '/error:/,/^[[:digit:]] errors? generated/ p'
Output:
main.c:16:5: error: use of undeclared identifier 'x'
x = 5;
^
main.c:17:5: error: use of undeclared identifier 'y'
y = 3;
^
2 errors generated.
I am building an expo project and there are a lot of warnings that come from libraries which I don't want to see. I was able to filter out the warnings with this command.
set -o pipefail \
&& xcodebuild build -workspace app.xcworkspace -scheme app \
CODE_SIGN_IDENTITY="" \
CODE_SIGNING_REQUIRED=NO \
CODE_SIGNING_ALLOWED=NO \
| xcpretty \
| grep --line-buffered -v -F "[-W" \
| grep --line-buffered -v -F "*" \
| grep --line-buffered -v -F "^" \
| grep --line-buffered -v -F ";" \
| grep --line-buffered -v -e "^$" \
| grep --line-buffered -v -F "#" \
| grep --line-buffered -v -F ")" \
| grep --line-buffered -v -F "/" \
| grep --line-buffered -v -F "}" \
| grep --line-buffered -v -F "{" \
| grep --line-buffered -v -F "\\" \
| grep --line-buffered -v -F "#" \
| grep --line-buffered -v -F ","
I'll admit it's a little sloppy but I couldn't get any of the other solutions to work. The -quiet option still printed hundreds of warnings I had no ability to resolve.
What is strange is that when I compile on the command line on the build machine I wasn't getting the warnings but when I compiled in my CI build I would get the warnings. Very annoying. I wish apple would provide a way to silence warnings for xcodebuild
I have been bitten by xcpretty swallowing way too much information in a CI environment, making it pretty hard to debug the error. -quiet hides output in a way that's a bit too aggressive, so I put together this one-liner and called it xcquiet.sh. It only hides specific lines, while preserving enough of the original output and not swallowing any other unexpected log entries.
if you want to remove warnings use
xcodebuild <command> | sed -e '/warning:/,/\^/d'
if you want to suppress warnings while using xcpretty try this
xcodebuild <command> \
| sed -e '/warning:/,/\^/d' \
| xcpretty -s
We run detox build command and it threw too many logs in CI.
-quite parameter failed with hiding logs.
The best solution is to hide and save logs in file:
npx detox build -c ios &> detox_build.log