Start server, run tests, stop server - makefile

I have a Makefile target that looks like the following
integration-test: git-hooks
java -Djava.library.path=$$(pwd)/test/integration/lib/DynamoDBLocal_lib \
-Djava.util.logging.config.file=/dev/null \
-Dorg.eclipse.jetty.LEVEL=WARN \
-Dlog4j.com.amazonaws.services.dynamodbv2.local.server.LocalDynamoDBServerHandler=OFF \
-jar $$(pwd)/test/integration/lib/DynamoDBLocal.jar \
-inMemory \
-port 8000 &
sleep 3
./node_modules/.bin/mocha --compilers coffee:coffee-script/register \
--reporter spec \
test/integration/main.coffee
ps -ef | grep [D]ynamoDBLocal_lib | awk '{print $$2}' | xargs kill
Here's what I'm doing:
the Java command starts a local instance of Amazon's DynamoDB.
I give it 3 seconds to start
I run my integration tests
I kill the database
What I would like is kill the database regardless of the fact that the tests passed or not.
To do that I suppose I need the exit status of the test command and return that, both if the tests fails or if they succeeded.
What is happening is that if tests pass the database is correctly killed, if the tests fail it's not.
I've read in the docs that you can prepend a - in front a command to have make ignore it if it produce a non zero exit status, the problem if I do that is that I don't know if the tests failed or not, since $? will always return 0.
What's the usual practice in this scenario? I'm fine in splitting the target in more targets if that solves my issue.
Thank you.

You'll have to run the entire thing in a single shell, which means you'll need to use command separators (e.g., ;) and backslashes to connect the lines. Then you can store the result and exit with it:
integration-test: git-hooks
{ java -Djava.library.path=$$(pwd)/test/integration/lib/DynamoDBLocal_lib \
-Djava.util.logging.config.file=/dev/null \
-Dorg.eclipse.jetty.LEVEL=WARN \
-Dlog4j.com.amazonaws.services.dynamodbv2.local.server.LocalDynamoDBServerHandler=OFF \
-jar $$(pwd)/test/integration/lib/DynamoDBLocal.jar \
-inMemory \
-port 8000 & }; \
sleep 3; \
./node_modules/.bin/mocha --compilers coffee:coffee-script/register \
--reporter spec \
test/integration/main.coffee; \
r=$$?; \
ps -ef | grep [D]ynamoDBLocal_lib | awk '{print $$2}' | xargs kill; \
exit $$r
However, you can actually do even better if you use a single shell, by killing only the exact process you want instead of using ps:
integration-test: git-hooks
{ java -Djava.library.path=$$(pwd)/test/integration/lib/DynamoDBLocal_lib \
-Djava.util.logging.config.file=/dev/null \
-Dorg.eclipse.jetty.LEVEL=WARN \
-Dlog4j.com.amazonaws.services.dynamodbv2.local.server.LocalDynamoDBServerHandler=OFF \
-jar $$(pwd)/test/integration/lib/DynamoDBLocal.jar \
-inMemory \
-port 8000 & }; \
pid=$$!; \
sleep 3; \
./node_modules/.bin/mocha --compilers coffee:coffee-script/register \
--reporter spec \
test/integration/main.coffee; \
r=$$?; \
kill $$pid; \
exit $$r

Related

How to execute bash variable with double quotes and get the output in realtime

I have a variable that has a command that I want to run.
It has a bunch of double-quotes. when I echo it, it looks beautiful.
I can copy-paste it and run it just fine.
I tried simply $cmd, but it doesn't work. I get an error as if the command is malformed.
I then tried running it via eval "$cmd" or similarly, bash -c "$cmd", which works, but I don't get any output until the command is done running.
Example with bash -c "$cmd":
This runs the command, BUT I don't get any output until the command is done running, which sucks and I'm trying to fix that:
cmd="docker run -v \"$PROJECT_DIR\":\"$PROJECT_DIR\" \
-v \"$PPI_ROOT_DIR/utilities/build_deploy/terraform/modules/\":/ppi_modules \
--workdir \"$PROJECT_DIR/terraform\" \
--env TF_VAR_aws_account_id=$AWS_ACCOUNT_ID \
--env TF_VAR_environment=${ENVIRONMENT} \
--env TF_VAR_region=${AWS_DEFAULT_REGION:-us-west-2} \
${OPTIONAL_AWS_ENV_VARS} \
${CUSTOM_TF_VARS} \
${TERRAFORM_BASE_IMAGE} \
init --plugin-dir=/.terraform/providers \
-reconfigure \
-backend-config=\"bucket=${AWS_ACCOUNT_ID}-tf-remote-state\" \
-backend-config=\"key=${ENVIRONMENT}/${PROJECT_NAME}\" \
-backend-config=\"region=us-west-2\" \
-backend-config=\"dynamodb_table=terraform-locks\" \
-backend=true"
# command output looks good. I can copy and paste it and run it my terminal too.
echo $cmd
# Running the command via bash works,
# but I don't get the output until the command is done running,
# which is what I'm trying to fix:
bash -c "$cmd"
Here is an example using bash array.
It prints it to screen perfectly, but just like running it like $cmd, it throws an error as if the command is malformed:
cmd=(docker run -v \"$PROJECT_DIR\":\"$PROJECT_DIR\" \
-v \"$PPI_ROOT_DIR/utilities/build_deploy/terraform/modules/\":/ppi_modules \
--workdir \"$PROJECT_DIR/terraform\" \
--env TF_VAR_aws_account_id=$AWS_ACCOUNT_ID \
--env TF_VAR_environment=${ENVIRONMENT} \
--env TF_VAR_region=${AWS_DEFAULT_REGION:-us-west-2} \
${OPTIONAL_AWS_ENV_VARS} \
${CUSTOM_TF_VARS} \
${TERRAFORM_BASE_IMAGE} \
init --plugin-dir=/.terraform/providers \
-reconfigure \
-backend-config=\"bucket=${AWS_ACCOUNT_ID}-tf-remote-state\" \
-backend-config=\"key=${ENVIRONMENT}/${PROJECT_NAME}\" \
-backend-config=\"region=us-west-2\" \
-backend-config=\"dynamodb_table=terraform-locks\" \
-backend=true)
echo "${cmd[#]}"
"${cmd[#]}"
How can I execute a bash variable that has double-quotes, but run it so I get the output in realtime, just as if I executed via $cmd (which doesn't work)
Similar to these questions, but my question is to run it AND get the output in realtime:
Execute command containing quotes from shell variable
Bash inserting quotes into string before execution
bash script execute command with double quotes, single quotes and spaces
In your array version, double quotes escaped by a backslash become part of the arguments, which is not intended.
So removing backslashes should fix the issue.

Mark Gitlab CI stage as passed based on condition in job log

I have a bash script that is executing a dataflow job like so
/deploy.sh
python3 main.py \
--runner DataflowRunner \
--region us-west1 \
--job_name name \
--project project \
--autoscaling_algorithm THROUGHPUT_BASED \
--max_num_workers 10 \
--environment prod \
--staging_location gs staging loc \
--temp_location temp loc \
--setup_file ./setup.py \
--subnetwork subnetwork \
--experiments use_network_tags=internal-ssh-server
So I use gitlab ci to run this
./gitlab-ci.yml
Deploy Prod:
stage: Deploy Prod
environment: production
script:
- *setup-project
- pip3 install --upgrade pip;
- pip install -r requirements.txt;
- chmod +x deploy.sh
- ./deploy.sh
only:
- master
So now my code runs and logs in the gitlab pipeline AND in the logs viewer in dataflow. What I want to be able to do is that once gitlab sees JOB_STATE_RUNNING, it marks the pipeline as passed and stops outputting logs to gitlab. Maybe there's a way to do this in the bash script? Or can it be done in gitlab ci?
GitLab doesn't have this capability as a feature, so your only is to script the solution.
Something like this should work to monitor the output and exit once that text is encountered.
python3 myscript.py > logfile.log &
( tail -f -n0 logfile.log & ) | grep -q "JOB_STATE_RUNNING"
echo "Job is running. We're done here"
exit 0
reference: https://superuser.com/a/900134/654263
You'd need to worry about some other things in the script, but that's the basic idea. GitLab unfortunately has no success condition based on trace output.
Little late, hope this helps someone, The way i solved it was I created two scripts
build.sh
#!/bin/bash
# This helps submit the beam dataflow job using nohup and parse the results for job submission
nohup stdbuf -oL bash ~/submit_beam_job.sh &> ~/output.log &
# Wait for log file to appear before checking on it
sleep 5
# Count to make sure the while loop is not stuck
cnt=1
while ! grep -q "JOB_STATE_RUNNING" ~/output.log; do
echo "Job has not started yet, waiting for it to start"
cnt=$((cnt+1))
if [[ $cnt -gt 5 ]]; then
echo "Job submission taking too long please check!"
exit 1
fi
# For other error on the log file, checking the keyword
if grep -q "Errno" ~/output.log || grep -q "Error" ~/output.log; then
echo "Error submitting Dataflow job, please check!!"
exit 1
fi
sleep 30
done
The submit beam job is like this
#!/bin/bash
# This submits individual beam jobs for each topics
export PYTHONIOENCODING=utf8
# Beam pipeline for carlistingprimaryimage table
python3 ~/dataflow.py \
--input_subscription=projects/data-warehouse/subscriptions/cars \
--window_size=2 \
--num_shards=2 \
--runner=DataflowRunner \
--temp_location=gs://bucket/beam-python/temp/ \
--staging_location=gs://bucket/beam-python/binaries/ \
--max_num_workers=5 \
--project=data-warehouse \
--region=us-central1 \
--gcs_project=data-warehouse \
--setup_file=~/setup.py

spark-submit: command not found

A very simple question:
I try to use a bash script to submit spark jobs. But somehow it keeps complaining that it cannot find spark-submit command.
But when I just copy out the command and run directly in my terminal, it runs fine.
My shell is fish shell, here's what I have in my fish shell config: ~/.config/fish/config.fish:
alias spark-submit='/Users/MY_NAME/Downloads/spark-2.0.2-bin-hadoop2.7/bin/spark-submit'
Here's my bash script:
#!/usr/bin/env bash
SUBMIT_COMMAND="HADOOP_USER_NAME=hdfs spark-submit \
--master $MASTER \
--deploy-mode client \
--driver-memory $DRIVER_MEMORY \
--executor-memory $EXECUTOR_MEMORY \
--num-executors $NUM_EXECUTORS \
--executor-cores $EXECUTOR_CORES \
--conf spark.shuffle.compress=true \
--conf spark.network.timeout=2000s \
$DEBUG_PARAM \
--class com.fisher.coder.OfflineIndexer \
--verbose \
$JAR_PATH \
--local $LOCAL \
$SOLR_HOME \
--solrconfig 'resource:solrhome/' \
$ZK_QUORUM_PARAM \
--source $SOURCE \
--limit $LIMIT \
--sample $SAMPLE \
--dest $DEST \
--copysolrconfig \
--shards $SHARDS \
$S3_ZK_ZNODE_PARENT \
$S3_HBASE_ROOTDIR \
"
eval "$SUBMIT_COMMAND"
What I've tried:
I could run this command perfectly fine on my Mac OS X fish shell when I copy this command literally out and directly run.
However, what I wanted to achieve is to be able to run ./submit.sh -local which executes the above shell.
Any clues please?
You seem to be confused about what a fish alias is. When you run this:
alias spark-submit='/Users/MY_NAME/Downloads/spark-2.0.2-bin-hadoop2.7/bin/spark-submit'
You are actually doing this:
function spark-submit
/Users/MY_NAME/Downloads/spark-2.0.2-bin-hadoop2.7/bin/spark-submit $argv
end
That is, you are defining a fish function. Your bash script has no knowledge of that function. You need to either put that path in your $PATH variable or put a similar alias command in your bash script.
Make sure this command is added to path:
export PATH=$PATH:/Users/{your_own_path_where_spark_installed}/bin
For mac, open either one of these files ~/.bash, ~/.zprofile, ~/.zshrc and add the command below in the file.

Vagrant keeps losing file doing provision

I'm running into an odd behavior on the latest version of vagrant in a Windows7/msys/Virtualbox environment setup, where after executing a vagrant up command I get an error with rsync; 'file has vanished: "/c/Users/spencerd/workspace/watcher/.LISTEN' doing the provisioning stage.
Since google, irc, and issue trackers have little to no documentation on this issue I wonder if anyone else ran into this and what would the fix be?
And for the record I have successfully build a box using the same vagrant file and provisioning script. For those that want to look, the project code is up at https://gist.github.com/denzuko/a6b7cce2eae636b0512d, with the debug log at gist.github.com/
After digging further into the directory structure and running into issues with git pushing code up I was able to find a non-existant file that needed to be removed after a reboot.
Thus, doing a reboot and a rm -rf -- "./.LISTEN\ \ \ \ \ 0\ \ \ \ \ \ 100\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ " did the trick.

Creating a zip for my app in bash script

I'm trying to use Xcode 5 bots for my Continuos Integration. I need to create a zip-file of my app-file.
In my scheme under Archive I use this script:
LATESTBUILD=$(ls -1rt /Library/Server/Xcode/Data/BotRuns | tail -1)
APP="/Library/Server/Xcode/Data/BotRuns/${LATESTBUILD}/output/Archive.xcarchive/Products/Applications/${PRODUCT_NAME}.app"
echo "Zipping .app for ${PRODUCT_NAME}"
/usr/bin/zip -r "${APP}.zip" "${APP}"
echo "Sending to *HockeyApp*"
curl \
-F "status=2" \
-F "notify=0" \
-F "notes=Testing CI" \
-F "notes_type=0" \
-F "ipa=${APP}.zip" \
-H "X-HockeyAppToken: myToken \
https://rink.hockeyapp.net/api/2/apps/myAppID/app_versions/upload
This will create a .zip-file. However, not the actual app but the whole folder structure for ${app}.
How can I create a zip-file that only contains the actual app?
To specify a file parameter with curl, you have to prefix the filename with #. Correct would be:
echo "Sending to *HockeyApp*"
curl \
-F "status=2" \
-F "notify=0" \
-F "notes=Testing CI" \
-F "notes_type=0" \
-F "ipa=#${APP}.zip" \
-H "X-HockeyAppToken: myToken \
https://rink.hockeyapp.net/api/2/apps/myAppID/app_versions/upload
Fixed this by stepping in to the directory before zipping it.
cd "/tmp/Archive.xcarchive/Products/Applications/"

Resources