I have a variable that has a command that I want to run.
It has a bunch of double-quotes. when I echo it, it looks beautiful.
I can copy-paste it and run it just fine.
I tried simply $cmd, but it doesn't work. I get an error as if the command is malformed.
I then tried running it via eval "$cmd" or similarly, bash -c "$cmd", which works, but I don't get any output until the command is done running.
Example with bash -c "$cmd":
This runs the command, BUT I don't get any output until the command is done running, which sucks and I'm trying to fix that:
cmd="docker run -v \"$PROJECT_DIR\":\"$PROJECT_DIR\" \
-v \"$PPI_ROOT_DIR/utilities/build_deploy/terraform/modules/\":/ppi_modules \
--workdir \"$PROJECT_DIR/terraform\" \
--env TF_VAR_aws_account_id=$AWS_ACCOUNT_ID \
--env TF_VAR_environment=${ENVIRONMENT} \
--env TF_VAR_region=${AWS_DEFAULT_REGION:-us-west-2} \
${OPTIONAL_AWS_ENV_VARS} \
${CUSTOM_TF_VARS} \
${TERRAFORM_BASE_IMAGE} \
init --plugin-dir=/.terraform/providers \
-reconfigure \
-backend-config=\"bucket=${AWS_ACCOUNT_ID}-tf-remote-state\" \
-backend-config=\"key=${ENVIRONMENT}/${PROJECT_NAME}\" \
-backend-config=\"region=us-west-2\" \
-backend-config=\"dynamodb_table=terraform-locks\" \
-backend=true"
# command output looks good. I can copy and paste it and run it my terminal too.
echo $cmd
# Running the command via bash works,
# but I don't get the output until the command is done running,
# which is what I'm trying to fix:
bash -c "$cmd"
Here is an example using bash array.
It prints it to screen perfectly, but just like running it like $cmd, it throws an error as if the command is malformed:
cmd=(docker run -v \"$PROJECT_DIR\":\"$PROJECT_DIR\" \
-v \"$PPI_ROOT_DIR/utilities/build_deploy/terraform/modules/\":/ppi_modules \
--workdir \"$PROJECT_DIR/terraform\" \
--env TF_VAR_aws_account_id=$AWS_ACCOUNT_ID \
--env TF_VAR_environment=${ENVIRONMENT} \
--env TF_VAR_region=${AWS_DEFAULT_REGION:-us-west-2} \
${OPTIONAL_AWS_ENV_VARS} \
${CUSTOM_TF_VARS} \
${TERRAFORM_BASE_IMAGE} \
init --plugin-dir=/.terraform/providers \
-reconfigure \
-backend-config=\"bucket=${AWS_ACCOUNT_ID}-tf-remote-state\" \
-backend-config=\"key=${ENVIRONMENT}/${PROJECT_NAME}\" \
-backend-config=\"region=us-west-2\" \
-backend-config=\"dynamodb_table=terraform-locks\" \
-backend=true)
echo "${cmd[#]}"
"${cmd[#]}"
How can I execute a bash variable that has double-quotes, but run it so I get the output in realtime, just as if I executed via $cmd (which doesn't work)
Similar to these questions, but my question is to run it AND get the output in realtime:
Execute command containing quotes from shell variable
Bash inserting quotes into string before execution
bash script execute command with double quotes, single quotes and spaces
In your array version, double quotes escaped by a backslash become part of the arguments, which is not intended.
So removing backslashes should fix the issue.
Related
I'm using a Makefile to run various docker-compose commands and I'm trying to capture the output of a script run on my local machine and pass that value to a Docker image.
start-service:
VERSION=$(shell aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$(VERSION) \
connect make run-service
When I run this I can see the variable being assigned but it still errors. Why is the value not getting passed into the -e argument:
VERSION=1.2.3-build342 && \
docker-compose -f ./compose/docker-compose.yml run --rm \
-e VERSION?=$(VERSION) \
connect make run-connect
/bin/sh: VERSION: command not found
You're mixing several different Bourne shell and Make syntaxes here. The Make $$(VERSION) translates to shell $(VERSION), which is command-substitution syntax; GNU Make $(shell ...) generally expands at the wrong time and isn't what you want here.
If you were writing this as an ordinary shell command it would look like
# Set VERSION using $(...) substitution syntax
# Refer to just plain $VERSION
VERSION=$(aws s3 ls s3://redact/downloads/1.2.3/) && ... \
-e VERSION=$VERSION ... \
So when you use this in a Make context, if none of the variables are Make variables (they get set and used in the same command), just double the $ to $$ not escape them.
start-service:
VERSION=$$(aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$VERSION \
connect make run-service
I'm running several psql commands inside a bash shell script. One of the commands imports a csv file to a table. The problem is, the CSV file is occasionally corrupt, it has invalid characters at the end and the import fails. When that happens, and I have the ON_ERROR_STOP=on flag set, my entire shell script stops at that point as well.
Here's the relevant bits of my bash script:
$(psql \
-X \
$POSTGRES_CONNECTION_STRING \
-w \
-b \
-L ./output.txt
-A \
-q \
--set ON_ERROR_STOP=on \
-t \
-c "\copy mytable(...) from '$input_file' csv HEADER"\
)
echo "import is done"
The above works fine as long as the csv file isn't corrupt. If it is however, psql spits out a message to the console that begins ERROR: invalid byte sequence for encoding "UTF8": 0xb1 and my bash script apparently stops cold at that point-- my echo statement above doesn't execute, and neither do any other subsequent commands.
Per the psql documentation, a hard stop in psql should return an error code of 3:
psql returns 0 to the shell if it finished normally, 1 if a fatal error of its own occurs (e.g. out of >memory, file not found), 2 if the connection to the server went bad and the session was not >interactive, and 3 if an error occurred in a script and the variable ON_ERROR_STOP was set
That's fine and good, but is there a reason returning a value of 3 should terminate my calling bash script? And can I prevent that? I'd like to keep ON_ERROR_STOP set to on because I actually have other commands I'd like to run in that psql statement if the intial import succeeds, but not if it doesn't.
ON_ERROR_STOP will not work with the -c option.
Also, the $(...) surronding the psql look wrong — do you want to execute the output as a command?
Finally, you forgot a backslash after the -L option
Try using a “here document”:
psql \
-X \
$POSTGRES_CONNECTION_STRING \
-w \
-b \
-L ./output.txt \
-A \
-q \
--set ON_ERROR_STOP=on \
-t <<EOF
\copy mytable(...) from '$input_file' csv HEADER
EOF
echo "import is done"
A very simple question:
I try to use a bash script to submit spark jobs. But somehow it keeps complaining that it cannot find spark-submit command.
But when I just copy out the command and run directly in my terminal, it runs fine.
My shell is fish shell, here's what I have in my fish shell config: ~/.config/fish/config.fish:
alias spark-submit='/Users/MY_NAME/Downloads/spark-2.0.2-bin-hadoop2.7/bin/spark-submit'
Here's my bash script:
#!/usr/bin/env bash
SUBMIT_COMMAND="HADOOP_USER_NAME=hdfs spark-submit \
--master $MASTER \
--deploy-mode client \
--driver-memory $DRIVER_MEMORY \
--executor-memory $EXECUTOR_MEMORY \
--num-executors $NUM_EXECUTORS \
--executor-cores $EXECUTOR_CORES \
--conf spark.shuffle.compress=true \
--conf spark.network.timeout=2000s \
$DEBUG_PARAM \
--class com.fisher.coder.OfflineIndexer \
--verbose \
$JAR_PATH \
--local $LOCAL \
$SOLR_HOME \
--solrconfig 'resource:solrhome/' \
$ZK_QUORUM_PARAM \
--source $SOURCE \
--limit $LIMIT \
--sample $SAMPLE \
--dest $DEST \
--copysolrconfig \
--shards $SHARDS \
$S3_ZK_ZNODE_PARENT \
$S3_HBASE_ROOTDIR \
"
eval "$SUBMIT_COMMAND"
What I've tried:
I could run this command perfectly fine on my Mac OS X fish shell when I copy this command literally out and directly run.
However, what I wanted to achieve is to be able to run ./submit.sh -local which executes the above shell.
Any clues please?
You seem to be confused about what a fish alias is. When you run this:
alias spark-submit='/Users/MY_NAME/Downloads/spark-2.0.2-bin-hadoop2.7/bin/spark-submit'
You are actually doing this:
function spark-submit
/Users/MY_NAME/Downloads/spark-2.0.2-bin-hadoop2.7/bin/spark-submit $argv
end
That is, you are defining a fish function. Your bash script has no knowledge of that function. You need to either put that path in your $PATH variable or put a similar alias command in your bash script.
Make sure this command is added to path:
export PATH=$PATH:/Users/{your_own_path_where_spark_installed}/bin
For mac, open either one of these files ~/.bash, ~/.zprofile, ~/.zshrc and add the command below in the file.
I'm stuck for houres on this strange issue.
I have a bashscript which is executing the following:
TEST="12.x.x.x"
echo ${TEST} gave me 12.x.x.x
So now I want to use this env var in my command:
oadm ca create-server-cert --signer-cert=ca.crt \
--signer-key=ca.key --signer-serial=ca.serial.txt \
--hostnames='docker-registry.default.svc.cluster.local,$TEST' \
--cert=registry.crt --key=registry.key
An echo of this command shows the content of $TEST in it.
But the command fails (it did not create the crt and key for my IP).
But it works when I'm just executing:
oadm ca create-server-cert --signer-cert=ca.crt \
--signer-key=ca.key --signer-serial=ca.serial.txt \
--hostnames='docker-registry.default.svc.cluster.local,12.x.x.x' \
--cert=registry.crt --key=registry.key
What could be the issue?
An echo of $TEST gave always my IP. Before and after the command.
Single quotes prevent variable expansion. Try with double quotes:
oadm ca create-server-cert --signer-cert=ca.crt \
--signer-key=ca.key --signer-serial=ca.serial.txt \
--hostnames="docker-registry.default.svc.cluster.local,${TEST}" \
--cert=registry.crt --key=registry.key
The variable is not valid between single quotes'', you should use double quotes "" , like this:
--hostnames="docker-registry.default.svc.cluster.local,$TEST"
I'm trying to send a notification via pushover using curl in a bash script.
I cannot get curl -F to interpret the line break correctly though.
curl -s \
-F "token=TOKEN" \
-F "user=USER" \
-F "message=Root Shell Access on HOST \n `date` \n `who` " \
https://api.pushover.net/1/messages.json > NUL
I've tried:
\n
\\\n
%A0
I'd rather push the message out directly, not through a file.
curl doesn't interpret backslash escapes, so you have to insert an actual newline into the argument which curl sees. In other words, you have to get the shell (bash in this case) to interpret the \n, or you need to insert a real newline.
A Posix standard shell does not interpret C escapes like \n, although the standard utility command printf does. However, bash does provide a way to do it: in the quotation form $'...' C-style backslash escapes will be interpreter. Otherwise, $'...' acts just like '...', so that parameter and command substitutions do not take place.
However, any shell -- including bash -- allows newlines to appear inside quotes, and the newline is just passed through as-is. So you could write:
curl -s \
-F "token=$TOKEN" \
-F "user=$USER" \
-F "message=Root Shell Access on $HOST
$(date)
$(who)
" \
https://api.pushover.net/1/messages.json > /dev/null
(Note: I inserted parameter expansions where it seemed like they were missing from the original curl command and changed the deprecated backtick command substitutions to the recommended $(...) form.)
The only problem with including literal newlines, as above, is that it messes up indentation, if you care about appearances. So you might prefer bash's $'...' form:
curl -s \
-F "token=$TOKEN" \
-F "user=$USER" \
-F "message=Root Shell Access on $HOST"$'\n'"$(date)"$'\n'"$(who)" \
https://api.pushover.net/1/messages.json > /dev/null
That's also a little hard to read, but it is completely legal. The shell allows a single argument ("word") to be composed of any number of quoted or unquoted segments, as long as there is no whitespace between the segments. But you can avoid the multiple quote syntax by predefining a variable, which some people find more readable:
NL=$'\n'
curl -s \
-F "token=$TOKEN" \
-F "user=$USER" \
-F "message=Root Shell Access on $HOST$NL$(date)$NL$(who)" \
https://api.pushover.net/1/messages.json > /dev/null
Finally, you could use the standard utility printf, if you are more used to that style:
curl -s \
-F "token=$TOKEN" \
-F "user=$USER" \
-F "$(printf "message=Root Shell Access on %s\n%s\n%s\n" \
"$HOST" "$(date)" "$(who)")" \
https://api.pushover.net/1/messages.json > /dev/null