I am trying to load the content of a file to avoid showing sensitive data like passwords while invoking a command with -p during a makefile task execution.
The command works when run without variable replacement. I am trying to run a docker login to a docker registry, but in theory this should not be relevant, I mean: the approach below could be used with any command not just the docker command.
I think I am messing up with the combination of double-quotes " and back-quotes ` or something like that, but I can not find a way to get this working.
So far no luck with this makefile content as a combination of bash/gnu-make:
REGISTRY_PSW_FILE=./volume/conf/docker-registry.conf
REGISTRY_PSW=`cat $(REGISTRY_PSW_FILE)`
REGISTRY_LOGIN_COMMAND=docker login my-own-registry.something.com -u myRegistryUsr -p $(REGISTRY_PSW)
docker-push:
#echo -e "\n\n\nPushing the docker image to the registry"
#echo -e "stuff: $(REGISTRY_PSW)"
#echo -e "more stuff: $(REGISTRY_LOGIN_COMMAND)"
$(REGISTRY_LOGIN_COMMAND)
#echo "DONE"
I've tried different things also stuff like:
REGISTRY_LOGIN_COMMAND=`docker login my-own-registry.something.com -u myRegistryUsr -p $(REGISTRY_PSW)`
Or this one using directly the cat command:
REGISTRY_LOGIN_COMMAND=docker login my-own-registry.something.com -u myRegistryUsr -p `cat $(REGISTRY_PSW_FILE)`
But I keep getting different errors like:
flag needs an argument: 'p' in -p \n See 'docker login --help'.
Error response from daemon: login attempt to https://my-own-registry.something.com/v2 failed with status: 401 Unauthorized
In the makefile task I am using $(REGISTRY_LOGIN_COMMAND) straight away (and not a combination of eval + echo) because I am trying to avoid showing the password in the output of the command (like in the echo that shows more stuff: <THE THING I AM TRYING TO HIDE>), anyway even when using a combination of echo and eval like this #eval "echo -e $(REGISTRY_LOGIN_COMMAND)" it still does not work with similar output errors.
I am using cygwin with the following version of gnu-make:
$ make -v
GNU Make 4.2.1
Built for x86_64-unknown-cygwin
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
What am I doing wrong?
I see no way to prevent the user from seeing the docker command (and the password it contains), if Make runs the docker directly.
But if Make constructs a script that contains the docker command, and executes it, that's another story:
DOCKER_SCRIPT = run_docker
docker-push:
sed -e 's/^/docker login my-own-registry.something.com -u myRegistryUsr -p /' -e 's|$$| >/dev/null/|' $(REGISTRY_PSW_FILE) > $(DOCKER_SCRIPT)
chmod +x $(DOCKER_SCRIPT)
./$(DOCKER_SCRIPT)
Related
I am writing an automation script to download the aws cloud-hsm client and pcks for doing a aws-cloudhsm-client init-container for a vault enterprise deployment.
The goal is to automate the config and setup of the HSM integration for vault to reference.
This is a guide that details how to do it.
https://github.com/jacobmammoliti/aws-vault-cloudhsm
My issue is that the cloud-hsm cli provided with the cloud-hsm client doesn't have a auto yes feature for when you execute the change password command. In order to automate this I have it in a EOF block for inline script execution to use the cloud-hsm cli inside of my start up script to configure it.
The issue is I'm trying to use yes | to answer the prompt but I don't think the EOF inline script method supports that and I am trying to find another way around it because the cloud-hsm cli doesn't support it which is kinda silly.
Here is a test bash script I'm running from a ubuntu:18.04 shell inside of my Kubernetes cluster to workout the automation. The HSM is on a private network so I'm using a pod to be inside the HSM network.
apt update -y
apt-get install wget -y
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -ab
export PATH=~/miniconda3/bin:${PATH}
python --version
wget https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/Bionic/cloudhsm-client_latest_u18.04_amd64.deb
wget https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/Bionic/cloudhsm-client-pkcs11_latest_u18.04_amd64.deb
apt install -y ./cloudhsm-client_latest_u18.04_amd64.deb
apt install -y ./cloudhsm-client-pkcs11_latest_u18.04_amd64.deb
export CLOUD_HSM_IP='HSM_IP'
export CUSTOMER_CA="base64encodedca"
export VAULT_DEFAULT_ADMIN_PASSWORD='password'
export VAULT_HSM_ADMIN_PASSWORD='myadminpassword'
export VAULT_USER='vault'
export VAULT_HSM_PASSWORD='myadminpassword'
echo "Configure Cloud HSM $CLOUD_HSM_IP"
/opt/cloudhsm/bin/configure -a ${CLOUD_HSM_IP}
echo "Config File"
cat /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg
echo "Echo Customer CA"
echo "${CUSTOMER_CA}" | base64 --decode > /opt/cloudhsm/etc/customerCA.crt
echo "Cat Customer CA"
cat /opt/cloudhsm/etc/customerCA.crt
echo "Execute cloudhsm cli"
echo "
yes | /opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg <<'EOF'
enable_e2e
loginHSM PRECO admin ${VAULT_DEFAULT_ADMIN_PASSWORD}
changePswd PRECO admin ${VAULT_HSM_ADMIN_PASSWORD}
logoutHSM
loginHSM CO admin ${VAULT_HSM_ADMIN_PASSWORD}
createUser CU ${VAULT_USER} ${VAULT_HSM_PASSWORD}
logoutHSM
quit
EOF
" > configure_hsm
cat configure_hsm
yes | bash configure_hsm
service cloudhsm-client start
Question:
How can I get around this issue because yes | doesn't work because cloud-hsm is its own cli?
Regardless of which shell you are using, a subprocess can only receive standard input from one place.
Anything which looks like
something | binary <<eof
...
eof
is basically an ambiguous redirect. Should binary receive standard input from the pipe, or from the here document? It can't be reading both.
(Quick testing on Bash and Alpine Linux indicates that these shells prefer the here document in this situation, and ignore the pipe. I'm too lazy to check if this is defined somewhere like POSIX.)
From a brief read of the documentation I'm guessing you want to respond y to the "are you sure?" prompts after several of these commands. Just add that in the here document just like all the other input.
/opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg <<'EOF'
enable_e2e
loginHSM PRECO admin ${VAULT_DEFAULT_ADMIN_PASSWORD}
changePswd PRECO admin ${VAULT_HSM_ADMIN_PASSWORD}
y
logoutHSM
loginHSM CO admin ${VAULT_HSM_ADMIN_PASSWORD}
createUser CU ${VAULT_USER} ${VAULT_HSM_PASSWORD}
y
logoutHSM
quit
EOF
For what it's worth, quoting the 'EOF' marker will prevent the shell from expanding the variables in the here document; I'm guessing probably you would not want that?
I have a super dumb script
$ cat script.sh
cat <<EOT > entrypoint.sh
#!/bin/bash
echo "$#"
EOT
docker run -it --rm -v $(pwd)/entrypoint.sh:/root/entrypoint.sh --entrypoint /root/entrypoint.sh bash:4 Hello World
But when I run script I got strange error:
$ sh script.sh
standard_init_linux.go:207: exec user process caused "no such file or directory"
Why script does not print Hello world ?
standard_init_linux.go:207: exec user process caused "no such file or directory"
The above error means one of:
Your script actually doesn't exist. This isn't likely with your volume mount but it doesn't hurt to run the container without the entrypoint, just open a shell with the same volume mount and list the file to be sure it's there. It's possible for the volume mount to fail on desktop versions of docker where the directory isn't shared to the docker VM and you end up with empty folders being created inside the container instead of mounting your file. When checking from inside of another container, also make sure you have execute permissions on the script.
If it's a script, the first line pointing to the interpreter is invalid. Make sure that command exists inside the container. E.g. alpine containers typically do not ship with bash and you need to use /bin/sh instead. This is the most common issue that I see.
If it's a script, similar to above, make sure your first line has linux linefeeds. A windows linefeed adds and extra \r to the name of the command trying to be run, which won't be found on the linux side.
If the command is a binary, it can refer to a missing library. I often see this with "statically" compiled go binaries that didn't have CGO disabled and have links to libc appear when importing networking libraries.
If you use json formatting to run your command, I often see this error with invalid json syntax. This doesn't apply to your use case, but may be helpful to others googling this issue.
This list is pulled from a talk I gave at last year's DockerCon: https://sudo-bmitch.github.io/presentations/dc2018/faq-stackoverflow.html#59
First of all:
Request
docker run -it --rm bash:4 which bash
Output
/usr/local/bin/bash
So
#!/bin/bash
Should be changed to
#!/usr/local/bin/bash
And
docker run -it --rm -v $(pwd)/entrypoint.sh:/root/entrypoint.sh --entrypoint /root/entrypoint.sh bash:4 Hello World
Gives you
Hello World
Update
Code
cat <<EOT > entrypoint.sh
#!/bin/bash
echo "$#"
EOT
Should be fixed as
#!/usr/bin/env bash
cat <<EOT > entrypoint.sh
#!/usr/bin/env bash
echo "\$#"
EOT
Did anyone succeed in setting up automated UIAutomation tests in Xcode?
I'm trying to set up a target in my Xcode project that should run all the UIAutomation scripts I prepared. Currently, the only Build Phase of this target is this Run Script block:
TEMPLATE="/Applications/Xcode.app/Contents/Applications/Instruments.app/Contents/PlugIns/AutomationInstrument.bundle/Contents/Resources/Automation.tracetemplate"
MY_APP="/Users/Me/Library/Application Support/iPhone Simulator/6.0/Applications/564ED15A-A435-422B-82C4-5AE7DBBC27DD/MyApp.app"
RESULTS="/Users/Me/Projects/MyApp/Tests/UI/Traces/Automation.trace"
SCRIPT="/Users/Me/Projects/MyApp/Tests/UI/SomeTest.js"
instruments -t $TEMPLATE $MY_APP -e UIASCRIPT $SCRIPT -e UIARESULTSPATH $RESULTS
When I build this target it succeeds after a few seconds, but the script didn't actually run. In the build log I get these errors:
instruments[7222:707] Failed to load Mobile Device Locator plugin
instruments[7222:707] Failed to load Simulator Local Device Locator plugin
instruments[7222:707] Automation Instrument ran into an exception while trying to run the script. UIATargetHasGoneAWOLException
+0000 Fail: An error occurred while trying to run the script.
Instruments Trace Complete (Duration : 1.077379s; Output : /Users/Me/Projects/MyApp/Tests/UI/Traces/Automation.trace)
I am pretty sure, that my javascript and my run script are both correct, because if I run the exact same instruments command in bash it works as expected.
Could this be a bug in Xcode?
I finally found a solution for this problem. It seems like Xcode is running the Run Scripts with limited rights. I'm not entirely sure, what causes the instruments command to fail, but using su to change to your user will fix it.
su $USER -l -c <instruments command>
Obviously, this will ask you for your password, but you can't enter it when running as a script. I didn't find a way to specify the password for su, however if you run it as root, you don't have to specify one. Luckily sudo can accept a password via the pipe:
echo <password> | sudo -S su $USER -l -c <instruments command>
If you don't want to hardcode your password (always a bad idea), you could use some AppleScript to ask for the password.
I posted the resulting script below. Copy that to a *.sh file in your project and run that script from a Run Script.
#!/bin/bash
# This script should run all (currently only one) tests, independently from
# where it is called from (terminal, or Xcode Run Script).
# REQUIREMENTS: This script has to be located in the same folder as all the
# UIAutomation tests. Additionally, a *.tracetemplate file has to be present
# in the same folder. This can be created with Instruments (Save as template...)
# The following variables have to be configured:
EXECUTABLE="TestApp.app"
# Optional. If not set, you will be prompted for the password.
#PASSWORD="password"
# Find the test folder (this script has to be located in the same folder).
ROOT="$( cd -P "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Prepare all the required args for instruments.
TEMPLATE=`find $ROOT -name '*.tracetemplate'`
EXECUTABLE=`find ~/Library/Application\ Support/iPhone\ Simulator | grep "${EXECUTABLE}$"`
SCRIPTS=`find $ROOT -name '*.js'`
# Prepare traces folder
TRACES="${ROOT}/Traces/`date +%Y-%m-%d_%H-%M-%S`"
mkdir -p "$TRACES"
# Get the name of the user we should use to run Instruments.
# Currently this is done, by getting the owner of the folder containing this script.
USERNAME=`ls -l "${ROOT}/.." | grep \`basename "$ROOT"\` | awk '{print $3}'`
# Bring simulator window to front. Depending on the localization, the name is different.
osascript -e 'try
tell application "iOS Simulator" to activate
on error
tell application "iOS-Simulator" to activate
end try'
# Prepare an Apple Script that promts for the password.
PASS_SCRIPT="tell application \"System Events\"
activate
display dialog \"Password for user $USER:\" default answer \"\" with hidden answer
text returned of the result
end tell"
# If the password is not set directly in this script, show the password prompt window.
if [ -z "$PASSWORD" ]; then
PASSWORD=`osascript -e "$PASS_SCRIPT"`
fi
# Run all the tests.
for SCRIPT in $SCRIPTS; do
echo -e "\nRunning test script $SCRIPT"
COMMAND="instruments -t \"$TEMPLATE\" \"$EXECUTABLE\" -e UIASCRIPT \"$SCRIPT\""
COMMAND="echo '$PASSWORD' | sudo -S su $USER -l -c '$COMMAND'"
echo "$COMMAND"
eval $COMMAND > results.log
SCRIPTNAME=`basename "$SCRIPT"`
TRACENAME=`echo "$SCRIPTNAME" | sed 's_\.js$_.trace_g'`
mv *.trace "${TRACES}/${TRACENAME}"
if [ `grep " Fail: " results.log | wc -l` -gt 0 ]; then
echo "Test ${SCRIPTNAME} failed. See trace for details."
open "${TRACES}/${TRACENAME}"
exit 1
break
fi
done
rm results.log
It seems as though this really might be an Xcode problem; at any rate, at least one person has filed a Radar report on it. Someone in this other thread claims you can work around this exception by disconnecting any iDevices that are currently connected to the computer, but I suspect that does not apply when you're trying to run the script as an Xcode target.
I would suggest filing a Radar report as well; you may get further details on the issue from Apple, or at least convince them that many people are having the problem and they ought to figure out what's going on.
Sorry for a not-terribly-helpful answer (should have been a comment, but comments and links/formatting do not mix very well). Please update this question with anything you find out on the issue.
Note: this is not a direct answer to the question, but it is an alternative solution to the underlying problem.
While searching for in-depth information about UIAutomation, I stumbled across a framework by Square called KIF (Keep it functional). It is a integration testing framework that allows for many of the same features as UIAutomation, but the great thing about is is that you can just write your integration tests in Objective-C.
It is very easy to setup (via CocoaPods), they have good examples too, and the best thing is that it's a breeze to set up with your CI system like Jenkins.
Have a look at: http://github.com/square/KIF
Late to the game but I have a solution that works for Xcode 5.1. Don't know if that's what broke the above solution or not. With the old solution I was still getting:
Failed to load Mobile Device Locator plugin, etc.
However, this works for the release version of Xcode 5.1.
echo <password> | sudo -S -u username xcrun instruments
Notice I removed the unneeded su command and added the xcrun command. The xcrun was the magic that was needed.
Here is my complete command:
echo <password> | sudo -S -u username xcrun instruments\
-w "iPhone Retina (3.5-inch) - Simulator - iOS 7.1"\
-D "${PROJECT_DIR}/TestResults/Traces/Traces.trace"\
-t "${DEVELOPER_DIR}/Instruments.app/Contents/PlugIns/AutomationInstrument.bundle/Contents/Resources/Automation.tracetemplate"\
"${BUILT_PRODUCTS_DIR}/MyApp.app"\
-e UIARESULTSPATH "${PROJECT_DIR}/TestResults"\
-e UIASCRIPT "${PROJECT_DIR}/UITests/main.js"
By the way if you type:
instruments -s devices
you will get a list of all the supported devices you can use for the -w option.
Edit: To make this work for different people checking out the project replace the following:
echo <password> | sudo -S -u username xcrun instruments
with
sudo -u ${USER} xcrun instruments
Since you are just doing an sudo to the same user no password is required.
Take a look at this tutorial that explains how to have Automated UI testing with Jenkins. It also uses Jasmine in the tutorial though. http://shaune.com.au/automated-ui-testing-for-ios-apps-uiautomation-jasmine-jenkins/ hope this helps. It has an example project file so you can download that as a template. Hope this helps.
In XCode - if you load up organizer (XCode->Window->Organizer)
Then select your machine under devices -> 'Enable Developer Mode'
This should remove the need for prompts with instruments.
I created an encrypted shell script with the tool shc.
The script works just fine on my computer but when I transfer it to
another one (solaris 10 to solaris 10) I get the following error:
invalid argument
It's not a permission problem and the encrypted script should be ok I guess it's a header/compiler problem.
The shc command used wasshc -rf <filename> so the script should work on another computer!?
According to The Geek Stuff you need to use the -r option to relax security and -f to specify your script file:
shc -r -f script.sh
After designing a simple shell/bash based backup script on my Ubuntu engine and making it work, I've uploaded it to my Debian server, which outputs a number of errors while executing it.
What can I do to turn on "error handling" in my Ubuntu machine to make it easier to debug?
ssh into the server
run the script by hand with either -v or -x or both
try to duplicate the user, group, and environment of the error run in your terminal window If necessary, run the program with something like "su -c 'sh -v script' otheruser
You might also want to pipe the result of the bad command, particularly if run by cron(8), into /bin/logger, perhaps something like:
sh -v -x badscript 2>&1 | /bin/logger -t badscript
and then go look at /var/log/messages.
Bash lets you turn on debugging selectively, or completely with the set command. Here is a good reference on how to debug bash scripts.
The command set -x will turn on debugging anywhere in your script. Likewise, set +x will turn it off again. This is useful if you only want to see debug output from parts of your script.
Change your shebang line to include the trace option:
#!/bin/bash -x
You can also have Bash scan the file for errors without running it:
$ bash -n scriptname