I'm on ubuntu 15.04, my version of ssh client is
OpenSSH_6.9p1 Ubuntu-2ubuntu0.2, OpenSSL 1.0.2d 9 Jul 2015
When I try to run the following command ssh admin#server bash -c 'cd /path/to/repo && git pull' the cd is not effective and i got
fatal: Not a git repository (or any of the parent directories): .git
However If I do
ssh admin#server bash -c 'echo test && cd /path/to/repo && git pull'
then it works
Already up-to-date.
Of course I'm well aware echo is not supposed to change anything but after trying several time, several days on several different servers (though all on debian) I'm now sure to have this error.
On other servers I tried the command cd /tmp && pwd , and I got my home directory, and if i do echo toto && /tmp && pwd I go /tmp printed...
Unfortunately, ssh passes through a single command line string to $SHELL -c on the remote. Your quotes aren't being effective.
When you run
ssh admin#server bash -c 'cd /path/to/repo && git pull'
this is being run on the remote server (with $SHELL -c):
bash -c cd /path/to/repo && git pull
So Bash is given single command (cd) and an unused argument, and then separately, you're also running git pull in the home directory.
On the other hand, when you run
ssh admin#server bash -c 'echo test && cd /path/to/repo && git pull'
this is being run on the remote server:
bash -c echo test && cd /path/to/repo && git pull
The first part is again useless, but the shell running the whole command then does cd /path/to/repo and git pull. Which works.
What you probably want to do is
ssh admin#server 'cd /path/to/repo && git pull'
The existing answer by ephemient is entirely correct in terms of cause.
To add an alternate solution -- one which works when your remote code contains constructs which sh -c will misinterpret -- consider:
repo=/path/to/repo ## here, this works even when your path contains
## nonprintable or otherwise surprising characters
printf -v repo_q '%q' "$repo" ## ...because we're asking your local copy of bash
## to generate a quoted/escaped copy of the value
## that will 'eval' back to its original meaning
## when interpreted by bash
## to ensure that it's interpreted by bash, we pass 'bash -s' as the command to ssh
## with an *unquoted* heredoc (<<EOF, vs <<'EOF'), with the escaped value expanded
ssh admin#server 'bash -s' <<EOF
cd $repo_q && git pull
EOF
Related
Bash script:
clonePath=/data/config/
git branch -r | fgrep -v 'origin/HEAD' | sed 's| origin/|git checkout |' > checkoutAllBranches.sh
chmod +x checkoutAllBranches.sh
echo "Fetch branch: `cat checkoutAllBranches.sh`"
./checkoutAllBranches.sh
git checkout master
git remote rm origin
rm checkoutAllBranches.sh
for config_dir in `ls -a`; do
cp -r $config_dir $clonePath/;
done
echo "API Config update complete..."
Dockerfile which issues this script execution
ENTRYPOINT ["sh","config-update-force.sh","|| true"]
The error below causes the container startup failure despite setting the command status to 0 manually using || true
ERROR:
Error:
cp: cannot create regular file '/data/./.git/objects/pack/pack-27a9d...fb5e368e4cf.pack': Permission denied
cp: cannot create regular file '/data/./.git/objects/pack/pack-27a9d...fbae25e368e4cf.idx': Permission denied
I am looking for 2 options here:
Change these file permissions and then store them in the remote with rwx permissions
Do something to the docker file to ignore this script failure error and start the container.
DOCKERFILE:
FROM docker.hub.com/java11-temurin:latest
USER root
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get install -y rsync telnet vim wget git
RUN mkdir -p /opt/config/clone/data
RUN chown -R 1001:1001 /opt/config
USER 1001
ADD build/libs/my-api-config-server.jar .
ADD config-update-force.sh .
USER root
RUN chmod +x config-update-force.sh
USER 1001
EXPOSE 8080
CMD java $BASE_JAVA_OPTS $JAVA_OPTS -jar my-api-config-server.jar
ENTRYPOINT ["sh","config-update-force.sh","|| true"]
BASH SCRIPT:
#!/bin/bash
set +e
set +x
clonePath=/opt/clone/data/data
#source Optumfile.properties
echo "properties loaded: example ${git_host}"
if [ -d my-api-config ]; then
rm -rf my-api-config;
echo "existing my-api-config dir deleted..."
fi
git_url=https://github.com/my-api-config-server
git clone https://github.com/my-api-config-server
cd my-api-config-server
git branch -r | fgrep -v 'origin/HEAD' | sed 's| origin/|git checkout |' > checkoutAllBranches.sh
chmod +x checkoutAllBranches.sh
echo "Fetch branch: `cat checkoutAllBranches.sh`"
./checkoutAllBranches.sh
git checkout master
git remote rm origin
rm checkoutAllBranches.sh
for config_dir in `ls -a`; do
cp -r $config_dir $clonePath/;
done
echo "My API Config update complete..."
When you do in the script...
chmod +x checkoutAllBranches.sh
...than why not before cp
chmod -R +rwx ${clonePath}
...or if the stderr message 'wont impact anything'...
cp -r $config_dir $clonePath/ 2>/dev/null;
...even cp dont copy -verbosly.
?
When your Dockerfile declares an ENTRYPOINT, that command is the only thing the container does. If it also declares a CMD, the CMD is passed as additional arguments to the ENTRYPOINT; it is not run on its own unless the ENTRYPOINT makes sure to execute it.
Shell errors are not normally fatal, and especially if you explicitly set +e, even if a shell command fails the shell script will keep running. You see this in your output where you get multiple cp errors; the first error does not terminate the script.
You need to do two things here. The first is to set the ENTRYPOINT to actually run the CMD; the simplest and most common way to do this is to end the script with
exec "$#"
The second is to remove the || true from the Dockerfile. As you have it written out currently, this is passed as the first argument to the entrypoint wrapper – it is not run through a shell and it is not interpreted as a "or" operator. If your script begins with a "shebang" line and is marked executable (both of these are correct in the question) the you do not explicitly need the sh interpreter.
# must be a JSON array; no additional "|| true" argument; no sh -c wrapper
ENTRYPOINT ["./config-update-force.sh"]
# any valid CMD will work with `exec "$#"
CMD java $BASE_JAVA_OPTS $JAVA_OPTS -jar my-api-config-server.jar
I'm trying to delete the contents of a remote directory in a bash script and leaving the folder intact by using ssh like this:
# First attempt
inboxResult=$(ssh -t -t username#host sudo -u rootUser rm -Rf /my/path/here/inbox/*)
# Second attempt
inboxResult=`ssh -t -t username#host sudo -u rootUser rm -Rf /my/path/here/inbox/*`
but it keeps failing silently. I've done my research and it seems like the '*' is being expanded before the command is sent via ssh to the remote host, but I would want the opposite. I couldn't find any solution and I've tried more than these two but they seem to be far from what I was looking.
I want to execute cd and scp commands on a remote server which have to be logged in with a different sudo user. Below code snippet asks for the password(echos on screen) for my user but hangs there. It doesn't execute cd
#!/bin/bash
server=myserver.com
ssh $server 'sudo -S -u <user> -i; cd dir1/dir2/; scp file1 user#local-sever'
The issue is that you have a semi colon before cd and so sudo has no command to execute. Remove the ; and it should work:
ssh $server 'sudo -S -u <user> -i scp dir1/dir2/file1 user#local-sever'
There are several ways to address this, but most boil down to wrapping up the commands into a set of instructions. Raman's solution is good since it handles the issue by using full paths, but sometimes that isn't an option. Here's another take -
Assuming your command list can afford the quotes, I like here-strings.
ssh -t sa-nextgen-jenkins.eng.rr.com <<< "
echo 'set -x; cd /tmp; whoami; touch foo; ls -l foo; rm -f foo;'|sudo -iSu user
"
If you need the quotes, try a here-doc.
ssh -t sa-nextgen-jenkins.eng.rr.com <<END
echo 'set -x; echo "$RANDOM"; cd /tmp; whoami; touch foo; ls -l foo; rm -f foo;'|sudo -iSu $user
END
You can also write a small script that has arbitrarily complex commands and scp it over, then use a remote ssh call to execute it as the relevant user.
I'm trying to write a script that's part of a much bigger automation script that configures the GitHub ssh key on a local dev machine.
This is the line I'm trying to run but for some reason the 'eval $(ssh-agent -s)' fails as it errors and outputs this message.
'eval' is not recognized as an internal or external command,
operable program or batch file.
cmd.exe /c "ssh-keygen -t rsa -b 4096 -C "$githubEmailAddress" && eval $(ssh-agent -s) && ssh-add ~/.ssh/id_rsa && clip < ~/.ssh/id_rsa.pub"
I have looked around and I'm having no luck getting past this issue. I can't work out how to launch the Git Bash terminal where the command works from the ps1 script.
You should consider creating an alias for bash.exe that way it is only referenced one time in the script and will be easier to change if you need to in the future.
You can then create the SSH key as shown:
New-Alias -Name gitBash -Value "$Env:ProgramFiles\Git\bin\bash.exe
gitBash -c "ssh-keygen -t rsa -b 4096 -C "blah#gmail.com" && eval $(ssh-agent -s) && ssh-add ~/.ssh/id_rsa && clip < ~/.ssh/id_rsa.pub && exit"
You also won't have to do as much encoding as the key gen parameters won't need to be wrapped in a string delimiter anymore.
You also avoid managing the current directory manually like you are with the cd command.
Also note the use of the $Env:ProgramFiles to get the base path of the program files directory, its not common for it to be configured differently than "C:\Program Files" but it can be and it avoids issues with spaces in the path name this way.
After a lot of trial and error the way you can open a git bash prompt from powershell is to find the bin\bash.exe file normally found in a folder called Git within the Program Files folder.
From the location of the ps1 file you might need to modify the ....\ part but the script below allow you to open a bash prompt execture a bunch of calls and then exit the prompt
cmd.exe /c 'cd "..\..\Program Files\Git\bin" && bash.exe -c "ssh-keygen -t rsa -b 4096 -C "blah#gmail.com" && eval $(ssh-agent -s) && ssh-add ~/.ssh/id_rsa && clip < ~/.ssh/id_rsa.pub && exit"'
I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.
.gitlab.ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master # Run this job only on changes for stage branch
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "${shell} git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
Error: .gitlab-deploy-prod.sh: line 7: -o: command not found
How can i check if directory existing??
What i've tried.
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"
if [-d "/flask-ci-cd" ] # check if directory exists
then
eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.
Your script has some errors.
Do not use eval. No, eval does not work that way. eval is evil
When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
Take your time and research how quoting and word splitting works in shell.
git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
Read documentation for the utilities you use.
No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.
How can i check if directory existing??
Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[#]}" [ -d "/flask-ci-cd" ]; then
...
In case of directory name with spaces I would go with:
if "${shell[#]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then
Pass set -x to sh to see what's happening also on the remote side.
For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF
But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:
echo "Deploy project on server ${SERVER_URL}"
work() {
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:"${git_token}"#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token) # transfer variables you need
$(declare -f work) # transfer function you need
work # call the function.
EOF
Updated answer for future reads.
.gitlab-ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
echo "\n Cloning into remote repo..."
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git
# Create and activate virtualenv
echo "\n Creating virtual env"
python3 -m venv env
else
echo "Pulling remote repo origin..."
cd flask-ci-cd
git pull
cd ..
fi
# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate
# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF
There is a test command which is explicit about checking files and directories:
test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands
Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)