I've looked around trying to figure out a way to get a Packer build to download a private repository for an ec2 ami build that will be used for spinning up new instances under an auto-launch configuration, having a newly created ec2 instance grab a private repo.
It seems the most secure way to grab a private repo is to use a deploy key, but I believe I would have to manually add each deploy key to the repo for each instance… which defeats the automation purpose, unless I'm doing something wrong.
I'm wondering how to clone a private repository through packer, be it through a shell script or other wise. I understand I can use Chef, but I don't think I should have to install another dependency when the only thing I'm trying to do is clone a github repository.
Do I have to write a send/expect type of script that uses the https github clone url?
Any and all help appreciated.
There's a "workaround" using ssh-agent. I say workaround because it's not particularly elegant. It would be better to have this part of a Puppet module (maybe there's one already).
The idea is that you need to generate a pair of Public/Private key for each of your private Github repository. Then you add the public key as a Deploy key in the Github project settings (Settings/Deploy Keys). Where you store the key pair is up to you.
Now in Packer, you could use a Shell provisioner and execute something along these lines:
#!/usr/bin/env bash
ssh-keyscan github.com >> /home/ec2-user/.ssh/known_hosts
eval `ssh-agent`
ssh-agent bash -c \
'ssh-add /home/ec2-user/.ssh/[privateKey]; git clone git#github.com:[account]/project.git'
The advantage with this approach is that you can clone multiple private repositories easily.
There are a few ways to upload your key pair on the EC2 box, either by using a file provisioner , Chef or Puppet.
I've been using this for Windows but in theory should be similar to this
{
"type": "powershell",
"inline": [
"Set-Location C:\\{{ user `LAB_ENVIRONMENT` }}; git clone https://{{ user `GITLAB_USERNAME` }}:{{ user `GITLAB_ACCESS_TOKEN` }}#gitlab.com/{{ user `REPOSITORY` }}"
]},
Related
I followed this and added the keys in SSH Permissions as well as the fingerprint in my circle config file.
I also added this to my ~/.gitconfig as part of my circle compile step.
[url "ssh://git#github.com/MYORGANIZATION/"]
insteadOf = https://github.com/MYORGANIZATION/
following the official recommendation
When I SSH into the circle image I can see the fingerprint being added using this command ssh-add -l -E md5. But there's no keys added in ~/.ssh/. I'd expect to have ~/.ssh/id_rsa_<fingerprint> in there.
However I still get access denied when I try to retrieve the package.
The easiest way to get this to work is to follow the instructions for adding a machine user: https://circleci.com/docs/2.0/gh-bb-integration/#enable-your-project-to-check-out-additional-private-repositories
For a more complicated solution, read on.
I recently attempted the same thing. The add_ssh_keys keys should (and did in my case) add the id_rsa_<fingerprint> file.
The problem I ran into was that the key is added with an ssh config that contains:
Host !github.com *
I believe the problem is that it uses the default CircleCI key to authenticate with github. That key is valid, so github accepts it, but it most likely does not have access to the private repo in your dependencies.
To get it to work what I had to do was:
# Disable ssh-agent which seemed to override `-i`
export SSH_AUTH_SOCK=none
# Tell get to ssh with the key I want to use
export GIT_SSH_COMMAND="ssh -i /root/.ssh/id_rsa_FINGERPRINT
# Run some command to pull dependencies
go test ./...
What is the best way to pull a playbook down from bitbucket and execute it using ansible?
description:
I wrote a playbook and have it checked into bitbucket. I have docker which is spinning up ansible(image) and I need it to pull down the playbook and run a command against it. Anyone have experience with this?? Any help would be appreciated.
You can use ansible-pull http://docs.ansible.com/ansible/playbooks_intro.html or just do a regular git checkout and then run the code. Both work well with a public repo but will require git credentials if the repo is private. If you want to do this with a private repo you will want some form of secrets management for storing a token or ssh key to authenticate with.
Alternatively, at build time you can copy the files you need into the Docker image from your dev machine or CI tool and run the playbook, instead of doing that at runtime. This will decrease the boot time of your docker image as well.
You can simply do:
# clone repo
git clone https://bitbucket.org/user/repo.git && cd repo
docker run --rm=true -v `pwd`:/repo:rw ansible/ubuntu14.04-ansible /bin/bash -c "ansible-playbook -i /repo/hosts /repo/main.yml"
PS:
I am using with Travis CI such a way: https://github.com/weldpua2008/ansible-pycharm/blob/master/.travis.yml
I'm using ruby-git to programmatically perform some operations git repos hosted on GitHub.
Everything works fine when I do it on public repos however, I'm not able to clone private repos even if I have created an ssh key on the machine when the code runs and added the public one on GitHub. To make sure the key is properly setup I have cloned the repo directly from the terminal and the repo was successfully cloned. The key has also been added to the ssh-agent and $SSH_AUTH_SOCK is set.
So, I think the issue is related to how I use the ruby-git gem.
Here my (simple) code
#repo = Git.clone("git#github.com:#{repo_full_name}.git", path)
And here, if it may be helpful teh error
Git::GitExecuteError: git clone '--' 'git#github.com:USER/REPO.git' '/var/www/repo-root/USER/REPO' 2>&1:Cloning into '/var/www/repo-root/USER/REPO'... Permission denied (publickey). fatal: Could not read
Am I missing anything here?
Thanks
Do you have your Git env config pointing at your ssh key path?
Git.configure do |config|
# If you want to use a custom git binary
config.binary_path = '/git/bin/path'
# If you need to use a custom SSH script
config.git_ssh = '/path/to/ssh/script'
end
I'm try to update my private repository via a cron job but nothing seems to be happening. I'm using Satis to create the repository for my private packages.
I can do this manually by logging into my account via SSH and running:
php bin/satis build satis.json ./ -n
which updates everything fine apart from I have to enter my passphrase a million times. I can get round this by using SSH Agent and think this may be my problem...
php /home/accountname/public_html/bin/satis build /home/accountname/public_html/satis.json /home/accountname/public_html/ -n
Is there anything I'm missing?
UPDATE
It is the SSH auth as I've received this error via email (shortened version)
Reading composer.json of vendor/package
(master)
Failed to update
git#bitbucket.org:vendor/package.git, package information from this
repository may be outdated (Permission denied (publickey). fatal: The
remote end hung up unexpectedly error: Could not fetch origin )
In the docs it says about using -n to use the SSH Key but I am using it.
Any ideas?
UPDATE
Thought I would add my satis file structure:
{
"name": "Name Of My Repo",
"homepage": "http://repodomain.co.uk",
"repositories": [
{ "type": "vcs", "url": "git#bitbucket.org:vendor/package.git" }
],
"require-all": true
}
really stuck on this one, the documentation is crap!
You are right that the SSH authentication is the problem here.
When the cronjob is running, it must access your Bitbucket repository somehow. You chose to use the "git" protocol, which uses SSH. When using SSH, using key-based authentication is way better than using passwords - and when it comes to git repos, it is the only way most of the time depending on the hosting.
Either you put your private key onto that cron server to get access to Bitbucket (might be a bad idea), or you create a new key pair and use that to allow access for your repo (is better, unless you experience some limitations of your bitbucket account, like too many users if that key counts as a user - on the other hand you could limit this key to be only allowed to read, not write).
Make sure that the user that is used to run your cronjob is using these keys, e.g. you should be able to manually start the script without any agent, and it should finish without asking for passwords. The correct place for the private key is ~/.ssh, the public key goes to Bitbucket. After that everything should run perfectly in the cronjob.
The other way could be to use a different protocol (like HTTPS) for the repo access and see what happens.
I have tried following how-set-up-your-own-private-git-server-linux and private-remote-git-repositories-ubuntu-linode but I am still having problems.
My local environment is windows and my remote linux. I have a couple of questions:
In the first article it describes setting up a public ssh key for the server - I've never done this before and I'm not sure where to type the commands (not even sure whether local or remote!!): ssh myuser#server.com mkdir .ssh and scp ~/.ssh/id_rsa.pub myuser#server.com:.ssh/authorized_keys
On my local dev machine should I be running msysgit? Is there an alternative because the version I installed is taking up 1.4 GB! I installed msysGit-netinstall-1.7.4-preview20110204.exe from http://code.google.com/p/msysgit/downloads/list
I tried skipping making the git user and public key, created the repositories on the remote machine but then when I try git remote add origin ssh://[username]#[domain/ip/hostname]/srv/git/[project-name] as root user it says: fatal: Not a git repository (or any of the parent directories): .git
I'm not sure if this should be here or if it would be best migrated over to a different site, but since I might be able to help I'll go ahead and answer.
I just skimmed the articles you linked. It looks like they both deal with accessing a git server over ssh, which you mention, so that's what I'll focus on.
First, on your server:
You need to set up an account on the server so that you can log in. This can be either a generic git account, or your own personal account. For the moment we'll assume that you are setting it up to work with a personal account. What you want to do is create your account, then somewhere accessible to that account (say, in your home directory), create the git repository.
mkdir myrepo.git
cd myrepo.git
git --bare init --shared=all
So now you have a git repository up on the server. Depending on the git client you are using, you might not need to mess with keys right now. If SSH is configured on your server to allow password login then you can probably just connect and enter your password when you need to interact with the server. If you want to set up keys, what you need to do is to generate an ssh public key. I don't know off hand how to do this in windows, but in linux you'd do something like:
ssh-keygen -t rsa -b 1024
That command will generate two files, "id_rsa" and "id_rsa.pub"; whatever tool you use should also generate two files, a public key and a private key. They might have different names, but assume for now that "id_rsa.pub" is the name of your public key file.
You want to copy the public key to the server, you can use scp, ftp, or just move it over using a thumbdrive. Either way, once you get it onto the server, and it's accessible by your user, log in as your user on the server. You want to add the public key to your "authorized_hosts" file, so after logging in to your account on the server, do this:
cd
mkdir .ssh
cat id_rsa.pub >> .ssh/authorized_hosts
rm id_rsa.pub
Now, from your workstation you need to configure your ssh client to use the private key your generated. Again, I don't know how to do this on Windows and it will probably vary depending on what ssh client you are using, so you'll need to get that information somewhere else.
Next, you need to create your local repository, add some files, and make a commit. Note that you can't clone the remote repository you made yet because there's nothing in there. Once you have some commits made locally, you need to set the remote server in your repository.
If you are using the git command line tools, you can run:
git remote add origin user#yourserver:myrepo.git
If you put the repository somewhere other than your home directory, use the full path:
git remote add origin user#yourserver:/path/to/repo.git
Note that you need the ".git" in there, since your directory name has ".git" as part of the name.
If you are using a GUI tool, then you instead just edit the configuration file for the repository. This will be in the top level of your repository under ".git/config". You'll want to add something like
[remote "origin"]
url = user#yourserver:/path/to/repo.git
fetch = +refs/heads/*:refs/remotes/origin/*
Now that your remote is configured, and you have some commits locally, you can push your master branch up to the server. If you're using the command line use:
git push origin master
Or, if you're working on a different branch:
git push origin mybranch
If you are using a GUI frontend for get then you'll need to look up the documentation for that tool on how to push.
I would like to recommend to use Gitolite that allows to setup git hosting on central srever with great features like access control and creating/managin repositories etc.
And for windows, the tool called git-scm can be used for setting up git client and also have GUI panel for git repositories. After installation of this git tool, you will get a git bash tool and a git GUI.
Gitolite makes settting up of a git hosting server and client very simple.
I hope this may help new people who are looking for an efficient solution for settting up git hosting server and clients in some easy to understand simple steps.
I followed the most voted answer and I didnt get it to work. I found this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2
and I got it working. I think the problem was that, in the server, the public key is to be copied to a file "authorized_keys" and not "authorized_hosts".
I had been trying to generate the key pairs using putty but couldn't get it to work. In the end I realised that I could generate the keys via the msysgit command line using ssh-keygen -C "git#example.com" -t rsa. Copy the public key to the server using nano /home/git/.ssh/authorized_keys and hey presto!
I had downloaded the full source code hence the size! The non-source app, Git-1.7.4-preview20110204.exe was just 13MB via http://code.google.com/p/msysgit/downloads/list?can=3