Chef, Vagrant and private git cloning - ruby

i have acces to git repo on host, and i have a Vagrantfile:
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu14.04"
config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64- vagrant-disk1.box"
config.vm.provider "virtualbox" do |vb|
vb.memory = 1024
vb.cpus = 2
end
config.ssh.forward_agent = true
config.vm.provision :chef_solo do |chef|
# chef.log_level = :debug
chef.cookbooks_path = "./cookbooks"
chef.add_recipe "git_sync"
end
end
if i run vagrant and ssh into it, i could also git clone my private repo, (recipe "install_pkgs" is to install git on vm) but the pecipe "git_sync" gets an error like:
[2015-05-08T18:40:26+00:00] ERROR: Running exception handlers
[2015-05-08T18:40:26+00:00] ERROR: Exception handlers complete
[2015-05-08T18:40:26+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-05-08T18:40:26+00:00] ERROR: git[/home/vagrant/geomongo] (git_sync::default line 1) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '128'
---- Begin output of git ls-remote "git#bitbucket.org:galiaf95/test.git" HEAD ----
STDOUT:
STDERR: Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
---- End output of git ls-remote "git#bitbucket.org:galiaf95/test.git" HEAD ----
Ran git ls-remote "git#bitbucket.org:galiaf95/test.git" HEAD returned 128
================================================================================
Error executing action `sync` on resource 'git[/home/vagrant/geomongo]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '128'
---- Begin output of git ls-remote "git#bitbucket.org:galiaf95/test.git" HEAD ----
STDOUT:
STDERR: Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
---- End output of git ls-remote "git#bitbucket.org:galiaf95/test.git" HEAD ----
Ran git ls-remote "git#bitbucket.org:galiaf95/test.git" HEAD returned 128
Resource Declaration:
---------------------
# In /tmp/vagrant-chef-1/chef-solo-1/cookbooks/git_sync/recipes/default.rb
1: git "/home/vagrant/geomongo" do
2: # repository "git#bitbucket.org:osll/geomongo.git"
3: # repository "https://github.com/galiaf95/test.git"
4: repository "git#bitbucket.org:galiaf95/test.git"
5: action :sync
6: end
Compiled Resource:
------------------
# Declared in /tmp/vagrant-chef-1/chef-solo-1/cookbooks/git_sync/recipes/default.rb:1:in `from_file'
git("/home/vagrant/geomongo") do
provider Chef::Provider::Git
action [:sync]
retries 0
retry_delay 2
destination "/home/vagrant/geomongo"
revision "HEAD"
remote "origin"
cookbook_name :git_sync
recipe_name "default"
repository "git#bitbucket.org:galiaf95/test.git"
end
[2015-05-08T18:38:31+00:00] INFO: Forking chef instance to converge...
[2015-05-08T18:40:26+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.
Here is my git_sync.rb recipe
git "/home/vagrant/geomongo" do"
repository "git#bitbucket.org:galiaf95/test.git"
action :sync
end
I'm new to chef and vagrant and it would be great to have some very comprehensive examples of how to clone private repo using chef.
Problem solved with this post https://stackoverflow.com/a/8191279/3564452 But can someone, please, discribe what's going on in this recipe and how this fixes my problem.

(I'm no chef expert, but I have a similar setup working with Salt...)
The issue is that when you log on, you set up the ssh known_host as the user "vagrant", and the SSH_AUTH_SOCK variable is set for that user too. When you run your Chef recipes in the provisioner, they run as root. So, you need to add your host to the /root/.ssh/known_hosts and also you need to edit /etc/sudoers to allow SSH_AUTH_SOCK to be passed through, something like so, although you'll obviously need to port my salt-call line to Chef.
config.vm.provision "setup-and-highstate", type: "shell" do |s|
s.inline = <<SCRIPT
grep -s SSH_AUTH_SOCK /etc/sudoers || echo 'Defaults env_keep="SSH_AUTH_SOCK"' | (EDITOR="tee -a" visudo)
ln -s /var/cache/salt/minion/extmods/outputters/ /var/cache/salt/minion/extmods/output
salt-call --local 'ssh.set_known_host' 'root' enc='ecdsa' fingerprint='ff:ff:ff:23:b4:20:93:d1:2e:91:ff:3c:a8:ff' hostname='git.xxxx.yyyy.com'
SCRIPT
end

With the error string Host key verification failed. to me it sounds like one of three things:
you didn't add your SSH keys to your BitBucket repository
the SSH key on your node is formatted incorrectly (a trailing space or newline for instance)
you didn't ssh-add your SSH key on your node
In regards to the HOSTS issue, you can use the ssh_known_hosts cookbook resource to add the git#bitbucket.org address prior to using the git resource.

Related

Packer error with build from vmx file - "file was not found"

I have a problem that I can't figure out. I'm trying to make my first clone of a VM from Packer (Hashicorp) here is my HCL :
source "vmware-vmx" "basic-example" {
source_path = "D:/HashiCorp/Packer1.8.0/Project/WindowsServer2022.vmx"
vm_name = "Test1"
communicator = "winrm"
winrm_host = "redteam"
winrm_username = "admin"
winrm_password = "Disc0very"
shutdown_command = "shutdown /s"
}
build {
sources = ["sources.vmware-vmx.basic-example"]
}
and when I run my build command I get this result :
D:\HashiCorp\Packer1.8.0\Project>packer build packer_test_win2022CARL.pkr.hcl
vmware-vmx.basic-example: output will be in this color.
==> vmware-vmx.basic-example: Configuring output and export directories...
==> vmware-vmx.basic-example: Creating required virtual machine disks
==> vmware-vmx.basic-example: Cloning source VM...
==> vmware-vmx.basic-example: Deleting output directory...
Build 'vmware-vmx.basic-example' errored after 521 milliseconds 605 microseconds: VMware error: Error: A file was not found
==> Wait completed after 521 milliseconds 605 microseconds
==> Some builds didn't complete successfully and had errors:
--> vmware-vmx.basic-example: VMware error: Error: A file was not found
==> Builds finished but no artifacts were created.
Solved, not enough to point the VMX file, VMware needs other files such as .vmdk, .vmxf etc ... to proceed with the cloning

Adding a sudoers file in chef fails at verification

goal
I'm trying to add/edit a sudoers file in Chef.
After a lot of serach (and broken sudoers) I found this question and the answer seemed to be exactly what I am after.
My cookbook
So in my chef I added the following visudo cookbook:
The recipe: ~/chef-repo/cookbook/visudo/recipes/allowUpgrade.rb
template '/etc/sudoers.d/allowUpgrade' do
cookbook 'visudo'
source 'allowUpgrade.erb'
owner'root'
group 'root'
mode '0440'
verify "visudo -c -f %{path}"
end
My template: ~/chef-repo/cookbooks/visudo/templates/allowUpgrade.erb
username ALL=(ALL) NOPASSWD: /usr/local/bin/upgrade
Template and verification works manually
When I put this line/file there manually using
sudo nano /etc/sudoers.d/allowUpgrade
(I know one shouldn't) and then verify it using
visudo -c -f /etc/sudoers.d/allowUpgrade
I get
/etc/sudoers.d/allowUpgrade: parsed OK
and it works meaning I can run
sudo upgrade
without beeing prompted for the sudo password.
Verification fails running Chef
However it is not working using Chef. I'm trying it first on the local machine using
sudo chef-client -z --runlist 'recipe[visudo::allowUpgrade]'
But I get this error
Error executing action `create` on resource 'template[/etc/sudoers.d/allowUpgrade]'
Chef::Exceptions::ValidationFailed
Why is the verification failing in chef? What am I doing wrong?
Here the complete error message
Recipe: visudo::allowUpgrade
* template[/etc/sudoers.d/allowUpgrade] action create[2017-12-07T08:24:50+01:00] INFO: Processing template[/etc/sudoers.d/allowUpgrade] action create (visudo:: allowUpgrade line 7)
================================================================================
Error executing action `create` on resource 'template[/etc/sudoers.d/allowUpgrade]'
================================================================================
Chef::Exceptions::ValidationFailed
----------------------------------
Proposed content for /etc/sudoers.d/allowUpgrade failed verification #<Chef::Resource::File::Verification:0x0000000004070c48>
Resource Declaration:
---------------------
# In /home/username/chef-repo/.chef/local-mode-cache/cache/cookbooks/visudo/recipes/allowUpgrade.rb
7: template '/etc/sudoers.d/allowUpgrade' do
8: owner'root'
9: group 'root'
10: mode '0440'
11: source 'allowUpgrade.erb'
12: verify 'visudo -c -f %{path}'
13: end
Compiled Resource:
------------------
# Declared in /home/username/chef-repo/.chef/local-mode-cache/cache/cookbooks/visudo/recipes/allowUpgrade.rb:7:in `from_file'
template("/etc/sudoers.d/allowUpgrade") do
action [:create]
default_guard_interpreter :default
source "allowUpgrade.erb"
declared_type :template
cookbook_name "visudo"
recipe_name "allowUpgrade"
owner "root"
group "root"
mode "0440"
verifications [#<Chef::Resource::File::Verification:0x0000000004070c48 #command_opts={},
#command="visudo -c -f %{path}", #block=nil, #parent_resource=<template[/etc/sudoers.d/allowUpgrade]
#name: "/etc/sudoers.d/allowUpgrade" #before: nil #params: {}
#provider: nil #allowed_actions: [:nothing, :create, :delete, :touch, :create_if_missing]
#action: [:create] #updated: false #updated_by_last_action: false
#source_line: "/home/username/chef-repo/.chef/local-mode-cache/cache/cookbooks/visudo/recipes/allowUpgrade.rb:7:in `from_file'"
#guard_interpreter: nil #default_guard_interpreter: :default
#elapsed_time: 0 #source: "allowUpgrade.erb" #cookbook: nil
#local: false #variables: {} #inline_helper_blocks: {}
#inline_helper_modules: [] #helper_modules: [] #declared_type: :template
#cookbook_name: "visudo" #recipe_name: "allowUpgrade" #owner: "root" #group: "root" #mode: "0440"
#verifications: [...] #path: "/etc/sudoers.d/allowUpgrade">>]
path "/etc/sudoers.d/allowUpgrade"
end
Update:
When I leave the verification out and just do
template '/etc/sudoers.d/allowUpgrade' do
cookbook 'visudo'
source 'allowUpgrade.erb'
owner 'root'
group 'root'
mode '0440'
verify { 1 == 1 }
end
The sudo is broken! In recovery mode and the root console I checked and it looks just the same as when I insert it manually (what works fine)?!
Thanks to the help of Tensibai here in the comments and the hint to lineendings I could finally solve this problem.
Indeed the issue was lineendings as noted in this ancient Issue
I generated the cookbooks, recipes and templates on an Ubuntu Server 16.04 but do all m editing on the repository in Brackets.io on Windows.
This made template (and other) files have CRLF instead of LF lineendings because Brackets seems to use automatically the lineendings of the OS it is running on. This ofcourse made the /etc/sudoers.d/allowUpgrade file brake the sudoers because it has to end in a new line.
After some research I found this was an old known Issue and could be solved by the Plug-In Newline.
After installing this Plug-In indeed I could see that the file had CRLF lineendings.
I switched it to LF thanks to the Plug-In by clicking on the CRLF. Now my cookbook runs as expected and I'm able to run
sudo upgrade
without beeng prompted for the password - meaning it works.

Chef LWRP recipe Errno::ENOENT: No such file or directory # dir_s_mkdir

I am using the chef community java cookbook to install java on CentOS 7.2. I have an LWRP recipe that is not working
I build up my install parameters via the java_ark section
op_sys = node['os']
# Used to get the required java update from the environment file
java_ver_update = node['java_ver']
# Logic for each OS
if op_sys == 'linux'
# Java_ark, which is used to define the correct install attributes for each OS type (win/linux)
install_dir = node['install_dir']
java_ark "jdk" do
url 'http://sv-dc01.sv.local/install_artifacts/java/oracle/JRE/jre-'+"#{java_ver_update}"+'-linux-x64.tar.gz'
app_home install_dir
owner 'root'
group 'wheel'
app_home_mode 774
action :install
end
# Set the folder permissions
execute "chown-dir" do
command "chmod -R 774 #{install_dir}"
action :run
end
end
Here is my environment file where I have set some node attributes to be called in the main recipe
name 'env_workstation_dubbo'
description "Environment Workstation Dubbo"
cookbook_versions({
"ohai" => "> 0.0.1",
"java" => "> 0.1.0",
"install_java" => "> 0.0.1"
})
$environment = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc) }
$override = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc) }
$override['java']['jdk_version'] = '8'
$override['java']['install_flavor'] = 'oracle'
$override['java']['oracle']['accept_oracle_download_terms'] = true
$override['java']['set_default'] = false
# Custom attributes/variables to be placed here
$override['java_ver'] = '8u77'
$override['install_dir'] = '/applications/'
default_attributes(Chef::Mixin::DeepMerge.merge($_default_environment, $environment))
override_attributes($override)
And here is what happens during the sudo chef-client run on the CentOS machine I am using for testing:
Starting Chef Client, version 12.16.42
resolving cookbooks for run list: ["install_java"]
Synchronizing Cookbooks:
- install_java (0.2.0)
- java (1.43.0)
- compat_resource (12.16.2)
- ohai (4.2.2)
- seven_zip (2.0.2)
- homebrew (2.1.2)
- apt (5.0.0)
- build-essential (7.0.2)
- windows (2.1.1)
- mingw (1.2.4)
- ark (2.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
[2016-12-12T11:04:24+13:00] WARN: Chef::Provider::AptRepository already exists! Cannot create deprecation class for LWRP provider apt_repository from cookbook apt
[2016-12-12T11:04:24+13:00] WARN: AptRepository already exists! Deprecation class overwrites Custom resource apt_repository from cookbook apt
Converging 8 resources
Recipe: install_java::default
* java_ark[jdk] action install
================================================================================
Error executing action `install` on resource 'java_ark[jdk]'
================================================================================
Errno::ENOENT
-------------
No such file or directory # dir_s_mkdir -
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/java/providers/ark.rb:116:in `block (2 levels) in class_from_file'
/var/chef/cache/cookbooks/java/providers/ark.rb:115:in `block in class_from_file'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/install_java/recipes/default.rb
18: java_ark "jdk" do
19: url 'http://sv-dc01.sv.local/install_artifacts/java/oracle/JRE/jre-'+"#{java_ver_update}"+'-linux-x64.tar.gz'
20: app_home install_dir
21: owner 'root'
22: group 'wheel'
23: app_home_mode 774
24: action :install
25: end
26:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/install_java/recipes/default.rb:18:in `from_file'
java_ark("jdk") do
action [:install]
supports {:report=>true, :exception=>true}
retries 0
retry_delay 2
default_guard_interpreter :default
declared_type :java_ark
cookbook_name "install_java"
recipe_name "default"
url "http://sv-dc01.sv.local/install_artifacts/java/oracle/JRE/jre-8u77-linux-x64.tar.gz"
app_home "/applications/"
owner "root"
group "wheel"
app_home_mode 774
end
Platform:
---------
x86_64-linux
Running handlers:
[2016-12-12T11:04:25+13:00] ERROR: Running exception handlers
Running handlers complete
[2016-12-12T11:04:25+13:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 17 seconds
[2016-12-12T11:04:25+13:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2016-12-12T11:04:25+13:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2016-12-12T11:04:25+13:00] ERROR: java_ark[jdk] (install_java::default line 18) had an error: Errno::ENOENT: No such file or directory # dir_s_mkdir -
[2016-12-12T11:04:25+13:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
I can't for the life of me figure out why it can't find the file or directory or get a completed install of Java working???
Maybe your environment is not applied correctly. I mean, if you machine is not using your 'env_workstation_dubbo' environment, the node['install_dir'] attribute will not be correctly set. You can read how to set the environment for a node here.
Another possibility is that you are using a modified version of the java cookbook that uses mkdir instead of mkdir_p. I say that because I have not been able to find your 2.0.0 java cookbook version in the supermarket. Where are you getting that cookbook from?
Update after downgraded to java cookbook version v1.43.0
The problem is that the install_dir must be at least 2 directory level deep, following the app_root/app_name format. For example "/applications/default".
If you use "/applications" as install_dir, app_name will be "applications", app_root will be empty and the latter will cause the mkdir error when trying to create the application root directory.

File files/default/plugins/README does not exist for cookbook ohai

I'm running a provisioning setup using vagrant and chef-solo to install the gitlab cookbook (however, this problem does not seem to be specific to the gitlab cookbook).
The run-list is simple:
{
"run_list": [
"recipe[ohai::default]"
]
}
The chef-solo run (chef-solo -c solo.rb -j dna.json) results in the following error:
Error executing action create on resource 'cookbook_file[/etc/chef/ohai_plugins/README]'
The file IS there:
# ls -l /tmp/vagrant-chef/e939c8a8cabcf9cdd72f5d7c3a98d728/cookbooks/ohai-2.0.1/files/default/plugins/README
-rwxrwxrwx. 1 vagrant vagrant 49 Oct 21 13:16 /tmp/vagrant-chef/e939c8a8cabcf9cdd72f5d7c3a98d728/cookbooks/ohai-2.0.1/files/default/plugins/README
When I check the process with 'strace' it looks like I can see the error source:
10444 0.000210 stat("/tmp/vagrant-chef/cookbooks/cookbooks/ohai/files/default/plugins/README", 0x7ffff0570bb0) = -1 ENOENT (No such file or directory)
If you notice, the cookbook is named 'ohai-2.0.1' however the process is trying to access the cookbook 'ohai' (i.e., without the version number).
Has anyone else encountered this before? I've seen one other post related to the issue that suggested putting ohai::default first in the run-list, which I've done (see dna.json above)
This happened to me while using Berkshelf. It names all of the cookbooks name-version instead of just name. To solve this, I did berks vendor and added that directory to my cookbooks path. Now everything works!

Recipe fails on crete_if_missing

I am following this tutorial: http://www.gotealeaf.com/blog/chef-basics-for-rails-developers , where they have us making our own recipe. The code bellow is the recipe. The problem is with the block of code that starts with cookbook_file "id_rsa" and ends right before the, # Add Github as known host, comment. I was able to get past the cookbook_file "id_rsa" block and the cookbook_file "id_rsa.pub" block by moving my id_rsa and id_rsa.pub files into the rails-stack/files/default/ directory, but now it breaks when it attempts the sudo_without_password block. Surprisingly, if I provision vagrant after every error thrown by the action: create_if_missing blocks the configuration gets as far as the cookbooks_file "authorization keys" block but it gets stuck there; even after provisioning when I get the error the first time. Any ideas about what is happening? Please be as descriptive as you can, I am relatively new to devops and only know a few of the ins and outs of vagrant and chef. Thanks in advance!
execute "apt-get update" do
command "apt-get update"
end
# OS Dendencies
%w(git ruby-dev build-essential libsqlite3-dev libssl-dev).each do |pkg|
package pkg
end
# Deployer user, sudoer and with known RSA keys
user_account 'deployer' do
create_group true
end
group "sudo" do
action :modify
members "deployer"
append true
end
cookbook_file "id_rsa" do
source "id_rsa"
path "/home/deployer/.ssh/id_rsa"
group "deployer"
owner "deployer"
mode 0600
action :create_if_missing
end
cookbook_file "id_rsa.pub" do
source "id_rsa.pub"
path "/home/deployer/.ssh/id_rsa.pub"
group "deployer"
owner "deployer"
mode 0644
action :create_if_missing
end
# Allow sudo command without password for sudoers
cookbook_file "sudo_without_password" do
source "sudo_without_password"
path "/etc/sudoers.d/sudo_without_password"
group "root"
owner "root"
mode 0440
action :create_if_missing
end
# Authorize yourself to connect to server
cookbook_file "authorized_keys" do
source "authorized_keys"
path "/home/deployer/.ssh/authorized_keys"
group "deployer"
owner "deployer"
mode 0600
action :create
end
# Add Github as known host
ssh_known_hosts_entry 'github.com'
# Install Ruby Version
include_recipe 'ruby_build'
ruby_build_ruby '2.1.2'
link "/usr/bin/ruby" do
to "/usr/local/ruby/2.1.2/bin/ruby"
end
gem_package 'bundler' do
options '--no-ri --no-rdoc'
end
# Install Rails Application
include_recipe "runit"
application 'capistrano-first-steps' do
owner 'deployer'
group 'deployer'
path '/var/www/capistrano-first-steps'
repository 'git#github.com:gotealeaf/capistrano-first-steps.git'
rails do
bundler true
database do
adapter "sqlite3"
database "db/production.sqlite3"
end
end
unicorn do
worker_processes 2
end
end
****EDIT*******
Since writing the question the first time, I've commented out the sudo_without_password block and was able to find a work around by adding
ssh_keygen true
to the user_account 'deployer' block.
I also put an empty authorized_keys file in rails-stack/files/default/ and that helps the cookbook_file 'authorized_keys' block run without errors.
Now I get this error when vagrant/chef tries to pull the example repo
==> default: [2014-12-04T22:44:18+00:00] ERROR: deploy_revision[capistrano-first-steps] (/tmp/vagrant-chef-3/chef-solo-2/cookbooks/application/providers/default.rb line 123) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '128'
==> default: ---- Begin output of git ls-remote "git#github.com:gotealeaf/capistrano-first-steps.git" "HEAD" ----
==> default: STDOUT:
==> default: STDERR: Warning: Permanently added the RSA host key for IP address '192.30.252.129' to the list of known hosts.
==> default: Permission denied (publickey).
==> default: fatal: Could not read from remote repository.
==> default:
==> default: Please make sure you have the correct access rights
==> default: and the repository exists.
==> default: ---- End output of git ls-remote "git#github.com:gotealeaf/capistrano-first-steps.git" "HEAD" ----
==> default: Ran git ls-remote "git#github.com:gotealeaf/capistrano-first-steps.git" "HEAD" returned 128
==> default: [2014-12-04T22:44:18+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
The answer is simple, after I remembered I had a similar issue with puppet. For some reason, not sure why using
git#github.com:gotealeaf/capistrano-first-steps.git
Does not sit well with vagrant/chef/puppet. So, what I did was change the above line to
https://github.com/gotealeaf/capistrano-first-steps
and that did it, my config of the box worked and there were no problems!
You probably have to point the application resource to the private key that will be used used to clone the repo.
application 'capistrano-first-steps' do
...
deploy_key lazy { File.read("/home/deployer/.ssh/id_rsa") }
...
end
More info -- https://supermarket.chef.io/cookbooks/application

Resources