How can I deploy a BOSH Director on BOSH Lite - vagrant

Question

I'm currently trying to install a BOSH Director on BOSH Lite - it's clear to me that BOSH Lite already ships with a Director, but I would like to test a release containing a Director "on top of that". Here is my setup:
Everything works fine until I add the warden_cpi job. I would like to configure the Warden CPI to connect to Warden running on the Virtual Machine hosting BOSH Lite and still being available to the Director . So what I tried is this:
releases:
- name: bosh-warden-cpi
url: https://bosh.io/d/github.com/cppforlife/bosh-warden-cpi-release?v=29
sha1: 9cc293351744f3892d4a79479cccd3c3b2cf33c7
version: latest
instance_groups:
- name: bosh-components
jobs:
- name: warden_cpi
release: bosh-warden-cpi
properties:
warden_cpi:
host_ip: 10.254.50.4 # host IP of BOSH Lite Vagrant Box
warden:
connect_network: tcp
connect_address: 10.254.50.4:7777 # again host IP and Port of garden-linux on BOSH Lite Vagrant Box
agent:
mbus: nats://user:password#127.0.0.1:4222
blobstore:
provider: dav
options:
endpoint: http://127.0.0.1:25250
user: user
password: password
where 10.254.50.4 is the IP address of the Vagrant Box and 7777 is the port of garden-linux.
During the deployment, I get this message from bosh vms
+----------------------------------------------------------+--------------------+-----+---------+--------------+
| VM | State | AZ | VM Type | IPs |
+----------------------------------------------------------+--------------------+-----+---------+--------------+
| bosh-components/0 (37a1938e-e1df-4650-bec6-460e4bc3916e) | unresponsive agent | n/a | small | |
| bosh-director/0 (2bb47ce1-0bba-49aa-b9a3-86e881e91ee9) | running | n/a | small | 10.244.102.2 |
| jumpbox/0 (51c895ae-3563-4561-ba3f-d0174e90c3f4) | running | n/a | small | 10.244.102.4 |
+----------------------------------------------------------+--------------------+-----+---------+--------------+
As an error message from bosh deploy, I get this:
Error 450002: Timed out sending `get_state' to e1ed3839-ade4-4e12-8f33-6ee6000750d0 after 45 seconds
After the error occurs, I can see the VM with bosh vms:
+----------------------------------------------------------+---------+-----+---------+--------------+
| VM | State | AZ | VM Type | IPs |
+----------------------------------------------------------+---------+-----+---------+--------------+
| bosh-components/0 (37a1938e-e1df-4650-bec6-460e4bc3916e) | running | n/a | small | 10.244.102.3 |
| bosh-director/0 (2bb47ce1-0bba-49aa-b9a3-86e881e91ee9) | failing | n/a | small | 10.244.102.2 |
| jumpbox/0 (51c895ae-3563-4561-ba3f-d0174e90c3f4) | running | n/a | small | 10.244.102.4 |
+----------------------------------------------------------+---------+-----+---------+--------------+
But when I ssh into the bosh-components VM, there are no jobs in /var/vcap/jobs.
When I remove the warden_cpi block from the jobs list, everything runs as expected. The full jobs list for my BOSH components VM:
nats
postgres
registry
blobstore
The Director itself runs on another machine. Without the Warden CPI the two machines can communicate as expected.
Can anybody point out to me how I have to configure the Warden CPI so that it connects to the Vagrant Box as expected?

Answer 1

The question is very old, it's a BOSH v1 CLI whereas now BOSH v2 is an established standard, Garden Linux had been deprecated a long time ago in favor of Garden runC, but still, having experimented a lot with BOSH-Lite, I'd like to answer this one.
First, a semantics remark: you shouldn't say “on top of that”, but “as instructed by” instead, because a BOSH Director just instructs some underlying (API-based) infrastructure to do something, that eventually makes it run some workloads.
Second, there are two hurdles you might hit here:
The main problem is that the Warden CPI talks to both the Garden backend and the local Linux kernel for setting up various things around those Garden containers. As a direct consequence, you cannot run a Warden CPI inside a BOSH-Lite container.
The filesystem used (here by the long-gone Garden Linux, but nowadays the issue would be similar with Garden runC) might not work inside a Garden container, as managed by the pre-existing Warden CPI.
All in all, then main thing to be aware of, is this idea that the Warden CPI not only talks to the Garden backend through some its REST API. More than that, the Warden CPI needs to be co-located with the Linux kernel that runs Garden, in order to make system calls and run local commands for mounting persistent storage and other things.