In my chef recipe, I am basically decrypting a couple of data bags:
1. test.json
2. sample.json
The data obtained after decryption will next be used to create files on my kitchen node. Basically, test.json and sample.json are encrypted using a secret key I have (test.json was obtained from test.txt and sample.json was obtained from sample.txt which are both plaintext files), within a script called gendatabags.rb that creates these files and puts them in their respective places. Note that the gendatabags.rb takes the secret key path and input file path as input parameters. Now as I want to integration-test this flow, I am looking forward to using a test secret key that I've generated. I would like to provide test versions of both test.txt and sample.txt which contain some dummy strings. The catch is, now I'd like to run this script automatically during compile time of my recipe. Can someone please provide some info on how to achieve this?
Thank you!
Strongly wouldn't recommend this. Technically you could do this with the execute resource but you'd have all sorts of timing issues and it would defeat the purpose of having the encrypted data bag anyway.
Now, if you're trying to test a dummy encrypted databag that is easy. You'll make a data bag as normal but with the addition of the -z switch
knife data bag create <data bag name> -z
knife data bag from file <data bag name> <path to .json file> --secret-file <path to encryption key file> -z
This will make a local directory with the name of your data bag and place the encrypted data bag item inside of it, with the name of the "id" value of the json file.
-z defaults to putting the data bag and items in /users//data_bags
From there you can edit your .kitchen.yml to point towards both your data bag and secret key thusly
Suites:
- name: default
run_list:
data_bags_path: <path to data_bags dir>
encrypted_data_bag_secret_key_path: <path to secret_file>
and if you have multiple suites using the same data_bags path you can move the declaration to
provisioner:
name: chef_zero
data_bags_path: <path to data_bags dir>
encrypted_data_bag_secret_key_path: <path to secret_file>
Hope this helps.
Related
I'm working with the LetsEncrypt dns-01 challenge system which entails dynamically creating a TXT record in Google Cloud DNS with specific content, so LE can assert proof of ownership for generating a wildcard certificate (so I can't use http-01). The problem is sometimes LE tells me to create a TXT record that starts with a "-", for example -E_DFDFHJKF1783FSHDJ. I cannot get the gcloud cli to properly accept this data no matter what I do.
Example:
gcloud dns record-sets transaction start --zone=myzone
gcloud dns record-sets transaction add "-E_ASDFSDF" --ttl=30 --zone=myzone --name=test --type=TXT
gcloud dns record-sets transaction remove "-A_DSFKHSDF" --ttl=30 --zone=myzone --name=test2 --type=TXT
If you run those commands and inspect the resulting transaction.yaml you can see whether it properly contains the right string. If it did it correct, you should see something like:
- kind: dns#resourceRecordSet
name: test.
rrdatas:
- '"ASDFASDF"'
ttl: 30
type: TXT
I am executing this via Node's child_process, but I have the issue even if I execute it directly from bash, so Node isn't really meaningful issue at the moment. I've tried echoing the value in. I've tried setting an environment variable and using that in the string.
No matter what I do I get an error like the following:
ERROR: (gcloud.dns.record-sets.transaction.add) unrecognized arguments: -E_ASDFSDF
It turns out some characters need to be escaped in the CLI. I can confirm that the following works:
gcloud dns --project=myprojectid record-sets transaction add "\-test123" --name=test.mydomain.com. --ttl=300 --type=TXT --zone=myzoneid
We have 20 worker on AWS and I want to parameterized CSV file name for each instance Please help
I have divided my CSV in to number of Load generator hosts
$ wc -l "youroriginalcsv.csv" /* this will return number of total rows in csv*/
$ split -l "count of above query"/"number of hosts" "youroriginalcsv.csv" /* this will split CSV with file name as xaa, xab ... */
Transfer each unique CSV to all available hosts
$ scp xaa host1_user#host1_ip:/csvpath/csvfile.csv
$ scp xab host2_user#host2_ip:/csvpath/csvfile.csv
$ scp xaz hostN_user#hostN_ip:/csvpath/csvfile.csv
Now I want to use specific file name for specific host
What do you mean by "specific file name for specific host"? Your CSV files are all named csvfile.csv so it's sufficient to specify /csvpath/csvfile.csv in the CSV Data Set Config and each JMeter slave will pick up its own file containing partial data from the "big" CSV file.
If you want to use different names for CSV files depending on the machine IP address or DNS hostname - go for combination of If Controller with __machineName() or __machineIP() function
Also if you don't want the same data to be re-used by different JMeter slaves you can consider using Redis Data Set Config or HTTP Simple Table Server, this way you won't have to "split" and "copy" CSV files and will be able to centrally manage your test data from a single location
I want to use Cloudera's MapReduceIndexerTool to understand how morphlines work. I created a basic morphline that just reads lines from the input file and I tried to run that tool using that command:
hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar org.apache.solr.hadoop.MapReduceIndexerTool \
--morphline-file morphline.conf \
--output-dir hdfs:///hostname/dir/ \
--dry-run true
Hadoop is installed on the same machine where I run this command.
The error I'm getting is the following:
net.sourceforge.argparse4j.inf.ArgumentParserException: Cannot write parent of file: hdfs:/hostname/dir
at org.apache.solr.hadoop.PathArgumentType.verifyCanWriteParent(PathArgumentType.java:200)
The /dir directory has 777 permissions on it, so it is definitely allowed to write into it. I don't know what I should do to allow it to write into that output directory.
I'm new to HDFS and I don't know how I should approach this problem. Logs don't offer me any info about that.
What I tried until now (with no result):
created a hierarchy of 2 directories (/dir/dir2) and put 777 permissions on both of them
changed the output-dir schema from hdfs:///... to hdfs://... because all the examples in the --help menu are built that way, but this leads to an invalid schema error
Thank you.
It states 'cannot write parent of file'. And the parent in your case is /. Take a look into the source:
private void verifyCanWriteParent(ArgumentParser parser, Path file) throws ArgumentParserException, IOException {
Path parent = file.getParent();
if (parent == null || !fs.exists(parent) || !fs.getFileStatus(parent).getPermission().getUserAction().implies(FsAction.WRITE)) {
throw new ArgumentParserException("Cannot write parent of file: " + file, parser);
}
}
In the message printed is file, in your case hdfs:/hostname/dir, so file.getParent() will be /.
Additionally you can try the permissions with hadoop fs command, for example you can try to create a zero length file in the path:
hadoop fs -touchz /test-file
I solved that problem after days of working on it.
The problem is with that line --output-dir hdfs:///hostname/dir/.
First of all, there are not 3 slashes at the beginning as I put in my continuous trying to make this work, there are only 2 (as in any valid HDFS URI). Actually I put 3 slashes because otherwise, the tool throws an invalid schema exception! You can easily see in this code that the schema check is done before the verifyCanWriteParent check.
I tried to get the hostname by simply running the hostname command on the Cent OS machine that I was running the tool on. This was the main issue. I analyzed the /etc/hosts file and I saw that there are 2 hostnames for the same local IP. I took the second one and it worked. (I also attached the port to the hostname, so the final format is the following: --output-dir hdfs://correct_hostname:8020/path/to/file/from/hdfs
This error is very confusing because everywhere you look for the namenode hostname, you will see the same thing that the hostname command returns. Moreover, the errors are not structured in a way that you can diagnose the problem and take a logical path to solve it.
Additional information regarding this tool and debugging it
If you want to see the actual code that runs behind it, check the cloudera version that you are running and select the same branch on the official repository. The master is not up to date.
If you want to just run this tool to play with the morphline (by using the --dry-run option) without connecting to Solr and playing with it, you can't. You have to specify a Zookeeper endpoint and a Solr collection or a solr config directory, which involves additional work to research on. This is something that can be improved to this tool.
You don't need to run the tool with -u hdfs, it works with a regular user.
Trying to extract xml data from multiple string outputs dynamically (the data changes) into a column format.
About 100 of these XML bits echo out when I run a query against an SQL database.
<?xml version="1.0"?>
<Connection>
<ConnectionType>Putty</ConnectionType>
<CreatedBy>Someone</CreatedBy>
<CreationDateTime>2014-10-27T11:53:59.8993492-04:00</CreationDateTime>
<Events>
<OpenCommentPrompt>true</OpenCommentPrompt>
<WarnIfAlreadyOpened>true</WarnIfAlreadyOpened>
</Events>
<Group>Cloud Services Client Delivery\Willis\Linux\Test - SJC</Group>
<ID>77e96d52-f165-482f-8389-ffb95b9d8ccd</ID>
<KeyboardHook>InFullScreenMode</KeyboardHook>
<MetaInformation />
<Name>Hostname-H-A10D</Name>
<OpenEmbedded>true</OpenEmbedded>
<PinEmbeddedMode>False</PinEmbeddedMode>
<Putty>
<PortFowardingArray />
<Scripting />
<SessionHost>10.0.0.100</SessionHost>
<SessionName>10.0.0.100</SessionName>
<TelnetEncoding>IBM437</TelnetEncoding>
</Putty>
<ScreenColor>C24Bits</ScreenColor>
<SoundHook>DoNotPlay</SoundHook>
<Stamp>771324d1-0c59-4f12-b81e-96edb5185ef7</Stamp>
</Connection>
And what I need is the and in a column format. And essentially where the hostname equal Hostname-H-A10D, I want to be able to match the D at the end and mark the First column with Dev, Q as Test and no letter at the end as Prod. So the output would look like -->
Dev Hostname-H-A10D 10.0.0.100
Dev Hostname-H-A11D 10.0.0.101
Prod Hostname-H-A12 10.0.0.201
Test Hostname-H-A13Q 10.0.0.10
I have played around with sed/awk/etc and not just cannot get the format I want without writing out temp flat files. I would prefer to get this into an array using something like xmlstarlet or xmllint. Of course better suggestions can be made and that is why I am here :) Thanks folks.
It would be better to use an XML parser.
Using awk:
$ awk -F'[<>]' 'BEGIN{a["D"]="Dev";a["Q"]="Test"} /Name/{name=$3; type=a[substr(name,length(name))]; if (length(type)==0) type="Prod";} /SessionHost/{print type, name, $3;}' s.xml
Dev Hostname-H-A10D 10.0.0.100
How it works
BEGIN{a["D"]="Dev";a["Q"]="Test"}
This defines associative array a.
/Name/{name=$3; type=a[substr(name,length(name))]; if (length(type)==0) type="Prod";}
On the line that has the host name, this captures the host name and, from it, determines the host type.
/SessionHost/{print type, name, $3;}
On the line that contains the host IP, this prints the type, name, and IP.
You have not mentioned any parameter in XML file whether the host is Dev or Prod or Test.
But from the above XML file you can get the name using the following way.
$cat test.xml |grep Name |awk -F '[<,>]' '{print $3}' |xargs
Hostname-H-A10D 10.0.0.100
I am new to Ansible. I am able to test it and its working fine with my test requirment. For making connection between management node and the client node I am using already created ssh key pair. How can I use another node with different SSH key pair? For reference I am considering 3 ec2-instance with different key pairs.
Good news- in a basic use case, this is fairly easy. Simply use the ansible_ssh_private_key_file parameter in your Ansible inventory.
Here are some examples purloined from my personal file:
$ cat hosts.ini
[server1]
54.1.2.3 ansible_ssh_private_key_file=~/.ssh/server1.pem
[testservers]
ec2-54-2-3-4.compute-1.amazonaws.com ansible_ssh_private_key_file=~/.ssh/aws-testserver.pem ansible_ssh_user=ubuntu
ec2-54-2-3-5.compute-1.amazonaws.com ansible_ssh_private_key_file=~/.ssh/aws-testserver.pem ansible_ssh_user=ubuntu
[piwall]
10.0.0.88 ansible_ssh_private_key_file=~/.ssh/raspberrypi.pem ansible_ssh_user=pi
tedder42 is correct, however, there is a better way of doing it.
See ansible_ssh_private_key_file here.
I have in my host files the following
# SSH Keys configuration
[all_servers:vars]
ansible_ssh_private_key_file = <YOUR PRIVATE KEY LOCATION>
# Server configuration
[all_servers:children]
elastic_servers
nginx_servers
[elastic_servers]
44.22.11.22
44.55.66.77
22.11.22.33
[nginx_servers]
22.24.123.123
233.111.222.11
If you have multiple keys configuration, you can do something like the following
[nginx:vars]
ansible_ssh_private_key_file = <YOUR PRIVATE KEY LOCATION>
[app:vars]
ansible_ssh_private_key_file = <YOUR 2nd PRIVATE KEY LOCATION>
[nginx:children]
nginx_servers
[app:children]
app_servers
[nginx_servers]
1.2.3.4
[app_servers]
5.5.5.5
6.6.6.6
That's way cleaner than tedder42 answer. This is useful if you have multiple keys for multiple servers.
Otherwise, you can include your key in ansible.cfg file instead.