I have a YAML file like below:
server:
scheme: http
host: localhost
port: 8080
context: myctx
report:
output-dir-base: /tmp
---
spring:
config:
activate:
on-profile: local
vertica:
datasource:
jdbc-url: jdbc:vertica://server:65534/database
username: user
password: db_pass
da-config:
user: username
password: da_pass
da-host:
scheme: http
server: server
port: 65535
From a bash script I want to replace Vertica password with a given value, let's say "first_pass" and da password with "second_pass".
I tried this but didn't work.
sed '/^vertica:\([[:space:]]*password: \).*/s//\1first_pass/' common-properties.yaml
Can this be helped with, please?
UPDATE:
This is what I did to handle the situation. In the bash script file
DB_PASSWORD="first_pass/here" ## I realized later that the password could contain a '/' char as well
DA_PASSWORD="second_pass"
sed -i '/vertica:/{n;n;n;n;s/\(password\).*/\1: '"${DB_PASSWORD//\//\\/}"'/}' my.yaml ## backslash-escape the '/' in the variable
sed -i '/da-config:/{n;n;s/\(password\).*/\1: '"${DA_PASSWORD//\//\\/}"'/}' my.yaml
Assumptions:
input is nicely formatted as displayed in question
OP does not have access to tools designed specifically for yaml editing
New passwords stored in variables:
pass1='first_pass'
pass2='second_pass'
One sed idea using ranges to zero in on the sections we're interested in:
sed -E "/^vertica:/,/password:/ s/(password:).*/\1 ${pass1}/; /^da-config:/,/password:/ s/(password:).*/\1 ${pass2}/; " common-properties.yaml
# or, as OP has mentioned in an update, when the password contains a forward slash we change the sed script delimiter:
sed -E "/^vertica:/,/password:/ s|(password:).*|\1 ${pass1}|; /^da-config:/,/password:/ s|(password:).*|\1 ${pass2}|; " common-properties.yaml
One awk idea:
awk -v p1="${pass1}" -v p2="${pass2}" '
$1 == "password:" { if ( found1 ) { sub(/password: .*/,"password: " p1); found1=0 }
if ( found2 ) { sub(/password: .*/,"password: " p2); found2=0 }
}
/^vertica:/ { found1=1 }
/^da-config:/ { found2=1 }
1
' common-properties.yaml
Both of these generate:
server:
scheme: http
host: localhost
port: 8080
context: myctx
report:
output-dir-base: /tmp
---
spring:
config:
activate:
on-profile: local
vertica:
datasource:
jdbc-url: jdbc:vertica://server:65534/database
username: user
password: first_pass
da-config:
user: username
password: second_pass
da-host:
scheme: http
server: server
port: 65535
Once OP is satisified with the output:
for sed add the -i option to update-in-place the file
if using GNU awk add -i inplace to update-in-place the file; if not running GNU awk then save the output to a temp file and then overwrite the input file with the contents of the temp file
Related
I am writing a script that creates a logstash conf file and adds the configuration, and removes the existing filebeat config file and creates a new one.
I am using cat, but when I run the script, I get:
./script.sh: /etc/logstash/conf.d/apache.conf : Permission denied
./script.sh: /etc/filebeat/filebeat.yml: Permission denied
This is the script. I have tried using sudo chown -R.
Am I missing something or is there a better way to configure my file?
#!/bin/bash
sudo rm /etc/filebeat/filebeat.yml
cat > "/etc/filebeat/filebeat.yml" <<EOF
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
paths:
- /home/ubuntu/logs/.*log
setup.kibana:
output.logstash:
hosts: ["169.254.169.254:5044"]
EOF
sudo touch /etc/logstash/conf.d/apache.conf
sudo cat > "/etc/logstash/conf.d/apache.conf " <<EOF
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["169.254.169.254"]
}
}
EOF
The main problem here is because of how redirections work.
According to this answer:
All redirections (including >) are applied before executing the actual command. In other words, your shell first tries to open /etc/php5/apache2/php.ini for writing using your account, then runs a completely useless sudo cat.
You can easily solve your problem by using tee (with sudo) instead of cat. Then, your script should be like this:
#!/bin/bash
sudo rm /etc/filebeat/filebeat.yml
sudo tee "/etc/filebeat/filebeat.yml" << EOF
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
paths:
- /home/ubuntu/logs/.*log
setup.kibana:
output.logstash:
hosts: ["169.254.169.254:5044"]
EOF
sudo touch /etc/logstash/conf.d/apache.conf
sudo tee "/etc/logstash/conf.d/apache.conf" << EOF
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["169.254.169.254"]
}
}
EOF
type: OS::Nova::Server
properties:
name: { get_param: hostname }
availability_zone: nova
image: { get_param: wisdom_image_id }
flavor: { get_param: flavor }
config_drive: true
networks:
- port: { get_param: public_EDN_port }
- port: { get_param: private_port }
user_data_format: RAW
user_data:
str_replace:
template: |
#!/bin/sh
#Any unix steps from IG leaving
set -e
cd wrongex
trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG
trap 'echo "\"${last_command}\" command filed with exit code $?."' EXIT
cd wrongex
printf "DEVICE=eth0\nBOOTPROTO=static\nONBOOT=yes\nTYPE=Ethernet\nUSERCTL=no\nIPADDR=public_EDNv4_ip\nNETMASK=public_EDNv4_Netmask\nGATEWAY=public_EDNv4_GATEWAY\nIPV6INIT=yes\nIPV6ADDR=public_EDNv6_ip\nIPV6_DEFAULTGW=public_EDNv6_IPV6_GATEWAY\n" > /etc/sysconfig/network-scripts/ifcfg-eth0
printf "DEVICE=eth1\nBOOTPROTO=static
Using "openstack stack create -e envYAML -f HEATyaml stackname" command to instantiate the VM.
This is the portion of HEAT YAML wherein we are trying to run some bash commands in the VM once it is instantiated (all what is part of user_data). Just need to know if there is a way to exit and report the error if any shell command fails.
Currently I have tried trap, set -e etc. but none of them I believe is able to provide the feedback and the stack still is created successfully (even if any of the commands inside user_data fails).
I need to register many thousands of users in ejabberd from csv file. For this, I wrote a simple script.
#!/bin/sh
OLDIFS=$IFS
IFS=','
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read username domain pass p1 p2 p3 p4
do
echo "ejabberdctl register $username $domain $pass"
ejabberdctl register $username $domain $pass
done < users.csv
IFS=$OLDIFS
But in the end, the answer is: Error: cannot_register
If I just run the line copied from the output, everything is ok. The user is created normally.
This is just a little trick for later: once you get your loop working, if you consider ejabberdctl is too slow, you can try using the ReST API. That should be a lot faster when doing many requests.
Configure temporarily something like this (remember to remove this when you finished):
listen:
-
port: 5280
module: ejabberd_http
tls: false
request_handlers:
/api: mod_http_api
api_permissions:
"console commands":
from:
- ejabberd_ctl
- mod_http_api
who: all
what: "*"
modules:
mod_http_api: {}
Then execute this in a shell to register an account:
curl 'localhost:5280/api/register?user=user2&host=localhost&password=somepass123'
I put the following to after.sh to autoconfigure the Xdebug form project:
#!/bin/sh
echo "Configuring Xdebug"
ip=$(netstat -rn | grep "^0.0.0.0 " | cut -d " " -f10)
xdebug_config="/etc/php/$(php -v | head -n 1 | awk '{print $2}'|cut -c 1-3)/mods-available/xdebug.ini"
echo "IP for the xdebug to connect back: ${ip}"
echo "Xdebug Configuration path: ${xdebug_config}"
echo "Port for the Xdebug to connect back: ${XDEBUG_PORT}"
echo "Optimize for ${IDE} ide"
if [ $IDE=='atom' ]; then
echo "Configuring xdebug for ATOM ide"
if [ -z ${xdebug_config} ]; then
sudo touch ${xdebug_config}
fi
sudo cat <<EOL >${xdebug_config}
zend_extension = xdebug.so
xdebug.remote_enable = 1
xdebug.remote_host=${ip}
xdebug.remote_port = ${XDEBUG_PORT}
xdebug.max_nesting_level = 1000
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_autostart=true
xdebug.remote_log=xdebug.log
EOL
fi
Also I have the following settings to Homestead.yaml:
ip: 192.168.10.10
memory: 2048
cpus: 1
provider: virtualbox
authorize: ~/.ssh/id_rsa.pub
timeout: 120
keys:
- ~/.ssh/id_rsa
folders:
-
map: /home/pcmagas/Kwdikas/php/apps/ellakcy_member_app/
to: /home/vagrant/code
sites:
-
map: homestead.test
to: /home/vagrant/code/web
type: symfony
databases:
- homestead
- homestead-test
variables:
- key: database_host
value: 127.0.0.1
- key: database_port
value: 3306
- key: database_name
value: homestead
- key: database_user
value: homestead
- key: database_password
value: secret
- key: smtp_host
value: localhost
- key: smtp_port
value: 1025
- key: smtp_user
value: no-reply#example.com
- key: IDE
value: atom
- key: XDEBUG_PORT
value: 9091
name: ellakcy-member-app
hostname: ellakcy-member-app
But for some reason it cannot read the values from enviromental variables defined in Homestead.yml as seen in the following output:
ellakcy-member-app: IP for the xdebug to connect back: 10.0.2.2
ellakcy-member-app: Xdebug Configuration path: /etc/php/7.2/mods-available/xdebug.ini
ellakcy-member-app: Port for the Xdebug to connect back:
ellakcy-member-app: Optimize for ide
ellakcy-member-app: Configuring xdebug for ATOM ide
As you can see it fails to read values from the IDE and XDEBUG_PORT do you knwo why and how I can fix that?
You can put a parse_yaml.sh:
#!/bin/sh
parse_yaml() {
local prefix=$2
local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo #|tr # '\034')
sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
-e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" $1 |
awk -F$fs '{
indent = length($1)/2;
vname[indent] = $2;
for (i in vname) {if (i > indent) {delete vname[i]}}
if (length($3) > 0) {
vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
}
}'
}
And into after.sh
#!/bin/sh
# include parse_yaml function
. parse_yaml.sh
# read yaml file
eval $(parse_yaml zconfig.yml "config_")
# access yaml content
echo $config_development_database
thanks -> https://gist.github.com/pkuczynski/8665367
In my case I tried the approach of having a file named xdebug.conf where I place anything that the default xdebug.conf needs to get rewritten:
zend_extension = xdebug.so
xdebug.remote_enable = 1
xdebug.remote_host = $ip
xdebug.remote_port = 9091
xdebug.max_nesting_level = 1000
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_autostart=true
xdebug.remote_log=xdebug.log
The $ip indicates the auto-replaced value with the correct ip in order for the xdebug to get connected into. The script that actually updates the xdebug configuration with the appropriate values is this one in my after.sh
#!/bin/sh
code_path="/home/vagrant/code"
cd $code_path
# Some other bootstrapping
echo "Configuring Xdebug"
ip=$(netstat -rn | grep "^0.0.0.0 " | cut -d " " -f10)
xdebug_config="/etc/php/$(php -v | head -n 1 | awk '{print $2}'|cut -c 1-3)/mods-available/xdebug.ini"
echo "Xdebug config file ${xdebug_config}"
if [ -f "${code_path}/xdebug.conf" ]; then
echo "Specifying the ip with ${ip}"
sed "s/\$ip/${ip}/g" xdebug.conf > xdebug.conf.tmp
echo "Moving Into ${xdebug_config}"
cat xdebug.conf.tmp
sudo cp ./xdebug.conf.tmp ${xdebug_config}
else
echo "File not found"
fi
The last step is to .gitignore any xdebug.conf* file. So now the developer has to create his own xdebug.conf.
I'm tasked with creating about a hundred files for use with puppet. I'm creating .yaml files with unique filenames that will contain site-specific IP and hostname information, these must have the same format (ideally from a template).
I want to create a file generator that fills in variables for IP, subnet, network, and hostname from an input file (.csv?) What's the best way to approach this?
sample format:
---
network::interfaces::interfaces:
eth0:
method: 'static'
address: '10.20.30.1'
netmask: '255.255.240.0'
broadcast: '10.20.30.255'
network: '10.20.30.0'
gateway: '10.20.30.1'
network::interfaces::auto:
- 'eth0'
hosts::host_entries:
HOSTNAME:
ip: '10.20.30.2'
hosts::purge_hosts: true
dhcpd::range_start: '10.20.30.11'
dhcpd::range_stop: '10.20.30.240'
dhcpd::gateway: '10.20.30.1'
hornetq::site: 'test'
Write a skeleton like this:
network::interfaces::interfaces:
eth0:
method: 'static'
address: '__IP__'
netmask: '__MASK__'
broadcast: '__BC__'
network: '__NET__'
gateway: '__GW__'
etc.
Generate files with a loop like
cat input-file | while read OUTPUT IP MASK BC NET GW ; do
sed -e s/__IP__/$IP/ \
-e s/__MASK__/$MASK/ \
-e s/__BC__/$BC/ \
-e s/__NET__/$NET/ \
-e s/__GW__/$GW/ \
<$SKELETON >$OUTPUT
done
This assumes that the fileds in the input file are separated by whitespace, with the name of the respective output file in the first column.