Openstack snapshot create and restore scripting with bash commands - bash

First of all it amazes me there is so little information about openstack and script examples but that is not the question i have
I want to create a snapshot and an simple way to restore the snapshot. Because the way of our hosting provider uses underlying storage i am unable to use the rebuild command so i need to destroy the running vm and recreate it with the snapshot image as a base. The creation of the image only works when all information about the running vm is provided as input parameters and here comes the troubles i have.
the information needed is provided by 3 commands
command1: nova show
Output:
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| NWFINFRA_1600 network | 10.0.0.39 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | gn3a |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-10-02T14:25:21.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2019-10-02T14:25:05Z |
| description | - |
| flavor:disk | 0 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"ostype": "win", "hw:cpu_cores": "1", "hw:cpu_sockets": "2"} |
| flavor:original_name | win.2large |
| flavor:ram | 8192 |
| flavor:swap | 0 |
| flavor:vcpus | 2 |
| hostId | 18aa94c61106a53b2d9e672e93619a6fce76abb1ee6ba9da471491f9 |
| id | 70941fbf-9143-4f1c-a5e7-979f818ace23 |
| image | IFW039-InstanceSnapshot (8ee1104d-55e4-4c99-93e5-ceb4a53ce13f) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | IWF039 |
| os-extended-volumes:volumes_attached | [{"id": "1134fe12-777b-4c26-ac2b-e6ecb6ad4f70", "delete_on_termination": false}, {"id": "f610a46e-46ad-460f-81b3-e2b34acfbbfc", "delete_on_termination": false}] |
| progress | 0 |
| status | ACTIVE |
| tags | [] |
| tenant_id | 4c15fd467dde4bd6a25427d6bab64a7f |
| trusted_image_certificates | - |
| updated | 2019-10-02T14:25:21Z |
| user_id | ddff2ce854114bef873bac9a1476805e |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Command 2: ./Scripts/openstack port show NWFINFRA_1600_IWF039
where NWFINFRA_1600_IWF039 is a combination of previous output NWFINFRA_1600 network and the server name IWF039
Output:
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | None |
| binding_profile | None |
| binding_vif_details | None |
| binding_vif_type | None |
| binding_vnic_type | normal |
| created_at | 2019-10-02T06:49:08Z |
| data_plane_status | None |
| description | |
| device_id | 70941fbf-9143-4f1c-a5e7-979f818ace23 |
| device_owner | compute:gn3a |
| dns_assignment | fqdn='iwf039.rijkscloud.local.', hostname='iwf039', ip_address='10.0.0.39' |
| dns_domain | |
| dns_name | iwf039 |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.0.39', subnet_id='3298e8d0-b317-465c-8757-c1a4f2cad298' |
| id | b49a7d3a-bb0d-49cb-a04b-64c5dbf9df20 |
| location | cloud='', project.domain_id='default', project.domain_name=, project.id='4c15fd467dde4bd6a25427d6bab64a7f', project.name='vws-pgb', region_name='Groningen3', zone= |
| mac_address | fa:16:3e:99:cc:3c |
| name | NWFINFRA_1600_IWF039 |
| network_id | 450dcc7a-5e55-4e38-9f4e-de9a9c685502 |
| port_security_enabled | False |
| project_id | 4c15fd467dde4bd6a25427d6bab64a7f |
| propagate_uplink_status | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 15 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2019-10-03T11:10:00Z |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
with these outputs i can create the build command to restore the snapshot:
nova boot --poll --flavor win.2large --image IFW039-InstanceSnapshot --security-groups default --availability-zone gn3a --nic net-id=450dcc7a-5e55-4e38-9f4e-de9a9c685502 IWF039
note: the image is the name of the created snapshot
I try to script this so i can have a simple snapshot create and restore procedure
but i get stuck on the table layout of the output. This shows really nice but i cannot use it in my scripting to redirect the output to input variables.
I tryed using this: { read foo ; read ID Name MAC IP status;} < <(./Scripts/openstack port list --server IWF039 | sed 's/+--------------------------------------+----------------------+-------------------+--------------------------------------------------------------------------+-------- +//' | sed 's/|//' | sed 's/MAC Address/MAC/' | sed 's/Fixed IP Addresses/IP/')
But the variables get contents like '|' char etc.
So echo $Name gives '|' as output.
There must be a simpler way but i am unable to see it.
Please help ...
I managed to get it almost working by using awk instead of grep:
i have now this code:
#/bin/bash # # Query needed variables # echo -e "\nQuery needed information" NETWORK=$(nova show IWF039 | awk '/network/ {print $2}') ZONE=$(nova show IWF039 | awk '/OS-EXT-AZ:availability_zone/ {print $4}') FLAVOR=$(nova show IWF039 | awk '/flavor:original_name/ {print $4}') SERVERID=$(nova show IWF039 | awk -F '|' '/id/ {print $3; exit}') NETWORKPORT=$(nova interface-list IWF039 | awk -F '|' '/ACTIVE/ {print $3}') # Print out variables echo "network: $NETWORK" echo "zone: $ZONE" echo "flavor: $FLAVOR" echo "server_id: $SERVERID" echo "network_port_id: $NETWORKPORT" # Remove current instance echo -e "\nRemove current instance" nova delete $SERVERID # Rebuild instance from snapshot image echo -e "\nRebuild instance from snapshot" nova boot --poll --flavor $FLAVOR --image IFW039-InstanceSnapshot2 --security-groups default --availability-zone $ZONE --nic port-id=$NETWORKPORT IWF039
If i run the script the last item however e.g. IWF039 which is the name of the instance i want to use throws me an error:
error: unrecognized arguments: IWF039
anyone can tell me why ?
If i run the line on the commandline it works, only not from the bash script

#/bin/bash
#
# Script restores snapshot created in OpenStack
# Script created by Lex IT, Alex Flora
# usage: snapshot-restore.sh <snapshot image name> <server name to restore to>
# to use: make sure to install the following python openstack modules:
# pip install python-openstackclient python-keystoneclient python-glanceclient python-novaclient python-neutronclient
#
#
# Query needed variables
#
if [ "$#" -eq "0" ]
then
echo -e "usage: restore_snapshot <name of snapshot> <name of server>"
echo -e "Querying available snapshots, one moment please ..."
glance image-list
echo -e "\n\033[0;33mGive name of snapshot to restore"
echo -e "\033[0m"
read SNAPSHOT
echo -e "\n\033[0;33mGive server name to restore"
echo -e "\033[0m"
read SERVER
else
SNAPSHOT=$1
SERVER=$2
fi
echo -e "\n\033[0mQuery needed server information from server $SERVER, one moment please ..."
NETWORK=$(nova show IWF039 | awk '/network/ {print $2}' | sed -e 's/^[[:space:]]*//')
ZONE=$(nova show IWF039 | awk '/OS-EXT-AZ:availability_zone/ {print $4}' | sed -e 's/^[[:space:]]*//')
FLAVOR=$(nova show IWF039 | awk '/flavor:original_name/ {print $4}' | sed -e 's/^[[:space:]]*//')
SERVERID=$(nova show IWF039 | awk -F '|' '/\<id\>/ {print $3; exit}' | sed -e 's/^[[:space:]]*//')
NETWORKPORT=$(nova interface-list IWF039 | awk -F '|' '/ACTIVE/ {print $3}' | sed -e 's/^[[:space:]]*//')
# Print out variables
echo -e "\033[0mnetwork: \033[0;32m$NETWORK"
echo -e "\033[0mzone: \033[0;32m$ZONE"
echo -e "\033[0mflavor: \033[0;32m$FLAVOR"
echo -e "\033[0mserver_id: \033[0;32m$SERVERID"
echo -e "\033[0mnetwork_port_id: \033[0;32m$NETWORKPORT"
echo -e "\033[0mSnapshot image: \033[0;32m$SNAPSHOT"
echo -e "\033[0mServer naam: \033[0;32m$SERVER"
# Ask confirmation
echo -e "\n\033[0mGoing to restore snapshot image $SNAPSHOT to server $SERVER"
read -p "Is this correct (y/n) ? " -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
# Remove current instance
echo -e "\n\033[0mRemove current instance"
nova delete $SERVERID
sleep 3
# Rebuild instance from snapshot image
echo -e "\nRebuild instance from snapshot using command:"
echo -e "\033[0mnova boot --poll --flavor $FLAVOR --image $SNAPSHOT --security-groups default --availability-zone $ZONE --nic port-id=$NETWORKPORT $SERVER"
nova boot --poll --flavor $FLAVOR --image $SNAPSHOT --security-groups default --availability-zone $ZONE --nic port-id=$NETWORKPORT $SERVER
fi
This is my complete script. Problem was that the output contained spaces in front of the string. I resolved this with the following sed command: sed -e 's/^[[:space:]]*//'
Hopefully someone has some use of the script.

Related

How to ssh into and issue command to list of ip addresses in a txt file Jenkins

I have a list of ip addresses in a text file that I wish to use in a script.
Here is the code outputting the ip addresses in the text file
openstack server list | grep agent | awk '{print \$9}' >> ${STACK}_list.txt
I would like to retrieve the ip addresses and use in a loop by ssh'ing in to them but not sure how to do that
Please refer this post.
script to read a file with IP addresses and login
Might be helpful for you.
Thanks.
Subhadeep
You can use a regex to filter all ip addresses from the server list-output:
openstack server list | grep -o '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*'
You could pipe this output into a file if you need or to use this within a bash-script you could make something like this, without writing it into a file:
#!/bin/bash
#
ADDRESSES=$(openstack server list | grep -o '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*')
for ADDRESS in $ADDRESSES
do
echo "ip: $ADDRESS"
done
It reads all ip-addresses from ther server-list output and iterate within the for-loop over this output and prints each ip separate on the terminal. Instead of the echo you could insert your ssh-command.
Example-server on my deployment:
root#m1r1:~# openstack server list
+--------------------------------------+-----------------------+--------+--------------------------+----------------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------------+--------+--------------------------+----------------+--------+
| 46d04a77-4d33-4bb3-8214-b1444eed33a3 | server1 | ACTIVE | l2-network=192.168.4.131 | cirros | XS |
| e9489aca-00c3-4fc9-afc5-515c08b17406 | server2 | ACTIVE | l2-network=192.168.4.61 | | XS |
| ea8cec6a-a8d5-4bbb-970e-aaf65d7374b2 | server3 | ACTIVE | l2-network=192.168.4.163 | cirros | S |
| 7d934ec4-1d53-467b-9220-d67b4b68a832 | server4 | ACTIVE | l2-network=192.168.4.184 | | XS |
| 74d3036e-372a-4566-8ba2-10a0760c5562 | server5 | ACTIVE | l2-network=192.168.4.232 | cirros | XS |
| e08e1637-f4df-478d-a478-6578d038cb22 | server6 | ACTIVE | l2-network=192.168.4.190 | | XS |
| 8307a481-679e-4df0-a64e-3a497b13ac81 | server7 | ACTIVE | l2-network=192.168.4.202 | | XS |
| 38d10b12-daa5-483e-b9a5-9a16ba14d841 | server8 | ACTIVE | l2-network=192.168.4.250 | cirros | XS |
+--------------------------------------+-----------------------+--------+--------------------------+----------------+--------+
Output of this example:
ip: 192.168.4.131
ip: 192.168.4.61
ip: 192.168.4.163
ip: 192.168.4.184
ip: 192.168.4.232
ip: 192.168.4.190
ip: 192.168.4.202
ip: 192.168.4.250
#!/bin/sh
openstack server list | grep -o '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*' > stack
while $(wc -l stack | cut -d' ' -f1) -gt 0 ]
do
ipnumber=$(sed -n '1p' stack)
echo "${ipnumber}"
sed -i '1d' stack
done
The echo command there is just a placeholder. You can replace it with ssh, or whatever else you want to do, with the IP number in the variable.

Find references to files, recursively

In a project where XML/JS/Java files can contain references to other such files, I'd like to be able to have a quick overview of what has to be carefully checked, when one file has been updated.
So, it means I need to eventually have a look at all files referencing the modified one, and all files referencing files which refer to the modified one, etc. (recursively on matched files).
For one level, it's quite simple:
grep -E -l -o --include=*.{xml,js,java} -r "$FILE" . | xargs -n 1 basename
But how can I automate that to match (grand-(grand-))parents?
And how can that be, maybe, made more readable? For example, with a tree structure?
For example, if the file that interests me is called modified.js...
show-referring-files-to modified.js
... I could wish such an output:
some-file-with-ref-to-modified.xml
|__ a-file-referring-to-some-file-with-ref-to-modified.js
another-one-with-ref-to-modified.xml
|__ a-file-referring-to-another-one-with-ref-to-modified.js
|__ a-grand-parent-file-having-ref-to-ref-file.xml
|__ another-file-referring-to-another-one-with-ref-to-modified.js
or any other output (even flat) which allows for quickly checking which files are potentially impacted by a change.
UPDATE -- Results of current proposed answer:
ahmsff.js
|__ahmsff.xml
| |__ahmsd.js
| | |__ahmsd.xml
| | | |__ahmst.xml
| | | | |__BESH.java
| |__ahru.js
| | |__ahru.xml
| | | |__ahrut.xml
| | | | |__ashrba.js
| | | | | |__ashrba.xml
| | | | | | |__STR.java
| | |__ahrufrp.xml
| | | |__ahru.js
| | | | |__ahru.xml
| | | | | |__ahrut.xml
| | | | | | |__ashrba.js
| | | | | | | |__ashrba.xml
| | | | | | | | |__STR.java
| | | | |__ahrufrp.xml
| | | | | |__ahru.js
| | | | | | |__ahru.xml
| | | | | | | |__ahrut.xml
| | | | | | | | |__ashrba.js
| | | | | | | | | |__ashrba.xml
| | | | | | | | | | |__STR.java
| | | | | | |__ahrufrp.xml
(...)
I'd use a shell function (for the recursion) inside an shell script:
Assuming the filenames are unique have no characters that need escaping in them:
File: /usr/local/bin/show-referring-files-to
#!/bin/sh
get_references() {
grep -F -l --include=*.{xml,js,java} -r "$1" . | grep -v "$3" | while read -r subfile; do
#read each line of the grep result into the variable subfile
subfile="$(basename "$subfile")"
echo "$2""$subfile"
get_references "$subfile" ' '"$2" "$3"'\|'"$subfile"
done
}
while test $# -gt 0; do
#loop so more than one file can be given as argument to this script
echo "$1"
get_references "$1" '|__' "$1"
shift
done
There still are lots of performance enhancements possible.
Edit: Added $3 to prevent infinite-loop.

Cleaning up IP output on command line [duplicate]

This question already has answers here:
How to clean up masscan output (-oL)
(4 answers)
Closed 6 years ago.
I have a problem with the output L options ("grep-able" output); for instance, it outputs this:
| 14.138.12.21:123 | unknown | disabled |
| 14.138.184.122:123 | unknown | disabled |
| 14.138.179.27:123 | unknown | disabled |
| 14.138.20.65:123 | unknown | disabled |
| 14.138.12.235:123 | unknown | disabled |
| 14.138.178.97:123 | unknown | disabled |
| 14.138.182.153:123 | unknown | disabled |
| 14.138.178.124:123 | unknown | disabled |
| 14.138.201.191:123 | unknown | disabled |
| 14.138.180.26:123 | unknown | disabled |
| 14.138.13.129:123 | unknown | disabled |
The above is neither very readable nor easy to understand.
How can I use Linux command-line utilities, e.g. sed, awk, or grep, to output something as follows, using the file above?
output
14.138.12.21
14.138.184.122
14.138.179.27
14.138.20.65
14.138.12.235
Using awk with field separator as space, and : and getting the second field:
awk -F '[ :]' '{print $2}' file.txt
Example:
% cat file.txt
| 14.138.12.21:123 | unknown | disabled |
| 14.138.184.122:123 | unknown | disabled |
| 14.138.179.27:123 | unknown | disabled |
| 14.138.20.65:123 | unknown | disabled |
| 14.138.12.235:123 | unknown | disabled |
| 14.138.178.97:123 | unknown | disabled |
| 14.138.182.153:123 | unknown | disabled |
| 14.138.178.124:123 | unknown | disabled |
| 14.138.201.191:123 | unknown | disabled |
| 14.138.180.26:123 | unknown | disabled |
| 14.138.13.129:123 | unknown | disabled |
% awk -F '[ :]' '{print $2}' file.txt
14.138.12.21
14.138.184.122
14.138.179.27
14.138.20.65
14.138.12.235
14.138.178.97
14.138.182.153
14.138.178.124
14.138.201.191
14.138.180.26
14.138.13.129
AWK is perfect for cases when you want to split the file by "columns", and you know exactly that the order of values/columns is constant. AWK splits the lines by a field separator (which can be a regular expression like '[: ]'). The column names are accessible by their positions from the left: $1, $2, $3, etc.:
awk -F '[ :]' '{print $2}' src.log
awk -F '[ :|]' '{print $3}' src.log
awk 'BEGIN {FS="[ :|]"} {print $3}' src.log
You can also filter the lines with a regular expression:
awk -F '[ :]' '/138\.179\./ {print $2}' src.log
However, it is impossible to capture substrings with the regular expression groups.
SED is more flexible in regard to regular expressions:
sed -r 's/^[^0-9]*([0-9\.]+)\:.*/\1/' src.log
However, it lacks many useful features of the Perl-like regular expressions we used to use in every day programming. For example, even the extended syntax (-r) fails to interpret \d as a number.
Perhaps, Perl is the most flexible tool for parsing files. You can opt to simple expressions:
perl -n -e '/^\D*([^:]+):/ and print "$1\n"' src.log
or make the matching as strict as you like:
perl -n -e '/^\D*((?:\d{1,3}\.){3}\d{1,3}):/ and print "$1\n"' src.log
using sed
sed -r 's/^ *[|] *([0-9]+[.][0-9]+[.][0-9]+[.][0-9]+):[0-9]{3}.*/\1/

shell script to extract the name and IP address

Is there a way to use shell script to get only the name and net from the result as below:
Result
6cb7f14e-6466-4211-9a09-2b8e7ad92703 | name-erkoev4ja3rv | 2e3900ff36574cf9937d88223403da77 | ACTIVE | Running | net0=10.1.1.2; ing-net=10.1.1.3; net=10.1.1.4;
Expected Result
name-erkoev4ja3rv: 10.1.1.4
$ input="6cb7f14e-6466-4211-9a09-2b8e7ad92703 | name-erkoev4ja3rv | 2e3900ff36574cf9937d88223403da77 | ACTIVE | Running | net0=10.1.1.2; ing-net=10.1.1.3; net=10.1.1.4;"
$ echo "$input" | sed -E 's,^[^|]+ \| ([^ ]+).* net=([0-9.]+).*$,\1: \2,g'
name-erkoev4ja3rv: 10.1.1.4
echo "6cb7f14e-6466-4211-9a09-2b8e7ad92703 | name-erkoev4ja3rv | 2e3900ff36574cf9937d88223403da77 | ACTIVE | Running | net0=10.1.1.2; ing-net=10.1.1.3; net=10.1.1.4;" | awk -F ' ' '{print $3}{print $13}'
Does this satisfy your case?

Linux - Postgres psql retrieving undesired table

I've got the following problem:
There is a Postgres database which I need to get data from, via a Nagios Linux distribution.
My intention is to make a resulting SELECT be saved to a .txt, that would be sent via email to me using MUTT.
Until now, I've done:
#!/bin/sh
psql -d roaming -U thdroaming -o saida.txt << EOF
\d
\pset border 2
SELECT central, imsi, mapver, camel, nrrg, plmn, inoper, natms, cba, cbaz, stall, ownms, imsi_translation, forbrat FROM vw_erros_mgisp_totalizador
EOF
My problem is:
The .txt "saida.txt" is bringing me info about the database, as follows:
Lista de relações
Esquema | Nome | Tipo | Dono
---------+----------------------------------+-----------+------------
public | apns | tabela | jmsilva
public | config_imsis_centrais | tabela | thdroaming
public | config_imsis_sgsn | tabela | postgres
(3 Registers)
+---------+---------+----------+---------+---------+--------+------------+-------+---------+----------+-------+-------+------------------+-----------+
| central | imsi | mapver | camel | nrrg | plmn | inoper | natms | cba | cbaz | stall | ownms | imsi_translation | forbrat |
+---------+---------+----------+---------+---------+--------+------------+-------+---------+----------+-------+-------+------------------+-----------+
| MCTA02 | 20210 | | | | | INOPER-127 | | | | | | | |
| MCTA02 | 20404 | | | | | INOPER-127 | | | | | | | |
| MCTA02 | 20408 | | | | | INOPER-127 | | | | | | | |
| MCTA02 | 20412 | | | | | INOPER-127 | | | | | | | |
.
.
.
How could I make the first table not to be imported to the .txt?
Remove the '\d' portion of the script which causing listing the tables in the DB you see at the top of your output. So your script will become:
#!/bin/sh
psql -d roaming -U thdroaming -o saida.txt << EOF
\pset border 2
SELECT central, imsi, mapver, camel, nrrg, plmn, inoper, natms, cba, cbaz, stall, ownms, imsi_translation, forbrat FROM vw_erros_mgisp_totalizador
EOF
To get the output to appear CSV formatted in a file named /tmp/output.csv do you can do the following:
#!/bin/sh
psql -d roaming -U thdroaming -o saida.txt << EOF
\pset border 2
COPY (SELECT central, imsi, mapver, camel, nrrg, plmn, inoper, natms, cba, cbaz, stall, ownms, imsi_translation, forbrat FROM vw_erros_mgisp_totalizador) TO '/tmp/output.csv' WITH (FORMAT CSV)
EOF

Resources