I have a file with the following entries:
foop07_bar2_20190423152612.zip
foop07_bar1_20190423153115.zip
foop08_bar2_20190423152612.zip
foop08_bar1_20190423153115.zip
where
foop0* = host
bar* = fp
I would like to read the file and create 3 variables, the whole file name, host and fp (which stands for file_path_differentiator).
I am using read to take the first line and get my whole file name variable, I though I could then feed this into awk to grab the next two variables, however the first method of variable insertion creates an error and the second gives me all the variables.
I would like to loop each line, as I wish to use these variables to ssh to the host and grab the file
#!/bin/bash
while read -r FILE
do
echo ${FILE}
host=`awk 'BEGIN { FS = "_" } ; { print $1 }'<<<<"$FILE"`
echo ${host}
path=`awk -v var="${FILE}" 'BEGIN { FS = "_" } ; { print $2 }'`
echo ${path}
done <zips_not_received.csv
Expected Result
foop07_bar2_20190423152612.zip
foop07
bar2
foop07_bar1_20190423153115.zip
foop07
bar1
Actual Result
foop07_bar2_20190423152612.zip
/ : No such file or directoryfoop07_bar2_20190423152612.zip
bar2 bar1 bar2 bar1
You can do this alone with bash, without using any external tool.
while read -r file; do
[[ $file =~ (.*)_(.*)_.*\.zip ]] || { echo "invalid file name"; exit 1; }
host="${BASH_REMATCH[1]}"
path="${BASH_REMATCH[2]}"
echo "$file"
echo "$host"
echo "$path"
done < zips_not_received.csv
typical...
Managed to work a solution after posting...
#!/bin/bash
while read -r FILE
do
echo ${FILE}
host=`echo "$FILE" | awk -F"_" '{print $1}'`
echo $host
path=`echo "$FILE" | awk -F"_" '{print $2}'`
echo ${path}
done <zips_not_received.csv
not sure on the elegance or its correctness as i am using echo to create variable...but i have it working..
Assuming there is no space or _ in your "file name" that are part of the host or path
just separate line before with sed, awk, ... if using default space separator (or use _ as argument separator in batch). I add the remove of empty line value as basic security seeing your sample.
sed 's/_/ /g;/[[:blank:]]\{1,\}/d' zips_not_received.csv \
| while read host path Ignored
do
echo "${host}"
echo "${path}"
done
Related
I have a below properties file and would like to parse it as mentioned below. Please help in doing this.
.ini file which I created :
[Machine1]
app=version1
[Machine2]
app=version1
app=version2
[Machine3]
app=version1
app=version3
I am looking for a solution in which ini file should be parsed like
[Machine1]app = version1
[Machine2]app = version1
[Machine2]app = version2
[Machine3]app = version1
[Machine3]app = version3
Thanks.
Try:
$ awk '/\[/{prefix=$0; next} $1{print prefix $0}' file.ini
[Machine1]app=version1
[Machine2]app=version1
[Machine2]app=version2
[Machine3]app=version1
[Machine3]app=version3
How it works
/\[/{prefix=$0; next}
If any line begins with [, we save the line in the variable prefix and then we skip the rest of the commands and jump to the next line.
$1{print prefix $0}
If the current line is not empty, we print the prefix followed by the current line.
Adding spaces
To add spaces around any occurrence of =:
$ awk -F= '/\[/{prefix=$0; next} $1{$1=$1; print prefix $0}' OFS=' = ' file.ini
[Machine1]app = version1
[Machine2]app = version1
[Machine2]app = version2
[Machine3]app = version1
[Machine3]app = version3
This works by using = as the field separator on input and = as the field separator on output.
I love John1024's answer. I was looking for exactly that. I have created a bash function that allows me to lookup sections or specific keys based on his idea:
function iniget() {
if [[ $# -lt 2 || ! -f $1 ]]; then
echo "usage: iniget <file> [--list|<section> [key]]"
return 1
fi
local inifile=$1
if [ "$2" == "--list" ]; then
for section in $(cat $inifile | grep "\[" | sed -e "s#\[##g" | sed -e "s#\]##g"); do
echo $section
done
return 0
fi
local section=$2
local key
[ $# -eq 3 ] && key=$3
# https://stackoverflow.com/questions/49399984/parsing-ini-file-in-bash
# This awk line turns ini sections => [section-name]key=value
local lines=$(awk '/\[/{prefix=$0; next} $1{print prefix $0}' $inifile)
for line in $lines; do
if [[ "$line" = \[$section\]* ]]; then
local keyval=$(echo $line | sed -e "s/^\[$section\]//")
if [[ -z "$key" ]]; then
echo $keyval
else
if [[ "$keyval" = $key=* ]]; then
echo $(echo $keyval | sed -e "s/^$key=//")
fi
fi
fi
done
}
So given this as file.ini
[Machine1]
app=version1
[Machine2]
app=version1
app=version2
[Machine3]
app=version1
app=version3
then the following results are produced
$ iniget file.ini --list
Machine1
Machine2
Machine3
$ iniget file.ini Machine3
app=version1
app=version3
$ iniget file.ini Machine1 app
version1
$ iniget file.ini Machine2 app
version2
version3
Again, thanks to #John1024 for his answer, I was pulling my hair out trying to create a simple bash ini parser that supported sections.
Tested on Mac using GNU bash, version 5.0.0(1)-release (x86_64-apple-darwin18.2.0)
You can try using awk:
awk '/\[[^]]*\]/{ # Match pattern like [...]
a=$1;next # store the pattern in a
}
NF{ # Match non empty line
gsub("=", " = ") # Add space around the = character
print a $0 # print the line
}' file
Excellent answers here. I made some modifications to #davfive's function to fit it better to my use case. This version is largely the same except it allows for whitespace before and after = characters, and allows values to have spaces in them.
# Get values from a .ini file
function iniget() {
if [[ $# -lt 2 || ! -f $1 ]]; then
echo "usage: iniget <file> [--list|<section> [key]]"
return 1
fi
local inifile=$1
if [ "$2" == "--list" ]; then
for section in $(cat $inifile | grep "^\\s*\[" | sed -e "s#\[##g" | sed -e "s#\]##g"); do
echo $section
done
return 0
fi
local section=$2
local key
[ $# -eq 3 ] && key=$3
# This awk line turns ini sections => [section-name]key=value
local lines=$(awk '/\[/{prefix=$0; next} $1{print prefix $0}' $inifile)
lines=$(echo "$lines" | sed -e 's/[[:blank:]]*=[[:blank:]]*/=/g')
while read -r line ; do
if [[ "$line" = \[$section\]* ]]; then
local keyval=$(echo "$line" | sed -e "s/^\[$section\]//")
if [[ -z "$key" ]]; then
echo $keyval
else
if [[ "$keyval" = $key=* ]]; then
echo $(echo $keyval | sed -e "s/^$key=//")
fi
fi
fi
done <<<"$lines"
}
For taking disparate sectional and tacking the section name (including 'no-section'/Default together) to each of its related keyword (along with = and its keyvalue), this one-liner AWK will do the trick coupled with a few clean-up regex.
ini_buffer="$(echo "$raw_buffer" | awk '/^\[.*\]$/{obj=$0}/=/{print obj $0}')"
Will take your lines and output them like you wanted:
+++ awk '/^\[.*\]$/{obj=$0}/=/{print obj $0}'
++ ini_buffer='[Machine1]app=version1
[Machine2]app=version1
[Machine2]app=version2
[Machine3]app=version1
[Machine3]app=version3'
A complete solution to the INI-format File
As Clonato, INI-format expert said that for the latest INI version 1.4 (2009-10-23), there are several other tricky aspects to the INI file:
character set constraint for section name
character set constraint for keyword
And lastly is for the keyvalue to be able to handle pretty much anthing that is not used in the section and keyword name; that includes nesting of quotes inside a pair of same single/double-quote.
Except for the nesting of quotes, a INI-format Github complete solution to parsing INI-format file with default section:
# syntax: ini_file_read <raw_buffer>
# outputs: formatted bracket-nested "[section]keyword=keyvalue"
ini_file_read()
{
local ini_buffer raw_buffer hidden_default
raw_buffer="$1"
# somebody has to remove the 'inline' comment
# there is a most complex SED solution to nested
# quotes inline comment coming ... TBA
raw_buffer="$(echo "$raw_buffer" | sed '
s|[[:blank:]]*//.*||; # remove //comments
s|[[:blank:]]*#.*||; # remove #comments
t prune
b
:prune
/./!d; # remove empty lines, but only those that
# become empty as a result of comment stripping'
)"
# awk does the removal of leading and trailing spaces
ini_buffer="$(echo "$raw_buffer" | awk '/^\[.*\]$/{obj=$0}/=/{print obj $0}')" # original
ini_buffer="$(echo "$ini_buffer" | sed 's/^\s*\[\s*/\[/')"
ini_buffer="$(echo "$ini_buffer" | sed 's/\s*\]\s*/\]/')"
# finds all 'no-section' and inserts '[Default]'
hidden_default="$(echo "$ini_buffer" \
| egrep '^[-0-9A-Za-z_\$\.]+=' | sed 's/^/[Default]/')"
if [ -n "$hidden_default" ]; then
echo "$hidden_default"
fi
# finds sectional and outputs as-is
echo "$(echo "$ini_buffer" | egrep '^\[\s*[-0-9A-Za-z_\$\.]+\s*\]')"
}
The unit test for this StackOverflow post is included in this file:
https://github.com/egberts/bash-ini-file
Source:
https://github.com/egberts/easy-admin/blob/main/test/section-regex.sh
https://cloanto.com/specs/ini/#escapesequences
I need to read a file that has lines like
user=username1
pass=password1
How can I read multiple lines like this into separate variables like username and password?
Would I use awk or grep? I have found ways to read lines into variables with grep but would I need to read the file for each individual item?
The end result is to use these variables to access a database via the command line. So I need to be able to read, store and use these values in other commands.
if the process which generates the file is safe and has shell syntax just source the file.
. ./file
Otherwise the file can be processes before to add quotes
perl -ne 'if (/^([A-Za-z_]\w*)=(.*)/) {$k=$1;$v=$2;$v=~s/\x27/\x27\\\x27\x27/g;print "$k=\x27$v\x27\n";}' <file >file2
. ./file2
If you want to use awk then
Input
$ cat file
user=username1
pass=password1
Reading
$ user=$(awk -F= '$1=="user"{print $2;exit}' file)
$ pass=$(awk -F= '$1=="pass"{print $2;exit}' file)
Output
$ echo $user
username1
$ echo $pass
password1
You could use a loop for your file perhaps, but this is probably the functionality you're looking for.
$ echo 'user=username1' | awk -F= '{print $2}'
username1
Using the -F flag sets the delimiter to = and we select the 2nd item from the row.
file.txt:
user=username1
pass=password1
user=username2
pass=password2
user=username3
pass=password3
Do to avoid browsing several times the file file.txt:
#!/usr/bin/env bash
func () {
echo "user:$1 pass:$2"
}
i=0
while IFS='' read -r line; do
if [ $i -eq 0 ]; then
i=1
user=$(echo ${line} | cut -f2 -d'=')
else
i=0
pass=$(echo ${line} | cut -f2 -d'=')
func "$user" "$pass"
fi
done < file.txt
Output:
user:username1 pass:password1
user:username2 pass:password2
user:username3 pass:password3
In the sample bellow, I would like to print text on the screen and also append this text into file, when the variable cpstdout is set to 1. Otherwise only print the text on screen. I need to have the echo part flexible to the append variable. Is there any way to correct my code?
#!/bin/ksh
cpstdout=1
if [ $cpstdout -eq 1 ]; then
append="| tee somefile"
else
append=""
fi
echo "test string" $append
Now the result is just like this:
./test.sh
test string | tee somefile
-no file is created of course
example of print function:
print_output(){
printf "\t/-------------------------------------------------\\ \n"
for i in "$#"; do
printf "\t| %-14s %-32s |\n" "$(echo $i | awk -F, '{print $1}')" "$(echo $i | awk -F, '{print $2}')"
shift
done
printf "\t\-------------------------------------------------/\n"
}
Define your appending command as a function:
output_with_append() {
tee -a somefile <<<"$1"
}
Then, in the if, set a variable to the appropriate outputting function:
if [ $cpstdout -eq 1 ]; then
output=output_with_append
else
output=echo
fi
Finally, use variable expansion to run the command:
$output "test_string"
Note that I've used tee -a since you said you wanted to append to a file and not overwrite it.
Setting cpstdout to $1 so we can control it through a command-line parameter:
cpstdout="$1"
A example session then looks like this:
$ ./test.sh 1
test_string
$ ./test.sh 1
test_string
$ cat somefile
test_string
test_string
$ ./test.sh 0
test_string
$ cat somefile
test_string
test_string
I've got a txt file with several lines, each one describing a remote server, like this:
user#server:port:remote_working_path:whether_using_VPN
The : char separates the 4 fields.
I need to operate batch actions within each server, hence I need to parse each line and set appropriate variables. Right now, what I've coded is this:
while read server;
do
echo "$server" | awk -F ':' '{print $1}' &&
echo "$server" | awk -F ':' '{print $2}' &&
echo "$server" | awk -F ':' '{print $3}'
echo "$VPN"
declare $( echo "$server" | awk -F ':' '{print $VPN=$4}' )
echo 'VPN: '$VPN
done < $CUSTOMER_SERVERS_FILE
This script only prints the first 3 fields, and in my intentions should also set $VPN variable as the 4th field. However this seems way broken, and I'm being unable to fix it. How should I modify it so that $VPN = $4?
First, you don't need to use awk in this case. You could try to use something like :
while IFS=':' read -ra array; do
# "${array[0]}" => first field
# "${array[1]}" => second field
# ...
# "${array[#]}" => all fields
done < "$CUSTOMER_SERVERS_FILE"
Then if you want to set VPN variable with the 4th field, you could use :
while IFS=':' read -ra array; do
# ...
VPN="${array[3]}"
done < "$CUSTOMER_SERVERS_FILE"
Another solution :
while IFS=':' read -r address port path vpn trash; do
# The variables $adress $port $path and $vpn are assigned.
# $trash is set with other fields if there are more than 4 fields
done
Finally, when you want to assign the output of a command in a variable, you could do :
var="$(command)"
# or
var="`command`"
I'm writing a bash script to modify a config file which contains a bunch of key/value pairs. How can I read the key and find the value and possibly modify it?
A wild stab in the dark for modifying a single value:
sed -c -i "s/\($TARGET_KEY *= *\).*/\1$REPLACEMENT_VALUE/" $CONFIG_FILE
assuming that the target key and replacement value don't contain any special regex characters, and that your key-value separator is "=". Note, the -c option is system dependent and you may need to omit it for sed to execute.
For other tips on how to do similar replacements (e.g., when the REPLACEMENT_VALUE has '/' characters in it), there are some great examples here.
Hope this helps someone. I created a self contained script, which required config processing of sorts.
#!/bin/bash
CONFIG="/tmp/test.cfg"
# Use this to set the new config value, needs 2 parameters.
# You could check that $1 and $2 is set, but I am lazy
function set_config(){
sudo sed -i "s/^\($1\s*=\s*\).*\$/\1$2/" $CONFIG
}
# INITIALIZE CONFIG IF IT'S MISSING
if [ ! -e "${CONFIG}" ] ; then
# Set default variable value
sudo touch $CONFIG
echo "myname=\"Test\"" | sudo tee --append $CONFIG
fi
# LOAD THE CONFIG FILE
source $CONFIG
echo "${myname}" # SHOULD OUTPUT DEFAULT (test) ON FIRST RUN
myname="Erl"
echo "${myname}" # SHOULD OUTPUT Erl
set_config myname $myname # SETS THE NEW VALUE
Assuming that you have a file of key=value pairs, potentially with spaces around the =, you can delete, modify in-place or append key-value pairs at will using awk even if the keys or values contain special regex sequences:
# Using awk to delete, modify or append keys
# In case of an error the original configuration file is left intact
# Also leaves a timestamped backup copy (omit the cp -p if none is required)
CONFIG_FILE=file.conf
cp -p "$CONFIG_FILE" "$CONFIG_FILE.orig.`date \"+%Y%m%d_%H%M%S\"`" &&
awk -F '[ \t]*=[ \t]*' '$1=="keytodelete" { next } $1=="keytomodify" { print "keytomodify=newvalue" ; next } { print } END { print "keytoappend=value" }' "$CONFIG_FILE" >"$CONFIG_FILE~" &&
mv "$CONFIG_FILE~" "$CONFIG_FILE" ||
echo "an error has occurred (permissions? disk space?)"
sed "/^$old/s/\(.[^=]*\)\([ \t]*=[ \t]*\)\(.[^=]*\)/\1\2$replace/" configfile
So I can not take any credit for this as it is a combination of stackoverflow answers and help from irc.freenode.net #bash channel but here are bash functions now to both set and read config file values:
# https://stackoverflow.com/a/2464883
# Usage: config_set filename key value
function config_set() {
local file=$1
local key=$2
local val=${#:3}
ensureConfigFileExists "${file}"
# create key if not exists
if ! grep -q "^${key}=" ${file}; then
# insert a newline just in case the file does not end with one
printf "\n${key}=" >> ${file}
fi
chc "$file" "$key" "$val"
}
function ensureConfigFileExists() {
if [ ! -e "$1" ] ; then
if [ -e "$1.example" ]; then
cp "$1.example" "$1";
else
touch "$1"
fi
fi
}
# thanks to ixz in #bash on irc.freenode.net
function chc() { gawk -v OFS== -v FS== -e 'BEGIN { ARGC = 1 } $1 == ARGV[2] { print ARGV[4] ? ARGV[4] : $1, ARGV[3]; next } 1' "$#" <"$1" >"$1.1"; mv "$1"{.1,}; }
# https://unix.stackexchange.com/a/331965/312709
# Usage: local myvar="$(config_get myvar)"
function config_get() {
val="$(config_read_file ${CONFIG_FILE} "${1}")";
if [ "${val}" = "__UNDEFINED__" ]; then
val="$(config_read_file ${CONFIG_FILE}.example "${1}")";
fi
printf -- "%s" "${val}";
}
function config_read_file() {
(grep -E "^${2}=" -m 1 "${1}" 2>/dev/null || echo "VAR=__UNDEFINED__") | head -n 1 | cut -d '=' -f 2-;
}
at first I was using the accepted answer's sed solution: https://stackoverflow.com/a/2464883/2683059
however if the value has a / char it breaks
in general it's easy to extract the info with grep and cut:
cat "$FILE" | grep "^${KEY}${DELIMITER}" | cut -f2- -d"$DELIMITER"
to update you could do something like this:
mv "$FILE" "$FILE.bak"
cat "$FILE.bak" | grep -v "^${KEY}${DELIMITER}" > "$FILE"
echo "${KEY}${DELIMITER}${NEWVALUE}" >> "$FILE"
this would not maintain the order of the key-value pairs obviously. add error checking to make sure you don't lose your data.
I have done this:
new_port=$1
sed "s/^port=.*/port=$new_port/" "$CONFIG_FILE" > /yourPath/temp.x
mv /yourPath/temp.x "$CONFIG_FILE"
This will change port= to port=8888 in your config file if you choose 8888 as $1 for example.
Suppose your config file is in below format:
CONFIG_NUM=4
CONFIG_NUM2=5
CONFIG_DEBUG=n
In your bash script, you can use:
CONFIG_FILE=your_config_file
. $CONFIG_FILE
if [ $CONFIG_DEBUG == "y" ]; then
......
else
......
fi
$CONFIG_NUM, $CONFIG_NUM2, $CONFIG_DEBUG is what you need.
After your read the values, write it back will be easy:
echo "CONFIG_DEBUG=y" >> $CONFIG_FILE