I want to break down a file that has multiple lines that follow this style:
e-mail;year/month/date;groups;sharedFolder
An example line from file:
alan.turing#cam.ac.uk;1912/06/23;visitor;/visitorData
Essentially I want to break each line up into four arrays that can be accessed later on in a loop to create a new user for each line.
I have declared the arrays already have a file saved as variable 'filename'
Usernames need to be the first three letters of the surname and the first three letters of the first name.
Passwords need to be the users birthdate as day/month/year.
So far this is what I have. Am I on the right track? Are there places I have gone wrong or could improve on?
#reads file and saves into appropriate arrays
while read -r line
do
IFS = $';' read -r -a array <<< "$line"
mailArray += "$(array[0])"
dateArray += "$(array[1])"
groupArray += "$(array[2])"
folderArray += "$(array[3])"
done < $filename
#create usernames from emails
for i in "$(mailArray[#])"
do
IFS=$'.' read -r -a array <<< "$i"
part1 = ${array[0]:0:3}
part2 = ${array[1]:0:3}
user = $part2
user .= $part1
userArray += ("$user")
done
#create passwords from birthdates
for i in "$(dateArray[#])"
do
IFS=$'/' read -r -a array <<< "$i"
password = $part3
password .= $part2
password .= $part1
passArray += ("$password")
done
Not sure if arrays are required here, and if you just want to create username, password from the lines in the desired format, please see below:
# Sample Input Data:
bash$> cat d
alan.turing#cam.ac.uk;1912/06/23;visitor;/visitorData
rob.zombie#cam.ac.uk;1966/06/23;metalhead;/metaldata
donald.trump#stupid.com;1900/00/00;idiot;/idiotique
bash$>
# Sample Output from script:
bash$> ./dank.sh "d"
After processing the line [alan.turing#cam.ac.uk;1912/06/23;visitor;/visitorData], we have the following parameters extracted:
Name: alan
Surname: turing
Birthdate: 1912/06/23
username: alatur
Password: 1912/06/23
After processing the line [rob.zombie#cam.ac.uk;1966/06/23;metalhead;/metaldata], we have the following parameters extracted:
Name: rob
Surname: zombie
Birthdate: 1966/06/23
username: robzom
Password: 1966/06/23
After processing the line [donald.trump#stupid.com;1900/00/00;idiot;/idiotique], we have the following parameters extracted:
Name: donald
Surname: trump
Birthdate: 1900/00/00
username: dontru
Password: 1900/00/00
Now the script, which does this operation.
# Script.
bash$> cat dank.sh
#!/bin/bash
cat "$1" | while read line
do
name=`echo $line | sed 's/^\(.*\)\..*\#.*$/\1/g'`
surname=`echo $line | sed 's/^.*\.\(.*\)\#.*$/\1/g'`
bdate=`echo $line | sed 's/^.*;\(.*\);.*;.*$/\1/g'`
temp1=`echo $name | sed 's/^\(...\).*$/\1/g'`
temp2=`echo $surname | sed 's/^\(...\).*$/\1/g'`
uname="${temp1}${temp2}"
echo "
After processing the line [$line], we have the following parameters extracted:
Name: $name
Surname: $surname
Birthdate: $bdate
username: $uname
Password: $bdate
"
done
bash$>
Basically, i am just running couple of sed commands to extract what is useful and required, and storing them in variables and then one could use them anyways they want to. You could redirect them to a file if you want or print out a pipe separated output.. upto you.
Let me know..
Related
$string="name: Destination Administrator
description: Manage the destination configurations, certificates and subaccount trust.
readOnly:
roleReferences:
- roleTemplateAppId: destination-xsappname!b62
roleTemplateName: Destination_Administrator
name: Destination Administrator"
I have above string each line is delimited by newline char, and I like to create array with two column after "-" as below
Col1 col2
roleTemplateAppId destination-xsappname!b62
roleTemplateName Destination_Administrator
name Destination Administrator
I tried below but it is not returning correct array
IFS='- ' read -r -a arrstring <<< "$string"
echo "${arrstring [1]}"
Assumptions:
OP is unable to use a yaml parser (per Léa's comment)
the input is guaranteed to have \n line endings (within the data)
the - only shows up in the one location (as depicted in OP's sample input); otherwise we need a better definition of where to start parsing the data
we're interested in parsing everything that follows the -
data is to be parsed based on a : delimiter, with the first field serving as the index in an associative array, while the 2nd field will be the value stored in the array
leading/trailing spaces to be removed from array indexes and values
One sed idea for pulling out just the lines we're interested in:
$ sed -n '/- /,${s/-//;p}' <<< "${string}"
roleTemplateAppId: destinationxsappname!b62
roleTemplateName: Destination_Administrator
name: Destination Administrator
Adding a few more bits to strip off leading/trailing spaces:
$ sed -n '/- /,${s/-//;s/^[ ]*//;s/[ ]*$//;s/[ ]*:[ ]*/:/;p}' <<< "${string}"
roleTemplateAppId:destination-xsappname!b62
roleTemplateName:Destination_Administrator
name:Destination Administrator
From here we'll feed this to a while loop where we'll populate the associative array
unset arrstring
declare -A arrstring # declare as an associative array
while IFS=':' read -r index value
do
arrstring["${index}"]="${value}"
done < <(sed -n '/- /,${s/-//;s/^[ ]*//;s/[ ]*$//;s/[ ]*:[ ]*/:/;p}' <<< "${string}")
Leaving us with:
$ typeset -p arrstring
declare -A arrstring=([roleTemplateAppId]="destination-xsappname!b62" [name]="Destination Administrator" [roleTemplateName]="Destination_Administrator" )
$ for i in "${!arrstring[#]}"
do
echo "$i : ${arrstring[$i]}"
done
roleTemplateAppId : destination-xsappname!b62
name : Destination Administrator
roleTemplateName : Destination_Administrator
Would first like to thank everyone for taking the time and reviewing this, and providing some assistance.
I am stuck on this bash script project I have been working on. This script is supposed to pull data from this file, export it to a csv, and then email it out. I was able to grab the required data and email it to myself but the problem is that the groups in the file have special characters. I need to have the lsgroup command executed on those groups in order to retrieve the users and then have it exported to the csv file.
For example, below is sample data that are in the file and how it looks like:
[Skyrim]
comment = Elder Scrolls
path = /export/skyrim/elderscrolls
valid users = #dawnstar nords #riften
invalid users = #lakers
[SONY]
comment = PS4
path = /export/Sony/PS4
valid users = #insomniac #activision
invalid users = peterparker controller #pspro
The script is supposed to be gathering the name, comment, path, valid users, invalid users, and exporting them to the csv
So far this is what I have that works,
out="/tmp/parsed-report.csv"
file="/tmp/file.conf"
echo "name,comment,path,valid_users,invalid_users" > $out;
scp -q server:/tmp/parse/file.conf $out
grep "^\[.*\]$" $file |grep -Ev 'PasswordPickup|global' | while read shr ; do
shr_regex=$(echo "$shr" | sed 's/[][]/\\&/g')
shr_print=$(echo "$shr"|sed 's/[][]//g')
com=$(grep -p "$shr_regex" $file|grep -v "#"| grep -w "comment"| awk -F'=' '{print $2}'|sed 's/,/ /g')
path=$(grep -p "$shr_regex" $file|grep -v "#"| grep -w "path"| awk -F'=' '{print $2}')
val=$(grep -p "$shr_regex" $file|grep -v "#"| grep -w "valid users"| awk -F'=' '{print $2}')
inv=$(grep -p "$shr_regex" $file|grep -v "#"| grep -w "invalid users"| awk -F'=' '{print$2}')
echo "$shr_print,$com,$path,$val,$inv" >> $out
done
exit 0
The text with '#' are considered groups so if $var3='#' then run the lsgroup command and export the data to csv file under the correct category, else if $vars3!='#' then export users to the csv file.
This is what I tried to come up with:
vars3="$val$inv"
Server="server_1"
for lists in $(echo "$vars3"); do
if [[ $lists = *[!\#]* ]]; then
ssh -q $Server "lsgroup -a users $(echo "$lists"|tr -d /#/)|awk -
F'=' '{print $1}'" > print to csv file as valid or invalid users
else [[ $lists != *[!\#]* ]]; then
echo "users without #" > to csv file as valid or invalid users
With the right commands the output should look like this
: skyrim
Comment: Elder Scrolls
Path: /export/skyrim/elderscrolls
Valid Users: dragonborn argonian kajit nords
Invalid Users : Shaq Kobe Phil Lebron
: SONY
Comment: PS4
Path: /export/Sony/PS4
Valid Users: spiderman ratchet&clank callofduty spyro
Invalid Users : peterparker controller 4k
Create a file file.sed with this content:
s/\[/: / # replace [ with : and one space
s/]// # remove ]
s/^ // # remove leading spaces
s/ = /: /
s/#lakers/Shaq Kobe Phil Lebron/
s/^comment/Comment/
# Can be completed by you here.
and then use
sed -f file.sed your_sample_data_file
Output:
: Skyrim
Comment: Elder Scrolls
path: /export/skyrim/elderscrolls
valid users: #dawnstar nords #riften
invalid users: Shaq Kobe Phil Lebron
: SONY
Comment: PS4
path: /export/Sony/PS4
valid users: #insomniac #activision
invalid users: peterparker controller #pspro
Parsing things is a hard problem and, in my opinion, writing your own parser is unproductive.
Instead, I highly advise you to take your time and learn about grammars and parsing generators. Then you can use some battle tested library such as textX to implement your parser.
Here is part of my data, I need new line characters to be removed from the lines, excluding those lines ending with a sequence matching the format |HH:MM:SS.
Here are the first two records. First record started with "Reset system" and second with "Collaborator informs".
Reset system password SISMED WE|Collaborator requests password reset of SISMED WEB system.
Login: John Doe
Nome: Jackie
Locat: D. XYZ – UA ABC Al
Setor/Depto: Administration
Floor: 1st
Tel./Ramal: 358-108|14/01/2015 |11:23:22
Collaborator informs that he can not open archiv ... |Collaborator informs you that you can not open files
Path: \\abc\def\ghi\jkl\mno
File: ESCALAS.xls
Name: Hutch cock
Locat: D. Al Mo
Setor/Depto: Hos
Floor: 2nd
Tel./Ramal: 1521
IP: 1.5.2.14|14/01/2015 |11:26:21
I need output some thing like below
Reset system password SISMED WE|Collaborator requests password reset of SISMED WEB system.Login: John Doe Nome: Jackie Locat: D. XYZ – UA ABC Al Setor/Depto: Administration Floor: 1st Tel./Ramal: 358-108|14/01/2015 |11:23:22
Collaborator informs that he can not open archiv ... |Collaborator informs you that you can not open files Path: \\abc\def\ghi\jkl\mno File: ESCALAS.xls Name: Hutch cock Locat: D. Al Mo Setor/Depto: Hos Floor: 2nd Tel./Ramal: 1521 IP: 1.5.2.14|14/01/2015 |11:26:21
Can some body please help me with UNIX commands.
Thank you.
In native bash, aiming for readability over terseness:
#!/usr/bin/env bash
# if we were passed a filename as an argument, read from that file
# otherwise, this script reads from stdin
[[ $1 ]] && exec <"$1"
# ERE-syntax regex matching end-of-record marker
end_of_record_re='[|][[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}[[:space:]]*$'
buffer='' # start out with an empty buffer
while IFS= read -r line; do # while we can, read a line.
if ! [[ $line =~ $end_of_record_re ]]; then # unless it has an end marker...
buffer+=" $line" # ...add to our buffer, preceded by a space
else # if the line has an end marker...
printf '%s\n' "${buffer# }${line}" # ...print buffer except for first space
buffer= # ...and reset the buffer to be empty
fi
done
# finally, if we have trailing content, print it out.
[[ $buffer ]] && printf '%s\n' "${buffer# }"
I sorted the output from conflicting numbers that are already existing in my DB with new insert request to a file name output.
This is the output of the file.
cat output
DataBase: (9999999999) NewDB_insert: (999-999-9999)
DataBase: (1111111111) NewDB_insert: (111-111-1111)
DataBase: (2222222222) NewDB_insert: (222-222-2222)
DataBase: (3333333333) NewDB_insert: (333-333-3333)
I want to run the same command that displays information from the same line that has the conflict. The command just pulls the DB information.
showinfo -id 9999999999
ID:9999999999
Name: Mr.Brown
City: New York
Company: Acme
Client: 1245
showinfo -id 999-999-9999
ID:999-999-9999
Name: Mr.Brown
City: New York
Company: Acme
Client: 1245
I want to use 2 variables to send the output to the screen from my file named output. So I would have the following information from my file.
**From DB:**
ID:9999999999
Name: Mr.Brown
City: New York
Company: Acme
Client: 1245
**From NewDB_insert:**
showinfo -id 999-999-9999
ID:999-999-9999
Name: Mr.Brown
City: New York
Company: Acme
Client: 1245
So basically I would cat the file in a for loop with one variable as so.
for i in `cat output | awk '{print $2}' | tr -d ")("`;do echo == From DB:==;showinfo -id $i; echo ==========;done
Which gives me the first part of what I need. I looked online and not able to use any examples to create a for loop with 2 variables. Any help would be appreciated.
The short answer is probably: "you don't".
You're probably best off using a while loop with read.
sed -e 's/[^(]*(\([^)]*\))[^(]*(\([^)]*\)).*/\1 \2/' output |
while read a b
do
echo == From DB ==
showinfo -id $a
echo =============
echo == From NewDB_Insert ==
showinfo -id $b
echo =======================
done
If you think there might be nasty characters, especially backslashes, in your data, you can use read -r and you can quote "$a" and "$b" in your showinfo lines. If you're sure there won't be any such characters, you're (just about) OK with the code as written.
Note that any variables set in the while loop as written won't be accessible when the loop completes. There are ways around that problem if it is, indeed, a problem.
The sed script looks for:
zero or more non-open-parentheses
an open parenthesis
captures zero or more non-close-parentheses
stops capturing
a close parenthesis
zero of more non-open-parentheses
an open parenthesis
captures zero of more non-close-parentheses
stops capturing
a close parenthesis
zero or more other characters
and replaces them with the two captured strings (e.g. 9999999999 and 999-999-9999) separated by a space.
I am creating a script that will take user input for a file name, and number of files to be created.
The script will then create the number of files required, and use the name given but increment the file name by one each iteration.
So lets say I want 3 files, named boogey. The output in the home folder would be; boogey1.txt, boogey2.txt and boogey3.txt.
Here is the code I have so far.
#!/bin/bash
var1=1
echo -n "Enter the number of your files to create: [ENTER]: "
read file_number
var2=file_number
echo -n "Enter the name of your file: [ENTER]: "
read file_name
var3=file_name
while [ "$var1" -le $file_number]
do
cat $file_name.txt
#var1=$(( var1+1 ))
done
Your script is almost working. You just need to:
uncomment that var1 assignment line
add a space between $file_number and ] on the while loop line
change cat to touch (or any of a number of other possibilities) since cat doesn't create files it read them.
Use $file_name$var1.txt in the cat (now touch) line to actually use the incremented index instead of the bare filename.
That being said this script could be dramatically improved. See below for an example.
#!/bin/bash
read -p 'Enter the number of your files to create: [ENTER]: ' file_number
read -p 'Enter the name of your file: [ENTER]: ' file_name
for ((i = 1; i < file_number; i++); do
: > "$file_name$i.txt"
done