I'm working on a long Bash script. I want to read cells from a CSV file into Bash variables. I can parse lines and the first column, but not any other column. Here's my code so far:
cat myfile.csv|while read line
do
read -d, col1 col2 < <(echo $line)
echo "I got:$col1|$col2"
done
It's only printing the first column. As an additional test, I tried the following:
read -d, x y < <(echo a,b,)
And $y is empty. So I tried:
read x y < <(echo a b)
And $y is b. Why?
You need to use IFS instead of -d:
while IFS=, read -r col1 col2
do
echo "I got:$col1|$col2"
done < myfile.csv
To skip a given number of header lines:
skip_headers=3
while IFS=, read -r col1 col2
do
if ((skip_headers))
then
((skip_headers--))
else
echo "I got:$col1|$col2"
fi
done < myfile.csv
Note that for general purpose CSV parsing you should use a specialized tool which can handle quoted fields with internal commas, among other issues that Bash can't handle by itself. Examples of such tools are cvstool and csvkit.
How to parse a CSV file in Bash?
Coming late to this question and as bash do offer new features, because this question stand about bash and because none of already posted answer show this powerful and compliant way of doing precisely this.
Parsing CSV files under bash, using loadable module
Conforming to RFC 4180, a string like this sample CSV row:
12,22.45,"Hello, ""man"".","A, b.",42
should be splitted as
1 12
2 22.45
3 Hello, "man".
4 A, b.
5 42
bash loadable .C compiled modules.
Under bash, you could create, edit, and use loadable c compiled modules. Once loaded, they work like any other builtin!! ( You may find more information at source tree. ;)
Current source tree (Oct 15 2021, bash V5.1-rc3) do contain a bunch of samples:
accept listen for and accept a remote network connection on a given port
asort Sort arrays in-place
basename Return non-directory portion of pathname.
cat cat(1) replacement with no options - the way cat was intended.
csv process one line of csv data and populate an indexed array.
dirname Return directory portion of pathname.
fdflags Change the flag associated with one of bash's open file descriptors.
finfo Print file info.
head Copy first part of files.
hello Obligatory "Hello World" / sample loadable.
...
tee Duplicate standard input.
template Example template for loadable builtin.
truefalse True and false builtins.
tty Return terminal name.
uname Print system information.
unlink Remove a directory entry.
whoami Print out username of current user.
There is an full working cvs parser ready to use in examples/loadables directory: csv.c!!
Under Debian GNU/Linux based system, you may have to install bash-builtins package by
apt install bash-builtins
Using loadable bash-builtins:
Then:
enable -f /usr/lib/bash/csv csv
From there, you could use csv as a bash builtin.
With my sample: 12,22.45,"Hello, ""man"".","A, b.",42
csv -a myArray '12,22.45,"Hello, ""man"".","A, b.",42'
printf "%s\n" "${myArray[#]}" | cat -n
1 12
2 22.45
3 Hello, "man".
4 A, b.
5 42
Then in a loop, processing a file.
while IFS= read -r line;do
csv -a aVar "$line"
printf "First two columns are: [ '%s' - '%s' ]\n" "${aVar[0]}" "${aVar[1]}"
done <myfile.csv
This way is clearly the quickest and strongest than using any other combination of bash builtins or fork to any binary.
Unfortunely, depending on your system implementation, if your version of bash was compiled without loadable, this may not work...
Complete sample with multiline CSV fields.
Conforming to RFC 4180, a string like this single CSV row:
12,22.45,"Hello ""man"",
This is a good day, today!","A, b.",42
should be splitted as
1 12
2 22.45
3 Hello "man",
This is a good day, today!
4 A, b.
5 42
Full sample script for parsing CSV containing multilines fields
Here is a small sample file with 1 headline, 4 columns and 3 rows. Because two fields do contain newline, the file are 6 lines length.
Id,Name,Desc,Value
1234,Cpt1023,"Energy counter",34213
2343,Sns2123,"Temperatur sensor
to trigg for alarm",48.4
42,Eye1412,"Solar sensor ""Day /
Night""",12199.21
And a small script able to parse this file correctly:
#!/bin/bash
enable -f /usr/lib/bash/csv csv
file="sample.csv"
exec {FD}<"$file"
read -ru $FD line
csv -a headline "$line"
printf -v fieldfmt '%-8s: "%%q"\\n' "${headline[#]}"
numcols=${#headline[#]}
while read -ru $FD line;do
while csv -a row "$line" ; (( ${#row[#]} < numcols )) ;do
read -ru $FD sline || break
line+=$'\n'"$sline"
done
printf "$fieldfmt\\n" "${row[#]}"
done
This may render: (I've used printf "%q" to represent non-printables characters like newlines as $'\n')
Id : "1234"
Name : "Cpt1023"
Desc : "Energy\ counter"
Value : "34213"
Id : "2343"
Name : "Sns2123"
Desc : "$'Temperatur sensor\nto trigg for alarm'"
Value : "48.4"
Id : "42"
Name : "Eye1412"
Desc : "$'Solar sensor "Day /\nNight"'"
Value : "12199.21"
You could find a full working sample there: csvsample.sh.txt or
csvsample.sh.
Note:
In this sample, I use head line to determine row width (number of columns). If you're head line could hold newlines, (or if your CSV use more than 1 head line). You will have to pass number or columns as argument to your script (and the number of head lines).
Warning:
Of course, parsing CSV using this is not perfect! This work for many simple CSV files, but care about encoding and security!! For sample, this module won't be able to handle binary fields!
Read carefully csv.c source code comments and RFC 4180!
From the man page:
-d delim
The first character of delim is used to terminate the input line,
rather than newline.
You are using -d, which will terminate the input line on the comma. It will not read the rest of the line. That's why $y is empty.
We can parse csv files with quoted strings and delimited by say | with following code
while read -r line
do
field1=$(echo "$line" | awk -F'|' '{printf "%s", $1}' | tr -d '"')
field2=$(echo "$line" | awk -F'|' '{printf "%s", $2}' | tr -d '"')
echo "$field1 $field2"
done < "$csvFile"
awk parses the string fields to variables and tr removes the quote.
Slightly slower as awk is executed for each field.
In addition to the answer from #Dennis Williamson, it may be helpful to skip the first line when it contains the header of the CSV:
{
read
while IFS=, read -r col1 col2
do
echo "I got:$col1|$col2"
done
} < myfile.csv
If you want to read CSV file with some lines, so this the solution.
while IFS=, read -ra line
do
test $i -eq 1 && ((i=i+1)) && continue
for col_val in ${line[#]}
do
echo -n "$col_val|"
done
echo
done < "$csvFile"
I have 2 text files. I want to loop in the first file to get a list, then using that list, loop from the second file to search for matching fields.
The first loop was fine, but when the second loop comes in, the variable $CLIENT_ABBREV cannot be read in the second loop, it's reading as blank. Output looks like does not match DOG where there's a blank before does.
while IFS=',' read CLIENT_ID NAME SERVER_NAME CLIENT_ABBREV
do
echo "\n------------"
echo Configuration in effect for this run
echo CLIENT_ID=$CLIENT_ID
echo NAME=$NAME
echo SERVER_NAME=$SERVER_NAME
echo CLIENT_ABBREV=$CLIENT_ABBREV
while IFS=',' read JOB_NAME CLIENT_ABBREV_FROMCOMMAND JOBTYPE JOBVER
do
if [ "$CLIENT_ABBREV" == "$CLIENT_ABBREV_FROMCOMMAND" ]; then
# do something
else
echo $CLIENT_ABBREV does not match $CLIENT_ABBREV_FROMCOMMAND
done <"$COMMAND_LIST"
done <"$CLIENT_LIST"
Is there a file with the name COMMAND_LIST ?
Or, actually do you want to use $COMMAND_LIST instead of COMMAND_LIST ?
I am trying to allow a user to input the number of ip addresses that they want to interface with, then enter each ip address and have it assigned to a variable. The script will ultimately write and execute a second script. The reason for the second script is that I am ssh-ing into an AP in order to cluster x number of AP's together, and once SSH occurs, bash/python variables are no longer passed through(the AP has it's own language), so they must be translated to plain text before the ssh script is run. The code below functions but allows only 2 ip addresses(I couldn't figure out how to use the $cvar to create multiple variables), and does not allow me to decide how many ip addresses to enter:
#!/bin/bash
echo -e "How many AP's do you want to Cluster?:"
#this is the variable to define how many ips to ask for
read cvar
echo -e "Enter last 2 digits of AP #1:"
read ip1
echo -e "Enter last 2 digits of AP #2:"
read ip2
#I want this to continue for x number of ip addresses(defined by $cvar)
echo -e "Enter a name for your Cluster:"
read cname
#code below is executed within the AP terminal(commands are unique to that shell)
echo "#!/bin/bash
ssh -T admin#192.168.0.$ip1 <<'eof'
configure
cluster
add $cname
add-member 192.168.0.$ip1 user ***** password ********
save
add-member 192.168.0.$ip2 user ***** password ********
save
exit
operate $cname
save
exit
" > ./2ndScript.sh
chmod a+x ./2ndScript.sh
/bin/bash ./2ndScript.sh
Without rewriting the entire script, here is a snippet.
#!/bin/bash
# IP is an array
IP=()
# Read number of IP Addresses to be read in
read -p "How many AP's do you want to Cluster?: " cvar
loop=1
while [ $loop -le $cvar ]
do
read -p "Enter last 2 digits of AP #${loop}: " IP[$loop]
loop=$((loop+1))
done
Arrays are your friend here. Take the following;
echo -e "Enter last 2 digits of AP #1:"
read ip1
echo -e "Enter last 2 digits of AP #2:"
read ip2
#I want this to continue for x number of ip addresses(defined by $cvar)
We can make a for loop, and then add an element to an array for each address. In this for loop, $i will tell us which cycle we're on, starting with 0. Since it auto-increments, we can just use it to specify what index of the array to update.
for (( i=0; i<$cvar; i++ )); do
echo -e "Enter last 2 digits of AP #$((i+1)):"
read #With no arguments, read assigns the output to $REPLY
#optional; this allows the user to enter "done" to end prematurely
#if [[ $REPLY == "done" ]]; then break; fi
ip[$i]=$REPLY #ip is the name of the array, and $i points to the index
done
If you use that optional code snipit, you don't even have to ask how many addresses the user wants. You could instead replace the for loop with while true; do, and just instruct the user to enter "done" (or any other exit command) to end address collection (though you'll need to define i=0 somewhere and then increment it at the end of the loop if you swap to while).
Now you have a list of values ordered from ${ip[0]} to ${ip[n]} of all the address that the user entered. You can extract them using another for loop later;
for ((i=0;i<${#ip[*]};i++)); do
#Insert code here. For example:
#echo "${ip[$i]} #echos the currently selected value of the array
#echo "${ip[$i]}" >> file.txt #appends file.txt with the current value
done
I need to create Bash script that generates text files named file001.txt through file050.txt
Of those files, all should have this text inserted "This if file number xxx" (where xxx is the assigned file number), except for file007.txt, which needs to me empty.
This is what I have so far..
#!/bin/bash
touch {001..050}.txt
for f in {001..050}
do
echo This is file number > "$f.txt"
done
Not sure where to go from here. Any help would be very appreciated.
#!/bin/bash
for f in {001..050}
do
if [[ ${f} == "007" ]]
then
# creates empty file
touch "${f}.txt"
else
# creates + inserts text into file
echo "some text/file" > "${f}.txt"
fi
done
The continue statement can be used to skip an iteration of a loop and go on to the next -- though since you actually do want to take an operation on file 7 (creating it), it makes just as much sense to have a conditional:
for (( i=1; i<50; i++ )); do
printf -v filename '%03d.txt' "$i"
if (( i == 7 )); then
# create file if it doesn't exist, truncate if it does
>"$filename"
else
echo "This is file number $i" >"$filename"
fi
done
A few words about the specific implementation decisions here:
Using touch file is much slower than > file (since it starts an external command), and doesn't truncate (so if the file already exists it will retain its contents); your textual description of the problem indicates that you want 007.txt to be empty, making truncation appropriate.
Using a C-style for loop, ie. for ((i=0; i<50; i++)), means you can use a variable for the maximum number; ie. for ((i=0; i<max; i++)). You can't do {001..$max}, by contrast. However, this does need meaning to add zero-padding in a separate step -- hence the printf.
Of course, you can costumize the files' name and the text, the key thing is the ${i}. I tried to be clear, but let us know if you don't understand something.
#!/bin/bash
# Looping through 001 to 050
for i in {001..050}
do
if [ ${i} == 007 ]
then
# Create an empty file if the "i" is 007
echo > "file${i}.txt"
else
# Else create a file ("file012.txt" for example)
# with the text "This is file number 012"
echo "This is file number ${i}" > "file${i}.txt"
fi
done
When i give the command awk NR==7 ABC.mod it gives me the title ('Add') I need which is present on the 7th line of the file Currently I am just able to read the title but I am not aware of how to append it to the output. Can somebody help me organize this so I can modify the code to get the expected menu output with minimal disruption (I hope) to the existing script?
Assuming you can pull out the "Add", "Delete" ... and other "titles" from the 7th line of each *.mod file, then you need to modify your script where it looks at the file a1.out somewhere before the line which seems to create the menu, namely: tr ' ' '\n' < ~/a1.out > ~/b.dat.
I say "assuming" because, even though you mention awk NR==7, I don't see where you are using it in the script. In any case, if you can get the "title" from the 7th line of a given *.mod file, then you can get the menu "name" from the file name (which seems to be the way you are constructing your menu) like this:
awk '{ln=length(ARGV[1]); if(NR==7) print substr(ARGV[1],0,ln-4)"..."$0}' ABC.mod
outputs:
ABC...Add
There's may be a shorter, easier way to do this using sed, but you mentioned awk.
For me at least, there's not really enough information to go on to help you much further. If you update your question someone may be able to give more concrete advice.
EDIT:
I'll post my work here in the hope that you find it useful. It is not a complete solution. I have to say, this is a strangely written application - with shell code and variables hardwired to temporary data to locations strewn about the file system. It's a bit hairy to try and set up a local version to try it out. Hopefully by experimenting and making modifications to the cod e you will learn more about how your application works and about shell programming in general. Extra advice: record your changes; sketch out how/where your application reads and writes its data; use comments in the source code to help you and others remember how the code works; make backups; use source control.
My assumptions:
pradee.sh looks like this (why does the file has a .sh extension - it seems more like a it defines some constants for your script)
% cat pradee.sh
HBKTM
ABC
HBKTM
CBC
HBKTM
DBC
HBKTM
IBC
HBKTM
MBCE
HBKTM
UBC
HBKTM
VBCM
Here's how I created my "test environment":
% for file in `grep -v HBKTM pradee.sh`; do touch $file.mod ; done
% ls
ABC.mod CBC.mod DBC.mod IBC.mod MBCE.mod UBC.mod VBCM.mod pradee.sh
% echo -e "_ctrl.jsp \n\n\n\n\n" > *.mod # mod files have required text+6 lines
% echo -e "_ctrl.jsp \n\n\n\n\n" > HBKTM.mod # this file seems special ?
% sed -i'' -e "7i\\[Ctrl-V Ctrl-J]
Add" ABC.mod
OR since the files now have 6 lines ... echo the menu title onto the last line:
% echo "Delete" >> DBC.mod
% echo "Insert" >> IBC.mod
... [continue inserting titles like "Add" "Delete" etc to the other *.mod files]
After that I think I have data files that mimic your set up. You tell me. Now, if I make a few small changes to your script (so the file locations don't remove overwrite my own files) and add the awk command I mentioned previously, here is what I end up with:
# menu_create.sh
# See http://stackoverflow.com/questions/17297671
rm -f *.dat
clear
cont="y"
while [ "$cont" = "y" ] # "$" is need for POSIX
do
echo -e "\n\nPlease Enter ONS Name : "
read ons
currpath=.
up=$(echo $ons|tr '[:lower:]' '[:upper:]')
#echo "\n ONS menu name \n"
#echo $up
if [ -f $up.mod ]; then
#in=$(grep -ri $up pradee.sh) # changed to following
# - how could this have worked ?
in=$(grep -v $up pradee.sh)
if [ -n "$in" ]; then
onsname=$(grep -ri "_ctrl.jsp" $up.mod)
#echo "onsname : $onsname"
if [ -n "$onsname" ]; then
echo -e "\n ONS menu name : $up "
echo $in > a1.dat
#echo "written to a1.dat\n"
#cat ~/a1.dat
#tr ' ' '\n' < ~/a1.dat > ~/a.dat
#cat ~/a.dat
sed "s/$up//g" a1.dat >a1.out
for i in `cat a1.dat`;
do
awk '{ln=length(ARGV[1]);if(NR==7) print substr(ARGV[1],0,ln-4)"..."$0}' $i.mod >> menu.dat ;
done
echo -e "\n FINUX Names \n"
#tr ' ' '\n' < a1.out > b.dat
tr ' ' '\n' < menu.dat > b.dat
cat b.dat
else
echo -e "ONS Name Not Valid !"
fi
else
echo -e "FINUX menu Name not found in our Repository"
fi
else
echo -e "\n Please Enter valid ONS name !!"
fi
echo -e "\n\n Press "y" to continue, Any other key to exit"
read cont
done
It gives me this output:
Please Enter ONS Name :
hbktm
ONS menu name : HBKTM
FINUX Names
ABC...Add
CBC...Cancel
DBC...Delete
IBC...Insert
MBCE...Modify
UBC...Undelete
VBCM...Verify
Press y to continue, Any other key to exit
q
I hope my response to your question helps you learn more about how to modify your application.