I want to cut CSV file by column names.
Sample file:
"column A","column B","column C","column D","column E",
aaa,bbb,ccc,ddd,eee,
111,222,333,444,555
column_A:
$ cat column_A
aaa
111
column_B:
$ cat column_B
bbb
222
when I using awk but it's not success. Some time when we meet \n in "".
Like this:
aaa,ssss,"zzz
xxx"
but this should be a line what should I do?
just 1 " it can be successful
but when csv file have too may \n and too may “ on line
it's
can't be success
function isallline(){
LineNumber=$1
LineInfo=`echo "$2"|tr -d '\n'`
FileName=$3
LastLineNumber=$4
GetInfo=`echo "$LineInfo"|awk -F '"' '{print NF-1}'`
IsAl=$((GetInfo%2))
if [[ $IsAl != 0 ]]
then
LineNumber=$((LineNumber+1))
LineInfo="$LineInfo""`sed -n ''$LineNumber'p' $FileName|tr -d '\n'`"
if [[ $LineNumber -le $LastLineNumber ]]
then
isallline $LineNumber "$LineInfo" $FileName $LastLineNumber
else
echo "error with not complte'\"'"
fi
else
echo "$LineInfo" >>CSVFile
return $LineNumber
fi
}
function GetCsvFile()
{
FileName=$1
>CSVFile #-- clearn file --
i=1 #-- declare i=1 --
LasetLineNumber=`wc -l $FileName|awk '{print $1 }'`
LineNumber=0 #-- declare LineNumber=0 --
while read LINE #-- read file --
do
getinfo=`echo $LINE|awk -F '"' '{print NF-1}'` #-- get count(") --
if [[ $getinfo != 0 ]]
then
if [[ $LineNumber == 0 ]]
then
isallline $i "$LINE" $FileName LasetLineNumber #-- call function isallline --
LineNumber=$? #-- get function isallline return(LineNumber) --
elif [[ $LineNumber -lt $i ]]
then
isallline $i "$LINE" $FileName LasetLineNumber
LineNumber=$?
fi
fi
if [[ $i -gt $LineNumber ]]
then
echo $LINE >> CSVFile
fi
let i++
done < $FileName
}
Theoretically it's possible to write such regexp, which will be able to parse CSV. But here is brilliant answer why it's more or less practically impossible. You better use specific parser. Actually, almost any linux system has python installed and python includes tooling to parse CSV files
Related
I'm trying to automate a build process by replacing .js chunks for particular lines in my main.config.php file. When I run the following code:
declare -a js_strings=("footer." "footerJQuery." "headerCSS." "headerJQuery.")
build_path="./build/build"
config_path="./system/Config/main.config.php"
while read -r line;
do
for js_string in ${js_strings[#]}
do
if [[ $line == *$js_string* ]]
then
for js_file in "$build_path"/*
do
result="${js_file//[^.]}"
if [[ $js_file == *$js_string* ]] && [[ ${#result} -eq 3 ]]
then
sed -i "s/$line/$line$(basename $js_file)\";/g" $config_path
fi
done
fi
done
done < "$config_path"
I get this message back, and file has not been updated/edited:
sed: 1: "./system/Config/main.co ...": invalid command code .
I haven't been able to find anything in my searches that pertain to this specific message. Does anyone know what I need to change/try to get the specific lines replaced in my .php file?
Updated script with same message:
declare -a js_strings=("footer." "footerJQuery." "headerCSS." "headerJQuery.")
build_path="./build/build"
config_path="./system/Config/main.config.php"
while read -r line;
do
for js_string in ${js_strings[#]}
do
if [[ $line == *$js_string* ]]
then
for js_file in "$build_path"/*
do
result="${js_file//[^.]}"
if [[ $js_file == *$js_string* ]] && [[ ${#result} -eq 3 ]]
then
filename=$(basename $js_file)
newline="${line//$js_string*/$filename\";}"
echo $line
echo $newline
sed -i "s\\$line\\$newline\\g" $config_path
echo ""
fi
done
fi
done
done < "$config_path"
Example $line:
$config['public_build_header_css_url'] = "http://localhost:8080/build/headerCSS.js";
Example $newline:
$config['public_build_header_css_url'] = "http://localhost:8080/build/headerCSS.7529a73071877d127676.js";
Updated script with changes suggested by #Vercingatorix:
declare -a js_strings=("footer." "footerJQuery." "headerCSS." "headerJQuery.")
build_path="./build/build"
config_path="./system/Config/main.config.php"
while read -r line;
do
for js_string in ${js_strings[#]}
do
if [[ $line == *$js_string* ]]
then
for js_file in "$build_path"/*
do
result="${js_file//[^.]}"
if [[ $js_file == *$js_string* ]] && [[ ${#result} -eq 3 ]]
then
filename=$(basename $js_file)
newline="${line//$js_string*/$filename\";}"
echo $line
echo $newline
linenum=$(grep -n "^${line}\$" ${config_path} | cut -d':' -f 1 )
echo $linenum
[[ -n "${linenum}" ]] && sed -i "${linenum}a\\
${newline}
;${linenum}d" ${config_path}
echo ""
fi
done
fi
done
done < "$config_path"
Using sed's s command to replace a line of that complexity is a losing proposition, because whatever delimiter you choose may appear in the line and mess things up. If these are in fact entire lines, it is better to delete them and insert a new one:
linenum=$(fgrep -nx -f "${line}" "${config_path}" | awk -F : "{print \$1}" )
[[ -n "${linenum}" ]] && sed -i "" "${linenum}a\\
${newline}
;${linenum}d" "${config_path}"
What this does is search for the line number of the line that matches $line in its entirety, then extracts the line number portion. fgrep is necessary otherwise the symbols in your file are interpreted as regular expressions. If there was a match, then it runs sed, appending the new line (a) and deleting the old one (d).
My code looks something like this:
for line in `cat fileName`
do
if [[ $line == "Marker 1" ]]
then
while [[ $line != "---" ]]
do
#basically I want to read all the data below "Marker 1" till "---"
echo $line
((line++)) #this is wrong
done
elif [[ $line == "Marker 2" ]]
then
while [[ $line != "---" ]]
do
echo $line
((line++))
done
fi
done
How do I increment the value of $line when in the while loop? ((line++)) doesn't work
Using sed
If the goal is to echo all the lines from a line with Marker 1 or Marker 2 to a line with ---, then that entire shell loop can be replaced with this simple sed command:
sed -n '/^Marker [12]$/,/^---$/p' File
Example
Consider this test file:
$ cat File
beginning
Marker 1
one
---
more
Marker 2
two
Two
---
end
Now, let's run our command:
$ sed -n '/^Marker [12]$/,/^---$/p' File
Marker 1
one
---
Marker 2
two
Two
---
Using awk
With the same test file:
$ awk '/^Marker [12]$/,/^---$/' File
Marker 1
one
---
Marker 2
two
Two
---
((line++)) is for incrementing an integer value.
But the value of line in your example is a string.
In fact what you seem to want is get the next line from the file.
You need a different approach for that, and use while read instead of a for-loop.
#!/usr/bin/env bash
read_and_print_until_dashes() {
while read -r line; do
[[ $line = '---' ]] && break
echo "$line"
done
}
while read -r line; do
if [[ $line = "Marker 1" ]]; then
echo "$line"
read_and_print_until_dashes
elif [[ $line = "Marker 2" ]]; then
echo "$line"
read_and_print_until_dashes
fi
done < file.txt
Can anyone see whats wrong here? If I put X|9 in lan.db (or any db in this directory) and run the following code, the IF statement does not work. It's weird! if you echo $LINE, it is indeed pulling X|9 out of lan.db (or any db in this directory) and setting it equal to LINE, but it wont do the comparison.
DBREGEX="^[0-9]|[0-9]$"
shopt -s nullglob
DBARRAY=(databases/*)
i=0
for i in "${!DBARRAY[#]}"; do
cat ${DBARRAY[$i]} | grep -v \# | while read LINE; do
echo "$LINE" (Whats weird is that LINE DOES contain X|9)
if [[ !( $LINE =~ $DBREGEX ) ]]; then echo "FAIL"; fi
done
done
If however I just manually sent LINE="X|9" the same code (minus the while) works fine. ie LINE=X|9 fails, but LINE=9|9 succeeds.
DBREGEX="^[0-9]|[0-9]$"
Comment shopt -s nullglob
Comment DBARRAY=(databases/*)
Comment i=0
Comment for i in "${!DBARRAY[#]}"; do
Comment cat ${DBARRAY[$i]} | grep -v \# | while read LINE; do
LINE="X|9"
if [[ !( $LINE =~ $DBREGEX ) ]]; then echo "FAIL"; fi
Comment done
Comment done
* UPDATE *
UGH I GIVE UP
Now not even this is working...
DBREGEX="^[0-9]|[0-9]$"
LINE="X|9"
if [[ ! $LINE =~ $DBREGEX ]]; then echo "FAIL"; fi
* UPDATE *
Ok, so it looks like I have to escape |
DBREGEX="^[0-9]\|[0-9]$"
LINE="9|9"
echo "$LINE"
if [[ ! $LINE =~ $DBREGEX ]]; then echo "FAIL"; fi
This seems to work ok again
| has a special meaning in a regular expression. ^[0-9]|[0-9]$ means "starts with a digit, or ends with a digit". If you want to match a literal vertical bar, backslash it:
DBREGEX='^[0-9]\|[0-9]$'
for LINE in 'X|9' '9|9' ; do
echo "$LINE"
if [[ ! $LINE =~ $DBREGEX ]] ; then echo "FAIL" ; fi
done
You don't need round brackets in regex evaluation. You script is also creating a sub shell and making a useless use of cat which can be avoided.
Try this script instead:
while read LINE; do
echo "$LINE"
[[ "$LINE" =~ $DBREGEX ]] && echo "PASS" || echo "FAIL"
done < <(grep -v '#' databases/lan.db)
i'm trying to get a file name, and a character index, and to print me the characters with that index from each line (and do it for each character index the user enters if such character exists).
This is my code:
#!/bin/bash
read file_name
while read x
do
if [ -f $file_name ]
then
while read string
do
counter=0
while read -n char
do
if [ $counter -eq $x ]
then
echo $char
fi
counter=$[$counter+1]
done < $(echo -n "$string")
done < $file_name
fi
done
But, it says an error:
line 20: abcdefgh: No such file or directory
line 20 is the last done, so it doesn't help me figure out where is the error.
So what's wrong in my code and how do I fix it?
Thanks a lot.
I think "cut" might fit the bill:
read file_name
if [ -f $file_name ]
then
while read -n char
do
cut -c $char $file_name
done
fi
This line seems to be problematic:
done < $(echo -n "$string")
Replace that with:
done < <(echo -n "$string")
replace
counter=0
while read -n char
do
if [ $counter -eq $x ]
then
echo $char
fi
counter=$[$counter+1]
done < $(echo -n "$string")
with
if [ $x -lt ${#string} ]
then
echo ${line:$x:1}
fi
It does the same, but allows to avoid such errors.
Another approach is using cut
cut -b $(($x+1)) $file_name | grep -v "^$"
It can replace two inner loops
I am trying to create an if statement that performs an action when it reads a blank line.
I would assume it would be something like this : if ($line=='\n');then
where line is the line that it is reading from a text file. But this is not working.
while read line; do
if [ "$line" = "" ]; then
echo BLANK
fi
done < filename.txt
or a slight variation:
while read line; do
if [ "$line" ]; then
echo NOT BLANK
else
echo BLANK
fi
done < filename.txt
try this:
if [[ "x$line" == "x" ]]; then...
or
if [[ "$line" =~ "^$" ]]; ...
Or also:
grep -q '.' <<< $line
Returns 1 if line is empty, 0 if non-empty