I try to retrieve a line from a file without using a loop.
myFile.txt
val1;a;b;c
val2;b;d;e
val3;c;r;f
I would like to get the line where the second column is b.
If I do grep "b" myFile.txt then both first and second line will be outputed.
If I do cat myFile.txt | cut -d ';' -f2 | grep "b" then the output will just be b whereas I'd like to get the full line val2;b;d;e.
Is there a way of reaching the desired results without using a loop as below ? My file being huge it wouldn't be nice looping through it again and again.
while read line; do
if [ `echo $line | cut -d ';' -f2` = "b" ]; then
echo $line
fi
done < myFile.txt
Given your input file, The below one-liner should work:
awk -F";" '$2 == "b" {print}' myFile.txt
Explanation:
awk -F";" ##Field Separator as ";"
'$2 == "b" ##Searches for "b" in the second column($2)
{print}' ##prints the searched line
Using:
grep:
grep '^[^;]*;b;' myFile.txt
sed:
sed '/^[^;]*;b;/!d' myFile.txt
Output is the same for both:
val2;b;d;e
Related
I am using xargs in conjuction with cut but I am unsure how to get the output of cut to a variable which I can pipe to use for further processing.
So, I have a text file like so:
test.txt:
/some/path/to/dir,filename.jpg
/some/path/to/dir2,filename2.jpg
...
I do this:
cat test.txt | xargs -L1 | cut -d, -f 1,2
/some/path/to/dir,filename.jpg
but what Id like to do is:
cat test.txt | xargs -L1 | cut -d, -f 1,2 | echo $1 $2
where $1 and $2 are /some/path/to/dir and filename.jpg
I am stumped that I cannot seem to able to achieve this..
You may want to say something like:
#!/bin/bash
while IFS=, read -r f1 f2; do
echo ./mypgm -i "$f1" -o "$f2"
done < test.txt
IFS=, read -r f1 f2 reads a line from test.txt one by one,
splits the line on a comma, then assigns the variables f1 and f2
to the fields.
The line echo .. is for the demonstration purpose. Replace the
line with your desired command using $f1 and $f2.
Try this:
cat test.txt | awk -F, '{print $1, $2}'
From man xargs:
xargs [-L number] [utility [argument ...]]
-L number
Call utility for every number non-empty lines read.
From man awk:
Awk scans each input file for lines that match any of a set of patterns specified literally in prog or in one or more files specified as -f progfile.
So you don't have to use xargs -L1 as you don't pass the utility to call.
Also from man awk:
The -F fs option defines the input field separator to be the regular expression fs.
So awk -F, can replace the cut -d, part.
The fields are denoted $1, $2, ..., while $0 refers to the entire line.
So $1 is for the first column, $2 is for the second one.
An action is a sequence of statements. A statement can be one of the following:
print [ expression-list ] [ > expression ]
An empty expression-list stands for $0.
The print statement prints its argument on the standard output (or on a file if > file or >> file is present or on a pipe if | cmd is present), separated by the current output field separator, and terminated by the output record separator.
Put all these together, cat test.txt | awk -F, '{print $1, $2}' would achieve that you want.
I have a config file where each line is in a format say UniqueOption = SomeValue:
$ cat somefile
option1sub1 = yes
option1sub2 = 1234
...
option1subn = xxxx
option2 = 2345
option3 = no
...
I want to deal with each value of "option1" in a loop. but, sed or grep give me all of option1 in one time.
How could I achieve that using sed or grep, getting a single option1 line at a time?
pipe the output of grep to a while loop:
grep 'option1' somefile | while read line
do
echo "single option is in var $line"
done
Solution 1st: Following awk may help you on same to get the value of option1 string's last value.
awk -F" = " '/^option1/{print $NF}' Input_file
Solution 2nd: Above will print all values of string option1 in case you need only very first value of string option1 then use following.
awk -F" = " '/^option1/{print $NF;exit}' Input_file
The following will parse out all sub-options for option1 in the file file.conf and save them in a bash array. The options are then easily accessed from that array.
#!/bin/bash
while IFS= read -r data; do
opt1+=( "$data" )
done < <( awk -F ' *= *' '$1 ~ /^option1/ { print $2 }' file.conf )
printf 'Option 1, sub-option 1 is "%s"\n' "${opt1[0]}"
Output:
Option 1, sub-option 1 is "yes"
The awk script will return everything after the = (and any spaces), which allows you to store data that contains multiple words. Only the lines starting with option1 in the configuration file are processed.
This would be adapted to parse the whole configuration file into a single structure, possibly using an associative array in a sufficiently recent version of bash.
Already we can see few awesome answers but as you asked something with grep, you can use one of the following if you want.
For all values
grep option1 m | cut -d "=" -f2 | awk '{$1=$1};1'
For first value
grep option1 m | cut -d "=" -f2 | awk '{$1=$1};1' | head -1
Here: cut is used to cut the second option uisng dilimiter =; awk is used to trim the spaces in output and head is used to print first occurrence
With sed
sed '/^option1.* = /!d;s///' somefile
With gnu grep 2.20 (support of pcre)
grep -oP '^option1.* = \K.*' somefile
If you want to get only the first match
sed '/^option1.* = /!d;s///;q' somefile
grep -m1 -oP '^option1.* = \K.*' somefile
The problem is the following I have a file that each line has this form:
id|lastName|firstName|gender|birthday|joinDate|IP|browser
i want to sort alphabetically all the firstnames in that file and print them one on each line but each name only once
i have created the following program but for some reason it creates an infinite loop:
array1=()
while read LINE
do
if [ ${LINE:0:1} != '#' ]
then
IFS="|"
array=($LINE)
if [[ "${array1[#]}" != "${array[2]}" ]]
then
array1+=("${array[2]}")
fi
fi
done < $3
echo ${array1[#]} | awk 'BEGIN{RS=" ";} {print $1}' | sort
NOTES
if [ ${LINE:0:1} != '#' ] : this command is used because there are comments in the file that i dont want to print
$3 : filename
array1 : is used for all the seperate names
Wow, there's a MUCH simpler and cleaner way to achieve this, without having to mess with the IFS variable or using arrays. You can use "for" to do this:
First I created a file with the same structure as yours:
$ cat file
id|lastName|Douglas|gender|birthday|joinDate|IP|browser
id|lastName|Tim|gender|birthday|joinDate|IP|browser
id|lastName|Andrew|gender|birthday|joinDate|IP|browser
id|lastName|Sasha|gender|birthday|joinDate|IP|browser
#id|lastName|Carly|gender|birthday|joinDate|IP|browser
id|lastName|Madson|gender|birthday|joinDate|IP|browser
Here's the script I wrote using "for":
#!/bin/bash
for LINE in `cat file | grep -v "^#" | awk -F'|' '{print$3}' | sort -u`
do
echo $LINE
done
And here's the output of this script:
$ ./script.sh
Andrew
Douglas
Madson
Sasha
Tim
Explanation:
for LINE in `cat file`
Creates a loop that reads each line of "file". The commands between ` are run by linux, for example, if you wanted to store the date inside of a variable you could use "VARDATE=`date`".
grep -v "^#"
The option -v is used to exclude results matching the pattern, in this case the pattern is "^#". The "^" character means "line begins with". So grep -v "^#" means "exclude lines beginning with #".
awk -F'|' '{print$3}'
The -F option switches the column delimiter from the default (the default is a space) to whatever you put between ' after it, in this case the "|" character.
The '{print$3}' prints the 3rd column.
sort -u
And the "sort -u" command to sort the names alphabetically.
I have a text file that looks like this:
input_file
1|abc
2|def
3|ghi
n|etc...
I need to split this up into two files on the pipe delimeter. So this is the expected output:
File_1:
1
2
3
n
File_2:
abc
def
ghi
etc
I do not know how many lines the input file will have. How do you achieve this in ksh or bash?
Thank you.
awk would be suitable for this task:
awk -F\| '{print $1 > "File_1"; print $2 > "File_2"}' input_file
This splits your text on the "|" and prints each column to the respective file.
If there were more than two fields, you may prefer to use a loop instead:
awk -F\| '{for(i=1;i<=NF;++i) print $i > "File_" i}' input_file
cut -d '|' -f 1 input_file > File_1
cut -d '|' -f 2 input_file > File_2
Only with bash:
while IFS='|' read A B; do echo "$A" >>File_1; echo "$B" >>File_2; done <input_file
Here is another solution using other bash commands
cat input_file | cut -d '|' -f1 > File_1
cat input_file | cut -d '|' -f2 > File_2
Or you can put them together in one line
cat input_file | tee >(cut -d '|' -f1 > File_1) | cut -d '|' -f2 > File_2
I am trying to grep out the lines in a file where the third field matches certain criteria.
I tried using grep but had no luck in filtering out by a field in the file.
I have a file full of records like this:
12794357382;0;219;215
12795287063;0;220;215
12795432063;0;215;220
I need to grep only the lines where the third field is equal to 215 (in this case, only the third line)
Thanks a lot in advance for your help!
Put down the hammer.
$ awk -F ";" '$3 == 215 { print $0 }' <<< $'12794357382;0;219;215\n12795287063;0;220;215\n12795432063;0;215;220'
12795432063;0;215;220
grep:
grep -E "[^;]*;[^;]*;215;.*" yourFile
in this case, awk would be easier:
awk -F';' '$3==215' yourFile
A solution in pure bash for the pre-processing, still needing a grep:
while read line; do
OLF_IFS=$IFS; IFS=";"
line_array=( $line )
IFS=$OLD_IFS
test "${line_array[2]}" = 215 && echo "$line"
done < file | grep _your_pattern_
Simple egrep (=grep -E)
egrep ';215;[0-d][0-d][0-d]$' /path/to/file
or
egrep ';215;[[:digit:]]{3}$' /path/to/file
How about something like this:
cat your_file | while read line; do
if [ `echo "$line" | cut -d ";" -f 3` == "215" ]; then
# This is the line you want
fi
done
Here is the sed version to grep for lines where 3rd field is 215:
sed -n '/^[^;]*;[^;]*;215;/p' file.txt
Simplify your problem by putting the 3rd field at the beginning of the line:
cut -d ";" -f 3 file | paste -d ";" - file
then grep for the lines matching the 3rd field and remove the 3rd field at the beginning:
grep "^215;" | cut -d ";" -f 2-
and then you can grep for whatever you want. So the complete solution is:
cut -d ";" -f 3 file | paste -d ";" - file | grep "^215;" | cut -d ";" -f 2- | grep _your_pattern_
Advantage: Easy to understand; drawback: many processes.