I know there has probably been a few hundred forms of this question asked on stackoverflow, but I can't seem to find a suitable answer to my question.
I'm trying to parse through the /etc/ldap.conf file on a Linux box so that I can specifically pick out the description fields from between (description= and ):
*-bash-3.2$ grep '^nss_base_passwd' /etc/ldap.conf
nss_base_passwd ou=People,dc=ca,dc=somecompany,dc=com?one?|(description=TD_FI)(description=TD_F6)(description=TD_F6)(description=TRI_142)(description=14_142)(description=REX5)(description=REX5)(description=1950)*
I'm looking to extract these into their own list with no duplicates:
TD_FI
TD_F6
TRI_142
14_142
REX5
1950
(or all on one line with a proper delimiter)
I had played with sed for a few hours but couldn't get it to work - I'm not entirely sure how to use the global option.
You could use grep with -P option,
$ grep '^nss_base_passwd' /etc/ldap.conf | grep -oP '(?<=description\=)[^)]*' | uniq
TD_FI
TD_F6
TRI_142
14_142
REX5
1950
Explanation:
A positive lookbehind is used in grep to print all the characters which was just after to the description= upto the next ) bracket. uniq command is used to remove the duplicates.
perl -nE 'say join(",", /description=\K([^)]+)/g) if /^nss_base_passwd/' /etc/ldap.conf
TD_FI,TD_F6,TD_F6,TRI_142,14_142,REX5,REX5,1950
Try this:
grep '^nss_base_passwd' /etc/ldap.conf |
grep -oE '[(]description=[^)]*' | sort -u |
cut -f2- -d=
Explanations:
With bash, if you end a line with | (or || or &&), the shell knows that the command continues on the next line, so you don't need to use \.
The second grep uses the -o flag to indicate that the matching expressions should be printed out, one per line. It also uses the -E flag to indicate that the pattern is an "Extended" (i.e. normal) regular expression.
Since -o will print the entire match, we need to extract the part after the prefix, for which we use cut, specifying a delimiter of =. -f2- means "all the fields starting with the second field", which we need in case there is an = in the description.
Avinash's answer was very close. Here is my improved version:
grep '^nss_base_passwd' /etc/ldap.conf | grep -Po '\(description=\K[^)]+' | sort -u
There is no need to use lookaround syntax when you can simply use \K (which is actually a shortcut for a corresponding zero-width assertion).
Also, you said that you want NO duplicates, but uniq will only remove duplicate adjacent lines, it will not remove duplicates if there is something in between. That's why I am using sort -u instead.
Related
<Table[\n\r\s]+showFilterBar=\{false\}
This regex when pasted in VSCode search works but when I grep it does not. I have also tried:
<Table[\n\r\s]\+showFilterBar=\{false\}
<Table[\n\r\s]\+
I want to basically find all Table components when showFilterBar={false} prop is used and set to false.
grep doesn't support \s for whitespace. Use [:space:] instead, or list out all of the whitespace characters individually.
grep -E '<Table[[:space:]]+showFilterBar={false}'
grep -E '<Table[ \t\r]+showFilterBar={false}'
I've omitted \n since grep only searches one line at a time.
I want to use grep in a bash script to get the line with only the word "version" from a file, the problem is that a few lines further down in the file that i want the script to look in there is another line with the word "dlversion".
I am piping the grep into a cut command, the output will be saved as a variable
The problem is that it either saves nothing into the variable or it saves both lines, I've already tried several methods that i found, though none of them have worked.
grep -Fx version /path/to/file.txt | cut -c9-
output = nothing
grep '^version$' /path/to/file.txt | cut -c9-
output = nothing
grep "version" /path/to/file.txt | cut -c9-
output = both lines
also tried
grep -w "version " /path/to/file.txt | cut -c9-
output = nothing
I also tried to use -F, -x on their own which also caused the variable to not have a value.
You have a few options, depending on your version of grep.
If supported, the best option is to use word boundaries \b either side of your word:
grep '\bversion\b' /path/to/file.txt
Or:
grep '\<version\>' /path/to/file.txt
Where \< and \> match the empty string at the start and end of a word respectively.
Otherwise, you can create your own set of characters that you consider to not be a word:
grep -E '(^|[[:space:][:punct:]])version' /path/to/file.txt
This matches "version", preceded by either the start of the line or any type of space or punctuation.
In your specific case, you could use something like this:
grep -E '(^|[^l])version' /path/to/file.txt
This matches "version" preceded by either the start of the line or anything other than an "l".
In response to your comment:
^ matches the start of the line.
| means "or".
[^l] is a bracket expression, where the ^ as the first character means "not" (so this matches every character other than "l").
The parentheses are used to create a group, so that the "or" only applies to this part of the pattern.
In case any sophisticated tricks fail, use the brute force: pipe the result through another grep process to filter out the lines with unwanted words (use -v option for that):
grep 'version' sourcefile | grep -v 'dlversion' > destination
I am writing shell script for embedded Linux in a small industrial box. I have a variable containing the text pid: 1234 and I want to strip first X characters from the line, so only 1234 stays. I have more variables I need to "clean", so I need to cut away X first characters and ${string:5} doesn't work for some reason in my system.
The only thing the box seems to have is sed.
I am trying to make the following to work:
result=$(echo "$pid" | sed 's/^.\{4\}//g')
Any ideas?
The following should work:
var="pid: 1234"
var=${var:5}
Are you sure bash is the shell executing your script?
Even the POSIX-compliant
var=${var#?????}
would be preferable to using an external process, although this requires you to hard-code the 5 in the form of a fixed-length pattern.
Here's a concise method to cut the first X characters using cut(1). This example removes the first 4 characters by cutting a substring starting with 5th character.
echo "$pid" | cut -c 5-
Use the -r option ("use extended regular expressions in the script") to sed in order to use the {n} syntax:
$ echo 'pid: 1234'| sed -r 's/^.{5}//'
1234
Cut first two characters from string:
$ string="1234567890"; echo "${string:2}"
34567890
pipe it through awk '{print substr($0,42)}' where 42 is one more than the number of characters to drop. For example:
$ echo abcde| awk '{print substr($0,2)}'
bcde
$
Chances are, you'll have cut as well. If so:
[me#home]$ echo "pid: 1234" | cut -d" " -f2
1234
Well, there have been solutions here with sed, awk, cut and using bash syntax. I just want to throw in another POSIX conform variant:
$ echo "pid: 1234" | tail -c +6
1234
-c tells tail at which byte offset to start, counting from the end of the input data, yet if the the number starts with a + sign, it is from the beginning of the input data to the end.
Another way, using cut instead of sed.
result=`echo $pid | cut -c 5-`
I found the answer in pure sed supplied by this question (admittedly, posted after this question was posted). This does exactly what you asked, solely in sed:
result=\`echo "$pid" | sed '/./ { s/pid:\ //g; }'\``
The dot in sed '/./) is whatever you want to match. Your question is exactly what I was attempting to, except in my case I wanted to match a specific line in a file and then uncomment it. In my case it was:
# Uncomment a line (edit the file in-place):
sed -i '/#\ COMMENTED_LINE_TO_MATCH/ { s/#\ //g; }' /path/to/target/file
The -i after sed is to edit the file in place (remove this switch if you want to test your matching expression prior to editing the file).
(I posted this because I wanted to do this entirely with sed as this question asked and none of the previous answered solved that problem.)
Rather than removing n characters from the start, perhaps you could just extract the digits directly. Like so...
$ echo "pid: 1234" | grep -Po "\d+"
This may be a more robust solution, and seems more intuitive.
This will do the job too:
echo "$pid"|awk '{print $2}'
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2`
outputs something like this
"Abc Inc";
What I want to do is I want to remove the trailing ";" as well. How can i do that? I am a beginner to bash. Any thoughts or suggestions would be helpful.
This will remove the last character contained in your COMPANY_NAME var regardless if it is or not a semicolon:
echo "$COMPANY_NAME" | rev | cut -c 2- | rev
I'd use sed 's/;$//'. eg:
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2 | sed 's/;$//'`
foo="hello world"
echo ${foo%?}
hello worl
I'd use head --bytes -1, or head -c-1 for short.
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2 | head --bytes -1`
head outputs only the beginning of a stream or file. Typically it counts lines, but it can be made to count characters/bytes instead. head --bytes 10 will output the first ten characters, but head --bytes -10 will output everything except the last ten.
NB: you may have issues if the final character is multi-byte, but a semi-colon isn't
I'd recommend this solution over sed or cut because
It's exactly what head was designed to do, thus less command-line options and an easier-to-read command
It saves you having to think about regular expressions, which are cool/powerful but often overkill
It saves your machine having to think about regular expressions, so will be imperceptibly faster
I believe the cleanest way to strip a single character from a string with bash is:
echo ${COMPANY_NAME:: -1}
but I haven't been able to embed the grep piece within the curly braces, so your particular task becomes a two-liner:
COMPANY_NAME=$(grep "company_name" file.txt); COMPANY_NAME=${COMPANY_NAME:: -1}
This will strip any character, semicolon or not, but can get rid of the semicolon specifically, too.
To remove ALL semicolons, wherever they may fall:
echo ${COMPANY_NAME/;/}
To remove only a semicolon at the end:
echo ${COMPANY_NAME%;}
Or, to remove multiple semicolons from the end:
echo ${COMPANY_NAME%%;}
For great detail and more on this approach, The Linux Documentation Project covers a lot of ground at http://tldp.org/LDP/abs/html/string-manipulation.html
Using sed, if you don't know what the last character actually is:
$ grep company_name file.txt | cut -d '=' -f2 | sed 's/.$//'
"Abc Inc"
Don't abuse cats. Did you know that grep can read files, too?
The canonical approach would be this:
grep "company_name" file.txt | cut -d '=' -f 2 | sed -e 's/;$//'
the smarter approach would use a single perl or awk statement, which can do filter and different transformations at once. For example something like this:
COMPANY_NAME=$( perl -ne '/company_name=(.*);/ && print $1' file.txt )
don't have to chain so many tools. Just one awk command does the job
COMPANY_NAME=$(awk -F"=" '/company_name/{gsub(/;$/,"",$2) ;print $2}' file.txt)
In Bash using only one external utility:
IFS='= ' read -r discard COMPANY_NAME <<< $(grep "company_name" file.txt)
COMPANY_NAME=${COMPANY_NAME/%?}
Assuming the quotation marks are actually part of the output, couldn't you just use the -o switch to return everything between the quote marks?
COMPANY_NAME="\"ABC Inc\";" | echo $COMPANY_NAME | grep -o "\"*.*\""
you can strip the beginnings and ends of a string by N characters using this bash construct, as someone said already
$ fred=abcdefg.rpm
$ echo ${fred:1:-4}
bcdefg
HOWEVER, this is not supported in older versions of bash.. as I discovered just now writing a script for a Red hat EL6 install process. This is the sole reason for posting here.
A hacky way to achieve this is to use sed with extended regex like this:
$ fred=abcdefg.rpm
$ echo $fred | sed -re 's/^.(.*)....$/\1/g'
bcdefg
Some refinements to answer above. To remove more than one char you add multiple question marks. For example, to remove last two chars from variable $SRC_IP_MSG, you can use:
SRC_IP_MSG=${SRC_IP_MSG%??}
cat file.txt | grep "company_name" | cut -d '=' -f 2 | cut -d ';' -f 1
I am not finding that sed 's/;$//' works. It doesn't trim anything, though I'm wondering whether it's because the character I'm trying to trim off happens to be a "$". What does work for me is sed 's/.\{1\}$//'.
[Editorial insertion: Possible duplicate of the same poster's earlier question?]
Hi, I need to extract from the file:
first
second
third
using the grep command, the following line:
second
third
How should the grep command look like?
Instead of grep, you can use pcregrep which supports multiline patterns
pcregrep -M 'second\nthird' file
-M allows the pattern to match more than one line.
Your question abstract "bash grep newline", implies that you would want to match on the second\nthird sequence of characters - i.e. something containing newline within it.
Since the grep works on "lines" and these two are different lines, you would not be able to match it this way.
So, I'd split it into several tasks:
you match the line that contains "second" and output the line that has matched and the subsequent line:
grep -A 1 "second" testfile
you translate every other newline into the sequence that is guaranteed not to occur in the input. I think the simplest way to do that would be using perl:
perl -npe '$x=1-$x; s/\n/##UnUsedSequence##/ if $x;'
you do a grep on these lines, this time searching for string ##UnUsedSequence##third:
grep "##UnUsedSequence##third"
you unwrap the unused sequences back into the newlines, sed might be the simplest:
sed -e 's/##UnUsedSequence##/\n'
So the resulting pipe command to do what you want would look like:
grep -A 1 "second" testfile | perl -npe '$x=1-$x; s/\n/##UnUsedSequence##/ if $x;' | grep "##UnUsedSequence##third" | sed -e 's/##UnUsedSequence##/\n/'
Not the most elegant by far, but should work. I'm curious to know of better approaches, though - there should be some.
I don't think grep is the way to go on this.
If you just want to strip the first line from any file (to generalize your question), I would use sed instead.
sed '1d' INPUT_FILE_NAME
This will send the contents of the file to standard output with the first line deleted.
Then you can redirect the standard output to another file to capture the results.
sed '1d' INPUT_FILE_NAME > OUTPUT_FILE_NAME
That should do it.
If you have to use grep and just don't want to display the line with first on it, then try this:
grep -v first INPUT_FILE_NAME
By passing the -v switch, you are telling grep to show you everything but the expression that you are passing. In effect show me everything but the line(s) with first in them.
However, the downside is that a file with multiple first's in it will not show those other lines either and may not be the behavior that you are expecting.
To shunt the results into a new file, try this:
grep -v first INPUT_FILE_NAME > OUTPUT_FILE_NAME
Hope this helps.
I don't really understand what do you want to match. I would not use grep, but one of the following:
tail -2 file # to get last two lines
head -n +2 file # to get all but first line
sed -e '2,3p;d' file # to get lines from second to third
(not sure how standard it is, it works in GNU tools for sure)
So you just don't want the line containing "first"? -v inverts the grep results.
$ echo -e "first\nsecond\nthird\n" | grep -v first
second
third
Line? Or lines?
Try
grep -E -e '(second|third)' filename
Edit: grep is line oriented. you're going to have to use either Perl, sed or awk to perform the pattern match across lines.
BTW -E tell grep that the regexp is extended RE.
grep -A1 "second" | grep -B1 "third" works nicely, and if you have multiple matches it will even get rid of the original -- match delimiter
grep -E '(second|third)' /path/to/file
egrep -w 'second|third' /path/to/file
you could use
$ grep -1 third filename
this will print a string with match and one string before and after. Since "third" is in the last string you get last two strings.
I like notnoop's answer, but building on AndrewY's answer (which is better for those without pcregrep, but way too complicated), you can just do:
RESULT=`grep -A1 -s -m1 '^\s*second\s*$' file | grep -s -B1 -m1 '^\s*third\s*$'`
grep -v '^first' filename
Where the -v flag inverts the match.