I am trying to find all instances of "type":"FollowEvent", and then within those instances, if the string "actor": is not followed by {, then capture the string enclosed in " that comes immediately after "actor":. Else, capture the string enclosed in " that comes immediately after "login:".
What I have so far:
zgrep -e '"type":"FollowEvent"' /path/to/dir/* | zgrep -o '"actor":(?!{)*' | cut -f2- -d: | cut -d',' -f1 > results_file.txt
What this does:
For all files in /path/to/dir, for all lines that contain "type":"FollowEvent", find "actor:" not followed by {. Then take everything after the :, and before the next ,. Put the results in results_file.txt.
A single line in the files that are being grep'd could look like this:
{"repo":{"url":"https://url","name":"/"},"type":"FollowEvent","public":true,"created_at":"2011-05-29","payload":{"target":{"gravatar_id":"73","id":64,"repos":35,"followers":58,"login":"username3"}},"actor":{"gravatar_id":"06","id":439,"url":"https://url","avatar_url":"https://.png","login":"username4"},"id":"14"}
or like this:
{"repo":{"url":"https://url/","name":"/"},"type":"FollowEvent","public":true,"created_at":"2011-04-01","payload":{"target":{"gravatar_id":"40","repos":2,"followers":1,"login":"username2"},"actor":"username1","actor_gravatar":"de4"},"actor":{"gravatar_id":"de4","id":716,"url":"https://url","avatar_url":"https://.png","login":"username2"},"id":"12"}
What I want:
a file containing only the usernames of actors. Here, I want, in results_file.txt:
username4
username1
Let's say:
JSON='{"repo":{"url":"https://url","name":"/"},"type":"FollowEvent","public":true,"created_at":"2011-05-29","payload":{"target":{"gravatar_id":"73","id":64,"repos":35,"followers":58,"login":"username3"}},"actor":{"gravatar_id":"06","id":439,"url":"https://url","avatar_url":"https://.png","login":"username4"},"id":"14"}'
For a simple answer, I do suggest you to use jq: https://stedolan.github.io/jq/
$ echo "$JSON" | jq -r '. | select(.type=="FollowEvent") | .actor.login'
username4
You can install it in most of distros with the default package manager.
Anyway if you need to do it with GNU tools.
$ echo "$JSON" | grep '"type":"FollowEvent"' | sed 's/.*"login":"\([^"]*\).*/\1/g'
username4
Related
I am trying to use grep to filter out the RDS snapshot identifier from the rds describe-db-snapshots command output below:
"arn:aws:rds:ap-southeast-1:123456789:snapshot:rds:apple-pie-2018-05-06-17-12",
"rds:apple-pie-2018-05-06-17-12",
how to return the exact output as in
rds:apple-pie-2018-05-06-17-12
tried using
grep -Eo ",rds:"
but not able to
Following awk may also help you on same.
awk 'match($0,/^"rds[^"]*/){print substr($0,RSTART+1,RLENGTH-1)}' Input_file
Your grep -Eo ",rds:" is failing for different reasons:
You did not add a " in the string to match
Between the comma and rds you need to match the character.
You are trying to match the comma that can be on the previous line
Your sample input is 2 lines (with a newline in between), perhaps the real input is without the newline.
You want to match until the next double quote.
You can support both input-styles (with/without newline) with
grep -Eo '(,|^)"rds:[^"]*' rdsfile |cut -d'"' -f2
You can do this in one command with
sed -rn 's/.*(,|^)"(rds:[^"]*).*/\2/p' rdsfile
EDIT: Manipulting stdout and not the file is with similar commands:
yourcommand | grep -Eo '(,|^)"rds:[^"]*' |cut -d'"' -f2
# or
yourcommand | sed -rn 's/.*(,|^)"(rds:[^"]*).*/\2/p'
You can also test the original commands with yourcommand > rdsfile.
You might notice that rdsfile is missing data that you have seen on the screen, in that case add 2>&1
yourcommand 2>&1 | grep -Eo '(,|^)"rds:[^"]*' |cut -d'"' -f2
# or
yourcommand 2>&1 | sed -rn 's/.*(,|^)"(rds:[^"]*).*/\2/p'
I have an env.txt file in the following format:
lDRIVER={ODBC Driver 13 for SQL Server};
PORT=1433;
SERVER=serveename;
DATABASE=db;
UID=username;
PWD=password!
I have a git bash script (.sh) that requires the UID and PWD from that file. I was thinking about getting it by the last/second last line number. How do I do this/ is there a better way (say looking for UID and PWD and assigning the git bash variable that way)
There's lots of ways to do this. You could use awk which I would personally use since it's sort of like an x-acto knife for this type of thing:
uid=$(awk -F"[=;]" '/UID/{print $2}' env.txt)
pwd=$(awk -F"[=;]" '/PWD/{print $2}' env.txt)
Or grep and sed. sed is nice because it allows you to get very specific about the piece of info you want to cut from the line, but it's regex which has its learning curve:
uid=$(grep "UID" env.txt | sed -r 's/^.*=(.*)(;|$)/\1/g' )
pwd=$(grep "PWD" env.txt | sed -r 's/^.*=(.*)(;|$)/\1/g' )
As #JamesK noted in the comments you can use sed and have it do the search instead of grep. This is super nice and I would definitely choose this instead of the grep | sed.
uid=$(sed -nr '/UID/s/^.*=(.*)(;|$)/\1/gp' )
pwd=$(sed -nr '/PWD/s/^.*=(.*)(;|$)/\1/gp' )
Or grep and cut. Bleh... we can all do better, but sometimes we just want to grep and cut and not have to think about it:
uid=$(grep "UID" env.txt | cut -d"=" -f2 | cut -d";" -f1)
pwd=$(grep "PWD" env.txt | cut -d"=" -f2 | cut -d";" -f1)
I definitely wouldn't go by line number though. That looks like and odbc.ini file and the order in which the parameters are listed in each odbc entry are irrelevant.
First rename PWD to something like PASSWORD. PWD is a special variable used by the shell. Even better is to use lowercase variable names for all your own variables.
When the password is without special characters (spaces, $, ), you can
source env.txt
When the password has something special, consider editing the env.txt:
lDRIVER="{ODBC Driver 13 for SQL Server}"
PORT="1433"
SERVER="serveename"
DATABASE="db"
UID="username"
PASSWORD="password!"
When you are only interested in lowercase uid and passwd, consider selecting only the interesting fields and change the keywords to lowercase
source <(sed -rn '/^(UID|PWD)=/ s/([^=]*)/\L\1/p' env.txt)
I have a CSV file with this lines:
----------+79975532211,----------+79975532212
4995876655,4995876658
I try to grep this lines in Bash script
#!/bin/bash
config='/test/config.conf'
sourcecsv=/test/sourse.csv
cat $sourcecsv | while read line
do
Oldnumber=$(echo $line | cut -d',' -f1)
cat $config | grep "\\$Oldnumber" -B 8
done
But when script grep value 4995876655 I get error:
grep: Invalid back reference
How I can grep all values in my file?
Instead of:
cat $config | grep "\\$Oldnumber" -B 8
You should do:
grep -B 8 -F -- "$Oldnumber" "$config"
If you really mean to grep for all strings between commas, you can do it all in one go.
tr ',' '\n' </test/sourse.csv |
grep -F -f - -B 8 /test/config.conf
If you need to obtain the matches in sequence (all matches for the first string followed by all matches for the second, etc) then maybe loop over them with a proper while loop:
tr ',' '\n' </test/sourse.csv |
while read -r Oldnumber; do
grep -F -B 8 -e "$Oldnumber" /test/config.conf
done
Keeping the file names in variables does not seem to offer any advantage here.
If you mean to search for the strings preceded by a literal backslash, you can add it back; the -F option I added turns all strings into literals. If you need metacharacters, and take out the -F option, you need to double the backslashes (inside double quotes, a single backslash needs to be represented as double; and to get a literal backslash in a regular expression, you need two of them).
Hey so got another predicament that I am stuck in. I wanted to see approximately how many Indian people are using the stampede computer. So I set up an indian txt file in vim that has about 50 of the most common surnames in india and I want to compare those names in the file to the user name list.
So far this is the code I have
getent passwd | cut -f 5 -d: | cut -f -d' '
getent passwd gets the userid list which is going to look like this
tg827313:x:827313:8144474:Brandon Williams
the cut functions will get just the last name so the output of the example will be
Williams
Now can use the grep function to compare files but how do I use it to compare the getent passwd list with the file?
To count how many of the last names of computer users appear in the file namefile, use:
getent passwd | cut -f 5 -d: | cut -f -d' ' | grep -wFf namefile | wc -l
How it works
getent passwd | cut -f 5 -d: | cut -f -d' '
This is your code which I will assume works as intended for you.
grep -wFf namefile
This selects names that match a line in namefile. The -F option tells grep not to use regular expressions for the names. The names are assumed to be fixed strings. The option -f tells grep to read the strings from file. -w tells grep to match whole words only.
wc -l
This returns a count of the lines in the output.
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2`
outputs something like this
"Abc Inc";
What I want to do is I want to remove the trailing ";" as well. How can i do that? I am a beginner to bash. Any thoughts or suggestions would be helpful.
This will remove the last character contained in your COMPANY_NAME var regardless if it is or not a semicolon:
echo "$COMPANY_NAME" | rev | cut -c 2- | rev
I'd use sed 's/;$//'. eg:
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2 | sed 's/;$//'`
foo="hello world"
echo ${foo%?}
hello worl
I'd use head --bytes -1, or head -c-1 for short.
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2 | head --bytes -1`
head outputs only the beginning of a stream or file. Typically it counts lines, but it can be made to count characters/bytes instead. head --bytes 10 will output the first ten characters, but head --bytes -10 will output everything except the last ten.
NB: you may have issues if the final character is multi-byte, but a semi-colon isn't
I'd recommend this solution over sed or cut because
It's exactly what head was designed to do, thus less command-line options and an easier-to-read command
It saves you having to think about regular expressions, which are cool/powerful but often overkill
It saves your machine having to think about regular expressions, so will be imperceptibly faster
I believe the cleanest way to strip a single character from a string with bash is:
echo ${COMPANY_NAME:: -1}
but I haven't been able to embed the grep piece within the curly braces, so your particular task becomes a two-liner:
COMPANY_NAME=$(grep "company_name" file.txt); COMPANY_NAME=${COMPANY_NAME:: -1}
This will strip any character, semicolon or not, but can get rid of the semicolon specifically, too.
To remove ALL semicolons, wherever they may fall:
echo ${COMPANY_NAME/;/}
To remove only a semicolon at the end:
echo ${COMPANY_NAME%;}
Or, to remove multiple semicolons from the end:
echo ${COMPANY_NAME%%;}
For great detail and more on this approach, The Linux Documentation Project covers a lot of ground at http://tldp.org/LDP/abs/html/string-manipulation.html
Using sed, if you don't know what the last character actually is:
$ grep company_name file.txt | cut -d '=' -f2 | sed 's/.$//'
"Abc Inc"
Don't abuse cats. Did you know that grep can read files, too?
The canonical approach would be this:
grep "company_name" file.txt | cut -d '=' -f 2 | sed -e 's/;$//'
the smarter approach would use a single perl or awk statement, which can do filter and different transformations at once. For example something like this:
COMPANY_NAME=$( perl -ne '/company_name=(.*);/ && print $1' file.txt )
don't have to chain so many tools. Just one awk command does the job
COMPANY_NAME=$(awk -F"=" '/company_name/{gsub(/;$/,"",$2) ;print $2}' file.txt)
In Bash using only one external utility:
IFS='= ' read -r discard COMPANY_NAME <<< $(grep "company_name" file.txt)
COMPANY_NAME=${COMPANY_NAME/%?}
Assuming the quotation marks are actually part of the output, couldn't you just use the -o switch to return everything between the quote marks?
COMPANY_NAME="\"ABC Inc\";" | echo $COMPANY_NAME | grep -o "\"*.*\""
you can strip the beginnings and ends of a string by N characters using this bash construct, as someone said already
$ fred=abcdefg.rpm
$ echo ${fred:1:-4}
bcdefg
HOWEVER, this is not supported in older versions of bash.. as I discovered just now writing a script for a Red hat EL6 install process. This is the sole reason for posting here.
A hacky way to achieve this is to use sed with extended regex like this:
$ fred=abcdefg.rpm
$ echo $fred | sed -re 's/^.(.*)....$/\1/g'
bcdefg
Some refinements to answer above. To remove more than one char you add multiple question marks. For example, to remove last two chars from variable $SRC_IP_MSG, you can use:
SRC_IP_MSG=${SRC_IP_MSG%??}
cat file.txt | grep "company_name" | cut -d '=' -f 2 | cut -d ';' -f 1
I am not finding that sed 's/;$//' works. It doesn't trim anything, though I'm wondering whether it's because the character I'm trying to trim off happens to be a "$". What does work for me is sed 's/.\{1\}$//'.