I have some data that looks like this
"thumbnailUrl": "http://placehold.it/150/adf4e1"
I want to know how I can get the trailing part of the URL, I want the output to be
adf4e1
I was trying to grep when starting with / and ending with " but I'm only a beginner in shell scripting and need some help.
I came up with a quick and dirty solution, using grep (with perl regex) and cut:
$ cat file
"thumbnailUrl": "http://placehold.it/150/adf4e1"
"anotherUrl": "http://stackoverflow.com/questions/3979680"
"thumbnailUrl": "http://facebook.com/12f"
"randortag": "http://google.com/this/is/how/we/roll/3fk19as1"
$ cat file | grep -o '/\w*"$' | cut -d'/' -f2- | cut -d'"' -f1
adf4e1
3979680
12f
3fk19as1
We could kill this with a thousand little cuts, or just one blow from Awk:
awk -F'[/"]' '{ print $(NF-1); }'
Test:
$ echo '"thumbnailUrl": "http://placehold.it/150/adf4e1"' \
| awk -F'[/"]' '{ print $(NF-1); }'
adf4e1
Filter thorugh Awk using double quotes and slashes as field separators. This means that the trailing part ../adf4e1" is separated as {..}</>{adf4e1}<">{} where curly braces denote fields and angle brackets separators. The Awk variable NF gives the 1-based number of fields and so $NF is the last field. That's not the one we want, because it is blank; we want $(NF-1): the second last field.
"Golfed" version:
awk -F[/\"] '$0=$(NF-1)'
If the original string is coming from a larger JSON object, use something like jq to extract the value you want.
For example:
$ jq -n '{thumbnail: "http://placehold.it/150/adf4e1"}' |
> jq -r '.thumbnail|split("/")[-1]'
adf4e1
(The first command just generates a valid JSON object representing the original source of your data; the second command parses it and extracts the desired value. The split function splits the URL into an array, from which you only care about the last element.)
You can also do this purely in bash using string replacement and substring removal if you wrap your string in single quotes and assign it to a variable.
#!/bin/bash
string='"thumbnailUrl": "http://placehold.it/150/adf4e1"'
string="${string//\"}"
echo "${string##*/}"
adf4e1 #output
You can do that using 'cut' command in linux. Cut it using '/' and keep the last cut. Try it, its fun!
Refer http://www.thegeekstuff.com/2013/06/cut-command-examples
Related
My input is test.txt which contains data in this format:
'X'=>'ABCDEF',
'X'=>'XYZ',
'X'=>'GHIJKLMN',
I want to get something like:
'ABCDEF',
'XYZ',
'GHIJKLMN',
How do I go about this in bash?
Thanks!
If the input never contains the character > elsewhere than in the "fat arrow", you can use cut:
cut -f2 -d\> file
-d specifies the delimiter, here > (backslash needed to prevent the shell from interpreting it as the redirection operator)
-f specifies which field to extract
Here's a solution using sed:
curl -sL https://git.io/fjeX4 | sed 's/^.*>//'
Sed is passed a single command: s///. is a regex that matches any characters (.*) from the beginning of the line (^) to the last '>'. The is an empty string, so essentially sed is just deleting all the characters on the line up to the last >. As with the other solutions, this solution assumes that there is only one '>' on the line.
If the data is really uniform, then you could just run cut (on example input):
$ curl -sL https://git.io/fjeX4 | cut -d '>' -f 2
'ABCDEF',
'XYZ',
'GHIJKLMN',
You can see flag explanations on explainshell.
With awk, it would look similar:
$ curl -sL https://git.io/fjeX4 | awk -F '>' '{ print $2 }'
'ABCDEF',
'XYZ',
'GHIJKLMN',
Using awk
awk 'BEGIN{FS="=>"}{print $2}' file
'ABCDEF',
'XYZ',
'GHIJKLMN',
FS in awk stands for field separator. The code inside BEGIN is executed only at the beginning, ie, before processing the first record. $2 prints the second field.
A more idiomatic way of putting the above stuff would be
awk 'BEGIN{FS="=>"}$2' file
'ABCDEF',
'XYZ',
'GHIJKLMN',
The default action in awk is to print the record. Here we explicitly mention what to print. ie $2.
I am having trouble figuring out how to grep the characters between two single quotes .
I have this in a file
version: '8.x-1.0-alpha1'
and I like to have the output like this (the version numbers can be various):
8.x-1.0-alpha1
I wrote the following but it does not work:
cat myfile.txt | grep -e 'version' | sed 's/.*\?'\(.*?\)'.*//g'
Thank you for your help.
Addition:
I used the sed command sed -n "s#version:\s*'\(.*\)'#\1#p"
I also like to remove 8.x- which I edited to sed -n "s#version:\s*'8.x-\(.*\)'#\1#p".
This command only works on linux and it does not work on MAC. How to change this command to make it works on MAC?
sed -n "s#version:\s*'8.x-\(.*\)'#\1#p"
If you just want to have that information from the file, and only that you can quickly do:
awk -F"'" '/version/{print $2}' file
Example:
$ echo "version: '8.x-1.0-alpha1'" | awk -F"'" '/version/{print $2}'
8.x-1.0-alpha1
How does this work?
An awk program is a series of pattern-action pairs, written as:
condition { action }
condition { action }
...
where condition is typically an expression and action a series of commands.
-F "'": Here we tell awk to define the field separator FS to be a <single quote> '. This means the all lines will be split in fields $1, $2, ... ,$NF and between each field there is a '. We can now reference these fields by using $1 for the first field, $2 for the second ... etc and this till $NF where NF is the total number of fields per line.
/version/{print $2}: This is the condition-action pair.
condition: /version/:: The condition reads: If a substring in the current record/line matches the regular expression /version/ then do action. Here, this is simply translated as if the current line contains a substring version
action: {print $2}:: If the previous condition is satisfied, then print the second field. In this case, the second field would be what the OP requests.
There are now several things that can be done.
Improve the condition to be /^version :/ && NF==3 which reads _If the current line starts with the substring version : and the current line has 3 fields then do action
If you only want the first occurance, you can tell the system to exit immediately after the find by updating the action to {print $2; exit}
I'd use GNU grep with pcre regexes:
grep -oP "version: '\\K.*(?=')" file
where we are looking for "version: '" and then the \K directive will forget what it just saw, leaving .*(?=') to match up to the last single quote.
Try something like this: sed -n "s#version:\s*'\(.*\)'#\1#p" myfile.txt. This avoids the redundant cat and grep by finding the "version" line and extracting the contents between the single quotes.
Explanation:
the -n flag tells sed not to print lines automatically. We then use the p command at the end of our sed pattern to explicitly print when we've found the version line.
Search for pattern: version:\s*'\(.*\)'
version:\s* Match "version:" followed by any amount of whitespace
'\(.*\)' Match a single ', then capture everything until the next '
Replace with: \1; This is the first (and only) capture group above, containing contents between single quotes.
When your only want to look at he quotes, you can use cut.
grep -e 'version' myfile.txt | cut -d "'" -f2
grep can almost do this alone:
grep -o "'.*'" file.txt
But this may also print lines you don't want to: it will print all lines with 2 single quotes (') in them. And the output still has the single quotes (') around it:
'8.x-1.0-alpha1'
But sed alone can do it properly:
sed -rn "s/^version: +'([^']+)'.*/\1/p" file.txt
I have a well-formed CSV file, which may or may not have a header line; and may or may not have quoted data. I want to determine the number of columns in it, using the shell.
Now, if I can be sure there are no quoted commas in the file, the following seems to work:
x=$(tail -1 00-45-19-tester-trace.csv | grep -o , | wc -l); echo $((x + 1))
but what if I can't make that assumption? That is, what if I can't assume a comma is always a field separator? How do I do it then?
If it helps, you're allowed to make the assumption of there being no quoted quotes (i.e. \"s between within quoted strings); but better not to make that one either.
If you cannot make any optimistic assumptions about the data, then there won't be a simple solution in Bash. It's not trivial to parse a general CSV format with possible embedded newlines and embedded separators. You're better off not writing that in bash, but using an existing proper CSV parse. For example Python has one built in its standard library.
If you can assume that there are no embedded newlines and no embedded separators, than it's simple to split in commas using awk:
awk -F, '{ print NF; exit }' input.csv
-F, tells awk to use comma as the field separator, and the automatic NF variable is the number of fields on the current line.
If you want to allow embedded separators, but you can assume no embedded double quotes, then you can eliminate the embedded separators with a simple filter, before piping to the same awk as earlier:
head -n 1 input.csv | sed -e 's/"[^"]*"//g' | awk ...
Note that both of these examples use the first line to decide the number of fields. If the input has a header line, this should work quite well, as the header should not contain embedded newlines
count fields in first row, then verify all rows have same number
CNT=$(head -n1 hhdata.csv | awk -F ',' '{print NF}')
cat hhdata.csv | awk -F ',' '{print NF}' | grep -v $CNT
Doesn't cope with embedded commas but will highlight if they exist
If File has not double quotes then use below command:
awk -F"," '{ print NF }' filename| sort -u
If File has every column enclosed with double quotes then use below command:
awk -F, '{gsub(/"[^"]*"/,x);print NF}' filename | sort -u
I am trying to get the value from the string:
define('__mypassword', 'value');
I am trying to use cut and grep for this.
grep "__mypassword'" myfile.php | cut -d ',' -f 2
This returns 'value');
I do not need the quotes or braces or semi column. How do I take the value out without using multiple cut statements?
Juse use awk!
$ awk -F"'" '/__mypassword/{print $4}' <<< "define('__mypassword', 'value');"
value
This sets the field separator to the single quote. This way, it is just a matter of printing the 4th element, which is the one after the 3rd quote. /__mypassword/ acts as grep "__mypassword".
In case you also need to match the single quote, use /__mypassword'\''/ (a bit picky, you need to close the awk statement to include a single quote).
Don't count commas, count single quotes:
grep "__mypassword'" myfile.php | cut -d"'" -f4
kent$ grep -oP "[^']+(?='\))" <<<"define('__mypassword', 'value');"
value
Lets say that I have a command list kittens that returns something in this multi line format in my terminal (in this exact layout):
[ 'fluffy'
'buster'
'bob1' ]
How can I fetch bob1 and assign to a variable for scripting use? Here's my non working try so far.
list kittens | grep "'([^']+)' \]"
I am not overly familiar with grepping on the cli and am running into issues of syntax with quotes and such.
If you know that bob1 will be in the last line, you can capture it like that:
myvar="$(list kittens | tail -n1 | grep -oP "'\K[^']+(?=')")"
This uses tail to find the last line and then grep with a lookahead and a lookbehind in the regular expression to extract the part inside the quotes.
Edit: The above assume that you are using GNU grep (for the -P mode). Here's an alternative with sed:
myvar="$(list kittens | tail -n1 | sed -e "s/^[^']*'//; s/'[^']*$//")"
Could be done by awk alone:
list kittens |awk 'END{gsub(/\047|[[:blank:]]|\]/,"");print $0}'
bob1
Example:
echo "$kit"
[ 'fluffy'
'buster'
'bob1' ]
echo "$kit" |awk 'END{gsub(/\047|[[:blank:]]|\]/,"");print $0}'
bob1
To Assign it to any variable:
var=$(list kittens |awk 'END{gsub(/\047|[[:blank:]]|\]/,"");print $0}'
Explanation:
END{}: End block is used to take data from last line as we are interested only for last line.
gsub: This is awk's inbuilt function for search and replacement tasks. Here white space and double quoted and single quotes are removed. Not that \047 is used for single quote replacement.