I'm trying to get the version of my compilation from a text file. I'm using this command
grep -w -m 1 "V1" server.h | sed $(VERSION) 's/#define V1[\t]//'
It works fine but now I'm trying to execute it from my Makefile using shell:
VERSION=$(shell grep -w -m 1 "V1" server.h | sed $(VERSION) 's/#define V1[\t]//')
but I'm not able to make it works because of the |. If I only put one command, grep for example it runs fine, there's another way to indicate the | to concatenate expressions? Or how else can I do this?
Edit:
Hi, thanks for the answer,
Reading your answer i realize that copy/paste has betrayed me jeje, this is the right expression i'm using:
VERSION:=$(shell grep -w -m 1 "V2" server.h | sed 's/#define V2[\t]//')
And this is the output of Makefile:
unterminated call to function shell': missing)'. Stop.
I have tested your recommendations and it also fails with the same Error.
Thanks.
It has nothing to do with the pipe. The reason it is not working is because you have defined it as a dynamic macro. By using '=' it will re-evaluate the macro on each reference. You have essentially created a recursive macro by trying to overload the VERSION variable as both the version and the filename. Try using different variable names and make it static:
VERSION_NUMBER:=$(shell grep -w -m 1 "V1" server.h | sed $(VERSION) 's/#define V1[\t]//')
$(error VERSION_NUMBER=$(VERSION_NUMBER))
Remove the $(error) once you have it working. Also, awk might be more efficient in this case anyway:
VERSION_NUMBER:=$(shell awk '/\<V1\>/ { print gensub(/^\#define V1[[:space:]]+/, "", ""); exit }' $(VERSION))
You also have the problem of the hash (pound for you US fellas) #. It is terminating the expression as a comment. Try this:
VERSION:=$(shell grep -w -m 1 "V2" server.h | sed 's/\#define V2[\t]//')
Or this:
VERSION:=$(shell awk '/\<V1\>/ { print gensub(/^\#define V1[[:space:]]+/, "", ""); exit }' server.h)
You will have this problem with many characters in make. The Dollar being the most problematic to escape. I have seen expressions like this:
V:=$(foreach f,$(list),$(eval $(shell create_rule $$$$(f))))
Sometimes it is better to write a shell script and call that:
script.sh:
#!/bin/sh
awk '/\<V1\>/ { print gensub(/^#define V1[[:space:]]+/, "", ""); exit }' "$#"
Makefile:
VERSION=$(shell script.sh server.h)
Related
I have a convenience function in my bashrc file that looks like following:
function get_pattern()
{
grep "P1" $1 | grep -v "BUT_NOT_THIS" | awk -F":" '{print $(1)}' | sort -u
}
alias gp='get_pattern'
This works fine if I run it on individual file like gp file_1.c. However I am unable to run it like find . -name "*.c" -type f | xargs gp. I also fail to run it like gp *.c. How do I code get_pattern so that I can have all these conveniences.
NOTE: I have simplified the function for easier understanding. Not expecting smart grep/awk/sed/sort hacks or tweaks. The question is I have an alias that takes filenames as arguments. Want it to work with pipes, and preferably with globs.
As pointed out by the experts, aliases are not suited for your requirement.
In your function, you are only passing the first argument to grep, as in grep "P1" $1. Change it to use all arguments, this way:
function get_pattern() {
grep "P1" "$#" | grep -v "BUT_NOT_THIS" | awk -F":" '{print $(1)}' | sort -u
}
Note:
When you invoke your function as get_pattern *.c and there are matching files, the function doesn't see *.c, it sees the list of matching files. The glob expansion is done by the shell while invoking the function, but not inside the function.
In the present format, the function doesn't read from stdin. So, piping the results of another command into your function may not work. To make it accept stdin, you need to change the first grep to:
grep "P1" - "$#"
That would mess up the invocation when you intend the function to only read the files. So, it would be better to rewrite the function this way:
function get_pattern() {
if (($# > 0)); then
# arguments passed, use them as file names to grep from
grep_args=("$#")
else
# no arguments passed, grep from stdin
grep_args=(-)
fi
grep "P1" "${grep_args[#]}" | grep -v "BUT_NOT_THIS" | awk -F":" '{print $(1)}' | sort -u
}
I'm just going to focus on the issue of piping data to xargs. That doesn't work because xargs doesn't know anything about the alias. However you can effectively pass the function definition in to make it work (non-portable solution which works in bash and possibly some other shells, but don't expect it to work everywhere) with export -f:
$ foo() { echo foo: $#; }
$ echo bar baz | xargs bash -c 'foo $#' _
_: foo: command not found
$ export -f foo
$ echo bar baz | xargs bash -c 'foo $#' _
foo: bar baz
I am not aware of any way to do this with an alias, but I'm also not aware of any reason to ever use an alias. Stop using aliases.
Given the following content:
title="Bar=1; Fizz=2; Foo_Bar=3;"
I'd like to match the first occurrence of Bar value which is 1. Also I don't want to rely on soundings of the word (like double quote in the front), because the pattern could be in the middle of the line.
Here is my attempt:
$ grep -o -m1 'Bar=[ ./0-9a-zA-Z_-]\+' input.txt
Bar=1
Bar=3
I've used -m/--max-count which suppose to stop reading the file after num matches, but it didn't work. Why this option doesn't work as expected?
I could mix with head -n1, but I wondering if it is possible to achieve that with grep?
grep is line-oriented, so it apparently counts matches in terms of lines when using -m[1]
- even if multiple matches are found on the line (and are output individually with -o).
While I wouldn't know to solve the problem with grep alone (except with GNU grep's -P option - see anubhava's helpful answer), awk can do it (in a portable manner):
$ awk -F'Bar=|;' '{ print $2 }' <<<"Bar=1; Fizz=2; Foo_Bar=3;"
1
Use print "Bar=" $2, if the field name should be included.
Also note that the <<< method of providing input via stdin (a so-called here-string) is specific to Bash, Ksh, Zsh; if POSIX compliance is a must, use echo "..." | grep ... instead.
[1] Options -m and -o are not part of the grep POSIX spec., but both GNU and BSD/OSX grep support them and have chosen to implement the line-based logic.
This is consistent with the standard -c option, which counts "selected lines", i.e., the number of matching lines:
grep -o -c 'Bar=[ ./0-9a-zA-Z_-]\+' <<<"Bar=1; Fizz=2; Foo_Bar=3;" yields 1.
Using perl based regex flavor in gnu grep you can use:
grep -oP '^(.(?!Bar=\d+))*Bar=\d+' <<< "Bar=1; Fizz=2; Foo_Bar=3;"
Bar=1
(.(?!Bar=\d+))* will match 0 or more of any characters that don't have Bar=\d+ pattern thus making sure we match first Bar=\d+
If intent is to just print the value after = then use:
grep -oP '^(.(?!Bar=\d+))*Bar=\K\d+' <<< "Bar=1; Fizz=2; Foo_Bar=3;"
1
You can use grep -P (assuming you are on gnu grep) and positive look ahead ((?=.*Bar)) to achieve that in grep:
echo "Bar=1; Fizz=2; Foo_Bar=3;" | grep -oP -m 1 'Bar=[ ./0-9a-zA-Z_-]+(?=.*Bar)'
First use a grep to make the line start with Bar, and then get the Bar at the start of the line:
grep -o "Bar=.*" input.txt | grep -o -m1 "^Bar=[ ./0-9a-zA-Z_-]\+"
When you have a large file, you can optimize with
grep -o -m1 "Bar=.*" input.txt | grep -o -m1 "^Bar=[ ./0-9a-zA-Z_-]\+"
How to extract a single value from a given json?
{
"Vpc": {
"InstanceTenancy": "default",
"State": "pending",
"VpcId": "vpc-123",
"CidrBlock": "10.0.0.0/16",
"DhcpOptionsId": "dopt-123"
}
}
Tried this but with no luck:
grep -e '(?<="VpcId": ")[^"]*'
You probably wanted -Po, which works with your regex:
$ grep -oP '(?<="VpcId": ")[^"]*' infile
vpc-123
If GNU grep with its -P option isn't available, we can't use look-arounds and have to resort to for example using grep twice:
$ grep -o '"VpcId": "[^"]*' infile | grep -o '[^"]*$'
vpc-123
The first one extracts up to and excluding the closing quotes, the second one searches from the end of the line for non-quotes.
But, as mentioned, you'd be better off properly parsing your JSON. Apart from jq mentioned in another answer, I know of
Jshon
JSON.sh
A jq solution would be as simple as this:
$ jq '.Vpc.VpcId' infile
"vpc-123"
Or, to get raw output instead of JSON:
$ jq -r '.Vpc.VpcId' infile
vpc-123
Something like
grep '^ *"VpcId":' json.file \
| awk '{ print $2 }' \
| sed -e 's/,$//' -e 's/^"//' -e 's/"$//'
you can do:
sed -r -n -e '/^[[:space:]]*"VpcId":/s/^[^:]*: *"(.*)", *$/\1/p'
but really, using any shell tools to run regexes over JSON content is a bad idea. you should consider a much saner language like python.
python -c 'import json, sys; print(json.loads(sys.stdin.read())["Vpc"]["VpcId"]);'
Try this regex pattern:
\"VpcId\":\s?(\"\S+\")
If you can install a tool I would suggest using jq jq. It allows very simple grep, with great support for piping too.
The OP asks for solutions using grep. In case he means using terminal, the node cli is an alternative, since support for JSON is total. One alternative could be the command node --eval "script"
echo '{"key": 42}' \
| node -e 'console.log(JSON.parse(require("fs").readFileSync(0).toString()).key)' //prints 42
I am not sure why i am getting the unexpected syntax '( err
#!/bin/bash
DirBogoDict=$1
BogoFilter=/home/nikhilkulkarni/Downloads/bogofilter-1.2.4/src/bogofilter
echo "spam.."
for i in 'cat full/index |fgrep spam |awk -F"/" '{if(NR>1000)print$2"/"$3}'|head -500'
do
cat $i |$BogoFilter -d $DirBogoDict -M -k 1024 -v
done
echo "ham.."
for i in 'cat full/index | fgrep ham | awk -F"/" '{if(NR>1000)print$2"/"$3}'|head -500'
do
cat $i |$BogoFilter -d $DirBogoDict -M -k 1024 -v
done
Error:
./score.bash: line 7: syntax error near unexpected token `('
./score.bash: line 7: `for i in 'cat full/index |fgrep spam |awk -F"/" '{if(NR>1000)print$2"/"$3}'|head -500''
Uh, because you have massive syntax errors.
The immediate problem is that you have an unpaired single quote before the cat which exposes the Awk script to the shell, which of course cannot parse it as shell script code.
Presumably you want to use backticks instead of single quotes, although you should actually not read input with for.
With a fair bit of refactoring, you might want something like
for type in spam ham; do
awk -F"/" -v type="$type" '$0 ~ type && NR>1000 && i++<500 {
print $2"/"$3 }' full/index |
xargs $BogoFilter -d $DirBogoDict -M -k 1024 -v
done
This refactors the useless cat | grep | awk | head into a single Awk script, and avoids the silly loop over each output line. I assume bogofilter can read file name arguments; if not, you will need to refactor the xargs slightly. If you can pipe all the files in one go, try
... xargs cat | $BogoFilter -d $DirBogoDict -M -k 1024 -v
or if you really need to pass in one at a time, maybe
... xargs sh -c 'for f; do $BogoFilter -d $DirBogoDict -M -k 1024 -v <"$f"; done' _
... in which case you will need to export the variables BogoFilter and DirBogoDict to expose them to the subshell (or just inline them -- why do you need them to be variables in the first place? Putting command names in variables is particularly weird; just update your PATH and then simply use the command's name).
In general, if you find yourself typing the same commands more than once, you should think about how to avoid that. This is called the DRY principle.
The syntax error is due to bad quoting. The expression whose output you want to loop over should be in command substitution syntax ($(...) or backticks), not single quotes.
I'm probably being stupid, but I cannot do the most basic variable assignment in my Makefile.
TEST = $(pwd);
all:
echo $(TEST)
When I run "make all" from Bash (v4.2.42) in FreeBSD (v9.1) I get:
echo
No idea what I'm screwing up. I've also tried to assign the variable using $(shell ...) with the same result.
If I use back ticks (`) then basic assignment works, but it doesn't store the result, it stores the command. This breaks in the following example Makefile:
SERVERIP = `ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $$2}'`
all:
echo $(SERVERIP)
sed -e 's/%LISTENIP%/${SERVERIP}/g' test.conf > work/test.conf.tmp
The result is:
[pete#pete] ~/make
echo `ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $2}'`
10.128.28.151
sed -e 's/%LISTENIP%/`ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $2}'`/g' test.conf > work/test.conf.tmp
sed: 1: "s/%LISTENIP%/`ifconfig ...": unescaped newline inside substitute pattern
*** [all] Error code 1
You can see that basic variable assignment appears to work, but when you insert the result into sed, it inserts the whole command rather than the result, which breaks!
Any ideas?
Pete.
Maybe the answer is too late but I hope I can help (somebody else).
If you don't want to install GNU's make you can use the !=:
TEST!=pwd
all:
echo ${TEST}
will work. The explanation is very simple, please read carefully FreeBSD's man make, especially the "Variable assignment modifiers" inside the "VARIABLE ASSIGNMENTS" section:
Variable assignment modifiers
The five operators that can be used to assign values to variables are
as follows: ...
!= Expand the value and pass it to the shell for execution and
assign the result to the variable. Any newlines in the result
are replaced with spaces.
The makefile contains:
TEST = $(pwd);
all:
echo $(TEST)
The first line assigns the value in the make variable pwd plus a semicolon to make variable TEST. The rule then echoes that. You forgot, I think, that $(...) is significant to both make and shell. Assuming you want to echo the output of the pwd command, then use a double-dollar so make expands that to one dollar:
TEST = $$(pwd)
all:
echo $(TEST)
To get the output of a shell command into a make variable, you have to use GNU make extensions. There are two extensions in use: the := immediate assignment that evaluates the value of SERVERIP at the point of definition, rather than at the point of use; and $(shell ...cmd...) that is used to run a shell command and capture its output.
SERVERIP := $(shell ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $$2}')
all:
echo ${SERVERIP}
sed -e 's/%LISTENIP%/${SERVERIP}/g' test.conf > work/test.conf.tmp
This should work. The original error, about a newline in the sed regular expression, is odd; the back-ticks should have removed the trailing newline from the output of the ifconfig pipeline. However, if there were in fact multiple lines returned, then you'd still have a newline inside, justifying the complaint from sed.
This generates the complaint:
SERVERIP = `printf "%s\n" a b c`
all:
echo $(SERVERIP)
sed -e "s/%LISTENIP%/${SERVERIP}/g" test.conf > work/test.conf.tmp
This works:
SERVERIP = `printf "%s\n" c`
all:
echo $(SERVERIP)
sed -e "s/%LISTENIP%/${SERVERIP}/g" test.conf > work/test.conf.tmp
Since you're using FreeBSD, it is quite possible that you don't have GNU make. In that case, you will need to:
Make sure the ifconfig command pipeline only produces one line of output.
Use make commands like this:
SERVERIP = `ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $$2}'`
all:
echo ${SERVERIP}
sed -e "s/%LISTENIP%/${SERVERIP}/g" test.conf > work/test.conf.tmp
That should work with any version of make provided condition 1 is met.
This is a test makefile, similar to yours. The ifconfig command has been placed in a $(shell ...) command and the result is stored in SERVERIP.
TOP:= $(shell pwd)
SERVERIP:= $(shell ifconfig en1 | grep -E 'inet.[0-9]' | awk '{ print $$2}')
LISTENIP:=240.12.4.63
all:
echo $(SERVERIP)
sed -e 's/$(LISTENIP)/$(SERVERIP)/g' test.conf > work/test.conf.tmp