just a quick question regarding a doubt. What's the difference between these two:
grep "$genre" "$i" | grep "$type" -c
and
grep "$genre" "$i" | grep -c "$type"
Do they do perhaps the same thing?
The POSIX standard-mandated behavior for grep "$type" -c, presuming that $type does not expand to a string starting with a dash, is to treat -c as a filename.
Only nonstandard versions of grep (such as the one built by the GNU project and commonly found on Linux systems) will treat -c as an option enabling "count" behavior when it isn't before the first positional argument.
It's bad practice to write your scripts to require nonstandard tools unless they gain some concrete benefit from those tools. Use grep -c "$type".
As you can see from man grep, the -c switch belongs to the so-called "General Output Control". Those switches can be placed on different places in the grep ... command and this has no impact on the general outcome, so indeed, both lines you mention are equal.
Related
My Goal
I'm writing a small Bash script, which uses entr, which is a utility to re-run arbitrary commands when it detects file-system events. My immediate goal is to pass entr a command which converts a given markdown file to HTML. entr will run this command every time the markdown file changes. A simplified but working script looks like:
# script 1
in="$1"
out="${in%.md}.html"
echo "$in" | entr pandoc "${in}" -o "${out}"
This works fine. The filename to be watched is supplied to entr on stdin. On detecting changes in that file, entr runs the command specified by its args. In this example that is pandoc, and all the args after it, to convert the markdown file to an HTML file.
For future reference, set -x shows that entr was invoked as we'd expect. (Throughout, lines starting with + show the output from set -x):
+ entr pandoc 'READ ME.md' -o 'READ ME.html'
The problem
I want to look-up the command given to entr depending on the file-type of the
given input file. So the file-conversion command ends up in a variable, and I want to use that variable as the command-line args to entr. But I can't get the quoting right.
Again, simplified:
# script 2
in="$1"
out="${in%.md}.html"
cmd="pandoc \"${in}\" -o \"${out}\""
echo "$in" | entr "$cmd"
(shellcheck.net detects no issues on the above)
This fails. Because "$cmd" in the final line is in quotes, the entirety of $cmd
is treated as a single arg to entr:
+ entr 'pandoc "READ ME.md" -o "READ ME.html"'
entr tries to interpret the whole thing as the name of an executable, which
it cannot find:
entr: exec pandoc "READ ME.md" -o "READ ME.html": No such file or directory
So how should I modify script 2, to use the content of $cmd as the args to
entr?
What have I tried?
Check that $cmd is being formed as I expect? If I echo "$cmd" right after
it is defined in script 2, it looks exactly how I'd hope:
pandoc "READ ME.md" -o "READ ME.html"
I tried messing around with alternate ways of constructing cmd, such as:
cmd='pandoc "'"${in}"'" -o "'"${out}"'"'
but variations like this produce identical values of $cmd, and identical
behavior as script2.
Try not quoting the use of $cmd?
Since the final line of script 2 erroneously treats the whole of "$cmd"
as a single arg, and we want it to split up the words into seprate args
instead, maybe removing the quotes and using a bare $cmd is a step in the
right direction?
echo "$in" | entr $cmd
Predictably enough though, this splits $cmd up on every space, even the
ones inside our double-quotes:
+ entr pandoc '"READ' 'ME.md"' -o '"READ' 'ME.html"'
This makes Pandoc try, and fail, to open a file called "READ:
pandoc: "READ: openBinaryFile: does not exist (No such file or directory)
Try constructing $cmd using printf?
I notice printf -v can store output in a variable. How about using that
instead of assiging to cmd?
printf -v cmd 'pandoc "%s" -o "%s"' "$in" "$out"
Predictably enough, this produces the same results as script2. I tried some
speculative variations, such as %q in the format string, or using $in
and $out directly in the format string, but didn't stumble on anything
that seemed to help.
Try using the ${var#Q} form of parameter expansion.
echo "$in" | entr ${cmd#Q}
Tried with and without double quotes around the use of ${cmd#q}. No joy,
I guess I'm misunderstanding what #Q is for.
+ entr ''\''pandoc' '"READ' 'ME.md"' -o '"READ' 'ME.html"'\'''
entr: exec 'pandoc: No such file or directory
Details
I'm using Bash v5.1.16, in Pop!_OS 22.04, derived from Ubuntu 22.04 (Jammy).
The current 'apt' version of entr (v5.1) in Ubuntu Jammy (22.04) is too old
for my needs (e.g. the -z flag doesn't work.) so I'm compiling my own from
the latest v5.3 source release.
I know there are a lot of questions about quoting in Bash, but I don't see any that seem to match this. Apologies if I'm wrong.
Assemble the command as an array, instead of a string.
I read somewhere that maybe $# might do what I need, so I put the parts of $cmd into an array:
in="$1"
out="${in%.md}.html"
cmd=(pandoc "$in" -o "$out")
echo "$in" | entr "${cmd[#]}"
This correctly quotes the items in ${cmd[#]} which require it (e.g. have spaces in.)
+ entr pandoc 'READ ME.md' -o 'READ ME.html'
So ‘entr’ successfully calls ‘pandoc’, which successfully converts the documents. It works! I confess I did not expect that.
This approach seems viable for other similar situations, not just when invoking entr.
So I have a solution. It doesn't seem completely ideal for my future plans. I had visions of these 'file conversion commands' being configurable, and hence defined in a text file somewhere, so that users (==me, probably) could override them and define their own, and I'm not fluent enough with Bash to be sure how to go about that when commands are defined as arrays instead of strings.
I can't help but feel I've overlooked something simpler.
Use a shell to interpret the value of "$cmd":
echo "$in" | entr sh -c "$cmd"
This approach seems viable for other similar situations, not just when invoking entr.
Similarly, entr has a -s option which invokes a shell for you (chosen using the first word in $SHELL):
echo "$in" | entr -s "$cmd"
These both work well, at the minor cost of spawning an extra shell process.
Coreutils stat have --format= switch which report different info about file (owner, size, etc) in easy form for readers.
POSIX ls utility provide most of this info, but its output is hard to parse. Compare with one-liner:
[ `stat -c '%U' $f` = $USER ] && echo "You own $f" || echo Error!
Are there stat utility analog in POSIX?
It is not possible :-(
Your options are:
Use ls and parse that with awk; the output of ls -l is in POSIX, so you
can rely on that. This works okay for some fields (such as the owner in your
example), and not so good for others (such as the mtime).
Detect the stat version and switch parameters; GNU stat has -c, BSD stat has
-f, other versions perhaps something else. stat is not in POSIX at all, and I don't know how widespread it is beyond Linux, BSD, and OSX, though.
Use a Perl or Python one-liner; this is not even remotely POSIX of course, but
making the assumption that at least one of these languages is present is a
fairly reasonable one in 2015, and it is easily detectable at startup if they are indeed present. It's also
not an option if performance is any concern.
Example, I've used mtime in all these examples, as this is difficult to get with ls:
#!/bin/sh
file="/etc/passwd"
perl -e "print((stat(\"$file\"))[9])"
echo
echo "$file" | perl -e '$i = <STDIN>; chomp($i); print((stat($i))[9])'
echo
python -c "import os; print(os.stat(\"$file\").st_mtime, end='')"
echo
echo "$file" | python -c "import os, sys; print(os.stat(sys.stdin.readline()[:-1]).st_mtime, end='')"
echo
I would recommend the Perl version; not because I like Perl, but because this
Python example only work correctly with Python 3 (specifically, the end=''
bit to prevent printing newlines.
A version to work with both Python 2 & 3 gets rather long:
python2 -c "from __future__ import print_function; import os; print(os.stat('/etc/passwd') .st_mtime, end='')"
You could also expand this with other languages (Ruby, PHP, Tcl, etc.), but Perl & Python are the most widespread by far.
Documentation for:
Perl stat(),
Perl lstat()
Python os.stat()
.
The short answer is no, POSIX does not provide a simple way to get the same output as stat. However, you can usually get relevant bits using other tools. To get the owner specifically:
ls -ld /path/of/file/or/directory | awk '{print $3}'
I'm trying to define bash auto-completion for teamocil so that when I type teamocil <tab> it should complete with the file names in the folder ~/.teamocil/ without the file extensions. There's an example for zsh in the website:
compctl -g '~/.teamocil/*(:t:r)' teamocil
how can I use this in bash?
Edit: Influenced by michael_n 's answer I have come up with a one-liner:
complete -W "$(teamocil --list)" teamocil
Here's a generalized version of another completion script I have that does something similar. It assumes a generic hypothetical command "flist", using some directory of files defined by FLIST_DIR to complete the command (omitting options).
Modify the following for your program (teamocil), change the default dir from $HOME/flist to $HOME/.teamocil), define your own filters/transformations, etc; and then just source it (e.g., . ~/bin/completion/bash_completion_flist), optionally adding it to your existing list of bash completions.
# bash_completion_flist:
# for some hypothetical command called "flist",
# generate completions using a directory of files
FLIST_DIR=${FLIST_DIR=:-"$HOME/flist"}
_flist_list_files() {
ls $FLIST_DIR | sed 's/\..*//'
}
_flist() {
local cur="${COMP_WORDS[COMP_CWORD]}"
COMPREPLY=()
[[ ${cur} != -* ]] \
&& COMPREPLY=($(compgen -W "$(_flist_list_files)" -- ${cur}))
}
complete -o bashdefault -o default -o nospace -F _flist flist 2>/dev/null \
|| complete -o default -o nospace -F _flist flist
Notes:
it could be shorter, but this is more or less a template for longer, more complicated completions. (Functions are Good.)
the actual completion command (complete -o ...) is a bit of a hack to work across different versions of bash.
the suffix stripping is over-simplfied if there are "." in the filename, and is left as an exercise for the reader :-) There are multiple ways to do this (sed, awk, etc); the best is via bash-isms (base=${filename%.*}), but the easiest is arguably the simple sed with some assumptions about the filename format.
Bash implements similar idea but another way, so commands and files for zsh won't work in bash. But you may write your own rules for autocompletion. More info:
An introduction to bash completion: part 1
An introduction to bash completion: part 2
SO: How to enable tab-completion of command line switches in bash?
SO: Auto-complete command line arguments
I am, like many non-engineers or non-mathematicians who try writing algorithms, an intuitive. My exact psychological typology makes it quite difficult for me to learn anything serious like computers or math. Generally, I prefer audio, because I can engage my imagination more effectively in the learning process.
That said, I am trying to write a shell script that will help me master Linux. To that end, I copied and pasted a list of Linux commands from the O'Reilly website's index to the book Python In a Nutshell. I doubt they'll mind, and I thank them for providing it. These are the textfile `massivelistoflinuxcommands,' not included fully below in order to save space...
OK, now comes the fun part. How do I get this script to work?
#/bin/sh
read -d 'massivelistoflinuxcommands' commands <<EOF
accept
bison
bzcmp
bzdiff
bzgrep
bzip2
bzless
bzmore
c++
lastb
lastlog
strace
strfile
zmore
znew
EOF
for i in $commands
do
$i --help | less | cat > masterlinuxnow
text2wave masterlinuxnow -o ml.wav
done
It really helps when you include error messages or specific ways that something deviates from expected behavior.
However, your problem is here:
read -d 'massivelistoflinuxcommands' commands <<EOF
It should be:
read -d '' commands <<EOF
The delimiter to read causes it to stop at the first character it finds that matches the first character in the string, so it stops at "bzc" because the next character is "m" which matches the "m" at the beginning of "massive..."
Also, I have no idea what this is supposed to do:
$i --help | less | cat > masterlinuxnow
but it probably should be:
$i --help > masterlinuxnow
However, you should be able to pipe directly into text2wave and skip creating an intermediate file:
$i --help | text2wave -o ml.wav
Also, you may want to prevent each file from overwriting the previous one:
$i --help | text2wave -o ml-$i.wav
That will create files named like "ml-accept.wav" and "ml-bison.wav".
I would point out that if you're learning Linux commands, you should prioritize them by frequency of use and/or applicability to a beginner. For example, you probably won't be using bison right away`.
The first problem here is that not every command has a --help option!! In fact the very first command, accept, has no such option! A better approach might be executing man on each command since a manual page is more likely to exist for each of the commands. Thus change;
$i --help | less | cat > masterlinuxnow
to
man $i >> masterlinuxnow
note that it is essential you use the append output operator ">>" instead of the create output operator ">" in this loop. Using the create output operator will recreate the file "masterlinuxnow" on each iteration thus containing only the output of the last "man $i" processed.
you also need to worry about whether the command exists on your version of linux (many commands are not included in the standard distribution or may have different names). Thus you probably want something more like this where the -n in the head command should be replace by the number of lines you want, so if you want only the first 2 lines of the --help output you would replace -n with -2:
if [ $(which $i) ]
then
$i --help | head -n >> masterlinuxnow
fi
and instead of the read command, simply define the variable commands like so:
commands="
bison
bzcmp
bzdiff
bzgrep
bzip2
bzless
bzmore
c++
lastb
lastlog
strace
strfile
zmore
znew
"
Putting this all together, the following script works quite nicely:
commands="
bison
bzcmp
bzdiff
bzgrep
bzip2
bzless
bzmore
c++
lastb
lastlog
strace
strfile
zmore
znew
"
for i in $commands
do
if [ $(which $i) ]
then
$i --help | head -1 >> masterlinuxnow 2>/dev/null
fi
done
You're going to learn to use Linux by listening to help descriptions? I really think that's a bad idea.
Those help commands usually list every obscure option to a command, including many that you will never use-- especially as a beginner.
A guided tutorial or book would be much better. It would only present the commands and options that will be most useful. For example, that list of commands you gave has many that I don't know-- and I've been using Linux/Unix extensively for 10 years.
I'm new to bash scripts (and the *nix shell altogether) but I'm trying to write this script to make grepping a codebase easier.
I have written this
#!/bin/bash
args=("$#");
for arg in args
grep arg * */* */*/* */*/*/* */*/*/*/*;
done
when I try to run it, this is what happens:
~/Work/richmond $ ./f.sh "\$_REQUEST\['a'\]"
./f.sh: line 4: syntax error near unexpected token `grep'
./f.sh: line 4: ` grep arg * */* */*/* */*/*/* */*/*/*/*;'
~/Work/richmond $
How do I do this properly?
And, I think a more important question is, how can I make grep recurse through subdirectories properly like this?
Any other tips and/or pitfalls with shell scripting and using bash in general would also be appreciated.
The syntax error is because you're missing do. As for searching recursively if your grep has the -R option you would do:
#!/bin/bash
for arg in "$#"; do
grep -R "$arg" *
done
Otherwise you could use find:
#!/bin/bash
for arg in "$#"; do
find . -exec grep "$arg" {} +
done
In the latter example, find will execute grep and replace the {} braces with the file names it finds, starting in the current directory ..
(Notice that I also changed arg to "$arg". You need the dollar sign to get the variable's value, and the quotes tell the shell to treat its value as one big word, even if $arg contains spaces or newlines.)
On recusive grepping:
Depending on your grep version, you can pass -R to your grep command to have it search Recursively (in subdirectories).
The best solution is stated above, but try putting your statement in back ticks:
`grep ...`
You should use 'find' plus 'xargs' to do the file searching.
for arg in "$#"
do
find . -type f -print0 | xargs -0 grep "$arg" /dev/null
done
The '-print0' and '-0' options assume you're using GNU grep and ensure that the script works even if there are spaces or other unexpected characters in your path names. Using xargs like this is more efficient than having find execute it for each file; the /dev/null appears in the argument list so grep always reports the name of the file containing the match.
You might decide to simplify life - perhaps - by combining all the searches into one using either egrep or grep -E. An optimization would be to capture the output from find once and then feed that to xargs on each iteration.
Have a look at the findrepo script which may give you some pointers
If you just want a better grep and don't want to do anything yourself, use ack, which you can get at http://betterthangrep.com/.