I am working with a PKGBUILD file for the AUR. I have a lot of colors that need to be replaced in different files in the $pkgsrc directory and I wanted to use an associative array.
declare -A _BLACKISH_REPLACEMENTS
_BLACKISH_REPLACEMENTS['#242424']='#1C1C1C'
_BLACKISH_REPLACEMENTS['#333333']='#292929'
_BLACKISH_REPLACEMENTS['#999999']='#787878'
_BLACKISH_REPLACEMENTS['#555555']='#4C4C4C'
_BLACKISH_REPLACEMENTS['#373737']='#2E2E2E'
_BLACKISH_REPLACEMENTS['#434343']='#383838'
_BLACKISH_REPLACEMENTS['#3E3E3E']='#333333'
_BLACKISH_REPLACEMENTS['#383838']='#2E2E2E'
_BLACKISH_REPLACEMENTS['#313131']='#262626'
_BLACKISH_REPLACEMENTS['#101010']='#101010'
_BLACKISH_REPLACEMENTS['#3B3B3B']='#303030'
_BLACKISH_REPLACEMENTS['#2A2A2A']='#1F1F1F'
_BLACKISH_REPLACEMENTS['#656565']='#575757'
_BLACKISH_REPLACEMENTS['#767676']='#5E5E5E'
_BLACKISH_REPLACEMENTS['#868686']='#787878'
_BLACKISH_REPLACEMENTS['#636363']='#595959'
_BLACKISH_REPLACEMENTS['#696969']='#5E5E5E'
_BLACKISH_REPLACEMENTS['#707070']='#666666'
_BLACKISH_REPLACEMENTS['#767676']='#6B6B6B'
_BLACKISH_REPLACEMENTS['#C1C1C1']='#B8B8B8'
_BLACKISH_REPLACEMENTS['#C6C6C6']='#BDBDBD'
That seems like a fairly clean solution, otherwise I would have many variables and that is less than ideal. Now, I iterate over these with the syntax found in other SO posts:
_blackish_replace() (
shopt -s globstar
echo "${!_BLACKISH_REPLACEMENTS[#]}"
echo "${_BLACKISH_REPLACEMENTS[#]}"
for file in "$1"/**/*.scss; do
echo "Replacing colors in file: $file"
for color in "${!_BLACKISH_REPLACEMENTS[#]}"; do
echo "$color"
sed -i "s;$color;${_BLACKISH_REPLACEMENTS["$color"]};gI" "$file"
done
done
)
It looks good to me, and when this is run in a standalone script, it does indeed replace the correct matches in the correct files.
However, when using it from makepkg, it fails silently, hence the four echo calls exhibited.
The first two output newlines. This leads me to believe they are undefined?
The iteration has proved to be working for the glob expansion, however echo "$color" is never reached; the loop iterates nothing.
I thought maybe makepkg was using the system shell, which in that case, running the code directly from my user shell zsh fails with event not found: _BLACKISH_REPLACEMENTS or something alike (off the top of my head).
I asked in the Arch Linux Discord server if makepkg uses the locally available bash, and was assured it does. I am very confused.
It is probably a good idea to turn your array into a sed script before iterating the files:
#!/usr/bin/env bash
declare -A _BLACKISH_REPLACEMENTS=(
['#242424']='#1C1C1C'
['#333333']='#292929'
['#999999']='#787878'
['#555555']='#4C4C4C'
['#373737']='#2E2E2E'
['#434343']='#383838'
['#3E3E3E']='#333333'
['#383838']='#2E2E2E'
['#313131']='#262626'
['#101010']='#101010'
['#3B3B3B']='#303030'
['#2A2A2A']='#1F1F1F'
['#656565']='#575757'
['#767676']='#5E5E5E'
['#868686']='#787878'
['#636363']='#595959'
['#696969']='#5E5E5E'
['#707070']='#666666'
['#767676']='#6B6B6B'
['#C1C1C1']='#B8B8B8'
['#C6C6C6']='#BDBDBD'
)
sed_script=
for k in "${!_BLACKISH_REPLACEMENTS[#]}"; do
v="${_BLACKISH_REPLACEMENTS[$k]}"
sed_script+="s/$k/$v/g;"
done
shopt -s globstar nullglob
for file in "$1"/**/*.scss; do
sed -i.bak -e "$sed_script" "$file"
done
Now in a more practical one-liner POSIX-shell friendly call:
find ./ -type f -name '*.scss' -exec sed -i.bak -e 's/#242424/#1C1C1C/g;s/#696969/#5E5E5E/g;s/#555555/#4C4C4C/g;s/#767676/#6B6B6B/g;s/#868686/#787878/g;s/#383838/#2E2E2E/g;s/#636363/#595959/g;s/#101010/#101010/g;s/#373737/#2E2E2E/g;s/#C6C6C6/#BDBDBD/g;s/#313131/#262626/g;s/#333333/#292929/g;s/#C1C1C1/#B8B8B8/g;s/#707070/#666666/g;s/#434343/#383838/g;s/#3E3E3E/#333333/g;s/#3B3B3B/#303030/g;s/#999999/#787878/g;s/#656565/#575757/g;s/#2A2A2A/#1F1F1F/g;' {} \;
To clarify the point of all the above:
As you are unsure about the shell brand running your makepkg, it is a safe route to choose the most portable shell code by sticking to POSIX-shell grammar, common tools and options.
Instead of choosing a quite over-engineered associative array here. The replacement instructions for sed can be layed-out as clearly as your associative array:
#!/usr/bin/env sh
# A plain string of sed replacement instructions
# is as compact and more portable than an associative array.
# It also saves from looping over each entry.
_BLACKISH_REPLACEMENTS='
s/#242424/#1C1C1C/g;
s/#696969/#5E5E5E/g;
s/#555555/#4C4C4C/g;
s/#767676/#6B6B6B/g;
s/#868686/#787878/g;
s/#383838/#2E2E2E/g;
s/#636363/#595959/g;
s/#101010/#101010/g;
s/#373737/#2E2E2E/g;
s/#C6C6C6/#BDBDBD/g;
s/#313131/#262626/g;
s/#333333/#292929/g;
s/#C1C1C1/#B8B8B8/g;
s/#707070/#666666/g;
s/#434343/#383838/g;
s/#3E3E3E/#333333/g;
s/#3B3B3B/#303030/g;
s/#999999/#787878/g;
s/#656565/#575757/g;
s/#2A2A2A/#1F1F1F/g;
'
_blackish_replace() {
# Instead of iterating a bash specific globstar,
# find -exec can replace it while sticking to the most
# genuine POSIX-shell grammar.
# find and sed tools are used with their most common options
# avoiding gnu-specific extensions.
find "$1" -type f -name '*.scss' -exec \
sed -i.bak -e "$_BLACKISH_REPLACEMENTS" {} \;
}
Related
When I declare an array in bash $ ARRAY=('ele1' ele2') I can append an element to it with $ ARRAY+=('ele3').
echo ${ARRAY[#]}
ele1 ele2 ele3
However, inside a script in a while loop I don't get it to work:
FOUNDFILES=$(ls -lA)
LINE_CNT=1
ARRAY=()
echo -e "$FOUNDFILES" | while read line
do
ARRAY+=("test")
LINE_CNT=$((LINE_CNT+1))
done
echo "${ARRAY[#]}"
echo $LINE_CNT
LINE_CNT variable delivers the amount of files that were found, but my array stays empty.
What am I doing wrong?
A few things:
Don't assume that find outputs exactly one file name per line; that breaks in the presence of filenames containing newlines.
Don't assume that newlines output by find are the only whitespace in the output.
Don't use find at all when a glob will do.
shopt -s globstar
foundfiles=(./**/"$1")
declare -a array
line_cnt=1
for f in "${foundfiles[#]}"; do
array+=(test)
line_count=$((line_count + 1))
done
If your call to find is more complex than a glob can handle, and your version of find can output null-delimited file names, use
# -d for readarray was introduced in bash 4.4; earlier versions
# require something more complex; see Gordan Davidson's answer at
# https://stackoverflow.com/a/1120952/1126841
readarray -t -d $'\0' < <(find . ... -name "$1" -print0)
If your find does not support outputting null-delimited file names, rethink writing this in bash. (You might consider using zsh, which has a vastly richer set of glob features that can eliminate many cases where you would otherwise need find.)
I'm trying to rename multiple files that match a pattern in one directory.
Files:
stack_overflow_one.xml
stack_overflow_two.xml
stack_overflow_one.html
I would like to rename stack_overflow to heap_graph
heap_graph_one.xml
heap_graph_two.xml
heap_graph_one.html
I have tried the following:
Using rename:
rename stack_overflow heap_graph stack_overflow* # returns 'The syntax of the command is incorrect.'
Using for loop in Bash
# how can I write this in one line? I've tried wrapping in one line, but also does not work
for i in stack_overflow* do
mv "$i" "${i/stack_overflow/heap_graph}"
done
However, none of these are working.
What you have is a trivial syntax error in the for loop. The rest of your script should work fine without any problem.
for i in stack_overflow*; do
# ^^^ missing semi-colon
# The below condition to handle graceful loop termination when no files are found
[ -f "$i" ] || continue
mv "$i" "${i/stack_overflow/heap_graph}"
done
As noted by ghoti below if you are in the bourne again shell bash and not the POSIX bourne shell (sh) for which the solution above is portable, you can use special globbing options to avoid the condition of having to deal with case when no files are returned by the glob.
shopt -s nullglob
for i in stack_overflow*; do
mv "$i" "${i/stack_overflow/heap_graph}"
done
shopt -u nullglob
The -s option sets it and -u unsets it. More on shopt built-in from the GNU bash page
The semi-common utility mmv
(article)
is useful for the "multi-move" case you've described.
$ mmv -n 'stack_overflow_*' 'heap_graph__#' # remove -n after testing
mv -- stack_overflow_one.html heap_graph_one.html
mv -- stack_overflow_one.xml heap_graph_one.xml
mv -- stack_overflow_two.xml heap_graph_two.xml
As you can see, it's just calling down to mv multiple times for the
pattern(s) matched.
The * is a wildcard like Bash uses, but it's quote-escaped to be
passed to mmv. The _# is the reference to the match, also escaped
(though the docs suggest #1 would work instead).
This family of commands is also handy for copying (cp) and linking
(ln).
If you happen to have Zsh (a common Bash upgrade/replacement), you've
already got zmv, which would similarly do the job with:
% zmv -n 'stack_overflow_(*)' 'heap_graph_$1'
From time to time I have to append some text at the end of a bunch of files. I would normally find these files with find.
I've tried
find . -type f -name "test" -exec tail -n 2 /source.txt >> {} \;
This however results in writing the last two lines from /source.txt to a file named {} however many times a file was found matching the search criteria.
I guess I have to escape >> somehow but so far I wasn't successful.
Any help would be greatly appreciated.
-exec only takes one command (with optional arguments) and you can't use any bash operators in it.
So you need to wrap it in a bash -c '...' block, which executes everything between '...' in a new bash shell.
find . -type f -name "test" -exec bash -c 'tail -n 2 /source.txt >> "$1"' bash {} \;
Note: Everything after '...' is passed as regular arguments, except they start at $0 instead of $1. So the bash after ' is used as a placeholder to match how you would expect arguments and error processing to work in a regular shell, i.e. $1 is the first argument and errors generally start with bash or something meaningful
If execution time is an issue, consider doing something like export variable="$(tail -n 2 /source.txt)" and using "$variable" in the -exec. This will also always write the same thing, unlike using tail in -exec, which could change if the file changes. Alternatively, you can use something like -exec ... + and pair it with tee to write to many files at once.
A more efficient alternative (assuming bash 4):
shopt -s globstar
to_augment=( **/test )
tail -n 2 /source.txt | tee -a "${to_augment[#]}" > /dev/null
First, you create an array with all the file names, using a simple pattern that should be equivalent to your call to find. Then, use tee to append the desired lines to all those files at once.
If you have more criteria for the find command, you can still use it; this version is not foolproof, as it assumes no filename contains a newline, but fixing that is best left to another question.
while read -r fname; do
to_augment+=( "$fname" )
done < <(find ...)
I am trying to loop through files of a list of specified extensions with a bash script. I tried the solution given at Matching files with various extensions using for loop but it does not work as expected. The solution given was:
for file in "${arg}"/*.{txt,h,py}; do
Here is my version of it:
for f in "${arg}"/*.{epub,mobi,chm,rtf,lit,djvu}
do
echo "$f"
done
When I run this in a directory with an epub file in it, I get:
/*.epub
/*.mobi
/*.chm
/*.rtf
/*.lit
/*.djvu
So I tried changing the for statement:
for f in "${arg}"*.{epub,mobi,chm,rtf,lit,djvu}
Then I got:
089281098X.epub
*.mobi
*.chm
*.rtf
*.lit
*.djvu
I also get the same result with:
for f in *.{epub,mobi,chm,rtf,lit,djvu}
So it seems that the "${arg}" argument is unnecessary.
Although either of these statements finds files of the specified extensions and can pass them to a program, I get read errors from the unresolved *. filenames.
I am running this on OS X Mountain Lion. I was aware that the default bash shell was outdated so I upgraded it from 3.2.48 to 4.2.45 using homebrew to see if this was the problem. That didn't help so I am wondering why I am getting these unexpected results. Is the given solution wrong or is the OS X bash shell somehow different from the *NIX version? Is there perhaps an alternate way to accomplish the same thing that might work better in the OS X bash shell?
This may be a BASH 4.2ism. It does not work in my BASH which is still 3.2. However, if you shopt -s extglob, you can use *(...) instead:
shopt -s extglob
for file in *.*(epub|mobi|chm|rtf|lit|djvu)
do
...
done
#David W.: shopt -s extglob for f in .(epub|mobi|chm|rtf|lit|djvu) results in: 089281098X.epub #kojiro: arg=. shopt -s nullglob for f in "${arg}"/.{epub,mobi,chm,rtf,lit,djvu} results in: ./089281098X.epub shopt -s nullglob for f in "${arg}".{epub,mobi,chm,rtf,lit,djvu} results in: 089281098X.epub So all of these variations work but I don't understand why. Can either of you explain what is going on with each variation and what ${arg} is doing? I would really like to understand this so I can increase my knowledge. Thanks for the help.
In mine:
for f in *.*(epub|mobi|chm|rtf|lit|djvu)
I didn't include ${arg} which expands to the value of $arg. The *(...) matches the pattern found in the parentheses which is one of any of the series of extensions. Thus, it matches *.epub.
Kojiro's:
arg=.
shopt -s nullglob
for f in "${arg}"/*.{epub,mobi,chm,rtf,lit,djvu}
Is including $arg and the slash in his matching. Thus, koriro's start with ./ because that's what they are asking for.
It's like the difference between:
echo *
and
echo ./*
By the way, you could do this with the other expressions too:
echo *.*(epub|mobi|chm|rtf|lit|djvu)
The shell is doing all of the expansion for you. It's really has nothing to do with the for statement itself.
A glob has to expand to an existing, found name, or it is left alone with the asterisk intact. If you have an empty directory, *.foo will expand to *.foo. (Unless you use the nullglob Bash extension.)
The problem with your code is that you start with an arg, $arg, which is apparently empty or undefined. So your glob, ${arg}/*.epub expands to /*.epub because there are no files ending in ".epub" in the root directory. It's never looking in the current directory. For it to do that, you'd need to set arg=. first.
In your second example, the ${arg}*.epub does expand because $arg is empty, but the other files don't exist, so they continue not to expand as globs. As I hinted at before, one easy workaround would be to activate nullglob with shopt -s nullglob. This is bash-specific, but will cause *.foo to expand to an empty string if there is no matching file. For a strict POSIX solution, you would have to filter out unexpanded globs using [ -f "$f" ]. (Then again, if you wanted POSIX, you couldn't use brace expansion either.)
To summarize, the best solutions are to use (most intuitive and elegant):
shopt -s extglob
for f in *.*(epub|mobi|chm|rtf|lit|djvu)
or, in keeping with the original solution given in the referenced thread (which was wrong as stated):
shopt -s nullglob
for f in "${arg}"*.{epub,mobi,chm,rtf,lit,djvu}
This should do it:
for file in $(find ./ -name '*.epub' -o -name '*.mobi' -o -name '*.chm' -o -name '*.rtf' -o -name '*.lit' -o -name '*.djvu'); do
echo $file
done
I'm new to bash scripts (and the *nix shell altogether) but I'm trying to write this script to make grepping a codebase easier.
I have written this
#!/bin/bash
args=("$#");
for arg in args
grep arg * */* */*/* */*/*/* */*/*/*/*;
done
when I try to run it, this is what happens:
~/Work/richmond $ ./f.sh "\$_REQUEST\['a'\]"
./f.sh: line 4: syntax error near unexpected token `grep'
./f.sh: line 4: ` grep arg * */* */*/* */*/*/* */*/*/*/*;'
~/Work/richmond $
How do I do this properly?
And, I think a more important question is, how can I make grep recurse through subdirectories properly like this?
Any other tips and/or pitfalls with shell scripting and using bash in general would also be appreciated.
The syntax error is because you're missing do. As for searching recursively if your grep has the -R option you would do:
#!/bin/bash
for arg in "$#"; do
grep -R "$arg" *
done
Otherwise you could use find:
#!/bin/bash
for arg in "$#"; do
find . -exec grep "$arg" {} +
done
In the latter example, find will execute grep and replace the {} braces with the file names it finds, starting in the current directory ..
(Notice that I also changed arg to "$arg". You need the dollar sign to get the variable's value, and the quotes tell the shell to treat its value as one big word, even if $arg contains spaces or newlines.)
On recusive grepping:
Depending on your grep version, you can pass -R to your grep command to have it search Recursively (in subdirectories).
The best solution is stated above, but try putting your statement in back ticks:
`grep ...`
You should use 'find' plus 'xargs' to do the file searching.
for arg in "$#"
do
find . -type f -print0 | xargs -0 grep "$arg" /dev/null
done
The '-print0' and '-0' options assume you're using GNU grep and ensure that the script works even if there are spaces or other unexpected characters in your path names. Using xargs like this is more efficient than having find execute it for each file; the /dev/null appears in the argument list so grep always reports the name of the file containing the match.
You might decide to simplify life - perhaps - by combining all the searches into one using either egrep or grep -E. An optimization would be to capture the output from find once and then feed that to xargs on each iteration.
Have a look at the findrepo script which may give you some pointers
If you just want a better grep and don't want to do anything yourself, use ack, which you can get at http://betterthangrep.com/.