Sed Issue - Bad flag in substitute command: '#' - bash

I am running a script in mac that was inherited by my team and it has some replacement code which doesnt seem to be running properly. I tried a variety of subsitute characters like %,\ etc but nothing seems to be working. Can someone give me an idea of what else should I change.
The script is run like this
./test.sh /Users/Apple/Test_Folder/ 425323 C6C7CB 425363 425363 425363 666666 666666 425363 425363 666666 425363 425363 a0a9b1 687582 a0a9b2
The issue inside the script(test.sh) is this for loop
for f in "$1"/*.svg;
do
sed -i '' 's\#000002\#'"$2"'/g;s\#000003\#'"$3"'/g;s\#000004\#'"$4"'/g;s\#000005\#'"$5"'/g;s\#000006\#'"$6"'/g;s\#000007\#'"$7"'/g;s\#000008\#'"$8"'/g;s\#000009\#'"$9"'/g;s\#000010\#'"$10"'/g;s\#000011\#'"$11"'/g;s\#000012\#'"$12"'/g;s\#000013\#'"$13"'/g;s\#000014\#'"$14"'/g;s\#000015\#'"$15"'/g;s\#000016\#'"$16"'/g' $f
done
The error i get is
sed: 1: "s%#000002%#425323/g;s%# ...": bad flag in substitute command: '#'
Any pointers?
Thanks,
Ram

Reformat so it's easier to read, find, and edit.
Use double-quotes as script delimiters so you don't need so many workarounds.
for f in "$1"/*.svg;
do
sed -i "
s/#000002/#$2/g;
s/#000003/#$3/g;
s/#000004/#$4/g;
s/#000005/#$5/g;
s/#000006/#$6/g;
s/#000007/#$7/g;
s/#000008/#$8/g;
s/#000009/#$9/g;
s/#000010/#$10/g;
s/#000011/#$11/g;
s/#000012/#$12/g;
s/#000013/#$13/g;
s/#000014/#$14/g;
s/#000015/#$15/g;
s/#000016/#$16/g;
" $f
done
If you're comfortable with heavy abstraction you can try something like this, but make sure you test it heavily...
for s in $(seq 2 16)
do n="$(printf "%06d" $s)"
script="$script s/#$n/#\${$s}/g; "
done
for f in "$1"/*.svg;
do eval " sed -i \" $script \" \"$f\" "
done

Related

Unexplained line break

I'm very sorry if someone already have ask this question or something near it, but I did not found anything close to it, but may be it's because of my weakness in coding.
I have bash this script:
...
...
## Read n line in vmlist
VMTORESTORE=$(sed -n "$yn p" ./vmlist)
## Replace " " between names by "\ "
VMTORESTORE="$(echo "$VMTORESTORE" | sed 's/ /\\ /g' )"
## Test variable
echo $VMTORESTORE
## Run Command
ls -1 /vmfs/volumes/BACKUP_SAN/$VMTORESTORE | sed 's/ /\\ /g'
This is the result:
Save\ Camera
ls: /vmfs/volumes/BACKUP_SAN/Save: No such file or
directory
Camera:
I don't understand why the echo gave the good result whereas the command insert a line break between Save\ and Camera.
thanks by advance for your help.
Proc
it is a good idea to quote the variables as in:
ls -1 "/vmfs/volumes/BACKUP_SAN/$VMTORESTORE"
Take this as a good practice and update the script wherever needed.
This is happening because \ is an escape character in bash. So shouldnt your path consists of / rather than \ to define the location save/camera. Also globbing happens when you are using commands without " " so use the address in " ".
Thanks you all, I did not expect a solution that easy.
You were right, the double quotes were needed.
But in my cas this was not enough.
The path to the searched folder was:
"/vmfs/volume/BACKUP_SAN/Save Camera/"
but usually in Linux to reach this kind of folder I need to put \ before spaces like that:
"/vmfs/volume/BACKUP_SAN/Save\ Camera/"
That's why I replaced in my script all " " with "\ "
But is seems in a bash script this is not needed so this works like a charm:
...
...
## Read n line in vmlist
VMTORESTORE=$(sed -n "$yn p" ./vmlist)
## Run Command
ls -1 "/vmfs/volumes/BACKUP_SAN/$VMTORESTORE"
I searched the complication whereas it was this simple.
Thank you all for your help.

Using regular expressions to get parts of file name

On Mac (OS X) I have a directory with many images named like this:
IMG_250x333_1.jpg
IMG_250x333_2.jpg
IMG_250x333_3.jpg
...
I need to rename all of them to:
IMG_1.jpg
IMG_2.jpg
IMG_3.jpg
...
I think using a UNIX command line with "mv" and a kind of regex would do the job, but I don't know how! Can someone please help?
Thanks!
What happens if there's a IMG_111x333_1.jpg and also a IMG_444x222_1.jpg? You risk mangling/overwriting something...
But if that is what you want, you can do it like this:
#!/bin/bash
for f in *.jpg; do
new=${f/_*_/_}
echo mv "$f" $new
done
If you like what it is doing, remove the word echo.
Here's an approach I like:
ls | sed 's/\(.*\)250x333_\(.*\)/mv "&" "\1\2"/' | sh
List the files with ls.
Then, transform the filenames with sed, and generate a mv command. Note that the & in the sed command outputs the full input string.
Finally, evaluate the mv command with sh
The nice thing about this approach is you can remove | sh and test that your regex is correct.

grep a string with some variables in bash

I have been struggling like for two hours to figure out the issue regarding this script of mine. When I used it statically without any variable it fetches the grep results, but when I put them with those variables, i keep receiving error and no results. I believe there is something wrong with the special character escape which I can not handle.
I have the file FLAGS_IN with this structure :
automotive_susan_s dataset1 -funsafe-math-optimizations -fno-guess-branch-probability -fno-ivopts -fno-inline-functions -fno-omit-frame-pointer -fselective-scheduling -fno-inline-small-functions -fno-tree-pre -ftracer -fno-move-loop-invariants
that have the flags for i in AppName and the dataset$j as structured above. Could anyone help me figure out what is wrong with this part of my sh script?
GCC_OPT="-O3"
OPT_FLAGS=$("grep $i\ dataset$j\ $FLAGS_IN|sed\ s/$i\ dataset$j//g")
echo $GCC_OPT
echo $OPT_FLAGS
echo "found the validated flags, they are \n $GCC_OPT $OPT_FLAGS"
make -f Makefile.gcc -j4 CCC_OPTS="$GCC_OPT\ $OPT_FLAG"
You're a little overzealous with your quoting. Also, it's a little easier to use cut here than sed.
OPT_FLAGS=$(grep "$i dataset$j" FLAGS_IN | cut -d " " -f3-)
and
make -f Makefile.gcc -j4 CCC_OPTS="$GCC_OPT $OPT_FLAG"
Is this what you're trying to do:
$ cat file
foo
automotive_susan_s dataset1 -funsafe-math-optimizations ...
bar
$ i=automotive_susan_s
$ j=1
$ sed -n "s/$i dataset$j//p" file
-funsafe-math-optimizations ...

xargs input involving spaces

I am working on a Mac using OSX and I'm using bash as my shell. I have a script that goes something to the effect of:
VAR1="pass me into parallel please!"
VAR2="oh me too, and there's actually a lot of us, but its best we stay here too"
printf "%s\n" {0..249} | xargs -0 -P 8 -n 1 . ./parallel.sh
I get the error: xargs: .: Permission denied. The purpose is to run a another script in parallel (called parallel.sh) which get's fed the numbers 0-249. Additionally I want to make sure that parallel can see and us VAR1 and VAR2. But when I try to source the script parallel with . ./parallel, xargs doesn't like that. The point of sourcing is because the script has other variables I wish parallel to have access to.
I have read something about using print0 since xargs separates it's inputs by spaces, but I really didn't understand what -print0 does and how to use it. Thanks for any help you guys can offer.
If you want the several processes running the script, then they can't be part of the parent process and therefore they can't access the exact same variables. However, if you export your variables, then each process can get a copy of them:
export VAR1="pass me into parallel please!"
export VAR2="oh me too, and there's actually a lot of us, but its best we stay here too"
printf "%s\n" {0..249} | xargs -P 8 -n 1 ./parallel.sh
Now you can just drop the extra dot since you aren't sourcing the parallel.sh script, you are just running it.
Also there is no need to use -0 since your input is just a series of numbers, one on each line.
To avoid the space problem I'd use new line character as separator for xargs with the -d option:
xargs -d '\n' ...
i think you have permission issues , try getting a execute permission for that file "parallel.sh"
command works fine for me :
Kaizen ~/so_test $ printf "%s\n" {0..4} | xargs -0 -P 8 -n 1 echo
0
1
2
3
4
man find :
-print0
True; print the full file name on the standard output, followed by a
null character (instead of the newline character that -print uses).
This allows file names that contain newlines or other types of white space to be correctly interpreted by programs that process the find
output. This option corresponds to the -0 option of xargs.
for print0 use : check the link out : there is a question for it in stack overflow
Capturing output of find . -print0 into a bash array
The issue of passing arguments is related to xarg's interpretation of white space. From the xargs man page:
-0 Change xargs to expect NUL (``\0'') characters as separators, instead of spaces and newlines.
The issue of environment variables can be solved by using export to make the variables available to subprocesses:
say.sh
echo "$1 $V"
result
bash$ export V=whatevs
bash$ printf "%s\n" {0..3} | xargs -P 8 -n 1 ./say.sh
1 whatevs
2 whatevs
0 whatevs
3 whatevs

Using sed to mass rename files

Objective
Change these filenames:
F00001-0708-RG-biasliuyda
F00001-0708-CS-akgdlaul
F00001-0708-VF-hioulgigl
to these filenames:
F0001-0708-RG-biasliuyda
F0001-0708-CS-akgdlaul
F0001-0708-VF-hioulgigl
Shell Code
To test:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/'
To perform:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/' | sh
My Question
I don't understand the sed code. I understand what the substitution
command
$ sed 's/something/mv'
means. And I understand regular expressions somewhat. But I don't
understand what's happening here:
\(.\).\(.*\)
or here:
& \1\2/
The former, to me, just looks like it means: "a single character,
followed by a single character, followed by any length sequence of a
single character"--but surely there's more to it than that. As far as
the latter part:
& \1\2/
I have no idea.
First, I should say that the easiest way to do this is to use the
prename or rename commands.
On Ubuntu, OSX (Homebrew package rename, MacPorts package p5-file-rename), or other systems with perl rename (prename):
rename s/0000/000/ F0000*
or on systems with rename from util-linux-ng, such as RHEL:
rename 0000 000 F0000*
That's a lot more understandable than the equivalent sed command.
But as for understanding the sed command, the sed manpage is helpful. If
you run man sed and search for & (using the / command to search),
you'll find it's a special character in s/foo/bar/ replacements.
s/regexp/replacement/
Attempt to match regexp against the pattern space. If success‐
ful, replace that portion matched with replacement. The
replacement may contain the special character & to refer to that
portion of the pattern space which matched, and the special
escapes \1 through \9 to refer to the corresponding matching
sub-expressions in the regexp.
Therefore, \(.\) matches the first character, which can be referenced by \1.
Then . matches the next character, which is always 0.
Then \(.*\) matches the rest of the filename, which can be referenced by \2.
The replacement string puts it all together using & (the original
filename) and \1\2 which is every part of the filename except the 2nd
character, which was a 0.
This is a pretty cryptic way to do this, IMHO. If for
some reason the rename command was not available and you wanted to use
sed to do the rename (or perhaps you were doing something too complex
for rename?), being more explicit in your regex would make it much
more readable. Perhaps something like:
ls F00001-0708-*|sed 's/F0000\(.*\)/mv & F000\1/' | sh
Being able to see what's actually changing in the
s/search/replacement/ makes it much more readable. Also it won't keep
sucking characters out of your filename if you accidentally run it
twice or something.
you've had your sed explanation, now you can use just the shell, no need external commands
for file in F0000*
do
echo mv "$file" "${file/#F0000/F000}"
# ${file/#F0000/F000} means replace the pattern that starts at beginning of string
done
I wrote a small post with examples on batch renaming using sed couple of years ago:
http://www.guyrutenberg.com/2009/01/12/batch-renaming-using-sed/
For example:
for i in *; do
mv "$i" "`echo $i | sed "s/regex/replace_text/"`";
done
If the regex contains groups (e.g. \(subregex\) then you can use them in the replacement text as \1\,\2 etc.
The easiest way would be:
for i in F00001*; do mv "$i" "${i/F00001/F0001}"; done
or, portably,
for i in F00001*; do mv "$i" "F0001${i#F00001}"; done
This replaces the F00001 prefix in the filenames with F0001.
credits to mahesh here: http://www.debian-administration.org/articles/150
The sed command
s/\(.\).\(.*\)/mv & \1\2/
means to replace:
\(.\).\(.*\)
with:
mv & \1\2
just like a regular sed command. However, the parentheses, & and \n markers change it a little.
The search string matches (and remembers as pattern 1) the single character at the start, followed by a single character, follwed by the rest of the string (remembered as pattern 2).
In the replacement string, you can refer to these matched patterns to use them as part of the replacement. You can also refer to the whole matched portion as &.
So what that sed command is doing is creating a mv command based on the original file (for the source) and character 1 and 3 onwards, effectively removing character 2 (for the destination). It will give you a series of lines along the following format:
mv F00001-0708-RG-biasliuyda F0001-0708-RG-biasliuyda
mv abcdef acdef
and so on.
Using perl rename (a must have in the toolbox):
rename -n 's/0000/000/' F0000*
Remove -n switch when the output looks good to rename for real.
There are other tools with the same name which may or may not be able to do this, so be careful.
The rename command that is part of the util-linux package, won't.
If you run the following command (GNU)
$ rename
and you see perlexpr, then this seems to be the right tool.
If not, to make it the default (usually already the case) on Debian and derivative like Ubuntu :
$ sudo apt install rename
$ sudo update-alternatives --set rename /usr/bin/file-rename
For archlinux:
pacman -S perl-rename
For RedHat-family distros:
yum install prename
The 'prename' package is in the EPEL repository.
For Gentoo:
emerge dev-perl/rename
For *BSD:
pkg install gprename
or p5-File-Rename
For Mac users:
brew install rename
If you don't have this command with another distro, search your package manager to install it or do it manually:
cpan -i File::Rename
Old standalone version can be found here
man rename
This tool was originally written by Larry Wall, the Perl's dad.
The backslash-paren stuff means, "while matching the pattern, hold on to the stuff that matches in here." Later, on the replacement text side, you can get those remembered fragments back with "\1" (first parenthesized block), "\2" (second block), and so on.
If all you're really doing is removing the second character, regardless of what it is, you can do this:
s/.//2
but your command is building a mv command and piping it to the shell for execution.
This is no more readable than your version:
find -type f | sed -n 'h;s/.//4;x;s/^/mv /;G;s/\n/ /g;p' | sh
The fourth character is removed because find is prepending each filename with "./".
Here's what I would do:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done
Then if that looks ok, add | sh to the end. So:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done | sh
for i in *; do mv $i $(echo $i|sed 's/AAA/BBB/'); done
The parentheses capture particular strings for use by the backslashed numbers.
ls F00001-0708-*|sed 's|^F0000\(.*\)|mv & F000\1|' | bash
Some examples that work for me:
$ tree -L 1 -F .
.
├── A.Show.2020.1400MB.txt
└── Some Show S01E01 the Loreming.txt
0 directories, 2 files
## remove "1400MB" (I: ignore case) ...
$ for f in *; do mv 2>/dev/null -v "$f" "`echo $f | sed -r 's/.[0-9]{1,}mb//I'`"; done;
renamed 'A.Show.2020.1400MB.txt' -> 'A.Show.2020.txt'
## change "S01E01 the" to "S01E01 The"
## \U& : change (here: regex-selected) text to uppercase;
## note also: no need here for `\1` in that regex expression
$ for f in *; do mv 2>/dev/null "$f" "`echo $f | sed -r "s/([0-9] [a-z])/\U&/"`"; done
$ tree -L 1 -F .
.
├── A.Show.2020.txt
└── Some Show S01E01 The Loreming.txt
0 directories, 2 files
$
2>/dev/null suppresses extraneous output (warnings ...)
reference [this thread]: https://stackoverflow.com/a/2372808/1904943
change case: https://www.networkworld.com/article/3529409/converting-between-uppercase-and-lowercase-on-the-linux-command-line.html

Resources