make: referencing shell variables in sed call - shell

I am using make and I'd like to reference a shell variable within sed. However, within the sed-call $$LASTMOD only expands to $LASTMOD but I'd like to see sth like
-rw-r--r-- 1 weiss Administ 5752 Apr 1 23:44 src/stata/ini-00.do
It works for echo but not for sed. .ONESHELL doesn't seem to work either.
.SUFFIXES : .do .tmp
DOFILES = $(shell ls src/stata/*.do | sed 's/.do/.tmp/')
all: $(DOFILES)
.do.tmp:
LASTMOD=`ls -l $*`; echo $$LASTMOD; sed -e \
"s/Last modified:.*/Last modified: $$LASTMOD/g" $*.do > $*.tmp
UPDATE:
After applying the changes as suggested by #geekosaur, #William Pursell and #eriktous it works like a charm:
.SUFFIXES : .do .tmp
DOFILES = $(shell ls src/stata/*.do | sed 's/\.do/\.tmp/')
all: $(DOFILES)
.do.tmp:
LASTMOD=`ls -l $<`; \
sed -e "s, Last modified:.*, Last modified: $$LASTMOD, g" $*.do > $*.tmp

Are you actually using double quotes there? I would expect that behavior from single quotes; but with double quotes, you will get sed syntax errors unless you escape the slashes or use different pattern delimiters (I generally use , when I know there will be / involved, as with pathnames).
LASTMOD=$$(ls -l "$*"); \
sed "s,Last modified:.*,Last modified: $$LASTMOD,g" "$*.do" >"$*.tmp"

Did you mean ls -l $*, or did you mean to write ls -l $<? What you are doing looks correct except for that. (Make will echo the command without expanding $LASTMOD, but sed will do the replacement you desire.)

Related

how to delete a file with quote in file name

When I do ls in my directory, get bunch of these:
data.log".2015-01-22"
data.log".2015-01-23"
However when I do this:
rm: cannot remove `data.log.2015-01-22': No such file or directory
If I could somehow do something line ls | escape quotes | xargs rm
So yeah, how do I remove these files containing "?
Update
While most answer work. I was actually trying to do this:
ls | rm
So it was failing for some files. How can I escape a quote in a pipe after ls? Most of the answers actually addresses the manual manipulation of file which works. But I was asking about the escaping/replacing quotes after the ls. Sorry if my question was confusing.
If you only need to do this once in a while interactively, use
rm -i -- *
and answer y or n as appropriate. This can be used to get rid of many files having funny characters in their name.
It has the advantage of not needing to type/escape funny characters, blanks, etc, since the shell globbing with * does that for you. It is also as short as it gets, so easy to memorize.
Use single quotes to quote the double quotes, or backslash:
rm data.log'"'*
rm data.log\"*
Otherwise, double quotes are interpreted by the shell and removed from the string.
Answer to the updated question:
Don't process the output of ls. Filenames can contain spaces, newlines, etc.
You could do like this.
find . -type f -name '*"*' -exec rm {} +
If you have a particularly stubborn file that has all kinds of characters
For example
''$'\336\017\200\336\024\b''J'$'\336\017\220\336\024\b''V'$'\336\017\240\336\024\b''^'$'\336\017\260\336\024\b''f'$'\336\017\300\336\024\b''p'$'\336\017\320\336\024\b''y'$'\336\017\340\336\024\b\202\336\017\360\336\024\b\212\336\017\337\024\b\223\336\017\020\337\024\b\234\336\017'
If it were me, I'd forget dealing with interpolation of strings (unless you are really good at using special characters) and use the file index number, or inode.
For example, assume your culprit is in the current working directory, then the following procedure would work:
# visually inspect list of files with index numbers (inodes) for the culprit
ls -i .
# remove the culprit by inode
find . -maxdepth 1 -inum <inode> -exec rm {} \;
and be done with it.
Escape the quote with single quotes
$ touch '" and spaces also "'
$ ls
" and spaces also "
$ rm '" and spaces also "'
$ ls
$
In your case:
$ rm 'data.log".2015-01-22"' 'data.log".2015-01-23"'
1st - suggestion - modify the tool creating file names with quotes in them... :)
Try a little wild-char magic - using your tool of choice, i.e I would use tr:
ls | escape quotes | xargs rm ## becomes something like:
ls | tr "[\",']" '?' | xargs rm ## does not work on my system but this does:
rm -v $(ls *sing* *doub* | tr "[\",']" '?')
Output is:
removed `"""double"""'
removed `\'\'\'single\'\'\''
Now:
$ touch "'''single'''" '"""double"""'
$ ls -l *sing* *doub*
-rw-rw-r-- 1 dale dale 0 Feb 15 09:48 """double"""
-rw-rw-r-- 1 dale dale 0 Feb 15 09:48 '''single'''
If your patterns are consistent the other way might be to simplify:
$ rm -v *sing* *doub*
removed `\'\'\'single\'\'\''
removed `"""double"""'
For your example:
rm -v data.*${YEAR}-${MONTH}-${DAY}* ## data.log".2015-01-22" OR
rm -v data.*${YEAR}-${MONTH}-${DAY}? ## data.log".2015-01-22"
ls | rm
doesn't work because rm gets the files to remove from the command line arguments, not from standard input. You can use xargs to translate standard input to arguments.
ls | xargs -d '\n' rm
But to just delete the files you want, quote the part of the name containing the single quote:
rm 'data.log"'.*

Removing last n characters from Unix Filename before the extension

I have a bunch of files in Unix Directory :
test_XXXXX.txt
best_YYY.txt
nest_ZZZZZZZZZ.txt
I need to rename these files as
test.txt
best.txt
nest.txt
I am using Ksh on AIX .Please let me know how i can accomplish the above using a Single command .
Thanks,
In this case, it seems you have an _ to start every section you want to remove. If that's the case, then this ought to work:
for f in *.txt
do
g="${f%%_*}.txt"
echo mv "${f}" "${g}"
done
Remove the echo if the output seems correct, or replace the last line with done | ksh.
If the files aren't all .txt files, this is a little more general:
for f in *
do
ext="${f##*.}"
g="${f%%_*}.${ext}"
echo mv "${f}" "${g}"
done
If this is a one time (or not very often) occasion, I would create a script with
$ ls > rename.sh
$ vi rename.sh
:%s/\(.*\)/mv \1 \1/
(edit manually to remove all the XXXXX from the second file names)
:x
$ source rename.sh
If this need occurs frequently, I would need more insight into what XXXXX, YYY, and ZZZZZZZZZZZ are.
Addendum
Modify this to your liking:
ls | sed "{s/\(.*\)\(............\)\.txt$/mv \1\2.txt \1.txt/}" | sh
It transforms filenames by omitting 12 characters before .txt and passing the resulting mv command to a shell.
Beware: If there are non-matching filenames, it executes the filename—and not a mv command. I omitted a way to select only matching filenames.

Remove everything but one file tcsh

I basically want to do the following bash command but in tcsh:
rm !(file1)
Thanks
You can use ls -1 (that's the number one, not the lowercase letter L) to list one file per line, and then use grep -vx <pattern> to exclude (-v) lines that exactly (-x) match <pattern>, and then xargs it to your command, rm. For example,
ls -1 | grep -vx file1 | xargs rm
In case your version of grep doesn't support the -x option, you can use anchors:
ls -1 | grep -vx '^file1$' | xargs rm
To use this with commands other than rm that may not take an arbitrary number of arguments, remember to add the -n 1 option to xargs so that arguments are handled one by one:
ls -1 | grep -vx '^file1$' | xargs -n 1 rm
I believe you can also achieve this using find's -name option to specify a parameter by negation, i.e. the find utility itself may support expressions like !(file1), though you'll still have to pipe the results to xargs.
tcsh has a special ^ syntax for glob patterns (not supported in csh, sh, or bash). Prefixing a glob pattern with ^ negates it, causing to match all file names that don't match the pattern.
Quoting the tcsh manual:
An entire glob-pattern can also be negated with `^':
> echo *
bang crash crunch ouch
> echo ^cr*
bang ouch
A single file name is not a glob pattern, and so the ^ prefix doesn't apply to it, but it can be turned into one by, for example, surrounding the first character with square brackets.
So this:
rm ^[f]ile1
should remove all files in the current directory other than file1.
I strongly recommend testing this before using it, either by using an echo command first:
echo ^[f]ile1
or by using Ctrl-X * to expand the pattern to a list of files before hitting Enter.
UPDATE: I've since learned that bash supports similar functionality but with a different syntax. In bash, !(PATTERN) matches anything not matched by the pattern. This is not recognized unless the extglob shell option is enabled. Unlike tcsh's ^ syntax, the pattern can be a single file name. This isn't relevant to what you're asking, but it could be useful if you ever decide to switch to bash.
zsh probably has something similar.

Using ls, how to list files without printing the extension (the part after the dot)?

Suppose I have a directory with some files:
$ ls
a.c b.c e.c k.cpp s.java
How can I display the result without the file extension(the part following the dot, including that dot)? Like this:
$ <some command>
a
b
e
k
s
using sed?
ls -1 | sed -e 's/\..*$//'
ls | while read fname
do
echo ${fname%%.*}
done
Try that.
ls -a | cut -d "." -f 1
man (1) cut
Very handy, the -d switch defines the delimiter and the -f which field you want.
EDIT: Include riverfall's scenario is also piece of cake as cut can start also from the end, though the logic is somewhat different. Here an example with a bunch of files with random names, some with two dots, some with a single dot and some without extension:
runlevel0#ubuntu:~/test$ ls
test.001.rpx test.003.rpx test.005.rpx test.007.rpx test.009.rpx testxxx
test.002.rpx test.004.rpx test.006.rpx test.008.rpx test_nonum test_xxx.rtv
runlevel0#ubuntu:~/test$ ls | cut -d "." -f -2
test.001
test.002
test.003
test.004
test.005
test.006
test.007
test.008
test.009
test_nonum
testxxx
test_xxx.rtv
Using the minus before the field number makes it eliminate all BUT the indicated fields (1,2 in this case) and putting it behind makes it start counting from the end.
This same notation can be used for offset and characters besides of fields (see the man page)
If you already know the extension of the file, you can use basename, from the man page:
basename - strip directory and suffix from filenames
Unfortunately, it's mostly useful if you're trying to filter a single extension, in your case the command is:
basename -s .c -a $(ls *.c) && basename -s .cpp -a $(ls *.cpp) && basename -s .java -a $(ls *.java)
output:
a
b
e
k
s
for f in *; do printf "%s\n" ${f%%.*}; done
Why it works?
${string%%substring} Deletes longest match of $substring from back of $string.
This would handle mypackage.pkg.tar.xz --> mypackage for instance.
In contrast:
${string%substring} Deletes shortest match of $substring from back of $string.
That is ${string%substring} would only delete the final extension, i.e.
mypackage.pkg.tar.xz --> mypackage.pkg.tar
On a side note, use printf preferentially to echo. The syntax is a little more complex, but it will work on a wider variety of systems.
If you only want to see files, not directories:
for f in *; do if [[ -f ${f} ]]; then printf "%s\n" ${f%%.*}; fi; done

Using sed to mass rename files

Objective
Change these filenames:
F00001-0708-RG-biasliuyda
F00001-0708-CS-akgdlaul
F00001-0708-VF-hioulgigl
to these filenames:
F0001-0708-RG-biasliuyda
F0001-0708-CS-akgdlaul
F0001-0708-VF-hioulgigl
Shell Code
To test:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/'
To perform:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/' | sh
My Question
I don't understand the sed code. I understand what the substitution
command
$ sed 's/something/mv'
means. And I understand regular expressions somewhat. But I don't
understand what's happening here:
\(.\).\(.*\)
or here:
& \1\2/
The former, to me, just looks like it means: "a single character,
followed by a single character, followed by any length sequence of a
single character"--but surely there's more to it than that. As far as
the latter part:
& \1\2/
I have no idea.
First, I should say that the easiest way to do this is to use the
prename or rename commands.
On Ubuntu, OSX (Homebrew package rename, MacPorts package p5-file-rename), or other systems with perl rename (prename):
rename s/0000/000/ F0000*
or on systems with rename from util-linux-ng, such as RHEL:
rename 0000 000 F0000*
That's a lot more understandable than the equivalent sed command.
But as for understanding the sed command, the sed manpage is helpful. If
you run man sed and search for & (using the / command to search),
you'll find it's a special character in s/foo/bar/ replacements.
s/regexp/replacement/
Attempt to match regexp against the pattern space. If success‐
ful, replace that portion matched with replacement. The
replacement may contain the special character & to refer to that
portion of the pattern space which matched, and the special
escapes \1 through \9 to refer to the corresponding matching
sub-expressions in the regexp.
Therefore, \(.\) matches the first character, which can be referenced by \1.
Then . matches the next character, which is always 0.
Then \(.*\) matches the rest of the filename, which can be referenced by \2.
The replacement string puts it all together using & (the original
filename) and \1\2 which is every part of the filename except the 2nd
character, which was a 0.
This is a pretty cryptic way to do this, IMHO. If for
some reason the rename command was not available and you wanted to use
sed to do the rename (or perhaps you were doing something too complex
for rename?), being more explicit in your regex would make it much
more readable. Perhaps something like:
ls F00001-0708-*|sed 's/F0000\(.*\)/mv & F000\1/' | sh
Being able to see what's actually changing in the
s/search/replacement/ makes it much more readable. Also it won't keep
sucking characters out of your filename if you accidentally run it
twice or something.
you've had your sed explanation, now you can use just the shell, no need external commands
for file in F0000*
do
echo mv "$file" "${file/#F0000/F000}"
# ${file/#F0000/F000} means replace the pattern that starts at beginning of string
done
I wrote a small post with examples on batch renaming using sed couple of years ago:
http://www.guyrutenberg.com/2009/01/12/batch-renaming-using-sed/
For example:
for i in *; do
mv "$i" "`echo $i | sed "s/regex/replace_text/"`";
done
If the regex contains groups (e.g. \(subregex\) then you can use them in the replacement text as \1\,\2 etc.
The easiest way would be:
for i in F00001*; do mv "$i" "${i/F00001/F0001}"; done
or, portably,
for i in F00001*; do mv "$i" "F0001${i#F00001}"; done
This replaces the F00001 prefix in the filenames with F0001.
credits to mahesh here: http://www.debian-administration.org/articles/150
The sed command
s/\(.\).\(.*\)/mv & \1\2/
means to replace:
\(.\).\(.*\)
with:
mv & \1\2
just like a regular sed command. However, the parentheses, & and \n markers change it a little.
The search string matches (and remembers as pattern 1) the single character at the start, followed by a single character, follwed by the rest of the string (remembered as pattern 2).
In the replacement string, you can refer to these matched patterns to use them as part of the replacement. You can also refer to the whole matched portion as &.
So what that sed command is doing is creating a mv command based on the original file (for the source) and character 1 and 3 onwards, effectively removing character 2 (for the destination). It will give you a series of lines along the following format:
mv F00001-0708-RG-biasliuyda F0001-0708-RG-biasliuyda
mv abcdef acdef
and so on.
Using perl rename (a must have in the toolbox):
rename -n 's/0000/000/' F0000*
Remove -n switch when the output looks good to rename for real.
There are other tools with the same name which may or may not be able to do this, so be careful.
The rename command that is part of the util-linux package, won't.
If you run the following command (GNU)
$ rename
and you see perlexpr, then this seems to be the right tool.
If not, to make it the default (usually already the case) on Debian and derivative like Ubuntu :
$ sudo apt install rename
$ sudo update-alternatives --set rename /usr/bin/file-rename
For archlinux:
pacman -S perl-rename
For RedHat-family distros:
yum install prename
The 'prename' package is in the EPEL repository.
For Gentoo:
emerge dev-perl/rename
For *BSD:
pkg install gprename
or p5-File-Rename
For Mac users:
brew install rename
If you don't have this command with another distro, search your package manager to install it or do it manually:
cpan -i File::Rename
Old standalone version can be found here
man rename
This tool was originally written by Larry Wall, the Perl's dad.
The backslash-paren stuff means, "while matching the pattern, hold on to the stuff that matches in here." Later, on the replacement text side, you can get those remembered fragments back with "\1" (first parenthesized block), "\2" (second block), and so on.
If all you're really doing is removing the second character, regardless of what it is, you can do this:
s/.//2
but your command is building a mv command and piping it to the shell for execution.
This is no more readable than your version:
find -type f | sed -n 'h;s/.//4;x;s/^/mv /;G;s/\n/ /g;p' | sh
The fourth character is removed because find is prepending each filename with "./".
Here's what I would do:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done
Then if that looks ok, add | sh to the end. So:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done | sh
for i in *; do mv $i $(echo $i|sed 's/AAA/BBB/'); done
The parentheses capture particular strings for use by the backslashed numbers.
ls F00001-0708-*|sed 's|^F0000\(.*\)|mv & F000\1|' | bash
Some examples that work for me:
$ tree -L 1 -F .
.
├── A.Show.2020.1400MB.txt
└── Some Show S01E01 the Loreming.txt
0 directories, 2 files
## remove "1400MB" (I: ignore case) ...
$ for f in *; do mv 2>/dev/null -v "$f" "`echo $f | sed -r 's/.[0-9]{1,}mb//I'`"; done;
renamed 'A.Show.2020.1400MB.txt' -> 'A.Show.2020.txt'
## change "S01E01 the" to "S01E01 The"
## \U& : change (here: regex-selected) text to uppercase;
## note also: no need here for `\1` in that regex expression
$ for f in *; do mv 2>/dev/null "$f" "`echo $f | sed -r "s/([0-9] [a-z])/\U&/"`"; done
$ tree -L 1 -F .
.
├── A.Show.2020.txt
└── Some Show S01E01 The Loreming.txt
0 directories, 2 files
$
2>/dev/null suppresses extraneous output (warnings ...)
reference [this thread]: https://stackoverflow.com/a/2372808/1904943
change case: https://www.networkworld.com/article/3529409/converting-between-uppercase-and-lowercase-on-the-linux-command-line.html

Resources