How can I convert a Windows dir path (say c:/libs/Qt-static) to the correct POSIX dir path (/c/libs/Qt-static) by means of standard msys features? And vice versa?
Cygwin, Git Bash, and MSYS2 have a readymade utility called cygpath.exe just for doing that.
Output type options:
-d, --dos print DOS (short) form of NAMEs (C:\PROGRA~1\)
-m, --mixed like --windows, but with regular slashes (C:/WINNT)
-M, --mode report on mode of file (binmode or textmode)
-u, --unix (default) print Unix form of NAMEs (/cygdrive/c/winnt)
-w, --windows print Windows form of NAMEs (C:\WINNT)
-t, --type TYPE print TYPE form: 'dos', 'mixed', 'unix', or 'windows'
I don't know msys, but a quick google search showed me that it includes the sed utility. So, assuming it works similar in msys than it does on native Linux, here's one way how to do it:
From Windows to POSIX
You'll have to replace all backslashes with slashes, remove the first colon after the drive letter, and add a slash at the beginning:
echo "/$pth" | sed 's/\\/\//g' | sed 's/://'
or, as noted by xaizek,
echo "/$pth" | sed -e 's/\\/\//g' -e 's/://'
From POSIX to Windows
You'll have to add a semi-colon, remove the first slash and replace all slashes with backslashes:
echo "$pth" | sed 's/^\///' | sed 's/\//\\/g' | sed 's/^./\0:/'
or more efficiently,
echo "$pth" | sed -e 's/^\///' -e 's/\//\\/g' -e 's/^./\0:/'
where $pth is a variable storing the Windows or POSIX path, respectively.
Just use cygpath:
$ cygpath -w "/c/foo/bar"
-> C:\foo\bar
$ cygpath -u "C:\foo\bar"
-> /c/foo/bar
You may wonder: "Do I have cygpath installed?" Well,
If you're using the git-bash shell, then yes.
If you're in cygwin or MSYS2, then yes.
If you're in another shell, but you have installed git-bash before, then cygpath can be found at git-bash-install-folder\usr\bin\cygpath.exe.
Else: maybe not, but I'm sure you can find a way to installed it.
The "correct" way in MSYS is:
$ MSYS_NO_PATHCONV=1 taskkill /F /T /IM ssh-agent.exe
This avoids having to manually translate slashes. It simply de-activates the path conversion.
Here is my implementation (tested on git bash).
From POSIX to Windows
sed '
\,/$, !s,$,/,
\,^/, s,/,:/,2
s,^/,,
s,/,\\,g
' <<< "$#"
Works for:
/c/git
relative/dir
c:/git
~
.
..
/c
/c/
./relative/dir
/sd0/some/dir/
except
/
<path with space>
Explanation:
\,^/, s,/,:/,2 (converts /drive/dir/ to /drive:/dir/) is the heart of it and inserts : before the 2nd /. I use , for delim instead of / for readability. If starting with / (\,^/,), then replace / with :/ for the 2nd occurrence. I do not want to assume drive letter length of 1 so this works for /sd0/some/dir.
s,^/,, removes the leading / and s,/,\\,g converts all / to \.
\,/$, !s,$,/, is to handle the corner case of /c and ensure 2nd / (/c/) for the next command to work.
Note:
If here string <<< does not work in your shell then you can echo and pipe as
echo "$#" | sed ...
Errata
Here e script
just FYI - at least for my git version 2.26.2.windows.1
e.g. if I have a path like C:\dev\work_setup\msk, I can go directly to Git Bash and type
cd "C:\dev\work_setup\msk"
this will result in current folder being changed to /c/dev/work_setup/msk - so this type of conversion seems to be done automatically, as long as I put the Windows path inside double quotes. Unfortunately I don't have references to original documentation that would back that up.
My solution works with a list of folders/files and it's done in 2 steps.
Suppose you would like to replace a path from D:\example to /example for a list of file where this Windows path has been repetead.
The first step it changes the backlashes into slashes
grep -lr "D:\\\\example" /parent-folder | xargs -d'\n' sed -i 's+\\+\/+g'
Note that parent-folder could be root (/) or whatever you like and -d'\n' parameter is necessary if you have filenames or folder names with white spaces.
Second step it substitutes the D:/example into /example:
grep -lr "D:/example" /parent-folder | xargs -d'\n' sed -i 's+D:+/example+g'
I wanted to share this solution since it tooks me some time to make this 2 lines but it has been really helpfull job (I'm migrating a Windows App to a Linux Server with tons of Windows paths inside').
The answer of #hello_earth is misleading, due to Windows path must be double backslashed like:
cd "e:\\dir\\subdir\\path"
otherwise the shell will find escape-sequences.
Related
I'm using the following command on Ubuntu to list all files containing a given pattern:
for f in *; do if grep -zoPq "foo\nbar" $f; then echo $f; fi; done
But on macos, I'm geting the following error:
grep: invalid option -- z
There's no -z option to treat files as a big string with macos grep, unlike gnu grep.
Is there another option on macos grep equivalent to `-z ? If not, what alternative can I use to get the same result ?
-P (PERL regex) is only supported in gnu grep but not on BSD grep found on Mac OS.
You can either use home brew to install gnu grep or else use this equivalent awk command:
awk 'p ~ /foo$/ && /^bar/ {print FILENAME; nextfile}; {p=$0}' *
Please note that this eliminates the need to use shell for loop.
You can install pcregrep via home brew, and then use it with the -M option:
By default, each line that matches a pattern is copied to the
standard output, and if there is more than one file, the file name is
output at the start of each line, followed by a colon. However, there
are options that can change how pcregrep behaves. In particular, the
-M option makes it possible to search for patterns that span line
boundaries. What defines a line boundary is controlled by the -N
(--newline) option.
With ripgrep
rg -lU 'foo\nbar'
This will list all filenames containing foo\nbar in the current directory. -U option allows to match multiple lines. Unlike grep -z, whole file isn't read in one-shot, so this is safe to use even for larger input files.
ripgrep recursively searches by default. Use rg -lU --max-depth 1 'foo\nbar' if you don't want to search sub-directories.
However, note that by default, rigprep ignores
files and directories that match rules specified by ignore files like .gitignore
hidden files and directories
binary files
You can change that by using:
-u or --no-ignore
-uu or --no-ignore --hidden
-uuu or --no-ignore --hidden --binary
It seems you are searching for files which have the sequence foo\nbar. With GNU awk (brew install gawk), you can set the record separatorRS to this sequence and check if the record matches:
gawk 'BEGIN{RS="foo\nbar"}{exit (RT!=RS)}' file
This will try to split your files in records which are separated by the record separator RS, if so, it will terminate with exit code 0, otherwise with exit code 1. The behaviour is the same as the proposed grep
If you just want the files listed, you can do:
gawk 'BEGIN{RS="foo\nbar"}(RT==RS){print FILENAME}{nextfile}' *
I would like to modify the name of several files in a folder. The name are in this format:
Name_name_name_name_XXXX.fa
I would like:
XXXX_Name_name_name_name.fa
I tired using
for f in *.fa; do mv "${f/.fa/Name_name_name_name.fa}";done
output Name_name_name_name_XXXX_Name_name_name_name.fa
and then
for f in *.fa; do mv "${f/Name_name_name_name_//}"; done
to remove the 4 name_name. but it didnt work.
Any idea how to do it?
You can use sed:
for f in *.fa; do
mv "$f" $(sed -E 's/^(.+)_([^.]+)\./\2_\1./' <<< "$f")
done
sed command uses 2 capture groups. 1st group contains string before last underscore and 2nd group contains XXXX part (part between last _` and dot).
Try this using perl's rename in just one command using regex capture groups:
rename -n 's/(.*?)_([^_]+)\.fa$/$2_$1/' *.fa
(remove -n switch when your tests are OK)
There are other tools with the same name which may or may not be able to do this, so be careful.
If you run the following command (GNU)
$ file "$(readlink -f "$(type -p rename)")"
and you have a result like
.../rename: Perl script, ASCII text executable
and not containing:
ELF
then this seems to be the right tool =)
If not, to make it the default (usually already the case) on Debian and derivative like Ubuntu :
$ sudo update-alternatives --set rename /path/to/rename
(replace /path/to/rename to the path of your perl's rename command.
If you don't have this command, search your package manager to install it or do it manually
Last but not least, this tool was originally written by Larry Wall, the Perl's dad.
I have a directory full of files with one extension (.txt in this case) that I want to automatically convert to another extension (.md).
Is there an easy terminal one-liner I can use to convert all of the files in this directory to a different file extension?
Or do I need to write a script with a regular expression?
You could use something like this:
for old in *.txt; do mv $old `basename $old .txt`.md; done
Make a copy first!
Alternatively, you could install the ren (rename) utility
brew install ren
ren '*.txt' '#1.md'
If you want to rename files with prefix or suffix in file names
ren 'prefix_*.txt' 'prefix_#1.md'
Terminal is not necessary for this... Just highlight all of the files you want to rename. Right click and select "Rename ## items" and just type ".txt" into to the "Find:" box and ".md" into the "Replace with:" box.
The preferred Unix way to do this (yes, OS X is based on Unix) is:
ls | sed 's/^\(.*\)\.txt$/mv "\1.txt" "\1.md"/' | sh
Why looping with for if ls by design loops through the whole list of filenames? You've got pipes, use them. You can create/modify not only output using commands, but also commands (right, that is commands created by a command, which is what Brian Kernighan, one of the inventors of Unix, liked most on Unix), so let's take a look what the ls and the sed produces by removing the pipe to sh:
$ ls | sed 's/^\(.*\)\.txt$/mv "\1.txt" "\1.md"/'
mv "firstfile.txt" "firstfile.md"
mv "second file.txt" "second file.md"
$
As you can see, it is not only an one-liner, but a complete script, which furthermore works by creating another script as output. So let's just feed the script produced by the one-liner script to sh, which is the script interpreter of OS X. Of course it works even for filenames with spaces in it.
BTW: Every time you type something in Terminal you create a script, even if it is only a single command with one word like ls or date etc. Everything running in a Unix shell is always a script/program, which is just some ASCII-based stream (in this case an instruction stream opposed to a data stream).
To see the actual commands being executed by sh, just add an -x option after sh, which turns on debugging output in the shell, so you will see every mv command being executed with the actual arguments passed by the sed editor script (yeah, another script inside the script :-) ).
However, if you like complexity, you can even use awk and if you like to install other programs to just do basic work, there is ren. I know even people who would prefer to write a 50-lines or so perl script for this simple every-day task.
Maybe it's easier in finder to rename files, but if connected remotely to a Mac (e.g. via ssh), using finder is not possible at all. That's why cmd line still is very useful.
Based on the selected and most accurate answer above, here's a bash function for reusability:
function change_all_extensions() {
for old in *."$1"; do mv $old `basename $old ."$1"`."$2"; done
}
Usage:
$ change_all_extensions txt md
(I couldn't figure out how to get clean code formatting in a comment on that answer.)
No need to write a script for it just hit this command
find ./ -name "*.txt" | xargs -I '{}' basename '{}' | sed 's/\.txt//' | xargs -I '{}' mv '{}.txt' '{}.md'
You do not need a terminal for this one; here is a sample demonstration in MacOS Big Sur.
Select all the files, right-click and select "rename..."
Add the existing file extension in "Find" and the extension you want to replace with "Replace with".
And done!
I had a similar problem where files were named .gifx.gif at the end and this worked in OS X to remove the last .gif:
for old in *.gifx.gif; do
mv $(echo "$old") $(echo "$old" | sed 's/x.gif//');
done
cd $YOUR_DIR
ls *.txt > abc
mkdir target // say i want to move it to another directory target in this case
while read line
do
file=$(echo $line |awk -F. '{ print $1 }')
cp $line target/$file.md // depends if u want to move(mv) or copy(cp)
done < abc
list=ls
for file in $list
do
newf=echo $file|cut -f1 -d'.'
echo "The newf is $newf"
mv $file $newf.jpg
done
I'm trying to use plink on winXP to connect to a linux server and execute a command. Here's the command line I'm running:
plink some_profile cd "$(echo 'T:\somedir\somesubdir with space in it\' | sed 's_\\_/_g' | sed 's_T:_/media/drive1_g')";unrar x 'somefile.rar'
If I execute the command portion of this (start with the cd) on the linux box directly it works perfectly. But when I run it through plink, it fails with the following error:
bash: line 0: cd: /media/drive1/somedir/somesubdir: No such file or directory
I think I'm not quoting the command correctly when going through plink so it's not transferring the quotes needed to handle the spaces in the directories. Any ideas on how this should be quoted so that it works?
In case you're wondering why I'm even doing this, it's basically a script I'm running on Directory Opus where I can unrar the file I currently have highlighted (a samba mount). The directory and filename are passed to the script which will the unrar that file on my remote box.
Thanks!
EDIT: Problem solved thanks to the response from Mikel. Here is the line that works now in case anyone else comes across this later...
plink some_profile cd \"$(echo 'T:\somedir\somesubdir with space in it\' | sed 's_\\_/_g' | sed 's_T:_/media/drive1_g')\";unrar x 'somefile.rar'
You need to add another level of quotes, e.g.
plink some_profile cd "\"$(echo 'T:\somedir\somesubdir with space in it\' | sed 's_\\_/_g' | sed 's_T:_/media/drive1_g')\"";unrar x 'somefile.rar'
This is because you need one level of quotes on the Windows side, and one level of quotes on the Linux side.
Objective
Change these filenames:
F00001-0708-RG-biasliuyda
F00001-0708-CS-akgdlaul
F00001-0708-VF-hioulgigl
to these filenames:
F0001-0708-RG-biasliuyda
F0001-0708-CS-akgdlaul
F0001-0708-VF-hioulgigl
Shell Code
To test:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/'
To perform:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/' | sh
My Question
I don't understand the sed code. I understand what the substitution
command
$ sed 's/something/mv'
means. And I understand regular expressions somewhat. But I don't
understand what's happening here:
\(.\).\(.*\)
or here:
& \1\2/
The former, to me, just looks like it means: "a single character,
followed by a single character, followed by any length sequence of a
single character"--but surely there's more to it than that. As far as
the latter part:
& \1\2/
I have no idea.
First, I should say that the easiest way to do this is to use the
prename or rename commands.
On Ubuntu, OSX (Homebrew package rename, MacPorts package p5-file-rename), or other systems with perl rename (prename):
rename s/0000/000/ F0000*
or on systems with rename from util-linux-ng, such as RHEL:
rename 0000 000 F0000*
That's a lot more understandable than the equivalent sed command.
But as for understanding the sed command, the sed manpage is helpful. If
you run man sed and search for & (using the / command to search),
you'll find it's a special character in s/foo/bar/ replacements.
s/regexp/replacement/
Attempt to match regexp against the pattern space. If success‐
ful, replace that portion matched with replacement. The
replacement may contain the special character & to refer to that
portion of the pattern space which matched, and the special
escapes \1 through \9 to refer to the corresponding matching
sub-expressions in the regexp.
Therefore, \(.\) matches the first character, which can be referenced by \1.
Then . matches the next character, which is always 0.
Then \(.*\) matches the rest of the filename, which can be referenced by \2.
The replacement string puts it all together using & (the original
filename) and \1\2 which is every part of the filename except the 2nd
character, which was a 0.
This is a pretty cryptic way to do this, IMHO. If for
some reason the rename command was not available and you wanted to use
sed to do the rename (or perhaps you were doing something too complex
for rename?), being more explicit in your regex would make it much
more readable. Perhaps something like:
ls F00001-0708-*|sed 's/F0000\(.*\)/mv & F000\1/' | sh
Being able to see what's actually changing in the
s/search/replacement/ makes it much more readable. Also it won't keep
sucking characters out of your filename if you accidentally run it
twice or something.
you've had your sed explanation, now you can use just the shell, no need external commands
for file in F0000*
do
echo mv "$file" "${file/#F0000/F000}"
# ${file/#F0000/F000} means replace the pattern that starts at beginning of string
done
I wrote a small post with examples on batch renaming using sed couple of years ago:
http://www.guyrutenberg.com/2009/01/12/batch-renaming-using-sed/
For example:
for i in *; do
mv "$i" "`echo $i | sed "s/regex/replace_text/"`";
done
If the regex contains groups (e.g. \(subregex\) then you can use them in the replacement text as \1\,\2 etc.
The easiest way would be:
for i in F00001*; do mv "$i" "${i/F00001/F0001}"; done
or, portably,
for i in F00001*; do mv "$i" "F0001${i#F00001}"; done
This replaces the F00001 prefix in the filenames with F0001.
credits to mahesh here: http://www.debian-administration.org/articles/150
The sed command
s/\(.\).\(.*\)/mv & \1\2/
means to replace:
\(.\).\(.*\)
with:
mv & \1\2
just like a regular sed command. However, the parentheses, & and \n markers change it a little.
The search string matches (and remembers as pattern 1) the single character at the start, followed by a single character, follwed by the rest of the string (remembered as pattern 2).
In the replacement string, you can refer to these matched patterns to use them as part of the replacement. You can also refer to the whole matched portion as &.
So what that sed command is doing is creating a mv command based on the original file (for the source) and character 1 and 3 onwards, effectively removing character 2 (for the destination). It will give you a series of lines along the following format:
mv F00001-0708-RG-biasliuyda F0001-0708-RG-biasliuyda
mv abcdef acdef
and so on.
Using perl rename (a must have in the toolbox):
rename -n 's/0000/000/' F0000*
Remove -n switch when the output looks good to rename for real.
There are other tools with the same name which may or may not be able to do this, so be careful.
The rename command that is part of the util-linux package, won't.
If you run the following command (GNU)
$ rename
and you see perlexpr, then this seems to be the right tool.
If not, to make it the default (usually already the case) on Debian and derivative like Ubuntu :
$ sudo apt install rename
$ sudo update-alternatives --set rename /usr/bin/file-rename
For archlinux:
pacman -S perl-rename
For RedHat-family distros:
yum install prename
The 'prename' package is in the EPEL repository.
For Gentoo:
emerge dev-perl/rename
For *BSD:
pkg install gprename
or p5-File-Rename
For Mac users:
brew install rename
If you don't have this command with another distro, search your package manager to install it or do it manually:
cpan -i File::Rename
Old standalone version can be found here
man rename
This tool was originally written by Larry Wall, the Perl's dad.
The backslash-paren stuff means, "while matching the pattern, hold on to the stuff that matches in here." Later, on the replacement text side, you can get those remembered fragments back with "\1" (first parenthesized block), "\2" (second block), and so on.
If all you're really doing is removing the second character, regardless of what it is, you can do this:
s/.//2
but your command is building a mv command and piping it to the shell for execution.
This is no more readable than your version:
find -type f | sed -n 'h;s/.//4;x;s/^/mv /;G;s/\n/ /g;p' | sh
The fourth character is removed because find is prepending each filename with "./".
Here's what I would do:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done
Then if that looks ok, add | sh to the end. So:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done | sh
for i in *; do mv $i $(echo $i|sed 's/AAA/BBB/'); done
The parentheses capture particular strings for use by the backslashed numbers.
ls F00001-0708-*|sed 's|^F0000\(.*\)|mv & F000\1|' | bash
Some examples that work for me:
$ tree -L 1 -F .
.
├── A.Show.2020.1400MB.txt
└── Some Show S01E01 the Loreming.txt
0 directories, 2 files
## remove "1400MB" (I: ignore case) ...
$ for f in *; do mv 2>/dev/null -v "$f" "`echo $f | sed -r 's/.[0-9]{1,}mb//I'`"; done;
renamed 'A.Show.2020.1400MB.txt' -> 'A.Show.2020.txt'
## change "S01E01 the" to "S01E01 The"
## \U& : change (here: regex-selected) text to uppercase;
## note also: no need here for `\1` in that regex expression
$ for f in *; do mv 2>/dev/null "$f" "`echo $f | sed -r "s/([0-9] [a-z])/\U&/"`"; done
$ tree -L 1 -F .
.
├── A.Show.2020.txt
└── Some Show S01E01 The Loreming.txt
0 directories, 2 files
$
2>/dev/null suppresses extraneous output (warnings ...)
reference [this thread]: https://stackoverflow.com/a/2372808/1904943
change case: https://www.networkworld.com/article/3529409/converting-between-uppercase-and-lowercase-on-the-linux-command-line.html