Bash locale sensitive globbing on OS X - macos

Scenario on both Linux(CentOS) and OS X(Yosemite):
$ touch A B C X Y Z a b c x y z
$ locale
LC_ALL=en_GB.UTF-8
On OS X I am using a case-sensitive filesystem.
Bash 4.1.2 on Linux:
$ echo [A-Z]
A b B c C x X y Y z Z
This is the expected output, the LC_COLLATE for this locale is followed.
Bash 4.3.39 on OS X:
$ echo [A-Z]
A B C X Y Z
This appears to be the same as locale ANSI, C, or POSIX. So, it looks like on OS X the locale is ignored by globbing.
Why is it inconsistent? Is there any way to get the locale sensitive result on OS X?
Edit:
OS X: LC_ALL is explicitly set in .bash_profile using:
export LC_ALL=en_GB.UTF-8
locale on OS X:
bash-4.3$ locale
LANG="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_CTYPE="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_ALL="en_GB.UTF-8"
locale on Linux:
$ locale
LANG=en_US.utf8
LC_CTYPE="en_GB.utf8"
LC_NUMERIC="en_GB.utf8"
LC_TIME="en_GB.utf8"
LC_COLLATE="en_GB.utf8"
LC_MONETARY="en_GB.utf8"
LC_MESSAGES="en_GB.utf8"
LC_PAPER="en_GB.utf8"
LC_NAME="en_GB.utf8"
LC_ADDRESS="en_GB.utf8"
LC_TELEPHONE="en_GB.utf8"
LC_MEASUREMENT="en_GB.utf8"
LC_IDENTIFICATION="en_GB.utf8"
LC_ALL=en_GB.utf8
Edit
Format of the filesystem is Mac OS Extended (Case-sensitive, Journaled).
#chepner suggested that I didn't have separate files on OS X. Two things here, the inodes are different, and the glob construct ? gave all the files in upper and lower case:
bash-4.3$ ls -li ?
559 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 A
560 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 B
561 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 C
562 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 X
563 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 Y
564 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 Z
565 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 a
566 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 b
567 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 c
568 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 x
569 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 y
570 -rw-r--r-- 1 clivedarke staff 0 Aug 4 17:23 z
Also:
bash-4.3$ echo 'lower' > a
bash-4.3$ echo 'upper' > A
bash-4.3$ cat a
lower
bash-4.3$ cat A
upper
bash-4.3$ diff a A
1c1
< lower
---
> upper

Digging int the source code I found in bash-4.3/lib/glob/smatch.c the following comment:
/* We use strcoll(3) for range comparisons in bracket expressions,
even though it can have unwanted side effects in locales
other than POSIX or US. For instance, in the de locale, [A-Z] matches
all characters. If GLOB_ASCIIRANGE is non-zero, and we're not forcing
the use of strcoll (e.g., for explicit collating symbols), we use
straight ordering as if in the C locale. */
In configure I found:
--enable-glob-asciiranges-default
force bracket range expressions in pattern matching
to use the C locale by default
So I guess that was set when compiled. Need more digging.

Related

"show-all-if-ambiguous" produces ^X and not intended output

I am trying to get "glob-expand-word" to work
pi#raspberrypi:~ $ cat .inputrc
set editing-mode vi
set keymap vi
set show-all-if-ambiguous on
Control-o: "> output"
pi#raspberrypi:~/tmp/test4 $ ls
total 8.0K
drwxr-xr-x 10 pi pi 4.0K Jan 27 08:48 ..
-rw-r--r-- 1 pi pi 0 Jan 27 08:48 3_eng
-rw-r--r-- 1 pi pi 0 Jan 27 08:48 2_eng
-rw-r--r-- 1 pi pi 0 Jan 27 08:48 1_eng
drwxr-xr-x 2 pi pi 4.0K Jan 27 08:48 .
pi#raspberrypi:~/tmp/test4 $ ls *eng^X
When trying to do the ctrlx, * (a two stroke combo) it doesn't work - instead of auto-completing and outputting "ls 1_eng 2_eng 3_eng" it instead appends ^X
What can I try next?
Try to bind ctrl-x :
bind '"\C-x": glob-list-expansions'
# Then -> ls *eng^X

rsync: failed to set times on "/cygdrive/e/.": Invalid argument (22)

I get the below error message when I try to rsync from a local hard disk to a USB disk mounted at E: on Windows 10.
rsync: failed to set times on "/cygdrive/e/.": Invalid argument (22)
My rsync command is as below (path shortened for brevity):
rsync -rtv --delete --progress --modify-window=5 /cygdrive/d/path/to/folder/ /cygdrive/e/
I actually need to set modification times (on directories as well) and rsync actually sets modification times perfectly. It only fails to set times on root of the USB disk.
I experienced exactly the same problem.
I created a dir containing one text file and when trying to rsync it to an removable (USB) drive, I got the error. However, the file was copied to the destination. The problem is not reproducible if the destination is a folder (other than root) on the removable drive
I then repeated the process using a fixed drive as destination, and the problem was not reproducible
The 1st difference that popped up between the 2 drives, was the file system (for more details, check [MS.Docs]: File Systems Technologies):
FAT32 - on the removable drive
NTFS - on the fixed one
So this was the cause of my failure. Formatting the USB drive as NTFS fixed the problem:
The USB drive formatted as FAT32 (default):
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 20
dr-xr-xr-x 1 cfati None 0 Jul 14 17:58 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxr-xr-x 1 cfati None 0 Dec 31 1979 w
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ rsync -rtv --progress --modify-window=5 ./dir/ /cygdrive/w
sending incremental file list
rsync: failed to set times on "/cygdrive/w/.": Invalid argument (22)
./
a.txt
3 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/2)
sent 111 bytes received 111 bytes 444.00 bytes/sec
total size is 3 speedup is 0.01
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 20
dr-xr-xr-x 1 cfati None 0 Jul 14 17:58 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxr-xr-x 1 cfati None 0 Dec 31 1979 w
After formatting the USB drive as NTFS:
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 24
dr-xr-xr-x 1 cfati None 0 Jul 14 17:59 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxrwxrwx+ 1 Administrators Administrators 0 Jul 14 17:59 w
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ rsync -rtv --progress --modify-window=5 ./dir/ /cygdrive/w
sending incremental file list
./
a.txt
3 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/2)
sent 111 bytes received 38 bytes 298.00 bytes/sec
total size is 3 speedup is 0.02
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 24
dr-xr-xr-x 1 cfati None 0 Jul 14 17:59 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxrwxrwx+ 1 Administrators Administrators 0 Jul 14 13:19 w
As a side note, when I was at step #2., I was an idiot and kept the --delete arg, so til I hit Ctrl + C, it deleted some data. Luckily, it didn't get to delete crucial files / folders.

TCL/Expect - exec - how to execute program with parameters

I am experimenting with TCL command exec in tclsh and here are my results:
% set show_me_dir "ls"
ls
% exec $show_me_dir
VboxSharedFolder
% set show_me_dir "ls -la"
ls -la
% exec $show_me_dir
couldn't execute "ls -la": no such file or directory
% set show_me_dir {ls -la}
ls -la
% exec $show_me_dir
couldn't execute "ls -la": no such file or directory
% ls -la
total 141
d---------+ 1 wakatana Domain Users 0 Jan 22 19:12 .
d---------+ 1 wakatana Domain Users 0 Apr 16 2014 ..
----------+ 1 wakatana Domain Users 20214 Jan 23 18:43 .bash_history
----------+ 1 wakatana Domain Users 1494 Apr 15 2014 .bash_profile
----------+ 1 wakatana Domain Users 7593 Jan 22 19:03 .bashrc
d---------+ 1 wakatana Domain Users 0 Jan 15 14:56 VboxSharedFolder
%
Can somebody please explain how can I execute command with arguments?
Edit:
The following example from Expanding a list of parameters in Tcl and eval article was big eye opener of what is going on here:
The variable $action is only expanded into the string "piemiddle apple" AFTER the command line has been split into its individual parameters:
% set action {piemiddle apple}
% set $action
can't read "piemiddle apple": no such variable
Result: set command "sees" one argument, equivalent to:
% set {piemiddle apple}
The expand operator allows you to specify that a variable is to be expanded BEFORE the command line is split into individual parameters:
% set action {piemiddle apple}
% set {*}$action
apple
Result: set command "sees" two arguments, equivalent to:
% set piemiddle apple
In earlier versions of Tcl, the eval command was the recommended alternative and it remains available today.
% set action {piemiddle apple}
% eval set $action
apple
Another examples which proves functionality of expansion operator:
% set {*}"name Linus"
Linus
% puts $name
Linus
%
%
% set distro Unbuntu
Unbuntu
% set {*}"linux $distro"
Unbuntu
% puts $linux
Unbuntu
%
%
Finally the discovery that exec needs command as it's first argument and first command option as it's second argument etc.
% exec "ls" "-la"
total 137
d---------+ 1 wakatana Domain Users 0 Jan 22 19:12 .
d---------+ 1 wakatana Domain Users 0 Apr 16 2014 ..
----------+ 1 wakatana Domain Users 20214 Jan 23 18:43 .bash_history
----------+ 1 wakatana Domain Users 1494 Apr 15 2014 .bash_profile
----------+ 1 wakatana Domain Users 7593 Jan 22 19:03 .bashrc
d---------+ 1 wakatana Domain Users 0 Jan 15 14:56 VboxSharedFolder
%
%
% exec "ls -la"
couldn't execute "ls -la": no such file or directory
The safest way to build a command for exec is to use Tcl's list. For example:
% set tcl_version
8.5
% set cmd [list ls -l tmp]
ls -l tmp
% eval exec $cmd
total 32
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 file.txt
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-1.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-2.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-3.dat
% exec {*}$cmd
total 32
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 file.txt
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-1.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-2.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-3.dat
%
Note that {*} is a new syntax of Tcl 8.5 which can help reduce the uses of eval.
As example for ls command you can do:
exec {*}ls -lsa {*}[glob *.cpp]
Please have a look at What does {*} do in TCL?

How do I batch renumber files in sequence?

I have a sequence of image files that look like this:
image-149454.jpg
image-149455.jpg
I have some other images that I want to append to the end of the sequence where it left off but currently they are numbered from 0 (i.e. image-000000 to image-010000).
What's a script I could use to rename those new images starting from a certain number and going on, in this case 149456 and onwards?
I would try something like this. You just have to adjust the offset and add a string in front of the name, just as you need it.
x=1
for i in *.jpg; do
temp=$(printf "%08d.jpg" ${x}) #padding since you seem to want it
mv ${i} ${temp}
let x=x+1
done
Here's an untested solution in Python. This should change all of the files in a directory to have sequentially higher numbers than the other.
To use it enter: script.py dir1 dir2
Assuming the original files have the higher number (149455) are in dir1 and the new files start from 000000 are in dir2:
import os, sys, re
max_image = 0
# check if current (with higher numbers) and other (with lower) directory is given
if len(sys.argv) == 3:
for files in os.listdir(sys.argv[1]): # first dir is current
if files.endswith(".jpg"):
#for all jpgs get the max
m = re.search("\-(\d+)", files)
number = int(m.group(1))
if max_image < number:
max_image = number
for files in os.listdir(sys.argv[2]): # second dir is other
if files.endswith(".jpg"):
#get current start point
m = re.search("\-(\d+)", files)
number = int(m.group(1))
os.rename(files, "image-" + str(number + max_image)+".jpg") # add the max from current folder
Here's a pure bash solution which uses the last image sequence as your starting point (per your request):
#!/bin/bash
last_seq=$(ls image-* | tail -1 | sort -n | cut -c7-) # grab the sequence number
last_seq=${last_seq%.jpg} # remove the trailing .jpg
if [ -z $last_seq ] ; then
echo "Unable to obtain the last image sequence number"
exit 1
fi
for image in image-0*.jpg ; do
[ -f ${image} ] || break # In case no files match image-0*.jpg
let last_seq=last_seq+1
mv -v ${image} image-$(printf "%06d.jpg" ${last_seq})
done
Here's how it worked locally:
before the run:
~/tmp › ls -l
total 80
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-00000.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-01000.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14908.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14909.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14910.jpg
-rw-r--r-- 1 dyoung staff 418 Dec 12 10:24 testh.sh
during the run:
~/tmp › sh ./testh.sh
image-00000.jpg -> image-14911.jpg
image-01000.jpg -> image-14912.jpg
after the run:
~/tmp › ls -l
total 80
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14908.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14909.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14910.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-14911.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-14912.jpg
-rw-r--r-- 1 dyoung staff 429 Dec 12 12:25 testh.sh

How to set world permissions to be the same as group permissions?

How would you go about changing permissions for a file or in a directory recursively in such a way that group permissions would be copied over to world permissions, with no other changes? For example, to go from this directory listing:
drwxr-x--- 2 septi septi 4096 Jun 29 01:14 example.d
-rw-r----- 1 septi septi 0 Jun 29 01:14 example.r
-rwxr-x--- 1 septi septi 0 Jun 29 01:14 example.x
...to:
drwxr-xr-x 2 septi septi 4096 Jun 29 01:14 example.d
-rw-r--r-- 1 septi septi 0 Jun 29 01:14 example.r
-rwxr-xr-x 1 septi septi 0 Jun 29 01:14 example.x
From the chmod(1) man page (relevant parts extracted):
-R Change the modes of the file hierarchies rooted in the files
instead of just the files themselves.
And:
The symbolic mode is described by the following grammar:
who ::= a | u | g | o
op ::= + | - | =
perm ::= r | s | t | w | x | X | u | g | o
The who symbols "u", "g", and "o" specify the user, group, and
other parts of the mode bits, respectively. The who symbol a is
equivalent to ugo.
The perm symbols represent the portions of the mode bits as follows:
g The group permission bits in the original mode of the file.
So for you:
chmod -R o=g *
Example:
$ ls -l
total 0
drwxr-x--- 2 carl staff 68 Jun 28 10:25 example.d
-rw-r----- 1 carl staff 0 Jun 28 10:25 example.r
-rwxr-x--- 1 carl staff 0 Jun 28 10:25 example.x
$ chmod -R o=g *
$ ls -l
total 0
drwxr-xr-x 2 carl staff 68 Jun 28 10:25 example.d
-rw-r--r-- 1 carl staff 0 Jun 28 10:25 example.r
-rwxr-xr-x 1 carl staff 0 Jun 28 10:25 example.x

Resources