I use sort file
ABC
AB-C
ABCDEFG-HI
I get
ABC
AB-C
ABCDEFG-HI
why does sort orders the string this way? how do I make it sort '-' alphabetically?
The solution provided by #cnicutar is correct, but the reason needs explanation which is why I'm giving a new answer.
After the discussion with #cnicutar where in the end I suspected a bug in coreutils' sort I found that this sorting behavior is expected:
At that point sort appears broken because case is folded and punctuation is ignored because ‘en_US.UTF-8’ specifies this behavior.
So to sort, your input seems to be mapped as follows:
ABC -> ABC
AB-C -> ABC
ABCDEFG-HI -> ABCDEFGHI
If you want pure ASCII sorting, you need to call LC_ALL=C sort (temporarily set the locale to C when calling sort which means "standard" behavior without localization; you can also use POSIX instead of C).
On other Unixes this behavior seems to be different (tested on Mac OS X which userland tools are derived from FreeBSD), but LC_ALL=C sort should yield the same behavior across all POSIX systems.
I remember this :)) try
[cnicutar#aiur ~]$ LANG=POSIX sort
ABC
AB-C
ABCDEFG-HI
AB-C
ABC
ABCDEFG-HI
Alternatively LANG=C should work.
Related
If I create a text file containing the following lines:
>TESTTEXT_10000000
>TESTTEXT_1000000
>TESTTEXT_10000002
>TESTTEXT_10000001
and perform sort myfile, my output is
>TESTTEXT_1000000
>TESTTEXT_10000000
>TESTTEXT_10000001
>TESTTEXT_10000002
However, if I append /1 and /2 to my lines the sort output changes drastically, and I do not know why.
Input:
>TESTTEXT_10000000/1
>TESTTEXT_1000000/1
>TESTTEXT_10000002/1
>TESTTEXT_10000001/1
Output:
>TESTTEXT_10000000/1
>TESTTEXT_1000000/1
>TESTTEXT_10000001/1
>TESTTEXT_10000002/1
Input:
>TESTTEXT_10000000/2
>TESTTEXT_1000000/2
>TESTTEXT_10000002/2
>TESTTEXT_10000001/2
Output:
>TESTTEXT_10000000/2
>TESTTEXT_10000001/2
>TESTTEXT_1000000/2
>TESTTEXT_10000002/2
Is the forward slash being recognised as a seperator? using --field-sperator did not alter the behaviour. If so, why is 1000000/2 in between the 1000001/2 and 1000002/2 entries? Using the human sort, numeric sort or other options never brought about consistency. Can anyone help me out here?
:edit:
Because it seems to be relevant, considering the answers, the value of LC_ALL on this machine is en_GB.UTF-8
/ is before 0 in your locale. Using LC_ALL=C or other locale will properly not change anything.
In your use case you would properly be able to use -Version sort:
sort -V myfile
Alternative can you specify the separator and keys to sort on:
sort -t/ -k1,1 myfile
I'm working on a some build scripts that I'd like to depend on only standardized features.
I need to sort some files by version. Say the files are bar-1.{0,2,3} bar-11.{0,2,3}.
By default, ls gives me:
bar-1_0
bar-11_0
bar-11_2
bar-11_3
bar-1_2
bar-1_3
Getting what I want is easy using 'ls -v':
bar-1_0
bar-1_2
bar-1_3
bar-11_0
bar-11_2
bar-11_3
The problem is that 'ls -v' is not standard. Standard sort also seems to lack the option I want, though I could be looking at old versions of the specs.
Can anyone suggest a portable way to achieve this effect short of writing my own sort routine?
Thanks,
Rhys
sort -n -t- -k2 seems to do what you want. -n gives you numeric (i.e. not alphabetic) sort; -t- sets the field separator to -, and -k2 selects the second field, i.e. the version number.
My sort, on Ubuntu, even does the part with the underscore correctly, but I'm not sure if that is standard. To be sure, you could sort by the minor version first, then by major version.
unix numeric sort gives strange results, even when I specify the delimiter.
$ cat example.csv # here's a small example
58,1.49270399401
59,0.000192136419373
59,0.00182092924724
59,1.49270399401
60,0.00182092924724
60,1.49270399401
12,13.080339685
12,14.1531049905
12,26.7613447051
12,50.4592437035
$ cat example.csv | sort -n --field-separator=,
58,1.49270399401
59,0.000192136419373
59,0.00182092924724
59,1.49270399401
60,0.00182092924724
60,1.49270399401
12,13.080339685
12,14.1531049905
12,26.7613447051
12,50.4592437035
For this example, sort gives the same result regardless if you specify the delimiter. I know if I set LC_ALL=C then sort starts to give expected behavior again. But I do not understand why the default environment settings, as shown below, would make this happen.
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
I've read from many other questions (e.g. here, here, and here) how to avoid this behavior in sort, but still, this behavior is incredibly weird and unpredictable and has caused me a week of heartache. Can someone explain why sort with default environment settings on Mac OS X (10.8.5) would behave this way? In other words: what is sort doing (with local variables set to en_US.UTF-8) to get that result?
I'm using
sort 5.93 November 2005
$ type sort
sort is /usr/bin/sort
UPDATE
I've discussed this on the gnu-coreutils list and now understand why sort with english unicode default locale settings gave the output it did. Because in English unicode, the comma character "," is considered a numeric (so as to allow for comma's as thousand's (or e.g. hundreds) separators), and sort defaults to "being greedy" when it interprets a line, it read the example numbers as approximately
581.491...
590.000...
590.001...
591.492...
600.001...
601.492...
1213.08...
1214.15...
1226.76...
1250.45...
Although this was not what I had intended and chepner is right that to get the actual result I want, I need to specify that I want sort to key on only the first field. sort defaults to interpreting more of the line as a key rather than just the first field as a key.
This behavior of sort has been discussed in gnu-coreutil's FAQ, and is further specified in the POSIX description of sort.
So that, as Eric Blake on the gnu-coreutil's list put it, if the field-separator is also a numeric (which a comma is) then "Without -k to stop things, [the field-separator] serves as BOTH a separator AND a numeric character - you are sorting on numbers that span multiple fields."
I'm not sure this is entirely correct, but it's close.
sort -n -t, will try to sort numerically by the given key(s). In this case, the key is a tuple consisting of an integer and a float. Such tuples cannot be sorted numerically.
If you explicitly specify which single keys to sort on with
sort -k1,1n -k2,2n -t,
it should work. Now you are explicitly telling sort to first sort on the first field (numerically), then on the second field (also numerically).
I suspect that -n is useful as a global option only if each line of the input consists of a single numerical value. Otherwise, you need to use the -n option in conjunction with the -k option to specify exactly which fields are numbers.
Use sort --debug to find out what's going on.
I've used that to explain in detail your issue at:
http://lists.gnu.org/archive/html/coreutils/2013-10/msg00004.html
If you use
cat example.csv | sort
instead of
cat example.csv | sort -n --field-separator=,
then it would give correct output. Use this command, hope this is helpful to you.
Note: I tested with "sort (GNU coreutils) 7.4"
I have a huge file and I split the big file into several small chunks and divide and conquer. Now I have a folder contains a list of files like below:
output_aa #(the output file done: cat input_aa | python parse.py > output_aa)
output_ab
output_ac
output_ad
...
I am wondering is there a way to merge those files back together FOLLOWING THE INDEX ORDER:
I know I could do it by using
cat * > output.all
but I am more curious another magical command already exist comes with split..
The magic command would be:
cat output_* > output.all
There is no need to sort the file names as the shell already does it (*).
As its name suggests, cat original design was precisely to conCATenate files which is basically the opposite of split.
(*) Edit:
Should you use an (hypothetical ?) locale that use a collating order where the a-z order is not abcdefghijklmnopqrstuvwxyz, here is one way to overcome the issue:
LC_ALL=C "sh -c cat output_* > output.all"
There are other ways to concat files together, but there is no magical "opposite of split" in "linux".
Of course, talking about "linux" in general is a bit far fetched, as many distributions have different tools (most of them use a different shell already by default, like sh, bash, csh, zsh, ksh, ...), but if you're talking about debian based linux at least, I don't know of any distribution which would provide such a tool.
For sorting you can use the linux command "sort" ;
Also be aware that using ">" for redirecting stdout will override maybe existing contents, while ">>" will concat to an existing file.
I don't want to copycat, but still make this answer complete, so what jlliagre said about the cat command should also be considered of course (that "cat" was made to con-"cat" files, effectively making it possible to reverse the split command - but that's only provided you use the same ordering of files, so it's not exactly the "opposite of split", but will work that way in close to 100% of the cases (see comments under jlliagre answer for specifics))
I have a folder with scripts with a name pattern of UPDATE[x.y.z] where x.y.z is the script's version.
What I need is a bash script that runs the scripts ordered by their version, hence alphabetic sort is not good.
For example UPDATE1.11.0 should be executed after UPDATE1.2.3.
is there a comparator I can use on order to dictate that sorting order? if not, how else can it be done?
And if you don't have GNU sort (i.e. you're in OSX or FreeBSD or NetBSD), you may be able to fake it by sorting different fields.
[ghoti ~]$ printf "foo1.2.3\nfoo1.11.0\nfoo1.4.1\nfoo1.4.0\n" | sort -nt. -k2,3
foo1.2.3
foo1.4.0
foo1.4.1
foo1.11.0
[ghoti ~]$
This misses out on the major version number because it's not delimited by a dot. But it may work for you anyway.
If you have GNU sort or another version that supports it, you can use version sort.
sort -V
or
sort --version-sort
Also, in my answer here, I posted a script which compares versions.