how to split by unicode chars in shell - shell

Question

using java:
File file = new File("C:/Users/Administrator/Desktop/es.txt");
List<String> lines = FileUtils.readLines(file, "utf-8");
for (String line : lines) {
String[] arr = line.split("\\u007C\\u001C");
System.out.println(arr.length);
System.out.println(Arrays.toString(arr));
}
how can I do it in shell(awk, tr, or sed)?
I've tried this, but it didn't work:
awk -F\u007c\u001c '{print $1}' es.txt
Thanks.

Answer 1

Obviously, U+007C and U+001C are plain old 7-bit ASCII characters, so splitting on those doesn't actually require any Unicode support (apart from possibly handling any ASCII-incompatible Unicode encoding in the files you are manipulating; but your question indicates that your data is in UTF-8, so that does not seem to be the case here. UTF-16 would require the splitting tool to be specifically aware of and compatible with the encoding).
Assuming your question can be paraphrased as "if I know the numeric Unicode code point I want to split on, how do I pass that to a tool which is capable of splitting on it", my recommendation would be Perl.
perl -CSD -aF'\N{U+1f4a9}' -nle 'print $F[0]' es.txt
using U+1F4A9 as the separator. (Perl's arrays are zero-based, so $F[0] corresponds to Awk's $1. The -a option requests field splitting to the array #F; normally, Perl does not explicitly split the input into fields.) If the hex code for the code point you want to use as the field separator is in a shell variable, use double quotes instead of single, obviously.
PIPE='007C'
FS='001C'
perl -CSD -aF"\N{U+$PIPE}\N{U+$FS}" -nle 'print $F[0]' es.txt
Alternatively, if the tool you want to use handles UTF-8 transparently, you can use the ANSI C quoting facility of Bash to specify the separator. Unicode support seems only to have been introduced in Bash 4.2 so e.g. Debian Squeeze (currently oldoldstable) does not have it.
awk -F$'\U0001f4a9' '{print $1}' es.txt # or $'\u007c' for 4-digit code points
However, because the quoting facility is a form of single quotes, you can't (easily) have the separator's code point value in a variable.

Answer 2

gawk 4.1.3
[root#test /tmp]$ more a
\u8BF7\u5C06\u60A8\u8981\u8F6C\u6362\u7684\u6C49\u6587\u8981\u8F6C\u5185\u5BB9\u
7C98\u8D34\u5728\u8FD9\u91CC\u3002
[root#test /tmp]$ awk -F '.u8981..8F6C' '{print $1}' a
\u8BF7\u5C06\u60A8
[root#test /tmp]$ awk -F '.u8981..8F6C' '{print $2}' a
\u6362\u7684\u6C49\u6587
[root#test /tmp]$ awk -F '.u8981..8F6C' '{print $3}' a
\u5185\u5BB9\u7C98\u8D34\u5728\u8FD9\u91CC\u3002

Answer 3

Pure bash:
As your question is tagged shell there is a pure bash way:
declare -a out=()
pnt=0
while IFS= read -d '' -n1 char ;do
LANG=C LC_ALL=C printf -v val %d "'$char"
(( val == 195 )) && out[pnt]+= &&
printf -v out[pnt+1] "%s" "${char}" &&
((pnt+=2)) ||
printf -v out[pnt] "%s%s" "${out[pnt]}" "${char}"
done <<<'Il est déjà très tard!'
Where submited string containg UTF8 chars and newlines, this will create an array of 7 strings:
declare -p o
declare -a o=([0]="Il est d" [1]="é" [2]="j" [3]="à" [4]=" tr" [5]="è" [6]=$'s tard!\n')
or
cat -n <(printf -- "<%s>\n" "${o[#]#Q}")
1 <'Il est d'>
2 <'é'>
3 <'j'>
4 <'à'>
5 <' tr'>
6 <'è'>
7 <$'s tard!\n'>
Where even fields are separators and odd fields are content.
As a function:
splitOnUnicod () {
local -n out=$1
out=()
local -i pnt=0 cval
local char
while IFS= read -d '' -rn1 char; do
LANG=C LC_ALL=C printf -v cval %d "'$char";
((cval==195)) && out[pnt]+= && printf -v out[++pnt] %s "$char" && pnt+=1 || printf -v out[pnt] %s%s "${out[pnt]}" "$char";
done
}
Then
splitOnUnicod myvar <<<"Généralités"
declare -p myvar
declare -a myvar=([0]="G" [1]="é" [2]="n" [3]="é" [4]="ralit" [5]="é" [6]=$'s\n')
splitOnUnicod myvar < <(printf "Iñès.")
declare -p myvar
declare -a myvar=([0]="I" [1]="ñ" [2]="" [3]="è" [4]="s.")
Where ñ as è are separators, they are in even fields.
paste <(printf %s\\n "${!myvar[#]}") <(printf %s\\n "${myvar[#]}")
0 I
1 ñ
2
3 è
4 s.