remove latin-1 character from large text file in bash - utf-8

Question

I have some large dataset plain text files (wikipedia articles) and I have to remove latin-1 characters like here:
kemer } şehir kır toplam }}
use specific terminology . for example , it is often more appropriate for people or things from ethiopia ( a country in africa ) to be described as ethiopian , not carelessly ( with the risk of stereotyping ) as african .
bat avg .
label ਕਾਲਜ
ਅਡੋਲਫ ਹਿਟਲਰ ਨੇ ਦੇਸ਼ ਵਿਚ ਕਮਿਊਨਿਸਟ ਪਾਰਟੀ ਬਣਾਉਣ ਦੀ ਇਜਾਜ਼ਤ ਦੇਣ ਤੋਂ ਨਾਂਹ ਕਰ ਦਿਤੀ।
alt }
if not extra_units then
utc_offset +
ਕਬਜਾ ( )
demographics _title regional
I want to get only like
ਕਾਲਜ
ਅਡੋਲਫ ਹਿਟਲਰ ਨੇ ਦੇਸ਼ ਵਿਚ ਕਮਿਊਨਿਸਟ ਪਾਰਟੀ ਬਣਾਉਣ ਦੀ ਇਜਾਜ਼ਤ ਦੇਣ ਤੋਂ ਨਾਂਹ ਕਰ ਦਿਤੀ।
ਕਬਜਾ
and eventually trim white space lines that is trivial.
The approach I have used was the following
<?php
$in = fopen('php://stdin','rb');
while($line = stream_get_line($in, 64000)) {
foreach(str_split($line) as $char) {
$ordChar = ord($char);
if($ordChar > 127 || $ordChar <= 31) {
echo $char;
}
}
}
used like cat wiki.hi.txt | php -d memory_limit=1024M escape_latin.php > wiki.hi.esc.txt
This approach works ok, the only issue is that performances are getting worst as the file size grows as I can see with a watch du -h filename on the file I'm working on with. I'm surprised because I'm working on a local disk and I'm using stream_get_line to get the lines in streaming.
I have tried the same approach in python, but I get pretty the same performances with file size of ~1GB.
see here for more details.
[UPDATE]
I'm reporting here some results from alternative approaches proposed
Using the regex approach, that seems to produce pretty much the same output file:
A ~50MB file
$ time tr -d "[:alnum:][:punct:]" < wiki.as.txt > wiki.as.test.txt
real 0m2.990s
user 0m2.818s
sys 0m0.088s
A ~100MB file
$ time tr -d "[:alnum:][:punct:]" < wiki.gu.txt > wiki.gu.test.txt
real 0m7.322s
user 0m6.772s
sys 0m0.282s
A ~600MB file
$ time tr -d "[:alnum:][:punct:]" < wiki.ta.txt > wiki.ta.test.txt
real 0m35.973s
user 0m33.498s
sys 0m1.254s
A ~1000MB (1GB) file
$ time tr -d "[:alnum:][:punct:]" < wiki.ja.1.txt > wiki.ja.1.test.txt
real 1m5.409s
user 1m0.669s
sys 0m2.068s

Answer 1

try a regex.
If you're running it from a CLI, try something like
tr -d "[:alnum:][:punct:]" < wiki.hi.txt > wiki.hi.esc.txt
If you prefer to do the same in php -
<?php
$in = fopen('php://stdin','rb');
while($line = stream_get_line($in, 64000)) {
echo preg_replace('/[:alnum:][:punct:]/', '', $line);
}
But please check these to make sure they are doing what you want - esp. the php, since I'm working without a test setup here. It's likely to have syntax issues and/or worse. With luck someone will edit it or offer a better solution, or at least comment and point out whatever I may have done wrong.
Hope it helps.