How to get rid of special characters in a text file? (*nix) - shell

I've got a source code file, that started as a copy of some sample code from a webpage.
It was created and edited under Windows and compiled with no problems.
But under Mac's I get a load of obscure errors, like:
../MyProgram.cpp:1: error: stray '\255' in program
../MyProgram.cpp:1: error: stray '\254' in program
../MyProgram.cpp:1: error: stray '#' in program
../MyProgram.cpp:3:4: error: invalid preprocessing directive #i
../MyProgram.cpp:5:4: error: invalid preprocessing directive #i
../MyProgram.cpp:7:4: error: invalid preprocessing directive #i
../MyProgram.cpp:23: error: missing terminating ' character
../MyProgram.cpp:369:6: error: invalid preprocessing directive #i
../MyProgram.cpp:371:8: error: invalid preprocessing directive #i
../MyProgram.cpp:375:8: error: invalid preprocessing directive #e
../MyProgram.cpp:381:8: error: invalid preprocessing directive #e
../MyProgram.cpp:383:6: error: invalid preprocessing directive #e
../MyProgram.cpp:385:8: error: invalid preprocessing directive #i
../MyProgram.cpp:389:8: error: invalid preprocessing directive #e
../MyProgram.cpp:1: error: 'i' does not name a type
../MyProgram.cpp:53: error: 'V' does not name a type
../MyProgram.cpp:75: error: 'v' does not name a type
../MyProgram.cpp:157: error: 'l' does not name a type
../MyProgram.cpp:169: error: 'l' does not name a type
../MyProgram.cpp:187: error: 'i' does not name a type
../MyProgram.cpp:197: error: 'v' does not name a type
Looks like the problem is with some special characters.
How can I strip them off with *nix command line?

Looks to me as if the file was saved as UTF-16. Opening it in a text-editor and reencoding to UTF-8 should, with some luck, fix the problem.

Originally I was just going say how to remove the \255 & \254 characters, but I agree with the comments, it's in unicode.
try
iconv -f iso-8859-1 -t utf-8 infile > outfile
iso-8859-1 is just a guess.

Related

llvm 14 fail to build linux kernel v5.10 or v5.11

When I use LLVM 14.0.3 to build Linux Kernel v5.11.22 or v5.10.118 for aarch64, it reports the below error:
CC arch/arm64/kernel/vdso/vgettimeofday.o
/tmp/vgettimeofday-0dc7a1.s: Assembler messages:
/tmp/vgettimeofday-0dc7a1.s:15: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:15: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:16: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:17: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:18: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:19: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:25: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:25: Error: junk at end of line, first unrecognized character is `0'
/tmp/vgettimeofday-0dc7a1.s:50: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:151: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:191: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:236: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:277: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:382: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:382: Error: junk at end of line, first unrecognized character is `2'
/tmp/vgettimeofday-0dc7a1.s:746: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:747: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:748: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:749: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:750: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:757: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:757: Error: junk at end of line, first unrecognized character is `0'
/tmp/vgettimeofday-0dc7a1.s:1063: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:1063: Error: junk at end of line, first unrecognized character is `2'
/tmp/vgettimeofday-0dc7a1.s:1155: Error: junk at end of line, first unrecognized character is `"'
/tmp/vgettimeofday-0dc7a1.s:1162: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:1162: Error: junk at end of line, first unrecognized character is `0'
/tmp/vgettimeofday-0dc7a1.s:1295: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:1295: Error: junk at end of line, first unrecognized character is `2'
/tmp/vgettimeofday-0dc7a1.s:1324: Error: file number less than one
/tmp/vgettimeofday-0dc7a1.s:1324: Error: junk at end of line, first unrecognized character is `2'
clang-14: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [scripts/Makefile.build:279: arch/arm64/kernel/vdso/vgettimeofday.o] Error 1
make: *** [arch/arm64/Makefile:191: vdso_prepare] Error 2
Below are the build commands that I use:
make ARCH=arm64 LLVM=1 CROSS_COMPILE=aarch64-linux-gnu- defconfig
make ARCH=arm64 LLVM=1 CROSS_COMPILE=aarch64-linux-gnu-
But I can build Linux Kernel v5.15.43 and the latest v5.18 by using the same build commands.
Also I tried LLVM 13.0.0, it can build Linux Kernel v5.11.22 for aarch64.
Pass LLVM_IAS=1 to kernel Makefile along with LLVM=-14 to ensure Clang use cross toolchain compatible for LLVM 14.x.x version.

Bash beeline alias

I use beeline from the command line quite a lot, so I would like to create a bash alias for the beeline command that takes care of all the boilerplate for me and does some argument parsing and things. Specifically, one thing I would like to do is create a showtables command which takes a single argument, the name of a database, and invokes the beeline command with the boiler plate and passes the -e argument along with the appropriate SQL, i.e. - showtables db1 should invoke /usr/bin/beeline -u $config -e "SHOW TABLES IN db1". My source file looks like this :
/usr/bin/beeline --showHeader=False --outputformat=tsv2 -u $config -e \"SHOW TABLES IN $1\"
but the output is
...
Error: Error while compiling statement: FAILED: ParseException line 1:4 cannot recognize input near 'SHOW' '<EOF>' '<EOF>' in ddl statement (state=42000,code=40000)
Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'TABLES' '<EOF>' '<EOF>' (state=42000,code=40000)
Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'IN' '<EOF>' '<EOF>' (state=42000,code=40000)
Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'db1' '<EOF>' '<EOF>' (state=42000,code=40000)
...
I've verified that just doing
/usr/bin/beeline --showHeader=False --outputformat=tsv2 -u $config -e "SHOW TABLES IN db1"
works as expected. I don't understand why my executable isn't working though.
The backslashes before your quotes make them behave like regular characters from the perspective of parsing, and not like quotes anymore. Thus, when you put backslashes in front of the quotes, SHOW TABLES IN db1 is no longer a string, but is four separate strings: "SHOW, TABLES, IN, and db1".
These words are not valid queries when run on their own -- hence your error. Take out the backslashes and the issue goes away.

#OCaml ocamldoc generation error

Hello I'm trying to generate ocaml documentation with ocamldoc. I'm trying the following command : ocamldoc -html alpha.mli -d doc/
But I have this error :
File "compiler/alpha.ml", line 16, characters 12-23:
Error: Unbound module Id
File "compiler/argHandler.ml", line 3, characters 17-27:
Error: Unbound module Parser
File "compiler/ARMGeneration.ml", line 7, characters 38-41:
Warning 10: this expression should have type unit.
File "compiler/ARMGeneration.ml", line 8, characters 47-59:
Warning 10: this expression should have type unit.
File "compiler/ARMGeneration.ml", line 11, characters 10-33:
Error: Unbound module Exception
File "compiler/ASMLGeneration.ml", line 7, characters 44-60:
Error: Unbound module Syntax
File "compiler/id.ml", line 44, characters 38-61:
Error: Unbound module Exception
File "compiler/knorm.ml", line 11, characters 4-14:
Error: Unbound module Syntax
File "compiler/letfold.ml", line 4, characters 4-15:
Error: Unbound module Syntax
File "compiler/main.ml", line 15, characters 4-24:
Error: Unbound module ArgHandler
File "compiler/regAllocation.ml", line 202, characters 18-29:
Error: Unbound module Id
File "compiler/SyntaxArm.ml", line 8, characters 12-16:
Error: Unbound module Id
File "compiler/syntax.ml", line 20, characters 12-16:
Error: Unbound module Id
11 error(s) encountered
This error will happen when ocamldoc can't locate the .cmi generated by the build.
In order for it to work you need to first compile your project and then point ocamldoc towards the _build/compiler folder where you .cmi are
A valid command for you might look like :
ocamldoc -html -I _build/compiler compiler/*.ml -d doc/
the -I option indicate that you will specify the folder where the .cmi are.

libprotobuf ERROR when training cifar10 on the windows Caffe

Hi I'm new to DNN and I'm trying Caffe on windows 7. I have successfully compiled Caffe and also made train and test leveldb files of CIFAR10 and mean.binaryproto file. Now it's time to start train and test the CIFAR10, but when I try train, I get the errors that I couldn't solve even after googling long time..Please help!
Here's my version info : Anaconda2, Cuda 7.5, Cudnn 4, Visual Studio 2013
C:\caffe-master\Build\x64\Release>caffe train -solver=cifar10_quick_solver.prototxt
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 4:10: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 4:23: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 4:32: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 4:40: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 24:22: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 24:35: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 24:44: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 24:52: Invalid escape sequence in string literal.
F1019 14:25:21.171140 5692 upgrade_proto.cpp:1063] Check failed: ReadProtoFromT
extFile(param_file, param) Failed to parse SolverParameter file: cifar10_quick_solver.prototxt
*** Check failure stack trace: ***
I also tried to update prototxt using upgrade_net_proto_binary, upgrade_net_proto_text, upgrade_solver_proto_text, I get the error...
C:\caffe-master\Build\x64\Release>upgrade_solver_proto_text c:\caffe-master\examples\cifar10\cifar10_full_solver.prototxt cifar10_full_new_solver.prototxt
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 25:22: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 25:35: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 25:44: Invalid escape sequence in string literal.
[libprotobuf ERROR ..\src\google\protobuf\text_format.cc:274] Error parsing text
-format caffe.SolverParameter: 25:52: Invalid escape sequence in string literal.
E1018 20:49:33.715873 5896 upgrade_solver_proto_text.cpp:30] Failed to parse in
put text file as SolverParameter: c:\caffe-master\examples\cifar10\cifar10_full_solver.prototxt
After all those trials and errors, I got the answer.
For those who suffer the same thing, please refer.
at the solver file, I have indicated the source file and mean file to absolute address.
But the mistake I made is, I have written the address with \ not /.
Even I type in the command in cmd window with using \, in the prototxt, it does not allow it.
So you should write the address in prototxt file, with /.
i.e. C:/caffe-master/examples/cifar10/cifar10_full_train.prototxt.
Cheers.

Invalid multibyte char(UTF-8)

I am trying compile this Ruby code with option --1.9:
\# encoding: utf-8
module Modd
def cpd
#"_¦+?" mySQL
"ñ,B˜"
end
end
I used the GVim editor and compiled then got the following error:
SyntaxError: f3.rb:6: invalid multibyte char (UTF-8)
After that I used Notepad++ and changed to Encode as UTF-8 and compiled with this option:
jruby --1.9 f3.rb
then I get:
SyntaxError: f3.rb:1: \273Invalid char `\273' ('╗') in expression
I have seen this happen when the BOM gets messed up during a charset conversion (the BOM in octal is 357 273 277). If you open the file with a hexadecimal editor (:%!xxd on vi), you will more than likely see characters at the beginning of the file, before the first #.
If you recreate that file directly in utf-8, or get rid of these spurious characters, this should solve your problem.

Resources