I have the following problem on WINDOWS (Italian):
my NLS_LANG parameter is: ITALIAN_ITALY.UTF8
i want to execute the following query:
INSERT INTO SCHEMA.MY_TABLE("NAME") VALUES('ò');
Doing it by using command line (pure sqlplus) stores invalid data inside DB.
Doing it by using SQLDEVELOPER stores correct data.
I cannot find any way to set this stuff correctly, what should I do? Using SQLPLUS from command line is required.
Any help is appreciated.
When you use sqlplus then it inherits the character set from command line window. You can interrogate and modify character set (aka encoding) with chcp, I assume it is CP850 - which is not UTF8.
Run chcp 65001 before you start sqlplus, then it should work. See also Converting German special characters to English equivalent one in Oracle SQL / PL-SQL or to read more details OdbcConnection returning Chinese Characters as "?"
My script fails on this bad encoding, even I brought all files to UTF-8 but still some won't convert or just have wrong chars inside.
It fails actually on var assignment step.
Can I set some kind of error handling for this case like below so my loop will continue. That ¿ causes all problem.
Need to run this script all the way without errors. Tried already encoding und force_encoding and shebang line. Is Ruby has any kind of error handling routing so I can handle that bad case and continue with the rest of script? How to get rid of this error invalid multibyte char (UTF-8)
line = '¿USE [Alpha]'
lineOK = ' USE [Alpha] OK line'
>ruby ReadFile_Test.rb
ReadFile_Test.rb:15: invalid multibyte char (UTF-8)
I could reproduce your issue by saving the file with ISO-8859-1 encoding.
Running your code with the file in this non UTF8-encoding the error popped up. My solution was to save the file as UTF-8.
I am using Sublime as text editor and there is the option 'file > save with encoding'. I have chosen 'UTF-8' and was able to run the script.
Using puts line.encoding showed me UTF-8 then and no error anymore.
I suggest to re-check the encoding of your saved script file again.
I am getting a json response array as below.
"[{\"id\":\"23886\",\"item_type\":2,\"name\":\"Equalizer\",\"label\":null,\"desc\":null,\"genre\":null,\"show_name\":null,\"img\":\"http:\\/\\/httpg3.scdn.arkena.com\\/10242\\/v2_images\\/tf1\\/0\\/tf1_media_ingest95290_image\\/tf1_media_ingest95290_image_0_208x277.jpg\",\"url\":\"\\/films\\/media-23886-Equalizer.html\",\"duration\":\"2h27mn\",\"durationtime\":\"8865\",\"audio_languages\":null,\"prod\":null,\"year\":null,\"vf\":\"1\",\"vost\":\"1\",\"sd\":true,\"hd\":false,\"sdprice\":\"4.99\",\"hdprice\":null,\"sdfile\":null,\"hdfile\":null,\"sdbundle\":\"12771\",\"hdbundle\":\"12771\",\"teaser\":\"23887\",\"att_getter\":\"Tout le monde a le droit \\u00e0 la justice\",\"orig_prod\":null,\"director\":null,\"actors\":null,\"csa\":\"CSA_6\",\"season\":null,\"episode\":null,\"typeid\":\"1\",\"isfav\":false,\"viewersrating\":\"4.0\",\"criticsrating\":\"3.0\",\"onThisPf\":1},{\"id\":\"23998\",\"item_type\":2,\"name\":\"Le Labyrinthe\",\"label\":null,\"desc\":null,\"genre\":null,\"show_name\":null,\"img\":\"http:\\/\\/httpg3.scdn.arkena.com\\/10242\\/v2_images\\/tf1\\/1\\/tf1_media_ingest94727_image\\/tf1_media_ingest94727_image_1_208x277.jpg\",\"url\":\"\\/films\\/media-23998-Le_Labyrinthe.html\",\"duration\":\"1h48mn\",\"durationtime\":\"6533\",\"audio_languages\":null,\"prod\":null,\"year\":null,\"vf\":\"1\",\"vost\":\"1\",\"sd\":true,\"hd\":false,\"sdprice\":\"4.99\",\"hdprice\":null,\"sdfile\":null,\"hdfile\":null,\"sdbundle\":\"12699\",\"hdbundle\":\"12699\",\"teaser\":\"23999\",\"att_getter\":\"Saurez-vous r\\u00e9chapper du labyrinthe ?\",\"orig_prod\":null,\"director\":null,\"actors\":null,\"csa\":\"CSA_1\",\"season\":null,\"episode\":null,\"typeid\":\"1\",\"isfav\":false,\"viewersrating\":\"3.5\",\"criticsrating\":\"4.0\",\"onThisPf\":1},{\"id\":\"23688\",\"item_type\":2,\"name\":\"Gone Girl\",\"label\":null,\"desc\":null,\"genre\":null,\"show_name\":null,\"img\":\"http:\\/\\/httpg3.scdn.arkena.com\\/10242\\/v2_images\\/tf1\\/0\\/tf1_media_ingest92895_image\\/tf1_media_ingest92895_image_0_208x277.jpg\",\"url\":\"\\/films\\/media-23688-Gone_Girl.html\",\"duration\":\"2h22mn\",\"durationtime\":\"8579\",\"audio_languages\":null,\"prod\":null,\"year\":null,\"vf\":\"1\",\"vost\":\"1\",\"sd\":true,\"hd\":false,\"sdprice\":\"4.99\",\"hdprice\":null,\"sdfile\":null,\"hdfile\":null,\"sdbundle\":\"12507\",\"hdbundle\":\"12507\",\"teaser\":\"23689\",\"att_getter\":\"Il ne faut pas se fier aux apparences...\",\"orig_prod\":null,\"director\":null,\"actors\":null,\"csa\":\"CSA_2\",\"season\":null,\"episode\":null,\"typeid\":\"1\",\"isfav\":false,\"viewersrating\":\"4.0\",\"criticsrating\":\"4.5\",\"onThisPf\":1}]"
While I try to parse it, I get Unexpected token Parser Error, which I believe is due to the quotes at the beginning and end of the response.
I was wrong to say that the parser error was due to the quotes at the beginning and end of response. But I am not sure why it happens. But when I try to parse the json response array, it does throw error.
Any idea whether there is anything wrong in the json respnse array.
I tried to parse it but it throws parser error. I tried as below
JSON.parse(File.read('demo')). The demo file contains the json
response which I pasted.
First of all, the json you posted is a ruby String. And ruby parses it as json without error. However, if you paste that string into a file, it will not be valid json because of the escape sequences, the most numerous of which is \".
In a ruby string, the sequence \", which is two characters long, is converted to one character; in a file that same sequence is two characters long: a \ and a ". In other words, escape sequences that are legal inside a ruby String do not represent the same thing when pasted into a file.
Another example: in a ruby String the escape sequence \20AC is a single character--the Euro sign. However, if you paste that sequence into a file, it will be five characters long: a \, and a 2, and a 0, and an A, and a C.
Response to comment:
There is an invisible byte order mark (BOM) at the start of the json, which you can see by executing:
p resp
...which produces the output:
\xEF\xBB\xBF[{\"id\":\"2388\" .....
The UTF-8 representation of the BOM is the byte sequence
0xEF,0xBB,0xBF
Byte order has no meaning in UTF-8,[4] so its only use in UTF-8 is to
signal at the start that the text stream is encoded in UTF-8.
You can skip the first 3 bytes/characters like this:
resp[3..-1]
I had this error with reading in JSON files and it turned out that the issue was that JSON.parse somehow did not like UTF-8-encoded files. When I first encoded the files to ASCII (= ISO 8859-1) everything went fine.
Try this. It works.
require 'json'
my_obj = JSON.parse("your json string", :symbolize_names => true)
While trying to print Duplicaci¾n out of a CSV file, I get the following error:
ArgumentError - invalid byte sequence in UTF-8
I'm using Ruby 1.9.3-p362 and opening the file using:
CSV.foreach(fpath, headers: true) do |row|
How can I skip an invalid character without using iconv or str.encode(undef: :replace, invalid: :replace, replace: '')?
I tried answers from the following questions, but nothing worked:
ruby 1.9: invalid byte sequence in UTF-8
Ruby Invalid Byte Sequence in UTF-8
ruby 1.9: invalid byte sequence in UTF-8
This is from the CSV.open documentation:
You must provide a mode with an embedded Encoding designator unless your data is in Encoding::default_external(). CSV will check the Encoding of the underlying IO object (set by the mode you pass) to determine how to parse the data. You may provide a second Encoding to have the data transcoded as it is read just as you can with a normal call to IO::open(). For example, "rb:UTF-32BE:UTF-8" would read UTF-32BE data from the file but transcode it to UTF-8 before CSV parses it.
That applies to any method in CSV that opens a file.
Also start reading in the documentation at the part beginning with:
CSV and Character Encodings (M17n or Multilingualization)
Ruby is expecting UTF-8 but is seeing characters that don't fit. I'd suspect WIN-1252 or ISO-8859-1 or a variant.
I got this SQL statement that I wish to run on SQLite:
INSERT INTO tEntity (name) VALUES ('Roger Café');
Note the é character. Using the SQLite browser, I can insert this statement with the proper encoding.
However, if I save the above statement as a file (my.sql) and then run it on the Windows command line, I am having an encoding problem. The é in Café is garbled up.
C:\somewhere> sqlite3.exe my.db
sqlite> .read my.sql
I'm using Notepad++ to create the file in ANSI encoding. I have tried to use UTF-8 encoding but sqlite3.exe gives me a syntax error while reading the SQL file.
Is there any solution to fix this?
The encoding UTF-8 in notepad++ has BOM, which sqlite3.exe does not know about. Try using UTF-8 without BOM.