I am inserting a record in aerospike. Everything is working fine on my local environment. However, on the staging environment, UTF-8 characters are not being stored correctly.
aql> select msg from test.msges
LIST('["{"message_id":"kxwFZHVBKj","title":"คำถามได้รับà¸à¸²à¸£à¸•à¸à¸šà¸à¸¥à¸±à¸šà¹à¸¥à¹‰à¸§!","actions":
|
+----------------------------------------------------------------------------------------------------------+ 1 row in set (0.038 secs)
Also, this is being done using a golang clinet which is pretty UTF-8 safe i.e. no problems of byte-to-string or string-to-byte conversions and this works fine on local, hence
I am assuming this has something to do with aerospike configuration or version. The aerospike version on my local is 3.14.1.1 and on staging is 3.9.1.1.
One more thing I noticed based on the c-client comments coming on this post is that on my local machine I get the following aql version:
Aerospike Query Client Version 3.13.0.1 C Client Version 4.1.6
But on the remote I have the following:
Aerospike Query Client Version 3.9.1.2
The C-client version is missing !
Anyone can help what configuration (or whatever) am I missing here ?
aql is a tool written in c-client. In c-client, the application is expected to handle utf-8 encoding as there is no "string" datatype similar to newer generation languages that support multibyte utf-8 characters. The application reading data should have prior knowledge that the data is indeed written with utf-8 encoding to handle that. aql cannot determine that as it is a generic tool. So, it ends up printing each byte as a character.
Related
I'm using Firebird 3 with encryption via the AES128 plugin found here. https://www.ibphoenix.com/products/software/encryptionplugin
I have successfully encrypted the database, and I am able to connect and query it using isql however I am not able to connect from within my ColdFusion CFIDE - Data Sources area. I previously used the dbcrypt plugin and it worked fine, it was just tremendously slow compared to the un-encrypted database. With the AES128 plugin though, I get the error
Connection verification failed for data source: devBBL
java.sql.SQLException: Unsupported operation code: 97 The root cause
was that: java.sql.SQLException: Unsupported operation code: 97
My connection information in the CFIDE Data Sources window is the same as it's always been.
JDBC URL: jdbc:firebirdsql:localhost/3050:C:\fbdb\MASTER25.FDB
Driver Class: org.firebirdsql.jdbc.FBDriver
Driver Name: JayBird
Username: SYSDBA
Password: PASSWORD
Does anyone have any idea what is causing me to receive this error when trying to connect? Do I need to include some additional argument?
This happens if the database is encrypted and the plugin requires a callback to get the key (instead of using a server-local encryption key).
If configured to use a callback to obtain the encryption key, during the attach phase, Firebird will send a packet with operation code op_crypt_key_callback (= 97) (and plugin specific data) to the client, and the client in turn should respond to that with the encryption key (or at least, with plugin specific data that the plugin can use to derive the encryption key).
Jaybird versions before 3.0.4 do not support handling of encryption key callbacks (operation code 97), so this error occurs.
Basic support to pass a fixed response to an encryption key callback has been introduced in Jaybird 3.0.4. For earlier versions, if you can't upgrade, the workaround would be to use a server-local encryption key.
Solution
The solution is to upgrade Jaybird to Jaybird 3.0.4, which introduced support for database encryption callbacks. If the encryption plugin performs a callback, but doesn't actually need to use the content of the response, then it will work out of the box.
It the encryption plugin needs a reply with the key, you can set the key in the dbCryptConfig connection property. You can either use a base64 encoded value by prefixing it with base64: or a string key, which will be converted to bytes using UTF-8 encoding.
For example in a connection string:
jdbc:firebirdsql://localhost/appdbalias?dbCryptConfig=base64:dmVyeXNlY3JldGtleQ==
or
jdbc:firebirdsql://localhost/appdbalias?dbCryptConfig=verysecretkey
The implementation currently does not support more advanced callbacks.
Jaybird 3.0.4 (or higher) can be downloaded from https://www.firebirdsql.org/en/jdbc-driver/
See Jaybird 3.0.x release notes, section Database encryption support for more information.
Workaround
NOTE Only use this workaround if you can't yet upgrade to Jaybird 3.0.4 or higher.
According to the instructions on https://www.ibphoenix.com/products/software/encryptionplugin you can configure a server-local key by using KeyHolderPlugin = KeyFile instead of KeyHolderPlugin = Callback.
Unfortunately, this doesn't work. It looks like the plugin in question is unconditionally performing a callback to the client, even if it has the necessary data available locally. This is possibly a bug in this encryption plugin, or maybe a deliberate design decision.
IBPhoenix has released a new version of this plugin, version 1.2.1, which allows you to disable this callback with an explicit setting in plugins/KeyFile.conf with setting DisableCallback = true.
I am using a scraper and uploading my data into redshift using EC2. I would prefer not to upload the data into S3 first. My code is in Jupyter Notebook. However, I get the "String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: 80 (error 3)" error that I see a lot of other people have asked about previously. I even found a page on redshift that walks through using a Remote Desktop. However, as I said before I would prefer not going through S3. Is this possible?
Currently using psycopg2 to connect to the database. I figured it wouldn't work but I tried just putting in acceptinvchars after the database user/password line, and it said that ACCEPTINVCHARS isn't defined.
If you want to copy data to Redshift right from your notebook you have to compose valid INSERT statements and execute them against the existing table in Redshift. However, throughput of this approach is quite low. I don't know how much data you plan to write but I guess scrappers should have higher throughput than that. You can first write the output of your Python script to the same EC2 instance and use COPY command.
More info on copying from an EC2 instance here: COPY from Remote Host (SSH)
As for your error, you likely have accented letters in your input and you need to use LATIN1 encoding everywhere
I have my Sinatra-Redis Blog which had ran pretty well on my computer before I uploaded it on server. But I got the error that saying when I am running it on my server
"Encoding::CompatibilityError - incompatible character encodings: US-ASCII and UTF-8"
I found there are some answer like putting "#encoding:UTF-8" ahead my code, it doesn't work for me
I've tried to put ".force_encoding("utf-8")" after the values in my erb file, it works, but I don't think this is the only way to solve the problem, because I don't want to put ".force_encoding("utf-8")" after all my values.
Ruby version: 2.3.1
redis version:3.0.7
and I am using erb as my templating system.
Thanks in advance!
I have encounter this problem many times. It runs pretty well on my localhost (MacOS) and found to have problem on the server(Debian). It is possibly because your server is using a different encoding setting.
You may want to change the encoding and restart your server.
Reference:
https://www.linuxquestions.org/questions/linux-newbie-8/how-to-change-the-default-encoding-203988/
I faced that issue before, using Liferay POrtal, after I import lar file, any Arabic character and any special character like "♦" ruined to be question mark "?", and when I check Database, i see its stored like that too, I mean "?" .
Its explained here : Import .lar file issue on Encoding with Glassfish as Application Server
adding -DFile.encoding=UTF8 does not helped me.
AND Using Liferay itself with Arabic has no issues (like adding web content), retrieving data from DB is working well.
These days, I'm doing upgrade from liferay 5.2.8 to 6.1.20, i faced that issue again, all Arabic characters touched with upgrade (extracted to temp directory as a temp file then re-inserted to DB) got corrupted..
So its seemed to me this is issue with file system with Glassfish.
I tried Tomcat 7.0 and this issue not happened, and my data survived, but i need to solve it in Glassfish Application Server.
FYI: same issue with Glassfish v2.1.1 , v3.1.2.2 , and ML versions.
Any idea?
Check what kind of encoding your database uses. The typical advice is to use UTF-8 for the database storage as well. Careful: When you check for correct characters in the database, your DB-tools might use a different encoding than the database itself. Sorry for the situation you're in, encoding issues are always a mess.
If you detect that your database uses a non-UTF-8 encoding, you might be lucky with specifying that encoding in the database connection string (depends on whatever database you're using), but you'd be lucky with this. You probably have a bit to repair - scripted or manually.
The -Dfile.encoding is a Oracle JVM specific setting how to read Java source files. This doesn't have any influence on the charset as specified in the Content-Type header of the HTTP response.
You can add the following to the file glassfish-web.xml (located in the WEB-INF folder):
<parameter-encoding default-charset="UTF-8"/>
If you have .jsp files try that in web.xml:
<jsp-config>
<jsp-property-group>
<url-pattern>*.jsp</url-pattern>
<page-encoding>UTF-8</page-encoding>
</jsp-property-group>
</jsp-config>
we are sending data trough MQ from a z/OS/CICS system to an AS400.
Original encoding of the message is CCSID 500 with a MQSTR Format. The client application is getting the message with the CONVERT option and CCSID 819.
Data is almost converted correctly except for the end of line caracter. Any idea?
The z/OS is sending 0D (CR) as end of line caracter. If they had 0D+0A (CR+LF), CCSID automatically change from 500 to 437, and the end of line still ain't right on the client side.
Have already you tried only "0A" (LF)? Have you tried without an expliciet CONVERT? We've never had any issues with this part of the conversion and our machine talks with windows, AIX, Linux and so. Almost all the time we do rely on automatic conversions.
Also, you may want to check the USER of your job that picks the MQ message from the Queue. Most of the time the USER uses the system attributes (as defined in the appropriate SYSVAL). Maybe this userprofile has been configured very specific and different from the SYSVAL which stops the iSeries in doing the correct automatic conversion.
As a simple test, FTP your data to the iSeries. That will also force an automatic conversion that uses the system tables (WRKTBL). That may show an error in that configuration.
We had a problem with the CCSID of the queue manager not being set correctly and the wrong conversion table used.