I have a requirement of inserting greek characters, such as 'ϕ', into Oracle. My existing DB structure wasn't supporting it. On investigating found various solutions and I adopted the solution of using NCLOB instead of CLOB. It works perfectly fine when I use unicode, 03A6, for 'ϕ', and use UNISTR function in SQL editor to insert. Like the one below.
UPDATE config set CLOB = UNISTR('\03A6')
However, it fails when I try to insert the character through my application, using hibernate. On debugging, I find that the string before inserting is '\u03A6'. After insert, I see it as ¿.
Can some one please help me how I can resolve this? How do I make use of UNISTR?
PN: I don't use any native sqls or hqls. I use entity object of the table.
Edit:
Hibernate version used is 3.5.6. Cannot change the version as there are so many other plugins dependent on this. Thus, cannot use #Nationalized or #Type(type="org.hibernate.type.NClobType") on my field in Hibernate Entity
After racking my brain on different articles and trying so many options, I finally decided to tweak my code a bit to handle this in Java and not through hibernate or Oracle DB.
Before inserting into the DB, I identify unicode characters in the string and format it to append &#x in the beginning and ; at the end. This encoded string will be displayed with its actual unicode in the UI (JSP, HTML) that is UTF-8 compliant. Below is the code.
Formatter formatter = new Formatter();
for (char c : text.toCharArray()) {
if (CharUtils.isAscii(c)) {
formatter.format("%c", c);
} else {
formatter.format("&#x%04x;", (int) c);
}
}
String result = formatter.toString();
Ex:
String test = "ABC \uf06c DEF \uf06cGHI";
will be formatted to
test = "ABC DEF GHI";
This string when rendered in UI (and even in word doc) it displays it as
ABC DEF GHI
I tried it with various unicode characters and it works fine.
Related
I am trying to create a report in BI. I have created a data model using a sql script and the data type is string. I then created a report using this data model and the sample data is showing as 06185 as shown below.
But when i run the report the output is showing as 6185.0 with decimal places.
I have tried to change the format by double clicking the value. But I cant choose any format except none in the report designer. What I need is the values to be displayed same as the sample data.
06185 is not a number - it's a string. As a number you will never have leading 0s. 6185 is a number.
So first of all you need to find out what your data is really supposed to be. "06185" = string. 6185 = integer. 6185.0 = double.
To be quick simply add ' character next to the number character... Else excel will make your life difficult specially if you have bank account numbers...
SELECT payees.supplier_site_id csupplier_site_id,
payees.org_id c_org_id,
payees.payee_party_id ,
cbbv.bank_name,
max(''''||to_char(ieb.bank_account_num)) as bank_account_num
FROM apps.iby_pmt_instr_uses_all instrument,
apps.iby_account_owners owners,
apps.iby_external_payees_all payees,
apps.iby_ext_bank_accounts ieb,
apps.ap_supplier_sites_all asa,
apps.ap_suppliers asp,
apps.ce_bank_branches_v cbbv
WHERE owners.primary_flag = 'Y'
AND owners.ext_bank_account_id = ieb.ext_bank_account_id
AND owners.ext_bank_account_id = instrument.instrument_id
AND payees.ext_payee_id = instrument.ext_pmt_party_id
AND payees.payee_party_id = owners.account_owner_party_id
AND payees.supplier_site_id = asa.vendor_site_id
AND asa.vendor_id = asp.vendor_id
and bank_party_id(+) = ieb.bank_id
--AND cbbv.branch_party_id(+) = ieb.branch_id
AND (
(INSTRUMENT.END_DATE IS NULL)
OR
( INSTRUMENT.END_DATE >= TO_DATE(SYSDATE) )
)
AND (
(IEB.END_DATE IS NULL)
OR
(IEB.END_DATE >= TO_DATE(SYSDATE))
)
group by payees.supplier_site_id ,
payees.org_id ,
payees.payee_party_id ,
cbbv.bank_name
You may have to edit this with the desktop Microsoft Word add-in. There's limited capabilities with the Oracle BI Publisher Stand-alone running on the server.
You have 100% control if you're editing the template in Word. Unfortunately, there's limited export capability to export your existing template. Oracle BI publisher stand-alone will export an XML file, which to my knowledge is not visually editable in Word. There is an ability to import an RTF file which you created in Word, though. So Install the BI Publisher desktop/Word add-in. Make a new file, save as .rtf format, use the tool to import an example XML file, add a table, and continue editing as you need to. If you need help with this, there's plenty of documentation out there to get you started. Also, there's a Template Viewer application that's also installed that will let you process the XML sample/data file and rtf file into Excel, PDF, or whichever you prefer.
I have an MDB file which contains a number of tables and forms. Each field has a validation rule such as Is Null Or >=0 And <=255.
This access database is being converted into an online system using MySQL. Exporting all the data is easy using MDBTools (https://github.com/brianb/mdbtools).
However I can't find any way of exporting the validation rules. There are thousands of fields across over 100 tables so it's going to be important to export and import them rather than rewrite each one.
I don't really mind what format they're exported in, any sort of text format so I could do a regular expression or something will be fine.
However I haven't been able to find any information anywhere on exporting these validation rules.
Perhaps if it's not built into access by default then a VB script could be used to find the info and write it to a text file? I'm not really familiar with access or windows at all so if anyone could suggest if that was a possibility that would be great.
Using VBA allows you to retrieve field validation rules directly.
I realize it's probably too late to help you now. And, although it may not seem appropriate for someone unfamiliar with Access and VBA, this approach requires only a table, copying the code below into a standard module, and running it. So someone else may benefit.
I created my table, field_validation_rules, to store the text of the validation rule properties. The table includes 3 text fields: table_name; field_name; and validation_rule.
Public Sub GatherValidationRules()
Dim db As DAO.Database
Dim fld As DAO.Field
Dim rs As DAO.Recordset
Dim tdf As DAO.TableDef
Set db = CurrentDb
Set rs = db.OpenRecordset("field_validation_rules", dbOpenTable, dbAppendOnly)
For Each tdf In db.TableDefs
If Not (tdf.Name Like "~*" Or tdf.Name Like "MSys*") Then
For Each fld In tdf.Fields
If Len(fld.ValidationRule) > 0 Then
rs.AddNew
rs!table_name.Value = tdf.Name
rs!field_name.Value = fld.Name
rs!validation_rule.Value = fld.ValidationRule
rs.Update
End If
Next
End If
Next
rs.Close
End Sub
The ValidationRule property is a string value. If the property has not been assigned for a given field, ValidationRule is an empty string. The code skips those, storing only validation rules for fields which have them assigned.
If you want the collected validation rules in a text file, there a several options. I dumped mine to CSV like this:
DoCmd.TransferText acExportDelim, , "field_validation_rules", "C:\share\Access\field_validation_rules.txt", False
To anyone else finding this, this is how I wound up doing it. This was in Access 2003, it may be different in other versions.
First I went to Tools > Analyze > Documenter selected the table I wanted and used these settings:
I was then presented with what looked like a pdf or word doc (I don't think it is, but it doesn't really matter).
I then did File > Export and selected "Text Files .txt" and saved it to a location on my computer.
I then opened the .txt file in PHP (anywhere you can do regular expressions should be fine).
In my instance not every single field had validation rules and the validation rules did not appear if they were not set, which meant a regular expression to fetch the fieldID had more results than to fetch the validation rules.
So I used two regular expressions.
/SourceField:\s+(\S+).*?AllowZeroLength/msi
This gets everything betwenen SourceField and AllowZeroLength. AllowZeroLength is the first bit of repeating text after the validation rules.
I then used this regular expression to get the validation rules from within that string.
/ValidationRule:\s+(.*)\\r/
I had to use \r instead of new line, probably something to do with moving it from Windows to Ubuntu.
In PHP it looked like this:
<?php
$file_contents = file_get_contents('validations.txt');
$response = [];
preg_match_all('/SourceField:\s+(\S+).*?AllowZeroLength/msi', $file_contents, $matches);
for($i = 0; $i < count($matches[0]); $i++) {
$id = $matches[1][$i];
preg_match('/ValidationRule:\s+(.*)\\r/', $matches[0][$i], $validation_match);
$response[$id] = $validation_match[1] ?? null;
}
There is almost certainly a cleaner regular expression than this, but this was incredibly quick and I got exactly what I wanted.
I would like to ask if a windows allowed filename is secure for SQL (Oracle and general), JS, xml, etc injections.
The following characters are checked and replaced: /:*?"<>| (also with their ascii values etc).
Also the length could not be more than ~180 characters
The checks are made securely, both in client (only for usability) and server side (for security)
The filename is saved to an oracle DB and it has a flow like oracle->java->xml->xslt->browser to be displayed.
EDIT: #Bohemian 's answer points out the insecure part of SQL injection (which could simply be prohibited with the use of prepared statements). How about JS or xml injection? (That could lead to XSS or access to hdd through xslt)
No, it isn't safe. Because windows allows the single quote ' and the semi colon ; in file names.
That combination is all you need to inject SQL:
SQL injection example (in java):
int userid = 999;
String filename = "foo';delete from users;update users set name = 'bar";
String sql = "update users set avatar = '" + filename + "' where id = " + userid;
System.out.println(sql);
Output:
update users set avatar = 'foo';delete from users;update users set name = 'bar' where id = 999
This is valid, and pernicious, SQL.
Windows also allows both curly {} and square [] brackets in filenames. I can't think of an example right now, but it seems that a javascript injection would be possible too, perhaps if passed to eval(). (Perhaps someone could provide a working example - feel free to edit one in here)
When I run the following .Net code:
using (var c = Shared.DataSources.BSS1.CreateCommand())
{
c.CommandText = "\r\nSelect c1, c2, c3, rowid \r\nFrom someSpecificTable \r\nWhere c3 = :p0";
var p = c.CreateParameter() as Oracle.DataAccess.Client.OracleParameter;
c.Parameters.Add(p);
p.OracleDbType = Oracle.DataAccess.Client.OracleDbType.Varchar2;
p.DbType = System.Data.DbType.AnsiString;
p.Size = 20;
p.Value = "007";
p.ParameterName = ":p0";
using (var r = c.ExecuteReader())
{
r.Read();
}
}
I get the following error:
ORA-01460: unimplemented or unreasonable conversion requested
ORA-02063: preceding line from XXX
This is not my database, and I don't have control over the select statements that I get, that table IS from a database link.
The funny thing is that if I add the following code just before the ExecuteReader it runs fine.
c.CommandText = c.CommandText.Replace("\r\n", " ");
Unfortunately that is not a good solution in my case as I can't control to SQL nore can I change it that way.
As for the table itself, the columns are:
c1 Number(5)
c2 varchar2(40)
c3 varchar2(20).
I know that ORA-02063 that comes after indicate something about a database link, but I looked in the synonim table and it didn't come from any database_link, and also I don't think that \r\n should affect database link.
I tried running the query without bound parameters, and it did work - but again bad practice to do so in a general term.
The trouble is that a competing tool that is not .Net based, is working and thus it's not a general problem.
I also couldn't reproduce the problem in my own environment, this is a customer database and site.
I am using instant client 11.1.6.20 and also tested it with instant client 11.2.3.0
The db is 10 and the db link is to an oracle v8 database
Any help would be appreciated
This problem can be recreated with straight forward steps. That is, any SQL query having a string literal, in where clause, more than 4000 characters in length gives an error "ORA-01704: string literal too long"
But, when the same query is executed through JDBC it gives "ORA-01460: unimplemented or unreasonable conversion requested"
Finally I found the answer!!!
After investigating and reflecting into the code I found that by changing the Direction of the Parameter to input output - the problem was resolved.
p.Direction = ParameterDirection.InputOutput;
After much investigation I found out that it's all about the fact that we have bound parameters that are used from ODP.NET and targeting tables from a DBLINK to a V8 Oracle server.
Once I eliminated the bound parameters it all worked.
It was a while back, but I think it's had something to do with varying string lengths of the strings sent to the bound parameter.
It seems that it ignored the size property, so if in the first query I sent a string with length 10 and in the second string I sent a string with length 12, I'll get that error.
I also found the oracle articles about it :
https://community.oracle.com/thread/2460796?tstart=0
and the patch for it:
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=745005.1
But - I found a fix in my code that actually solved it - see my next answer.
Hope this helps anyone.
The accepted answer didn't work for me. However, after reading the attached links, I applied the following – although it does involve editing the SQL.
In my case, I knew the maximum left of the bind variable (the length reducing after the first call is what causes the issue). So I padded the .NET string, and added a TRIM in the SQL. Following your example:
c.CommandText = "\r\nSelect c1, c2, c3, rowid \r\nFrom someSpecificTable \r\nWhere c3 = TRIM(:p0)";
...
p.Value = "007".PadRight(10);
I use the ruby-gem sequel to read utf-8-encoded data from a MSSQL-Server table.
The fields of the table are defined as nvarchar, they look correct in the Microsoft Server Management Studio (Cyrillic is Cyrillic, Chinese looks chinese).
I connect my database with
db = Sequel.connect(
:adapter=>'ado',
:host =>connectiondata[:server],
:database=>connectiondata[:dsn],
#Login via SSO
)
sel = db[:TEXTE].filter(:language=> 'EN')
sel.each{|data|
data.each{|key, val|
puts "#{val.encoding}: #{val.inspect}" #-> CP850: ....
puts val.encode('utf-8')
}
}
This works fine for English, German returns also a useable result:
CP850: "(2 St\x81ck) f\x81r
(2 Stück) für ...
But the result is converted to CP850, it is not the original UTF-8.
Cyrillic languages (I tested with Bulgarian) and Chinese produce only '?'
(reasonable, because CP850 doesn't include Chinese and Bulgarian characters).
I also connected via a odbc-connection:
db = Sequel.odbc(odbckey,
:db_type => 'mssql', #necessary
#:encoding => 'utf-8', #Only MySQL-Adapter
)
The result is ASCII-8BIT, I have to convert the data with force_encoding to CP1252 (not CP850!).
But Cyrillic and Chinese is still not possible.
What I tried already:
The MySQL-adapter seems to have an encoding option, with MSSQL I detected no effect.
I did similar tests with sqlite and sequel and I had no problem with unicode.
I installed SQLNCLI10.dll and used it as provider. But I get a Invalid connection string attribute-error (same with sqlncli).
So my closing question: How can I read UTF-8 data in MS-SQL via ruby and sequel?
My environment:
Client:
Windows 7
Ruby 1.9.2
sequel-3.33.0
Database:
SQL Server 2005
Database has collation Latin1_General_CI_AS
After preparing my question I found a solution. I will post it as an answer.
But I still hope, there is a better way.
If you can avoid it, you really don't want to use the ado adapter (it's OK for read-only workloads, but I wouldn't recommend it for other workloads). I would try the tinytds adapter, as I believe that will handle encodings properly, and it defaults to UTF-8.
Sequel itself does not do any transcoding, it leaves the handling of encodings to the lower level driver.
After preparing my question I found a solution on my own.
When I add a
Encoding.default_external='utf-8'
to my code, I get the correct results.
As a side effect each File.open expects now also UTF-8-encoded files (This can be overwritten by additional parameters in File.open).
As an alternative, this works also:
Encoding.default_internal='utf-8'
As I mentioned in my question, I don't like to change global settings, only to change the behaviour of one interface.
So I still hope on a better solution.