I was doing some tests, where I inserted some records of java bigDecimal to a varchar2 column in Oracle.
What I wanted to do was insert java bigDecimal to number column in Oracle.
I am wondering how the 2 work differently and what interim conversion steps does Oracle take in the scenarios.
BigDecimal =>Varchar2 column
BigDecimal=>Number column
Can I still use the findings from my previous tests. I am mostly looking at latency, throughput etc.
Remember the golden rule: You should never ever under no circumstances store numbers in varchar columns.
Storing numbers in character columns will give you a lot of trouble in the long run.
Always store numbers as numbers.
To store the numbers, use a PreparedStatement and use the setBigDecimal() method to send the number to the database. This will take care of any conversion and will guarantee that the correct value is stored in the database and you don't have to worry e.g. about different decimal separators in different locales when sending a number as a string to the database.
I did not find any measurable performace difference. This was just a test of a prototype so I can use the results.
Related
So we have in our data schema column of type NUMBER(15). How to map it in hibernate without changing schema? int data type is too small for that, long data type is too big for that(ORA-01438), even when persisting numbers withing DB column bounds.
Assume, that schema is unmodifiable and we do not want to use BigInteger. Say, that we know, that number wont be bigger than 372036854775807, thus can fit into both long and NUMBER(15).
Oracle DB.
so actually this was my fault, due to misread of oracle documentation, regarding number scale and precision. I thought, that precision is the part before '.', and scale after it, but no, precision is number of all digits. That was causing my problem, and on different column.
To answer my own question: ORA-01438 won't be thrown for numbers which fit into type used in DB layer. Therefore if I know, that data type used on app layer cannot be bigger than number(15), I can safely use long in java and NUMBER(15) in DB. Until this constraint is violated, everything is fine. And adhering to this constraint is important in first place, and BigInteger won't help without it, since BigInteger can be bigger than NUMBER(37).
I have 2 timestamp columns stored in an HDFS that I can access through Impala, hive, etc...
The timestamps that I need to compare may look like this example:
2014-04-08 00:23:21.687000000
2014-04-08 00:23:21.620000000
With differences in the milliseconds, and need to build a new column that in this example should have a value of 0.067000
I've tried using impala's built in time functions but none of them seem to make the cut.
I've tried:
casting the string to a timestamp and then substracting the 2 values. This returns an error "AnalysisException: Arithmetic operation requires numeric operands"
using the unix_timestamp function. This truncates the values to an int that represent seconds, so subsecond values are lost.
While writting this question I found the answer :)
The way to do it was using a double cast.
Cast(cast(time_stamp) as timestamp) as double)
this makes the times_stamp into a number without truncating sub-second values.
Once there it becomes a trivial arithmetic operation.
I'm working with an SQLite DB where all columns are of NVARCHAR data type.
Coming from MS SQL background I know that NVARCHAR has additional baggage associated with it, my first impulse is to refactor most column types to have concrete string lengths enforced (most are under 50 chars long).
But at the same time I know that SQLite treats things a bit differently.
So my question is should I change/refactor the column types? And by doing so is there anything to gain in terms of disk space or performance in SQLite?
DB runs on Android/iOS devices.
Thanks!
You should read https://www.sqlite.org/datatype3.html.
CHARACTER(20), VARCHAR(255), VARYING CHARACTER(255), NCHAR(55)
NATIVE CHARACTER(70), NVARCHAR(100), TEXT and CLOB are treated as TEXT.
Also SQLite does not enforce lengths.
So instead of digging through critic documentation I did a bit of experimenting with column types.
Database I'm working with has well over 1 million records, with most columns as NVARCHAR, so any change on column datatypes was easily seen in file size deltas.
Here are the results I found in effort to reduce DB size:
NVARCHAR:
Biggest savings came from switching column types where possible from NVARCHAR to plain INT or FLOAT. On a DB file of 80MB savings were in Megabytes, very noticable. With some additional refactoring I dropped the size down to 47MB.
NVARCHAR vs. VARCHAR:
Made very little difference, perhaps a few KBs on a DB of a size of 80MBs
NVARCHAR vs. OTHER String Types:
Switching between various string based types made almost no difference, as documentation points out all string types are stored all the same in SQLite, as TEXT
INT vs OTHER numerics
No Difference here, SQLite stores all as NUMBER in the end.
Indexes based on NVARCHAR columns also took up more space, once re-indexed on INT columns I shedded a few MBs
I've been developing a DotNet project on oracle (Ver 10.2) for the last couple of months and was using Varchar2 for my string data fields. This was fine and when navigating the project page refreshes were never more than a half second if even (it's quiet a data intensive project). The data is referenced from 2 different schemas, one a centralised store of data and one of which is my own. Now the centralised schema will be changing to be unicode compliant (but hasn't yet) so all Varchar2 fields will become NVarchar2, in preparation for this I changed all the fields in my schema to be NVarchar2 and since then performance has been horrible .. up to 30/40 second page refreshes.
Could this be because Varchar2 fields in the centralised schema will be joined against NVarchar2 fields in my schema on some stored procedures. I know NVarchar2 is twice the size of Varchar2 but that wouldn't explain the sudden massive change. As I said any tips for what to look for to improve would be great, if I haven't explained the scenario well enough do ask for more information.
Firstly, do a
select * from v$nls_parameters where parameter like '%SET%';
Character sets can be complicated. You can have single-byte charactersets, fixed-size multibyte character set sand variable-sized multi-byte character sets. See the unicode descriptions here
Secondly, if you are joining a string in a single-byte characterset to a string in a two-byte characters set, you have a choice. You can do a binary/byte comparison (which generally won't match anything if you compare between a single-byte character set and a two-byte characterset). Or you can do a linguistic comparison, which will generally mean some CPU cost, as one value is converted into another, and often the failure to use an index.
Indexes are ordered, A,B,C etc. But a character like Ä may fall in different places depending on the Linguistic order. Say the index structure puts Ä between A and B. But then you do a linguistic comparison. The language of that comparison may put Ä after Z, in which case the index can't be used. (Remember your condition could be a BETWEEN rather than an = ).
In short, you'll need a lot of preparation, both in your schema and the central store, to enable efficient joins between different charactersets.
It is difficult to say anything based on what you have provided. Did you manage to check if the estimated cardinalities and/or explain plan changed when you changed the datatype to NVARCHAR2? You may want to read the following blog post to see if you can find a lead
http://joze-senegacnik.blogspot.com/2009/12/cbo-oddities-in-determing-selectivity.html
It is likely no longer able to use indexes that it previously could. As Narendra suggests check the explain plan to see what changed. It is possible that once the centeralized store is changed the indexes will again be usable. I suggest testing that path.
Setting the NLS_LANG initialization parameter properly is essential to proper data conversion. The character set that is specified by the NLS_LANG initialization parameter should reflect the setting for the client operating system. Setting NLS_LANG correctly enables proper conversion from the client operating system code page to the database character set. When these settings are the same, Oracle assumes that the data being sent or received is encoded in the same character set as the database character set, so no validation or conversion is performed. This can lead to corrupt data if conversions are necessary.
Does the SCN itself encode a timestamp or is it a lookup from some table.
From an AskTom post he explains that the timestamp to +/-3seconds is stored in raw field in smon_scn_time. IS that where the function is going to get the value?
If so, when is that table purged if ever? If so, what triggers that purge?
If it is, does that make it impossible to translate old SCN's to Timestamps?
If it's impossible, then it eliminates any uses of that field that are long term things (read: auditing).
If I put that function in a query, would joining to that table be faster?
If so, anyone know how to covert that Raw column?
The SCN does not encode a time value. I believe it is an autoincrementing number.
I would guess that SMON is inserting a row into SMON_SCN_TIME (or whatever table underlies it) every time it increments the SCN, including the current timestamp.
I queried for the minimum recorded timestamp in several databases and they all go back about 5 days and have a little under 1500 rows in the table. So it is less than the instance lifetime.
I imagine the lower bound on how long the data is kept might be determined by the DB_FLASHBACK_RETENTION_TARGET parameter, which defaults to 1 day.
I would recommend using the function, they've probably provided it so they can change the internals at will.
No idea what the RAW column TIM_SCN_MAP contains, but the TIME_DP and SCN column would appear to give you the mapping.