So, I am trying to build a query in RMAN Catalogue ( using RC_RMAN_BACKUP_JOB_DETAILS) to compare the most recent backup duration (TIME_TAKEN_DISPLAY) for each database (DB_NAME) with its historical average AVG backup duration (TIME_TAKEN_DISPLAY).
How do I convert TIME_TAKEN_DISPLAY(timestamp; HH:MM:SS), i.e. in VARCHAR2 Format to a minute format, i.e number only, so as to run the query against the entire RC_RMAN_BACKUP_JOB_DETAILS to compare AVG time taken in past with time takes for last backup for each DB.
One thing that may work is converting String(Time_taken_display)->To_TIME(Time_taken_display in Time format)->TO_NUM(Time_taken_display in minutes in number format), but this will be so highly inefficient.
The solution can be pretty simple and complex depending on the requirements:
One simple solution is:
select avg(substr(TIME_TAKEN_DISPLAY, 0,2)*60 + substr(TIME_TAKEN_DISPLAY, 4,2) + substr(TIME_TAKEN_DISPLAY, 7,2)/60) from RC_RMAN_BACKUP_JOB_DETAILS;
Using Type Casting Functions:
Cast TIME_TAKEN_DISPLAY into time format using TO_TIMESTAMP and then cast to TO_NUMBER, but I did not want to take this approach as I plan to run my scripts against all databases logged in the view, and multiple casting will leave the performance highly inefficient.
But as per #alex Poole comment, I will be using ENLAPSED_SECONDS field as it is readily available in seconds and number data type.
Related
I've got a 3GB SQLite database file with a single table with 40 million rows and 14 fields (mostly integers and very short strings and one longer string), no indexes or keys or other constraints -- so really nothing fancy. I want to check if there are entries where a specific integer field has a specific value. So of course I'm using
SELECT EXISTS(SELECT 1 FROM FooTable WHERE barField=?)
I haven't got much experience with SQLite and databases in general and on my first test query, I was shocked that this simple query took about 30 seconds. Subsequent tests showed that it is much faster if a matching row occurs at the beginning, which of course makes sense.
Now I'm thinking of doing an initial SELECT DISTINCT barField FROM FooTable at application startup, and caching the results in software. But I'm sure there must be a cleaner SQLite way to do this, I mean, that should be part of what a DBMS's job right?
But so far, I've only created primary keys for speeding up queries, which doesn't work here because the field values are non-unique. So how can I speed up this query so that it works at constant time? (It doesn't have to be lightning fast, I'd be completely fine if it was under one second.)
Thanks in advance for answering!
P.S. Oh, and there will be about 500K new rows every month for an indefinite period of time, and it would be great if that doesn't significantly increase query time.
Adding an index on barField should speed up the subquery inside the EXISTS clause:
CREATE INDEX barIdx ON FooTable (barField);
To satisfy the query, SQLite would only have to seek the index once and detect that there is at least one matching value.
ClickHouse doesn't support, yet, DateTime with milliseconds.
I saw two possible suggestion regarding fields like: 2019-03-17T14:00:32.296Z
multiply by 100 an store it in UInt32/64. How do I use the multiply by 100 and store as UInt32?
to store milliseconds separately. Is there a way to remove milliseconds from 2019-03-17T14:00:32.296Z => 2019-03-17 14:00:32?
Thanks for your help!
Should use the datetime64 type - https://clickhouse.com/docs/en/sql-reference/data-types/datetime64/
In my mind, the main idea, why ClickHouse does not support milliseconds in DateTime is worse compression.
Long story short: use DateTime and precession by seconds. If you want to store milliseconds, you can go ahead with two ways:
Store milliseconds separately, so you will have a DateTime with your date, that you could use in all possible DateTime functions, as well as primary keys. And put milliseconds part in separate column with type UInt16. You have to prepare data separately before storing. Depends on what language do you use for preprocess data before storing, it could be different ways to do it. In golang it could be done:
time.Now().UnixNano() / 1e6 % 1e3
Another way, is to store whole as timestamp. This means you should convert your date to unix timestamp with milliseconds by your own and put it into ClickHouse as Uint64. It also depends on what do you use for preparing inserts. For golang it could like:
time.Now().UnixNano() / 1e6
I have 2 timestamp columns stored in an HDFS that I can access through Impala, hive, etc...
The timestamps that I need to compare may look like this example:
2014-04-08 00:23:21.687000000
2014-04-08 00:23:21.620000000
With differences in the milliseconds, and need to build a new column that in this example should have a value of 0.067000
I've tried using impala's built in time functions but none of them seem to make the cut.
I've tried:
casting the string to a timestamp and then substracting the 2 values. This returns an error "AnalysisException: Arithmetic operation requires numeric operands"
using the unix_timestamp function. This truncates the values to an int that represent seconds, so subsecond values are lost.
While writting this question I found the answer :)
The way to do it was using a double cast.
Cast(cast(time_stamp) as timestamp) as double)
this makes the times_stamp into a number without truncating sub-second values.
Once there it becomes a trivial arithmetic operation.
I've writing this thread as I've fought this problem for three whole days now!
Basically, I have a program that collects a big CSV-file and uses that as input to a local SQLCE-database.
For every row in this CSV-file (which represents some sort of object, lets call it "dog"), I need to know whether this dog already exists in the database.
If it already exists, don't add it to the database.
If it doesn't exists, add a new row in the database.
The problem is, every query takes around 60 milliseconds (in the beginning, when the database is empty) and it goes up to about 80ms when the database is around 1000 rows big.
When I have to go thru 1000 rows (which in my opinion is not much), this takes around 70000 ms = 1 minute and 10 seconds (just to check if the database is up to date), way too slow! Considering this amount will probably some day be more than 10000 rows, I cannot expect my user to wait for over 10 minutes before his DB is synchronized.
I've tried to use the compiled query instead, but that does not improve performance.
The field which im searching for is a string (which is the primary key), and it's indexed.
If it's necessary, I can update this thread with code so you can see what I do.
SQL CE on Windows Phone isn't the fastest of creatures but you can optimise it:
This article covers a number of things you can do:WP7 Local DB Best Practices
They als provide a WP7 project that can be downloaded so you can play with the code.
On top of this article I'd suggest changing your PK from a string to an int; strings take up more space than ints so your index will be larger and take more time to load from isolated storage. Certainly in SQL Server searchs of strings are slower than searches of ints/longs.
Does the SCN itself encode a timestamp or is it a lookup from some table.
From an AskTom post he explains that the timestamp to +/-3seconds is stored in raw field in smon_scn_time. IS that where the function is going to get the value?
If so, when is that table purged if ever? If so, what triggers that purge?
If it is, does that make it impossible to translate old SCN's to Timestamps?
If it's impossible, then it eliminates any uses of that field that are long term things (read: auditing).
If I put that function in a query, would joining to that table be faster?
If so, anyone know how to covert that Raw column?
The SCN does not encode a time value. I believe it is an autoincrementing number.
I would guess that SMON is inserting a row into SMON_SCN_TIME (or whatever table underlies it) every time it increments the SCN, including the current timestamp.
I queried for the minimum recorded timestamp in several databases and they all go back about 5 days and have a little under 1500 rows in the table. So it is less than the instance lifetime.
I imagine the lower bound on how long the data is kept might be determined by the DB_FLASHBACK_RETENTION_TARGET parameter, which defaults to 1 day.
I would recommend using the function, they've probably provided it so they can change the internals at will.
No idea what the RAW column TIM_SCN_MAP contains, but the TIME_DP and SCN column would appear to give you the mapping.