Does Oracle NoSQL provide a function allowing to generate the system timestamp for a column - oracle-nosql

does Oracle NoSQL provide SDK API or sql build-in function which can automatically generate the system timestamp for a column?
We have a column called timeUpdated which keep the time when any data updated in the same row, we can pass the system time when updating that row, but it could be a little gap with the time when the data is committed at database finally. The timestamp passed by will be the local time of our application servers

you can use current_time() - Returns the current time in UTC, as a timestamp value with millisecond precision.
sql-> create table TEST ( a integer, b timestamp(3), primary key (a));
Statement completed successfully
sql-> insert into TEST values (1, current_time());
{"NumRowsInserted":1}
1 row returned
sql-> select * from TEST;
{"a":1,"b":"2021-06-15T09:52:01.561Z"}
sql-> update TEST set b = current_time() where a =1;
{"NumRowsUpdated":1}
1 row returned
sql-> select * from ping;
{"a":1,"b":"2021-06-15T09:53:24.101Z"}
But you can also use the modification_time function allows you to see the most recent modification time (in UTC) of a row.
sql-> select id, firstname, lastname, modification_time($u) FROM users $u ;
{"id":4,"firstname":"Peter","lastname":"Smith","Column_4":"2021-06-08T08:27:19.640Z"}
{"id":2,"firstname":"John","lastname":"Anderson","Column_4":"2021-06-08T08:27:19.624Z"}
{"id":5,"firstname":"Dana","lastname":"Scully","Column_4":"2021-06-08T08:27:19.644Z"}
{"id":1,"firstname":"David","lastname":"Morrison","Column_4":"2021-06-08T08:27:19.613Z"}
{"id":3,"firstname":"John","lastname":"Morgan","Column_4":"2021-06-08T08:27:19.630Z"}
Here the table, this table does not have a timestamp column
sql-> desc as json table users;
{
"json_version" : 1,
"type" : "table",
"name" : "Users",
"fields" : [{
"name" : "id",
"type" : "INTEGER",
"nullable" : false
}, {
"name" : "firstname",
"type" : "STRING",
"nullable" : true
}, {
"name" : "lastname",
"type" : "STRING",
"nullable" : true
}, {
"name" : "income",
"type" : "INTEGER",
"nullable" : true
}],
"primaryKey" : ["id"],
"shardKey" : ["id"]
}
In this case, this is the NoSQL system that is managing the date but it seems that in your use-case, you want to manage your own column

Related

Cosmos DB Collection not using _id index when querying by _id?

I have a CosmosDb - MongoDb collection that I'm using purely as a key/value store for arbitrary data where the _id is the key for my collection.
When I run the query below:
globaldb:PRIMARY> db.FieldData.find({_id : new BinData(3, "xIAPpVWVkEaspHxRbLjaRA==")}).explain(true)
I get this result:
{
"_t" : "ExplainResponse",
"ok" : 1,
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "data.FieldData",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [ ]
},
"winningPlan" : {
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1,
"executionTimeMillis" : 106,
"totalKeysExamined" : 0,
"totalDocsExamined" : 3571,
"executionStages" : {
},
"allPlansExecution" : [ ]
},
"serverInfo" : #REMOVED#
}
Notice that the totalKeysExamined is 0 and the totalDocsExamined is 3571 and the query took over 106ms. If i run without .explain() it does find the document.
I would have expected this query to be lightning quick given that the _id field is automatically indexed as a unique primary key on the collection. As this collection grows in size, I only expect this problem to get worse.
I'm definitely not understanding something about the index and how it works here. Any help would be most appreciated.
Thanks!

Not getting incremental data with jdbc importer from sql to elastic search

As per jdbc importer :
It is recommended to use timestamps in UTC for synchronization. This example fetches all product rows which has added since the last run, using a millisecond resolution column mytimestamp:
{
"type" : "jdbc",
"jdbc" : {
"url" : "jdbc:mysql://localhost:3306/test",
"user" : "",
"password" : "",
"sql" : [
{
"statement" : "select * from \"products\" where \"mytimestamp\" > ?",
"parameter" : [ "$metrics.lastexecutionstart" ]
}
],
"index" : "my_jdbc_index",
"type" : "my_jdbc_type"
}
}
I want to input data incrementally based on a column modified data whose format is 2015-08-20 14:52:09 also i use a scheduler which runs every minute . I tried with the value of sql key as
"statement" : "select * from \"products\" where \"modifiedDate\" > ?",
But data was not loaded.
Am I missing out something ?
the format of lastexecutionstart like this "2016-03-27T06:37:09.165Z".
it contain 'T' and 'Z' . So that is why your data was not loaded.
If you want to know more.
here is link
https://github.com/jprante/elasticsearch-jdbc

Delete records not updating in elastic search river plugin

In elastic search river , if i deleted a record in mysql , its still showing in index . I have enabled auto-commit also . How make mysql and elastic search in sync and also how to make delta-imports in elastic ?
{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/testrivet",
"user" : "root",
"password" : "Gemini*123",
"sql" : [
{
"statement" : "select *,empid as _id from empdata"
}
],
"strategy" : "simple",
"schedule" : "0 0-59 0-23 ? * *",
"autocommit" : true,
"metrics": {enabled:true}
},
"index" : {
"autocommit":true
}
}
Indeed, if a record is deleted from your database, there's no way your JDBC river will be able to retrieve it anymore in order to delete the corresponding record in ES.
An alternative is to "soft-delete" records from your database by setting a flag (i.e. a new boolean column). The flag would be true when the record is active and false when the record is deleted. That way when your import process runs, you'd get all records and based on that flag you know you have to delete the documents from Elasticsearch.
There are other ways but they involve adding another component to the mix, so if this would do the job I'd suggest doing like that.

ElasticSearch Fields Mapping to String by default when indexing

Let me first explain my scenario.
I am fetching data from RDBMS and pushing it into ElasticSearch.
Fetched Results are in the form of List and i am preparing bulk index request like this:
BulkRequestBuilder bulkRequest = client.prepareBulk();
for (Map<String,Object> singleDataRow : ResultSet)
{
IndexRequest indexRequest = new IndexRequest("testindex","testtype",singleDataRow.getObject("NAME"));
bulkRequest.add(indexRequest);
}
bulkRequest.execute().actionGet();
My Map = includes Map of string to string, string to big decimal, string to big integer etc.
eg.
{ BIRTHDATE : 2015-03-05 , NAME : deepankar , AGE : 22 , AMOUNT : 15.5 }
But when i see the mapping of my testtype in testindex, all mapping of fields are of "type" : "string"
Why the fields does not maps to "type": "string" , or "type" : "long" , and even "type" : "date" as elasticsearch does it by default?
Elasticsearch will attempt to 'guess' the field type by the first insert, unless you create and map fields beforehand.
There are two possible reasons why your fields are being indexed as string instead of long or any other type:
You're not really sending these fields as int, so you're sending '10' instead of 10
You've already inserted at least 1 document that had a string value for that field, so if you've inserted your first document with AGE: '22' Elasticsearch will set that field to type: string and any future inserts will have a string value.
If you want to make sure, you can delete the current index, re-create it and manually set up mapping before inserting the first document, like so:
curl -XPUT 'http://localhost:9200/testindex/_mapping/testmapping' -d '
{
"testmapping" : {
"properties" : {
"birthdate" : { "type" : "date", "format": "dateOptionalTime" },
"name" : { "type" : "string" },
"age" : { "type" : "long" },
"amount" : { "type" : "double" }
}
}
}
'

How to add multiple object types to elasticsearch using jdbc river?

I'm using the jdbc river to successfully add one object type, "contacts", to elasticsearch. How can I add another contact type with different fields? I'd like to add "companies" as well.
What I have is below. Do I need to do a separate PUT statement? If I do, no new data appears to be added to elasticsearch.
PUT /_river/projects_river/_meta
{
"type" : "jdbc",
"index" : {
"index" : "ALL",
"type" : "project",
"bulk_size" : 500,
"max_bulk_requests" : 1,
"autocommit": true
},
"jdbc" : {
"driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"poll" : "30s",
"strategy" : "poll",
"url" : "jdbc:sqlserver://connectionstring",
"user":"username","password":"password",
"sql" : "select ContactID as _id, * from Contact"
}
}
Also, when search returns results, how can I tell if they are of type contact or company? Right now they all have a type of "jdbc", and changing that in the code above throws an error.
You can achieve what you want with inserting several columns to your sql query.
Like ContactID AS _id you can also define indexName AS _index and indexType AS _type in your sql query.
Also, if you need another river, add rivers with different _river types.
In your case such as,
PUT /_river/projects_river2/_meta + Query ....
PUT /_river/projects_river3/_meta + Query ....
Anyone else who stumbles across this, please see official documentation for syntax first: https://github.com/jprante/elasticsearch-river-jdbc/wiki/How-bulk-indexing-isused-by-the-JDBC-river
Here's the final put statement I used:
PUT /_river/contact/_meta
{
"type":"jdbc",
"jdbc": {
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"connectionstring",
"user":"username",
"password":"password",
"sql":"select ContactID as _id,* from Contact",
"poll": "5m",
"strategy": "simple",
"index": "contact",
"type": "contact"
}
}

Resources