Index created successfully, but won't reindex new data - elasticsearch

I'm totally bamboozled, since this code was working previously.
I'm trying to create a Mixed Edge index on an edge property in my graph (trxn_dt_int).
It shows, from the output of mgmt.printSchema(), that it was created successfully (see, Status: Enabled, I've highlighted the relevant rows).
mgmt.printSchema() output:
==>------------------------------------------------------------------------------------------------
Vertex Label Name | Partitioned | Static |
---------------------------------------------------------------------------------------------------
entity | false | false |
---------------------------------------------------------------------------------------------------
Edge Label Name | Directed | Unidirected | Multiplicity |
---------------------------------------------------------------------------------------------------
transacts_with | true | false | MULTI |
---------------------------------------------------------------------------------------------------
Property Key Name | Cardinality | Data Type |
---------------------------------------------------------------------------------------------------
benef_nm | SINGLE | class java.lang.String |
orig_nm | SINGLE | class java.lang.String |
trxn_dt_int | SINGLE | class java.lang.Float |
---------------------------------------------------------------------------------------------------
Vertex Index Name | Type | Unique | Backing | Key: Status |
---------------------------------------------------------------------------------------------------
***************************************************************************************************
Edge Index (VCI) Name | Type | Unique | Backing | Key: Status |
---------------------------------------------------------------------------------------------------
byDate | Mixed | false | search | trxn_dt_int: ENABLED|
***************************************************************************************************
---------------------------------------------------------------------------------------------------
Relation Index | Type | Direction | Sort Key | Order | Status |
Here's my groovy script to create the index:
graph.tx().rollback()
mgmt = graph.openManagement()
mgmt.printSchema()
// get index count
index_count = mgmt.getGraphIndexes(Edge.class).size()
if (index_count==0) {
// create edge schema for graph (only for properties to be indexed)
if (!mgmt.getEdgeLabel('transacts_with') & !mgmt.getPropertyKey('trxn_dt_int')) {
transacts_with = mgmt.makeEdgeLabel('transacts_with').make()
trxn_dt = mgmt.makePropertyKey('trxn_dt_int').dataType(Float.class).make()
mgmt.addProperties(transacts_with, trxn_dt)
}
mgmt.commit()
mgmt = graph.openManagement()
if (!mgmt.containsGraphIndex("byDate")) {
trxn_dt = mgmt.getPropertyKey('trxn_dt_int')
mgmt.buildIndex('byDate', Edge.class).addKey(trxn_dt).buildMixedIndex('search')
mgmt.commit()
ManagementSystem.awaitGraphIndexStatus(graph, 'byDate').call()
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex("byDate"), SchemaAction.REINDEX).get()
mgmt.commit()
}
}
When I query the elasticsearch server, it shows it's there:
curl --silent 'http://127.0.0.1:9200/_cat/indices' | cut -d\ -f3
> janusgraph_bydate
When I query for the number of documents, it returns 0.
curl --silent 'http://127.0.0.1:9200/janusgraph_bydate/_count?q=*'
> {"count":0,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
My question is, why aren't any documents being added?
I initially thought I had to reindex, but when I tried, it still didn't work. No errors at all.
mgmt = graph.openManagement()
index = mgmt.getGraphIndex('byDate')
mgmt.updateIndex(index, SchemaAction.REINDEX).get()
mgmt.commit()
Setup:
Janusgraph 0.5.1
Elasticsearch 6.0.1

Well, this is embarrassing; it wasn't updating the index with new data because my machine lacked sufficient memory, but also didn't offer warnings to suggest this. After freeing up memory elsewhere it now works again.

Related

How to properly apply index_desc hint for a remote database object in oracle database?

We were facing some issues with execution plans while accessing remote database objects with dblink. Here is the query itself run on the remote database:
select --+ index_desc (d DAY_OPERATIONAL_PK)
d.oper_day
from day_operational d
where rownum = 1
The plan for this query is the following :
Plan Hash Value : 2761870770
---------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost | Time |
---------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 8 | 2 | 00:00:01 |
| * 1 | COUNT STOPKEY | | | | | |
| 2 | INDEX FULL SCAN DESCENDING | DAY_OPERATIONAL_PK | 1 | 8 | 2 | 00:00:01 |
---------------------------------------------------------------------------------------------
This one works correct that is it returns the last operational day. In this case 14.09.2021. However if execute this exact same query from other database connecting to this one via dblink, wrong results are returned . In this case the first row of the table is returned - 05.09.2009.
Here is the query:
select --+ index_desc (d DAY_OPERATIONAL_PK)
d.oper_day
from day_operational#iabs d
where rownum = 1
The plan generated for this query in local database is the following:
Plan Hash Value :
---------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost | Time |
---------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT REMOTE | | 1 | 8 | 2 | 00:00:01 |
| * 1 | COUNT STOPKEY | | | | | |
| 2 | INDEX FAST FULL SCAN | XPKDAY_OPERATIONAL | 1 | 8 | 2 | 00:00:01 |
---------------------------------------------------------------------------------------
As it can be seen, the plan generated when connected via dblink uses full table scan and ignores index_desc hint. How could we enforce oracle to use this index? Tried adding driving_site hint but it didn't help
Sorry for confusions. It occured that index names in local database and remote database are not same and therefore hint was ignoring since there was no index with that name in the remote db. With right index names, both the plan and result were correct

Laravel | Return rows in order based off pivot

I have the following setup:
stages:
| id | name | order |
commands:
| id | name | body |
------------------------------=
| 1 | test | phpunit |
| 2 | style | echo "style" |
| 3 | deploy | deploy |
command_stage:
| command_id | stage_id | order |
---------------------------------
| 1 | 1 | 1 |
| 2 | 2 | 1 |
| 3 | 1 | 2 |
Basically, I would like to create a method on the stage model which allows me to get all of the commands back based off of the order, but commands have a specific order and so do the stages.
So each command is stored under a stage so we know which part to run but each command also has an order in that stage. Now I know I can do something like the following:
$inOrder = collect();
Stage::get()->each(function ($stage) {
$commands = $stage->commands()->orderByPivot('order')->get();
$inOrder->push($commands);
});
return $inOrder;
But I was wondering if there is a nicer way to do this? Or even a way to do this solely on one database hit?
Pre-load and sort the relationship:
$stages = Stage::with(['commands' => function ($subQuery) {
$subQuery->orderBy('command_stage', 'order');
}])->get();
foreach($stages as $stage) {
$inOrder->push($stage->commands);
}
This method of Eager-Loading will reduce the N+1 query issue of doing $stage->commands() inside the foreach() loop.

SqlQuery returns no results but ScanQuery contains result

The cache in Ignite contains the following data:
+=======================================================================================================================================================+
| Key Class | Key | Value Class | Value |
+=======================================================================================================================================================+
| java.lang.Long | 1 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=-533910214, Domain=x.blub.com, id=1, advId=31] |
| java.lang.Long | 2 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=-482120537, Domain=x.blub.de, id=2, advId=31] |
| java.lang.Long | 3 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=-1060766828, Domain=x.blub.at, id=3, advId=31] |
| java.lang.Long | 4 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=376860969, Domain=y.blab.com, id=4, advId=94] |
| java.lang.Long | 5 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=1641697092, Domain=y.blib.com, id=5, advId=94] |
| java.lang.Long | 6 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=-757055874, Domain=z.bla.com, id=6, advId=1] |
| java.lang.Long | 7 | o.a.i.i.binary.BinaryObjectImpl | com.test.domain.Domain [hash=-1536137434, Domain=h.bla.com, id=7, advId=20] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------+
In my spring-boot project I do the following query:
SqlQuery<Long, Domain> domainSqlQuery = new SqlQuery<>(Domain.class, "advId = ?");
trackdomainSqlQuery.setArgs(31);
List<Cache.Entry<Long, Domain>> domainCacheEntrySqlQueryResults = domainConfigurationCache.query(domainSqlQuery).getAll();
the query returns an empty list, but if I run a ScanQuery I get them all
List<Cache.Entry<Long, Domain>> domainCacheEntryScanQueryResults = domainConfigurationCache.query(new ScanQuery()).getAll();
Here the result from the ScanQuery:
[IgniteBiTuple [val1=1, val2=Domain{id=1, domain='blib.blub.com', advId=31}],
IgniteBiTuple [val1=2, val2=Domain{id=2, domain='blib.blub.de', advId=31}],
IgniteBiTuple [val1=3, val2=Domain{id=3, domain='Domain=blib.blub.at', advId=31}],
IgniteBiTuple [val1=4, val2=Domain{id=4, domain='blab.blab.com', advId=94}],
IgniteBiTuple [val1=5, val2=Domain{id=5, domain='blib.blab.com', advId=94}],
IgniteBiTuple [val1=6, val2=Domain{id=6, domain='blo.bla.com', advId=1}],
IgniteBiTuple [val1=7, val2=Domain{id=7, domain='blu.bla.com', advId=20}]]
Also just for the info the version I'm using for ignite-core and ignite-indexing is 2.6.0
Anyone has an idea what I'm doing wrong?
As you have indicated:
Ignite will only index cache entries where key and value types match what your tables expect.
You can store any types in cache but table will only accept single key type and single value type. You can also have multiple tables per cache, btw, provided their types do not overlap.
Entries whose types did not match any table will be present in cache but not obtainable via SQL. This is what you observed when package name was com.test.domain.cache.Domain in configuration but com.test.domain.Domain in practice (or vice versa).
Try updating your second line:
trackdomainSqlQuery.setArgs(31L);
Your key class is a long but you're passing in an integer. (I've not seen this in Java but I had the same problem using the Python API.)

Automatically generating documentation about the structure of the database

There is a database that contains several views and tables.
I need create a report (documentation of database) with a list of all the fields in these tables indicating the type and, if possible, an indication of the minimum/maximum values and values from first row. For example:
.------------.--------.--------.--------------.--------------.--------------.
| Table name | Column | Type | MinValue | MaxValue | FirstRow |
:------------+--------+--------+--------------+--------------+--------------:
| Table1 | day | date | ‘2010-09-17’ | ‘2016-12-10’ | ‘2016-12-10’ |
:------------+--------+--------+--------------+--------------+--------------:
| Table1 | price | double | 1030.8 | 29485.7 | 6023.8 |
:------------+--------+--------+--------------+--------------+--------------:
| … | | | | | |
:------------+--------+--------+--------------+--------------+--------------:
| TableN | day | date | ‘2014-06-20’ | ‘2016-11-28’ | ‘2016-11-16’ |
:------------+--------+--------+--------------+--------------+--------------:
| TableN | owner | string | NULL | NULL | ‘Joe’ |
'------------'--------'--------'--------------'--------------'--------------'
I think the execution of many queries
SELECT MAX(column_name) as max_value, MIN(column_name) as min_value
FROM table_name
Will be ineffective on the huge tables that are stored in Hadoop.
After reading documentation found an article about "Statistics in Hive"
It seems I must use request like this:
ANALYZE TABLE tablename COMPUTE STATISTICS FOR COLUMNS;
But this command ended with error:
Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.ColumnStatsTask
Do I understand correctly that this request add information to the description of the table and not display the result? Will this request work with view?
Please suggest how to effectively and automatically create documentation for the database in HIVE?

How many leafref is possible inside a leaf in Yang modelling?

According to RFC - RFC 6020 - LeafRef I can understand that the leaf can contain a leafref which inturn have the path pointing to the instance which is referenced but question is how many leafrefs are possible for one leaf? Only one or many?
Ex.
leaf mgmt-interface {
type leafref {
path "../interface/name";
}
type leafref {
path "../interface/ip";
}
}
Is the above possible?
A leafref may only target a single leaf or leaf-list node via path. There may only be one type substatement to a leaf (also applies to leaf-list, typedef) and there may only be a single path substatement to type.
7.6.2. The leaf's Substatements
+--------------+---------+-------------+
| substatement | section | cardinality |
+--------------+---------+-------------+
| config | 7.19.1 | 0..1 |
| default | 7.6.4 | 0..1 |
| description | 7.19.3 | 0..1 |
| if-feature | 7.18.2 | 0..n |
| mandatory | 7.6.5 | 0..1 |
| must | 7.5.3 | 0..n |
| reference | 7.19.4 | 0..1 |
| status | 7.19.2 | 0..1 |
| type | 7.6.3 | 1 | <--
| units | 7.3.3 | 0..1 |
| when | 7.19.5 | 0..1 |
+--------------+---------+-------------+
12. YANG ABNF Grammar
type-stmt = type-keyword sep identifier-ref-arg-str optsep
(";" /
"{" stmtsep
type-body-stmts
"}")
type-body-stmts = numerical-restrictions /
decimal64-specification /
string-restrictions /
enum-specification /
leafref-specification /
identityref-specification /
instance-identifier-specification /
bits-specification /
union-specification
leafref-specification =
;; these stmts can appear in any order
path-stmt stmtsep
[require-instance-stmt stmtsep]
path-stmt = path-keyword sep path-arg-str stmtend
Note: it is not possible to use union for leafref types in YANG 1.0. This has changed in YANG 1.1 however, where any built-in YANG type may appear inside a union.
9.12. The union Built-In Type
A member type can be of any built-in or derived type, except it MUST
NOT be one of the built-in types "empty" or "leafref".

Resources