Misaligned UV coordinates after Draco compression - three.js

All extension attributes are misaligned after the very first primitive in the asset after exporting to glTF using Blender's inbuilt Draco compression.
Example: first one is good but the two next are misaligned which causes error.
"extensions" : {
"KHR_draco_mesh_compression" : {
"bufferView" : 0,
"attributes" : {
"POSITION" : 0,
"NORMAL" : 1,
"TEXCOORD_0" : 2,
"TEXCOORD_1" : 3
}
}
},
"indices" : 4,
"material" : 0,
"mode" : 4
},
{
"attributes" : {
"POSITION" : 5,
"NORMAL" : 6,
"TEXCOORD_0" : 7,
"TEXCOORD_1" : 8
},
"extensions" : {
"KHR_draco_mesh_compression" : {
"bufferView" : 4,
"attributes" : {
"POSITION" : 0,
"NORMAL" : 1,
"TEXCOORD_0" : 7,
"TEXCOORD_1" : 8
}
}
},
"indices" : 9,
"material" : 1,
"mode" : 4
},
{
"attributes" : {
"POSITION" : 10,
"NORMAL" : 11,
"TEXCOORD_0" : 12,
"TEXCOORD_1" : 13
},
"extensions" : {
"KHR_draco_mesh_compression" : {
"bufferView" : 5,
"attributes" : {
"POSITION" : 0,
"NORMAL" : 1,
"TEXCOORD_0" : 12,
"TEXCOORD_1" : 13
}
}
},
I learned that one solution could be to change the attributes within the extensions of the second and further primitives so they follow an incremental order. However, when dealing with a lot of objects this manual process may cause some issues as it will require a lot of manual work which may result in bugs and errors in the result.
My question is does the most recent version of Draco and Three.js have these bug fixed? Our app is currently running on three.js r107.

Solution part 1: Yes, you should manually update extensions._compression.attributes so they follow a numerical order, at least until this bug report is fixed.
However, I'm almost certain that the reason you're still running into this problem is because you're using Three.js r107. I say this because GLTF support for KHR_Mesh_Quantization was added on r111, which Blender's DRACO compression uses. Additionally, there have been substantial improvements to both DRACO and GLTF between r107 and r117. You can see this in the changelog.
When updating to the newest version, don't forget to update the files for dracoLoader.setDecoderPath() as outlined in the docs.

Related

How do I index non-standard version numbers in Elasticsearch?

Our documents have a field of keyword type called "version_numbers", which essentially stores an array of version numbers. For example: ["1.00.100.01", "2.00.470.00"].
This versioning follows a specific pattern, where each group should be associated with search keywords. Here's a breakdown of the versioning pattern:
1.00.240.15
\ / | |
\/ \ \_ maintenance version
major \_ minor version
version
I want to to build an analyzer such that:
major version is associated with keyword APP_VERSION and can be searched with queries like APP_VERSION1.0, APP_VERSION1.00, APP_VERSION1 etc.
minor version is associated with keyword XP and can be searched with queries like XP24, XP 24, XP240 etc.
maintenance version is associated with keyword Rev and can be searched with queries like Rev15, Rev 15 etc.
documents also can be queried in a combination of all three, like APP_VERSION1 XP240 Rev15 etc.
How do I associate each group of version pattern with keywords specified above?
Here's what I've tried so far to tokenize versions:
{
"analysis": {
"analyzer": {
"version_analyzer": {
"tokenizer": "version_tokenizer"
}
},
"tokenizer": {
"version_tokenizer": {
"type": "simple_pattern_split",
"pattern": "\\d"
}
}
}
}
But this seems to be splitting by dots only.
{
"tokens" : [
{
"token" : "2",
"start_offset" : 0,
"end_offset" : 1,
"type" : "word",
"position" : 0
},
{
"token" : "00",
"start_offset" : 2,
"end_offset" : 4,
"type" : "word",
"position" : 1
},
{
"token" : "470",
"start_offset" : 5,
"end_offset" : 8,
"type" : "word",
"position" : 2
},
{
"token" : "00",
"start_offset" : 9,
"end_offset" : 11,
"type" : "word",
"position" : 3
}
]
}
I'm new to Elasticsearch, so I highly appreciate any hints/guidance on this.

How to design my elastic index for my application?

My application is a survey creation app where user can create survey with many questions of different types. Each survey can then be shared to any number of people whose responses are recorded as below...
{
"id" : 256, // submission id
"timeTaken" : "39.00",
"startTime" : "2020-07-19T05:37:38.873Z",
"state" : "COMPLETED",
"completedTime" : "2020-07-19T05:38:17.873Z",
"deviceType" : "COMPUTER",
"ip" : null,
"account_id" : 2,
"channel_id" : 48,
"contact_id" : null,
"survey_id" : 10,
"trigger_id" : 93,
"trigger_contact_id" : null,
"locked" : false,
"location" : null,
"language" : null,
"submission_id" : 256,
"question_90" : {
"skipped" : false,
"answer_choices" : [ 79 ]
},
"question_122" : {
"skipped" : false,
"otherChoice" : null,
"answer_choices" : [ 115, 113, 111, 110, 114 ]
},
"question_106" : {
"skipped" : false,
"answer_choices" : [
85
]
},
"question_120" : {
"answer_txt": "Great service",
"skipped" : false
},
"question_118" : {
"answer_txt": "Hello people",
"skipped" : false
},
"question_121" : {
"skipped" : false,
"answer_date" : "2020-06-04T20:01:49.783Z",
"answer_timezone" : 330
},
"question_108" : {
"skipped" : false,
"answer_int" : "93"
},
"question_105" : {
"skipped" : false,
"answer_string" : "+1 202 9932219"
},
"question_93" : {
"skipped" : false,
"answer_string" : "Kyra60#yahoo.com"
},
"question_117" : {
"skipped" : false
},
"question_92" : {
"skipped" : false,
"answer_txt" : "composite"
},
"question_107" : {
"skipped" : false,
"answer_bool" : true
},
}
Initially i had created one index per survey but it turned out to be a bad idea since each index allocated 5 shards and my application had nearly 20k surveys created by users. Amazon elastic service broke down and responded 60k shards were created in my 2 nodes..
In this dilemma, I have no idea on how to create my index or meaningfully partition it for efficient querying in the later stage.
Can anyone share some insights and ask me more question so that I can update question for clarity?
Looks like you are using elasticsearch version < 7.X where default number of primary shards were 5 which is changed to 1 and one of the reason was your problem of having a lot of smaller size shards which impacts the Elasticsearch performance.
You should ideally create just one index for all your survey and based on time-range or size you can roll-over to a new index.
you need to have survey_id(unique identification of survey) in your single index and when querying against the index, use survey_id in filter context to get the better query performance as filter contexts are cached by default.

Elasticsearch deleted document reappears using logstash

I am running ES on single node cluster for development.
I am deleting a document using delete api from kibana. It is deleted for a second and immediately reappears. Any help would be appreciated
Here is api command I use:
DELETE test/_doc/12345
{
"_index" : "test",
"_type" : "_doc",
"_id" : "12345",
"_version" : 231,
"result" : "deleted",
"_shards" : {
"total" : 3,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 899,
"_primary_term" : 1
}
GET test/_count
{
"count" : 3,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}
Immediately deleted doc is re-indexed
GET test/_count
{
"count" : 4,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}
According to the documentation:
...If clean_run is set to true, this value will be ignored and
sql_last_value will be set to Jan 1, 1970
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_state
That may explain why all your data are added each 10 minutes. Remove the clean_run and test again or check if the _version filed is updated.
I found that it was an data issue. my logstash jdbc statement checks for modificationdate greater than sql_last_value. and scheduler is set to run every 10 seconds. The reappeared documents have modificationdate in the future, changing it to current date solved the problem

elasticsearch: How to interpret log file (cluster went to yellow status)?

Elasticsearch 1.7.2 on CentOS, 8GB RAM, 2 node cluster.
We posted the whole log here: http://pastebin.com/zc2iG2q4
When we look at /_cluster/health , we see 2 unassigned shards:
{
"cluster_name" : "elasticsearch-prod",
"status" : "yellow", <--------------------------
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2, <--------------------------
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
In the log, we see:
marking and sending shard failed due to [failed to create shard]
java.lang.OutOfMemoryError: Java heap space
And other errors.
The only memory related config value we have is:
indices.fielddata.cache.size: 75%
We are looking to:
understand the log more completely
understand what action we need to take to address the situation now (recover) and prevent it in the future
Additional details:
1) ES_HEAP_SIZE is stock, no changes. (Further, looking around, it is not clear where best to change it.... /etc/init.d/elasticsearch ?)
2) Our jvm stats are below. (And please note, as a test, I modded "/etc/init.d/elasticsearch" and and added export ES_HEAP_SIZE=4g [in place of the existing "export ES_HEAP_SIZE" line] and restarted ES.... Comparing two identical nodes, one with the changed elasticsearch file, and one stock, the values below appear identical)
"jvm" : {
"timestamp" : 1448395039780,
"uptime_in_millis" : 228297,
"mem" : {
"heap_used_in_bytes" : 81418872,
"heap_used_percent" : 7,
"heap_committed_in_bytes" : 259522560,
"heap_max_in_bytes" : 1037959168,
"non_heap_used_in_bytes" : 50733680,
"non_heap_committed_in_bytes" : 51470336,
"pools" : {
"young" : {
"used_in_bytes" : 52283368,
"max_in_bytes" : 286326784,
"peak_used_in_bytes" : 71630848,
"peak_max_in_bytes" : 286326784
},
"survivor" : {
"used_in_bytes" : 2726824,
"max_in_bytes" : 35782656,
"peak_used_in_bytes" : 8912896,
"peak_max_in_bytes" : 35782656
},
"old" : {
"used_in_bytes" : 26408680,
"max_in_bytes" : 715849728,
"peak_used_in_bytes" : 26408680,
"peak_max_in_bytes" : 715849728
}
}
},
"threads" : {
"count" : 81,
"peak_count" : 81
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 250,
"collection_time_in_millis" : 477
},
"old" : {
"collection_count" : 1,
"collection_time_in_millis" : 22
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 112,
"used_in_bytes" : 20205138,
"total_capacity_in_bytes" : 20205138
},
"mapped" : {
"count" : 0,
"used_in_bytes" : 0,
"total_capacity_in_bytes" : 0
}
}
},
Solved.
The key here is the error "java.lang.OutOfMemoryError: Java heap space"
Another day, another gem from the ES docs:
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
says (emphasis mine):
The default installation of Elasticsearch is configured with a 1 GB heap. For just about every deployment, this number is far too small. If you are using the default heap values, your cluster is probably configured incorrectly.
Resolution:
Edit: /etc/sysconfig/elasticsearch
Set ES_HEAP_SIZE=4g // this system has 8GB RAM
Restart ES
And tada.... the unassigned shards are magically assigned, and the cluster goes green.

Elastic Search Index Status

I am trying to setup a scripted reindex operation as suggested in: http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/
To go with the suggestion of creating a new index, aliasing then deleting the old index I would need to have a way to tell when the indexing operation on the new index was complete. Ideally via the REST interface.
It has 80 million rows to index and can take a few hours.
I can't find anything helpful in the docs..
You can try with _stats : http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-stats.html
Eg :
{
"_shards" : {
"total" : 10,
"successful" : 5,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 575,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
"index_time_in_millis" : 0,
"index_current" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
I think, you can compare _all.total.docs.count and _all.total.indexing.index_current

Resources