Rocksdb compaction not triggered or not happening - apache-kafka-streams
I have 2 kafka streams state stores implemented. They are both persistent key value stores. The problem that I am facing is the compaction of rocksdb is happening only in one of the state stores and the other state store is just piling on more sst files and in turn increasing the disk space. I tried to debug the rocksdb logs. Everything looked fine there. The compactions were happening fine for one of the state stores and there was no info related to another store. Can anyone please help on this aspect?
To add more helpful info, this is happening only inside the container. I tried to replicate this with a sample app in my local developer environment, there were no issues there.
Edit:
I was able to print the rocksdb logs at the KV store level and found some interesting aspects. The compaction was happening but the files were not getting deleted. Here are the logs. I found the levels kept increasing. Can anyone explain why the levels are increasing?
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.182015) [db/db_impl_compaction_flush.cc:140] [default] Level summary: **files[9 0 0 0 0 0 1]** max score 1.00
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183102) [db/compaction_picker_universal.cc:473] [default] Universal: Possible candidate file 58[0].
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183113) [db/compaction_picker_universal.cc:525] [default] Universal: Skipping file 58[0] with size 4244 (compensated size 4244)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183116) [db/compaction_picker_universal.cc:473] [default] Universal: Possible candidate file 57[1].
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183298) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 57[1] with size 6522 (compensated size 6522)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183301) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 56[2] with size 4245 (compensated size 4245)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183302) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 55[3] with size 6506 (compensated size 6506)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183303) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 54[4] with size 4244 (compensated size 4244)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183305) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 53[5] with size 6505 (compensated size 6505)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183306) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 52[6] with size 4244 (compensated size 4244)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183307) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 51[7] with size 6503 (compensated size 6503)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183309) [db/compaction_picker_universal.cc:594] [default] Universal: Picking file 50[8] with size 4245 (compensated size 4245)
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.183429) [db/compaction_picker_universal.cc:289] [default] Universal: compacting for size ratio
example_store: INFO_LEVEL: [db/compaction_job.cc:1437] [default] [JOB 51] Compacting 8#0 files to L5, score 1.00
example_store: INFO_LEVEL: [db/compaction_job.cc:1441] [default] Compaction start summary: Base version 46 Base level 0, inputs: [57(6522B) 56(4245B) 55(6506B) 54(4244B) 53(6505B) 52(4244B) 51(6503B) 50(4245B)], [], [], [], [], [], []
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770177193012, "job": 51, "event": "compaction_started", "files_L0": [57, 56, 55, 54, 53, 52, 51, 50], "files_L1": [], "files_L2": [], "files_L3": [], "files_L4": [], "files_L5": [], "files_L6": [], "score": 1, "input_data_size": 43014}
example_store: INFO_LEVEL: [db/compaction_job.cc:1132] [default] [JOB 51] Generated table #59: 32 keys, 9878 bytes
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770177315362, "cf_name": "default", "job": 51, "event": "table_file_creation", "file_number": 59, "file_size": 9878, "table_properties": {"data_size": 8826, "index_size": 296, "filter_size": 0, "raw_key_size": 910, "raw_average_key_size": 28, "raw_value_size": 29633, "raw_average_value_size": 926, "num_data_blocks": 8, "num_entries": 32, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/compaction_job.cc:1199] [default] [JOB 51] Compacted 8#0 files to L5 => 9878 bytes
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.357595) [db/compaction_job.cc:621] [default] compacted to: files[1 0 0 0 0 1 1] max score 0.30, MB/sec: 0.3 rd, 0.1 wr, level 5, files in(8, 0) out(1) MB in(0.0, 0.0) out(0.0), read-write-amplify(1.2) write-amplify(0.2) OK, records in: 128, records dropped: 96
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.357647) EVENT_LOG_v1 {"time_micros": 1560770177357627, "job": 51, "event": "compaction_finished", "compaction_time_micros": 142703, "output_level": 5, "num_output_files": 1, "total_output_size": 9878, "num_input_records": 128, "num_output_records": 32, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 0, 0, 0, 0, 1, 1]}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.358280) [db/compaction_picker_universal.cc:261] [default] Universal: nothing to do
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:16:17.358281) [db/db_impl_compaction_flush.cc:1518] Compaction nothing to do
example_store: INFO_LEVEL: [db/db_impl_write.cc:1127] [default] New memtable created with log file: #12. Immutable memtables: 0.
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:17:47.147896) [db/db_impl_compaction_flush.cc:1216] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
example_store: INFO_LEVEL: [db/flush_job.cc:264] [default] [JOB 53] Flushing memtable with next log file: 12
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770267148077, "job": 53, "event": "flush_started", "num_memtables": 1, "num_entries": 16, "num_deletes": 0, "memory_usage": 19704}
example_store: INFO_LEVEL: [db/flush_job.cc:293] [default] [JOB 53] Level-0 flush table #60: started
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770267150374, "cf_name": "default", "job": 53, "event": "table_file_creation", "file_number": 60, "file_size": 6502, "table_properties": {"data_size": 5514, "index_size": 186, "filter_size": 0, "raw_key_size": 455, "raw_average_key_size": 28, "raw_value_size": 17717, "raw_average_value_size": 1107, "num_data_blocks": 5, "num_entries": 16, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/flush_job.cc:324] [default] [JOB 53] Level-0 flush table #60: 6502 bytes OK
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:17:47.152018) [db/memtable_list.cc:372] [default] Level-0 commit table #60 started
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:17:47.153419) [db/memtable_list.cc:395] [default] Level-0 commit table #60: memtable #1 done
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:17:47.153437) EVENT_LOG_v1 {"time_micros": 1560770267153430, "job": 53, "event": "flush_finished", "lsm_state": [2, 0, 0, 0, 0, 1, 1], "immutable_memtables": 0}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:17:47.153456) [db/db_impl_compaction_flush.cc:140] [default] Level summary: files[2 0 0 0 0 1 1] max score 0.40
example_store: INFO_LEVEL: [db/db_impl_write.cc:1127] [default] New memtable created with log file: #12. Immutable memtables: 0.
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:18:17.170673) [db/db_impl_compaction_flush.cc:1216] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
example_store: INFO_LEVEL: [db/flush_job.cc:264] [default] [JOB 54] Flushing memtable with next log file: 12
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770297170853, "job": 54, "event": "flush_started", "num_memtables": 1, "num_entries": 16, "num_deletes": 0, "memory_usage": 14224}
example_store: INFO_LEVEL: [db/flush_job.cc:293] [default] [JOB 54] Level-0 flush table #61: started
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770297174979, "cf_name": "default", "job": 54, "event": "table_file_creation", "file_number": 61, "file_size": 4244, "table_properties": {"data_size": 3293, "index_size": 107, "filter_size": 0, "raw_key_size": 455, "raw_average_key_size": 28, "raw_value_size": 11916, "raw_average_value_size": 744, "num_data_blocks": 3, "num_entries": 16, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/flush_job.cc:324] [default] [JOB 54] Level-0 flush table #61: 4244 bytes OK
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:18:17.177091) [db/memtable_list.cc:372] [default] Level-0 commit table #61 started
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:18:17.178570) [db/memtable_list.cc:395] [default] Level-0 commit table #61: memtable #1 done
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:18:17.178590) EVENT_LOG_v1 {"time_micros": 1560770297178582, "job": 54, "event": "flush_finished", "lsm_state": [3, 0, 0, 0, 0, 1, 1], "immutable_memtables": 0}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:18:17.178614) [db/db_impl_compaction_flush.cc:140] [default] Level summary: files[3 0 0 0 0 1 1] max score 0.50
example_store: INFO_LEVEL: [db/db_impl_write.cc:1127] [default] New memtable created with log file: #12. Immutable memtables: 0.
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:20:17.360151) [db/db_impl_compaction_flush.cc:1216] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
example_store: INFO_LEVEL: [db/flush_job.cc:264] [default] [JOB 55] Flushing memtable with next log file: 12
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770417360328, "job": 55, "event": "flush_started", "num_memtables": 1, "num_entries": 32, "num_deletes": 0, "memory_usage": 32304}
example_store: INFO_LEVEL: [db/flush_job.cc:293] [default] [JOB 55] Level-0 flush table #62: started
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770417365009, "cf_name": "default", "job": 55, "event": "table_file_creation", "file_number": 62, "file_size": 9865, "table_properties": {"data_size": 8813, "index_size": 296, "filter_size": 0, "raw_key_size": 910, "raw_average_key_size": 28, "raw_value_size": 29633, "raw_average_value_size": 926, "num_data_blocks": 8, "num_entries": 32, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/flush_job.cc:324] [default] [JOB 55] Level-0 flush table #62: 9865 bytes OK
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:20:17.371008) [db/memtable_list.cc:372] [default] Level-0 commit table #62 started
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:20:17.373221) [db/memtable_list.cc:395] [default] Level-0 commit table #62: memtable #1 done
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:20:17.373274) EVENT_LOG_v1 {"time_micros": 1560770417373259, "job": 55, "event": "flush_finished", "lsm_state": [4, 0, 0, 0, 0, 1, 1], "immutable_memtables": 0}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:20:17.373318) [db/db_impl_compaction_flush.cc:140] [default] Level summary: files[4 0 0 0 0 1 1] max score 0.60
example_store: INFO_LEVEL: [db/db_impl_write.cc:1127] [default] New memtable created with log file: #12. Immutable memtables: 0.
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:22:17.625738) [db/db_impl_compaction_flush.cc:1216] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
example_store: INFO_LEVEL: [db/flush_job.cc:264] [default] [JOB 56] Flushing memtable with next log file: 12
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770537625921, "job": 56, "event": "flush_started", "num_memtables": 1, "num_entries": 32, "num_deletes": 0, "memory_usage": 32360}
example_store: INFO_LEVEL: [db/flush_job.cc:293] [default] [JOB 56] Level-0 flush table #63: started
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770537647724, "cf_name": "default", "job": 56, "event": "table_file_creation", "file_number": 63, "file_size": 9858, "table_properties": {"data_size": 8807, "index_size": 296, "filter_size": 0, "raw_key_size": 910, "raw_average_key_size": 28, "raw_value_size": 29633, "raw_average_value_size": 926, "num_data_blocks": 8, "num_entries": 32, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/flush_job.cc:324] [default] [JOB 56] Level-0 flush table #63: 9858 bytes OK
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:22:17.666254) [db/memtable_list.cc:372] [default] Level-0 commit table #63 started
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:22:17.685689) [db/memtable_list.cc:395] [default] Level-0 commit table #63: memtable #1 done
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:22:17.685728) EVENT_LOG_v1 {"time_micros": 1560770537685714, "job": 56, "event": "flush_finished", "lsm_state": [5, 0, 0, 0, 0, 1, 1], "immutable_memtables": 0}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:22:17.685761) [db/db_impl_compaction_flush.cc:140] [default] Level summary: files[5 0 0 0 0 1 1] max score 0.70
example_store: INFO_LEVEL: [db/db_impl_write.cc:1127] [default] New memtable created with log file: #12. Immutable memtables: 0.
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:24:18.003063) [db/db_impl_compaction_flush.cc:1216] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
example_store: INFO_LEVEL: [db/flush_job.cc:264] [default] [JOB 57] Flushing memtable with next log file: 12
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770658007107, "job": 57, "event": "flush_started", "num_memtables": 1, "num_entries": 32, "num_deletes": 0, "memory_usage": 32352}
example_store: INFO_LEVEL: [db/flush_job.cc:293] [default] [JOB 57] Level-0 flush table #64: started
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770658069543, "cf_name": "default", "job": 57, "event": "table_file_creation", "file_number": 64, "file_size": 9863, "table_properties": {"data_size": 8811, "index_size": 296, "filter_size": 0, "raw_key_size": 910, "raw_average_key_size": 28, "raw_value_size": 29633, "raw_average_value_size": 926, "num_data_blocks": 8, "num_entries": 32, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/flush_job.cc:324] [default] [JOB 57] Level-0 flush table #64: 9863 bytes OK
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:24:18.099968) [db/memtable_list.cc:372] [default] Level-0 commit table #64 started
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:24:18.136935) [db/memtable_list.cc:395] [default] Level-0 commit table #64: memtable #1 done
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:24:18.136965) EVENT_LOG_v1 {"time_micros": 1560770658136956, "job": 57, "event": "flush_finished", "lsm_state": [6, 0, 0, 0, 0, 1, 1], "immutable_memtables": 0}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:24:18.136988) [db/db_impl_compaction_flush.cc:140] [default] Level summary: files[6 0 0 0 0 1 1] max score 0.80
example_store: INFO_LEVEL: [db/db_impl_write.cc:1127] [default] New memtable created with log file: #12. Immutable memtables: 0.
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:26:18.014309) [db/db_impl_compaction_flush.cc:1216] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
example_store: INFO_LEVEL: [db/flush_job.cc:264] [default] [JOB 58] Flushing memtable with next log file: 12
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770778014601, "job": 58, "event": "flush_started", "num_memtables": 1, "num_entries": 32, "num_deletes": 0, "memory_usage": 32312}
example_store: INFO_LEVEL: [db/flush_job.cc:293] [default] [JOB 58] Level-0 flush table #65: started
example_store: INFO_LEVEL: EVENT_LOG_v1 {"time_micros": 1560770778036112, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 65, "file_size": 9858, "table_properties": {"data_size": 8807, "index_size": 296, "filter_size": 0, "raw_key_size": 910, "raw_average_key_size": 28, "raw_value_size": 29633, "raw_average_value_size": 926, "num_data_blocks": 8, "num_entries": 32, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
example_store: INFO_LEVEL: [db/flush_job.cc:324] [default] [JOB 58] Level-0 flush table #65: 9858 bytes OK
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:26:18.063192) [db/memtable_list.cc:372] [default] Level-0 commit table #65 started
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:26:18.082165) [db/memtable_list.cc:395] [default] Level-0 commit table #65: memtable #1 done
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:26:18.082203) EVENT_LOG_v1 {"time_micros": 1560770778082193, "job": 58, "event": "flush_finished", "lsm_state": **[7, 0, 0, 0, 0, 1, 1]**, "immutable_memtables": 0}
example_store: INFO_LEVEL: (Original Log Time 2019/06/17-11:26:18.082231) [db/db_impl_compaction_flush.cc:140] [default] Level summary: files[7 0 0 0 0 1 1] max score 0.90
The iterator has to be closed explicitly after using it. This was the main cause of the issue.
Related
Store numbers in a list that has a range
Write a program that will store the numbers from a list called random_number which are less than or equal to 300 into a new list. Print the results. random_numbers = [100, 34, 10, 17, 111, 304, 99, 87, 55, 0, 5, 303, 399, 354, 121, 208, 267, 406, 13] My attempt rand_count=0 for rand_value in random_numbers: if rand_value <=300: print (rand_value) rand_count+=1 else: random_numbers.append(rand_value)
avformat code produce slightly different output than ffmpeg with same parameters - why?
I would like to achieve the exact same result as this ffmpeg command line call does from code: ffmpeg -i CAMERARTSPLINK -c:v copy -an -movflags +frag_keyframe+empty_moov -f mp4 When I run the above command it gives this binary result: got 36 bytes: 0, 0, 0, 36, 102, 116, 121, 112, ..., 111, 54, 109, 112, 52, 49, got 512 bytes: 0, 0, 3, 76, 109, 111, 111, 118, 0, 0, 0, ..., 132, 0, 0, 3, 0, 4, 0, 0, 3, 0, 202, The code can utilize ffmpeg libraries and includes, but I don't want to use ffmpeg as a program call (i.e. exec* functions are not preferred). I have created a small demonstration code example with avformat for an RTSP H264 to MP4 remux. The code is highly reuses horgh's nice videostreamer library. I posted the sample code to pastebin.com (400 loc). It builds successfully but you need to link it against avformat, avdevice, avcodec and avutil. I tried to do my best to reach the same result, however when I run this code, the first few bytes after byte #38 are different (maybe not just those, I did not compare anything after byte #548): writeOutput: writing 36 bytes: 0, 0, 0, 36, 102, 116, 121, 112, ..., 111, 54, 109, 112, 52, 49, writeOutput: writing 512 bytes: 0, 0, 0, 0, 109, 111, 111, 118, 0, 0, 0, ..., 132, 0, 0, 3, 0, 4, 0, 0, 3, 0, 202, You can see on the second line of my code's output starts with 0 0 0 0 109, whereas the ffmpeg gave 0 0 3 76 109. All the rest (even the bytes are not pasted here) data are totally the same (at least for the first 548 bytes). What is wrong with my code? These 2 bytes seems super-important for decoding this stream.
102, 116, 121, 112 in ascii is ftyp This is the mp4 format type box. 0, 0, 0, 36 is the size of the box 109, 111, 111, 118 in ascii is mdat This is the data box. 0, 0, 0, 0 is the size of the box. In this case the size of the mdat box is unknown because we don't know the size of all the video and audio frames yet. So a placeholder of zero is used. When the file is finished the size value should be overwritten with the correct size
This was a silly mistake of ffmpeg's limited logging functionality plus my limited knowledge in video codecs. The problem was that ffmpeg (with h264 input) when writes an atom to the output buffer: first puts 0 for the atom's size (movenc.c#L3981) then fills the rest of the buffer then at the end it seeks back to atom's size in buffer and update the size (movenc.c#L4049) This is all good, but during filling the rest of the buffer, it uses avio_w8 and avio_w8 will flush the buffer, if it reaches the end of it. If you are using custom IO with avio_alloc_context AND you don't define a buffer which is large enough AND you don't define the seek operation either AND the atom being written is larger than your buffer size, it will be flushed out and ffmpeg will not be able to seek back to update the atom's size. And this will probably result in a corrupted (and unplayable) video output file or stream. So the solution was to increase the buffer size from 512 to 4096 in this case moov atom can be fit even without seeking operation. This is pretty straightforward if you know that the moov atom is larger that 512 bytes in length and look my sample code.
How to extract frame types along with motion vectors using extract_mvs.c from ffmpeg
I have been researching ways to get frame types (I, P, B) along with the motion vector data returned from extract_mvs.c in the examples folder in ffmpeg. The extract_mvs.c file after it is compiled, returns information like this: framenum,source,blockw,blockh,srcx,srcy,dstx,dsty,flags 2,-1,16,16, 8, 8, 8, 8,0x0 2, 1,16,16, 8, 8, 8, 8,0x0 2, 1,16,16, 24, 8, 24, 8,0x0 2, 1,16,16, 40, 8, 40, 8,0x0 2, 1,16,16, 56, 8, 56, 8,0x0 2, 1,16,16, 72, 8, 72, 8,0x0 2, 1,16,16, 88, 8, 88, 8,0x0 ... 297, 1,16,16, 248, 280, 248, 280,0x0 297, 1,16,16, 264, 280, 264, 280,0x0 297,-1,16,16, 278, 279, 280, 280,0x0 297, 1,16,16, 280, 280, 280, 280,0x0 297, 1,16,16, 296, 280, 296, 280,0x0 297, 1,16,16, 312, 280, 312, 280,0x0 297, 1,16,16, 328, 280, 328, 280,0x0 297, 1,16,16, 344, 280, 344, 280,0x0 Along with this information, I would like to output frame type so that I know framenum = 2 is, for example, a 'B' frame. I tried different things, one of which was using a separate command: ffprobe input.mp4 -show_frames | grep -E 'pict_type|coded_picture_number' But the problem with this command is that it returns data like: pict_type=I coded_picture_number=0 pict_type=B coded_picture_number=2 pict_type=P coded_picture_number=1 pict_type=B coded_picture_number=4 pict_type=P coded_picture_number=3 .... pict_type=P coded_picture_number=293 pict_type=B coded_picture_number=297 pict_type=B coded_picture_number=296 And there is no much I can relate here between coded_picture_number and framenum. The former starts counting from 0 and the later from 2. I assume framenum starting from 2, means the count from this variable is actually from 1, and it ignored 1 in the extraction process as it is maybe an I frame thus no motion vectors. So, how can we use only extract_mvs.c to get not only that information it provides but also the frame types in the returned table. Any hints either syntax/command-wise or in editing the c file would be appreciated. Thanks in advance.
I found a way to get the frames, had posted on this question, where I replaced the flags argument with frametypes.
Elasticsearch not using all storage drives
I have a 6-node elasticsearch 5.5 cluster with their path.data set to the following:
path.data: "/data/0/elasticsearch,/data/1/elasticsearch,/data/2/elasticsearch,/data/3/elasticsearch,/data/4/elasticsearch"
I have restarted all the nodes after the config change however only 1 node in the cluster actually writes to /data/0/elasticsearch and /data/1/elasticsearch. All the other 5 nodes have only been writing to /data/0/elasticsearch
To give you more insight into my cluster settings:
For my index => "number_of_shards":"5"
How do I configure the cluster to write evenly across all drives /data/0/ through /data/4/ and across all nodes.
Update 1:
Here is my health check report.
{
"cluster_name": "elasticsearch",
"status": "green",
"timed_out": false,
"number_of_nodes": 6,
"number_of_data_nodes": 6,
"active_primary_shards": 6,
"active_shards": 12,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
What are some algorithms for finding a closed form function given an integer sequence?
I'm looking form a programatic way to take an integer sequence and spit out a closed form function. Something like: Given: 1,3,6,10,15 Return: n(n+1)/2 Samples could be useful; the language is unimportant.
This touches an extremely deep, sophisticated and active area of mathematics. The solution is damn near trivial in some cases (linear recurrences) and damn near impossible in others (think 2, 3, 5, 7, 11, 13, ....) You could start by looking at generating functions for example and looking at Herb Wilf's incredible book (cf. page 1 (2e)) on the subject but that will only get you so far. But I think your best bet is to give up, query Sloane's comprehensive Encyclopedia of Integer Sequences when you need to know the answer, and instead spend your time reading the opinions of one of the most eccentric personalities in this deep subject. Anyone who tells you this problem is solvable is selling you snake oil (cf. page 118 of the Wilf book (2e).)
There is no one function in general.
For the sequence you specified, The On-Line Encyclopedia of Integer Sequences finds 133 matches in its database of interesting integer sequences. I've copied the first 5 here.
A000217 Triangular numbers: a(n) = C(n+1,2) = n(n+1)/2 = 0+1+2+...+n.
0, 1, 3, 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91, 105, 120, 136, 153, 171, 190, 210, 231, 253, 276, 300, 325, 351, 378, 406, 435, 465, 496, 528, 561, 595, 630, 666, 703, 741, 780, 820, 861, 903, 946, 990, 1035, 1081, 1128, 1176, 1225, 1275, 1326, 1378, 1431
A130484 Sum {0<=k<=n, k mod 6} (Partial sums of A010875).
0, 1, 3, 6, 10, 15, 15, 16, 18, 21, 25, 30, 30, 31, 33, 36, 40, 45, 45, 46, 48, 51, 55, 60, 60, 61, 63, 66, 70, 75, 75, 76, 78, 81, 85, 90, 90, 91, 93, 96, 100, 105, 105, 106, 108, 111, 115, 120, 120, 121, 123, 126, 130, 135, 135, 136, 138, 141, 145, 150, 150, 151, 153
A130485 Sum {0<=k<=n, k mod 7} (Partial sums of A010876).
0, 1, 3, 6, 10, 15, 21, 21, 22, 24, 27, 31, 36, 42, 42, 43, 45, 48, 52, 57, 63, 63, 64, 66, 69, 73, 78, 84, 84, 85, 87, 90, 94, 99, 105, 105, 106, 108, 111, 115, 120, 126, 126, 127, 129, 132, 136, 141, 147, 147, 148, 150, 153, 157, 162, 168, 168, 169, 171, 174, 178, 183
A104619 Write the natural numbers in base 16 in a triangle with k digits in the k-th row, as shown below. Sequence gives the leading diagonal.
1, 3, 6, 10, 15, 2, 1, 1, 14, 3, 2, 2, 5, 12, 4, 4, 4, 13, 6, 7, 11, 6, 9, 9, 10, 7, 12, 13, 1, 0, 1, 10, 5, 1, 12, 8, 1, 1, 14, 1, 9, 7, 1, 4, 3, 1, 2, 2, 1, 3, 4, 2, 7, 9, 2, 14, 1, 2, 8, 12, 2, 5, 10, 3, 5, 11, 3, 8, 15, 3, 14, 6, 3, 7, 0, 4, 3, 13, 4, 2, 13, 4, 4, 0, 5, 9, 6, 5, 1, 15, 5, 12, 11, 6
A037123 a(n) = a(n-1) + Sum of digits of n.
0, 1, 3, 6, 10, 15, 21, 28, 36, 45, 46, 48, 51, 55, 60, 66, 73, 81, 90, 100, 102, 105, 109, 114, 120, 127, 135, 144, 154, 165, 168, 172, 177, 183, 190, 198, 207, 217, 228, 240, 244, 249, 255, 262, 270, 279, 289, 300, 312, 325, 330, 336, 343, 351, 360, 370, 381
If you restrict yourself to polynomial functions, this is easy to code up, and only mildly tedious to solve by hand.
Let , for some unknown
Now solve the equations
…
which simply a system of linear equations.
If your data is guaranteed to be expressible as a polynomial, I think you would be able to use R (or any suite that offers regression fitting of data). If your correlation is exactly 1, then the line is a perfect fit to describe the series. There's a lot of statistics that goes into regression analysis, and I am not familiar enough with even the basics of calculation to give you much detail. But, this link to regression analysis in R might be of assistance
The Axiom computer algebra system includes a package for this purpose. You can read its documentation here. Here's the output for your example sequence in FriCAS (a fork of Axiom): (3) -> guess([1, 3, 6, 10, 15]) 2 n + 3n + 2 (3) [[function= -----------,order= 0]] 2 Type: List(Record(function: Expression(Integer),order: NonNegativeInteger))
I think your problem is ill-posed. Given any finite number of integers in a sequence with no generating function, the next element can be anything. You need to assume something about the sequence. Is it geometric? Arithmetic?
If your sequence comes from a polynomial then divided differences will find that polynomial expressed in terms of the Newton basis or binomial basis. See this.
There is no general answers; a simple method can be implemented bu using Pade approximants; in two words, assume your sequence is a sequence of coefficients of the Taylor expansion of an unknown function, then apply an algorithm (similar to the continued-fraction algorithm) in order to "simplify" this Taylor-expansion (more precisely: find a rational function very close to the initial (and truncated) function. The Maxima program can do it: look at "pade" on the page: http://maxima.sourceforge.net/docs/manual/maxima_28.html Another answer tells about the "guess" package in the FriCAS fork of Axiom (see previous answer by jmbr). If I am not wrong; this package is itself inspired from the Rate program by Christian Krattenthaler; you can find it here: http://www.mat.univie.ac.at/~kratt/rate/rate.html Maybe looking at its source could tell you about other methods.