Elasticsearch: empty slices when using scroll api with slice - elasticsearch

I have a simple setup with a green cluster (v7.5.2) of 1 instance with 1 index (replica 0) with 8 shards.
61,500 documents indexed.
If I create 8 slices with these queries (POST queries)
http://localhost:9202/products_dev/_search?scroll=10m: {'slice': {'field': 'created_at', 'id': 0, 'max': 8}, 'size': 1000}
http://localhost:9202/products_dev/_search?scroll=10m: {'slice': {'field': 'created_at', 'id': 1, 'max': 8}, 'size': 1000}
...
http://localhost:9202/products_dev/_search?scroll=10m: {'slice': {'field': 'created_at', 'id': 7, 'max': 8}, 'size': 1000}
for each slice I collect the first hits of each slice and get those lengths:
[1000, 0, 0, 0, 0, 0, 0, 0]
Only 1 slice has results which is equivalent to not slicing the index in the first place.
I tried a max of 32 with those commands:
http://localhost:9202/products_dev/_search?scroll=10m: {'slice': {'field': 'created_at', 'id': 0, 'max': 32}, 'size': 1000}
http://localhost:9202/products_dev/_search?scroll=10m: {'slice': {'field': 'created_at', 'id': 1, 'max': 32}, 'size': 1000}
...
http://localhost:9202/products_dev/_search?scroll=10m: {'slice': {'field': 'created_at', 'id': 31, 'max': 32}, 'size': 1000}
for each slice I collect the first hits of each slice and get those lengths:
[1000, 0, 0, 0, 0, 0, 0, 0, 1000, 0, 0, 0, 0, 0, 0, 0, 1000, 0, 0, 0, 0, 0, 0, 0, 1000, 0, 0, 0, 0, 0, 0, 0]
This way I can scroll my index using 4 different (non-empty) slices but I had to create 32 contexts which is not ideal.
What am I doing wrong?
I tried to use a date as slice.field
"slice": {
"field": "created_at",
but it did not help.


My local index had so few documents, most of the documents where saved in 1 shard

Related

Laravel sortBy multiple values and multiple keys

I'm trying to figure out the best way to sort by multiple values and multiple keys at once.
I have the following "sort sequence";
sort by "finished or not"
sort by "out of time or not"
sort by "points in total"
sort by "points for your time"
sort by "points for each 'secret time control', can be null, then you end up at the end of that sorting"
sort by "the distance driven"
I currently have something like this of response (sorted);
[
{
"team_number": 201,
"points_for_time": 0,
"detail_points_for_time": {
"tc_start": 0,
"tc_round_in": 0,
"tc_stop": 0
},
"points_for_gtc": 1,
"detail_points_for_gtc": [
0,
1
],
"points_for_distance": 0,
"missed_controls": 100,
"out_of_time": false,
"dnf": false,
"total": 101
},
{
"team_number": 202,
"points_for_time": 2,
"detail_points_for_time": {
"tc_start": 0,
"tc_round_in": 0,
"tc_stop": 2
},
"points_for_gtc": 0,
"detail_points_for_gtc": [],
"points_for_distance": 0,
"missed_controls": 100,
"out_of_time": false,
"dnf": false,
"total": 102
},
{
"team_number": 203,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
},
{
"team_number": 204,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
},
{
"team_number": 205,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
},
{
"team_number": 206,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
},
{
"team_number": 207,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
},
{
"team_number": 208,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
},
{
"team_number": 209,
"points_for_time": 0,
"detail_points_for_time": 0,
"points_for_gtc": 0,
"detail_points_for_gtc": 0,
"points_for_distance": 0,
"missed_controls": 0,
"out_of_time": false,
"dnf": true,
"total": 0
}
]
So the goal here is, that I get a full sorting based on the above "sequence".
I currentmy use this;
$results = $results->sortBy('total')
->sortBy('points_for_time')
->sortBy('points_for_gtc') // this is an issue, since it puts you below someone else when you have less points
->sortBy('points_for_distance')
->sortBy('out_of_time')
->sortBy('dnf');
There is another "field" I have, detail_points_for_gtc, which holds the points (in order) for each "time control".
The possibility exists that there is nothing filled in, if so, then you should be at the end of the list. it's also possible that the value is null. Then it should be sorted at the end.
I have no idea on how I can sort on the detail_points_for_gtc, and if this sorting is "the correct sorting"?
Anyone who understands the question and may be able to help?

I think you can try this;
// you can send column and direction from request or where you need
$results = $request->direction == 'asc' ? $results->sortBy($request->column) : $results->sortByDesc($request->column);

how to add to different field values and store it in another field in elastic search python (elastic search field operation)

I have written this elastic search query:
es.search(index=['ind1'],doc_type=['doc'])
I am getting following result:
{'_shards': {'failed': 0, 'skipped': 0, 'successful': 5, 'total': 5},
'hits': {'hits': [{'_id': '1327',
'_index': 'ind1',
'_score': 1.0,
'_source': {'val1': 1,
'val2': None,
'value1': 1327,
'value2': 1531,
'new_values': {'nv1': 1,
'nv2': 0},
{'_id': '1349',
'_index': 'ind1',
'_score': 1.0,
'_source': {'val1': 2,
'val2': 3,
'value1': 1328,
'value2': 1539,
'new_values': {'nv1': 1,
'nv2': 3}},.......
I want all the add val1, value1 and nv1 and store it in another field Let's call total. I want the result will be like:
{'_shards': {'failed': 0, 'skipped': 0, 'successful': 5, 'total': 5},
'hits': {'hits': [{'_id': '1327',
'_index': 'ind1',
'_score': 1.0,
'_source': {'val1': 1,
'val2': None,
'value1': 1327,
'value2': 1531,
'new_values': {'nv1': 1,
'nv2': 0},
'total1': 1329},
{'_id': '1349',
'_index': 'ind1',
'_score': 1.0,
'_source': {'val1': 2,
'val2': 3,
'value1': 1328,
'value2': 1539,
'new_values': {'nv1': 1,
'nv2': 3},
'total': 1331},.......

What you can do is use a script field in order to compute that value on the fly:
es.search(index=['ind1'],doc_type=['doc'], body={
'_source': true,
'script_fields': {
'total': {
'script': {
'source': 'doc.val1 + doc.value1 + doc.new_values.nv1'
}
}
}
})
The best way, though, would be to pre-compute that at index time and store the value in a new total field.

JQPlot Grouped Stacked graph with different colours

I am trying to produce the following graph with JQPlot
I have attempted this as best I can however the script causes an excsessive runtime and eventually firefox comes up with the stop script notice.
Any suggestions?
$(document).ready(function () {
var assigned1 = [5, 12, 2, 0, 0, 4, 8];
var assigned2 = [4, 0, 2, 0, 0, 2, 4];
var assigned3 = [1, 0, 2, 0, 0, 2, 0];
var assigned4 = [0, 0, 5, 0, 0, 3, 0];
var assigned5 = [0, 0, 1, 0, 0, 0, 0];
var assigned = [assigned1, assigned2, assigned3, assigned4, assigned5];
var numCompleted = [assigned1, assigned2, assigned3, assigned4, assigned5]; //[4,3,6,5,5,4,0,0,5,4,0,0,0,0];
//[1,1,0,0,2,2,8,7,4,4,11,10,1,1];
//var test3 = [4,6,3,8,6,3,1,9,1,5,3,6,7,4];
var dates = ['Assigned', 'Employee', 'Machine', 'Assigned', 'Employee', 'Machine', 'Assigned', 'Employee', 'Machine', 'Assigned', 'Employee', 'Machine', 'Assigned', 'Employee', 'Machine', 'Assigned', 'Employee', 'Machine', 'Assigned', 'Employee', 'Machine'];
plot3 = $.jqplot('chart3', [assigned, numCompleted], {
// Tell the plot to stack the bars.
seriesColors: ["#81a7d4", "#d0d0d0", "#fac"],
stackSeries: true,
shadow: false,
title: 'Cell Name',
animate: false,
captureRightClick: false,
grid: {
drawGridLines: false,
shadow: false
},
legend: {
show: false,
},
seriesDefaults: {
renderer: $.jqplot.BarRenderer,
rendererOptions: {
barMargin: 10,
barPadding: 0,
shadowOffset: 0,
groups: 7,
highlightMouseDown: true
},
pointLabels: {
show: true,
hideZeros: true,
edgeTolerance: 10
}
},
grid: {
gridLineColor: '#fff',
borderWidth: 0,
shadow: false,
background: "white"
},
axes: {
xaxis: {
renderer: $.jqplot.CategoryAxisRenderer,
ticks: dates,
showTickMarks: false
},
yaxis: {
showTicks: false
}
}
});
});
http://jsfiddle.net/K5cjj/1/

I have managed to do this by drawing multiple graphs to the same element. This has other issue but appears to work, can anyone expand on this to enable the click function to work on all bars
However this works in Firefox, chrome and IE11 only, IE 10 and below is all messed up (as usual)
http://jsfiddle.net/LADzf/1
$(document).ready(function(){
/* graph config */
var maxVal = 13;
/* graph vals */
var assigned1 = [5, 0, 0, 0, 12, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 8, 0, 0];
var assigned2 = [3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 2, 0, 0];
var assigned3 = [2, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0];
var assigned4 = [0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
var employee1 = [0, 5, 0, 0, 0, 10, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 12, 0];
var employee2 = [0, 5, 0, 0, 0, 2, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
var machine = [0, 0, 12, 0, 0, 0, 12, 0, 0, 0, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 12];
var basevals = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
/* graph ticks */
var ticks = ['Assigned', 'Employee', 'Machine', ' ', 'Assigned', 'Employee', 'Machine', ' ', 'Assigned', 'Employee', 'Machine', ' ', 'Assigned', 'Employee', 'Machine', ' ', 'Assigned', 'Employee', 'Machine', ' ', 'Assigned', 'Employee', 'Machine', ' ', 'Assigned', 'Employee', 'Machine'];
var blankticks = [' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '];
var dateticks = ['<br><br><br><br>01/01/2014', '<br><br><br><br>02/01/2014', '<br><br><br><br>03/01/2014', '<br><br><br><br>04/01/2014', '<br><br><br><br>05/01/2014', '<br><br><br><br>06/01/2014', '<br><br><br><br>07/01/2014'];
/* plot the base graph */
plotbase = $.jqplot('chart3', [basevals], {
seriesDefaults:{
renderer:$.jqplot.BarRenderer,
rendererOptions: {barMargin: 10},
pointLabels: {show: false}
},
axesDefaults: {show: false},
tickOptions: {showMark: false, angle: 90},
axes: {
showLabel: false,
xaxis: {
renderer: $.jqplot.CategoryAxisRenderer,
ticks: dateticks,
tickOptions: {markSize: 0}
},
yaxis: {
padMin: 0,
min: 0,
max: maxVal,
showLabel: false,
show: false
}
}
});
plot3 = $.jqplot('chart3', [machine], {
seriesColors: ["#a62525"],
stackSeries: true,
captureRightClick: true,
seriesDefaults:{
renderer:$.jqplot.BarRenderer,
rendererOptions: {
barMargin: 10,
highlightMouseOver: true
},
pointLabels: {show: false}
},
axesDefaults: {show: false},
tickOptions: {showMark: false, angle: 90},
axes: {
showLabel: false,
xaxis: {
renderer: $.jqplot.CategoryAxisRenderer,
ticks: blankticks,
tickOptions: {markSize: 0}
},
yaxis: {
padMin: 0,
min: 0,
max: maxVal,
showLabel: false,
show: false
}
}
});
plot2 = $.jqplot('chart3', [employee1, employee2], {
seriesColors: ["#67ce64", "#da9831"],
stackSeries: true,
captureRightClick: true,
seriesDefaults:{
renderer:$.jqplot.BarRenderer,
rendererOptions: {barMargin: 10, highlightMouseOver: true},
pointLabels: {show: false}
},
axesDefaults: {show: false},
tickOptions: {showMark: false},
axes: {
showLabel: false,
xaxis: {
renderer: $.jqplot.CategoryAxisRenderer,
ticks: blankticks
},
yaxis: {
padMin: 0,
min: 0,
max: maxVal,
showLabel: false,
show: false
}
},
grid: {background: 'transparent', drawGridLines: false, gridLineColor: 'transparent', borderColor: 'transparent'}
});
plot1 = $.jqplot('chart3', [assigned1, assigned2, assigned3, assigned4], {
stackSeries: true,
captureRightClick: true,
seriesColors: ["#effa38", "#37d1f8", "#5129b6"],
seriesDefaults:{
renderer:$.jqplot.BarRenderer,
rendererOptions: {barMargin: 10, highlightMouseOver: true },
pointLabels: {show: false}
},
axes: {
xaxis: {
renderer: $.jqplot.CategoryAxisRenderer,
tickRenderer:$.jqplot.CanvasAxisTickRenderer,
ticks: ticks,
tickOptions: {
angle: -90,
}
},
yaxis: {
padMin: 0,
min: 0,
max: maxVal
}
}, grid: {background: 'transparent', drawGridLines: false, gridLineColor: 'transparent', borderColor: 'transparent'}
});
$('#chart3').bind('jqplotDataClick', function (ev, seriesIndex, pointIndex, data) {
$('#info3').html('series: '+seriesIndex+', point: '+pointIndex+', data: '+data);
});
});

Obtain sub-cv::Mat from a given image such that

I have a processed an image that returns a Mat say:
[
[ 0, 0, 255, 255, 255, 0, 0, 0],
[ 0, 0, 0, 255, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 255, 255, 0],
[ 0, 0, 0, 0, 0, 255, 255, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0]
]
are there any segmentation functions that can help me isolate individually ?
if not so what would be the best approach ?
i.e. say if i do mysegment(Mat a,0) should return
[
[ 0, 0, 255, 255, 255, 0, 0, 0],
[ 0, 0, 0, 255, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0]
]
then mysegment(Mat a,1) will return
[
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 255, 255, 0],
[ 0, 0, 0, 0, 0, 255, 255, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0]
]
I consider continuous 255 pixels to be a block. Now I want to iterate over or process these blocks ... since more than one block can be formed in a matrix.

You can use contours
findContours( canny_output, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0) );
the link to the library

3D skew transformation matrix along one coordinate axis

Is there a way to calculate the skew transformation matrix along one coordinate axis, given the skew angle, as follows

This should work for the most part for skewing an object with a transformation matrix, in particular using glMultMatrix(matrix)
matrix1[] = {
1, 0, 0, 0,
tan(a), 1, 0, 0,
0, 0, 1, 0,
0, 0, 0, 1
};
matrix2[] = {
1, 0, 0, 0,
0, 1, 0, 0,
tan(a), 0, 1, 0,
0, 0, 0, 1
};
matrix3[] = {
1, tan(a), 0, 0,
0, 1, 0, 0,
0, 0, 1, 0,
0, 0, 0, 1
};
matrix4[] = {
1, 0, 0, 0,
0, 1, 0, 0,
0, tan(a), 1, 0,
0, 0, 0, 1
};
matrix5[] = {
1, 0, tan(a), 0,
0, 1, 0, 0,
0, 0, 1, 0,
0, 0, 0, 1
};
matrix6[] = {
1, 0, 0, 0,
0, 1, tan(a), 0,
0, 0, 1, 0,
0, 0, 0, 1
};

Resources