I'm using jsLPSolver to solve an integer-programming problem.
I'm having trouble adjusting the model to contain incompatibility constraints.
I've got the following model:
{
"optimize": "cost",
"opType": "min",
"constraints": {
"c1": { "min": 36000.0, "max": 36800.0 },
"c2": { "min": 12000.0, "max": 12800.0 },
"c3": { "equal": 1000.0 }
},
"variables": {
"p1": { "c1": 0, "c2": 0, "c3": 1, "cost": 437.47, },
"p2": { "c1": 0, "c2": 60.0, "c3": 1, "cost": 1964.49, },
"p3": { "c1": 34.0, "c2": 0, "c3": 1, "cost": 1428.98, },
"p4": { "c1": 46.0, "c2": 0, "c3": 1, "cost": 1973.11, }
},
"ints": { "p1": 1, "p2": 1, "p3": 1, "p4": 1 }
}
and the feasible result
{ bounded: true, feasible: true, p2: 200, p3: 66, p4: 734, result: 1935473.42 }
However, it exists a constraint that p3 and p4 could not be together in the solution, because they are incompatible.
Is it possible to define an incompatibility constraint to define that p3 and p4 is incompatible variables?
EDIT
I'm thinking about use a constraint like p3 + p4 = 0:
{
"optimize": "cost",
"opType": "min",
"constraints": {
"c1": { "min": 36000.0, "max": 36800.0 },
"c2": { "min": 12000.0, "max": 12800.0 },
"c3": { "equal": 1000.0 },
"incompatible": { "equal": 0.0 }
},
"variables": {
"p1": { "c1": 0, "c2": 0, "c3": 1, "cost": 437.47, "incompatible": 0.0 },
"p2": { "c1": 0, "c2": 60.0, "c3": 1, "cost": 1964.49, "incompatible": 0.0 },
"p3": { "c1": 34.0, "c2": 0, "c3": 1, "cost": 1428.98, "incompatible": 1.0 },
"p4": { "c1": 46.0, "c2": 0, "c3": 1, "cost": 1973.11, "incompatible": 1.0 }
},
"ints": { "p1": 1, "p2": 1, "p3": 1, "p4": 1 }
}
but see what happens to the solution in this case:
{ bounded: true, feasible: false, p2: 200, p3: -3000, p4: 3000, result: 0 }
as seen on https://runkit.com/tetrimesquita/incompatible-contraint, which is correct but unfeasible.
I see now that p3 and p4 are continuous. (If they are actually binary, see this answer.) In that case, add two new binary variables, say z3 and z4, that equal 1 if p3 and p4 (respectively) are greater than zero:
p3 <= Mz3
p4 <= Mz4
where M is a large number. Then add a constraint
z3 + z4 <= 1
The first two constraints say that if p3 > 0, then z3 must equal 1 (and similarly for p4 and z4). The third constraint says at most one of z3 and z4 can equal 1, i.e., at most one of p3 and p4 can be positive.
If p3 and p4 are unrestricted (allowed to be positive or negative) and you want to require at most one of them to be nonzero, then also add:
-p3 <= Mz3
-p4 <= Mz4
Probably not the most elegant version out there, but this should do it:
var solver = require("javascript-lp-solver");
var model = {
"optimize": "cost",
"opType": "min",
"constraints": {
"c1": { "min": 36000.0, "max": 36800.0 },
"c2": { "min": 12000.0, "max": 12800.0 },
"c3": { "equal": 1000.0 }
},
"variables": {
"p1": { "c1": 0, "c2": 0, "c3": 1, "cost": 437.47, },
"p2": { "c1": 0, "c2": 60.0, "c3": 1, "cost": 1964.49, },
"p3": { "c1": 34.0, "c2": 0, "c3": 1, "cost": 1428.98, },
},
"ints": { "p1": 1, "p2": 1, "p3": 1}
}
var results_p3 = solver.Solve(model);
var model = {
"optimize": "cost",
"opType": "min",
"constraints": {
"c1": { "min": 36000.0, "max": 36800.0 },
"c2": { "min": 12000.0, "max": 12800.0 },
"c3": { "equal": 1000.0 }
},
"variables": {
"p1": { "c1": 0, "c2": 0, "c3": 1, "cost": 437.47, },
"p2": { "c1": 0, "c2": 60.0, "c3": 1, "cost": 1964.49, },
"p4": { "c1": 46.0, "c2": 0, "c3": 1, "cost": 1973.11, }
},
"ints": { "p1": 1, "p2": 1, "p4": 1 }
}
var results_p4 = solver.Solve(model);
var finalResults = (results_p3.feasible && (results_p3.result < results_p4.result)) ? results_p3 : results_p4;
console.log(results_p3);
console.log(results_p4);
console.log(finalResults);
The idea is that you run the model twice - once where p3 is fixed, and once where p4 is fixed. Then you take the solution that obeys the constraints and yields the better result [1].
See output here:
https://runkit.com/dionhaefner/so-incompatible-constraint
[1] You should probably polish that last check a little. E.g., what should happen if both results are infeasible? What if both have a result of 0? You might have to check the bounded property, too.
If p3 and p4 are binary, you can just use
p3 + p4 <= 1
Or if you want exactly one of them to equal 1, then use
p3 + p4 = 1
I'm not familiar with the syntax for jsLPSolver, but the constraints above provide the logic.
Related
I can see that Elasticsearch support both Lucene syntax and it's own query language.
You can use both and get same kinds of results.
Example (might be done differently maybe but to show what I mean):
Both of these queries produce the same result but use Lucene or Elastic query syntax.
GET /index/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "field101:Denmark"
}
}
]
}
}
}
GET /index/_search
{
"query": {
"match": {
"field101": {
"query": "Denmark"
}
}
}
}
I was wondering are there any kind of implications when choosing one approach over the other (like performance or some kinds of optimizations)? Or is Elastic query syntax just translated to Lucene query somewhere since Elastic runs Lucene as its underlying search engine ?
I was wondering are there any kind of implications when choosing one approach over the other (like performance or some kinds of optimizations)?
Elasticsearch DSL will convert into Lucene query under the hood, you can set "profile":true in the query to see how that works and exactly how much time it takes to convert.
I would say there are no important performance implications and you should always use the DSL, because in many cases Elasticsearch will do optimizations for you. Also, query_string will expect well written Lucene queries, and you can have syntax errors (try doing "Denmark AND" as query_string.
Or is Elastic query syntax just translated to Lucene query somewhere since Elastic runs Lucene as its underlying search engine ?
Yes. You can try it yourself:
GET test_lucene/_search
{
"profile": true,
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "field101:Denmark"
}
}
]
}
}
}
will produce:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
},
"profile": {
"shards": [
{
"id": "[KGaFbXIKTVOjPDR0GrI4Dw][test_lucene][0]",
"searches": [
{
"query": [
{
"type": "TermQuery",
"description": "field101:denmark",
"time_in_nanos": 3143,
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 0,
"shallow_advance_count": 0,
"set_min_competitive_score": 0,
"next_doc": 0,
"match": 0,
"next_doc_count": 0,
"score_count": 0,
"compute_max_score_count": 0,
"compute_max_score": 0,
"advance": 0,
"advance_count": 0,
"score": 0,
"build_scorer_count": 0,
"create_weight": 3143,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 0
}
}
],
"rewrite_time": 2531,
"collector": [
{
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 1115
}
]
}
],
"aggregations": []
}
]
}
}
And
GET /test_lucene/_search
{
"profile": true,
"query": {
"match": {
"field101": {
"query": "Denmark"
}
}
}
}
Will produce the same
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
},
"profile": {
"shards": [
{
"id": "[KGaFbXIKTVOjPDR0GrI4Dw][test_lucene][0]",
"searches": [
{
"query": [
{
"type": "TermQuery",
"description": "field101:denmark",
"time_in_nanos": 3775,
"breakdown": {
"set_min_competitive_score_count": 0,
"match_count": 0,
"shallow_advance_count": 0,
"set_min_competitive_score": 0,
"next_doc": 0,
"match": 0,
"next_doc_count": 0,
"score_count": 0,
"compute_max_score_count": 0,
"compute_max_score": 0,
"advance": 0,
"advance_count": 0,
"score": 0,
"build_scorer_count": 0,
"create_weight": 3775,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 0
}
}
],
"rewrite_time": 3483,
"collector": [
{
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 1780
}
]
}
],
"aggregations": []
}
]
}
}
As you see, times are in nanoseconds, not even miliseconds, that says conversion is fast.
You can read more about here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
Right now I have a repository method annotated with #RestResource(path = "xx") which return a json in the following form
{
"_embedded": {
"myResponse": []
},
"_links": {},
"page": {
"size": 24,
"totalElements": 238,
"totalPages": 10,
"number": 0
}
}
but I would like something with the form :
{
"content": [],
"pageable": {
"sort": {
"sorted": false,
"unsorted": true,
"empty": true
},
"pageSize": 24,
"pageNumber": 0,
"offset": 0,
"paged": true,
"unpaged": false
},
"last": false,
"totalElements": 238,
"totalPages": 10,
"size": 24,
"number": 0,
"first": true,
"numberOfElements": 24,
"sort": {
"sorted": false,
"unsorted": true,
"empty": true
},
"empty": false
}
But when I create a GetMapping in my controller which is simply calling my repo method like so :
repo.findByTitleContaining(searchedWord, PageRequest.of(page, size));
I have the desired form of response..
My question is : is there a way to get it directly from the repo with restresource annotation ?
I am using Mesos 1.0.1. I have added an agent with a new role docker_gpu_worker. I register a framework with this role. The framework does not receive any offers. Other frameworks (same Java code) using other roles are working fine. I have not restarted the three Mesos masters. Does anyone have an idea about what might be going wrong?
At master/frameworks, I see my framework:
"{
"id": "fd01b1b0-eb73-4d40-8774-009171ae1db1-0701",
"name": "/data4/Users/mikeb/jobs/999",
"pid": "scheduler-77345362-b85c-4044-8db5-0106b9015119#x.x.x.x:57617",
"used_resources": {
"disk": 0,
"mem": 0,
"gpus": 0,
"cpus": 0
},
"offered_resources": {
"disk": 0,
"mem": 0,
"gpus": 0,
"cpus": 0
},
"capabilities": [],
"hostname": "x-x-x-x.ec2.internal",
"webui_url": "",
"active": true,
"user": "mikeb",
"failover_timeout": 10080,
"checkpoint": true,
"role": "docker_gpu_worker",
"registered_time": 1507028279.18887,
"unregistered_time": 0,
"principal": "test-framework-java",
"resources": {
"disk": 0,
"mem": 0,
"gpus": 0,
"cpus": 0
},
"tasks": [],
"completed_tasks": [],
"offers": [],
"executors": []
}"
At master/roles I see my role:
"{
"frameworks": [
"fd01b1b0-eb73-4d40-8774-009171ae1db1-0701",
"fd01b1b0-eb73-4d40-8774-009171ae1db1-0673",
"fd01b1b0-eb73-4d40-8774-009171ae1db1-0335"
],
"name": "docker_gpu_worker",
"resources": {
"cpus": 0,
"disk": 0,
"gpus": 0,
"mem": 0
},
"weight": 1
}"
At master/slaves I see my agent:
"{
"id": "fd01b1b0-eb73-4d40-8774-009171ae1db1-S5454",
"pid": "slave(1)#x.x.x.x:5051",
"hostname": "x-x-x-x.ec2.internal",
"registered_time": 1506692213.24938,
"resources": {
"disk": 35056,
"mem": 59363,
"gpus": 4,
"cpus": 32,
"ports": "[31000-32000]"
},
"used_resources": {
"disk": 0,
"mem": 0,
"gpus": 0,
"cpus": 0
},
"offered_resources": {
"disk": 0,
"mem": 0,
"gpus": 0,
"cpus": 0
},
"reserved_resources": {
"docker_gpu_worker": {
"disk": 35056,
"mem": 59363,
"gpus": 4,
"cpus": 32,
"ports": "[31000-32000]"
}
},
"unreserved_resources": {
"disk": 0,
"mem": 0,
"gpus": 0,
"cpus": 0
},
"attributes": {},
"active": true,
"version": "1.0.1",
"reserved_resources_full": {
"docker_gpu_worker": [
{
"name": "gpus",
"type": "SCALAR",
"scalar": {
"value": 4
},
"role": "docker_gpu_worker"
},
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 32
},
"role": "docker_gpu_worker"
},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 59363
},
"role": "docker_gpu_worker"
},
{
"name": "disk",
"type": "SCALAR",
"scalar": {
"value": 35056
},
"role": "docker_gpu_worker"
},
{
"name": "ports",
"type": "RANGES",
"ranges": {
"range": [
{
"begin": 31000,
"end": 32000
}
]
},
"role": "docker_gpu_worker"
}
]
},
"used_resources_full": [],
"offered_resources_full": []
}"
We have tracked the problem to this Mesos agent config:
--isolation="filesystem/linux,cgroups/devices,gpu/nvidia"
Removing that, the agent works properly, but without access to GPU resources. This config is a requirement according to the docs for Nvidia GPU support and those docs seem to indicate that version 1.0.1 supports it. We are continuing to investigate.
The GPU_RESOURCES capability must be enabled for frameworks.
As illustrated in http://mesos.readthedocs.io/en/latest/gpu-support/,
this can be achieved for example by specifying --framework_capabilities="GPU_RESOURCES" in the mesos-execute command, or with code like this in C++:
FrameworkInfo framework;
framework.add_capabilities()->set_type(
FrameworkInfo::Capability::GPU_RESOURCES);
For Marathon frameworks instead, the Marathon service must be started with the option --enable_features "gpu_resources" as indicated in Enable GPU resources (CUDA) on DC/OS
you can register the roles with master statically,
if you add an agent role at run time it would not be known to master
and it would require mesos master restart for master to see this role.
Try restarting the mesos master.
I want to use Elasticsearch 5's profile api to find the search execution plan.
Here are my two different querys:
1:
"query": {
"bool": {
"should": [
{
"term": {
"field1": "f1"
}
},
{
"range": {
"field3": {
"gt": "1"
}
}
}
]
}
}
2:
"query": {
"bool": {
"filter": [
{
"term": {
"field1": "f1"
}
},
{
"range": {
"field3": {
"gt": "1"
}
}
}
]
}
}
The difference between the two queries is just that query1 uses should and query2 use filter.
But the children profile results of two queries are the same.
Such as:
"children": [
{
"type": "BooleanQuery",
"description": "field1:f1 field3:[2 TO 9223372036854775807]",
"time": "0.06531500000ms",
"breakdown": {
"score": 0,
"build_scorer_count": 0,
"match_count": 0,
"create_weight": 65314,
"next_doc": 0,
"match": 0,
"create_weight_count": 1,
"next_doc_count": 0,
"score_count": 0,
"build_scorer": 0,
"advance": 0,
"advance_count": 0
},
"children": [
{
"type": "TermQuery",
"description": "field1:f1",
"time": "0.05287500000ms",
"breakdown": {
"score": 0,
"build_scorer_count": 0,
"match_count": 0,
"create_weight": 52874,
"next_doc": 0,
"match": 0,
"create_weight_count": 1,
"next_doc_count": 0,
"score_count": 0,
"build_scorer": 0,
"advance": 0,
"advance_count": 0
}
},
{
"type": "",
"description": "field3:[2 TO 9223372036854775807]",
"time": "0.001556000000ms",
"breakdown": {
"score": 0,
"build_scorer_count": 0,
"match_count": 0,
"create_weight": 1555,
"next_doc": 0,
"match": 0,
"create_weight_count": 1,
"next_doc_count": 0,
"score_count": 0,
"build_scorer": 0,
"advance": 0,
"advance_count": 0
}
}
]
}]
Does ElasticSearch create multiple threads, then each thread executes one filter item and then the results are merged in the end?
But in my mind,i think es will use a filter pipeline,execute filter items one by one.Such as elasticsearch-order-of-filters-for-best-performance says
In case 1, the execution will be slower because all documents from the past month will need to go through filter A first, which is not cached.
In case 2, you first filter out all the documents without the type XYZ, which is fast because filter B is cached. Then the documents that made it through filter B can go through filter A. So even though filter A is not cached, the execution will still be faster since there are mush less documents left in the filter pipeline.
How es to execute multiple filter item?
Contrary to the current ES documentation http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-mlt-field-query.html, stop_words is an unsupported field for more_like_this_field queries in my ES installation.
version: 0.90.5
build_hash: c8714e8e0620b62638f660f6144831792b9dedee
build_timestamp: 2013-09-17T12:50:20Z
build_snapshot: false
lucene_version: 4.4
can anyone confirm this?
This is what I am sending to the server
{
"query": {
"bool": {
"must": [
{
"match_all": {
"boost": 1
}
},
{
"more_like_this_field": {
"FieldA": {
"like_text": "House",
"boost": 1,
"min_doc_freq": 0,
"min_word_len": 0,
"min_term_freq": 0
}
}
}
],
"should": [
{
"more_like_this_field": {
"Equipped": {
"like_text": "pool garage",
"boost": 0,
"min_doc_freq": 0,
"min_word_len": 0,
"min_term_freq": 0,
"stop_words": "garden"
}
}
},
{
"more_like_this_field": {
"Neighbourhood": {
"like_text": "school",
"boost": 5,
"min_doc_freq": 0,
"min_word_len": 0,
"min_term_freq": 0
}
}
}
],
"minimum_number_should_match": 2
}
}
}
and this is what I get back
QueryParsingException[[data] [mlt_field] query does not support [stop_words]];
Same happens with the more_like_this query, the reason is that stop_words must be an array.
{
"more_like_this_field": {
"Equipped": {
"like_text": "pool garage",
"boost": 0,
"min_doc_freq": 0,
"min_word_len": 0,
"min_term_freq": 0,
"stop_words": ["garden"]
}
}
}