I have created an elasticsearch index against a news table on sqlserver using logstash via the JDBC Driver. This all looks good in Elasticsearch.
Using Index Server, the type of query that gets built for that takes the following form:
SELECT News.*, fulltextsearch.rank FROM News INNER JOIN CONTAINSTABLE(News, ( Headline, BodyText ), 'ISABOUT("car track race" WEIGHT(0.65), car NEAR track NEAR race)') fulltextsearch ON News.NewsID = fulltextsearch.[Key] WHERE DateSubmitted <= '01/11/2017' ORDER BY fulltextsearch.rank DESC
Is there any kind of query that I can do in Elasticsearch to give a similar/same outcome as the above.
No, elasticsearch (version 5.3) do not support JOIN like this. See https://www.elastic.co/guide/en/elasticsearch/reference/current/joining-queries.html.
Related
can anyone help me to translate below SQL query to ELasticSearch DSL form for document selections?
SELECT * FROM table1 WHERE source_id IN (select source_id FROM table2)
This JOIN semantics does not translate to Elasticsearch. If your index had parent/child mapping - you might be able to achieve what you want to do. Another option is to do a two-pass query.
This blog post gives an idea of how relations are modeled in Elasticsearch - Managing Relations in Elasticsearch
I have data set(questions) which are mapped to multiple tags and these tags are hierarchical in nature.
So there is A question which is mapped to t1 and t2 tag.
t1 has parent p1 and p1 has parent p2.(p2 -> p1 - >t1 --mapped to--->A)
So i was storing my data in neo4j and I want to get A as result for p2 tag. I am getting result easily using cypher. but now i have sort and limit by in the same query and since neo4j cant use index in such queries, i am thinking of integrating neo4j with elasticsearch, but I am not able to get how to query?
$query = "MATCH p=(n:messages)-[r:TAGGED_TO]->(k:tags{tag_id:{tag_id}}) RETURN p,n ORDER by n.msgId desc limit 5";
$params['tag_id'] = (int)$tag_id;
$result = $this->dbHandle->run($query,$params);
Now sort and limit are not using index. I want to run this query in optimized way.
You can use Graphaware plugin for connecting neo4j to elastic or the apoc plugin,specifically apoc.es.* functions... see documentation for more.
I going to do a query that related to two _index ( conn, http). In sql it can be do like following, may I know how DSL query like in elasticSearch ? Please assist me, I really out of idea.
SELECT * FROM elastic INNER JOIN conn ON conn.uid=http.uid
WHERE elastic.date BETWEEN '2016-08-20 00:00:00' and '2016-08-22 13:41:09'
AND conn.service='http' ;
elasticsearch doesn't have support for joins like SQL.
They discuss the handling of relationships in the definitive guide: https://www.elastic.co/guide/en/elasticsearch/guide/current/relations.html
Can you nest queries logically in ElasticSearch, so the output of one query is the input to another query.
Another, way to ask is how can I chain or pipe queries together?
This should be analogous to the IN operator or subqueries in SQL
i.e.:-
select au_lname, au_fname, title from
(select au_lname, au_fname, au_id from pubs.dbo.authors
where state = 'CA')
or
SELECT Name
FROM AdventureWorks2008R2.Production.Product
WHERE ListPrice =
(SELECT ListPrice
FROM AdventureWorks2008R2.Production.Product
WHERE Name = 'Chainring Bolts' );
Elasticsearch doesn't support subqueries; you would need to perform your first query, then construct a second query using the results of the first query as an input.
this is not supported in elastic-search you must normalize your data and have all field you need in one setting
That is totally correct, you must programm a subquery in your favorite programming language. An example can be found here:
https://sebastianviereck.de/elasticsearch-subquery-scoring-optimization/
I'm working on a simple side project, and have a tech stack that involves both a SQL database and ElasticSearch. I only have ElasticSearch because I assumed that as my project grows, my full text searching would be most efficiently performed by ES. My ES schema is very simple - documents that I insert into ES have 2 fields, one being the id and the other being the field with the body of text to search. The id being inserted into ES corresponds to that document's primary key id from the SQL database.
insert record into SQL -> insert record into ES using PK from SQL
Searching would be the reverse of that. Query ES and grab all the matching ids, and then turn around and use those ids to get records from SQL.
search ES can get all PK ids -> use those ids to get documents from SQL
The problem that I am facing though, is that ES can only return documents in a paginated manner. This is a problem because I also have a WHERE clause on my SQL query, beyond just the ids. My SQL query might look like this ...
SELECT * FROM foo WHERE id IN (1,2,3,4,5) AND bar != 'baz'
Well, with ES paginating the results, my WHERE clause will always only be querying a subset of the full results from ES. Even if I utilize ES' skip and take, I'm still only querying SQL using a subset of document ids.
Is there a way to get Elastic Search to only return the entire list of matching document ids? I realize this is here to not allow me to shoot myself in the foot, because doing this across all shards and many many documents is not efficient. Is there no way, though?
After putting in some hours on this project, I've only now realized that I've poorly engineered this, unless I can get all of these ids from ES. Some alternative implementations that I've thought of would be to store the things that I'm filtering on, in SQL, in ES as well. A problem there is that I'd have to update the ES document every time I update the document in SQL. This would require a pretty big rewrite to some of my data access code. I could scrap ElasticSearch all together and just perform searching in Postgres, for now, until I can think of a better way to structure this.
The elasticsearch not support return each and every doc match to you queries. Because it Ll overload the system. Instead of this.. Use scroll concept in elasticsearch.. It's lik cursor concept in db's..
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html
For more examples refer the Github repo. https://github.com/sidharthancr/elasticsearch-java-client
Hope it helps..
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
please have a look into the elastic search document where you can specify only particular fields that return from the match documents
hope this resolves your problem
{
"fields" : ["user", "postDate"],
"query" : {
"term" : { "user" : "kimchy" }
}
}