Spring Data Solr - Multiple FilterQueries separated by OR - spring

I'm trying to implement a filter search using spring data solr. I've following filters types and all have a set of filters.
A
aa in (1,2,3)
ab between (2016-08-02 TO 2016-08-10)
B
ba in (2,3,4)
bb between (550 TO 1000)
The Solr query which I want to achieve using Spring data solr is:
q=*:*&fq=(type:A AND aa:(1,2,3) AND ab:[2016-08-02 TO 2016-08-10]) OR (type:B AND ba:(2,3,4) AND bb:[550 TO 1000])
I'm not sure how to group a number of clauses of a type of filter and then have an OR operator.
Thanks in advance.

The trick is to flag the second Criteria via setPartIsOr(true) with an OR-ish nature. This method returns void, so it cannot be chained.
First aCriteria and bCriteria are defined as described. Then bCriteria is flagged as OR-ish. Then both are added to a SimpleFilterQuery. That in turn can be added to the actual Query. That is left that out in the sample.
The DefaultQueryParser in the end is used only to generate a String that can be used in the assertion to check that the query is generated as desired.
import org.junit.jupiter.api.Test;
import org.springframework.data.solr.core.DefaultQueryParser;
import org.springframework.data.solr.core.query.Criteria;
import org.springframework.data.solr.core.query.FilterQuery;
import org.springframework.data.solr.core.query.SimpleFilterQuery;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class CriteriaTest {
#Test
public void generateQuery() {
Criteria aCriteria =
new Criteria("type").is("A")
.connect().and("aa").in(1,2,3)
.connect().and("ab").between("2016-08-02", "2016-08-10");
Criteria bCriteria =
new Criteria("type").is("B")
.connect().and("ba").in(2,3,4)
.connect().and("bb").between("550", "1000");
bCriteria.setPartIsOr(true); // that is the magic
FilterQuery filterQuery = new SimpleFilterQuery();
filterQuery.addCriteria(aCriteria);
filterQuery.addCriteria(bCriteria);
// verify the generated query string
DefaultQueryParser dqp = new DefaultQueryParser(null);
String actualQuery = dqp.getQueryString(filterQuery, null);
String expectedQuery =
"(type:A AND aa:(1 2 3) AND ab:[2016\\-08\\-02 TO 2016\\-08\\-10]) OR "
+ "((type:B AND ba:(2 3 4) AND bb:[550 TO 1000]))";
System.out.println(actualQuery);
assertEquals(expectedQuery, actualQuery);
}
}

Related

natural language logic in stanford corenlp

How does one use the natural logic component of Stanford CoreNLP?
I am using CoreNLP 3.9.1 and I fed natlog as an annotator in command line, but I don't seem to see any natlog result in the output, i.e. OperatorAnnotation and PolarityAnnotation, according to this link. Does that have anything to do with the outputFormat? I've tried xml and json, but neither has any output on natural logic. The other stuff (tokenization, dep parse) is in there though.
Here is my command:
./corenlp.sh -annotators tokenize,ssplit,pos,lemma,depparse,natlog -file natlog.test -outputFormat xml
Thanks in advance.
I don't think any of the output options show the natlog stuff. This is more designed if you have a Java system and are working with the Annotations themselves in Java code. You should be able to see them by looking at the CoreLabel for each token.
This code snippet works for me:
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.pipeline.StanfordCoreNLP;
import edu.stanford.nlp.util.CoreMap;
import edu.stanford.nlp.ling.CoreAnnotations.NamedEntityTagAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.PartOfSpeechAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.SentencesAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.TextAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.TokensAnnotation;
import edu.stanford.nlp.pipeline.Annotation;
// this is the polarity annotation!
import edu.stanford.nlp.naturalli.NaturalLogicAnnotations.PolarityDirectionAnnotation;
// not the one below!
// import edu.stanford.nlp.ling.CoreAnnotations.PolarityAnnotation;
import edu.stanford.nlp.util.PropertiesUtils;
import java.io.*;
import java.util.*;
public class test {
public static void main(String[] args) throws FileNotFoundException, UnsupportedEncodingException {
// code from: https://stanfordnlp.github.io/CoreNLP/api.html#generating-annotations
StanfordCoreNLP pipeline = new StanfordCoreNLP(
PropertiesUtils.asProperties(
// **add natlog here**
"annotators", "tokenize,ssplit,pos,lemma,parse,depparse,natlog",
"ssplit.eolonly", "true",
"tokenize.language", "en"));
// read some text in the text variable
String text = "Every dog sees some cat";
Annotation document = new Annotation(text);
// run all Annotators on this text
pipeline.annotate(document);
// these are all the sentences in this document
// a CoreMap is essentially a Map that uses class objects as keys and has values with custom types
List<CoreMap> sentences = document.get(SentencesAnnotation.class);
for(CoreMap sentence: sentences) {
// traversing the words in the current sentence
// a CoreLabel is a CoreMap with additional token-specific methods
for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
// this is the text of the token
String word = token.get(TextAnnotation.class);
// this is the POS tag of the token
String pos = token.get(PartOfSpeechAnnotation.class);
// this is the NER label of the token
String ne = token.get(NamedEntityTagAnnotation.class);
// this is the polarity label of the token
String pol = token.get(PolarityDirectionAnnotation.class);
System.out.print(word + " [" + pol + "] ");
}
System.out.println();
}
}
}
The output will be:
Every [up] dog [down] sees [up] some [up] cat [up]

How to get the cursor of each entity in a App Engine datastore query without performance hit?

I have a Datastore query using cursor (Objectify v5) and I want to get the cursor after each item in the result list. Code looks like this:
public List<Puzzle> queryWithCursor(String cursor, String order, int limit) {
Query<Puzzle> query = ObjectifyService.ofy()
.load()
.type(Puzzle.class)
.order(order)
.limit(limit);
query = query.startAt(Cursor.fromWebSafeString(cursor));
List<Puzzle> puzzles = new ArrayList<>();
QueryResultIterator<Puzzle> iterator = query.iterator();
while (iterator.hasNext()) {
Puzzle puzzle = iterator.next();
puzzle.setCursor(iterator.getCursor().toWebSafeString());
puzzles.add(puzzle);
}
return puzzles;
}
While the method works correctly, it triggers so many Datastore queries behind the scene. Basically, every time iterator.getCursor() runs, it triggers an additional query. I learnt from Stackdriver Trace that if limit is 20, the method triggers 19 queries in total (it seems that the last .getCursor() does not trigger additional query). So this method is even slower and more costly than the similar query using offset.
Is this really a bug? Is there a way to avoid the performance hit?
This is actually a fundamental behavior of the datastore, at least in the old sdk (as opposed to the new sdk that Objectify 6 uses, which may be the same maybe not). Calling getCursor() at non-batch boundaries restarts the query. You can try it with the low-level API.
There is a workaround: Make up your own Cursor class. It should consist of the low level Cursor and an offset. Explicitly set a chunk() size, then your cursor should consist of the Cursor at index 0 plus an offset into the chunk.
Then when you want to restart a query at that cursor, use .cursor(batchStartCursor).offset(offsetIntoBatch).
import com.google.appengine.api.datastore.Cursor;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.FetchOptions;
import com.google.appengine.api.datastore.PreparedQuery;
import com.google.appengine.api.datastore.Query;
import com.google.appengine.api.datastore.Query.SortDirection;
import com.google.appengine.api.datastore.QueryResultList;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class ListPeopleServlet extends HttpServlet {
static final int PAGE_SIZE = 15;
private final DatastoreService datastore;
public ListPeopleServlet() {
datastore = DatastoreServiceFactory.getDatastoreService();
}
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(PAGE_SIZE);
// If this servlet is passed a cursor parameter, let's use it.
String startCursor = req.getParameter("cursor");
if (startCursor != null) {
fetchOptions.startCursor(Cursor.fromWebSafeString(startCursor));
}
Query q = new Query("Person").addSort("name", SortDirection.ASCENDING);
PreparedQuery pq = datastore.prepare(q);
QueryResultList<Entity> results;
try {
results = pq.asQueryResultList(fetchOptions);
} catch (IllegalArgumentException e) {
// IllegalArgumentException happens when an invalid cursor is used.
// A user could have manually entered a bad cursor in the URL or there
// may have been an internal implementation detail change in App Engine.
// Redirect to the page without the cursor parameter to show something
// rather than an error.
resp.sendRedirect("/people");
return;
}
resp.setContentType("text/html");
resp.setCharacterEncoding("UTF-8");
PrintWriter w = resp.getWriter();
w.println("<!DOCTYPE html>");
w.println("<meta charset=\"utf-8\">");
w.println("<title>Cloud Datastore Cursor Sample</title>");
w.println("<ul>");
for (Entity entity : results) {
w.println("<li>" + entity.getProperty("name") + "</li>");
}
w.println("</ul>");
String cursorString = results.getCursor().toWebSafeString();
// This servlet lives at '/people'.
w.println("<a href='/people?cursor=" + cursorString + "'>Next page</a>");
}
}

Implicit not found on store

I have followed the pattern from examples on GitHub. When I call store on the model object, passing an instance of the entity, I get a compile error indicating one of the implicit parameters is missing as shown below.
could not find implicit value for parameter sg: com.outworkers.phantom.macros.SingleGeneric.Aux[com.ss.wuhu.settlement.entity.Settlement,Repr,HL,Out]
I guess I am missing something obvious. Could someone please point out how to bring the implicit into scope?
Regards
Meeraj
This is the code snippet where I am storing the data.
import akka.Done
import com.outworkers.phantom.dsl._
import com.outworkers.phantom.connectors.{CassandraConnection, ContactPoints}
import com.ss.wuhu.settlement.entity.Settlement
import com.ss.wuhu.settlement.entity.mapping.{SettlementForCourierModel, SettlementForVendorModel}
object Connector {
private val hosts = Seq("127.0.0.1") // TODO from environment
lazy val connector: CassandraConnection = ContactPoints(hosts).keySpace("wuhu_order")
}
class SettlementDatabase(override val connector: CassandraConnection) extends Database[SettlementDatabase](connector) {
object SettlementForCourierModel extends SettlementForCourierModel with connector.Connector
object SettlementForVendorModel extends SettlementForVendorModel with connector.Connector
def truncateAll() = {
Database.truncate()
}
def store(set: Settlement) = {
for {
v <- Database.SettlementForVendorModel.store(set)
d <- Database.SettlementForCourierModel.store(set)
} yield (Done)
}
}
object Database extends SettlementDatabase(Connector.connector)
This is a known bug with an open issue: https://github.com/outworkers/phantom/issues/774
I suggest either using the workaround as described in the link above, or my workaround which was creating my own .store() using .insert().
example:
def myStore(person: Person) : Future[ResultSet] =
insert
.value(_.name, person.name)
.value(_.age, person.age)
.value(_.timeCreate, person.timeCreate)
.future()

How to search for a part of a word with spring data elasticSearch

In my Spring Data Elasticsearch application I'd like to implement autocomplete functionality when user types a few chars and application will show him all possible variants with query*.
Right now I can't find a way how to properly implement it with Spring Data Elasticsearch.
For example I tried the following:
Criteria c = new Criteria("name").startsWith(query);
return elasticsearchTemplate.queryForPage(new CriteriaQuery(c, pageRequest), ESDecision.class);
It works for a single word query but in case of two or more words it returns error:
"Cannot constructQuery '*"security windows"'. Use expression or multiple clauses instead."
How to properly implement it in this case?
I have same requirement , I have implemented same . Querystring will work for you .
If you have two token like "security windows" than you have to pass "*security* *windows*" than Querystring will return all possible data available . If you have one token like "security" than you have to pass "*security*" .
One more explaination for this scenario ,check this answer -
https://stackoverflow.com/a/43278852/2357869
String aQueryString = "security windows" ;
String aQueryWithPartialSerach = null;
List<ESDecision> aESDecisions = null;
// Enabling partial sarch
if (aQueryString.contains(" ")) {
List<String> aTokenList = Arrays.asList(aQueryString.split(" "));
aQueryWithPartialSerach = String.join(" ", aTokenList.stream().map(p -> "*" + p + "*").collect(Collectors.toList()));
} else {
aQueryWithPartialSerach = "*" + aQueryString + "*";
}
NativeSearchQueryBuilder aNativeSearchQueryBuilder = new NativeSearchQueryBuilder();
aNativeSearchQueryBuilder.withIndices(indexName).withTypes(type).withPageable(new PageRequest(0, iPageRequestCount));
final BoolQueryBuilder aQuery = new BoolQueryBuilder();
aQuery.must(QueryBuilders.queryStringQuery(aQueryWithPartialSerach).defaultField("name"));
NativeSearchQuery nativeSearchQuery = aNativeSearchQueryBuilder.withQuery(aQuery).build();
aESDecisions = elasticsearchTemplate.queryForList(nativeSearchQuery, ESDecision.class);
return aESDecisions;
Imports need to be done :-
import org.springframework.data.elasticsearch.core.query.NativeSearchQuery;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.SearchQuery;
import org.springframework.data.domain.PageRequest;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;

How to get distance - MongoDB Template Near function

I'm trying to find Near by places.
Below code is working fine.
But i'm not able to get actual distance of place from my given lat,lng.
Criteria criteria = new Criteria("coordinates")
.near(new Point(searchRequest.getLat(),searchRequest.getLng()));
Query query = new Query();
query.addCriteria(criteria);
query.addCriteria(criteriaName);
query.limit(5);
List<Place> ls = (List<Place>) mongoTemplate.find(query, Place.class);
You can do it with geoNear aggregation. In spring-data-mongodb GeoNearOperation is representing this aggregation.
Extend or create inherit Place class with field where you would like to have distance information (example with inheritance):
public class PlaceWithDistance extends Place {
private double distance;
public double getDistance() {
return distance;
}
public void setDistance(final double distance) {
this.distance = distance;
}
}
Instead of Criteria with Query use aggregation. Second argument of geoNear is name of field where distance should be set:
final NearQuery nearQuery = NearQuery
.near(new Point(searchRequest.getLat(), searchRequest.getLng()));
nearQuery.num(5);
nearQuery.spherical(true); // if using 2dsphere index, otherwise delete or set false
// "distance" argument is name of field for distance
final Aggregation a = newAggregation(geoNear(nearQuery, "distance"));
final AggregationResults<PlaceWithDistance> results =
mongoTemplate.aggregate(a, Place.class, PlaceWithDistance.class);
// results.forEach(System.out::println);
List<PlaceWithDistance> ls = results.getMappedResults();
Just to make it easier - associated imports:
import static org.springframework.data.mongodb.core.aggregation.Aggregation.geoNear;
import static org.springframework.data.mongodb.core.aggregation.Aggregation.newAggregation;
import org.springframework.data.mongodb.core.aggregation.Aggregation;
import org.springframework.data.mongodb.core.aggregation.AggregationResults;
import org.springframework.data.mongodb.core.aggregation.GeoNearOperation;
import org.springframework.data.mongodb.core.query.NearQuery;
Walery Strauch's example was useful for me...
However I wanted to :
run aggregate query to get all the points in 2dsphere index with-in given distance in Kilometers or Meters. You can use Metrics.KILOMETERS & Metrics.MILES
collection name is not specified as part of pojo
I have 2dsphere index with old way of representation in MongoDB. I am using Mongo as sharded databased for Geo-Spatial queries. My nearSphere query (without aggregation) was failing only when there is a shard key added into the same collection where I have 2dsphere index.
After using below implementation with shard key in the same collection. I am successfully able to fetch the required data.
Here is the sample :
import org.springframework.data.geo.Metrics;
final NearQuery query = NearQuery.near(new Point(longitude, latitude), Metrics.KILOMETERS)
.num(limit)
.minDistance(distanceInKiloMeters)
.maxDistance(maxNearByUEDistanceInKiloMeters)
.spherical(true);
final Aggregation a = newAggregation(geoNear(query, "distance"));
final AggregationResults<PlaceWithDistance> results = offlineMongoTemplate.aggregate(a, "myCollectionName", PlaceWithDistance.class);
final List<PlaceWithDistance> measurements = new ArrayList<PlaceWithDistance>(results.getMappedResults());

Resources