I am wondering about an alternative method to TermFilter() in ES 7.5, it's deprecated since ES 2.x.
My current method:
#Override
public boolean deleteAnnotation(String userName, String id) throws Exception {
final IndexAndType annotationIndex = indexConfiguration.getIndex(Annotation);
final SearchResponse searchResponse = esClient.prepareSearch(annotationIndex.getIndex()).setTypes(annotationIndex.getType()).setQuery(
filteredQuery(QueryBuilders.termQuery(AnnotationField.id.field(), id), termFilter(AnnotationField.user.field(), userName))
).execute().actionGet();
if (0 < searchResponse.getTotalShards()) {
final SearchHit hit = searchResponse.getHits().getAt(0);
final DeleteResponse deleteResponse = esClient.prepareDelete(annotationIndex.getIndex(), annotationIndex.getType(), hit.getId()).execute().actionGet();
return deleteResponse.isFound();
}
return false;
}
Also isFound() is deprecated, is there any alternative?
Related
I have SearchRequest object with all the Elasticsearch(ES) query data set. I cannot use RestHighLevel client for my usecase because it requires endpoint need to be passed at the time of instantiation. I gets ES endpoint dynamically based on some condition. One way is to always create new RestHighLevel client which will be inefficient approach. Other way is to create static CloseableHttpClient on service start and make HttpPost request with dynamic endpoint. I wanted to take later approach but don't know how to convert SearchRequest object into json query string.
Any code reference/snippet would be very helpful
private final CloseableHttpClient client;
public GenericElasticSearchResponse search(#Nonnull final SearchRequest searchRequest,
#Nonnull final RoutingConfig route) {
final URIBuilder builder = new URIBuilder()
.setScheme(route.getScheme())
.setHost(route.getESEndpoint())
.setPort(Optional.ofNullable(route.getPort())
.orElse(80))
.setPath("/sessions*/_search");
final URI uri = builder.build();
final ContentType contentType = ContentType.create("application/json", "UTF-8");
final HttpPost httpPost = new HttpPost(uri);
httpPost.setEntity(entity);
final CloseableHttpResponse response = client.execute(httpPost);
final String responseEntity;
try (final Reader reader = new InputStreamReader(response.getEntity().getContent(), Charsets.UTF_8)) {
responseEntity = CharStreams.toString(reader);
}
final SearchResponse searchResponse = objectMapper.readValue(responseEntity, SearchResponse.class);
return new ElasticSearchResponse(searchResponse);
}
I found searchRequest.source().toString() was actually returning json form of SearchRequest. Following is complete code snippet for hitting ES via Apache client
final EndpointConfig endpoint = route.getEndpoint();
final URIBuilder builder = new URIBuilder()
.setScheme(endpoint.getScheme())
.setHost(endpoint.getHost())
.setPort(Optional.ofNullable(endpoint.getPort())
.orElse(HTTPS_PORT))
.setPath(Optional.ofNullable(endpoint.getQueryPath())
.orElse(StringUtils.EMPTY));
final URI uri = builder.build();
final ContentType contentType = ContentType.create("application/json", "UTF-8");
final String queryString = searchRequest.source().toString();
final StringEntity entity = new StringEntity(queryString, contentType);
final HttpPost httpPost = new HttpPost(uri);
httpPost.setEntity(entity);
final CloseableHttpResponse response = sendRequest(httpPost);
final String responseEntity;
try (final Reader reader = new InputStreamReader(response.getEntity().getContent(), Charsets.UTF_8)) {
responseEntity = CharStreams.toString(reader);
}
log.info("ElasticSearchClient response: Code: {}, Entity {}", response.getCode(), responseEntity);
SearchResponse searchResponse = null;
if (Objects.nonNull(responseEntity)) {
searchResponse = parseResponse(responseEntity, searchRequest, response.getCode());
log.info("ElasticSearchClient searchResponse- {} ", searchResponse);
}
return new ElasticSearchResponse(searchResponse);
} catch (final URISyntaxException e) {
throw new IllegalStateException(
String.format("Invalid URI. host: %s", route.getEndpoint()), e);
} catch (final IOException e) {
throw new IllegalStateException("ElasticSearch Request failed.", e);
}
private SearchResponse parseResponse(#Nonnull final String responseEntity,
#Nonnull final SearchRequest searchRequest,
final int responseCode) {
if (responseCode >= 400 || responseCode < 200) {
log.info("ES error response - {} ", responseEntity);
final ESErrorResponse response = GSON.fromJson(responseEntity, ESErrorResponse.class);
throw new IllegalStateException();
}
SearchResponse searchResponse = null;
final NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
final XContentParser parser;
try {
parser = JsonXContent.jsonXContent.createParser(registry,
DeprecationHandler.THROW_UNSUPPORTED_OPERATION, responseEntity);
searchResponse = SearchResponse.fromXContent(parser);
} catch (IOException e) {
throw new IllegalStateException("Error while parsing response ", e);
}
return searchResponse;
}
public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
final Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
return map.entrySet().stream()
.map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
.collect(Collectors.toList());
}
private CloseableHttpResponse sendRequest(final HttpPost httpPost) throws IOException {
return client.execute(httpPost);
}
Below is my test case,
#Test
public void aTestOfMocks() throws SQLException
{
when(dbTemplate.query(anyString(), anyMap(), Matchers.<ResultSetExtractor<List<Order>>>any())).thenReturn(list);
final List<Employee> actual = xxx.getEmployeeByDate("2020-05-29");
assertNotNull(actual, "error message");
}
Below is my actual code,
#Override
public List<Order> getEmployeeByDate(String startDate)
{
final Map<String, String> parameters = new HashMap<>();
parameters.put("startDate", startDate);
final String limiterQuery = "query";
final boolean includeTest = false;
return queryForEmployee(limiterQuery, parameters, includeTest);
}
private List<Employee> queryForEmployee(
String limiterQuery,
Map<String, String> parameters,
boolean includeTest)
{
ResultSetExtractor<List<Order>> resultExtractor;
final String query = limiterQuery + commonQueryForOrder(includeTestOrders);
final List<Order> result = dbTemplate.query(query, parameters, resultExtractor);
if (result == null)
{
return Collections.emptyList();
}
else
{
return result;
}
}
When I run this test getting error as
org.mockito.exceptions.misusing.InvalidUseOfMatchersException:
Invalid use of argument matchers!
3 matchers expected, 4 recorded:
Is there anything i am missing here? I have searched anc verified similar posts in stackoverflow but nothing helped me out to solve this.
I want to prevent not logged user form accessing the proxy. I can throw an exception but the response is 404 instead of `401 or '403'. It it possible?
Filter code:
#Component
public class CustomZuulFilter extends ZuulFilter {
//FIXME - if 401,403 get the new token??, fallbackMethod = "fall",
#HystrixCommand(
commandProperties = {
#HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000"),
#HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "60")
}
)
#Override
public Object run() {
logger.debug("Adding zulu header");
String userName = getLoggedUser();
RequestContext ctx = RequestContext.getCurrentContext();
if (userName == null) {
// throw new RuntimeException("User not authenticated");
logger.info("User not authenticated");
ctx.setResponseStatusCode(401);
ctx.sendZuulResponse();
return null;
}
return null;
}
private String getLoggedUser() {
[...]
}
#Override
public boolean shouldFilter() {
return true;
}
#Override
public String filterType() {
return PRE_TYPE;
}
#Override
public int filterOrder() {
return PRE_DECORATION_FILTER_ORDER - 1;
}
}
It might be a bit late, but i think you can remove ctx.sendZuulResponse();
and add ctx.setSendZuulResponse(false);
I have a set of node id's (Set< Long >) and want to restrict or filter the results of an query to only the nodes in this set. Is there a performant way to do this?
Set<Node> query(final GraphDatabaseService graphDb, final Set<Long> nodeSet) {
final Index<Node> searchIndex = graphdb.index().forNodes("search");
final IndexHits<Node> hits = searchIndex.query(new QueryContext("value*"));
// what now to return only index hits that are in the given Set of Node's?
}
Wouldn't be faster the other way round? If you get the nodes from your set and compare the property to the value you are looking for?
for (Iterator it=nodeSet.iterator();it.hasNext();) {
Node n=db.getNodeById(it.next());
if (!n.getProperty("value","").equals("foo")) it.remove();
}
or for your suggestion
Set<Node> query(final GraphDatabaseService graphDb, final Set<Long> nodeSet) {
final Index<Node> searchIndex = graphdb.index().forNodes("search");
final IndexHits<Node> hits = searchIndex.query(new QueryContext("value*"));
Set<Node> result=new HashSet<>();
for (Node n : hits) {
if (nodeSet.contains(n.getId())) result.add(n);
}
return result;
}
So the fastest solution I found was directly using lucenes IndexSearcher on the index created by neo4j and use an custom Filter to restrict the search to specific nodes.
Just open the neo4j index folder "{neo4j-database-folder}/index/lucene/node/{index-name}" with the lucene IndexReader. Make sure to use not add a lucene dependency to your project in another version than the one neo4j uses, which currently is lucene 3.6.2!
here's my lucene Filter implementation that filters all query results by the given Set of document id's. (Lucene Document id's (Integer) ARE NOT Neo4j Node id's (Long)!)
import java.io.IOException;
import java.util.PriorityQueue;
import java.util.Set;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.search.DocIdSet;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.search.Filter;
public class DocIdFilter extends Filter {
public class FilteredDocIdSetIterator extends DocIdSetIterator {
private final PriorityQueue<Integer> filterQueue;
private int docId;
public FilteredDocIdSetIterator(final Set<Integer> filterSet) {
this(new PriorityQueue<Integer>(filterSet));
}
public FilteredDocIdSetIterator(final PriorityQueue<Integer> filterQueue) {
this.filterQueue = filterQueue;
}
#Override
public int docID() {
return this.docId;
}
#Override
public int nextDoc() throws IOException {
if (this.filterQueue.isEmpty()) {
this.docId = NO_MORE_DOCS;
} else {
this.docId = this.filterQueue.poll();
}
return this.docId;
}
#Override
public int advance(final int target) throws IOException {
while ((this.docId = this.nextDoc()) < target)
;
return this.docId;
}
}
private final PriorityQueue<Integer> filterQueue;
public DocIdFilter(final Set<Integer> filterSet) {
super();
this.filterQueue = new PriorityQueue<Integer>(filterSet);
}
private static final long serialVersionUID = -865683019349988312L;
#Override
public DocIdSet getDocIdSet(final IndexReader reader) throws IOException {
return new DocIdSet() {
#Override
public DocIdSetIterator iterator() throws IOException {
return new FilteredDocIdSetIterator(DocIdFilter.this.filterQueue);
}
};
}
}
To map the set of neo4j node id's (the query result should be filtered with) to the correct lucene document id's i created an inmemory bidirectional map:
public static HashBiMap<Integer, Long> generateDocIdToNodeIdMap(final IndexReader indexReader)
throws LuceneIndexException {
final HashBiMap<Integer, Long> result = HashBiMap.create(indexReader.numDocs());
for (int i = 0; i < indexReader.maxDoc(); i++) {
if (indexReader.isDeleted(i)) {
continue;
}
final Document doc;
try {
doc = indexReader.document(i, new FieldSelector() {
private static final long serialVersionUID = 5853247619312916012L;
#Override
public FieldSelectorResult accept(final String fieldName) {
if ("_id_".equals(fieldName)) {
return FieldSelectorResult.LOAD_AND_BREAK;
} else {
return FieldSelectorResult.NO_LOAD;
}
}
};
);
} catch (final IOException e) {
throw new LuceneIndexException(indexReader.directory(), "could not read document with ID: '" + i
+ "' from index.", e);
}
final Long nodeId;
try {
nodeId = Long.valueOf(doc.get("_id_"));
} catch (final NumberFormatException e) {
throw new LuceneIndexException(indexReader.directory(),
"could not parse node ID value from document ID: '" + i + "'", e);
}
result.put(i, nodeId);
}
return result;
}
I'm using the Google Guava Library that provides an bidirectional map and the initialization of collections with an specific size.
I have a REST API that accepts 3 query params. When the query is called without any one of the query parameters, the API executes and returns the result. How do we make the queryparams mandatory? How can I add validation to check if all the parameters are present? Also, please let me know the best approach.
On a very simple level you could just inject the HttpServletRequest and check yourself:
#GET
public Response example(#Context HttpServletRequest request,
#QueryParam("name") String name) {
if (null == request.getParameter("name")) {
ResponseBuilder builder = Response.status(404);
return builder.build();
}
// Do something with name
}
Or you can implement something more elaborate using AOP. Here's a blog post about further options.
jersey doesn't give a mandatory parameter checking functionality out of the box. however you can do something like implementing your own annotation to achieve it.
Below is the annotation code:
#Target(value = ElementType.METHOD)
#Retention(value = RetentionPolicy.RUNTIME)
public #interface Required {
String[] value();
}
You also need a filter, below is the code:
public class RequiredParamResourceFilterFactory implements ResourceFilterFactory {
#Context
private transient HttpServletRequest servletRequest;
private class RequiredParamFilter implements ResourceFilter, ContainerRequestFilter {
private final String[] requiredParams;
protected List<String> parametersValueMissing;
private RequiredParamFilter(String[] requiredParams) {
this.requiredParams = requiredParams;
}
#Override
public ContainerRequest filter(ContainerRequest containerRequest) {
boolean missingMandatoryParameter = false;
List<String> missingParameters = new ArrayList<String>();
List<String> requiredParametersValueMissing = new ArrayList<String>();
List<String> URLParameters = getURLParameters(containerRequest.getQueryParameters());
List<String> methodRequiredParameters = Arrays.asList(requiredParams);
if (methodRequiredParameters != null) {
for (String methodRequiredParam : methodRequiredParameters) {
if (URLParameters == null) {
missingMandatoryParameter = true; //we will check this flag before returning result set to caller
missingParameters.add(methodRequiredParam);
} else if (!URLParameters.contains(methodRequiredParam)) {
missingMandatoryParameter = true; //we will check this flag before returning result set to caller
missingParameters.add(methodRequiredParam);
//Add to required parameters value missing List, only if the parameter is mandatory and value is not provided
// in the URL
} else if (parametersValueMissing.contains(methodRequiredParam)) {
requiredParametersValueMissing.add(methodRequiredParam);
}
}
if (missingMandatoryParameter && requiredParametersValueMissing.size() > 0) {
throw new YourCustomException("Missing Parameters = " + StringHelper.ArrayToString(missingParameters) +
"\nParameter value missing for " + StringHelper.ArrayToString(requiredParametersValueMissing));
} else if (missingMandatoryParameter) {
throw new YourCustomException("Missing Parameters = " + StringHelper.ArrayToString(missingParameters), MisbarErrorCode.VALIDATION_WRONG_INPUT_ERROR, "Customers");
} else if (requiredParametersValueMissing != null &&
requiredParametersValueMissing.size() > 0) {
throw new YourCustomException("Parameter value missing for " + StringHelper.ArrayToString(requiredParametersValueMissing));
}
}
return containerRequest;
}
#Override
public ContainerRequestFilter getRequestFilter() {
return this;
}
#Override
public ContainerResponseFilter getResponseFilter() {
return null;
}
/**
* To fetch the parameters sent to webservice call, these will be used to find if required parameter
* are present or not
*
* #param queryParams the queryparams sent
* #return all the parameters sent in URL
*/
private List<String> getURLParameters(MultivaluedMap<String,String> queryParams) {
parametersValueMissing = new ArrayList<String>();
List<String> arr = new ArrayList<String>();
for(String key:queryParams.keySet())
{
arr.add(key);
if(queryParams.get(key)==null)
parametersValueMissing.add(key);
}
if(!arr.isEmpty())
return arr;
return null;
}
}
#Override
public List<ResourceFilter> create(AbstractMethod am) {
Required required = am.getAnnotation(Required.class);
if(required!=null)
{
return Collections.<ResourceFilter>singletonList(new RequiredParamFilter(required.value()));
}
return null;
}
}
Below sample shows how to use this annotation, so in below webservice; file_id and count are mandatory parameters:
#GET
#Produces(MediaType.APPLICATION_JSON+";charset=utf-8")
#Cacheable(isCacheable = true)
#Path("posts/clusters")
#Required({"file_id","count"})
#Timed
public Response getClusters(
#QueryParam("file_id") Integer fileId,
#QueryParam("count") Integer count,
#DefaultValue("-1")#QueryParam("start_time") Long startTime){
;
}
If mandatory parameters are not provided in webservice call, you receive an error like below, mentioning the parameter names that are missing:
{
message: "Missing Parameters = file_id, count",
errorCode: "600"
}
Hope this solves your problem.