Steps to write elastic search count api using spring boot.?
how to use countRequest to get count of documents...??
need code snippets.
Using new Java API Client:
var countRequestResponse = client.count(CountRequest.of(c -> c.index("idx_name")));
System.out.println(countRequestResponse.count());
Using Java High Client Library
var countRequest = new CountRequest();
countRequest.indices("idx_name");
var response = getClient().count(countRequest, RequestOptions.DEFAULT);
System.out.println(response.getCount());
Related
How to use elasticsearch java client api to transform the following code?
code
NestedAggregationBuilder attrs = AggregationBuilders.nested("attrs", "attrs");
TermsAggregationBuilder attrIds = AggregationBuilders.terms("attrIds").field(".attrId").size(10);
attrs.subAggregation(attrIds);
TermsAggregationBuilder attrValueIds = AggregationBuilders.terms("attrValueIds").field("attrs.attrValueId").size(10);
attrIds.subAggregation(attrValueIds);
String[] include = {"attrs..attrName", "attrs..attrValueId", "attrs..attrValueName"};
attrValueIds.subAggregation(AggregationBuilders
.topHits("top_hits_data")
.fetchSource(include, null)
.sort("attrs..attrValueName", SortOrder.ASC)
.size(1));
I have to upgrade our ES from 6.2 (TCP client) to 7.4.2 (REST API)
And I have a little problem with UpdateByQueryRequestBuilder, that looks like need to be change to UpdateByQueryRequest (doc). The old code looks like this:
BoolQueryBuilder dateQueryBuilder = ...
QueryBuilder a = ...
BoolQueryBuilder b = ...
UpdateByQueryRequestBuilder updateByQuery = new UpdateByQueryRequestBuilder(tcpClient, UpdateByQueryAction.INSTANCE);
updateByQuery.filter(dateQueryBuilder.filter(a).filter(b)).script(updateScript);
As I wrote, I'm understanding UpdateByQueryRequestBuilder (that using the oldest client) should be replace with UpdateByQueryRequest but this new API haven't filter method (just setQuery that will replace the current query in chain case...)
UpdateByQueryRequest updateRequest = new UpdateByQueryRequest();
updateRequest.setQuery(dateQueryBuilder)
// .setQuery(a) - will replace dateQueryBuilder instead of chain new filter...
// .filter - not exist in the new API
So the question, how should I replace this code with newest ES REST API (or chain the queries)?
I have the following simple table I've created from the following source: https://hortonworks.com/hadoop-tutorial/introduction-apache-hbase-concepts-apache-phoenix-new-backup-restore-utility-hbase/#start-hbase
using the following:
create 'driver_dangerous_event','events'
put 'driver_dangerous_event','4','events:driverId','78'
put 'driver_dangerous_event','4','events:driverName','Carl'
put 'driver_dangerous_event','4','events:eventTime','2016-09-23 03:25:03.567'
put 'driver_dangerous_event','4','events:eventType','Normal'
put 'driver_dangerous_event','4','events:latitudeColumn','37.484938'
put 'driver_dangerous_event','4','events:longitudeColumn','-119.966284'
put 'driver_dangerous_event','4','events:routeId','845'
put 'driver_dangerous_event','4','events:routeName','Santa Clara to San Diego'
put 'driver_dangerous_event','4','events:truckId','637'
I need to query this row but using a where filter (for future use), I have a rest api or thrift api running on my server.
I tried using rest api but failed to do it, is it possible?
also I tried using this nuget: https://hbasenet.codeplex.com/releases/view/133288 but I can't understand how to filter the data with where clause, I can only select a specific row:
Hbase.Client c = new Hbase.Client(serverHostName, port, 10000);
var res = c.Scan<Driver>("driver_dangerous_event", "events", "1");
Is there any option to do a simple filtered query with REST api/ Thrift API/ some other C# library?
I used Microsoft.HBase.Client for preforming a simple query (https://github.com/hdinsight/hbase-sdk-for-net)
// Connection
RequestOptions scanOptions = RequestOptions.GetDefaultOptions();
scanOptions.Port = int.Parse(hbaseDataConnection.Port);
scanOptions.AlternativeEndpoint = "/";
var nodeIPs = new List<string>();
nodeIPs.Add(hbaseDataConnection.Address);
HBaseClient client = new HBaseClient(null, scanOptions, new LoadBalancerRoundRobin(nodeIPs));
Scanner scanner = new Scanner { batch = 10 };
ScannerInformation scannerInfo = await client.CreateScannerAsync(_tableName, scanner, scanOptions);
var options = RequestOptions.GetDefaultOptions();
options.Port = int.Parse(hbaseDataConnection.Port);
options.AlternativeEndpoint = "/";
options.AlternativeHost = scannerInfo.Location.Host;
var f1 = new SingleColumnValueFilter(
Encoding.UTF8.GetBytes(ColumnFamilyName),
Encoding.UTF8.GetBytes("driverName"),
CompareFilter.CompareOp.Equal,
new SubstringComparator(fld.Values[0].ToString()))
var filter = new FilterList(FilterList.Operator.MustPassAll, f1);
scanner.filter = filter.ToEncodedString();
ScannerInformation scanInfo = client.CreateScannerAsync(_tableName, scanner, scanOptions).Result;
result = RetrieveResults(client, scanInfo, scanOptions).ToList();
Make sure REST API is running on the HBase machine, e.g.
hbase rest start -p 20050 --infoport 20051
Is it possible to get the raw search query from the NEST client?
var result = client.Search<SomeType>(s => s
.AllIndices()
.Type("SomeIndex")
.Query(query => query
.Bool(boolQuery => BooleanQuery(searchRequest, mustMatchQueries)))
);
I'd really like to debug why I am getting certain results.
The methods to do this seem to change with each major version, hence the confusing number of answers. If you want this to work in NEST 6.x, AND you want to see the deserialized request BEFORE it's actually sent, it's fairly easy:
var json = elasticClient.RequestResponseSerializer.SerializeToString(request);
If you're debugging in Visual Studio, it's handy to put a breakpoint right after this line, and when you hit it, hover over the json variable above and hit the magnifying glass thingy. You'll get a nice formatted view of the JSON.
You can get raw query json from RequestInformation:
var rawQuery = Encoding.UTF8.GetString(result.RequestInformation.Request);
Or enable trace on your ConnectionSettings object, so NEST will print every request to trace output
var connectionSettings = new ConnectionSettings(new Uri(elasticsearchUrl));
connectionSettings.EnableTrace(true);
var client = new ElasticClient(connectionSettings);
NEST 7.x
Enable debug mode when creating settings for a client:
var settings = new ConnectionSettings(connectionPool)
.DefaultIndex("index_name")
.EnableDebugMode()
var client = new ElasticClient(settings);
then your response.DebugInformation will contain information about request sent to elasticsearch and response from elasticsearch. Docs.
For NEST / Elasticsearch.NET v6.0.2, use the ApiCall property of the IResponse object. You can write a handy extension method like this:
public static string ToJson(this IResponse response)
{
return Encoding.UTF8.GetString(response.ApiCall.RequestBodyInBytes);
}
Or, if you want to log all requests made to Elastic, you can intercept responses with the connection object:
var node = new Uri("https://localhost:9200");
var pool = new SingleNodeConnectionPool(node);
var connectionSettings = new ConnectionSettings(pool, new HttpConnection());
connectionSettings.OnRequestCompleted(call =>
{
Debug.Write(Encoding.UTF8.GetString(call.RequestBodyInBytes));
});
In ElasticSearch 5.x, the RequestInformation.Request property does not exist in ISearchResponse<T>, but similar to the answer provided here you can generate the raw query JSON using the Elastic Client Serializer and a SearchDescriptor. For example, for the given NEST search query:
var results = elasticClient.Search<User>(s => s
.Index("user")
.Query(q => q
.Exists(e => e
.Field("location")
)
)
);
You can get the raw query JSON as follows:
SearchDescriptor<User> debugQuery = new SearchDescriptor<User>()
.Index("user")
.Query(q => q
.Exists(e => e
.Field("location")
)
)
;
using (MemoryStream mStream = new MemoryStream())
{
elasticClient.Serializer.Serialize(debugQuery, mStream);
string rawQueryText = Encoding.ASCII.GetString(mStream.ToArray());
}
Before making Request, from Nest Query - For Nest 5.3.0 :
var stream = new System.IO.MemoryStream();
elasticClient.Serializer.Serialize(query, stream );
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());
Edit: It's changed from from Nest 6.x, and you can do below:
var json = elasticClient.RequestResponseSerializer.SerializeToString(request);
on nest version 6 use
connextionString.DisableDirectStreaming();
then on response.DebugInformation you can see all information.
Use result.ConnectionStatus.Request.
When using NEST 7 and you don't want to enable debug mode.
public static string GetQuery<T>(this IElasticClient client, SearchDescriptor<T> searchDescriptor) where T : class
{
using (System.IO.MemoryStream ms = new System.IO.MemoryStream())
{
client.RequestResponseSerializer.Serialize(searchDescriptor, ms);
return Encoding.UTF8.GetString(ms.ToArray());
}
}
While it's possible to get raw request/response through code, I find it much easier to analyze it with fiddler.
The reason is that I can easily analyze raw request, response, headers, Full URL, execution time - all together without any hassle of code changes.
Here's some reference links in case someone unfamiliar with fiddler wants to check details:
#1 https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/logging-with-fiddler.html
#2 NEST 1.0: See request on Fiddler
#3 https://newbedev.com/how-to-get-nest-to-work-with-proxy-like-fiddler
How about using Fiddler ?! :)
I'm doing a small research about a client for elastic search in .net and I found that NEST is one of the most supported solutions for this matter.
I was looking at Nest's docummentation and I couldn´t find a way to output a raw json from a query and avoid the serialization into an object, because I'm using angularJs in the front end I don´t want to overload the process of sending the information to the client with some unnecessary steps.
......and also I'd like to know how can I overrdide the serialization process?
I found that NEST uses Json.NET which I would like to change for the servicestack json serielizer.
thanks!
Hi Pedro you can do this with NEST
var searchDescriptor = new SearchDescriptor<ElasticSearchProject>()
.Query(q=>q.MatchAll());
var request = this._client.Serializer.Serialize(searchDescriptor);
ConnectionStatus result = this._client.Raw.SearchPost(request);
Assert.NotNull(result);
Assert.True(result.Success);
Assert.IsNotEmpty(result.Result);
This allows you to strongly type your queries, but return the string .Result which is the raw response from elasticsearch as string to your
request can be an object or the string so if you are OK with the internal json serialize just pass searchDescriptor directly
Use RequestResponseSerializer instead of Serializer.
var searchDescriptor = ...;
...
byte[] b = new byte[60000];
using (MemoryStream ms = new MemoryStream(b))
{
this._client.RequestResponseSerializer.Serialize(searchDescriptor , ms);
}
var rawJson = System.Text.Encoding.Default.GetString(b);