I have an application which uses EWS Managed API to search and retrieve items from Exchange server. On one client environment (Exchange 2016 on-premise server), when application tries to search and retrieve items from Archive MailBox folders using AQS query, the API seems to only return 250 items even though folder has more items that meets search query. The pagesize i request is 500, but the response only has 250 items.
Anyone know why FindItems() method only returning only 250 items?
Is there any throttling policy we need to disable? This is for Exchange 2016 Server.
By default, Exchange limits the maximum number of items that can be returned in a single request to 1000. But setting the maximum number of entries to a smaller number results in quicker response times, at the cost of having to send more requests to get all items. So, it seems Exchange admin reduced the max number of items that can be returned in a single request.
Related
I'm new at Grafana and I'm trying to create a graph that shows the requests count together with the average response time for the requests, I was able to create my requests count but now I'm struggling to add the information with the requests time, there is an option to show both information inside a panel? Or do I need to create two panels, one with the request count and another with the average time?
And another question, there is an option to show the average time in milliseconds?
I have a use case:
I need to use _search API to fetch whole bunch of records in a paginated way.
But at the same time, I would want to get the total hit number in the same _search API call.
Example:
The pagination number is 50, that is, I want to fetch result in a 50 batch manner. At the same time, I want to get the total hit number, let's say 5000 for each search call.
I have 2 questions:
Is this possible? get total hit number as the result of a _search API call?
Would the total hit number be impacted due to the pagination?
you can get total hit in search API with adding track_total_hits=true option.
GET localhost:9200/_search?pretty&track_total_hits=true
if you are using search API with from=X&size=50 for pagination, yes it is possible that the number of docs change during of pagination. but it depends of refresh interval. you can increase the refresh interval. there is another solution for this problem. Pit API.
https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html
also from=X&size=50 with you have limit for pagination(I think you can only fetch 10000 docs) you could increase this limitation. or use scroll API.
Image from Search API ES-DOCs.. You can use hits -> total.
I have a microservice with elasticsearch as a backend store. Now I have multiple indices with hundreds of thousands of documents inserted in those indices.
Now I need to expose GET APIs for those indices. GET /employees/get.
I have gone through ES pagination using scroll and search_after. But both of them require meta information like scroll_id and search_after(key) to do pagination.
Now the concern is my microservice shouldn't expose these scroll_ids or search_after. With current approach, I can list up to 10k docs but not after that. And I don't want users of microservice to know about the backend store or anything about it. So How can I achieve this in elasticservice?
I have below approach in mind:
Store the scroll_id in-memory and retrieve the results based on that for subsequent queries. Get query will be as below:
GET /employees/get?page=1 By default each page will have 10k documents.
Implement scroll API internally over GET API and return all matching documents to users. But this increases the latency and memory. Because at times I may end up returning 100k docs to user in a single call.
Expose GET API with search string. By default return 10k documents and further the results will be refreshed with searchstring as explained:
Lets say GET /employees/get return 10k documents. And accept query_string to enrich the 10k like auto suggestion using n gram. Then we show most valid 10k docs everytime. I know this is not actual pagination but somehow this too solves the problem in a hacky way. This is my Plan-B.
Edited:
This is my usecase:
Return list of employees of a company. There are more than 100k employees. So I have to return the results in pages. GET /employees/get?from=0&size=1000 and GET /employees/get?from=1001&size=1000
But once I reach from+size to 10k, ES rejects the query.
Please suggest what would be the ideal way to implement pagination in microservice with ES as backend store and not letting user to know about internals of ES.
There is orderdocuments stored in our elasticsearch. There is a field lets say phone-number in each document.
what we want to find out is phone number counts(how many times a order was received from a phone number) .
Each (i repeat each) phone numbers have made 10 million+ bookings across all our data.
So aggregating across all the data is taking too much time. we came to a conclusion that a limit should be put because data is too much. we introduced terminate_after in the query and the response times improved to desired limit but then a new problem got introduced. If we are going to put a limit, we want to limit results from most recent 2000 documents instead of any 2000 docs.
So in nutshell we want to find out ordercounts by phonenumber in most recent 2000 documents.
How to achieve this in elasticsearch.
what we have tried so far:
bucket sort (it applies only after the buckets are already created)
terminate_after property in query(this does not take into account that
we want counts in most recent 2000 matches)
TopNhits aggregator (This also works post the hits are calculated)
I am trying to fetch about 2.5 million records from elastic search using elastic search's Java High Level Client. Which is taking too much time (15 to 22 minutes based on number of records) to fetch all the record using scroll API as it has a limitation of fetch 10,000 record in one request. I tried sliced scroll also but that is taking more time than normal scroll. Following is my assumption about sliced scroll API:
I divided my scroll request into five slices. Which creates 5 requests.
I send 5 request in different threads.
Because every sliced scroll request is an individual request. I guess for each sliced scroll request first it fetches all the records (2.5 million) then filters out the records which belongs to that particular slice.
Which is resulting in more time.
Can anyone tell me more efficient way to fetch all the records.