Using rest I got this exception
http://localhost:8080/customgroups?_queryFilter=(members/uid+co+%22test%22)
{"code":413,"reason":"Request Entity Too Large","message":"Administrative Limit Exceeded"}
I turned all limits off:
ds-cfg-lookthrough-limit: 0
ds-cfg-size-limit: 0
Is there another constrain? The result should be 1-3 entries. Other requests like get all customGroups = 83 or users = 1300 works fine, so why does the query_filter making problems?
Thank You
There are a few things you might try out:
can you check the ds-rlim-lookthrough-limit operational attribute is correctly set? Especially for cn=Directory Manager if you are using it to make requests.
I can see there is a special config for collective attributes ds-rlim-lookthrough-limit;collective: 0. Maybe does it apply to your request?
References:
http://ludopoitou.com/2012/04/10/tips-resource-limits-in-opendj/
http://opendj.forgerock.org/opendj-server/configref/global.html#lookthrough-limit
http://docs.forgerock.org/en/opendj/2.6.0/admin-guide/index/chap-resource-limits.html
Related
My use case is the following :
Once every day I upload 1000 single page pdf to Azure Storage and process them with Form Recognizer via python azure-form-recognizer latest client.
So far I’m using the Async version of the client and I send the 1000 coroutines concurrently.
tasks = {asyncio.create_task(analyse_async(doc)): doc for doc in documents}
pending = set(tasks)
# Handle retry
while pending:
# backoff in case of 429
time.sleep(1)
# concurrent call return_when all completed
finished, pending = await asyncio.wait(
pending, return_when=asyncio.ALL_COMPLETED
)
# check if task has exception and register for new run.
for task in finished:
arg = tasks[task]
if task.exception():
new_task = asyncio.create_task(analyze_async(doc))
tasks[new_task] = doc
pending.add(new_task)
Now I’m not really comfortable with this setup. The main reason being the unpredictable successive states of the service in the same iteration. Can be up then throw 429 then up again. So not enough deterministic for me. I was wondering if another approach was possible. Do you think I should rather increase progressively the transactions. Start with 15 (default TPS) then 50 … 100 until the queue is empty ? Or another option ?
Thx
We need to enable the CORS and make some changes to that CORS to make it available to access the heavy workload.
Follow the procedure to implement the heavy workload in form recognizer.
Make it for page blobs here for higher and best performance.
Redundancy is also required. Make it ZRS for better implementation.
Create a storage account to upload the files.
Go to CORS and add the URL required.
Set the Allowed origins to https://formrecognizer.appliedai.azure.com
Go to containers and upload the documents.
Upload the documents. Use the container and blob information to give as the input for the recognizer. If the case is from Form Recognizer studio, the size of the total documents is considered and also the number of characters limit is there. So suggested to use the python code using the container created as the input folder.
I have run a analysis on Sonarqube with Codescan. The number of issues returned, is way above the 10000 web api limit. Therefore, within my client/code I wanted to loop through all of the rules within a quality profile and return all the issues per rule.
How can I get a list of rules using the web api from java?
You can use api/qualityprofiles/backup. It takes a quality profile key as parameter and returns an xml containing all "active rules".
Newer SonarQube versions do not have the 10K issues limitation.
You have to loop n-times to collect all results.
For example:
Consider a project with 44K issues.
You have to discover first how many issues you have to read, call one time the /api/issues/search with only you project key and the parameter ps ( pagesize ) equal to 100
http:///api/issues/search?componentKeys=&ps=100
You could receive an answer like this
{"total":44130,"p":1,"ps":100,"paging":{"pageIndex":1,"pageSize":100,"total":44130},"issues":[{"key":"AVtoCSNP6OwvnmtEJjae","ru..........
So we have to claim 44130 issues, using a pagesize of 100 then you must call (44130 / 100 ) + 1 times the /api/issues/search for your project and for every request remember to increase by 1 the p ( page ) parameter ( so you can point the right portion of results )
Your sequence of command will be like this
http:///api/issues/search?componentKeys=YOUR_PROJECT_KEY>&ps=100&p=1
http:///api/issues/search?componentKeys=YOUR_PROJECT_KEY>&ps=100&p=2
http:///api/issues/search?componentKeys=YOUR_PROJECT_KEY>&ps=100&p=3
....
http:///api/issues/search?componentKeys=YOUR_PROJECT_KEY>&ps=100&p=442
Parse the result of every call and you will be obtained the list of your issues.
Cheers
Massimo
I am using translate API to translate some texts in my page, those texts are large html formated texts, so I had to develop a function that splits these texts into smaller pieces less than 4500 characters (including html tags) to avoid the limit of 5000 characters per request, also I had to modify the Google PHP API to allow send requests via POST.
I have enabled the paid version of the api in Goole Developers Console, and changed the total quota to 50M of characters per day and 500 requests/second/urser.
Now I am translating the whole database of texts with a script, it works fine but at some random points I revive the error "(403) User Rate Limit Exceeded", and I have to wait some minutes to re-run the script because when reached the error the api is returning the same error over and over until I wait some time.
I don't know why it keeps returning the error if I don't pass the number of requests, it's like it has some kind of maximum chaaracters per each interval of time or something...
You probably exceed the quota limits you set before: this is either the daily billable or the limit on the request characters per second.
To change the usage limits or request an increase to your quota, do the following:
1. Go to the Google Developers Console "https://console.developers.google.com/".
2. Select a project.
3. On the left sidebar, expand APIs & auth.
4. Click APIs.
5. Click the name of an activated API you're interested in "i.e. The Translate API".
6. Near the top of the info page for the API, click Quota.
If you have the billing enabled, just click Quota and it will take you to the quota page where you can view and change the quota-related settings.
If not, clicking Quota shows information about any free quota and limits that apply to the Translate API.
Google Developer Console has a rate limit of 10 requests per second, regardless of the settings or limits you may have changed.
You may be exceeding this limit.
I was unable to find any documentation around this, but could verify it myself with various API requests.
You control the characters limitation but not the concurrency
You are either making more than 500 concurrent request/second or you are using another Google API that is hitting such concurrency limitation.
The referer header is not set by default, but it is possible to add the headers to a request like so:
$result = $t->translate('Hola Mundo', [
'restOptions' => [
'headers' => [
'referer' => 'https://your-uri.com'
]
]
]);
If it makes more sense for you to set the referer at the client level (so all requests flowing through the client receive the header), this is possible as well:
$client = new TranslateClient([
'key' => 'my-api-key',
'restOptions' => [
'headers' => [
'referer' => 'https://your-uri.com'
]
]
]);
This worked for me!
Reference
In my case, this error was caused by my invalid payment information. Go to Billing area and make sure everything is ok.
I used Elasticsearch-1.1.0 to index tweets.
The indexing process is okay.
Then I upgraded the version. Now I use Elasticsearch-1.3.2, and I get this message randomly:
Exception happened: Error raised when there was an exception while talking to ES.
ConnectionError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)).
Snapshot of the randomness:
Happened --33s-- Happened --27s-- Happened --22s-- Happened --10s-- Happened --39s-- Happened --25s-- Happened --36s-- Happened --38s-- Happened --19s-- Happened --09s-- Happened --33s-- Happened --16s-- Happened
--XXs-- = after XX seconds
Can someone point out on how to fix the Read timed out problem?
Thank you very much.
Its hard to give a direct answer since the error your seeing might be associated with the client you are using. However a solution might be one of the following:
1.Increase the default timeout Globally when you create the ES client by passing the timeout parameter. Example in Python
es = Elasticsearch(timeout=30)
2.Set the timeout per request made by the client. Taken from Elasticsearch Python docs below.
# only wait for 1 second, regardless of the client's default
es.cluster.health(wait_for_status='yellow', request_timeout=1)
The above will give the cluster some extra time to respond
Try this:
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)
It might won't fully avoid ReadTimeoutError, but it minimalize them.
Read timeouts can also happen when query size is large. For example, in my case of a pretty large ES index size (> 3M documents), doing a search for a query with 30 words took around 2 seconds, while doing a search for a query with 400 words took over 18 seconds. So for a sufficiently large query even timeout=30 won't save you. An easy solution is to crop the query to the size that can be answered below the timeout.
For what it's worth, I found that this seems to be related to a broken index state.
It's very difficult to reliably recreate this issue, but I've seen it several times; operations run as normal except certain ones which periodically seem to hang ES (specifically refreshing an index it seems).
Deleting an index (curl -XDELETE http://localhost:9200/foo) and reindexing from scratch fixed this for me.
I recommend periodically clearing and reindexing if you see this behaviour.
Increasing various timeout options may immediately resolve issues, but does not address the root cause.
Provided the ElasticSearch service is available and the indexes are healthy, try increasing the the Java minimum and maximum heap sizes: see https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html .
TL;DR Edit /etc/elasticsearch/jvm.options -Xms1g and -Xmx1g
You also should check if all fine with elastic. Some shard can be unavailable, here is nice doc about possible reasons of unavailable shard https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/
I am running a very simple performance experiment where I post 2000 documents to my application.
Who in tern persists them to a relational DB and sends them to Solr for indexing (Synchronously, in the same request).
I am testing 3 use cases:
No indexing at all - ~45 sec to post 2000 documents
Indexing included - commit after each add. ~8 minutes (!) to post and index 2000 documents
Indexing included - commitWithin 1ms ~55 seconds (!) to post and index 2000 documents
The 3rd result does not make any sense, I would expect the behavior to be similar to the one in point 2. At first I thought that the documents were not really committed but I could actually see them being added by executing some queries during the experiment (via the solr web UI).
I am worried that I am missing something very big. Is it possible that committing after each add will degrade performance by a factor of 400?!
The code I use for point 2:
SolrInputDocument = // get doc
SolrServer solrConnection = // get connection
solrConnection.add(doc);
solrConnection.commit();
Where as the code for point 3:
SolrInputDocument = // get doc
SolrServer solrConnection = // get connection
solrConnection.add(doc, 1); // According to API documentation I understand there is no need to call an explicit commit after this
According to this wiki:
https://wiki.apache.org/solr/NearRealtimeSearch
the commitWithin is a soft-commit by default. Soft-commits are very efficient in terms of making the added documents immediately searchable. But! They are not on the disk yet. That means the documents are being committed into RAM. In this setup you would use updateLog to be solr instance crash tolerant.
What you do in point 2 is hard-commit, i.e. flush the added documents to disk. Doing this after each document add is very expensive. So instead, post a bunch of documents and issue a hard commit or even have you autoCommit set to some reasonable value, like 10 min or 1 hour (depends on your user expectations).