I am trying to configure warmup queries in solrconfig.xml on Solr version 4.10.3, but no matter how we do it the cache seems to disappear after about a minute or so, and then the first search again takes about 20 secs., with subsequent searches coming straight away.
The query looks like this (filter is the variable search-term):
solr/Nyheder/select?q=overskrift:" & filter & "+OR+underrubrik:" & filter & "+OR+tekst:" & filter&fl=id+oprettet+overskrift+underrubrik+tekst+pix
&sort=oprettet+desc
And the solrConfig.xml section (which seems to help nothing) looks like this (it is similar for the event="firstSearcher"):
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">*:*</str>
<str name="sort">oprettet desc</str>
<str name="fl">id oprettet overskrift underrubrik tekst pix</str>
</lst>
<lst>
<str name="q">overskrift:* OR underrubrik:* OR tekst:*</str>
<str name="sort">oprettet desc</str>
<str name="fl">id oprettet overskrift underrubrik tekst pix</str>
</lst>
</arr>
</listener>
Edit: added commit configuration
<autoCommit>
<maxTime>120000</maxTime>
<openSearcher>true</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
Everytime you do a softcommit or a hard commit, your caches are more or less invalidated, since any type of commit will generally create a "newsearcher".
You probably have a softcommit or hardcommit set to 1 minute.
Check this link and see the sections Soft Commits and Hard commits.
It turned out that this was not related to SolR at all but to internal routing.
For those interested it turned out that ipv6 DNS was checked and needed to timeout before the IPv4 address was used, so the delay was from a DNS lookup+timeout and NOT from SOLr.
Related
I have a web-based frontend (localhost, currently) that uses Ajax to query Solr.
It's working well, but if I submit a single space (nothing else) in the input/search box, the URL in the browser shows
...#q=%20
and in that circumstance I get a 400 error, and my web page stalls (doesn't refresh), apparently waiting for a response from Solr.
By comparison, if I submit a semicolon (;) rather than a space, then the page immediately refreshes, albeit with no results (displaying 0 to 0 of 0; expected).
My question is what is triggering the " " (%20) query fault in Solr, and how do I address it in solrconfig.xml?
Update. This solrconfig.xml entry resolved the q=%20 issue I described.
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="hl">off</str>
<str name="wt">html</str>
<str name="df">query</str>
<!-- *** ADDITION: *** -->
<str name="defType">edismax</str>
</lst>
</requestHandler>
Note: development code, dummy data.
Before: https://www.youtube.com/watch?v=GhkA4XlqWds&list=PLTtHvbtVBhk6zAqSJ3A1-shsD_h5i3IuP&index=2
After: https://www.youtube.com/watch?v=507DdPOx1xA&list=PLTtHvbtVBhk6zAqSJ3A1-shsD_h5i3IuP&index=1
Additional discussion (reddit:Solr): https://old.reddit.com/r/Solr/comments/kt97ql/solr_query_with_space_only_q20_gives_error/
I used Elasticsearch-1.1.0 to index tweets.
The indexing process is okay.
Then I upgraded the version. Now I use Elasticsearch-1.3.2, and I get this message randomly:
Exception happened: Error raised when there was an exception while talking to ES.
ConnectionError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)).
Snapshot of the randomness:
Happened --33s-- Happened --27s-- Happened --22s-- Happened --10s-- Happened --39s-- Happened --25s-- Happened --36s-- Happened --38s-- Happened --19s-- Happened --09s-- Happened --33s-- Happened --16s-- Happened
--XXs-- = after XX seconds
Can someone point out on how to fix the Read timed out problem?
Thank you very much.
Its hard to give a direct answer since the error your seeing might be associated with the client you are using. However a solution might be one of the following:
1.Increase the default timeout Globally when you create the ES client by passing the timeout parameter. Example in Python
es = Elasticsearch(timeout=30)
2.Set the timeout per request made by the client. Taken from Elasticsearch Python docs below.
# only wait for 1 second, regardless of the client's default
es.cluster.health(wait_for_status='yellow', request_timeout=1)
The above will give the cluster some extra time to respond
Try this:
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)
It might won't fully avoid ReadTimeoutError, but it minimalize them.
Read timeouts can also happen when query size is large. For example, in my case of a pretty large ES index size (> 3M documents), doing a search for a query with 30 words took around 2 seconds, while doing a search for a query with 400 words took over 18 seconds. So for a sufficiently large query even timeout=30 won't save you. An easy solution is to crop the query to the size that can be answered below the timeout.
For what it's worth, I found that this seems to be related to a broken index state.
It's very difficult to reliably recreate this issue, but I've seen it several times; operations run as normal except certain ones which periodically seem to hang ES (specifically refreshing an index it seems).
Deleting an index (curl -XDELETE http://localhost:9200/foo) and reindexing from scratch fixed this for me.
I recommend periodically clearing and reindexing if you see this behaviour.
Increasing various timeout options may immediately resolve issues, but does not address the root cause.
Provided the ElasticSearch service is available and the indexes are healthy, try increasing the the Java minimum and maximum heap sizes: see https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html .
TL;DR Edit /etc/elasticsearch/jvm.options -Xms1g and -Xmx1g
You also should check if all fine with elastic. Some shard can be unavailable, here is nice doc about possible reasons of unavailable shard https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/
I am running a very simple performance experiment where I post 2000 documents to my application.
Who in tern persists them to a relational DB and sends them to Solr for indexing (Synchronously, in the same request).
I am testing 3 use cases:
No indexing at all - ~45 sec to post 2000 documents
Indexing included - commit after each add. ~8 minutes (!) to post and index 2000 documents
Indexing included - commitWithin 1ms ~55 seconds (!) to post and index 2000 documents
The 3rd result does not make any sense, I would expect the behavior to be similar to the one in point 2. At first I thought that the documents were not really committed but I could actually see them being added by executing some queries during the experiment (via the solr web UI).
I am worried that I am missing something very big. Is it possible that committing after each add will degrade performance by a factor of 400?!
The code I use for point 2:
SolrInputDocument = // get doc
SolrServer solrConnection = // get connection
solrConnection.add(doc);
solrConnection.commit();
Where as the code for point 3:
SolrInputDocument = // get doc
SolrServer solrConnection = // get connection
solrConnection.add(doc, 1); // According to API documentation I understand there is no need to call an explicit commit after this
According to this wiki:
https://wiki.apache.org/solr/NearRealtimeSearch
the commitWithin is a soft-commit by default. Soft-commits are very efficient in terms of making the added documents immediately searchable. But! They are not on the disk yet. That means the documents are being committed into RAM. In this setup you would use updateLog to be solr instance crash tolerant.
What you do in point 2 is hard-commit, i.e. flush the added documents to disk. Doing this after each document add is very expensive. So instead, post a bunch of documents and issue a hard commit or even have you autoCommit set to some reasonable value, like 10 min or 1 hour (depends on your user expectations).
I want a cache that checks its own items if they are expired or not. My cache config is below:
<?xml version="1.0" encoding="UTF-8"?>
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>subscriberinfo</cache-name>
<scheme-name>distributed-scheme</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>distributed-scheme</scheme-name>
<lease-granularity>member</lease-granularity>
<service-name>DistributedCache</service-name>
<serializer>
<instance>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>rbm-shovel-pof-config.xml</param-value>
</init-param>
</init-params>
</instance>
</serializer>
<backing-map-scheme>
<local-scheme>
<unit-calculator>BINARY</unit-calculator>
<expiry-delay>24h</expiry-delay>
<flush-delay>180</flush-delay>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
</caching-schemes>
</cache-config>
But the thing is, flush-delay can not be set. Any ideas?
Thanks
which version of Coherence do you use ?
In Coherence 3.7, the flush-delay has been removed from dtd as deprecated since version 3.5.
Flushing is just active when inserting new objects (have a look at eviction-policy) or accessing expired objects (look at expiry-delay).
Coherence deprecated the FlushDelay and related settings in 3.5. All of that work is done automatically now:
Expired items are automatically removed, and the eviction / expiry events are raised accordingly
You will never see expired data in the cache; even if you try to access it just as it expires, the expiry will occur as an event before the data access occurs
Eviction (for memory limits) is now done asynchronously, so that the "sharp edges" of the side-effects of eviction (such as event storms) are spread out across natural cycles (with the cycle length calculated based on the estimated rate of eviction)
I hope that helps.
For the sake of full disclosure, I work at Oracle. The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.
But the thing is, flush-delay can not be set. Any ideas?
What do you mean by this? Does the system throw errors or the expired items not getting removed from cache. Based on the configuration you have, the entry should be removed from cache after 24hours and 180seconds since last update to the entry.
Delay in initial retrieving of xml data using XRM
I am using this xml query
<fetch mapping='logical'> <entity name='de_municipality'> <order attribute='de_name' ascending='true' /> <attribute name='de_municipalityid'/> <attribute name='de_name'/> </entity> </fetch>"
For the Fetch method of XRM. It is taking around 10 seconds to get the result (though there are only limited number of entities).
ResultsXml = dc.UsingService(service => (string)service.Fetch(oFetchXml.InnerXml));
This is line of code is making the delay!!!
Next time execution will give results in 120 milli seconds.
I have experimented this query with the CRM4, the delay is only 200 milli seconds for all the attempts.
Any tips or trick or analysis method will be appreciated.
I got a reply from MSDN like this...
Hi Vinu,
we made a call to Microsoft about this issue and it was confirmed to be a design problem. This shouldn't be an issue anymore with CRM 2011.
Our current workaround is to keep the web application wich consumes the DataContext alive for as long as possible, because once the Metadata is cached the call does not occur any further.
Be careful - the DataContext not only caches the Metadata but also the content itself - such as attributes and relationships. If you want to refresh these you can clear the cache partially for specific entities as described here: Empty CRM Client DataContext cache
Kind regards
Markus
Markus Wolff
Senior Software Developer CRM Systems
Gruner & Jahr & Co. KG Hamburg, Germany