I'm trying to read a lot of bulk data (probably around 1-2G) into neo4j with a simple ruby script and Neography. My code mostly just consists of a lot of create_node and create_relationship methods.
It seems to work fine, but after around 5,000 create methods I reach an error:
/home/earlz/.gem/ruby/2.1.0/gems/excon-0.44.3/lib/excon/socket.rb:127:in `connect_nonblock': Cannot assign requested address - connect(2) for 127.0.0.1:7474 (Errno::EADDRNOTAVAIL) (Excon::Errors::SocketError)
How do I fix this? I've tried increasing the HTTP timeouts and such, but this didn't help anything
It seems like your script is opening so many connections that it ran out of ephemeral ports to pick from. Try this:
echo "32768 61000" >/proc/sys/net/ipv4/ip_local_port_range
Related
What would make a write call to a TCPSocket hang indefinitely?
lotsOfBytes = # a really large number of bytes, like 1 or 2 MB of data
socket = TCPSocket.new # some config
socket.write(lotsOfBytes) # this line hangs
I am trying to debug an issue where a get_multi operation sent to memcached with a large number of keys hangs indefinitely, and it does so on a line that resembles that code snippet. I'm trying to better understand how the low-level sockets on which this library is built are expected to work.
What are the values of following attributes on your TCPSocket:
Keep-alive activated and what value is set?
Timeout set and what value is set?
If you will do a Wireshark dump, it's much better so see what happens before hanging connection.
tcpdump? are there any attempts to send anything?
netstat - for see output queue.
does it work on a small number of bytes in your environment?
We are running JMeter for connecting TCP Socket thorugh BinaryTCPClientImpl , We are getting the response code : 500
Response message: org.apache.jmeter.protocol.tcp.sampler.ReadException
JMeter Version : 2.9
Help out
If this is the error
ERROR - jmeter.protocol.tcp.sampler.TCPSampler: org.apache.jmeter.protocol.tcp.sampler.ReadException:
at org.apache.jmeter.protocol.tcp.sampler.BinaryTCPClientImpl.read(BinaryTCPClientImpl.java:140)
at org.apache.jmeter.protocol.tcp.sampler.TCPSampler.sample(TCPSampler.java:414)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
at java.lang.Thread.run(Unknown Source)
then you have 2 options. The first (and much easier if it applies
to you) is to use the LengthPrefixedBinaryTCPClientImpl. If this
applies to you, that is, if your responses are always the same fixed
sizes, you can simply set the tcp.binarylength.prefix.length property
and go about your business.
If that is not the case, then your other option is to extend
org.apache.jmeter.protocol.tcp.sampler.TCPClient. It may help to get in
touch with the client team of this proprietary protocol, because after
all, they have implemented something that works. You'll probably have
to extend it to look something like LengthPrefixedBinaryTCPClientImpl
read N bytes. Although, this runs the risks of reading too many or too
few bytes. If your application server ever miscalculates the size of
it's output, you suffer the consequences by getting another timeout or
leaving extra bytes in the buffer and reading them on the next iteration
(and then cascading errors).
I used Elasticsearch-1.1.0 to index tweets.
The indexing process is okay.
Then I upgraded the version. Now I use Elasticsearch-1.3.2, and I get this message randomly:
Exception happened: Error raised when there was an exception while talking to ES.
ConnectionError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=10)).
Snapshot of the randomness:
Happened --33s-- Happened --27s-- Happened --22s-- Happened --10s-- Happened --39s-- Happened --25s-- Happened --36s-- Happened --38s-- Happened --19s-- Happened --09s-- Happened --33s-- Happened --16s-- Happened
--XXs-- = after XX seconds
Can someone point out on how to fix the Read timed out problem?
Thank you very much.
Its hard to give a direct answer since the error your seeing might be associated with the client you are using. However a solution might be one of the following:
1.Increase the default timeout Globally when you create the ES client by passing the timeout parameter. Example in Python
es = Elasticsearch(timeout=30)
2.Set the timeout per request made by the client. Taken from Elasticsearch Python docs below.
# only wait for 1 second, regardless of the client's default
es.cluster.health(wait_for_status='yellow', request_timeout=1)
The above will give the cluster some extra time to respond
Try this:
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)
It might won't fully avoid ReadTimeoutError, but it minimalize them.
Read timeouts can also happen when query size is large. For example, in my case of a pretty large ES index size (> 3M documents), doing a search for a query with 30 words took around 2 seconds, while doing a search for a query with 400 words took over 18 seconds. So for a sufficiently large query even timeout=30 won't save you. An easy solution is to crop the query to the size that can be answered below the timeout.
For what it's worth, I found that this seems to be related to a broken index state.
It's very difficult to reliably recreate this issue, but I've seen it several times; operations run as normal except certain ones which periodically seem to hang ES (specifically refreshing an index it seems).
Deleting an index (curl -XDELETE http://localhost:9200/foo) and reindexing from scratch fixed this for me.
I recommend periodically clearing and reindexing if you see this behaviour.
Increasing various timeout options may immediately resolve issues, but does not address the root cause.
Provided the ElasticSearch service is available and the indexes are healthy, try increasing the the Java minimum and maximum heap sizes: see https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html .
TL;DR Edit /etc/elasticsearch/jvm.options -Xms1g and -Xmx1g
You also should check if all fine with elastic. Some shard can be unavailable, here is nice doc about possible reasons of unavailable shard https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/
I have installed FastRWeb 1.1-0 on an installation of R 2.15.2 (Trick or Treat) running on an Ubuntu 10.04 box. I hope to use the resulting system to run a web service.
I've configured the system by setting http.port to 8181 in rserve.conf and unsetting the socket destination. I've assigned .http.request to FastRWeb::.http.request. I exchange JSON blobs between the client and the server using HTTP POST (the second blob can exceed 150KB in size, and will not fit in an HTTP GET query string.)
Everything works end to end -- I have a little client-side R script which generates JSON RPC calls across the channel. I see the run function invoked, and see it returned.
I've run into a significant performance problem, however: the return path takes in excess of 12 seconds from the time run() returns (including the call to done()) and the time that the R client gets the return value. RCurl doesn't seem to be the culprit; it appears that something is taking twelve seconds to do a return.
Does anybody have any suggestions of where to look? I can easily shift over to using Apache 2.0 and CGI, but, honestly, I'd rather keep everything R centric.
Answering my own question.
I wrapped .http.request with an Rprof()/Rprof(NULL) pair and looked at the time spent in each routine. It turns out that the system spends ~11 seconds inside URLDecode in the standard implementation of .run. This looks like a scaling problem in URLDecode in the core.
I'm having trouble with an old legacy app that recently started crashing. I'm trying to investigate the DebugDiag analysis, but don't have much luck. Either there is a sql query that locks and somehow the calling thread doesn't die away? Then again callstack points to oledb32!CImpIErrorInfo::GetHelpFile+a1.
Here's the info from DebugDiag which I think is relevant to this problem:
The following threads in w3wp.exe_MyApp_PID_7572_Date__10_21_2010__Time_08_43_22AM_720_Manual Dump.dmp are making a database operation using ADO.
The call to MSADO15!CERRORLOOKUP::GETHELPINFO originated from oledb32!CImpIErrorInfo::GetHelpFile+a1
...clip...clip...
Thread 17 - System ID 4160
Entry point msvcrt!_endthreadex+2f
Create time 21.10.2010 0:08:16
Time spent in user mode 0 Days 00:11:22.781
Time spent in kernel mode 0 Days 00:27:49.953
This thread is making a database operation using ADO.
The call to MSADO15!CERRORLOOKUP::GETHELPINFO originated from oledb32!CImpIErrorInfo::GetHelpFile+a1
Function Source
ntdll!GetUILangID+31
ntdll!LdrpSearchResourceSection_U+186
ntdll!LdrFindResource_U+18
kernel32!FindResourceExW+65
user32!LoadStringOrError+31
user32!LoadStringW+18
msado15!FetchInfo+ba
msado15!CErrorLookup::GetHelpInfo+1e
oledb32!CImpIErrorInfo::GetHelpFile+a1
msvbvm60!ExecProj::SetModuleCount+a
msvbvm60!CEcProjTypeComp::Release+4
msvbvm60!RcmConstructModuleInstance+8f
oleaut32!DispCallFunc+16a
msvbvm60!VBStrToLong+cf
msvbvm60!FileOutString+bb
msvbvm60!_vbaPrintObj+51
MSWCRUN!DllUnregisterDesigner+8ad3
MSWCRUN!DllUnregisterDesigner+accb
MSWCRUN!DllUnregisterDesigner+af8c
MSWCRUN!DllUnregisterDesigner+a7de
MSWCRUN!DllUnregisterDesigner+7b51
MyApp!DllCanUnloadNow+212e
oleaut32!DispCallFunc+16a
msvbvm60!VBStrToLong+cf
msvbvm60!FileOutString+bb
msvbvm60!_vbaPrintObj+51
MSWCRUN!DllUnregisterDesigner+8ad3
MSWCRUN!DllUnregisterDesigner+7d13
MSWCRUN!DllUnregisterDesigner+6e64
MSWCRUN!DllUnregisterDesigner+9097
MSWCRUN!DllUnregisterDesigner+8fa6
vbscript!IDispatchInvoke2+b2
vbscript!IDispatchInvoke+59
vbscript!InvokeDispatch+13a
vbscript!InvokeByName+42
vbscript!CScriptRuntime::RunNoEH+234c
vbscript!CScriptRuntime::Run+62
vbscript!CScriptEntryPoint::Call+51
vbscript!CSession::Execute+c8
vbscript!COleScript::ExecutePendingScripts+144
vbscript!COleScript::SetScriptState+14d
asp!CActiveScriptEngine::TryCall+19
asp!CActiveScriptEngine::Call+31
asp!CallScriptFunctionOfEngine+5b
asp!ExecuteRequest+17e
asp!Execute+24c
asp!CHitObj::ViperAsyncCallback+3f0
asp!CViperAsyncRequest::OnCall+92
comsvcs!CSTAActivityWork::STAActivityWorkHelper+32
ole32!EnterForCallback+c4
ole32!SwitchForCallback+1a3
ole32!PerformCallback+54
ole32!CObjectContext::InternalContextCallback+159
ole32!CObjectContext::DoCallback+1c
comsvcs!CSTAActivityWork::DoWork+12d
comsvcs!CSTAThread::DoWork+18
comsvcs!CSTAThread::ProcessQueueWork+37
comsvcs!CSTAThread::WorkerLoop+190
msvcrt!_endthreadex+a3
kernel32!BaseThreadStart+34
...clip...clip...
Client connection from 194.241.111.228:26238 to 81.175.250.2:80
Host Header 81.175.250.2:80
GET request for /MyApp/netk.asp
HTTP Version HTTP/1.1
SSL Request False
Time alive 00:49:33
QueryString
Request mapped to
HTTP Request State HTR_READING_CLIENT_REQUEST
Native Request State NREQ_STATE_PROCESS
Hard to say, but I'd start with throwing ProcessMonitor/RegMon/FileMon/TcpViewer from live.sysinternals.com. Fiddler wouldn't be a bad idea either.
Then, if you still get no clues, I'd break out WinDBG, which is always my nuclear option, because the learning curve is so massive. But, assuming, you learn the commands, you could break on the crash, then walk the stack backwards and potentially figure out where the error is coming from.
And of course, you could just reinstall everything and that will probably solve all your problems.