Rainbows worker is being killed after timeout even though it replied - ruby

I have a Sinatra app running on Rainbows.
I log the following :
before do
logger.info("#{Process.pid} #{Time.now} #{request.ip} #{request.path_info} # {params.to_s}")
end
and
after do
logger.info("#{Process.pid} #{Time.now} #{request.ip} #{request.path_info} #{params.to_s} => #{response.headers['X-API-Status']} (#{response.successful?})")
end
and in my logs I can read:
25988 2012-11-13 11:57:52 +0100 192.168.90.1 /req {"u"=>"810000027"}
25988 2012-11-13 11:57:59 +0100 192.168.90.1 /req {"u"=>"810000027"} => 200 (true)
192.168.90.1 - - [13/Nov/2012 11:57:59] "POST /req HTTP/1.1" **200** 14 7.5862
25988 2012-11-13 11:57:59 +0100 192.168.90.1 /req {"u"=>"810000027"}
25988 2012-11-13 11:57:59 +0100 192.168.90.1 /req {"u"=>"810000027"} => 200 (true)
192.168.90.1 - - [13/Nov/2012 11:57:59] "POST /req HTTP/1.1" 200 14 0.0223
E, [2012-11-13T11:58:04.099913 #25875] ERROR -- : worker=2 PID:**25988** timeout (12s > 11s), killing
E, [2012-11-13T11:58:04.106428 #25875] ERROR -- : reaped #<Process::Status: pid 25988 SIGKILL (signal 9)> worker=2
My worker (pid 25988) is being killed as if it had not responded to the first request... But it has, obviously! It even handled another request (and I use Base concurrency model -> no concurrency)
My Rainbows configuration is:
Rainbows! do
timeout(10)
end
listen(3000)
pid('/tmp/rainbows.pid')
stderr_path('/var/log/rainbows.log')
stdout_path('/var/log/rainbows.log')
working_directory('/opt/app')
worker_processes(4)
Do you have any idea of what happens ? Or how I could investigate further ?
Thanks !

Actually the issue lied in the request that was "kept alive" too long by the client (flash). Apparently there is no way to properly close the tcp connection in AS3...
I fixed my issue with:
Rainbows! do
keepalive_timeout(0)
end
which seems appropriate for me anyway.

Related

The debugger does not pause at the next breakpoint and trace doesn't report anything

When I use curl to post a SOAP message into message flow, it takes 9 seconds to respond. Debug won't stop at breakpoints and User Trace doesn't report anything. Meanwhile when I do the same request from Postman or SoapUI (the first message takes the same amount of time, later all messages take around 70 - 200ms) debugger and user trace work as intended. What is cause of this behavior?
IBM App Connect Enterprise 11.0.0.8
curl --trace-time output:
03:29:07.484000 * Trying <host>...
03:29:07.484000 * TCP_NODELAY set
03:29:07.531000 * Connected to <host_name> (<host>) port 7800 (#0)
03:29:07.531000 > POST /service HTTP/1.1
03:29:07.531000 > Host: <host_name>:7800
03:29:07.531000 > User-Agent: curl/7.55.1
03:29:07.531000 > Accept: */*
03:29:07.531000 > Content-Length: 508
03:29:07.531000 > Content-Type: application/x-www-form-urlencoded
03:29:07.531000 >
03:29:07.546000 * upload completely sent off: 508 out of 508 bytes
03:29:16.671000 < HTTP/1.1 200 OK
03:29:16.671000 < Cache-Control: no-cache
03:29:16.671000 < Pragma: no-cache
03:29:16.687000 < Expires: -1
03:29:16.687000 < X-AspNet-Version: 4.0.30319
03:29:16.687000 < X-Powered-By: ASP.NET
03:29:16.687000 < Date: Fri, 22 May 2020 01:29:14 GMT
03:29:16.687000 < Content-Type: text/xml; charset=utf-8
03:29:16.703000 < Server: IBM App Connect Enterprise
03:29:16.703000 < Content-Length: 243
EDIT: I'm still trying to resolve the problem - this time with help of Wireshark and user trace:
curl:
02:56:37.781000 > POST /service HTTP/1.1
after few milliseconds Wireshark detects POST message from "curl machine" - that means there are no problems with the connection
after around 10s delay SoapInput receives data. Why it takes so long?
2020-05-23 02:56:37.257076 6220 UserTrace BIP11304I: The Parser of type 'MQROOT' has been deleted from address '0x131f1312190'. This thread now has '0' cached parsers.
2020-05-23 02:56:40.591580 3684 UserTrace BIP11303I: A Parser of type 'MQROOT' has been created at address '0x131f13144a0'. This thread now has '36' cached parsers.
2020-05-23 02:56:45.143380 3684 UserTrace BIP11501I: Received data from input node 'SOAP Input'.
The input node 'SOAP Input' has received data and has propagated it to the message flow 'link'.
2020-05-23 02:56:45.143880 3684 UserTrace BIP6060I: Node 'link.SOAP Input' used parser type 'Properties' to process a portion of the incoming message of length '0' bytes beginning at offset '0'.
Fixed by restarting the machine. Any clue what exactly caused this problem?

jersey+springboot can't response correct http status [duplicate]

I've encountered the same issue as in this question, using Spring Boot 1.3.0 and not having my controllers annotated with #RestController, just #Path and #Service. As the OP in that question says,
this is, to me, anything but sensible
I also can't understand why would they have it redirect to /error. And it is very likely that I'm missing something, because I can only give back 404s or 200s to the client.
My problem is that his solution doesn't seem to work with 1.3.0, so I have the following request flow: let's say my code throws a NullPointerException. It'll be handled by one of my ExceptionMappers
#Provider
public class GeneralExceptionMapper implements ExceptionMapper<Throwable> {
private static final Logger LOGGER = LoggerFactory.getLogger(GeneralExceptionMapper.class);
#Override
public Response toResponse(Throwable exception) {
LOGGER.error(exception.getLocalizedMessage());
return Response.status(Response.Status.INTERNAL_SERVER_ERROR).build();
}
}
And my code returns a 500, but instead of sending it back to the client, it tries to redirect it to /error. If I don't have another resource for that, it'll send back a 404.
2015-12-16 18:33:21.268 INFO 9708 --- [nio-8080-exec-1] o.glassfish.jersey.filter.LoggingFilter : 1 * Server has received a request on thread http-nio-8080-exec-1
1 > GET http://localhost:8080/nullpointerexception
1 > accept: */*
1 > host: localhost:8080
1 > user-agent: curl/7.45.0
2015-12-16 18:33:29.492 INFO 9708 --- [nio-8080-exec-1] o.glassfish.jersey.filter.LoggingFilter : 1 * Server responded with a response on thread http-nio-8080-exec-1
1 < 500
2015-12-16 18:33:29.540 INFO 9708 --- [nio-8080-exec-1] o.glassfish.jersey.filter.LoggingFilter : 2 * Server has received a request on thread http-nio-8080-exec-1
2 > GET http://localhost:8080/error
2 > accept: */*
2 > host: localhost:8080
2 > user-agent: curl/7.45.0
2015-12-16 18:33:37.249 INFO 9708 --- [nio-8080-exec-1] o.glassfish.jersey.filter.LoggingFilter : 2 * Server responded with a response on thread http-nio-8080-exec-1
2 < 404
And client's side (curl):
$ curl -v http://localhost:8080/nullpointerexception
* STATE: INIT => CONNECT handle 0x6000572d0; line 1090 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* Trying ::1...
* STATE: CONNECT => WAITCONNECT handle 0x6000572d0; line 1143 (connection #0)
* Connected to localhost (::1) port 8080 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x6000572d0; line 1240 (connection #0)
* STATE: SENDPROTOCONNECT => DO handle 0x6000572d0; line 1258 (connection #0)
> GET /nullpointerexception HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.45.0
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x6000572d0; line 1337 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x6000572d0; line 1464 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x6000572d0; line 1474 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 404 Not Found
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Content-Length: 0
< Date: Wed, 16 Dec 2015 17:33:37 GMT
<
* STATE: PERFORM => DONE handle 0x6000572d0; line 1632 (connection #0)
* Curl_done
* Connection #0 to host localhost left intact
So it's always a 404. Unless I do have such an /error resource, then what? what am I supposed to return? All I have at that point is a GET request to /error. And I don't want those extra requests consuming resources and polluting my logs.
What am I missing? And if nothing, what should I do with my exception handling?
You can set the Jersey property ServerProperties.RESPONSE_SET_STATUS_OVER_SEND_ERROR to true.
Whenever response status is 4xx or 5xx it is possible to choose between sendError or setStatus on container specific Response implementation. E.g. on servlet container Jersey can call HttpServletResponse.setStatus(...) or HttpServletResponse.sendError(...).
Calling sendError(...) method usually resets entity, response headers and provide error page for specified status code (e.g. servlet error-page configuration). However if you want to post-process response (e.g. by servlet filter) the only way to do it is calling setStatus(...) on container Response object.
If property value is true the method Response.setStatus(...) is used over default Response.sendError(...).
Type of the property value is boolean. The default value is false.
You can set Jersey property simply by calling property(key, value) in your ResourceConfig subclass constructor.

regex match one instance of a pattern that repeats

Given the following regex
/(500 Internal Server Error)/
How do we match only the 1st occurrence of this pattern in a single string that has many repeats of the same match?
For example:
Match 1
1. 500 Internal Server Error
Match 2
1. 500 Internal Server Error
Match 3
1. 500 Internal Server Error
How do we get 1. as our only answer?
Sample text for the match as follows (one big string on purpose)
[SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 500 Internal Server Error12:09:26,638 INFO [SoapUIProTestCaseRunner] Assertion [Match content of [question]] has status VALID12:09:26,638 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Connection 0.0.0.0:41494<->23.6.55.1:80 shut down12:09:26,638 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Connection 0.0.0.0:41494<->23.6.55.1:80 closed12:09:26,638 INFO [SoapUIProTestCaseRunner] Finished running SoapUI testcase [Test_fieldsParameter], time taken: 384ms, status: FINISHED12:09:26,640 INFO [SoapUIProTestCaseRunner] Assertion [Match content of [errorCode]] has status VALID12:09:26,641 INFO [SoapUIProTestCaseRunner] Assertion [Match content of [message]] has status VALID12:09:26,641 INFO [SoapUIProTestCaseRunner] Assertion [Valid HTTP Status Codes] has status VALID12:09:26,641 INFO [SoapUIProTestCaseRunner] running step [OtherSortBy]12:09:26,643 DEBUG [HttpClientSupport$SoapUIHttpClient] Stale connection check12:09:26,645 DEBUG [HttpClientSupport$SoapUIHttpClient] Attempt 1 to execute request12:09:26,646 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Sending request: GET /api/review/v1/questions?prodId=570043&_sortby=other HTTP/1.112:09:26,666 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 500 Internal Server Error12:09:26,667 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Connection 0.0.0.0:41475<->23.6.55.1:80 shut down12:09:26,667 DEBUG
Please have a look, this proves the string captured is the first:
str = "Match 1\n1. 500 Internal Server Error\nMatch 2\n1. 500 Internal Server Error\nMatch 3\n1. 500 Internal Server Error"
re = /(500 Internal Server Error)/
mdata = re.match(str)
puts mdata.begin(1)
puts mdata
Output of a sample program proving it is the first occurrence (index of 12 in the input string):
12
500 Internal Server Error

how to download large file on G-wan?

I use G-wan download files from directory 'www' ,it can download the files size less than 16M,if the files size is 16M, the G-wan is wrong:
Signal : 11:Unknown SIGSEGV problem
Signal src : 128:.
errno : 0
Thread : 1
Code Pointer: 0000004081a8 (module:gwan, function:??, line:0)
Access Address: 000000000000
Registers : EAX=7f1dbc0afa4e CS=00000033 EIP=0000004081a8 EFLGS=000000010287
EBX=7f1dc24af000 SS=d5ab0400 ESP=7f1dd59acd30 EBP=7f1dbc0afa4e
ECX=ffe3e5a1746a4230 DS=d5ab0400 ESI=1c997c47a33a4e FS=00000033
EDX=1c1a5e8b984000 ES=d5ab0400 EDI=7f1dbc0afa3e CS=00000033
Module :Function :Line # PgrmCntr(EIP) RetAddress FramePtr(EBP)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Segmentation fault
thanks for Gil.
Os: debian 7,rhel6.4
dwonload the file size is 80M ,file:0.0.0.0_8080/#0.0.0.0/www/MFC-7340-inst-B2-zh.EXE.this is trace content:
Wed, 14 Aug 2013 02:57:36 GMT: start
Wed, 14 Aug 2013 02:58:00 GMT 12 127.0.0.1
GET /MFC-7340-inst-B2-zh.EXE HTTP/1.1^M
Host: localhost:8080^M
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130806 Firefox/17.0 Iceweasel/17.0.8^M
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8^M
Accept-Language: en-US,en;q=0.5^M
Accept-Encoding: gzip, deflate^M
Connection: keep-alive^M
Cookie: SESS49960de5880e8c687434170f6476605b=iOQ_BgGEgAWbhB9zA0U5jZbEOXzmu3nkzhyBW00GJGM; Drupal.tableDrag.showWeight=0; Drupal.toolbar.collapsed=1; MOIN_SESSION_800_ROOT=8a77c4a58fefc1ad6b6830b5bc51ca680a480bae^M
^M
Wed, 14 Aug 2013 02:58:02 GMT: signal 11: Unknown SIGSEGV problem
source : 128:Kernel
thread : 0/2
state : SEND
client : 127.0.0.1:58472
request : /MFC-7340-inst-B2-zh.EXE
----------------
----------------
Signal : 11:Unknown SIGSEGV problem
Signal src : 128:.
errno : 0
Thread : 0
Code Pointer: 0000004081a8 (module:gwan, function:??, line:0)
Access Address: 000000000000
Registers : EAX=7f85482fd0c0 CS=00000033 EIP=0000004081a8 EFLGS=000000010293
EBX=7f8565501000 SS=6cd00400 ESP=7f856cbfcd30 EBP=7f85482fd0c0
ECX=ffeea9eeb46863d5 DS=6cd00400 ESI=11d59693c9ef4b FS=00000033
EDX=1156114b9a1e8b ES=6cd00400 EDI=7f85482fd0b0 CS=00000033
Module :Function :Line # PgrmCntr(EIP) RetAddress FramePtr(EBP)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
G-WAN has been used to serve much larger file sizes. Could you specify the exact file size you have used so we can try to duplicate this issue?
Generating a graceful crash report (your trace) is as easy as using the crash script examples given with the download archive so we need a bit more information to start answering your question.

Where's my time going mysteriously?

I have a ruby script that uses rsolr rubygem to generate XMLs and POST them to Apache Solr (javadoc Update Command) surfaced by Jetty Server. My script logs certain time using the following code
405 unless docs.empty?
406 begin
407 log.info("Adding to solr")
408 response = solr.add(docs)
409 log.info("#{(id_2*100.0)/last_id}% Done")
410 if response['responseHeader']['status'] != 0
411 log.fatal("Document ids not sent")
412 #log.fatal(Solr::Request::AddDocument.new(docs_single).to_s)
413 log.close
414 exit
415 end
416 log.info("#{Time.now.to_f - starttime}s to feed Solr. #{id_1} to #{id_2}")
417 rescue Exception => e
418 log.fatal("Document ids not sent => ")
419 #log.fatal(Solr::Request::AddDocument.new(docs_single).to_s)
420 #log.fatal(docs)
421 log.close
422 exit
423 end
The log generated goes like
I, [2011-10-09T15:03:42.617048 #30092] INFO -- : Executing - SELECT * FROM solr_feeddata_2 WHERE id >= 5879999 AND id < 5881999
I, [2011-10-09T15:03:44.086661 #30092] INFO -- : External Data1 fetch time: 1.45462989807129
I, [2011-10-09T15:03:44.109514 #30092] INFO -- : External Data2 fetch time: 0.0226790904998779
I, [2011-10-09T15:03:44.109611 #30092] INFO -- : 1.49255704879761s to fetch details from database. 5879999 to 5881999
I, [2011-10-09T15:03:44.109702 #30092] INFO -- : Adding data1, data2, building docs
I, [2011-10-09T15:03:45.912603 #30092] INFO -- : 3.29554414749146s to build documents. 5879999 to 5881999
I, [2011-10-09T15:03:45.912730 #30092] INFO -- : Adding to solr
I, [2011-10-09T15:04:24.797620 #30092] INFO -- : 61.180855194502% Done
I, [2011-10-09T15:04:24.797744 #30092] INFO -- : 42.180694103241s to feed Solr. 5879999 to 5881999
According to this log, Solr took (42.18 - 3.29 - 1.49 - 2) 35.4s to respond. (See below comment)
At the same time my Solr log for this particular update goes like
INFO: {add=[5879999, 5880000, 5880001, 5880002, 5880003, 5880004, 5880005, 5880007, ... (1468 adds)]} 0 5780
Oct 9, 2011 3:04:24 PM org.apache.solr.core.SolrCore execute
INFO: [core0] webapp=/solr path=/update params={wt=ruby} status=0 QTime=5780
Oct 9, 2011 3:04:42 PM org.apache.solr.update.processor.LogUpdateProcessor finish
This clearly shows that Solr took 5.78s to add the docs, initiate response send and closed the log updater.
Both the services run on same machine, inside the network, and their ping summary is
rtt min/avg/max/mdev = 0.008/0.010/0.022/0.006 ms
This pattern is clearly visible for every batch data processed. Despite my sincere efforts to get this mystery out, I am not able to get the reason for this behavior.
My Solr mergeFactor is 10, autoCommit is off.

Resources