H2 Console starts on IP address which is not mine - h2

If I run
H2JAR=/common/home/jjs/.m2/repository/com/h2database/h2/1.3.168/h2-1.3.168.jar
java -cp $H2JAR org.h2.tools.Server $*
I get
Web Console server running at http://68.178.232.99:8082 (only local connections)
TCP server running at tcp://68.178.232.99:9092 (only local connections)
PG server running at pg://68.178.232.99:5435 (only local connections)
But I have
1004 ~\>traceroute 68.178.232.99
traceroute to 68.178.232.99 (68.178.232.99), 30 hops max, 60 byte packets
1 190.33.189.161 (190.33.189.161) 9.145 ms 9.023 ms 9.467 ms
2 172.31.36.254 (172.31.36.254) 171.169 ms 171.083 ms 170.976 ms
3 10.255.6.9 (10.255.6.9) 170.811 ms 170.641 ms 170.529 ms
4 ge-0-0-0.bal1-int-1.jf1-agr-1.cwpanama.net (201.224.254.237) 170.416 ms 170.306 ms 170.193 ms
5 so-7-1-3.mia11.ip4.tinet.net (216.221.158.49) 185.066 ms 186.763 ms 188.797 ms
6 xe-2-2-0.mia10.ip4.tinet.net (89.149.184.254) 189.751 ms xe-8-0-0.mia10.ip4.tinet.net (89.149.180.185) 202.777 ms xe-1-0-0.mia10.ip4.tinet.net (89.149.183.21) 202.611 ms
7 ge-0-2-2.mpr2.mia1.us.above.net (64.125.13.81) 211.130 ms 215.839 ms 217.518 ms
8 xe-4-0-0.cr2.iah1.us.above.net (64.125.30.202) 219.719 ms 221.003 ms 228.238 ms
9 xe-1-1-0.mpr4.phx2.us.above.net (64.125.30.149) 219.337 ms 225.518 ms 228.868 ms
10 209.66.64.6.t01121-04.above.net (209.66.64.6) 228.763 ms 214.909 ms 215.359 ms
my host file is:
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

H2 tries to detect the IP address of your computer. It seems it doesn't work correctly in your case. Could you run the network test of the H2 database? You would need to download the .zip file of H2, expand it, chmod the build.sh file, and then run:
./build.sh testNetwork
In my case the result is:
Target: testNetwork
localhost:
localhost/127.0.0.1
localhost/127.0.0.1
localhost/0:0:0:0:0:0:0:1
localhost/fe80:0:0:0:0:0:0:1%1
getLocalHost:Thomass-MacBook-Pro.local/192.168.0.104
/192.168.0.104
byName:/192.168.0.104
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=63498]
time: 0
server accepting
client:/192.168.0.104:63498
time: 8
server accepted:Socket[addr=/192.168.0.104,port=63499,localport=63498]
client:Socket[addr=/192.168.0.104,port=63498,localport=63499]
time: 2
server read:123
client read:234
server closing
server done
time: 202
done
Done in 1626 ms
This will not solve the problem, but it will give more information about what H2 tries to do to detect the IP address.

Related

Dronekit-sitl fails to bind on default port 5760

I have dronekit-sitl installed in a python3 virtual environment on my Windows 10 machine and have used it before by running dronekit-sitl copter with no issues. However, as of today I am running across what seems to be a permission issue when trying to execute the ArduCopter sitl.
$ dronekit-sitl copter
os: win, apm: copter, release: stable
SITL already Downloaded and Extracted.
Ready to boot.
Execute: C:\Users\kyrlon\.dronekit\sitl\copter-3.3\apm.exe --home=-35.363261,149.165230,584,353 --model=quad -I 0
SITL-0> Started model quad at -35.363261,149.165230,584,353 at speed 1.0
SITL-0.stderr> bind port 5760 for 0
Starting sketch 'ArduCopter'
bind failed on port 5760 - Operation not permitted
Starting SITL input
Not sure what might have triggered a new operation permission issue, and I tried to start over with a fresh Python environment, but even after a complete PC shutdown, I am still having the error as shown above.
It turns out that having docker on my system was the culprit and excluding the port I was attempting to use as mentioned in this SO post that led me to this github issue. Running the command in an elevated terminal:
netsh interface ipv4 show excludedportrange protocol=tcp
Provided me the results of the following excluded ports:
Protocol tcp Port Exclusion Ranges
Start Port End Port
---------- --------
1496 1595
1658 1757
1758 1857
1858 1957
1958 2057
2058 2157
2180 2279
2280 2379
2380 2479
2480 2579
2702 2801
2802 2901
2902 3001
3002 3101
3102 3201
3202 3301
3390 3489
3490 3589
3590 3689
3693 3792
3793 3892
3893 3992
3993 4092
4093 4192
4193 4292
4293 4392
4393 4492
4493 4592
4593 4692
4768 4867
4868 4967
5041 5140
5141 5240
5241 5340
5357 5357
5358 5457
5458 5557
5558 5657
5700 5700
5701 5800
8005 8005
8884 8884
15202 15301
15302 15401
15402 15501
15502 15601
15602 15701
15702 15801
15802 15901
15902 16001
16002 16101
16102 16201
16202 16301
16302 16401
16402 16501
16502 16601
16602 16701
16702 16801
16802 16901
16993 17092
17093 17192
50000 50059 *
* - Administered port exclusions.
Turns out that docker or possibly Hyper-V excluded the range that included 5760:
5701 5800
And as mentioned from the github issue, I probably resolved this issue before after a set number of restarts that incremented the port ranges, or possibly got lucky in the past starting dronekit-sitl before docker ran on my system.
Either way, to resolve this issue of Operation not permitted, running the command as admin:
net stop winnat
net start winnat
solved the issue with dronekit-sitl without having to specify a different port besides the default 5760.

NPM Install is giving Error: connect ETIMEDOUT (Without proxy)

I just created a new .Net Core 2.0 Angualar project template and get a strange error on restoring the packages.
Part of the output:
npm ERR! fetch failed https://registry.npmjs.org/extend/-/extend-3.0.1.tgz
npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 151.101.0.162:443
Later part of the output:
23980 error argv "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\Web\\External\\Node.exe" "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\Web\\External\\node_modules\\npm\\bin\\npm-cli.js" "install"
23981 error node v5.4.1
23982 error npm v3.3.4
23983 error code ETIMEDOUT
23984 error errno ETIMEDOUT
23985 error syscall connect
23986 error network connect ETIMEDOUT 151.101.0.162:443
23986 error network This is most likely not a problem with npm itself
23986 error network and is related to network connectivity.
23986 error network In most cases you are behind a proxy or have bad network settings.
23986 error network
23986 error network If you are behind a proxy, please make sure that the
23986 error network 'proxy' config is set properly. See: 'npm help config'
23987 verbose exit [ 1, true ]
Most solutions I found was about setting up the proxy. I have no proxy.
I also found this answer:
npm config set proxy false
npm cache clean
Which does nothing change.
I also resetted my network adapter and pc. Also I tried ipconfig -flushdns.
tracert is also working fine:
Routenverfolgung zu a.sni.fastly.net [151.101.0.162]
über maximal 30 Hops:
1 4 ms 1 ms <1 ms 192.168.1.1
2 28 ms 28 ms 28 ms loopback0.80.rdsl.99.nue.de.net.telefonica.de [62.52.195.28]
3 34 ms 29 ms 27 ms bundle-ether3.0001.dbrx.01.nue.de.net.telefonica.de [62.53.3.154]
4 * * * Zeitüberschreitung der Anforderung.
5 34 ms 36 ms 32 ms ae14-0.0001.corx.01.fra.de.net.telefonica.de [62.53.22.86]
6 32 ms 32 ms 32 ms bundle-ether15.0002.dbrx.02.fra.de.net.telefonica.de [62.53.26.0]
7 31 ms 38 ms 36 ms ae6-0.0001.prrx.11.fra.de.net.telefonica.de [62.53.19.146]
8 31 ms 30 ms 31 ms fastly2.fra.ecix.net [62.69.146.89]
9 33 ms 31 ms 35 ms 151.101.0.162
Ablaufverfolgung beendet.
If I call https://registry.npmjs.org/extend/-/extend-3.0.1.tgz in browser, I get the package offered for download.
Any pointers?
I have no idea what is happening...
But as a workaround, I have installed the Node MSI, and configured visual studio to use this one instead.
Tools > Options > Projects & Solutions > Web Package Management > External Web Tools
Now it is working. But I feel wrong using this workaround. If anybody has some pointers, how to fix the root of the problem, I would be more than happy.

dotnet core 2 long build time because of long restore time

I noticed that building in dotnet core 2 seemed a lot slower.
But the timing after the build always showed 'only' 15 seconds.
I couldn't believe that so I timed it with time.
> time dotnet build
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
hrm -> /Users/r/dev/hrm/bin/Debug/netcoreapp2.0/hrm.dll
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:15.45
real 0m52.366s
user 0m36.851s
sys 0m15.458s
That seemed more correct. Almost a minute.
I then tried without restore and it was a lot faster:
> time dotnet build --no-restore
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
hrm -> /Users/r/dev/hrm/bin/Debug/netcoreapp2.0/hrm.dll
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:15.39
real 0m15.795s
user 0m11.397s
sys 0m4.238s
But dotnet also shows 15 seconds.
Could it be that only building is counted in the timings?
Not sure why a restore is always slow when everything is already restored.
Are there other ways I could speed up the building process? Disable telemetry? (I'm using osx, my environment is set to development)
I prefer to use dotnet watch run but that seems even slower.
Running dotnet watch to view the parameters is taking 12 seconds.
> time dotnet watch
Microsoft DotNet File Watcher 2.0.0-rtm-26452
Usage: dotnet watch [options] [[--] <arg>...]
Options:
....
real 0m12.631s
user 0m8.880s
sys 0m3.816s
Is this only on my system?
Update:
Here is the result from dotnet restore /clp:PerformanceSummary
> dotnet restore /clp:PerformanceSummary
Restore completed in 43.95 ms for /Users/roeland/dev/hrm/hrm.csproj.
Restore completed in 52.73 ms for /Users/roeland/dev/hrm/hrm.csproj.
Restore completed in 38.48 ms for /Users/roeland/dev/hrm/hrm.csproj.
Project Evaluation Performance Summary:
36252 ms /Users/roeland/dev/hrm/hrm.csproj 3 calls
Project Performance Summary:
36424 ms /Users/roeland/dev/hrm/hrm.csproj 9 calls
24359 ms Restore 1 calls
1 ms _IsProjectRestoreSupported 2 calls
12011 ms _GenerateRestoreProjectPathWalk 1 calls
1 ms _GenerateRestoreProjectPathItemsPerFramework 1 calls
43 ms _GenerateRestoreGraphProjectEntry 1 calls
0 ms _GetRestoreSettingsPerFramework 1 calls
6 ms _GenerateProjectRestoreGraph 1 calls
3 ms _GenerateProjectRestoreGraphPerFramework 1 calls
Target Performance Summary:
0 ms _GenerateRestoreGraphProjectEntry 1 calls
0 ms _GenerateProjectRestoreGraph 1 calls
0 ms _GetRestoreTargetFrameworksAsItems 1 calls
0 ms _GetRestoreProjectStyle 2 calls
0 ms CheckForImplicitPackageReferenceOverridesBeforeRestore 2 calls
0 ms _CheckForUnsupportedNETCoreVersion 1 calls
0 ms _IsProjectRestoreSupported 1 calls
0 ms _GetRestoreSettingsPerFramework 1 calls
0 ms _GetProjectJsonPath 2 calls
0 ms _GetRestoreSettingsOverrides 1 calls
1 ms _GenerateRestoreProjectPathWalk 1 calls
1 ms _GenerateRestoreProjectPathItemsPerFramework 1 calls
1 ms _GenerateRestoreSpecs 1 calls
1 ms _GenerateRestoreProjectSpec 1 calls
2 ms _GenerateProjectRestoreGraphPerFramework 1 calls
2 ms _GetRestoreTargetFrameworksOutput 1 calls
5 ms _GenerateRestoreDependencies 1 calls
10 ms _LoadRestoreGraphEntryPoints 1 calls
20 ms _GenerateDotnetCliToolReferenceSpecs 1 calls
21 ms _GetRestoreSettings 1 calls
54 ms _GenerateRestoreGraph 1 calls
216 ms Restore 1 calls
12007 ms _GenerateRestoreProjectPathItems 1 calls
12014 ms _GetAllRestoreProjectPathItems 1 calls
12058 ms _FilterRestoreGraphProjectInputItems 1 calls
Task Performance Summary:
1 ms Message 3 calls
1 ms ConvertToAbsolutePath 2 calls
1 ms GetRestorePackageReferencesTask 1 calls
1 ms GetRestoreProjectReferencesTask 1 calls
2 ms GetRestoreProjectFrameworks 1 calls
3 ms RemoveDuplicates 5 calls
4 ms WarnForInvalidProjectsTask 1 calls
18 ms GetRestoreSettingsTask 1 calls
20 ms GetRestoreDotnetCliToolsTask 1 calls
216 ms RestoreTask 1 calls
36121 ms MsBuild 9 calls
Long story short: MSBuild scans the entire folder structure based on glob patterns defined by the SDK used. This is done for each project evaluation and the NuGet restore seems to trigger at least three full evaluations.
Since it is slow to scan large directories, the SDKs define globbing patterns used to exclude some known large directories that are usually not wanted as part of the project anyway (node_modules, bower_components etc.).
It has been known that special circumstances may circumvent these optimisations and or even trigger performance bugs in the include/exclude glob pattern expansion / matching.
As a precaution, add all folders known to be excluded to the DefaultItemExcludes property (inside of a <PropertyGroup> element):
<DefaultItemExcludes>custom\node_modules\**;$(DefaultItemExcludes)</DefaultItemExcludes>
For me excluding.git folder helped to make build around 10x faster.
<PropertyGroup>
<DefaultItemExcludes>.git\**;$(DefaultItemExcludes)</DefaultItemExcludes>
</PropertyGroup>

Unable to access tomcat manager 8080 in Google Cloud

I have been using Amazon EC2 to run my Tomcat+MySQL website for a while and is now migrating to Google Cloud Platform. I start a compute engine instance (Ubuntu 16.04), connect to it via ssh and use apt-get to install mysql/tomcat7.
The problem I encountered is that tomcat will not start. The catalina.out log didn't have a "Server startup at xxxms" message, and I can't connect to 8080 port via browser.
The last several lines of catalina.out is
Jul 10, 2017 7:06:20 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 928 ms
Jul 10, 2017 7:06:20 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina
Jul 10, 2017 7:06:20 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.68 (Ubuntu)
Jul 10, 2017 7:06:20 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/host-manager.xml
Jul 10, 2017 7:06:21 PM org.apache.catalina.startup.TldConfig execute INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
When I use netstat to check, it shows user tomcat7 is listening to 8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 115 32984 -
$ id -u tomcat7
$ 115
I try to wget localhost:8080 in ssh terminal, it shows
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response...
and just hang there.
Any idea or suggestion will be greatly appreciated!
Update
It turns out that firewall is not the root cause of the problem, and even without allowing port 8443 Tomcat will work (Of coz you need to allow 8080). The reason that there's no "Server started" message showing up is Tomcat take extremely long time to startup (1346049 ms the first time, 354034 ms when restarted, no web app installed except for the default index.html), and the reason for no responding to request is also that it has not finished starting up yet.
This is the first time I have seen that Tomcat takes so long to start and also the reason I didn't realize it in the first place. I suspect (with some search) this is caused by Tomcat Jar scanning. Will keep update this question once I have more detail.
Update - Problem Solved
It turns out that I encounter the same problem here and the solution is here. In short, much of the time is consumed by the following task:
Creation of SecureRandom instance for session ID generation using [SHA1PRNG]
which require Java to load /dev/random to get random numbers. /dev/random typically get its entropy source from keyboard/mouse input, which cannot provide enough randomness on a headless virtual machine. This causes the random number to be "used up" during computation and cause a lot of wait. The solution is to install haveged, which use some other source to provide randomness (details in the link).
I installed haveged, and now tomcat only takes 1 sec to startup and everything works normal.
Thanks for asking such interesting question.
I've never used Google Cloud services but I managed to replicate your issue.
After reading a little I found that you need to update your Firewall Rules to enable access to 8080 port.
Go to:
1) (Hamburguer Icon, upper left)
2) Networking
3) Firewall Rules
4) Add new
I created one called 'allow-tomcat7' with this properties:
Descripción
Enables Tomcat 7 access
Red
default
Prioridad
1000
Dirección
Entrada
Acción tras coincidencia
Permitir
Filtros de origen
Intervalos de IP
0.0.0.0/0
Protocolos y puertos
tcp:8080
tcp:8443
udp:8080
There's an option for 'target tags' when you edit the configuration, although I've created a 'tag' and applied it only to my new EC instance it didn't work. I had to remove target tags and it worked like a charm:
Make sure you allow access only for your IP address!
You'll need to adjust your security settings, otherwise, you'll become a honeypot, once I've enabled the port for everyone several bots started to scan it:
daychuzleo#testing-tomcat:~$ sudo tcpdump -i ens4 port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), capture size 262144 bytes
20:39:31.437634 IP 170.251.221.183.54162 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [.], seq 1638030511:1638030512, ack 1250919796, win 259, length 1: HTTP
20:39:31.437665 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.54162: Flags [.], ack 1, win 231, options [nop,nop,sack 1 {0:1}], length 0
20:39:37.133899 IP 170.251.221.183.53878 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [.], seq 2436191518:2436191519, ack 4071767590, win 259, length 1: HTTP
20:39:37.133930 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.53878: Flags [.], ack 1, win 222, options [nop,nop,sack 1 {0:1}], length 0
20:39:51.379839 IP 170.251.221.183.54162 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [F.], seq 1, ack 1, win 259, length 0
20:39:51.392375 IP 170.251.221.183.47923 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [S], seq 1420913913, win 8192, options [mss 1386,nop,wscale 8,nop,nop,sackOK,unknown-76 0x01010a18e9680005,unknown-76 0x0c01,nop,eol], length 0
20:39:51.392410 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.47923: Flags [S.], seq 507557961, ack 1420913914, win 28400, options [mss 1420,nop,nop,sackOK,nop,wscale 7], length 0
20:39:51.421934 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.54162: Flags [.], ack 2, win 231, length 0
20:39:51.586555 IP 170.251.221.183.47923 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [.], ack 1, win 259, length 0
20:39:51.590317 IP 170.251.221.183.47923 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [P.], seq 1:389, ack 1, win 259, length 388: HTTP: GET / HTTP/1.1
20:39:51.590337 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.47923: Flags [.], ack 389, win 231, length 0
I was unable to make it work with wget, but I think with this you'll found it out.
UPDATE:
I forgot to mention some things you may have not configured:
-Allowing Firewall for HTTP and HTTPS in your VM instance:
Try using a web navigator (Chrome, Firefox) don't use wget.
Verify that you're not being filtered by your company firewall, try testing with 4g in your cell phone or an unrestricted network, or just ask your IT team to allow you access to the temporary public IP (and port) generated (each time).
Start the service using:
sudo service tomcat7 start
Try reinstalling tomcat
Other things I did (in the research process)
Moving the service to IPV4 instead of IPV6
daychuzleo#testing-tomcat:~$ netstat -ntpl
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
To do it, edit the default tomcat and add in JavaOPTS the IPV4 option:
vim /etc/default/tomcat
JAVA_OPTS="-Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true"
Disable the 8443 redirection
Comment the section "redirectPort" in server.xml:
vim /etc/tomcat/server.xml
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
address="0.0.0.0"/>
<!--redirectPort="8443" />-->
Verify each change by restarting your tomcat instance.

Why does a simple Thin server stop responding at 16500 requests when benchmarking? [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
'ab' program freezes after lots of requests, why?
Here's a simple test server:
require 'rubygems'
require 'rack'
require 'thin'
class HelloWorld
def call(env)
[200, {"Content-Type" => "text/plain"}, "OK"]
end
end
Rack::Handler::Thin.run HelloWorld.new, :Port => 9294
#I've tried with these added too, 'rack.multithread' => true, 'rack.multiprocess' => true
Here's a test run:
$ ab -n 20000 http://0.0.0.0:9294/sdf
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 0.0.0.0 (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
Completed 12000 requests
Completed 14000 requests
Completed 16000 requests
apr_poll: The timeout specified has expired (70007)
Total of 16347 requests completed
It breaks down at around 16500. Why? How can I find out what's going on. Is it GC in ruby or is it something with number of available network sockets on an OS X machine. I have a MPB 2.5 Ghz 6G memory.
Edit
After some discussion here and testing various things, it seems like changing net.inet.tcp.msl from 15000 to 1000ms makes the problem of testing high frequency web servers with ab go away.
sudo sysctl -w net.inet.tcp.msl=1000 # this is only good for local development
See referenced question with the answer to this problem. 'ab' program freezes after lots of requests, why?
I'll add the solution here for claritys sake. The correct solution for managing to do high frequency tests with ab on os X is to change the 'net.inet.tcp.msl' setting from 15000ms to 1000ms. This should only be done on development boxes.
sudo sysctl -w net.inet.tcp.msl=1000 # this is only good for local development
This answer was found after the good detective work performed in the comments here and comes from an answer to a very similar question here's the answer: https://stackoverflow.com/a/6699135/155031
I think I've got it.
When ab makes connections to your test server, it opens a source port (say, 50134) and makes a connection to the destination port (9294).
The ports that ab opens for the source port are determined by the sysctl settings net.inet.ip.portrange.first and net.inet.ip.portrange.last. For example, on my machine:
philippotter ~ $ sysctl -a | grep ip.portrange
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535
This means that ab's source ports will be in the range from 49152 to 65535, which is a total of 16384.
HTTP is a TCP protocol. When a TCP connection is closed, it goes into the TIME_WAIT state, while it waits for any remaining in-transit packets to reach their destinations. This means that the port is not usable for any other purpose until the timeout is reached.
So, putting all of this together, ab uses up all available source ports very quickly; they go into the TIME_WAIT state; they can't be reused; ab is unable to create any more connections.
You can see this if you kill ab when it hangs, and run it again -- it won't be able to create any connections!

Resources