select * from foo (100 000 rows) takes 4 seconds, is this normal? - performance

I have postgres 8.4 installed on ubuntu server with 4 GB Ram and Intel E5504 2Ghz
I've created one table
create table foo
(
id serial primary key,
fname varchar(30),
lname varchar(30)
)
the insert of 10 000 rows takes about 4 seconds first time and 1 second after
but the select of 100 000 rows takes 4 seconds always,
select * from foo
is this normal or my configuration could be wrong ?
could it be that my ubuntu remote bandwith is limited or something like that ?

100000 rows of your table use 6.4 MBytes (or 12.4 MBytes if they are Unicode). This corresponds to 64 MBits, which takes about 6.4 seconds in a 10 MBit/s network. Therefore, the available network bandwidth may cause the 4 second delay that you are experiencing.

Try to check how much time will take a query ran locally:
Connect to local database using psql:
psql -U username dbname
Turn on displaying of timing information:
dbname=> \timing
Timing is on.
Set output to local temporary file:
dbname=> \o /tmp/foo.txt
Select all rows from your table to temporary file:
dbname=> select * from foo;
Time: 104.442 ms
Check TCP bandwidth between your server and your client:
(On client - I assume Linux) Temporarily disable firewall:
# service iptables stop
(On client) Start listening on port 10000:
# nc -l 10000 > /dev/null
(On server) Send a file to client using plain TCP:
# time nc client_ip 10000 < /tmp/foo.txt
real 0m0.190s
user 0m0.004s
sys 0m0.078s
(On client) Enable firewall back:
# service iptables start

Related

Large result is slow anywhere but local

I have a fairly large query running on Clickhouse. The problem is when running on localhost using cmd line it takes about 0.7 sec to complete. This is consistently fast. Issue is when querying from C# / HTTP / Postman. Here it takes about 10 times to return the data. (the size is about 3-4mb) so I dont think its a size issue.
I have tried to monitor network latency, but nothing to notice here.
On the host it works like a charm, but outside it does not :(.... what to do.
I exptect the latency to be a few 100 ms, but turns out to be 7 sec :/
check timings with curl https://clickhouse.yandex/docs/en/interfaces/http/
https://stackoverflow.com/a/22625150
and compare local vs remote
CH HTTP usually provides almost the same performance as TCP and HTTP could be faster for small resultsets (like 10 rows)
Again. The problem is not the HTTP.
Example:
time clickhouse-client -q "select number, arrayMap(x->sipHash64(number,x), range(10)) from numbers(10000)" >native.out
real 0m0.034s
time curl -S -o http.out 'http://localhost:8123/?query=select%20number%2C%20arrayMap(x-%3EsipHash64(number%2Cx)%2C%20range(10))%20from%20numbers(10000)'
real 0m0.017s
ls -l http.out native.out
2108707 Oct 1 16:17 http.out
2108707 Oct 1 16:17 native.out
10 000 rows - 2Mb
HTTP is faster 0.017s VS 0.034s
Canada -> Germany (openvpn)
time curl -S -o http.out 'http://user:xxx#cl.host.x:8123/?query=select%20number%2C%20arrayMap(x-%3EsipHash64(number%2Cx)%2C%20range(10))%20from%20numbers(10000)'
real 0m1.619s
ping cl.host.x
PING cl.host.x (10.253.52.6): 56 data bytes
64 bytes from 10.253.52.6: icmp_seq=0 ttl=61 time=131.710 ms
64 bytes from 10.253.52.6: icmp_seq=1 ttl=61 time=133.711 ms

Is my VPS under DDoS atack?

When i run this command on my VPS:
netstat -n|grep :80|cut -c 45-|cut -f 1 -d ':'|sort|uniq -c|sort -nr|more
i get this result:
207 222.73.144.194
89 191.96.249.54
58 191.96.249.53
21 2400
15 51.255.64.23
6 143.137.103.251
3 103.27.72.36
1 89.180.150.168
1 66.102.7.137
1 5.189.170.167
1 191.181.39.208
1 183.2.246.218
I think this command is showing the number of connections per IP to port 80.
Is this a DDoS atack?
Have you check another aspect (eg. cpu load, network throughput to specific ip(s) using iftop or iptraf)? If there's normal, maybe it's just web/http scanner to your web.
If you're use nginx, you can use limit_conn module, to queue the rest of ip(s) if didn't obey your policy.

HikariCP: What database level timeouts should be considered to set maxLifetime for Oracle 11g

In the documentation for HikariCP, it is mentioned that
We strongly recommend setting this value, and it should be at least 30 seconds less than any database-level connection timeout.
What are those database-level connection timeouts that should be taken into account for Oracle11.2 database? And how could I find those timeouts (queries to execute)?
Short answer: none (by default).
For the record (to include details here in case the link changes), we're talking about property maxLifetime of HikariCP:
This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed. We strongly recommend setting this value, and it should be at least 30 seconds less than any database or infrastructure imposed connection time limit. A value of 0 indicates no maximum lifetime (infinite lifetime), subject of course to the idleTimeout setting. Default: 1800000 (30 minutes)
In my experience, it's a good thing that HikariCP does that. As far as I can tell by default Oracle does not enforce a max lifetime for connections (neither on JDBC driver side (1), nor on server side(2)). So in this respect, the "infrastructure-imposed connection time limit" is +infinity -- and that was a problem for us, as we did observe issues with long-lived connections. It also means any value is "at least 30 seconds less", including the default :)
I suppose the connection layer does not do anything about this because it counts on the pool layer above to handle such things. It was not possible with (now deprecated) implicit connection pool, and I don't know if UCP (the replacement) does that, but if you use HikariCP you don't use those.
Now, after 30 minutes (usually after many reuses for various purposes) for a given connection, HikariCP closes it and creates a fresh one. That has a very minor cost, and fixed our issues with long-lived connections. We're happy with that default, but still make it configurable just in case (see 2 below).
(1) OracleDataSource does not offer any configuration point (property or system property) to control that, and I observed infinite lifetime.
(2) For server-side limits, see profile parameter IDLE_TIME. Quoting
this answer:
Oracle by default will not close a connection due to inactivity. You can configure a profile with an IDLE_TIME to cause Oracle to close inactive connections.
To verify what is the value of IDLE_TIME for your user, combining answers from this Q&A:
select p.limit
from dba_profiles p, dba_users u
where p.resource_name = 'IDLE_TIME' and p.profile = u.profile and u.username = '...'
;
Default value is UNLIMITED.
Please note there can be other limits enforced elsewhere (firewall... yes I've been bitten by that, although most DB systems have a keep-alive mechanism) that might interfere. So you'd better make it configurable, in case such issues are discovered when you deploy your product.
On Linux, you can verify max lifetime of physical connections by monitoring TCP sockets connected to your database. I've been running script below on my server (from the DB point of view that's the client host), it takes 1 argument, the ip:port of your oracle node, as it appears in output of netstat -tan (or a pattern if you have several nodes).
#!/bin/bash
target="$1"
dir=$(mktemp -d)
while sleep 10
do
echo "------------ "$(date)
now=$(date +%s)
netstat -tan | grep " $target " | awk '{print $4}' | cut -f2 -d: | while read port
do
file="p_$port"
[ ! -e $file ] && touch $file
ftime=$(stat -c %Z "$file")
echo -e "$port :\t "$(( now - ftime))
done
done
\rm "$dir"/p_*
\rmdir "$dir"
If you run that and stop it with ctrl-c during sleep time, it should exit the loop and clean up the temp directory, but this is not 100% foolproof
In the results none of the ports should show a value that exceeds 1800 seconds (ie 30 minutes), give or take a minute. See example output below, first sample shows 2 sockets above 1800s, they're gone 10s later.
------------ Thu Jul 6 16:09:00 CEST 2017
49806 : 1197
49701 : 1569
49772 : 1348
49782 : 1317
49897 : 835
49731 : 1448
49620 : 1830
49700 : 1569
49986 : 523
49722 : 1498
49715 : 1509
49711 : 1539
49629 : 1820
49732 : 1448
50026 : 332
49849 : 1036
49858 : 1016
------------ Thu Jul 6 16:09:10 CEST 2017
49806 : 1207
49701 : 1579
49772 : 1358
49782 : 1327
49897 : 845
49731 : 1458
49700 : 1579
49986 : 533
49722 : 1508
49715 : 1519
49711 : 1549
49732 : 1458
50026 : 342
49849 : 1046
49858 : 1026
You'll need to run the script for more than 30 minutes to see that, because it does not know age of sockets that existed before

Oracle Database CPU Usage on AIX

I want to find the CPU process usage for all Oracle processes on an AIX box.
On Solaris I can do the following:
prstat -n 400 -c -s cpu -p 9013 1 1
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
9013 oracle 3463M 2928M sleep 53 0 0:00:35 0.9% oracle/2
Total: 1 processes, 2 lwps, load averages: 2.25, 2.32, 2.40
This basically reports the CPU usage for a given process ID (in this case 9013). Given a list of all Oracle PID’s I can use this command to get the CPU usage for each one, sum them up and hey presto I have my Oracle database CPU usage.
How can I get the same with AIX?
Thanks
You can try nmon or topas, which will show the current %CPU. You might also want to look into using WLM to create a class for all the Oracle processes, then use wlmstat to see the CPU usage for that class. That would save you the trouble of adding them up manually.

What parameters to tweak for a text-based PG restore?

Every night we dump and restore a 200 GB database using:
# Production, PG 9:
pg_dump DATNAME | some-irrelevant-pipe
# QA, PG 8.3:
some-irrelevant-pipe | psql -d DATNAME
I had to go for text-based backups in order to restore a dump from 9 on 8.3.
The restore is painfully and unreasonably slow. I noticed my log is full of these:
2011-05-22 08:02:47 CDT LOG: checkpoints are occurring too frequently (9 seconds apart)
2011-05-22 08:02:47 CDT HINT: Consider increasing the configuration parameter "checkpoint_segments".
2011-05-22 08:02:54 CDT LOG: checkpoints are occurring too frequently (7 seconds apart)
2011-05-22 08:02:54 CDT HINT: Consider increasing the configuration parameter "checkpoint_segments".
My question is: Is it possible that the setting of checkpoint_segments is the bottleneck? What other parameters can I tweak to speed up the process?
That machine has 4 GB RAM. Other possibly relevant settings in postgresql.conf are:
shared_buffers = 1000MB
work_mem = 200MB
maintenance_work_mem = 200MB
effective_cache_size = 2000MB
# fsync and checkpoint settings are default
Did you read this ? See specially sec 14.4.9
For the purposes of restoring a database, change:
# I don't think PostgreSQL 8.3 supports synchronous_commit
synchronous_commit = off
# only change fsync = off if your version of PG is too old to support synchronous_commit. If you do support synchronous_commit, don't ever change fsync to anything but on. Ever.
#fsync = off
checkpoint_segments = 25
Regarding checkpoint_segments, set that value to the size of your disk controller's write buffer. 25 = 400MB
Also, make sure your psql is loading everything in a single transaction:
some-irrelevant-pipe | psql -1 -d DATNAME

Resources