How to understand ORA-00060 deadlock trace file - oracle
Recently I got a ORA-00060 deadlock error.
I read this post and this post and this post but I'm not sure what the problem is: unindexed FK cause this or other problem.
My question is how to understand this trace file and how to solve it?
Below is the trace file:
*** 2015-06-02 14:53:45.513
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TM-00014d94-00000000 497 556 S 332 1414 SX
TM-00014d94-00000000 332 1414 SX 416 1038 S
TX-0011000f-000000ab 416 1038 X 302 457 S
TM-00014d94-00000000 302 457 SX 497 556 S
session 556: DID 0001-01F1-0000000D session 1414: DID 0001-014C-00000022
session 1414: DID 0001-014C-00000022 session 1038: DID 0001-01A0-0000000C
session 1038: DID 0001-01A0-0000000C session 457: DID 0001-012E-00000028
session 457: DID 0001-012E-00000028 session 556: DID 0001-01F1-0000000D
Rows waited on:
Session 556: obj - rowid = 00014D94 - AAAAAAAAAAAAAAAAAA
(dictionary objn - 85396, file - 0, block - 0, slot - 0)
Session 1414: obj - rowid = 00014D94 - AAAAAAAAAAAAAAAAAA
(dictionary objn - 85396, file - 0, block - 0, slot - 0)
Session 1038: obj - rowid = 00014D94 - AAAAAAAAAAAAAAAAAA
(dictionary objn - 85396, file - 0, block - 0, slot - 0)
Session 457: obj - rowid = 00014FA0 - AAAU+gAAEAAAft+AAA
(dictionary objn - 85920, file - 4, block - 129918, slot - 0)
----- Information for the OTHER waiting sessions -----
Session 1414:
sid: 1414 ser: 1424 audsid: 100128 user: 91/SW flags: 0x45
pid: 332 O/S info: user: oracle, term: UNKNOWN, ospid: 10179
image: oracle#jwdb
client details:
O/S info: user: root, term: unknown, ospid: 1234
machine: localhost.localdomain program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
insert into t_course_takes (created_at, updated_at, attend, course_id, course_take_type_id, election_mode_id, lesson_id, limit_group_id, paid, remark, semester_id, state, std_id, turn, virtual_cost, id) values (:1 , :2 , :3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 , :12 , :13 , :14 , :15 , :16 )
Session 1038:
sid: 1038 ser: 951 audsid: 100212 user: 91/SW flags: 0x45
pid: 416 O/S info: user: oracle, term: UNKNOWN, ospid: 10343
image: oracle#jwdb
client details:
O/S info: user: root, term: unknown, ospid: 1234
machine: localhost.localdomain program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
delete from t_course_takes where id=:1
Session 457:
sid: 457 ser: 2983 audsid: 100099 user: 91/SW flags: 0x45
pid: 302 O/S info: user: oracle, term: UNKNOWN, ospid: 10111
image: oracle#jwdb
client details:
O/S info: user: root, term: unknown, ospid: 1234
machine: jdbcclient program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
insert into t_elect_loggers (created_at, updated_at, course_code, course_name, course_take_type_id, course_type, credits, election_mode_id, ip_address, lesson_no, operator_code, operator_name, project_id, remark, screening, semester_id, std_code, std_name, turn, type, virtual_orig, virtual_rest, id) values (:1 , :2 , :3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 , :12 , :13 , :14 , :15 , :16 , :17 , :18 , :19 , :20 , :21 , :22 , :23 )
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=ca9jc1g44ap41) -----
delete from t_course_takes where id=:1
===================================================
PROCESS STATE
-------------
Process global information:
process: 0x9d0fd98c8, call: 0x95429a500, xact: 0x922b10710, curses: 0x94110e198, usrses: 0x94110e198
----------------------------------------
SO: 0x9d0fd98c8, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x9d0fd98c8, name=process, file=ksu.h LINE:11459, pg=0
(process) Oracle pid:497, ser:7, calls cur/top: 0x95429a500/0x95429a500
flags : (0x0) -
flags2: (0x0), flags3: (0x0)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 9
last post received-location: ksq.h LINE:1877 ID:ksqrcl
last process to post me: 900fd0348 12 0
last post sent: 0 0 9
last post sent-location: ksq.h LINE:1877 ID:ksqrcl
last process posted by me: 900fce2c8 17 0
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 0x90102ea98
O/S info: user: oracle, term: UNKNOWN, ospid: 10507
OSD pid info: Unix process pid: 10507, image: oracle#jwdb
Dump of memory from 0x0000000921009C90 to 0x0000000921009E98
921009C90 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
921009E90 00000000 00000000 [........]
(FOB) flags=2050 fib=0x902c3c2b8 incno=0 pending i/o cnt=0
fname=/home/jwdb/oracle/oradata/orcl/undotbs01.dbf
fno=3 lblksz=8192 fsiz=238080
(FOB) flags=2050 fib=0x902c3c8b8 incno=0 pending i/o cnt=0
fname=/home/jwdb/oracle/oradata/orcl/users01.dbf
fno=4 lblksz=8192 fsiz=603680
(FOB) flags=2050 fib=0x902c3b6a0 incno=0 pending i/o cnt=0
fname=/home/jwdb/oracle/oradata/orcl/system01.dbf
fno=1 lblksz=8192 fsiz=92160
----------------------------------------
SO: 0x94110e198, type: 4, owner: 0x9d0fd98c8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x9d0fd98c8, name=session, file=ksu.h LINE:11467, pg=0
(session) sid: 556 ser: 3224 trans: 0x922b10710, creator: 0x9d0fd98c8
flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40008) -/-
DID: , short-term DID:
txn branch: (nil)
oct: 7, prv: 0, sql: 0x9cfed0be0, psql: 0x9afeb20c8, user: 91/SW
ksuxds FALSE at location: 0
service name: SYS$USERS
client details:
O/S info: user: root, term: unknown, ospid: 1234
machine: localhost.localdomain program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
Current Wait Stack:
0: waiting for 'enq: TM - contention'
name|mode=0x544d0004, object #=0x14d94, table/partition=0x0
wait_id=787 seq_num=896 snap_id=53
wait times: snap=0.023828 sec, exc=4 min 18 sec, total=4 min 18 sec
wait times: max=infinite, heur=4 min 18 sec
wait counts: calls=87 os=87
in_wait=1 iflags=0x15a0
There is at least one session blocking this session.
Dumping first 3 direct blockers:
inst: 1, sid: 457, ser: 2983
inst: 1, sid: 1353, ser: 5618
inst: 1, sid: 907, ser: 5215
Dumping final blocker:
inst: 1, sid: 1168, ser: 1194
There are 1 sessions blocked by this session.
Dumping one waiter:
inst: 1, sid: 1136, ser: 2212
wait event: 'enq: TM - contention'
p1: 'name|mode'=0x544d0004
p2: 'object #'=0x14d94
p3: 'table/partition'=0x0
row_wait_obj#: 85396, block#: 0, row#: 0, file# 0
min_blocked_time: 12 secs, waiter_cache_ver: 32536
Wait State:
fixed_waits=0 flags=0x23 boundary=(nil)/-1
Session Wait History:
elapsed time of 0.000000 sec since current wait
0: waited for 'latch: enqueue hash chains'
address=0x9313226a0, number=0x1c, tries=0x0
wait_id=839 seq_num=895 snap_id=1
wait times: snap=0.226082 sec, exc=0.226082 sec, total=0.226082 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
1: waited for 'enq: TM - contention'
name|mode=0x544d0004, object #=0x14d94, table/partition=0x0
wait_id=787 seq_num=894 snap_id=52
wait times: snap=3.000901 sec, exc=4 min 18 sec, total=4 min 18 sec
wait times: max=infinite
wait counts: calls=87 os=87
occurred after 0.000000 sec of elapsed time
2: waited for 'latch: enqueue hash chains'
address=0x6000cf38, number=0x1c, tries=0x0
wait_id=838 seq_num=893 snap_id=1
wait times: snap=0.000142 sec, exc=0.000142 sec, total=0.000142 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
3: waited for 'enq: TM - contention'
name|mode=0x544d0004, object #=0x14d94, table/partition=0x0
wait_id=787 seq_num=892 snap_id=51
wait times: snap=12.003822 sec, exc=4 min 15 sec, total=4 min 15 sec
wait times: max=infinite
wait counts: calls=86 os=86
occurred after 0.000000 sec of elapsed time
4: waited for 'latch: enqueue hash chains'
address=0x6000cf38, number=0x1c, tries=0x0
wait_id=837 seq_num=891 snap_id=1
wait times: snap=0.000170 sec, exc=0.000170 sec, total=0.000170 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
5: waited for 'enq: TM - contention'
name|mode=0x544d0004, object #=0x14d94, table/partition=0x0
wait_id=787 seq_num=890 snap_id=50
wait times: snap=6.001627 sec, exc=4 min 3 sec, total=4 min 3 sec
wait times: max=infinite
wait counts: calls=82 os=82
occurred after 0.000000 sec of elapsed time
6: waited for 'latch: enqueue hash chains'
address=0x6000cf38, number=0x1c, tries=0x0
wait_id=836 seq_num=889 snap_id=1
wait times: snap=0.000378 sec, exc=0.000378 sec, total=0.000378 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
7: waited for 'enq: TM - contention'
name|mode=0x544d0004, object #=0x14d94, table/partition=0x0
wait_id=787 seq_num=888 snap_id=49
wait times: snap=3.000543 sec, exc=3 min 57 sec, total=3 min 57 sec
wait times: max=infinite
wait counts: calls=80 os=80
occurred after 0.000000 sec of elapsed time
8: waited for 'latch: enqueue hash chains'
address=0x6000cf38, number=0x1c, tries=0x0
wait_id=835 seq_num=887 snap_id=1
wait times: snap=0.000350 sec, exc=0.000350 sec, total=0.000350 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
9: waited for 'enq: TM - contention'
name|mode=0x544d0004, object #=0x14d94, table/partition=0x0
wait_id=787 seq_num=886 snap_id=48
wait times: snap=3.000880 sec, exc=3 min 54 sec, total=3 min 54 sec
wait times: max=infinite
wait counts: calls=79 os=79
occurred after 0.000000 sec of elapsed time
Updated on 6/5/2015
The whole trace file is here
This ORA-00060 was causing problem to my team also. The trace file was showing the blocking of two session each other as TX--- X --------S. In our case we were sure that the multithreaded environment is distributed on unique_number column such that two threads will not update the same record. Further down the trace file it showed "enq: TX - allocate ITL entry". When the processing was running I checked the kind of waits:
select blocking_session, sid, serial#, wait_class, seconds_in_wait
from
v$session
where
blocking_session is not NULL
order by
blocking_session;
And it was showing the wait class is "Configure". So, I increased the initrans of the tables used by the processing. Then also reduced the batch size so that commit happens more frequently then earlier. After that the problem is solved.
Related
Can't add node to the cockroachde cluster
I'm staking to join a CockroachDB node to a cluster.
I've created first cluster, then try to join 2nd node to the first node, but 2nd node created new cluster as follows.
Does anyone knows whats are wrong steps on the following my steps, any suggestions are wellcome.
I've started first node as follows:
cockroach start --insecure --advertise-host=163.172.156.111
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
*
CockroachDB node starting at 2019-05-11 01:11:15.45522036 +0000 UTC (took 2.5s)
build: CCL v19.1.0 # 2019/04/29 18:36:40 (go1.11.6)
webui: http://163.172.156.111:8080
sql: postgresql://root#163.172.156.111:26257?sslmode=disable
client flags: cockroach <client cmd> --host=163.172.156.111:26257 --insecure
logs: /home/ueda/cockroach-data/logs
temp dir: /home/ueda/cockroach-data/cockroach-temp449555924
external I/O path: /home/ueda/cockroach-data/extern
store[0]: path=/home/ueda/cockroach-data
status: initialized new cluster
clusterID: 3e797faa-59a1-4b0d-83b5-36143ddbdd69
nodeID: 1
Then, start secondary node to join to 163.172.156.111, but can't join:
cockroach start --insecure --advertise-addr=128.199.127.164 --join=163.172.156.111:26257
CockroachDB node starting at 2019-05-11 01:21:14.533097432 +0000 UTC (took 0.8s)
build: CCL v19.1.0 # 2019/04/29 18:36:40 (go1.11.6)
webui: http://128.199.127.164:8080
sql: postgresql://root#128.199.127.164:26257?sslmode=disable
client flags: cockroach <client cmd> --host=128.199.127.164:26257 --insecure
logs: /home/ueda/cockroach-data/logs
temp dir: /home/ueda/cockroach-data/cockroach-temp067740997
external I/O path: /home/ueda/cockroach-data/extern
store[0]: path=/home/ueda/cockroach-data
status: restarted pre-existing node
clusterID: a14e89a7-792d-44d3-89af-7037442eacbc
nodeID: 1
The cockroach.log of joining node shows some gosip error:
cat cockroach-data/logs/cockroach.log
I190511 01:21:13.762309 1 util/log/clog.go:1199 [config] file created at: 2019/05/11 01:21:13
I190511 01:21:13.762309 1 util/log/clog.go:1199 [config] running on machine: amfortas
I190511 01:21:13.762309 1 util/log/clog.go:1199 [config] binary: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
I190511 01:21:13.762309 1 util/log/clog.go:1199 [config] arguments: [cockroach start --insecure --advertise-addr=128.199.127.164 --join=163.172.156.111:26257]
I190511 01:21:13.762309 1 util/log/clog.go:1199 line format: [IWEF]yymmdd hh:mm:ss.uuuuuu goid file:line msg utf8=✓
I190511 01:21:13.762307 1 cli/start.go:1033 logging to directory /home/ueda/cockroach-data/logs
W190511 01:21:13.763373 1 cli/start.go:1068 RUNNING IN INSECURE MODE!
- Your cluster is open for any client that can access <all your IP addresses>.
- Any user, even root, can log in without providing a password.
- Any user, connecting as root, can read or write any data in your cluster.
- There is no network encryption nor authentication, and thus no confidentiality.
Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
I190511 01:21:13.763675 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 992 MiB, using system memory
W190511 01:21:13.763752 1 cli/start.go:944 Using the default setting for --cache (128 MiB).
A significantly larger value is usually needed for good performance.
If you have a dedicated server a reasonable setting is --cache=.25 (248 MiB).
I190511 01:21:13.764011 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 992 MiB, using system memory
W190511 01:21:13.764047 1 cli/start.go:957 Using the default setting for --max-sql-memory (128 MiB).
A significantly larger value is usually needed in production.
If you have a dedicated server a reasonable setting is --max-sql-memory=.25 (248 MiB).
I190511 01:21:13.764239 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 992 MiB, using system memory
I190511 01:21:13.764272 1 cli/start.go:1082 CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
I190511 01:21:13.866977 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 992 MiB, using system memory
I190511 01:21:13.867002 1 server/config.go:386 system total memory: 992 MiB
I190511 01:21:13.867063 1 server/config.go:388 server configuration:
max offset 500000000
cache size 128 MiB
SQL memory pool size 128 MiB
scan interval 10m0s
scan min idle time 10ms
scan max idle time 1s
event log enabled true
I190511 01:21:13.867098 1 cli/start.go:929 process identity: uid 1000 euid 1000 gid 1000 egid 1000
I190511 01:21:13.867115 1 cli/start.go:554 starting cockroach node
I190511 01:21:13.868242 21 storage/engine/rocksdb.go:613 opening rocksdb instance at "/home/ueda/cockroach-data/cockroach-temp067740997"
I190511 01:21:13.894320 21 server/server.go:876 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I190511 01:21:13.894813 21 storage/engine/rocksdb.go:613 opening rocksdb instance at "/home/ueda/cockroach-data"
W190511 01:21:13.896301 21 storage/engine/rocksdb.go:127 [rocksdb] [/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/version_set.cc:2566] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
W190511 01:21:13.905666 21 storage/engine/rocksdb.go:127 [rocksdb] [/go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/version_set.cc:2566] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
I190511 01:21:13.911380 21 server/config.go:494 [n?] 1 storage engine initialized
I190511 01:21:13.911417 21 server/config.go:497 [n?] RocksDB cache size: 128 MiB
I190511 01:21:13.911427 21 server/config.go:497 [n?] store 0: RocksDB, max size 0 B, max open file limit 10000
W190511 01:21:13.912459 21 gossip/gossip.go:1496 [n?] no incoming or outgoing connections
I190511 01:21:13.913206 21 server/server.go:926 [n?] Sleeping till wall time 1557537673913178595 to catches up to 1557537674394265598 to ensure monotonicity. Delta: 481.087003ms
I190511 01:21:14.251655 65 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:322 [n?] circuitbreaker: gossip [::]:26257->163.172.156.111:26257 tripped: initial connection heartbeat failed: rpc error: code = Unknown desc = client cluster ID "a14e89a7-792d-44d3-89af-7037442eacbc" doesn't match server cluster ID "3e797faa-59a1-4b0d-83b5-36143ddbdd69"
I190511 01:21:14.251695 65 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 [n?] circuitbreaker: gossip [::]:26257->163.172.156.111:26257 event: BreakerTripped
W190511 01:21:14.251763 65 gossip/client.go:122 [n?] failed to start gossip client to 163.172.156.111:26257: initial connection heartbeat failed: rpc error: code = Unknown desc = client cluster ID "a14e89a7-792d-44d3-89af-7037442eacbc" doesn't match server cluster ID "3e797faa-59a1-4b0d-83b5-36143ddbdd69"
I190511 01:21:14.395848 21 gossip/gossip.go:392 [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"128.199.127.164:26257" > attrs:<> locality:<> ServerVersion:<major_val:19 minor_val:1 patch:0 unstable:0 > build_tag:"v19.1.0" started_at:1557537674395557548
W190511 01:21:14.458176 21 storage/replica_range_lease.go:506 can't determine lease status due to node liveness error: node not in the liveness table
I190511 01:21:14.458465 21 server/node.go:461 [n1] initialized store [n1,s1]: disk (capacity=24 GiB, available=18 GiB, used=2.2 MiB, logicalBytes=41 MiB), ranges=20, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=6467.00 p90=26940.00 pMax=43017435.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I190511 01:21:14.458775 21 storage/stores.go:244 [n1] read 0 node addresses from persistent storage
I190511 01:21:14.459095 21 server/node.go:699 [n1] connecting to gossip network to verify cluster ID...
W190511 01:21:14.469842 96 storage/store.go:1525 [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I190511 01:21:14.474785 21 server/node.go:719 [n1] node connected via gossip and verified as part of cluster "a14e89a7-792d-44d3-89af-7037442eacbc"
I190511 01:21:14.475033 21 server/node.go:542 [n1] node=1: started with [<no-attributes>=/home/ueda/cockroach-data] engine(s) and attributes []
I190511 01:21:14.475393 21 server/status/recorder.go:610 [n1] available memory from cgroups (8.0 EiB) exceeds system memory 992 MiB, using system memory
I190511 01:21:14.475514 21 server/server.go:1582 [n1] starting http server at [::]:8080 (use: 128.199.127.164:8080)
I190511 01:21:14.475572 21 server/server.go:1584 [n1] starting grpc/postgres server at [::]:26257
I190511 01:21:14.475605 21 server/server.go:1585 [n1] advertising CockroachDB node at 128.199.127.164:26257
W190511 01:21:14.475655 21 jobs/registry.go:341 [n1] unable to get node liveness: node not in the liveness table
I190511 01:21:14.532949 21 server/server.go:1650 [n1] done ensuring all necessary migrations have run
I190511 01:21:14.533020 21 server/server.go:1653 [n1] serving sql connections
I190511 01:21:14.533209 21 cli/start.go:689 [config] clusterID: a14e89a7-792d-44d3-89af-7037442eacbc
I190511 01:21:14.533257 21 cli/start.go:697 node startup completed:
CockroachDB node starting at 2019-05-11 01:21:14.533097432 +0000 UTC (took 0.8s)
build: CCL v19.1.0 # 2019/04/29 18:36:40 (go1.11.6)
webui: http://128.199.127.164:8080
sql: postgresql://root#128.199.127.164:26257?sslmode=disable
client flags: cockroach <client cmd> --host=128.199.127.164:26257 --insecure
logs: /home/ueda/cockroach-data/logs
temp dir: /home/ueda/cockroach-data/cockroach-temp067740997
external I/O path: /home/ueda/cockroach-data/extern
store[0]: path=/home/ueda/cockroach-data
status: restarted pre-existing node
clusterID: a14e89a7-792d-44d3-89af-7037442eacbc
nodeID: 1
I190511 01:21:14.541205 146 server/server_update.go:67 [n1] no need to upgrade, cluster already at the newest version
I190511 01:21:14.555557 149 sql/event_log.go:135 [n1] Event: "node_restart", target: 1, info: {Descriptor:{NodeID:1 Address:128.199.127.164:26257 Attrs: Locality: ServerVersion:19.1 BuildTag:v19.1.0 StartedAt:1557537674395557548 LocalityAddress:[] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} ClusterID:a14e89a7-792d-44d3-89af-7037442eacbc StartedAt:1557537674395557548 LastUp:1557537671113461486}
I190511 01:21:14.916458 59 gossip/gossip.go:1510 [n1] node has connected to cluster via gossip
I190511 01:21:14.916660 59 storage/stores.go:263 [n1] wrote 0 node addresses to persistent storage
I190511 01:21:24.480247 116 storage/store.go:4220 [n1,s1] sstables (read amplification = 2):
0 [ 51K 1 ]: 51K
6 [ 1M 1 ]: 1M
I190511 01:21:24.480380 116 storage/store.go:4221 [n1,s1]
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 1/0 50.73 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0 1 0.006 0 0
L6 1/0 1.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Sum 2/0 1.31 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0 1 0.006 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0 1 0.006 0 0
Uptime(secs): 10.6 total, 10.6 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
estimated_pending_compaction_bytes: 0 B
I190511 01:21:24.481565 121 server/status/runtime.go:500 [n1] runtime stats: 170 MiB RSS, 114 goroutines, 0 B/0 B/0 B GO alloc/idle/total, 14 MiB/16 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (7x), 50 KiB/1.5 MiB (r/w)net
What is the possibly cause to block to join? Thank you for your suggestion!
It seems you had previously started the second node (the one running on 128.199.127.164) by itself, creating its own cluster. This can be seen in the error message: W190511 01:21:14.251763 65 gossip/client.go:122 [n?] failed to start gossip client to 163.172.156.111:26257: initial connection heartbeat failed: rpc error: code = Unknown desc = client cluster ID "a14e89a7-792d-44d3-89af-7037442eacbc" doesn't match server cluster ID "3e797faa-59a1-4b0d-83b5-36143ddbdd69" To be able to join the cluster, the data directory of the joining node must be empty. You can either delete cockroach-data or specify an alternate directory with --store=/path/to/data-dir
How to high concurency in Spring Boot
I have a requirement to create a product which should support 40 concurrent users per second (I am new to working on concurrency)
To achieve this, I tried to developed one hello world spring-boot project.
i.e.,
spring-boot (1.5.9)
jetty 9.4.15
rest controller which has get endpoint
code below:
#GetMapping
public String index() {
return "Greetings from Spring Boot!";
}
App running on machine Gen10 DL360
Then I tried to benchmark using apachebench
75 concurrent users:
ab -t 120 -n 1000000 -c 75 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 75
Time taken for tests: 37.184 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 26893.28 [#/sec] (mean)
Time per request: 2.789 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3755.61 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 23.5 0 3006
Processing: 0 2 7.8 1 404
Waiting: 0 2 7.8 1 404
Total: 0 3 24.9 2 3007
100 concurrent users:
ab -t 120 -n 1000000 -c 100 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 100
Time taken for tests: 36.708 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27241.77 [#/sec] (mean)
Time per request: 3.671 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3804.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 35.7 1 3007
Processing: 0 2 9.4 1 405
Waiting: 0 2 9.4 1 405
Total: 0 4 37.0 2 3009
500 concurrent users:
ab -t 120 -n 1000000 -c 500 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 500
Time taken for tests: 36.222 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27607.83 [#/sec] (mean)
Time per request: 18.111 [ms] (mean)
Time per request: 0.036 [ms] (mean, across all concurrent requests)
Transfer rate: 3855.39 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 14 126.2 1 7015
Processing: 0 4 22.3 1 811
Waiting: 0 3 22.3 1 810
Total: 0 18 129.2 2 7018
1000 concurrent users:
ab -t 120 -n 1000000 -c 1000 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 1000
Time taken for tests: 36.534 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27372.09 [#/sec] (mean)
Time per request: 36.534 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3822.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 30 190.8 1 7015
Processing: 0 6 31.4 2 1613
Waiting: 0 5 31.4 1 1613
Total: 0 36 195.5 2 7018
From above test run, I achieved ~27K per second with 75 users itself but it looks increasing the users also increasing the latency. Also, we can clearly note connect time is increasing.
I have requirement for my application to support 40k concurrent users (assume all are using own separate browsers) and request should be finished within 250 milliseconds.
Please help me on this
I am also not a grand wizard in the topic myself but here is some advice: there is a hard limit how many request can handle one instance so if you want to support a lot of user you need more instance if you work with multiple instance then you have to somehow distribute the requests among the instances. One popular solution is Netflix Eureka if you don't want to maintain additional resources and the product will run in cloud then use the provided load balancing services (e.g. LoadBalancer on AWS) also you can fine-tune your server's connection pool settings
dbms_alert Polling intervals
Explain, please, here this paragraph from the documentation
WAITANY procedure. If you use the WAITANY procedure, and if a signalling session does a signal but does not commit within one second of the signal, a polling loop is required so that this uncommitted alert does not camouflage other alerts. The polling loop begins at a one second interval and exponentially backs off to 30-second intervals.
I correctly understand that here it is said that when WAITANY is called on the server, the thread queries the presence of events at certain intervals? And if I called WAITANY with a sufficiently long timeout, when an event occurs, will I be notified only after the current request interval expires? Those. on the server is about like this code
function WaitAny(ATimeout) {
const intervals = [0, 1, ....., 30);
for (i = 0; i < intervals.length; i++) {
Sleep(min(intervals[i], ATimeout))
if (IsExistsEvents())
return 0;
ATimeout -= intervals[i];
if (ATimeout <= 0)
return 1;
}
maxInterval = intervals[intervals.length - 1];
while (ATimeout > 0) {
Sleep(min(maxInterval, ATimeout))
if (IsExistsEvents())
return 0;
ATimeout -= maxInterval;
}
return 1;
}
No, it will still be "immediate". When someones signals an alert, it will flick out a pipe message (a non-transactional blip) which the waiter will see. But of course, the waiter has not "officially" received the alert until the signaller commits. So the waiter gets the blip, then waits on a lock (alerts are implemented with Oracle's standard locking mechanism). Because we're expecting the commit to come quickly. If it doesn't, that's a problem because what if we get a signal from a different alert (because we are doing a WAITANY here). So when that commit does not come, we'll acquiesce and do the exponential backoff. But it is not "check for the alert at 1, 2, 4, etc seconds", it is "re-enter the wait for lock state every 1,2,4, ...seconds". So when that commit comes, we'll pretty much see it straightaway. For example, I traced a 2 minute wait on an alert from someone who signaled but did not commit declare n varchar2(100); m varchar2(100); s int; begin dbms_alert.WAITANY(n,m,s); end; call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 127.49 0 0 0 1 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 2 0.00 127.50 0 0 0 1 Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 107 Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ enq: UL - contention 9 32.01 127.47 log file sync 1 0.00 0.00 SQL*Net message to client 1 0.00 0.00 SQL*Net message from client 1 0.00 0.00 ******************************************************************************** You can see the 2 minute wait was almost all on lock waits.
Varnish 4: High cache-hit but high backend traffic
When running Varnish 4.0.3 and looking at varnishstat, it reports a cache-hit ratio of about 0.9 - 0.95 which I interpret as a 90-95% cache-hit ratio (incorrect?). I have a very low hit-pass count, so I cannot see that there is a lot of traffic that is not cacheable. The part that bothers me is that the backend servers are working quite a bit despite this. When I look at iftop I see that the server (which isn't running any other applications) is transmitting at about 19Mbit (includes answers to the clients and request headers to backend servers) and receiving about 14Mbit from the backend servers. I can add that I also have a small grace period in my cache and I see that about 1-2 requests per second get counted as hits but still make a request to the backend, but that is an insignificant number of requests. I must be missing something that should be obvious. But what? Edit: MAIN.uptime 70613 1.00 Child process uptime MAIN.sess_conn 43313 0.61 Sessions accepted MAIN.sess_drop 0 0.00 Sessions dropped MAIN.sess_fail 0 0.00 Session accept failures MAIN.sess_pipe_overflow 0 0.00 Session pipe overflow MAIN.client_req_400 0 0.00 Client requests received, subject to 400 errors MAIN.client_req_411 0 0.00 Client requests received, subject to 411 errors MAIN.client_req_413 0 0.00 Client requests received, subject to 413 errors MAIN.client_req_417 0 0.00 Client requests received, subject to 417 errors MAIN.client_req 6438096 91.17 Good client requests received MAIN.cache_hit 5231719 74.09 Cache hits MAIN.cache_hitpass 149819 2.12 Cache hits for pass MAIN.cache_miss 619678 8.78 Cache misses MAIN.backend_conn 19491 0.28 Backend conn. success MAIN.backend_unhealthy 0 0.00 Backend conn. not attempted MAIN.backend_busy 0 0.00 Backend conn. too many MAIN.backend_fail 0 0.00 Backend conn. failures MAIN.backend_reuse 1279156 18.12 Backend conn. reuses MAIN.backend_toolate 5046 0.07 Backend conn. was closed MAIN.backend_recycle 1284221 18.19 Backend conn. recycles MAIN.backend_retry 5 0.00 Backend conn. retry MAIN.fetch_head 7 0.00 Fetch no body (HEAD) MAIN.fetch_length 166348 2.36 Fetch with Length MAIN.fetch_chunked 861097 12.19 Fetch chunked MAIN.fetch_eof 0 0.00 Fetch EOF MAIN.fetch_bad 0 0.00 Fetch bad T-E MAIN.fetch_close 862 0.01 Fetch wanted close MAIN.fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed MAIN.fetch_zero 192834 2.73 Fetch zero len body MAIN.fetch_1xx 0 0.00 Fetch no body (1xx) MAIN.fetch_204 0 0.00 Fetch no body (204) MAIN.fetch_304 77295 1.09 Fetch no body (304) MAIN.fetch_failed 0 0.00 Fetch failed (all causes) MAIN.fetch_no_thread 0 0.00 Fetch failed (no thread) MAIN.pools 2 . Number of thread pools MAIN.threads 400 . Total number of threads MAIN.threads_limited 0 0.00 Threads hit max MAIN.threads_created 400 0.01 Threads created MAIN.threads_destroyed 0 0.00 Threads destroyed MAIN.threads_failed 0 0.00 Thread creation failed MAIN.thread_queue_len 0 . Length of session queue MAIN.busy_sleep 22 0.00 Number of requests sent to sleep on busy objhdr MAIN.busy_wakeup 22 0.00 Number of requests woken after sleep on busy objhdr MAIN.sess_queued 0 0.00 Sessions queued for thread MAIN.sess_dropped 0 0.00 Sessions dropped for thread MAIN.n_object 61732 . object structs made MAIN.n_vampireobject 0 . unresurrected objects MAIN.n_objectcore 61791 . objectcore structs made MAIN.n_objecthead 59996 . objecthead structs made MAIN.n_waitinglist 100 . waitinglist structs made MAIN.n_backend 8 . Number of backends MAIN.n_expired 412530 . Number of expired objects MAIN.n_lru_nuked 0 . Number of LRU nuked objects MAIN.n_lru_moved 1938371 . Number of LRU moved objects MAIN.losthdr 0 0.00 HTTP header overflows MAIN.s_sess 43313 0.61 Total sessions seen MAIN.s_req 6438096 91.17 Total requests seen MAIN.s_pipe 192 0.00 Total pipe sessions seen MAIN.s_pass 586507 8.31 Total pass-ed requests seen MAIN.s_fetch 1206185 17.08 Total backend fetches initiated MAIN.s_synth 0 0.00 Total synthethic responses made MAIN.s_req_hdrbytes 4609529442 65278.77 Request header bytes MAIN.s_req_bodybytes 615275 8.71 Request body bytes MAIN.s_resp_hdrbytes 2888897421 40911.69 Response header bytes MAIN.s_resp_bodybytes 95537815896 1352977.72 Response body bytes MAIN.s_pipe_hdrbytes 57835 0.82 Pipe request header bytes MAIN.s_pipe_in 45698 0.65 Piped bytes from client MAIN.s_pipe_out 1305816 18.49 Piped bytes to client MAIN.sess_closed 24747 0.35 Session Closed MAIN.sess_pipeline 0 0.00 Session Pipeline MAIN.sess_readahead 0 0.00 Session Read Ahead MAIN.sess_herd 813362 11.52 Session herd MAIN.shm_records 512140676 7252.78 SHM records MAIN.shm_writes 18119050 256.60 SHM writes MAIN.shm_flushes 788498 11.17 SHM flushes due to overflow MAIN.shm_cont 89983 1.27 SHM MTX contention MAIN.shm_cycles 277 0.00 SHM cycles through buffer MAIN.sms_nreq 0 0.00 SMS allocator requests MAIN.sms_nobj 0 . SMS outstanding allocations MAIN.sms_nbytes 0 . SMS outstanding bytes MAIN.sms_balloc 0 . SMS bytes allocated MAIN.sms_bfree 0 . SMS bytes freed MAIN.backend_req 1298448 18.39 Backend requests made MAIN.n_vcl 1 0.00 Number of loaded VCLs in total MAIN.n_vcl_avail 1 0.00 Number of VCLs available MAIN.n_vcl_discard 0 0.00 Number of discarded VCLs MAIN.bans 1 . Count of bans MAIN.bans_completed 1 . Number of bans marked 'completed' MAIN.bans_obj 0 . Number of bans using obj.* MAIN.bans_req 0 . Number of bans using req.* MAIN.bans_added 1 0.00 Bans added MAIN.bans_deleted 0 0.00 Bans deleted MAIN.bans_tested 0 0.00 Bans tested against objects (lookup) MAIN.bans_obj_killed 0 0.00 Objects killed by bans (lookup) MAIN.bans_lurker_tested 0 0.00 Bans tested against objects (lurker) MAIN.bans_tests_tested 0 0.00 Ban tests tested against objects (lookup) MAIN.bans_lurker_tests_tested 0 0.00 Ban tests tested against objects (lurker) MAIN.bans_lurker_obj_killed 0 0.00 Objects killed by bans (lurker) MAIN.bans_dups 0 0.00 Bans superseded by other bans MAIN.bans_lurker_contention 0 0.00 Lurker gave way for lookup MAIN.bans_persisted_bytes 13 . Bytes used by the persisted ban lists MAIN.bans_persisted_fragmentation 0 . Extra bytes in persisted ban lists due to fragmentation MAIN.n_purges 0 . Number of purge operations executed MAIN.n_obj_purged 0 . Number of purged objects MAIN.exp_mailed 949607 13.45 Number of objects mailed to expiry thread MAIN.exp_received 949607 13.45 Number of objects received by expiry thread MAIN.hcb_nolock 6001341 84.99 HCB Lookups without lock MAIN.hcb_lock 447556 6.34 HCB Lookups with lock MAIN.hcb_insert 447556 6.34 HCB Inserts MAIN.esi_errors 0 0.00 ESI parse errors (unlock) MAIN.esi_warnings 0 0.00 ESI parse warnings (unlock) MAIN.vmods 2 . Loaded VMODs MAIN.n_gzip 558747 7.91 Gzip operations MAIN.n_gunzip 22866 0.32 Gunzip operations MAIN.vsm_free 970832 . Free VSM space MAIN.vsm_used 83963776 . Used VSM space MAIN.vsm_cooling 0 . Cooling VSM space MAIN.vsm_overflow 0 . Overflow VSM space MAIN.vsm_overflowed 0 0.00 Overflowed VSM space MGT.uptime 70613 1.00 Management process uptime MGT.child_start 1 0.00 Child process started MGT.child_exit 0 0.00 Child process normal exit MGT.child_stop 0 0.00 Child process unexpected exit MGT.child_died 0 0.00 Child process died (signal) MGT.child_dump 0 0.00 Child process core dumped MGT.child_panic 0 0.00 Child process panic MEMPOOL.vbc.live 24 . In use MEMPOOL.vbc.pool 10 . In Pool MEMPOOL.vbc.sz_wanted 88 . Size requested MEMPOOL.vbc.sz_needed 120 . Size allocated MEMPOOL.vbc.allocs 19491 0.28 Allocations MEMPOOL.vbc.frees 19467 0.28 Frees MEMPOOL.vbc.recycle 19436 0.28 Recycled from pool MEMPOOL.vbc.timeout 11998 0.17 Timed out from pool MEMPOOL.vbc.toosmall 0 0.00 Too small to recycle MEMPOOL.vbc.surplus 0 0.00 Too many for pool MEMPOOL.vbc.randry 55 0.00 Pool ran dry MEMPOOL.busyobj.live 6 . In use MEMPOOL.busyobj.pool 9 . In Pool MEMPOOL.busyobj.sz_wanted 65536 . Size requested MEMPOOL.busyobj.sz_needed 65568 . Size allocated MEMPOOL.busyobj.allocs 1298643 18.39 Allocations MEMPOOL.busyobj.frees 1298637 18.39 Frees MEMPOOL.busyobj.recycle 1298436 18.39 Recycled from pool MEMPOOL.busyobj.timeout 41750 0.59 Timed out from pool MEMPOOL.busyobj.toosmall 0 0.00 Too small to recycle MEMPOOL.busyobj.surplus 0 0.00 Too many for pool MEMPOOL.busyobj.randry 207 0.00 Pool ran dry MEMPOOL.req0.live 4 . In use MEMPOOL.req0.pool 9 . In Pool MEMPOOL.req0.sz_wanted 65536 . Size requested MEMPOOL.req0.sz_needed 65568 . Size allocated MEMPOOL.req0.allocs 423162 5.99 Allocations MEMPOOL.req0.frees 423158 5.99 Frees MEMPOOL.req0.recycle 423143 5.99 Recycled from pool MEMPOOL.req0.timeout 36313 0.51 Timed out from pool MEMPOOL.req0.toosmall 0 0.00 Too small to recycle MEMPOOL.req0.surplus 0 0.00 Too many for pool MEMPOOL.req0.randry 19 0.00 Pool ran dry MEMPOOL.sess0.live 4 . In use MEMPOOL.sess0.pool 9 . In Pool MEMPOOL.sess0.sz_wanted 384 . Size requested MEMPOOL.sess0.sz_needed 416 . Size allocated MEMPOOL.sess0.allocs 21655 0.31 Allocations MEMPOOL.sess0.frees 21651 0.31 Frees MEMPOOL.sess0.recycle 21642 0.31 Recycled from pool MEMPOOL.sess0.timeout 10076 0.14 Timed out from pool MEMPOOL.sess0.toosmall 0 0.00 Too small to recycle MEMPOOL.sess0.surplus 0 0.00 Too many for pool MEMPOOL.sess0.randry 13 0.00 Pool ran dry MEMPOOL.req1.live 6 . In use MEMPOOL.req1.pool 9 . In Pool MEMPOOL.req1.sz_wanted 65536 . Size requested MEMPOOL.req1.sz_needed 65568 . Size allocated MEMPOOL.req1.allocs 418423 5.93 Allocations MEMPOOL.req1.frees 418417 5.93 Frees MEMPOOL.req1.recycle 418406 5.93 Recycled from pool MEMPOOL.req1.timeout 35924 0.51 Timed out from pool MEMPOOL.req1.toosmall 0 0.00 Too small to recycle MEMPOOL.req1.surplus 0 0.00 Too many for pool MEMPOOL.req1.randry 17 0.00 Pool ran dry MEMPOOL.sess1.live 6 . In use MEMPOOL.sess1.pool 10 . In Pool MEMPOOL.sess1.sz_wanted 384 . Size requested MEMPOOL.sess1.sz_needed 416 . Size allocated MEMPOOL.sess1.allocs 21659 0.31 Allocations MEMPOOL.sess1.frees 21653 0.31 Frees MEMPOOL.sess1.recycle 21645 0.31 Recycled from pool MEMPOOL.sess1.timeout 10040 0.14 Timed out from pool MEMPOOL.sess1.toosmall 0 0.00 Too small to recycle MEMPOOL.sess1.surplus 0 0.00 Too many for pool MEMPOOL.sess1.randry 14 0.00 Pool ran dry SMA.s0.c_req 1349644 19.11 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 22210720975 314541.53 Bytes allocated SMA.s0.c_freed 19148751042 271178.83 Bytes freed SMA.s0.g_alloc 105992 . Allocations outstanding SMA.s0.g_bytes 3061969933 . Bytes outstanding SMA.s0.g_space 2306739187 . Bytes available SMA.Transient.c_req 1343403 19.02 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 24207322266 342816.79 Bytes allocated SMA.Transient.c_freed 24201619786 342736.04 Bytes freed SMA.Transient.g_alloc 7499 . Allocations outstanding SMA.Transient.g_bytes 5702480 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.backend1([IP]).vcls 1 . VCL references VBE.backend1([IP]).happy 18446744073709551615 . Happy health probes VBE.backend1([IP]).bereq_hdrbytes 173320400 2454.51 Request header bytes VBE.backend1([IP]).bereq_bodybytes 238971 3.38 Request body bytes VBE.backend1([IP]).beresp_hdrbytes 124704577 1766.03 Response header bytes VBE.backend1([IP]).beresp_bodybytes 19209840560 272043.97 Response body bytes VBE.backend1([IP]).pipe_hdrbytes 14288 0.20 Pipe request header bytes VBE.backend1([IP]).pipe_out 8069 0.11 Piped bytes to backend VBE.backend1([IP]).pipe_in 335893 4.76 Piped bytes from backend VBE.backend2([IP]).vcls 1 . VCL references VBE.backend2([IP]).happy 18446744073709551615 . Happy health probes VBE.backend2([IP]).bereq_hdrbytes 173379423 2455.35 Request header bytes VBE.backend2([IP]).bereq_bodybytes 76962 1.09 Request body bytes VBE.backend2([IP]).beresp_hdrbytes 124652416 1765.29 Response header bytes VBE.backend2([IP]).beresp_bodybytes 18690565655 264690.15 Response body bytes VBE.backend2([IP]).pipe_hdrbytes 14629 0.21 Pipe request header bytes VBE.backend2([IP]).pipe_out 8558 0.12 Piped bytes to backend VBE.backend2([IP]).pipe_in 284168 4.02 Piped bytes from backend VBE.backend3([IP]).vcls 1 . VCL references VBE.backend3([IP]).happy 18446744073709551615 . Happy health probes VBE.backend3([IP]).bereq_hdrbytes 173448768 2456.33 Request header bytes VBE.backend3([IP]).bereq_bodybytes 75294 1.07 Request body bytes VBE.backend3([IP]).beresp_hdrbytes 124641060 1765.13 Response header bytes VBE.backend3([IP]).beresp_bodybytes 18738758550 265372.64 Response body bytes VBE.backend3([IP]).pipe_hdrbytes 17940 0.25 Pipe request header bytes VBE.backend3([IP]).pipe_out 16704 0.24 Piped bytes to backend VBE.backend3([IP]).pipe_in 301836 4.27 Piped bytes from backend VBE.backend4([IP]).vcls 1 . VCL references VBE.backend4([IP]).happy 18446744073709551615 . Happy health probes VBE.backend4([IP]).bereq_hdrbytes 173265357 2453.73 Request header bytes VBE.backend4([IP]).bereq_bodybytes 227023 3.22 Request body bytes VBE.backend4([IP]).beresp_hdrbytes 124724341 1766.31 Response header bytes VBE.backend4([IP]).beresp_bodybytes 18642562615 264010.35 Response body bytes VBE.backend4([IP]).pipe_hdrbytes 22537 0.32 Pipe request header bytes VBE.backend4([IP]).pipe_out 12367 0.18 Piped bytes to backend VBE.backend4([IP]).pipe_in 383919 5.44 Piped bytes from backend LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 5272603 74.67 Lock Operations LCK.hcb.locks 837080 11.85 Lock Operations LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 2604799 36.89 Lock Operations LCK.sess.creat 43306 0.61 Created locks LCK.sess.destroy 43304 0.61 Destroyed locks LCK.sess.locks 4765 0.07 Lock Operations LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 3421304 48.45 Lock Operations LCK.wq.locks 6449131 91.33 Lock Operations LCK.objhdr.creat 447638 6.34 Created locks LCK.objhdr.destroy 387644 5.49 Destroyed locks LCK.objhdr.locks 42615452 603.51 Lock Operations LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 4462106 63.19 Lock Operations LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 4259219 60.32 Lock Operations LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 23549 0.33 Lock Operations LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 1594008 22.57 Lock Operations LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 112922 1.60 Lock Operations LCK.backend.locks 2626872 37.20 Lock Operations LCK.vcapace.creat 1 0.00 Created locks *Had to remove some (uneventful parts) from the log due to message limit. In this I calculated "other non-cacheable requests" by client_req - (cache_hit + cache_hitpass + cache_miss) = 436880. That along with cache_hitpass and cache misses (or just client_req-cache-hit) which are not cached brings the total to 586699, or about 18.7%. If they are heavy requests I suppose that could account for the traffic. But I'm a bit unconvinced.
Have you tried to look at the access log of one of the servers? Maybe you have requests coming in not through Varnish? Also, I know that at least in Varnish 3, the cache hit rate is calculated from hits that can come from cache to begin with. For example, with a standard configuration, POST requests are not counted there. The access log may give you some insight...
Deadlock in Oracle 3
I have found following errors on Oracle.. I cant understand what is happening here. Cant understand which session is waiting for what.... Also is it about Raw level lock or something? Can anyone please help me? *** 2015-03-10 10:23:30.288 DEADLOCK DETECTED ( ORA-00060 ) [Transaction Deadlock] The following deadlock is not an ORACLE error. It is a deadlock due to user error in the design of an application or from issuing incorrect ad-hoc SQL. The following information may aid in determining the deadlock: Deadlock graph: ---------Blocker(s)-------- ---------Waiter(s)--------- Resource Name process session holds waits process session holds waits TX-001c0006-0000a440 94 2903 X 123 2284 X TX-001d0011-0000fac7 123 2284 X 94 2903 X session 2903: DID 0001-005E-00018DE5 session 2284: DID 0001-007B-0001863D session 2284: DID 0001-007B-0001863D session 2903: DID 0001-005E-00018DE5 Rows waited on: Session 2903: obj - rowid = 0001285E - AAASheAAIAABMBIAAV (dictionary objn - 75870, file - 8, block - 311368, slot - 21) Session 2284: obj - rowid = 0001285E - AAASheAAIAABL9/AAq (dictionary objn - 75870, file - 8, block - 311167, slot - 42) ----- Information for the OTHER waiting sessions ----- Session 2284: sid: 2284 ser: 207 audsid: 2893404 user: 87/ESS_ADMIN flags: 0x45 pid: 123 O/S info: user: oracle, term: UNKNOWN, ospid: 3209 image: oracle#pbifmsdb2 client details: O/S info: user: root, term: unknown, ospid: 1234 machine: pbifmsapp2 program: JDBC Thin Client application name: JDBC Thin Client, hash value=2546894660 current SQL: UPDATE SEC_LOGIN_AUDIT SET LOGOUT_TIME= SYSDATE,LOGOUT_TYPE=:1 ,LST_UPDT_DT= SYSDATE WHERE UPPER(USER_NAME) IN UPPER(:2 ) AND LOGOUT_TYPE IS NULL AND TRUNC(LOGIN_DATE_TIME) BETWEEN TRUNC(SYSDATE)-1 AND TRUNC(SYSDATE) ----- End of information for the OTHER waiting sessions ----- Information for THIS session: ----- Current SQL Statement for this session (sql_id=cm6x9vt91y3v3) ----- UPDATE SEC_LOGIN_AUDIT SET LOGOUT_TIME= SYSDATE,LOGOUT_TYPE=:1 ,LST_UPDT_DT= SYSDATE WHERE UPPER(USER_NAME) IN UPPER(:2 ) AND LOGOUT_TYPE IS NULL AND TRUNC(LOGIN_DATE_TIME) BETWEEN TRUNC(SYSDATE)-1 AND TRUNC(SYSDATE) =================================================== PROCESS STATE ------------- Process global information: process: 0xc000001154e37f70, call: 0xc00000113a5ee668, xact: 0xc0000011489d4808, curses: 0xc0000011598a5760, usrses: 0xc0000011598a5760 ---------------------------------------- SO: 0xc000001154e37f70, type: 2, owner: 0x0000000000000000, flag: INIT/-/-/0x00 if: 0x3 c: 0x3 proc=0xc000001154e37f70, name=process, file=ksu.h LINE:11459 ID:, pg=0 (process) Oracle pid:94, ser:1, calls cur/top: 0xc00000113a5ee668/0xc00000113a5ee668 flags : (0x0) - flags2: (0x0), flags3: (0x0) intr error: 0, call error: 0, sess error: 0, txn error 0 intr queue: empty ksudlp FALSE at location: 0 (post info) last post received: 134 0 2 last post received-location: ksl2.h LINE:2165 ID:kslpsr last process to post me: c00000115cda7358 10 6 last post sent: 0 0 26 last post sent-location: ksa2.h LINE:282 ID:ksasnd last process posted by me: c00000115cda7358 10 6 (latch info) wait_event=0 bits=0 Process Group: DEFAULT, pseudo proc: 0xc000001170f9b220 O/S info: user: oracle, term: UNKNOWN, ospid: 9438 OSD pid info: Unix process pid: 9438, image: oracle#pbifmsdb2 Dump of memory from 0xC000001154DE40C8 to 0xC000001154DE42D0 C000001154DE40C0 00000000 00000000 [........] C000001154DE40D0 00000000 00000000 00000000 00000000 [................] Repeat 31 times (FOB) flags=2066 fib=c000001173a56518 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/ess_data.dbf fno=8 lblksz=8192 fsiz=339200 (FOB) flags=2066 fib=c000001173a55900 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/iwas_data.dbf fno=6 lblksz=8192 fsiz=3678848 (FOB) flags=2050 fib=c000001173a58348 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/temp01.dbf fno=201 lblksz=8192 fsiz=4194176 (FOB) flags=2050 fib=c000001173a56518 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/ess_data.dbf fno=8 lblksz=8192 fsiz=339200 (FOB) flags=2050 fib=c000001173a546d0 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/undotbs01.dbf fno=3 lblksz=8192 fsiz=231040 (FOB) flags=2050 fib=c000001173a55f00 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/iwas_index.dbf fno=7 lblksz=8192 fsiz=470400 (FOB) flags=2050 fib=c000001173a55900 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/iwas_data.dbf fno=6 lblksz=8192 fsiz=3678848 (FOB) flags=2050 fib=c000001173a53ab8 incno=0 pending i/o cnt=0 fname=/iwdmsdata/PIWDMS/DATA/system01.dbf fno=1 lblksz=8192 fsiz=204800 ---------------------------------------- SO: 0xc0000011598a5760, type: 4, owner: 0xc000001154e37f70, flag: INIT/-/-/0x00 if: 0x3 c: 0x3 proc=0xc000001154e37f70, name=session, file=ksu.h LINE:11467 ID:, pg=0 (session) sid: 2903 ser: 7 trans: 0xc0000011489d4808, creator: 0xc000001154e37f70 flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/- flags2: (0x40008) -/- DID: , short-term DID: txn branch: 0xc00000114052f870 oct: 6, prv: 0, sql: 0xc00000116796d070, psql: 0xc0000011675cd1f0, user: 87/ESS_ADMIN ksuxds FALSE at location: 0 service name: SYS$USERS client details: O/S info: user: root, term: unknown, ospid: 1234 machine: pbifmsapp2 program: JDBC Thin Client application name: JDBC Thin Client, hash value=2546894660 Current Wait Stack: 0: waiting for 'enq: TX - row lock contention' name|mode=0x54580006, usn<<16 | slot=0x1d0011, sequence=0xfac7 wait_id=471131 seq_num=12392 snap_id=1 wait times: snap=3.004466 sec, exc=3.004466 sec, total=3.004466 sec wait times: max=1 min 0 sec, heur=3.004466 sec wait counts: calls=1 os=1 in_wait=1 iflags=0x15a0 There is at least one session blocking this session. Dumping 1 direct blocker(s): inst: 1, sid: 2284, ser: 207 Dumping final blocker: inst: 1, sid: 2284, ser: 207 Wait State: fixed_waits=0 flags=0x22 boundary=0x0000000000000000/-1 Session Wait History: elapsed time of 1.625218 sec since current wait 0: waited for 'SQL*Net message from client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471130 seq_num=12391 snap_id=1 wait times: snap=0.000630 sec, exc=0.000630 sec, total=0.000630 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000003 sec of elapsed time 1: waited for 'SQL*Net message to client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471129 seq_num=12390 snap_id=1 wait times: snap=0.000000 sec, exc=0.000000 sec, total=0.000000 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000039 sec of elapsed time 2: waited for 'SQL*Net message from client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471128 seq_num=12389 snap_id=1 wait times: snap=0.002370 sec, exc=0.002370 sec, total=0.002370 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000001 sec of elapsed time 3: waited for 'SQL*Net message to client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471127 seq_num=12388 snap_id=1 wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000069 sec of elapsed time 4: waited for 'SQL*Net message from client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471126 seq_num=12387 snap_id=1 wait times: snap=0.000659 sec, exc=0.000659 sec, total=0.000659 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000004 sec of elapsed time 5: waited for 'SQL*Net message to client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471125 seq_num=12386 snap_id=1 wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.001259 sec of elapsed time 6: waited for 'SQL*Net message from client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471124 seq_num=12385 snap_id=1 wait times: snap=0.000447 sec, exc=0.000447 sec, total=0.000447 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000036 sec of elapsed time 7: waited for 'SQL*Net message to client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471123 seq_num=12384 snap_id=1 wait times: snap=0.000006 sec, exc=0.000006 sec, total=0.000006 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 1.741446 sec of elapsed time 8: waited for 'SQL*Net message from client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471122 seq_num=12383 snap_id=1 wait times: snap=0.001060 sec, exc=0.001060 sec, total=0.001060 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000002 sec of elapsed time 9: waited for 'SQL*Net message to client' driver id=0x74637000, #bytes=0x1, =0x0 wait_id=471121 seq_num=12382 snap_id=1 wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000038 sec of elapsed time Sampled Session History of session 2903 serial 7 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [3 samples, 10:23:27 - 10:23:29] waited for 'enq: TX - row lock contention', seq_num: 12392 p1: 'name|mode'=0x54580006 p2: 'usn<<16 | slot'=0x1d0011 p3: 'sequence'=0xfac7 time_waited: >= 2 sec (still in wait) [3 samples, 10:23:24 - 10:23:26] not in wait at each sample [13 samples, 10:23:11 - 10:23:23] idle wait at each sample [1 sample, 10:23:10] not in wait at each sample [8 samples, 10:23:02 - 10:23:09] idle wait at each sample [2 samples, 10:23:00 - 10:23:01] not in wait at each sample [5 samples, 10:22:54 - 10:22:59] idle wait at each sample [2 samples, 10:22:52 - 10:22:53] not in wait at each sample [1 sample, 10:22:51] idle wait at each sample [1 sample, 10:22:50] not in wait at each sample [2 samples, 10:22:48 - 10:22:49] idle wait at each sample [2 samples, 10:22:46 - 10:22:47] not in wait at each sample [9 samples, 10:22:37 - 10:22:45] idle wait at each sample [2 samples, 10:22:35 - 10:22:36] not in wait at each sample [1 sample, 10:22:34] idle wait at each sample [2 samples, 10:22:32 - 10:22:33] not in wait at each sample [1 sample, 10:22:31] idle wait at each sample [1 sample, 10:22:30] not in wait at each sample [1 sample, 10:22:29] idle wait at each sample [2 samples, 10:22:27 - 10:22:28] not in wait at each sample [1 sample, 10:22:26] idle wait at each sample [1 sample, 10:22:25] waited for 'db file scattered read', seq_num: 9258 p1: 'file#'=0x6 p2: 'block#'=0x1fecd0 p3: 'blocks'=0x4 time_waited: 0.007933 sec (sample interval: 0 sec) [5 samples, 10:22:20 - 10:22:24] idle wait at each sample [4 samples, 10:22:16 - 10:22:19] not in wait at each sample [3 samples, 10:22:13 - 10:22:15] idle wait at each sample [2 samples, 10:22:11 - 10:22:12] not in wait at each sample [7 samples, 10:22:04 - 10:22:10] idle wait at each sample [6 samples, 10:21:58 - 10:22:03] not in wait at each sample [30 samples, 10:21:30 - 10:21:57] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'enq: TX - row lock contention' [3 samples, 10:23:27 - 10:23:29] time_waited: >= 2 sec (still in wait) Can anyone please help me?
You have had two session that each waited for a resource that was held by the other session. For example: Session 1 update table_ .... where x = 100; Session 1 has now locked the record x=100. No other session can modify this record until Session1 either commits or rolls back (that is: ends the transaction). Session 2 update table_ .... where x = 200; Session 2 has now locked the record x=200. Then, still session 2: update table_ .... where x=100; Since Session 1 has locked that record, this statement will wait until it can do the modification (that is, until Session 1 end its transaction). Session 1 update table_ .... where x=200; Now, this is the deadlock. Since x=200 is locked by Session 2, Session 1 cannot proceed until the transaction of Session 2 ends. But Session 2 cannot proceed because it is also waiting for Session 1. Hence the Deadlock Error ORA-00060.