Postgresql query is performing really bad - performance

Question

I have a huge issue with query performance at my postgresql database.
The version of postgresql is: "PostgreSQL 8.4.3 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20070115 (prerelease) (SUSE Linux), 64-bit"
I have config file set as follows:
shared_buffers = 8GB
effective_cache_size = 24GB
work_mem = 419430kB
maintenance_work_mem = 2GB
checkpoint_segments = 128
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500
constraint_exclusion = on
The query which I need to execute is:
SELECT events.evt_id, events.evt_time, events.device_evt_time , to_ip_char(events.sip) , evt_agent.port , events.rv40 , events.evt , events.msg , events.sun , events.rv35 , events.dun , events.rv45 , events.fn , events.dp , events.trgt_trust_name , events.trgt_trust_domain , events.rv36 , events.rv43 , events.cv21 , events.cv40 , events.cv41 , events.cv42 , events.cv43 , events.cv44 , events.cv50 , events.cv51 , events.cv52 , events.cv53 , events.cv54 , events.cv55 , events.cv56 , events.cv35 , events.cv60 , events.cv61 , events.cv62
FROM events, evt_agent
WHERE
events.agent_id = evt_agent.agent_id AND (evt_agent.port::text = ANY (ARRAY['x'::character varying, 'y'::character varying, 'z'::character varying]::text[]))
AND events.evt::text <> 'Internal Message'::text
AND event_time > '2015-12-31 13:23:55.767+00'::timestamptz limit 10;
ORDER BY events.evt_time;
Table events is a partitioned table, one partition for each day.
Each partition has two constrains:
Name events_p_YYYYMMDDHHMISS_events_p_max_pk
Columns evt_time, evt_id
Name events_p_YYYYMMDDHHMISS_dc
Definition evt_time > '2015-12-04 13:24:25.267973+00'::timestamp with time zone AND evt_time <= '2015-12-05 13:24:25.267973+00'::timestamp with time zone
and seven indexes:
Name events_p_YYYYMMDDHHMISS_events_p_max_identity_ix1
Columns evt_time, init_usr_identity_guid, rid02
Operator classes timestamptz_ops, uuid_ops, int8_ops
Name events_p_YYYYMMDDHHMISS_events_p_max_identity_ix2
Columns evt_time, trgt_usr_identity_guid, rid02
Operator classes timestamptz_ops, uuid_ops, int8_ops
Name events_p_YYYYMMDDHHMISS_events_p_max_ix1
Columns evt_time, sev, agent_id
Operator classes timestamptz_ops, int4_ops, int8_ops
Name events_p_YYYYMMDDHHMISS_events_p_max_ix2
Columns evt_time, dip, sev
Operator classes timestamptz_ops, int4_ops, int4_ops
Name events_p_YYYYMMDDHHMISS_events_p_max_ix3
Columns evt_time, res, sev
Operator classes timestamptz_ops, text_ops, int4_ops
Name events_p_YYYYMMDDHHMISS_events_p_max_ix4
Columns evt_time, sip, sev
Operator classes timestamptz_ops, int4_ops, int4_ops
Name events_p_YYYYMMDDHHMISS_events_p_max_ix5
Columns evt_time, txnmy_id, agent_id
Operator classes timestamptz_ops, int8_ops, int8_ops
Table evt_agent is a dictionary table with about 20-30 rows only.
constrains:
Name evt_agent_pk
Columns agent_id
indexes:
Name evt_agent_ak1
Columns agent, port, rn, pn, sn, st, device_ctgry, src_id, cust_id
Operator classes text_ops, text_ops, text_ops, text_ops, text_ops, text_ops, text_ops, uuid_ops, int8_ops
Name evt_agent_ix1
Columns device_ctgry, agent_id
Operator classes text_ops, int8_ops
When I execute the query I got this explain plan:
http://explain.depesz.com/s/3xcu
Name evt_agent_ix2
Columns st, agent_id
Operator classes text_ops, int8_ops
Name test_ev_ag_indx1
Columns agent_id
Operator classes int8_ops
I thought there is a problem with statistics so I "vaccuum analyze" all related tables, but no improvement in the query performance, explain plan.
I tried with "inner query" trick, explain plan is here:
http://explain.depesz.com/s/FAkH
From what I can understand it got kind of worse.
Do you have nay idea how to get better execution plan for this query?
Now it takes about 42 minutes to get any results from the query.
Thanks in advance!

Answer 1

We will have to try and eliminate the hash join. A lateral join might just be the answer:
SELECT events.evt_id, events.evt_time, events.device_evt_time , to_ip_char(events.sip) , evt_agent.port , events.rv40 , events.evt , events.msg , events.sun , events.rv35 , events.dun , events.rv45 , events.fn , events.dp , events.trgt_trust_name , events.trgt_trust_domain , events.rv36 , events.rv43 , events.cv21 , events.cv40 , events.cv41 , events.cv42 , events.cv43 , events.cv44 , events.cv50 , events.cv51 , events.cv52 , events.cv53 , events.cv54 , events.cv55 , events.cv56 , events.cv35 , events.cv60 , events.cv61 , events.cv62
FROM evt_agent,
LATERAL (SELECT *
FROM events AS e
WHERE
e.agent_id = evt_agent.agent_id
AND e.evt::text <> 'Internal Message'::text
AND e.event_time > '2015-12-31 13:23:55.767+00'::timestamptz
LIMIT 10000) AS events
WHERE (evt_agent.port::text = ANY (ARRAY['x'::character varying, 'y'::character varying, 'z'::character varying]::text[]))
ORDER BY events.evt_time
LIMIT 10000;
Please try it with and without the inner limit and post the explain plans for both.