I'm doing multiple scans in HBase with a java/spring boot API. However I have a doubt about performances of the way hbase-client manages connections (I use hbase-client 1.3.0).
Here is the code I use for scan :
#Repository
public abstract class AbstractHBaseRepository implements InitializingBean {
#Autowired
private HbaseTemplate hbaseTemplate;
private Connection connection;
#Override
public void afterPropertiesSet() throws Exception {
connection = ConnectionFactory.createConnection(hbaseTemplate.getConfiguration(), (ExecutorService) null, (User) null);
}
protected ResultScanner scan(String tableName, FilterList filters) {
Scan scan = new Scan();
// If list is not empty, apply filters
if (null != filters && !filters.getFilters().isEmpty()) {
scan.setFilter(filters);
}
try {
return connection.getTable(TableName.valueOf(tableName)).getScanner(scan);
} catch (IOException e) {
throw new RuntimeException("Unexpected HBase scan error", e);
}
}
}
What I see when debugging, and performing multiple API requests is :
HBase connection (Connection object) is kept alive and shared by all
threads => Good
For each scan performed, a new zookeeper session is created => Not good ?
For the second point, I have following logs each time :
INFO o.a.h.h.zookeeper.RecoverableZooKeeper : Process identifier=hconnection-0x2634f5f3 connecting to ZooKeeper ensemble=zoo.node:2181
INFO org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=zoo.node:2181,noeyyet7.noe.edf.fr:2181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher#3467699e
INFO org.apache.zookeeper.ClientCnxn : Opening socket connection to server zoo.node/localhost:2181
INFO org.apache.zookeeper.ClientCnxn : Socket connection established to zoo.node/localhost:2181, initiating session
INFO org.apache.zookeeper.ClientCnxn : Session establishment complete on server zoo.node/localhost:2181, sessionid = 0x17f9803c1b2212c, negotiated timeout = 120000
INFO nectionManager$HConnectionImplementation : Closing zookeeper sessionid=0x17f9803c1b2212c
INFO org.apache.zookeeper.ZooKeeper : Session: 0x17f9803c1b2212c closed
INFO org.apache.zookeeper.ClientCnxn : EventThread shut down for session: 0x17f9803c1b2212c
Is it normal that a new zookeeper session is created for each HBase query ? Is there a way to keep it alive ?
Related
Please can someone why the Patho MQTT v3 Message Driven Adapter is continuously loosing connection when trying to subscribe to an HiveMQ 4.9 topic
Versions :
org.springframework.integration:spring-integration-mqtt:5.5.2
org.springframework.boot:spring-boot-starter:2.5.3
org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.2.5
Here is the config:
#Bean
public MqttPahoClientFactory mqttClientFactory() {
DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory();
MqttConnectOptions options = new MqttConnectOptions();
options.setServerURIs(new String[] { "tcp://localhost:1883" });
factory.setConnectionOptions(options);
return factory;
}
#Bean
public MqttPahoMessageDrivenChannelAdapter inboundAdapter(MqttPahoClientFactory clientFactory) {
return new MqttPahoMessageDrivenChannelAdapter("MyApp", clientFactory, "MyTopic");
}
Each time the ChannelAdapter issue this error :
2022-10-26 15:22:24.580 INFO 28190 --- [nio-8091-exec-1] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.endpoint.EventDrivenConsumer#9490cb2
2022-10-26 15:22:24.917 ERROR 28190 --- [TT Rec: MyoApp] .m.i.MqttPahoMessageDrivenChannelAdapter : Lost connection: Connection lost; retrying...
2022-10-26 15:22:35.241 ERROR 28190 --- [TT Rec: MyoApp] o.s.i.m.outbound.MqttPahoMessageHandler : Lost connection; will attempt reconnect on next request
Please does somebody know the real impact of this situation and what I can do to stop these erros ?
Thanks
I have a Spring Boot application, implementing Websocket as well as Redis stream.
The flow is, Subscriber (who subscribes Redis stream) upon receiving message will then send that information into Websocket (using STOMP protocol and AmazonMQ - ActiveMQ as an external message broker).
Example of 1 consumer group
public Subscription fExecutionResponse(RedisTemplate<String, Object> redisTemplate) {
try {
String groupName = FStreamName.fExecutionConsumerGroup;
String streamKey = FStreamName.fExecutionStream;
createConsumerGroup(streamKey, groupName, redisTemplate);
val listenerContainer = listenerContainer(redisTemplate, FTradeDataRedisEvent.class);
val subscription = listenerContainer.receiveAutoAck(
Consumer.from(groupName, FStreamName.fExecutionConsumer),
StreamOffset.create(streamKey, ReadOffset.lastConsumed()),
message -> {
log.info("[Subscription F_EXECUTION]: {}", message.getValue());
FTradeDataRedisEvent mTradeEvent = message.getValue();
try {
if (ExternalConfiguration.futureTradeDataSource.equals(ExecutionSource.ALL) || ExternalConfiguration.futureTradeDataSource.equals(mTradeEvent.getSource())) {
futureProductService.updateProductByDummy(mTradeEvent);
futureExecutionToTickHistoricalService.transform(mTradeEvent);
}
} catch (Exception ex) {
log.error("[fTradeEventResponse] error: {}", ex.getMessage());
}
redisTemplate.opsForStream().acknowledge(streamKey, groupName, message.getId());
redisTemplate.opsForStream().delete(streamKey, message.getId());
});
listenerContainer.start();
log.info("A stream key `{}` had been successfully set up in the group `{}`", streamKey, groupName);
return subscription;
} catch (Exception ex) {
log.error(ex.getMessage(), ex);
}
return null;
}
futureExecutionToTickHistoricalService.transform will send the data to web socket using SimpMessageSendingOperations
public void transform(FTradeDataRedisEvent tradeData) {
if (lastUpdateTickHistoricalDataTime == 0L) {
Calendar calendar = Calendar.getInstance();
this.lastUpdateTickHistoricalDataTime = calendar.getTimeInMillis();
}
List<FTickHistorical> res = separateFTickHistorical(tradeData);
res.forEach(tickHistorical -> {
List<KlineResponse> klineResponses = new ArrayList<>();
klineResponses.add(new KlineResponse(tickHistorical));
messageTemplate.convertAndSend(
PUBLIC_TOPIC_PREDIX + Constants.F_PRODUCTS_DESTINATION + "/" + tickHistorical.getProductId() + "/klines" + "_" + tickHistorical.getResolution().getCode(),
new HistoryResponse(klineResponses)
);
});
}
There are two problems with this setup, I have resolved one of them.
The Redis stream subscriber is started up before the connection to the external message broker is ready. Solved (listen to BrokerAvailabilityEvent and only then start Redis subscriptions)
When redeploy or shutdown application on IDE (like Intellij). The connection to the broker is again destroyed first (before the Redis stream subscribers), at the same time, there are still some data sending to the socket. This cause error: Message broker not active
I don't know how to configure the Spring boot application, so that when the application is stopped, it first stops consuming messages from Redis stream, process all of the pending messages, then close the broker connection.
This is the error log when the application is destroyed.
14.756 INFO 41184 --- [extShutdownHook] c.i.c.foapi.SpotStreamSubscriber : Broker not available
2022-10-09 14:14:14.757 INFO 41184 --- [extShutdownHook] c.i.c.f.config.RedisSubConfiguration : Broker not available
2022-10-09 14:14:14.781 ERROR 41184 --- [cTaskExecutor-1] c.i.c.foapi.consumer.SpotStreamConsumer : [executionResponse] error: Message broker not active. Consider subscribing to receive BrokerAvailabilityEvent's from an ApplicationListener Spring bean.; value of message: ExecutionRedisEvent(id=426665086, eventTime=2022-10-09T14:13:45.056809, productId=2, price=277.16, quantity=0.08, buyerId=2, sellerId=3, createdDate=2022-10-09T14:13:45.056844, takerSide=SELL, orderBuyId=815776680, orderSellId=815776680, symbol=bnbusdt, source=BINANCE)
2022-10-09 14:14:14.785 ERROR 41184 --- [cTaskExecutor-1] c.i.c.foapi.consumer.SpotStreamConsumer : [dummyOrderBookResponse] error: Message broker not active. Consider subscribing to receive BrokerAvailabilityEvent's from an ApplicationListener Spring bean.; value of message: DummyOrderBookEvent(productId=1, bids=[DummyOrderBook(price=1941.6, qty=0), DummyOrderBook(price=18827.3, qty=0.013), DummyOrderBook(price=18938.8, qty=5.004), DummyOrderBook(price=18940.3, qty=22.196), DummyOrderBook(price=18982.5, qty=20.99), DummyOrderBook(price=19027.2, qty=0.33), DummyOrderBook(price=19045.8, qty=8.432)
This is the code of SpotStreamSubscriber
#Component
#RequiredArgsConstructor
#Slf4j
public class SpotStreamSubscriber implements ApplicationListener<BrokerAvailabilityEvent> {
private final SpotStreamConsumer spotStreamConsumer;
#Override
public void onApplicationEvent(BrokerAvailabilityEvent event) {
if (event.isBrokerAvailable()) {
log.info("Broker ready");
spotStreamConsumer.subscribe();
} else {
log.info("Broker not available");
}
}
}
As you can see, the message broker is destroyed before the pending Redis messages have a chance to be processed.
Current architecture,
We use external message broker so that we can scale horizontally the api.
I have a Java JDBC application, after 100 queries of Select Jetty server crashed and return below error:
ERROR com.zaxxer.hikari.pool.HikariPool - dev-server - Exception during pool initialization.
java.sql.SQLNonTransientConnectionException: Too many connections
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:526)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:513)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:115)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:1606)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:633)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:347)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:219)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:443)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:514)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:111)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at com.jrg.platform.commons.hikari.HikariDataSourceSupport.getDataSource(HikariDataSourceSupport.java:70)
at com.jrg.platform.commons.hikari.HikariDataSourceSupport.getConnection(HikariDataSourceSupport.java:82)
at com.jrg.platform.services.internal.pcms.data.campaign.CampaignAlertDaoImpl.getCampaignAlerts(CampaignAlertDaoImpl.java:40)
at com.jrg.platform.services.internal.pcms.service.CampaignAlertServiceImpl.campaignAlerts(CampaignAlertServiceImpl.java:43)
at com.jrg.platform.services.internal.pcms.resource.CampaignAlertResource.getCampaignAlerts(CampaignAlertResource.java:52)
at com.jrg.platform.services.internal.pcms.resource.CampaignAlertResource_$$_jvstf5a_6._d5getCampaignAlerts(CampaignAlertResource_$$_jvstf5a_6.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Unable to find the solution. I am just getting the connection from HikariDataSourceSupport and performing the query.
the sample code of getting connection is given below:
default HikariDataSource getDataSource(E e, boolean readOnly) throws IOException {
String key = this.getKey(e, readOnly);
Map<String, HikariDataSource> sources = this.getDataSources();
if(!sources.containsKey(key)) {
synchronized(this.getMonitor()) {
if(!sources.containsKey(key)) {
if(logger.isDebugEnabled()) {
logger.debug("creating new DataSource for {}", key);
}
Config config = this.getConfig(e, readOnly);
if(!config.hasPathOrNull("jdbcUrl")) {
throw new EnvironmentNotConfigured(e, readOnly);
}
sources.put(key, new HikariDataSource(new HikariConfig(ConfigUtils.asProperties(config))));
}
}
}
return (HikariDataSource)sources.get(key);
}
default Connection getConnection(E env, boolean readOnly) throws SQLException, IOException {
return this.getDataSource(env, readOnly).getConnection();
}
There was problem in configuration HikariCP. When request generated and try to get connection, the create connection method was creating the new datasource because the last state was destroy due to view scope of the bean.
now i have converted that class into to singalton and created the implementation of that interface to inject in the code. it is saving the state of previous data source connection. Now it is working perfect with minimum idle connections.
the code can be seen here:
APP.java
bind(HikariLucktasticDataSourceSupportImpl.class)
.to(HikariDataSourceSupport.class)
.named("hdsSupport")
.in(Singleton.class);
in DAO Layer:
#Inject
#Named("hdsSupport")
private HikariDataSourceSupport hdsSupport;
I'm currently trying to use kafka and spring-kafka in order to consumer messages.
But I have trouble executing several consumers for the same topic and have several questions:
1 - My consumers tends to disconnect after some time and have trouble reconnecting
The following WARN is raised regularly on my consumers:
2017-09-06 15:32:35.054 INFO 5203 --- [nListener-0-C-1] f.b.poc.crawler.kafka.KafkaListener : Consuming {"some-stuff": "yes"} from topic [job15]
2017-09-06 15:32:35.054 INFO 5203 --- [nListener-0-C-1] f.b.p.c.w.services.impl.CrawlingService : Start of crawling
2017-09-06 15:32:35.054 INFO 5203 --- [nListener-0-C-1] f.b.p.c.w.services.impl.CrawlingService : Url has already been treated ==> skipping
2017-09-06 15:32:35.054 WARN 5203 --- [nListener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Auto-commit of offsets {job15-3=OffsetAndMetadata{offset=11547, metadata=''}, job15-2=OffsetAndMetadata{offset=15550, metadata=''}} failed for group group-3: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
2017-09-06 15:32:35.054 INFO 5203 --- [nListener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [job15-3, job15-2] for group group-3
2017-09-06 15:32:35.054 INFO 5203 --- [nListener-0-C-1] s.k.l.ConcurrentMessageListenerContainer : partitions revoked:[job15-3, job15-2]
2017-09-06 15:32:35.054 INFO 5203 --- [nListener-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group group-3
This cause the consumer to stop and wait for several seconds.
As mentionned in the message, I increased the consumers session.timeout.ms to something like 30000. I still get the message.
As you can see in the provided logs the disconnection occurs right after a record has finished its process.
So ... a lot before 30s of innactivity.
2- Two consumers application receives the same message REALLY often
While looking at my consumers' logs I saw that they tend to treat the same message. I understood Kafka is at-least-once but I never thought I would encounter a lot of duplication.
Hopefully I use redis but I probably have missunderstood some tuning / properties I need to do.
THE CODE
Note: I'm using ConcurrentMessageListenerContainer with auto-commit=true but run with 1 Thread. I just start several instances of the same application because the consumer uses services that aren't thread-safe.
KafkaContext.java
#Slf4j
#Configuration
#EnableConfigurationProperties(value = KafkaConfig.class)
class KafkaContext {
#Bean(destroyMethod = "stop")
public ConcurrentMessageListenerContainer kafkaInListener(IKafkaListener listener, KafkaConfig config) {
final ContainerProperties containerProperties =
new ContainerProperties(config.getIn().getTopic());
containerProperties.setMessageListener(listener);
final DefaultKafkaConsumerFactory<Integer, String> defaultKafkaConsumerFactory =
new DefaultKafkaConsumerFactory<>(consumerConfigs(config));
final ConcurrentMessageListenerContainer messageListenerContainer =
new ConcurrentMessageListenerContainer<>(defaultKafkaConsumerFactory, containerProperties);
messageListenerContainer.setConcurrency(config.getConcurrency());
messageListenerContainer.setAutoStartup(false);
return messageListenerContainer;
}
private Map<String, Object> consumerConfigs(KafkaConfig config) {
final String kafkaHost = config.getHost() + ":" + config.getPort();
log.info("Crawler_Worker connecting to kafka at {} with consumerGroup {}", kafkaHost, config.getIn().getGroupId());
final Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaHost);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
props.put(ConsumerConfig.GROUP_ID_CONFIG, config.getIn().getGroupId());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JacksonNextSerializer.class);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000);
return props;
}
}
Listener
#Slf4j
#Component
class KafkaListener implements IKafkaListener {
private final ICrawlingService crawlingService;
#Autowired
public KafkaListener(ICrawlingService crawlingService) {
this.crawlingService = crawlingService;
}
#Override
public void onMessage(ConsumerRecord<Integer, Next> consumerRecord) {
log.info("Consuming {} from topic [{}]", JSONObject.wrap(consumerRecord.value()), consumerRecord.topic());
consumerService.apply(consumerRecord.value());
}
}
The main issue here is that your consumer group is continuously being rebalanced. You are right about increasing session.timeout.ms, but I don't see this config applied in your configuration. Try removing:
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000);
and setting:
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 10);
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
You can increase MAX_POLL_RECORDS_CONFIG to get a better performance on communication with brokers. But if you process messages in one thread only it is safer to keep this value low.
I am using wildfly jms queue... I'm using wildfly 9.0.2.Final
I make the producer like this :
#Inject
private JMSContext jmsContext;
private JMSProducer jmsProducer;
#Resource(mappedName = "java:/jboss/exported/jms/queue/TosDownloadReport")
private Queue queueDownloadReport;
public void downloadReport(String adminId, DownloadReportFilter filter){
try {
jmsProducer = jmsContext.createProducer();
String requestParam = Json.getInstance().getObjectMapper().writeValueAsString(filter);
LOG.info("requestParam {}", requestParam);
String id = UUID.randomUUID().toString().replace("-", "");
RoutingRequest request = new RoutingRequest();
request.putProperty("id", id);
request.putProperty("adminId", adminId);
request.putProperty("parameterRequest", requestParam);
QueueMsgDownloadReport message = new QueueMsgDownloadReport();
message.setId(id);
message.setAdminId(adminId);
String jsonMsg = Json.getInstance().getObjectMapper().writeValueAsString(message);
gatewayService.send(RESOURCE, METHOD, request);
jmsProducer.send(queueDownloadReport, jsonMsg);
} catch (Exception e) {
LOG.error(e.getMessage(),e);
}
}
But sometimes i get exception like this and i must restart wildfly
2017-05-05 11:08:20,004 ERROR [com.daksa.tos.infrastructure.api.TosTimer] (EJB default - 7) Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA: javax.jms.JMSRuntimeException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA
Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
From what i read, i don't need to call jmsContext.close() if i'm using inject right?
Please tell me what i do wrong...Thx