What is the appropriate way to handle live events (i.e. service/component should keep on listening to events and save it to offchain db (h2/postgres))
How to close event subscription gracefully?
Implementation tried so far:
#Component
public class ERC20Listener implements Listener {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
/**
* Do something useful with the state update received
*/
#Override
#PostConstruct
public void listen() throws Exception {
Web3j web3j = null;
Disposable flowableEvent = null;
try {
WebSocketService web3jService = new WebSocketService("ws://", true);
web3jService.connect();
web3j = Web3j.build(web3jService);
ERC20Token token= ... //creating contract instance
flowableEvent = token.transferEventFlowable(DefaultBlockParameterName.LATEST, DefaultBlockParameterName.LATEST)
.subscribe(event -> {
try {
System.out.printf("hash=%s from=%s to=%s amount=%s%n",
event.log.getTransactionHash(),
event.from,
event.to,
event.value);
//process event data save to offchain db ==> service call
}catch(Throwable e) {
e.printStackTrace();
}
});
} catch (Exception e) {
logger.error("Unknown Exception " + e.getMessage());
throw new Exception(e.getMessage());
} finally {
web3j.shutdown();
flowableEvent.dispose();
}
}
}
Related
My app should do the next:
Send a POST request to server to get the token.
Connect to the websocket using this token in the headers while handshake.
Short question: To activate WebSocketClientProtocolHandler I have to fire event ctx.fireChannelActive() but from channelRead method because in this method I receive token from server . Is it correct place?
I implemented custom ChannelInboundHandlerAdapter and override:
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
authenticator.authenticate(ctx.channel()).addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture channelFuture) throws Exception {
if (!channelFuture.isSuccess()) {
authPromise.tryFailure(channelFuture.cause());
ctx.fireExceptionCaught(new RuntimeException("Auth is failed."));
} else {
ctx.fireUserEventTriggered("Auth is successful");
}
}
});
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
if (!(msg instanceof FullHttpResponse)) {
ctx.fireChannelRead(msg);
}
FullHttpResponse response = (FullHttpResponse) msg;
try {
authenticator.finishAuthentication(ctx.channel(), response);
authPromise.trySuccess();
ctx.pipeline().remove(this);
ctx.fireChannelActive();
} finally {
response.release();
}
}
Authenticator class adds needed handlers, sends POST request and then it should parse response from server and change the pipeline.
public class Authenticator {
private final Logger LOGGER = LoggerFactory.getLogger(this.getClass());
private final ObjectMapper mapper = new ObjectMapper();
private final MessengerConfig messengerConfig;
public Authenticator(MessengerConfig messengerConfig) {
this.messengerConfig = messengerConfig;
}
public ChannelFuture authenticate(Channel channel) {
this.preCheck(channel);
return this.authenticate(channel, channel.newPromise());
}
private void preCheck(Channel channel) {
ChannelPipeline pipeline = channel.pipeline();
HttpClientCodec httpClientCodec = pipeline.get(HttpClientCodec.class);
if (httpClientCodec == null) {
LOGGER.warn("Pipeline does not contain HttpClientCodec.");
pipeline.addFirst(HttpClientCodec.class.getName(), new HttpClientCodec());
LOGGER.info("HttpClientCodec was added to pipeline.");
}
HttpObjectAggregator httpObjectAggregator = pipeline.get(HttpObjectAggregator.class);
if (httpObjectAggregator == null) {
LOGGER.warn("Pipeline does not contain HttpObjectAggregator.");
pipeline.addAfter(
HttpClientCodec.class.getName(),
HttpObjectAggregator.class.getName(),
new HttpObjectAggregator(8192)
);
LOGGER.info("HttpObjectAggregator was added to pipeline.");
}
}
private ChannelFuture authenticate(Channel channel, ChannelPromise promise) {
HttpRequest request = createAuthRequest();
try {
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture channelFuture) throws Exception {
if (channelFuture.isSuccess()) {
promise.setSuccess();
} else {
promise.setFailure(new RuntimeException(""));
}
}
});
} catch (Exception e) {
LOGGER.error("Error", e);
}
return promise;
}
public void finishAuthentication(Channel channel, FullHttpResponse response) {
ByteBuf content = response.content();
AuthenticationData authenticationData = null;
try {
authenticationData = this.mapper.readValue(content.toString(CharsetUtil.UTF_8), AuthenticationData.class);
} catch (JsonProcessingException e) {
LOGGER.error("Can't parse authentication data.", e);
throw new RuntimeException((e));
}
LOGGER.info(Objects.toString(authenticationData));
DefaultWebSocketClientProtocolHandlerFactory factory = new DefaultWebSocketClientProtocolHandlerFactory();
WebSocketClientProtocolHandler handler = factory.getHandler(this.messengerConfig, authenticationData);
ChannelPipeline pipeline = channel.pipeline();
pipeline.addLast(WebSocketClientProtocolHandler.class.getName(), handler);
LOGGER.info("WebSocketClientProtocolHandler was added.");
pipeline.addLast(MessageHandler.class.getName(), new MessageHandler());
LOGGER.info("MessageHandler was added.");
}
So here I have two stages:
Auth stage with a pipeline:
io.netty.handler.codec.http.HttpClientCodec
io.netty.handler.codec.http.HttpObjectAggregator
AuthenticationHandler
2 Web-socket stage with a pipeline:
io.netty.handler.codec.http.HttpClientCodec
io.netty.handler.codec.http.HttpObjectAggregator
io.netty.handler.codec.http.websocketx.WebSocketClientProtocolHandshakeHandler
io.netty.handler.codec.http.websocketx.Utf8FrameValidator
io.netty.handler.codec.http.websocketx.WebSocketClientProtocolHandler
com.github.apsyvenko.client.messaging.MessageHandler
To activate second stage I have to fire event - ctx.fireChannelActive() but from channelRead.
As a result I got an exception:
18:19:37.055 [nioEventLoopGroup-2-1] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.net.SocketException: Connection reset
after hand-shake had started.
We are trying to use google cloud datastore in our project and trying to use objectify as the ORM since google recommends it. I have carefully used and tried everything i could read about and think of but somehow the transactions don't seem to work. Following is my code and setup.
#RunWith(SpringRunner.class)
#EnableAspectJAutoProxy(proxyTargetClass = true)
#ContextConfiguration(classes = { CoreTestConfiguration.class })
public class TestObjectifyTransactionAspect {
private final LocalServiceTestHelper helper = new LocalServiceTestHelper(
// Our tests assume strong consistency
new LocalDatastoreServiceTestConfig().setApplyAllHighRepJobPolicy(),
new LocalMemcacheServiceTestConfig(), new LocalTaskQueueTestConfig());
private Closeable closeableSession;
#Autowired
private DummyService dummyService;
#BeforeClass
public static void setUpBeforeClass() {
// Reset the Factory so that all translators work properly.
ObjectifyService.setFactory(new ObjectifyFactory());
}
/**
* #throws java.lang.Exception
*/
#Before
public void setUp() throws Exception {
System.setProperty("DATASTORE_EMULATOR_HOST", "localhost:8081");
ObjectifyService.register(UserEntity.class);
this.closeableSession = ObjectifyService.begin();
this.helper.setUp();
}
/**
* #throws java.lang.Exception
*/
#After
public void tearDown() throws Exception {
AsyncCacheFilter.complete();
this.closeableSession.close();
this.helper.tearDown();
}
#Test
public void testTransactionMutationRollback() {
// save initial list of users
List<UserEntity> users = new ArrayList<UserEntity>();
for (int i = 1; i <= 10; i++) {
UserEntity user = new UserEntity();
user.setAge(i);
user.setUsername("username_" + i);
users.add(user);
}
ObjectifyService.ofy().save().entities(users).now();
try {
dummyService.mutateDataWithException("username_1", 6L);
} catch (Exception e) {
e.printStackTrace();
}
List<UserEntity> users2 = this.dummyService.findAllUsers();
Assert.assertEquals("Size mismatch on rollback", users2.size(), 10);
boolean foundUserIdSix = false;
for (UserEntity userEntity : users2) {
if (userEntity.getUserId() == 1) {
Assert.assertEquals("Username update failed in transactional context rollback.", "username_1",
userEntity.getUsername());
}
if (userEntity.getUserId() == 6) {
foundUserIdSix = true;
}
}
if (!foundUserIdSix) {
Assert.fail("Deleted user with userId 6 but it is not rolledback.");
}
}
}
Since I am using spring, idea is to use an aspect with a custom annotation to weave objectify.transact around the spring service beans methods that are calling my daos.
But somehow the update due to ObjectifyService.ofy().save().entities(users).now(); is not gettign rollbacked though the exception throws causes Objectify to run its rollback code. I tried printing the ObjectifyImpl instance hashcodes and they are all same but still its not rollbacking.
Can someone help me understand what am i doing wrong? Havent tried the actual web based setup yet...if it cant pass transnational test cases there is no point in actual transaction usage in a web request scenario.
Update: Adding aspect, services, dao as well to make a complete picture. The code uses spring boot.
DAO class. Note i am not using any transactions here because as per code of com.googlecode.objectify.impl.TransactorNo.transactOnce(ObjectifyImpl<O>, Work<R>) a transnational ObjectifyImpl is flushed and committed in this method which i don't want. I want commit to happen once and rest all to join in on that transaction. Basically this is the wrong code in com.googlecode.objectify.impl.TransactorNo ..... i will try to explain my understanding a later in the question.
#Component
public class DummyDaoImpl implements DummyDao {
#Override
public List<UserEntity> loadAll() {
Query<UserEntity> query = ObjectifyService.ofy().transactionless().load().type(UserEntity.class);
return query.list();
}
#Override
public List<UserEntity> findByUserId(Long userId) {
Query<UserEntity> query = ObjectifyService.ofy().transactionless().load().type(UserEntity.class);
//query = query.filterKey(Key.create(UserEntity.class, userId));
return query.list();
}
#Override
public List<UserEntity> findByUsername(String username) {
return ObjectifyService.ofy().transactionless().load().type(UserEntity.class).filter("username", username).list();
}
#Override
public void update(UserEntity userEntity) {
ObjectifyService.ofy().save().entity(userEntity);
}
#Override
public void update(Iterable<UserEntity> userEntities) {
ObjectifyService.ofy().save().entities(userEntities);
}
#Override
public void delete(Long userId) {
ObjectifyService.ofy().delete().key(Key.create(UserEntity.class, userId));
}
}
Below is the Service class
#Service
public class DummyServiceImpl implements DummyService {
private static final Logger LOGGER = LoggerFactory.getLogger(DummyServiceImpl.class);
#Autowired
private DummyDao dummyDao;
public void saveDummydata() {
List<UserEntity> users = new ArrayList<UserEntity>();
for (int i = 1; i <= 10; i++) {
UserEntity user = new UserEntity();
user.setAge(i);
user.setUsername("username_" + i);
users.add(user);
}
this.dummyDao.update(users);
}
/* (non-Javadoc)
* #see com.bbb.core.objectify.test.services.DummyService#mutateDataWithException(java.lang.String, java.lang.Long)
*/
#Override
#ObjectifyTransactional
public void mutateDataWithException(String usernameToMutate, Long userIdToDelete) throws Exception {
//update one
LOGGER.info("Attempting to update UserEntity with username={}", "username_1");
List<UserEntity> mutatedUsersList = new ArrayList<UserEntity>();
List<UserEntity> users = dummyDao.findByUsername(usernameToMutate);
for (UserEntity userEntity : users) {
userEntity.setUsername(userEntity.getUsername() + "_updated");
mutatedUsersList.add(userEntity);
}
dummyDao.update(mutatedUsersList);
//delete another
UserEntity user = dummyDao.findByUserId(userIdToDelete).get(0);
LOGGER.info("Attempting to delete UserEntity with userId={}", user.getUserId());
dummyDao.delete(user.getUserId());
throw new RuntimeException("Dummy Exception");
}
/* (non-Javadoc)
* #see com.bbb.core.objectify.test.services.DummyService#findAllUsers()
*/
#Override
public List<UserEntity> findAllUsers() {
return dummyDao.loadAll();
}
Aspect which wraps the method annoted with ObjectifyTransactional as a transact work.
#Aspect
#Component
public class ObjectifyTransactionAspect {
private static final Logger LOGGER = LoggerFactory.getLogger(ObjectifyTransactionAspect.class);
#Around(value = "execution(* *(..)) && #annotation(objectifyTransactional)")
public Object objectifyTransactAdvise(final ProceedingJoinPoint pjp, ObjectifyTransactional objectifyTransactional) throws Throwable {
try {
Object result = null;
Work<Object> work = new Work<Object>() {
#Override
public Object run() {
try {
return pjp.proceed();
} catch (Throwable throwable) {
throw new ObjectifyTransactionExceptionWrapper(throwable);
}
}
};
switch (objectifyTransactional.propagation()) {
case REQUIRES_NEW:
int limitTries = objectifyTransactional.limitTries();
if(limitTries <= 0) {
Exception illegalStateException = new IllegalStateException("limitTries must be more than 0.");
throw new ObjectifyTransactionExceptionWrapper(illegalStateException);
} else {
if(limitTries == Integer.MAX_VALUE) {
result = ObjectifyService.ofy().transactNew(work);
} else {
result = ObjectifyService.ofy().transactNew(limitTries, work);
}
}
break;
case NOT_SUPPORTED :
case NEVER :
case MANDATORY :
result = ObjectifyService.ofy().execute(objectifyTransactional.propagation(), work);
break;
case REQUIRED :
case SUPPORTS :
ObjectifyService.ofy().transact(work);
break;
default:
break;
}
return result;
} catch (ObjectifyTransactionExceptionWrapper e) {
String packageName = pjp.getSignature().getDeclaringTypeName();
String methodName = pjp.getSignature().getName();
LOGGER.error("An exception occured while executing [{}.{}] in a transactional context."
, packageName, methodName, e);
throw e.getCause();
} catch (Throwable ex) {
String packageName = pjp.getSignature().getDeclaringTypeName();
String methodName = pjp.getSignature().getName();
String fullyQualifiedmethodName = packageName + "." + methodName;
throw new RuntimeException("Unexpected exception while executing ["
+ fullyQualifiedmethodName + "] in a transactional context.", ex);
}
}
}
Now the problem code part that i see is as follows in com.googlecode.objectify.impl.TransactorNo:
#Override
public <R> R transact(ObjectifyImpl<O> parent, Work<R> work) {
return this.transactNew(parent, Integer.MAX_VALUE, work);
}
#Override
public <R> R transactNew(ObjectifyImpl<O> parent, int limitTries, Work<R> work) {
Preconditions.checkArgument(limitTries >= 1);
while (true) {
try {
return transactOnce(parent, work);
} catch (ConcurrentModificationException ex) {
if (--limitTries > 0) {
if (log.isLoggable(Level.WARNING))
log.warning("Optimistic concurrency failure for " + work + " (retrying): " + ex);
if (log.isLoggable(Level.FINEST))
log.log(Level.FINEST, "Details of optimistic concurrency failure", ex);
} else {
throw ex;
}
}
}
}
private <R> R transactOnce(ObjectifyImpl<O> parent, Work<R> work) {
ObjectifyImpl<O> txnOfy = startTransaction(parent);
ObjectifyService.push(txnOfy);
boolean committedSuccessfully = false;
try {
R result = work.run();
txnOfy.flush();
txnOfy.getTransaction().commit();
committedSuccessfully = true;
return result;
}
finally
{
if (txnOfy.getTransaction().isActive()) {
try {
txnOfy.getTransaction().rollback();
} catch (RuntimeException ex) {
log.log(Level.SEVERE, "Rollback failed, suppressing error", ex);
}
}
ObjectifyService.pop();
if (committedSuccessfully) {
txnOfy.getTransaction().runCommitListeners();
}
}
}
transactOnce is by code / design always using a single transaction to do things. It will either commit or rollback the transaction. there is no provision to chain transactions like a normal enterprise app would want.... service -> calls multiple dao methods in a single transaction and commits or rollbacks depending on how things look.
keeping this in mind, i removed all annotations and transact method calls in my dao methods so that they don't start an explicit transaction and the aspect in service wraps the service method in transact and ultimately in transactOnce...so basically the service method is running in a transaction and no new transaction is getting fired again. This is a very basic scenario, in actual production apps services can call other service methods and they might have the annotation on them and we could still end up in a chained transaction..but anyway...that is a different problem to solve....
I know NoSQLs dont support write consistency at table or inter table levels so am I asking too much from google cloud datastore?
I'm using this library(https://github.com/rdavisau/sockets-for-pcl) to communicate with a TCP Server, that sends me when a event was generated, then, I have to verify all the time if the TCP Server sent to me a event, but if I try read anything before the TCP Server sends me, it's thrown the UnhandledException, but it only happens if I read inside a Task, in the main thread it thrown a timeout exception, the exception that I expected to happen in Task.
Someone can help me? Thanks. below is my code.
public class CentralTcpService
{
#region ConnectTcpAsync
public async void ConnectTcpAsync()
{
try
{
_sockecClient = new TcpSocketClient();
await _sockecClient.ConnectAsync(Central.Ip, Central.Port);
_writter = new ExtendedBinaryWriter(_sockecClient.WriteStream);
_reader = new ExtendedBinaryReader(_sockecClient.ReadStream);
_writter.WriteString(EvenNotProtocol.MobileReceiverCommand);
_sockecClient.ReadStream.ReadTimeout = int.MaxValue;
EnableTcpService();
}
catch (Exception e)
{
throw new Exception(e.Message);
}
}
#endregion
#region TcpService
private void EnableTcpService()
{
_cancelationTcpService = new CancellationTokenSource();
new Task(StartService, _cancelationTcpService.Token, TaskCreationOptions.LongRunning).Start();
}
private void StartService()
{
while (!_cancelationTcpService.Token.IsCancellationRequested)
{
var ev = EvenNotProtocol.DeserializeEvent(_reader);
if (ev == null) continue;
_writter.WriteString(EvenNotProtocol.MobileOkCommand);
EventReceived?.Invoke(this, new CentralTcpEventArgs(ev));
}
}
}
public class EvenNotProtocol
{
public static Event DeserializeEvent(ExtendedBinaryReader reader)
{
try
{
reader.SkipBytes(1);
.....
}
catch (IOException e)
{
return null;
}
}
}
I have a question regarding event listener. We have a event listener which listen to delete node event and perform some activity say "send email".
While code review i found this, although this code is working fine i am not convinced with the session being handled here :
#Activate
protected void activate(ComponentContext context) {
try{
final String path="/content/dam/";
Session session = repository.loginAdministrative(repository.getDefaultWorkspace());
observationManager = session.getWorkspace().getObservationManager();
observationManager.addEventListener(this, Event.PROPERTY_REMOVED, path, true, null, null, true);
checkOutProperty = OsgiUtil.toString(context.getProperties()
.get(ASSET_LOCK_PROPNAME_UPDATE), ASSET_LOCK_PROPNAME_DEFAULT);
if (session != null && session.isLive()) {
session.save();
}
} catch (RepositoryException e) {
if(LOG.isErrorEnabled()){
LOG.error("Error Occured in activate method of Property Removed Listener class:" + e.getMessage());
}
}catch (Exception e) {
if(LOG.isErrorEnabled()){
LOG.error("Error Occured in activate method of Property Removed Listener class:"+e.getMessage());
}
}
}
#Deactivate
protected void deactivate(ComponentContext componentContext) {
try {
if (observationManager != null) {
observationManager.removeEventListener(this);
}
} catch (RepositoryException e) {
if(LOG.isErrorEnabled()){
LOG.error("Error Occured " + e);
}
} catch (Exception e) {
if(LOG.isErrorEnabled()){
LOG.error(e.getMessage());
}
}
}
Questions:
Best practice would be to create session object private to this class and should be logout in deactivate method?
Once an event is added in Observation Manager, do we really need session object? I was expecting if we should logout from session there.
EventListener are a bit cumbersome here. I fought many battles with JCR Sessions and Sling ResourceResolvers within them. The problem is, you need to keep the Session active as long as the Event Listener is active. So the only thing missing in your code is a logout on deactivate.
I created an AbstractEventListener which takes care of this and provides the following two methods and has two private members:
private Session session;
private ObservationManager observationManager;
protected void addEventListener(final EventListener eventListener,
final int eventTypes, final String path, final String[] nodeTypes) {
try {
session = getRepositorySession();
observationManager = session.getWorkspace().getObservationManager();
observationManager.addEventListener(eventListener, eventTypes,
path, true, null, nodeTypes, true);
} catch (RepositoryException e) {
LOGGER.error("Repository error while registering observation: ", e);
}
}
protected void removeEventListener(final EventListener eventListener) {
if (observationManager != null) {
try {
observationManager.removeEventListener(eventListener);
} catch (RepositoryException e) {
LOGGER.error(
"Repository error while unregistering observation: ", e);
} finally {
logoutSession(session);
}
}
}
And then in the actual EventListener I just call them:
protected void activate(ComponentContext context) {
addEventListener(this, Event.PROPERTY_ADDED| Event.PROPERTY_CHANGED, "/content/mysite", null);
}
}
protected void deactivate(ComponentContext componentContext) {
removeEventListener(this);
}
I am trying to implement a simple pub sub example where I have a server and am publishing periodic notifications about uptime to clients.
This is being run as a part of a Windows service - bundled with InnoSetup and launch4j and Apache procrun/prunsrv.
The thread does not go beyond the creation of the context. What could be going wrong?
import java.io.IOException;
import java.util.Date;
import org.msgpack.MessagePack;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.zeromq.ZMQ;
import org.ocpsoft.prettytime.*;
/**
* Notification service for updates to configuration in the logger
* #author Aalhad
*/
public class NotificationServer extends Thread {
private final Logger log = LoggerFactory.getLogger(this.getClass());
private volatile boolean shouldRun;
private PrettyTime upTime;
private PreferenceManager prefMgr = PreferenceManager.getInstance();
public ZMQ.Context context;
public ZMQ.Socket pubSocket;
public NotificationServer() {
log.debug("Entered notification server constructor ......................");
context = ZMQ.context(1);
log.debug("THIS DOES NOT GET PRINTED ... it is as if we are blocking in ZMQ.context!!!");
pubSocket = context.socket(ZMQ.PUB);
pubSocket.bind("tcp://*:"+prefMgr.getNotificationPort());
pubSocket.bind("ipc://powerlogger");
log.debug("NotificationServer created");
}
#Override
public void run() {
log.debug("Entering run loop of Notification Server");
setStarting();
log.debug("Writing to tcp port: {}", prefMgr.getNotificationPort());
upTime = new PrettyTime();
ConfigMessage msg = prefMgr.getConfigMessage();
MessagePack msgPack = new MessagePack();
byte[] sendBytes;
try {
log.debug("Going ahead and sending: {}", msg);
sendBytes = msgPack.write(msg);
pubSocket.send(sendBytes);
log.debug("Finished sending msg");
} catch (IOException ex) {
log.error("Could not send first config notification",ex);
}
//On starts and restarts, we send the current configuration to our
//subscribers
String upSince;
while (shouldRun()) {
log.trace("In the notification loop");
upSince = upTime.format(new Date(0));
log.trace("============================================================== Started: {}", upSince );
ConfigMessage cfgMsg = new ConfigMessage();
cfgMsg.msgType = MessageType.UPSINCE;
cfgMsg.message = upSince;
try {
// ..... code here to write the time into a
// messagepack structure and publishing it
sleep(5000);
log.trace("After sleeping in notification loop");
} catch (InterruptedException ex) {
log.error("Notification thread disturbed when sleeping.");
}
}
}
public synchronized void shutDown() {
shouldRun = false;
log.trace("Set shouldRun to false in discovery server");
try {
if (pubSocket != null) {
pubSocket.close();
context.term();
}
}
catch(Exception e) {
log.error("Interesting situation when trying to close the discovery socket when shutting down",e);
}
}
public synchronized void setStarting() {
shouldRun = true;
}
private synchronized boolean shouldRun() {
return shouldRun;
}
}
Found the bug. It was a simple issue of forgetting to provide the jar for the service when creating the installer. The logs did not show that the class could not be found.
Fixed.