Where should a class like this add try-with-resources so that it correctly closes resources? I presume the appender should be reused as shown, right?
public class ChronicleWriter implements UpdateListener {
private ExcerptAppender appender;
private SingleChronicleQueue queue;
public ChronicleWriter(Path p) {
queue = SingleChronicleQueueBuilder.binary(p).build();
appender = queue.acquireAppender();
}
#Override
public void onUpdate(String text) {
appender.writeText(text);
}
}
queue.acquireAppender() call will acquire appender from thread local pool of weak references so there's little point reusing it, moreover it's dangerous unless your component is guaranteed to be used only in single thread as appenders are not thread safe.
The usual try-with-resources pattern we use is as follows:
try (final DocumentContext dc = queue.acquireAppender().writingDocument()) {
dc.wire().write("eventName").float32(Float.NaN);
}
Additionally you might want to make your class closeable and add:
public void close() {
net.openhft.chronicle.core.io.Closeable.closeQuietly(queue);
}
Related
I'm trying to declare multiple SFTP sessions, wrap them in a DelegatingSessionFactory, then later use SftpRemoteFileTemplate.execute(...) during a cron job.
On the execute part of things, the code is very simple, it is already used for a single session, but I want to expand it to multiple possible sessions.
Below I extended my single session code. I just copied the methods for reference. At the end I'll show how I think the new methods should look.
public class XSession extends SftpSession {
#Scheduled(cron = "${sftp.scan.x.schedule}")
void scan() {
List<FileHistoryEntity> fileList = template.execute(this::processFiles);
...
}
private List<FileHistoryEntity> processFiles(Session<ChannelSftp.LsEntry> session) {
List.of(session.list(this.remoteDir)).forEach(file -> doWhatever());
...
}
}
But now I have multiple sessions. So I declare the following class:
#Slf4j
#Configuration
#RequiredArgsConstructor
public class DelegateSftpSessionHandler {
private final SessionFactory<ChannelSftp.LsEntry> session1;
private final SessionFactory<ChannelSftp.LsEntry> session2;
private final SessionFactory<ChannelSftp.LsEntry> session3;
private final SessionFactory<ChannelSftp.LsEntry> session4;
private final SessionFactory<ChannelSftp.LsEntry> session5;
#RequiredArgsConstructor
public enum DelegateSessionConfig {
SESSION_1("IN_REALITY_A_RELEVANT_NAME_1");
SESSION_2("IN_REALITY_A_RELEVANT_NAME_2");
SESSION_3("IN_REALITY_A_RELEVANT_NAME_3");
SESSION_4("IN_REALITY_A_RELEVANT_NAME_4");
SESSION_5("IN_REALITY_A_RELEVANT_NAME_5");
public final String threadKey;
}
#Bean
#Primary
public DelegatingSessionFactory<ChannelSftp.LsEntry> delegatingSessionFactory() {
Map<Object, SessionFactory<ChannelSftp.LsEntry>> sessionMap = new HashMap<>();
sessionMap.put(DelegateSessionConfig.SESSION_1.threadKey, session1);
sessionMap.put(DelegateSessionConfig.SESSION_2.threadKey, session2);
sessionMap.put(DelegateSessionConfig.SESSION_3.threadKey, session3);
sessionMap.put(DelegateSessionConfig.SESSION_4.threadKey, session4);
sessionMap.put(DelegateSessionConfig.SESSION_5.threadKey, session5);
DefaultSessionFactoryLocator<ChannelSftp.LsEntry> sessionLocator = new DefaultSessionFactoryLocator<>(sessionMap);
return new DelegatingSessionFactory<>(sessionLocator);
}
#Bean
SftpRemoteFileTemplate ftpRemoteFileTemplate(DelegatingSessionFactory<ChannelSftp.LsEntry> dsf) {
return new SftpRemoteFileTemplate(dsf);
}
}
Ting is, I have no idea how any of this works, and the spring sftp / fpt documentation is by no means clear. The code is virtually undocumented. And I'm just guessing. I think that I have to do the following:
public class XSession extends SftpSession {
#Autowire
DelegatingSessionFactory<ChannelSftp.LsEntry> delegatingSessionFactory;
#Autowired
SftpRemoteFileTemplate template;
#Scheduled(cron = "${sftp.scan.x.schedule}") // x == SESSION_1
#Async // for thread key
void scan() {
delegatingSessionFactory.setThreadKey(DelegateSessionConfig.SESSION_1.threadKey);
// because thread key changes the session globally? So I don't need specify
// which session this template is working with???
List<FileHistoryEntity> fileList = template.execute(this::processFiles);
...
delegatingSessionFactory.clearThreadKey();
}
private List<FileHistoryEntity> processFiles(Session<ChannelSftp.LsEntry> session) {
List.of(session.list(this.remoteDir)).forEach(file -> doWhatever());
...
}
}
I'm basing what I'm saying on the following link, github spring integration test
Honestly, I hardly understand what is happening. But it seems like setting the thread key, changes the session globally.
My only other idea is to just ... create the RemoteFileTemplate on demand
public static SftpRemoteFileTemplate getTemplateFor(DelegatingSessionFactory<ChannelSftp.LsEntry> dsf, DelegateSessionConfig session) {
return new SftpRemoteFileTemplate(dsf.getFactoryLocator().getSessionFactory(session.threadKey));
}
It does not set it globally. That's how a ThreadLocal variable works: you set a value in some thread and only this thread can see it. If you use the same object concurrently, other threads don't see that value because it does not belong to their thread state.
Not sure what is your concern, but pattern to extend an SftpSession for custom logic is not right. You should consider to use an SftpRemoteFileTemplate.execute(SessionCallback<F, T> callback) instead, but thread key must be set into a DelegatingSessionFactory before anyway and in the same thread you going to call that execute().
Is it possible to process multiple amqp - messages in parallel with the same method annotated with #Incoming("queue") with quarkus and smallrye-reactive-messaging?
To be more precise, I have following class:
#ApplicationScoped
public class Receiver {
#Incoming("test-queue")
public void process(String input) {
System.out.println("start processing:" + input);
try {
Thread.sleep(10_000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("end processing:" + input);
}
}
With the configuration in the application.properties:
amqp-host: localhost
amqp-port: 5672
amqp-username: quarkus
amqp-password: quarkus
mp.messaging.incoming.test-queue.connector: smallrye-amqp
mp.messaging.incoming.test-queue.address: test-queue
Now I'd like define by configuration how many parallel processing of messages are possible. For example, on a 4 core cpu it should run 4 in parallel.
Currently I can just add 4 copies of the method with different names to allow this parallelism, but that is not configurable.
I'm not sure, but I don't think Reactive Messaging supports what you're asking for.
You can, however, do what you want another way. I think it's also a better overall pattern for using messaging.
http://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/2.5/amqp/amqp.html#amqp-inbound
Find the example with the CompletionStage and the explicit ack(). That variant is asynchronous, so if you combine it with Java's existing concurrency facilities, you'll get efficient parallel processing.
I would send the incoming work to an executor, and then have the executing task ack() when it completes.
I just came across the same scenario and here is how the spec intends for you to handle concurrency:
From eclipse Microprofile spec
Basically, instead of having a class with a method like this:
#Incoming("test-queue")
public void process(String input) {}
You have 2 classes like this:
#ApplicationScoped
public class MessageSubscriberProducer {
#Incoming("test-queue")
public Subscriber<String> createSubscriber() {
return new SubscriberImpl();
}
}
public class SubsciberImpl implements Subscriber<String> {
private Subscription subscription;
#Override
public void onSubscribe(Subscription subscription) {
this.subscription = subscription;
this.subscription.request(4); // this tells how many messages to grab right away
}
#Override
public void onNext(String val) {
// do processing
this.subscription.request(1); // grab 1 more
}
}
This has the additional advantage of moving your processing code from the vert.x event-loop thread to a worker thread pool.
I have a simple scenario in which am trying to verify some behavior when a method is called (i.e. that a certain method was called with given parameter, a function pointer in this scenario). Below are my classes:
#SpringBootApplication
public class Application {
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
AppBootStrapper bootStrapper = context.getBean(AppBootStrapper.class);
bootStrapper.start();
}
}
#Component
public class AppBootStrapper {
private NetworkScanner networkScanner;
private PacketConsumer packetConsumer;
public AppBootStrapper(NetworkScanner networkScanner, PacketConsumer packetConsumer) {
this.networkScanner = networkScanner;
this.packetConsumer = packetConsumer;
}
public void start() {
networkScanner.addConsumer(packetConsumer::consumePacket);
networkScanner.startScan();
}
}
#Component
public class NetworkScanner {
private List<Consumer<String>> consumers = new ArrayList<>();
public void startScan(){
Executors.newSingleThreadExecutor().submit(() -> {
while(true) {
// do some scanning and get/parse packets
consumers.forEach(consumer -> consumer.accept("Package Data"));
}
});
}
public void addConsumer(Consumer<String> consumer) {
this.consumers.add(consumer);
}
}
#Component
public class PacketConsumer {
public void consumePacket(String packet) {
System.out.println("Packet received: " + packet);
}
}
#RunWith(JUnit4.class)
public class AppBootStrapperTest {
#Test
public void start() throws Exception {
NetworkScanner networkScanner = mock(NetworkScanner.class);
PacketConsumer packetConsumer = mock(PacketConsumer.class);
AppBootStrapper appBootStrapper = new AppBootStrapper(networkScanner, packetConsumer);
appBootStrapper.start();
verify(networkScanner).addConsumer(packetConsumer::consumePacket);
verify(networkScanner, times(1)).startScan();
}
}
I want to verify that bootStrapper did in fact do proper setup by registering the packet consumer(there might be other consumers registered later on, but this one is mandatory) and then called startScan. I get the following error message when I execute the test case:
Argument(s) are different! Wanted:
networkScanner bean.addConsumer(
com.spring.starter.AppBootStrapperTest$$Lambda$8/438123546#282308c3
);
-> at com.spring.starter.AppBootStrapperTest.start(AppBootStrapperTest.java:24)
Actual invocation has different arguments:
networkScanner bean.addConsumer(
com.spring.starter.AppBootStrapper$$Lambda$7/920446957#5dda14d0
);
-> at com.spring.starter.AppBootStrapper.start(AppBootStrapper.java:12)
From the exception, clearly the function pointers aren't the same.
Am I approaching this the right way? Is there something basic I am missing? I played around and had a consumer injected into PacketConsumer just to see if it made a different and that was OK, but I know that's certainly not the right way to go.
Any help, perspectives on this would be greatly appreciated.
Java doesn't have any concept of "function pointers"; when you see:
networkScanner.addConsumer(packetConsumer::consumePacket);
What Java actually compiles is (the equivalent of):
networkScanner.addConsumer(new Consumer<String>() {
#Override void accept(String packet) {
packetConsumer.consumePacket(packet);
}
});
This anonymous inner class happens to be called AppBootStrapper$$Lambda$7. Because it doesn't (and shouldn't) define an equals method, it will never be equal to the anonymous inner class that the compiler generates in your test, which happens to be called AppBootStrapperTest$$Lambda$8. This is regardless of the fact that the method bodies are the same, and are built in the same way from the same method reference.
If you generate the Consumer explicitly in your test and save it as a static final Consumer<String> field, then you can pass that reference in the test and compare it; at that point, reference equality should hold. This should work with a lambda expression or method reference just fine.
A more apt test would probably verify(packetConsumer, atLeastOnce()).consumePacket(...), as the contents of the lambda are an implementation detail and you're really more concerned about how your component collaborates with other components. The abstraction here should be at the consumePacket level, not at the addConsumer level.
See the comments and answer on this SO question.
We are experiencing a problem in production where consumers are having zero utilization and the queues keep growing and performance degrades.
Each of the consumers is a container which contains a single instance of a non thread-safe listener bean.
Each listener needs to write to its own set of files. In order to avoid thread contention I would like only one thread to write to its own set of files.
Each listener is only instantiated once by using #Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
I'm using a configuration similar to the one in this question
Each container is also configured with a retry advice which has the following code:
public class RetryMessageAdvice extends StatelessRetryOperationsInterceptorFactoryBean {
private static final int DEFAULT_RETRY_COUNT = 5;
private static final int DEFAULT_BACKOFF_MS = 250;
private int retryCount;
private int backOffPeriodInMS;
public RetryMessageAdvice() {
this.retryCount = DEFAULT_RETRY_COUNT;
this.backOffPeriodInMS = DEFAULT_BACKOFF_MS;
initializeRetryPolicy();
}
public RetryMessageAdvice(int retryCount, int backoff) {
this.retryCount = retryCount;
this.backOffPeriodInMS = backoff;
initializeRetryPolicy();
}
public void initializeRetryPolicy() {
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(this.retryCount);
FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
backOffPolicy.setBackOffPeriod(backOffPeriodInMS);
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setRetryPolicy(retryPolicy);
retryTemplate.setBackOffPolicy(backOffPolicy);
this.setRetryOperations(retryTemplate);
this.setMessageRecoverer(new RetryMessageRecoverer());
}
public int getRetryCount() {
return retryCount;
}
public void setRetryCount(int retryCount) {
this.retryCount = retryCount;
}
}
The consumer looks something like this:
#Component("exportListenerImpl")
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public class ExportListenerImpl extends ExportListenerBase {
private static final Logger LOG = LoggerFactory.getLogger(ExportListenerImpl.class);
private final ExportMapper exportMapper;
private final ExportFormatter exportFormatter;
#Autowired
public ExportListenerImpl(#Qualifier("exportFormatter") ExportFormatter exportFormatter,
#Qualifier("exportMapper") ExportedMapper exportedMapper,
#Value("${export.root.dir}") String exportDirectory) {
super(exportDirectory);
this.exportedFormatter = exportFormatter;
this.exportedMapper = exportedMapper;
}
#Override
public void handle(AnalyticsEvent analyticsEvent) throws Exception {
ExportedEvent exportedEvent = exportMapper.mapPlace(analyticsEvent);
File csvFile = getCsvFile(exportedEvent);
String csvRow = exportFormatter.writeAsString(exportedEvent);
writeCsvRow(csvRow, csvFile);
}
}
Other things to note
Export mapper and export formatter are thread-safe but not using #Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
The method writeCsvRow is synchronized.
There is a high number of errors which cause the exportMapper to throw an exception and trigger the retry advice
The incoming message rage is 120/s
The ratio between the incoming and deliver rate is usually 5:1
My theories on what is wrong are
The high number of errors is causing a large number of retries and
degrading performance. I would be better off putting the bad message
in an error queue.
Somehow the synchronized method in writeCsvRow is
causing problems with some higher level thread managed by
spring-amqp.
My question is, which theory is right? Is the impact of the retry advice the problem?
If those beans are also not thread-safe, they must also be prototype scope.
Since there's only one thread, synchronizing that method is not necessary but it shouldn't hurt.
If the errors are irrecoverable, you should configure the retry policy to not retry those exceptions.
.
With those retry settings, you will suspend the container thread for 250ms each time you get an error. So, yes; it will hurt performance.
Shouldn't be a significant overhead.
I have an app that uses the "task:scheduler" and "task:scheduled-tasks" elements (the latter containing "task:scheduled" elements). This is all working fine.
I'm trying to write some code that introspects the "application configuration" to get a short summary of some important information, like what tasks are scheduled and what their schedule is.
I already have a class that has a bunch of "#Autowired" instance variables so I can iterate through all of this. It was easy enough to add a "List" to get all of the TaskScheduler objects. I only have two of these, and I have a different set of scheduled tasks in each of them.
What I can't see in those TaskScheduler objects (they are actually ThreadPoolTaskScheduler objects) is anything that looks like a list of scheduled tasks, so I'm guessing the list of scheduled tasks is recorded somewhere else.
What objects can I use to introspect the set of scheduled tasks, and which thread pool they are in?
This functionality will be available in Spring 4.2
https://jira.spring.io/browse/SPR-12748 (Disclaimer: I reported this issue and contributed code towards its solution).
// Warning there may be more than one ScheduledTaskRegistrar in your
// application context. If this is the case you can autowire a list of
// ScheduledTaskRegistrar instead.
#Autowired
private ScheduledTaskRegistrar scheduledTaskRegistrar;
public List<Task> getScheduledTasks() {
List<Task> result = new ArrayList<Task>();
result.addAll(this.scheduledTaskRegistrar.getTriggerTaskList());
result.addAll(this.scheduledTaskRegistrar.getCronTaskList());
result.addAll(this.scheduledTaskRegistrar.getFixedRateTaskList());
result.addAll(this.scheduledTaskRegistrar.getFixedDelayTaskList());
return result;
}
// You can this inspect the tasks,
// so for example a cron task can be inspected like this:
public List<CronTask> getScheduledCronTasks() {
List<CronTask> cronTaskList = this.scheduledTaskRegistrar.getCronTaskList();
for (CronTask cronTask : cronTaskList) {
System.out.println(cronTask.getExpression);
}
return cronTaskList;
}
If you are using a ScheduledMethodRunnable defined in XML:
<task:scheduled method="run" cron="0 0 12 * * ?" ref="MyObject" />
You can access the underlying target object:
ScheduledMethodRunnable scheduledMethodRunnable = (ScheduledMethodRunnable) task.getRunnable();
TargetClass target = (TargetClass) scheduledMethodRunnable.getTarget();
I have a snippet for pre spring 4.2 since it is still sitting at release candidate level.
The scheduledFuture interface is implemented by every runnable element in the BlockingQueue.
Map<String, ThreadPoolTaskScheduler> schedulers = applicationContext
.getBeansOfType(ThreadPoolTaskScheduler.class);
for (ThreadPoolTaskScheduler scheduler : schedulers.values()) {
ScheduledExecutorService exec = scheduler.getScheduledExecutor();
ScheduledThreadPoolExecutor poolExec = scheduler
.getScheduledThreadPoolExecutor();
BlockingQueue<Runnable> queue = poolExec.getQueue();
Iterator<Runnable> iter = queue.iterator();
while (iter.hasNext()) {
ScheduledFuture<?> future = (ScheduledFuture<?>) iter.next();
future.getDelay(TimeUnit.MINUTES);
Runnable job = iter.next();
logger.debug(MessageFormat.format(":: Task Class is {0}", JobDiscoverer.findRealTask(job)));
}
Heres a reflective way to get information about which job class is in the pool as threadPoolNamePrefix didn't return a distinct name for me:
public class JobDiscoverer {
private final static Field syncInFutureTask;
private final static Field callableInFutureTask;
private static final Class<? extends Callable> adapterClass;
private static final Field runnableInAdapter;
private static Field reschedulingRunnable;
private static Field targetScheduledMethod;
static {
try {
reschedulingRunnable = Class
.forName(
"org.springframework.scheduling.support.DelegatingErrorHandlingRunnable")
.getDeclaredField("delegate");
reschedulingRunnable.setAccessible(true);
targetScheduledMethod = Class
.forName(
"org.springframework.scheduling.support.ScheduledMethodRunnable")
.getDeclaredField("target");
targetScheduledMethod.setAccessible(true);
callableInFutureTask = Class.forName(
"java.util.concurrent.FutureTask$Sync").getDeclaredField(
"callable");
callableInFutureTask.setAccessible(true);
syncInFutureTask = FutureTask.class.getDeclaredField("sync");
syncInFutureTask.setAccessible(true);
adapterClass = Executors.callable(new Runnable() {
public void run() {
}
}).getClass();
runnableInAdapter = adapterClass.getDeclaredField("task");
runnableInAdapter.setAccessible(true);
} catch (NoSuchFieldException e) {
throw new ExceptionInInitializerError(e);
} catch (SecurityException e) {
throw new PiaRuntimeException(e);
} catch (ClassNotFoundException e) {
throw new PiaRuntimeException(e);
}
}
public static Object findRealTask(Runnable task) {
if (task instanceof FutureTask) {
try {
Object syncAble = syncInFutureTask.get(task);
Object callable = callableInFutureTask.get(syncAble);
if (adapterClass.isInstance(callable)) {
Object reschedulable = runnableInAdapter.get(callable);
Object targetable = reschedulingRunnable.get(reschedulable);
return targetScheduledMethod.get(targetable);
} else {
return callable;
}
} catch (IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
throw new ClassCastException("Not a FutureTask");
}
With #Scheduled based configuration the approach from Tobias M’s answer does not work out-of-the-box.
Instead of autowiring a ScheduledTaskRegistrar instance (which is not available for annotation based configuration), you can instead autowire a ScheduledTaskHolder which only has a getScheduledTasks() method.
Background:
The ScheduledAnnotationBeanPostProcessor used to manage #Scheduled tasks has an internal ScheduledTaskRegistrar that’s not available as a bean. It does implement ScheduledTaskHolder, though.
Every Spring XML element has a corresponding BeanDefinitionReader. For <task:scheduled-tasks>, that's ScheduledTasksBeanDefinitionParser.
This uses a BeanDefinitionBuilder to create a BeanDefinition for a bean of type ContextLifecycleScheduledTaskRegistrar. Your scheduled tasks will be stored in that bean.
The tasks will be executing in either a default TaskScheduler or one you provided.
I've given you the class names so you can look at the source code yourself if you need more fine grained details.