How to correctly recover when WatchKey.reset() returns false? - java-8

I have a long running process that 'watches' a directory for file changes.
The new Java 8 'watcher' API makes it very clear that WatchKey.reset() must be called on references returned by WatchService.take(). Note: The return result from WatchKey.reset() matches WatchKey.isValid().
Yesterday, my long running process observed many file updates and WatchKey.reset() normally returned true. However, for reasons I do not understand, a one call to WatchKey.reset() returned false.
What is the meaning of return value false from WatchKey.reset()? The official tutorial says: "If the key is no longer valid, the directory is inaccessible so exit the loop."
How do I recover from this state? I wish to continue monitoring the directory for file changes. The official tutorial and documentation does not explain how to do this.
For those less familiar with this new API, here is some setup code:
public static void main(String[] argArr)
throws IOException, InterruptedException {
final File dirPath = new File(argArr[0]);
if (! dirPath.isDirectory()) {
throw new IllegalArgumentException(argArr[0]);
}
final Path dirPath2 = dirPath.toPath();
try (final WatchService watchService = FileSystems.getDefault().newWatchService()) {
final WatchKey registerWatchKey =
dirPath2.register(
watchService,
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_MODIFY,
StandardWatchEventKinds.ENTRY_DELETE);
try {
while (true) {
// blocking
final WatchKey watchKey = watchService.take();
final List<WatchEvent<?>> watchEventList = watchKey.pollEvents();
for (final WatchEvent<?> watchEvent : watchEventList) {
// use 'watchEvent'
}
final boolean isValid = watchKey.reset();
if (! isValid) {
// How to recover here? I want to continue monitoring dirPath2.
}
}
}
finally {
registerWatchKey.cancel();
}
}
}

Related

How to use HttpContext inside Task.Run

There is some posts explain how to tackle, but couldnt help me much..
Logging Request/Response in middleware, it works when use 'await' with Task.Run() but since its awaited current operation to complete there is performance issue.
When I remove await as below, it runs fast but not logging anything, since HttpContext instance not available to use inside parallel thread
public class LoggingHandlerMiddleware
{
private readonly RequestDelegate next;
private readonly ILoggerManager _loggerManager;
public LoggingHandlerMiddleware(RequestDelegate next, ILoggerManager loggerManager)
{
this.next = next;
_loggerManager = loggerManager;
}
public async Task Invoke(HttpContext context, ILoggerManager loggerManager, IWebHostEnvironment environment)
{
_ = Task.Run(() =>
{
AdvanceLoggingAsync(context, _loggerManager, environment);
});
...
}
private void AdvanceLoggingAsync(HttpContext context, ILoggerManager loggerManager, IWebHostEnvironment environment, bool IsResponse = false)
{
{
context.Request.EnableBuffering(); // Throws ExecutionContext.cs not found
result += $"ContentType:{context.Request.ContentType},";
using (StreamReader reader = new StreamReader(context.Request.Body, Encoding.UTF8, true, 1024, true))
{
result += $"Body:{await reader.ReadToEndAsync()}";
context.Request.Body.Position = 0;
}
loggerManager.LogInfo($"Advance Logging Content(Request)-> {result}");
}
How can I leverage Task.Run() performance with accessing HttpContext?
Well, you can extract what you need from the context, build your string you want to log, and then pass that string to the task you run.
However, firing and forgetting a task is not good. If it throws an exception, you risk of bringing down the server, or at least you will have very hard time getting information about the error.
If you are concerned about the logging performance, better add what you need to log to a message queue, and have a process that responds to new messages in the queue and logs the message to the log file.

Downlolad and save file from ClientRequest using ExchangeFunction in Project Reactor

I have problem with correctly saving a file after its download is complete in Project Reactor.
class HttpImageClientDownloader implements ImageClientDownloader {
private final ExchangeFunction exchangeFunction;
HttpImageClientDownloader() {
this.exchangeFunction = ExchangeFunctions.create(new ReactorClientHttpConnector());
}
#Override
public Mono<File> downloadImage(String url, Path destination) {
ClientRequest clientRequest = ClientRequest.create(HttpMethod.GET, URI.create(url)).build();
return exchangeFunction.exchange(clientRequest)
.map(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()))
//.flatMapMany(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()))
.flatMap(dataBuffer -> {
AsynchronousFileChannel fileChannel = createFile(destination);
return DataBufferUtils
.write(dataBuffer, fileChannel, 0)
.publishOn(Schedulers.elastic())
.doOnNext(DataBufferUtils::release)
.then(Mono.just(destination.toFile()));
});
}
private AsynchronousFileChannel createFile(Path path) {
try {
return AsynchronousFileChannel.open(path, StandardOpenOption.CREATE);
} catch (Exception e) {
throw new ImageDownloadException("Error while creating file: " + path, e);
}
}
}
So my question is:
Is DataBufferUtils.write(dataBuffer, fileChannel, 0) blocking?
What about when the disk is slow?
And second question about what happens when ImageDownloadException occurs ,
In doOnNext I want to release the given data buffer, is that a good place for this kind operation?
I think also this line:
.map(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()))
could be blocking...
Here's another (shorter) way to achieve that:
Flux<DataBuffer> data = this.webClient.get()
.uri("/greeting")
.retrieve()
.bodyToFlux(DataBuffer.class);
Path file = Files.createTempFile("spring", null);
WritableByteChannel channel = Files.newByteChannel(file, StandardOpenOption.WRITE);
Mono<File> result = DataBufferUtils.write(data, channel)
.map(DataBufferUtils::release)
.then(Mono.just(file));
Now DataBufferUtils::write operations are not blocking because they use non-blocking IO with channels. Writing to such channels means it'll write whatever it can to the output buffer (i.e. may write all the DataBuffer or just part of it).
Using Flux::map or Flux::doOnNext is the right place to do that. But you're right, if an error occurs, you're still responsible for releasing the current buffer (and all the remaining ones). There might be something we can improve here in Spring Framework, please keep an eye on SPR-16782.
I don't see how your last sample shows anything blocking: all methods return reactive types and none are doing blocking I/O.

Log all methods called in an app by Xposed

As title said, I want to use xposed to log all methods called in an app from it start till I stop it. I only want to log Class name, Method name, don't want to hook all method.
I try this code, but get error getMethod not found.
findAndHookMethod("java.lang.Class", lpparam.classLoader, "getMethod", String.class, Object.class, new XC_MethodHook()
Thanks in advance!
There is no one line solution like what you seem to be searching.
Hooking all methods will let log what methods were called by app from it start till stop (sort of - see below), but if (for some reason) you don't want to hook all methods, the only solution I can think of is modifying the java VM itself (NOT something I would recommend.)
A solution that (sort of) works
What I did was first use apktool to decompile my apk and get the names of all the methods in all the classes.
Then I used xposed to hook into every single method of every class and print to the dlog the current function name.
Why it only sort of works
Xposed has an overhead whenever it hook a methods. For general usage of xposed apps, it isnt much. But when you start hooking each and every methods of an app, the overhead very quickly becomes ridiculously large - So much so that while the above methods works for small apps, for any large app it very quickly causes the app to hang and then crash.
An alternative that also sort-of works
FRIDA is a way to inject javascript to native apps. Here they show you how to log all function calls. While in the above link they log all function calls in a piece of python code, the same code also works for Android.
There is a way to log all Java methods.Modify XposedBridge.
Xposed hook java method through XposedBridge.java's method
"handleHookedMethod(Member method, int originalMethodId, Object additionalInfoObj, thisObject, Object[] args)"
Log.v(TAG, "className " + method.getClass().getName() + ",methodName " + method.getName());
As mentioned before Xposed is not the way to go in this situation due to its overhead.
The simplest solution is just to use dmtracedump as provided by Google. Most x86 Android images and emulator come with the debuggable flag on (ro.debuggable) so you can even use it for closed source apps.
Additionally other tools such as Emma are known to work with Android as well, but these might need modifications to the source code.
I found a solution.
See this code snippet below.
package com.kyunggi.logcalls;
import android.content.pm.*;
import android.util.*;
import dalvik.system.*;
import de.robv.android.xposed.*;
import de.robv.android.xposed.callbacks.XC_LoadPackage.*;
import java.io.*;
import java.lang.reflect.*;
import java.util.*;
import static de.robv.android.xposed.XposedHelpers.findAndHookMethod;
import android.app.*;
public class Main implements IXposedHookLoadPackage {
private String TAG = "LogCall";
public void handleLoadPackage(final LoadPackageParam lpparam) throws Throwable {
if (!lpparam.packageName.equals("com.android.bluetooth")) {
Log.i(TAG, "Not: " + lpparam.packageName);
return;
}
Log.i(TAG, "Yes " + lpparam.packageName);
//Modified https://d3adend.org/blog/?p=589
ApplicationInfo applicationInfo = AndroidAppHelper.currentApplicationInfo();
if (applicationInfo.processName.equals("com.android.bluetooth")) {
Set<String> classes = new HashSet<>();
DexFile dex;
try {
dex = new DexFile(applicationInfo.sourceDir);
Enumeration entries = dex.entries();
while (entries.hasMoreElements()) {
String entry = (String) entries.nextElement();
classes.add(entry);
}
dex.close();
} catch (IOException e) {
Log.e("HookDetection", e.toString());
}
for (String className : classes) {
boolean obex = false;
if (className.startsWith("com.android.bluetooth") || (obex = className.startsWith("javax.obex"))) {
try {
final Class clazz = lpparam.classLoader.loadClass(className);
for (final Method method : clazz.getDeclaredMethods()) {
if (obex) {
if (!Modifier.isPublic(method.getModifiers())) {
continue; //on javax.obex package, hook only public APIs
}
}
XposedBridge.hookMethod(method, new XC_MethodHook() {
final String methodNam = method.getName();
final String classNam = clazz.getName();
final StringBuilder sb = new StringBuilder("[");
final String logstr = "className " + classNam + ",methodName " + methodNam;
#Override
protected void beforeHookedMethod(MethodHookParam param) throws Throwable {
//Method method=(Method)param.args[0];
sb.setLength(0);
sb.append(logstr);
//Log.v(TAG,logstr);
for (Object o : param.args) {
String typnam = "";
String value = "null";
if (o != null) {
typnam = o.getClass().getName();
value = o.toString();
}
sb.append(typnam).append(" ").append(value).append(", ");
}
sb.append("]");
Log.v(TAG, sb.toString());
}
});
}
} catch (ClassNotFoundException e) {
Log.wtf("HookDetection", e.toString());
}
}
}
}
// ClassLoader rootcl=lpparam.classLoader.getSystemClassLoader();
//findAndHookMethod("de.robv.android.xposed.XposedBridge", rootcl, "handleHookedMethod", Member.class, int.class, Object.class, Object.class, Object[].class, );
}
}

How to close Java Formatter, in finally or not?

I know that normally streams and formatters (particularly java.util.Formatter) in Java should be closed in finally to avoid from resource leaks. But here I am a little bit confused, because I see a lot of examples where people just close it without any finally block, especially the formatters. This question may have no sense to some people, but I want to be sure in what I am asking about.
Some examples from java2s.com and from tutorialspoint.com where the formatters are just closed without any block.
Please consider that my question is only for Java 6 and lower versions, because I know about try with resources.
Example:
public static void main(String[] args) {
StringBuffer buffer = new StringBuffer();
Formatter formatter = new Formatter(buffer, Locale.US);
// format a new string
String name = "from java2s.com";
formatter.format("Hello %s !", name);
// print the formatted string
System.out.println(formatter);
// close the formatter
formatter.close();
// attempt to access the formatter results in exception
System.out.println(formatter);
}
In this specific example, it is not necessary to call close(). You only need to close the formatter if the underlying appender is Closable. In this case you are using a StringBuffer, which is not Closable so the call to close() does nothing. If you were to use Writer or PrintStream, those are closable and the call to close() would be necessary to avoid leaving the stream open.
If you are ever unsure if it is Closable it is best to just call close() anyway. No harm in doing so.
How about this, without further comments:
public static void main(String[] args) {
StringBuffer buffer = new StringBuffer();
Formatter formatter = null;
try {
formatter = new Formatter(buffer, Locale.US);
String name = "from java2s.com";
formatter.format("Hello %s !", name);
System.out.println(formatter);
}
finally {
if (formatter != null) {
formatter.close();
}
}
}

Non-Blocking Endpoint: Returning an operation ID to the caller - Would like to get your opinion on my implementation?

Boot Pros,
I recently started to program in spring-boot and I stumbled upon a question where I would like to get your opinion on.
What I try to achieve:
I created a Controller that exposes a GET endpoint, named nonBlockingEndpoint. This nonBlockingEndpoint executes a pretty long operation that is resource heavy and can run between 20 and 40 seconds.(in the attached code, it is mocked by a Thread.sleep())
Whenever the nonBlockingEndpoint is called, the spring application should register that call and immediatelly return an Operation ID to the caller.
The caller can then use this ID to query on another endpoint queryOpStatus the status of this operation. At the beginning it will be started, and once the controller is done serving the reuqest it will be to a code such as SERVICE_OK. The caller then knows that his request was successfully completed on the server.
The solution that I found:
I have the following controller (note that it is explicitely not tagged with #Async)
It uses an APIOperationsManager to register that a new operation was started
I use the CompletableFuture java construct to supply the long running code as a new asynch process by using CompletableFuture.supplyAsync(() -> {}
I immdiatelly return a response to the caller, telling that the operation is in progress
Once the Async Task has finished, i use cf.thenRun() to update the Operation status via the API Operations Manager
Here is the code:
#GetMapping(path="/nonBlockingEndpoint")
public #ResponseBody ResponseOperation nonBlocking() {
// Register a new operation
APIOperationsManager apiOpsManager = APIOperationsManager.getInstance();
final int operationID = apiOpsManager.registerNewOperation(Constants.OpStatus.PROCESSING);
ResponseOperation response = new ResponseOperation();
response.setMessage("Triggered non-blocking call, use the operation id to check status");
response.setOperationID(operationID);
response.setOpRes(Constants.OpStatus.PROCESSING);
CompletableFuture<Boolean> cf = CompletableFuture.supplyAsync(() -> {
try {
// Here we will
Thread.sleep(10000L);
} catch (InterruptedException e) {}
// whatever the return value was
return true;
});
cf.thenRun(() ->{
// We are done with the super long process, so update our Operations Manager
APIOperationsManager a = APIOperationsManager.getInstance();
boolean asyncSuccess = false;
try {asyncSuccess = cf.get();}
catch (Exception e) {}
if(true == asyncSuccess) {
a.updateOperationStatus(operationID, Constants.OpStatus.OK);
a.updateOperationMessage(operationID, "success: The long running process has finished and this is your result: SOME RESULT" );
}
else {
a.updateOperationStatus(operationID, Constants.OpStatus.INTERNAL_ERROR);
a.updateOperationMessage(operationID, "error: The long running process has failed.");
}
});
return response;
}
Here is also the APIOperationsManager.java for completness:
public class APIOperationsManager {
private static APIOperationsManager instance = null;
private Vector<Operation> operations;
private int currentOperationId;
private static final Logger log = LoggerFactory.getLogger(Application.class);
protected APIOperationsManager() {}
public static APIOperationsManager getInstance() {
if(instance == null) {
synchronized(APIOperationsManager.class) {
if(instance == null) {
instance = new APIOperationsManager();
instance.operations = new Vector<Operation>();
instance.currentOperationId = 1;
}
}
}
return instance;
}
public synchronized int registerNewOperation(OpStatus status) {
cleanOperationsList();
currentOperationId = currentOperationId + 1;
Operation newOperation = new Operation(currentOperationId, status);
operations.add(newOperation);
log.info("Registered new Operation to watch: " + newOperation.toString());
return newOperation.getId();
}
public synchronized Operation getOperation(int id) {
for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
return op;
}
}
Operation notFound = new Operation(-1, OpStatus.INTERNAL_ERROR);
notFound.setCrated(null);
return notFound;
}
public synchronized void updateOperationStatus (int id, OpStatus newStatus) {
iteration : for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
op.setStatus(newStatus);
log.info("Updated Operation status: " + op.toString());
break iteration;
}
}
}
public synchronized void updateOperationMessage (int id, String message) {
iteration : for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
op.setMessage(message);
log.info("Updated Operation status: " + op.toString());
break iteration;
}
}
}
private synchronized void cleanOperationsList() {
Date now = new Date();
for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if((now.getTime() - op.getCrated().getTime()) >= Constants.MIN_HOLD_DURATION_OPERATIONS ) {
log.info("Removed operation from watchlist: " + op.toString());
iterator.remove();
}
}
}
}
The questions that I have
Is that concept a valid one that also scales? What could be improved?
Will i run into concurrency issues / race conditions?
Is there a better way to achieve the same in boot spring, but I just didn't find that yet? (maybe with the #Async directive?)
I would be very happy to get your feedback.
Thank you so much,
Peter P
It is a valid pattern to submit a long running task with one request, returning an id that allows the client to ask for the result later.
But there are some things I would suggest to reconsider :
do not use an Integer as id, as it allows an attacker to guess ids and to get the results for those ids. Instead use a random UUID.
if you need to restart your application, all ids and their results will be lost. You should persist them to a database.
Your solution will not work in a cluster with many instances of your application, as each instance would only know its 'own' ids and results. This could also be solved by persisting them to a database or Reddis store.
The way you are using CompletableFuture gives you no control over the number of threads used for the asynchronous operation. It is possible to do this with standard Java, but I would suggest to use Spring to configure the thread pool
Annotating the controller method with #Async is not an option, this does not work no way. Instead put all asynchronous operations into a simple service and annotate this with #Async. This has some advantages :
You can use this service also synchronously, which makes testing a lot easier
You can configure the thread pool with Spring
The /nonBlockingEndpoint should not return the id, but a complete link to the queryOpStatus, including id. The client than can directly use this link without any additional information.
Additionally there are some low level implementation issues which you may also want to change :
Do not use Vector, it synchronizes on every operation. Use a List instead. Iterating over a List is also much easier, you can use for-loops or streams.
If you need to lookup a value, do not iterate over a Vector or List, use a Map instead.
APIOperationsManager is a singleton. That makes no sense in a Spring application. Make it a normal PoJo and create a bean of it, get it autowired into the controller. Spring beans by default are singletons.
You should avoid to do complicated operations in a controller method. Instead move anything into a service (which may be annotated with #Async). This makes testing easier, as you can test this service without a web context
Hope this helps.
Do I need to make database access transactional ?
As long as you write/update only one row, there is no need to make this transactional as this is indeed 'atomic'.
If you write/update many rows at once you should make it transactional to guarantee, that either all rows are updated or none.
However, if two operations (may be from two clients) update the same row, always the last one will win.

Resources