apache VFS2 uriStyle - root absolute path ends with double slash - ftp

while working on an ftp server with the vfs2 library I noticed, that I had to enable VFS.setUriStyle(true) so the library would change the working directory to parent directory of the target file I am operating on (cwd directoryName).
But if UriStyle is enabled, everything is being resolved relativly to the root. Which would not be a Problem if the root was not "//".
The class GenericFileName sets the absolutePath of the root to "/", which makes the Method getPath() return "/"+getUriTrailer() which in the case of the root always returns "//". Everything that is resolved relativly to // has two dots proceeding to their path.
Which means if I execute the following code:
public class RemoteFileTest {
public static void main(String[] args) {
// Options for a RemoteFileObject connection
VFS.setUriStyle(true);
FileSystemOptions options = new FileSystemOptions();
// we doing an ftp connection, hence we use the ftpConfigBuilder
// we want to work in passive mode
FtpFileSystemConfigBuilder.getInstance().setPassiveMode(options, true);
FtpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(options, false);
// DefaultFileSystemConfigBuilder.getInstance().setRootURI(options, "/newRoot/");
// System.out.println(DefaultFileSystemConfigBuilder.getInstance().getRootURI(options));
// ftp://localhost:21/
StaticUserAuthenticator auth = new StaticUserAuthenticator("", "user", "pass");
try {
DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(options, auth);
} catch (FileSystemException e) {
e.printStackTrace();
return;
}
// A FileSystemManager creates an abstract FileObject linked to are desired RemoteFile.
// That link is just simulated and not yet real.
FileSystemManager manager;
try {
manager = VFS.getManager();
} catch (FileSystemException e) {
e.printStackTrace();
return;
}
try (FileObject remoteFile = manager.resolveFile("ftp://localhost:21/sub_folder/test.txt", options)) {
System.out.println("Is Folder " + remoteFile.isFolder());
System.out.println("Is File " + remoteFile.isFile());
} catch (FileSystemException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
}}
I receive this interaction with the ftp server:
USER user
PASS ****
TYPE I
CWD //
SYST
PASV
LIST ..sub_folder/
PWD
CWD ..sub_folder/
I want the interaction to be just like this, but without the two dots infront of the directory.
Kind regards
Barry

Fixed it as described below:
Disabled uriStyle again.
Wrote my own VFS class which creates my custom written Manager.
That Manager overwrites the FtpFileProvider with my custom one, which simply sets the root to a custom selected one, which causes the desired behaviour.
import org.apache.commons.vfs2.FileName;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.FileSystem;
import org.apache.commons.vfs2.FileSystemException;
import org.apache.commons.vfs2.FileSystemOptions;
import org.apache.commons.vfs2.impl.DefaultFileSystemConfigBuilder;
import org.apache.commons.vfs2.provider.ftp.FtpFileProvider;
public class AdvancedFtpFileProvider extends FtpFileProvider {
public AdvancedFtpFileProvider() {
super();
// setFileNameParser(AdvancedFtpFileNameParser.getInstance());
}
#Override
protected FileObject findFile(FileName name, FileSystemOptions fileSystemOptions) throws FileSystemException {
// Check in the cache for the file system
//getContext().getFileSystemManager().resolveName... resolves the configured RootUri relative to the selected root (name.getRoot()). This calls cwd to the selectedRoot and operates from there with relatives urls towards the new root!
final FileName rootName = getContext().getFileSystemManager().resolveName(name.getRoot(), DefaultFileSystemConfigBuilder.getInstance().getRootURI(fileSystemOptions));
final FileSystem fs = getFileSystem(rootName, fileSystemOptions);
// Locate the file
// return fs.resolveFile(name.getPath());
return fs.resolveFile(name);
}
}

Came across this question because I was having the same issue with the following
ftp://user:pass#host//home/user/file.txt
becoming... (note the single slash after 'home')
ftp://user:pass#host/home/user/file.txt
I did this to solve the issue...
// Setup some options, add as many as you need
FileSystemOptions opts = new FileSystemOptions( );
// This line tells VFS to treat the URI as the absolute path and not relative
FtpsFileSystemConfigBuilder.getInstance( ).setUserDirIsRoot( opts, false );
// Retrieve the file from the remote FTP server
FileObject realFileObject = fileSystemManager.resolveFile( fileSystemUri, opts );
I hope this can help someone, if not then provide a reference for the next time this stumps me.

Related

Implementing Jump Hosts with SSHJ

Somebody asked for this and there is a pull-request which contains code that somehow was rewritten before it got merged and somebody managed to code a solution based on the pull-request. However, there is no example for the final version in that library.
Therefore, that doesn't really help me with my limited understanding of ssh and all. Basically there are two scenarios I want to solve:
common SSH-session via some jump-hosts:
user1#jump1.com
user2#jump2.com
user3#jump3.com
admin#server.com
ending in an ssh-session where the connecting user is free to work around in that ssh-shell at server.com, i.e. what a normal ssh admin#server.com-command would do in the shell on jump3.com.
like the above but ending in a port forwarding to server.com:80
That is possible with ssh's ProxyCommand, but I want to code this with SSHJ. And that's where I fail to figure out how to do this.
What I have now is
SSHClient hop1 = new SSHClient();
try {
Path knownHosts = rootConfig.getKnownHosts();
if (knownHosts != null) {
hop1.loadKnownHosts(knownHosts.toFile());
} else {
hop1.loadKnownHosts();
}
Path authenticationFile = hop1Config.getAuthenticationFile();
if (authenticationFile != null) {
KeyProvider keyProvider = hop1.loadKeys(authenticationFile.toString(), (String) null);
hop1.authPublickey(hop1Config.getUser(), keyProvider);
} else {
hop1.authPassword(hop1Config.getUser(), hop1Config.getPassword());
}
// I found these methods:
hop1.getConnection();
hop1.getSocket();
// and now what?
} catch (IOException e) {
logger.error("Failed to open ssh-connection to {}", hop1Config, e);
}
I noticed class LocalPortForwarder.DirectTCPIPChannel, but I don't know with what values I should instantiate it or how to use it with the rest afterwards.

How to correctly recover when WatchKey.reset() returns false?

I have a long running process that 'watches' a directory for file changes.
The new Java 8 'watcher' API makes it very clear that WatchKey.reset() must be called on references returned by WatchService.take(). Note: The return result from WatchKey.reset() matches WatchKey.isValid().
Yesterday, my long running process observed many file updates and WatchKey.reset() normally returned true. However, for reasons I do not understand, a one call to WatchKey.reset() returned false.
What is the meaning of return value false from WatchKey.reset()? The official tutorial says: "If the key is no longer valid, the directory is inaccessible so exit the loop."
How do I recover from this state? I wish to continue monitoring the directory for file changes. The official tutorial and documentation does not explain how to do this.
For those less familiar with this new API, here is some setup code:
public static void main(String[] argArr)
throws IOException, InterruptedException {
final File dirPath = new File(argArr[0]);
if (! dirPath.isDirectory()) {
throw new IllegalArgumentException(argArr[0]);
}
final Path dirPath2 = dirPath.toPath();
try (final WatchService watchService = FileSystems.getDefault().newWatchService()) {
final WatchKey registerWatchKey =
dirPath2.register(
watchService,
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_MODIFY,
StandardWatchEventKinds.ENTRY_DELETE);
try {
while (true) {
// blocking
final WatchKey watchKey = watchService.take();
final List<WatchEvent<?>> watchEventList = watchKey.pollEvents();
for (final WatchEvent<?> watchEvent : watchEventList) {
// use 'watchEvent'
}
final boolean isValid = watchKey.reset();
if (! isValid) {
// How to recover here? I want to continue monitoring dirPath2.
}
}
}
finally {
registerWatchKey.cancel();
}
}
}

Log all methods called in an app by Xposed

As title said, I want to use xposed to log all methods called in an app from it start till I stop it. I only want to log Class name, Method name, don't want to hook all method.
I try this code, but get error getMethod not found.
findAndHookMethod("java.lang.Class", lpparam.classLoader, "getMethod", String.class, Object.class, new XC_MethodHook()
Thanks in advance!
There is no one line solution like what you seem to be searching.
Hooking all methods will let log what methods were called by app from it start till stop (sort of - see below), but if (for some reason) you don't want to hook all methods, the only solution I can think of is modifying the java VM itself (NOT something I would recommend.)
A solution that (sort of) works
What I did was first use apktool to decompile my apk and get the names of all the methods in all the classes.
Then I used xposed to hook into every single method of every class and print to the dlog the current function name.
Why it only sort of works
Xposed has an overhead whenever it hook a methods. For general usage of xposed apps, it isnt much. But when you start hooking each and every methods of an app, the overhead very quickly becomes ridiculously large - So much so that while the above methods works for small apps, for any large app it very quickly causes the app to hang and then crash.
An alternative that also sort-of works
FRIDA is a way to inject javascript to native apps. Here they show you how to log all function calls. While in the above link they log all function calls in a piece of python code, the same code also works for Android.
There is a way to log all Java methods.Modify XposedBridge.
Xposed hook java method through XposedBridge.java's method
"handleHookedMethod(Member method, int originalMethodId, Object additionalInfoObj, thisObject, Object[] args)"
Log.v(TAG, "className " + method.getClass().getName() + ",methodName " + method.getName());
As mentioned before Xposed is not the way to go in this situation due to its overhead.
The simplest solution is just to use dmtracedump as provided by Google. Most x86 Android images and emulator come with the debuggable flag on (ro.debuggable) so you can even use it for closed source apps.
Additionally other tools such as Emma are known to work with Android as well, but these might need modifications to the source code.
I found a solution.
See this code snippet below.
package com.kyunggi.logcalls;
import android.content.pm.*;
import android.util.*;
import dalvik.system.*;
import de.robv.android.xposed.*;
import de.robv.android.xposed.callbacks.XC_LoadPackage.*;
import java.io.*;
import java.lang.reflect.*;
import java.util.*;
import static de.robv.android.xposed.XposedHelpers.findAndHookMethod;
import android.app.*;
public class Main implements IXposedHookLoadPackage {
private String TAG = "LogCall";
public void handleLoadPackage(final LoadPackageParam lpparam) throws Throwable {
if (!lpparam.packageName.equals("com.android.bluetooth")) {
Log.i(TAG, "Not: " + lpparam.packageName);
return;
}
Log.i(TAG, "Yes " + lpparam.packageName);
//Modified https://d3adend.org/blog/?p=589
ApplicationInfo applicationInfo = AndroidAppHelper.currentApplicationInfo();
if (applicationInfo.processName.equals("com.android.bluetooth")) {
Set<String> classes = new HashSet<>();
DexFile dex;
try {
dex = new DexFile(applicationInfo.sourceDir);
Enumeration entries = dex.entries();
while (entries.hasMoreElements()) {
String entry = (String) entries.nextElement();
classes.add(entry);
}
dex.close();
} catch (IOException e) {
Log.e("HookDetection", e.toString());
}
for (String className : classes) {
boolean obex = false;
if (className.startsWith("com.android.bluetooth") || (obex = className.startsWith("javax.obex"))) {
try {
final Class clazz = lpparam.classLoader.loadClass(className);
for (final Method method : clazz.getDeclaredMethods()) {
if (obex) {
if (!Modifier.isPublic(method.getModifiers())) {
continue; //on javax.obex package, hook only public APIs
}
}
XposedBridge.hookMethod(method, new XC_MethodHook() {
final String methodNam = method.getName();
final String classNam = clazz.getName();
final StringBuilder sb = new StringBuilder("[");
final String logstr = "className " + classNam + ",methodName " + methodNam;
#Override
protected void beforeHookedMethod(MethodHookParam param) throws Throwable {
//Method method=(Method)param.args[0];
sb.setLength(0);
sb.append(logstr);
//Log.v(TAG,logstr);
for (Object o : param.args) {
String typnam = "";
String value = "null";
if (o != null) {
typnam = o.getClass().getName();
value = o.toString();
}
sb.append(typnam).append(" ").append(value).append(", ");
}
sb.append("]");
Log.v(TAG, sb.toString());
}
});
}
} catch (ClassNotFoundException e) {
Log.wtf("HookDetection", e.toString());
}
}
}
}
// ClassLoader rootcl=lpparam.classLoader.getSystemClassLoader();
//findAndHookMethod("de.robv.android.xposed.XposedBridge", rootcl, "handleHookedMethod", Member.class, int.class, Object.class, Object.class, Object[].class, );
}
}

Accessing Distributed Cache in Pig StoreFunc

I have looked at all the other threads on this topic and still have not found an answer...
Put simply, I want to access hadoop distributed cache from a Pig StoreFunc, and NOT from within a UDF directly.
Relevant PIG code lines:
DEFINE CustomStorage KeyValStorage('param1','param2','param3');
...
STORE BLAH INTO /path/ using CustomStorage();
Relevant Java Code:
public class KeyValStorage<M extends Message> extends BaseStoreFunc /* ElephantBird Storage which inherits from StoreFunc */ {
...
public KeyValStorage(String param1, String param2, String param3) {
...
try {
InputStream is = new FileInputStream(configName);
try {
prop.load(is);
} catch (IOException e) {
System.out.println("PROPERTY LOADING FAILED");
e.printStackTrace();
}
} catch (FileNotFoundException e) {
System.out.println("FILE NOT FOUND");
e.printStackTrace();
}
}
...
}
configName is the name of the LOCAL file that I should be able to read from distributed cache, however, I am getting a FileNotFoundException. When I use the EXACT same code from within a PIG UDF directly, the file is found, so I know the file is being shipped via distributed cache. I set the appropriate param to make sure this happens:
<property><name>mapred.cache.files</name><value>/path/to/file/file.properties#configName</value></property>
Any ideas how I can get around this?
Thanks!
StroreFunc's constructor is called both at frontend and backend. When it is called from the frontend, (before the job is launched) then you'll get FileNotFoundException because at this point the files from the distributed cache are not yet copied to the nodes' local disk.
You may check whether you are at the backend (when the job is being executed) and load the file only in this case e.g:
DEFINE CustomStorage KeyValStorage('param1','param2','param3');
set mapreduce.job.cache.files hdfs://host/user/cache/file.txt#config
...
STORE BLAH INTO /path/ using CustomStorage();
public KeyValStorage(String param1, String param2, String param3) {
...
try {
if (!UDFContext.getUDFContext().isFrontend()) {
InputStream is = new FileInputStream("./config");
BufferedReader br = new BufferedReader(new InputStreamReader(is));
...
...
}

Is spring Resource a file or directory?

I am using the spring Resource API and using a ResourcePatternResolver to scan my classpath for files.
In one situation the scan is picking up some directories and files that are in a pre-built jar and some that are on the file system.
In either case a 'resource' will either be a file or a directory. How can I reliably detect whether a resource points to a directory or file, whether in a jar file or not? Calling getFile() on a Resource inside a jar throws an Exception so I can't use that plus isFile() as I initially tried.
Spring’s Resource interface is meant to be a more capable interface for abstracting access to low-level resources.
It wraps File sometimes while sometimes not.
It has six built-in implements: UrlResource, ClassPathResource, FileSystemResource, ServletContextResource, InputStreamResource, ByteArrayResource.
You can implement yourself resource form.
The UrlResource wraps a java.net.URL, and may be used to access any Object that is normally accessible via a URL. If you use http: prefix ,the resource is a URL.
The ClassPathResource represents a resource which should be obtained from the classpath. This Resource implementation supports resolution as java.io.File if the class path resource resides in the file system, but not for classpath resources which reside in a jar and have not been expanded (by the servlet engine, or whatever the environment is) to the filesystem. To address this the various Resource implementations always support resolution as a java.net.URL.
FileSystemResource is an implement for java.io.File handles.It obviously supports resolution as a File and as a URL.
InputStreamResource is a resource implements for a given InputStream. Do not use it if you need to keep the resource descriptor somewhere, or if you need read a stream multiple times.
ByteArrayResource is a Resource implement for a given byte array. It creates a ByteArrayInputStream for the given byte array.
So you should not always use getFile() as Spring's Resource doesn't always represent a file system resource.For this reason, we recommend that you use getInputStream() to access resource contents because it is likely to function for all possible resource types.
Refer to: Resources
I think you can just surround the code checking for file by a try catch block:
boolean isFile = true;
try {
resource.getFile()
...
} catch (...Exception e) {
ifFile = false
}
I had a similar requirement, and solved it by excluding directories from my search pattern. Then for each resource found I lookup the parent item in the path, and ensure the directory has been created before writing the file.
In my case the file could be in the filesystem, or in the classpath, so I check the scheme of the URI first..
Although my search pattern may still pickup dirs if they have a dot in the name, so it would be better to catch the exception in that case -
search pattern - classpath*:/**/sprout/plugins/**/*.*
Example code -
private void extractClientPlugins() throws IOException {
Resource[] resourcePaths = resolver.getResourcePaths(sproutPluginSearchPattern);
Path pluginFolderPath = Paths.get(sproutHome, "./plugins/");
pluginFolderPath.toFile().mkdirs();
if (resourcePaths.length == 0) {
log.info("No Sprout client side plugins found");
}
for (Resource resource : resourcePaths) {
try {
Path destinationPath = generateDestinationPath(pluginFolderPath, resource);
File parentFolder = destinationPath.getParent().toFile();
if (!parentFolder.exists()) {
parentFolder.mkdirs();
}
destinationPath.toFile().mkdirs();
copy(resource, destinationPath.toFile());
} catch (IOException e) {
log.error("could not access resource", e);
throw e;
}
}
}
private Path generateDestinationPath(Path rootDir, Resource resource) throws IOException {
String relativePath = null;
String scheme = resource.getURI().getScheme();
if ("JAR".contains(scheme.toUpperCase())) {
String[] uriParts = resource.getURL().toString().split("!");
relativePath = trimPluginPathPrefix(uriParts[1]);
} else {
String filePath = resource.getFile().getAbsolutePath();
relativePath = trimPluginPathPrefix(filePath);
}
return Paths.get(rootDir.toString(), relativePath);
}
private String trimPluginPathPrefix(String filePath) {
String[] pathParts = filePath.split("sprout/plugins/");
if (pathParts.length != 2) {
throw new RuntimeException("The plugins must be located in a path containing '**/sprout/plugins/*'");
}
return pathParts[1];
}
Using it in this project -
https://github.com/savantly-net/sprout-platform/blob/master/sprout-core/src/main/java/net/savantly/sprout/core/ui/UiLoader.java

Resources