I am using the spring Resource API and using a ResourcePatternResolver to scan my classpath for files.
In one situation the scan is picking up some directories and files that are in a pre-built jar and some that are on the file system.
In either case a 'resource' will either be a file or a directory. How can I reliably detect whether a resource points to a directory or file, whether in a jar file or not? Calling getFile() on a Resource inside a jar throws an Exception so I can't use that plus isFile() as I initially tried.
Spring’s Resource interface is meant to be a more capable interface for abstracting access to low-level resources.
It wraps File sometimes while sometimes not.
It has six built-in implements: UrlResource, ClassPathResource, FileSystemResource, ServletContextResource, InputStreamResource, ByteArrayResource.
You can implement yourself resource form.
The UrlResource wraps a java.net.URL, and may be used to access any Object that is normally accessible via a URL. If you use http: prefix ,the resource is a URL.
The ClassPathResource represents a resource which should be obtained from the classpath. This Resource implementation supports resolution as java.io.File if the class path resource resides in the file system, but not for classpath resources which reside in a jar and have not been expanded (by the servlet engine, or whatever the environment is) to the filesystem. To address this the various Resource implementations always support resolution as a java.net.URL.
FileSystemResource is an implement for java.io.File handles.It obviously supports resolution as a File and as a URL.
InputStreamResource is a resource implements for a given InputStream. Do not use it if you need to keep the resource descriptor somewhere, or if you need read a stream multiple times.
ByteArrayResource is a Resource implement for a given byte array. It creates a ByteArrayInputStream for the given byte array.
So you should not always use getFile() as Spring's Resource doesn't always represent a file system resource.For this reason, we recommend that you use getInputStream() to access resource contents because it is likely to function for all possible resource types.
Refer to: Resources
I think you can just surround the code checking for file by a try catch block:
boolean isFile = true;
try {
resource.getFile()
...
} catch (...Exception e) {
ifFile = false
}
I had a similar requirement, and solved it by excluding directories from my search pattern. Then for each resource found I lookup the parent item in the path, and ensure the directory has been created before writing the file.
In my case the file could be in the filesystem, or in the classpath, so I check the scheme of the URI first..
Although my search pattern may still pickup dirs if they have a dot in the name, so it would be better to catch the exception in that case -
search pattern - classpath*:/**/sprout/plugins/**/*.*
Example code -
private void extractClientPlugins() throws IOException {
Resource[] resourcePaths = resolver.getResourcePaths(sproutPluginSearchPattern);
Path pluginFolderPath = Paths.get(sproutHome, "./plugins/");
pluginFolderPath.toFile().mkdirs();
if (resourcePaths.length == 0) {
log.info("No Sprout client side plugins found");
}
for (Resource resource : resourcePaths) {
try {
Path destinationPath = generateDestinationPath(pluginFolderPath, resource);
File parentFolder = destinationPath.getParent().toFile();
if (!parentFolder.exists()) {
parentFolder.mkdirs();
}
destinationPath.toFile().mkdirs();
copy(resource, destinationPath.toFile());
} catch (IOException e) {
log.error("could not access resource", e);
throw e;
}
}
}
private Path generateDestinationPath(Path rootDir, Resource resource) throws IOException {
String relativePath = null;
String scheme = resource.getURI().getScheme();
if ("JAR".contains(scheme.toUpperCase())) {
String[] uriParts = resource.getURL().toString().split("!");
relativePath = trimPluginPathPrefix(uriParts[1]);
} else {
String filePath = resource.getFile().getAbsolutePath();
relativePath = trimPluginPathPrefix(filePath);
}
return Paths.get(rootDir.toString(), relativePath);
}
private String trimPluginPathPrefix(String filePath) {
String[] pathParts = filePath.split("sprout/plugins/");
if (pathParts.length != 2) {
throw new RuntimeException("The plugins must be located in a path containing '**/sprout/plugins/*'");
}
return pathParts[1];
}
Using it in this project -
https://github.com/savantly-net/sprout-platform/blob/master/sprout-core/src/main/java/net/savantly/sprout/core/ui/UiLoader.java
Related
I am getting a classic well-known error in my Springboot app when it tries to load a static resource using getResourceAsStream():
java.io.FileNotFoundException: class path resource [blah blah blah] cannot be opened because it does not exist
While doing some basic troubleshooting, I opened the contents of the generated jar file and found that everything in the jar is contained inside a folder named BOOT-INF.
Why is everything inside BOOT-INF?
How do I get rid of this?
UPDATE
Here's the actual relevant part of the code
ClassPathResource cpr = new ClassPathResource(RESOURCE_CLASSPATH_PREFIX + url);
try (InputStream is = cpr.getInputStream()) {
return DigestUtils.sha1Hex(is);
} catch (IOException e) {
throw new RuntimeException(e);
}
Additional Clue
The failing code is inside a ServletFilter. I think this has something to do with class loaders.
while working on an ftp server with the vfs2 library I noticed, that I had to enable VFS.setUriStyle(true) so the library would change the working directory to parent directory of the target file I am operating on (cwd directoryName).
But if UriStyle is enabled, everything is being resolved relativly to the root. Which would not be a Problem if the root was not "//".
The class GenericFileName sets the absolutePath of the root to "/", which makes the Method getPath() return "/"+getUriTrailer() which in the case of the root always returns "//". Everything that is resolved relativly to // has two dots proceeding to their path.
Which means if I execute the following code:
public class RemoteFileTest {
public static void main(String[] args) {
// Options for a RemoteFileObject connection
VFS.setUriStyle(true);
FileSystemOptions options = new FileSystemOptions();
// we doing an ftp connection, hence we use the ftpConfigBuilder
// we want to work in passive mode
FtpFileSystemConfigBuilder.getInstance().setPassiveMode(options, true);
FtpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(options, false);
// DefaultFileSystemConfigBuilder.getInstance().setRootURI(options, "/newRoot/");
// System.out.println(DefaultFileSystemConfigBuilder.getInstance().getRootURI(options));
// ftp://localhost:21/
StaticUserAuthenticator auth = new StaticUserAuthenticator("", "user", "pass");
try {
DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(options, auth);
} catch (FileSystemException e) {
e.printStackTrace();
return;
}
// A FileSystemManager creates an abstract FileObject linked to are desired RemoteFile.
// That link is just simulated and not yet real.
FileSystemManager manager;
try {
manager = VFS.getManager();
} catch (FileSystemException e) {
e.printStackTrace();
return;
}
try (FileObject remoteFile = manager.resolveFile("ftp://localhost:21/sub_folder/test.txt", options)) {
System.out.println("Is Folder " + remoteFile.isFolder());
System.out.println("Is File " + remoteFile.isFile());
} catch (FileSystemException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
}}
I receive this interaction with the ftp server:
USER user
PASS ****
TYPE I
CWD //
SYST
PASV
LIST ..sub_folder/
PWD
CWD ..sub_folder/
I want the interaction to be just like this, but without the two dots infront of the directory.
Kind regards
Barry
Fixed it as described below:
Disabled uriStyle again.
Wrote my own VFS class which creates my custom written Manager.
That Manager overwrites the FtpFileProvider with my custom one, which simply sets the root to a custom selected one, which causes the desired behaviour.
import org.apache.commons.vfs2.FileName;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.FileSystem;
import org.apache.commons.vfs2.FileSystemException;
import org.apache.commons.vfs2.FileSystemOptions;
import org.apache.commons.vfs2.impl.DefaultFileSystemConfigBuilder;
import org.apache.commons.vfs2.provider.ftp.FtpFileProvider;
public class AdvancedFtpFileProvider extends FtpFileProvider {
public AdvancedFtpFileProvider() {
super();
// setFileNameParser(AdvancedFtpFileNameParser.getInstance());
}
#Override
protected FileObject findFile(FileName name, FileSystemOptions fileSystemOptions) throws FileSystemException {
// Check in the cache for the file system
//getContext().getFileSystemManager().resolveName... resolves the configured RootUri relative to the selected root (name.getRoot()). This calls cwd to the selectedRoot and operates from there with relatives urls towards the new root!
final FileName rootName = getContext().getFileSystemManager().resolveName(name.getRoot(), DefaultFileSystemConfigBuilder.getInstance().getRootURI(fileSystemOptions));
final FileSystem fs = getFileSystem(rootName, fileSystemOptions);
// Locate the file
// return fs.resolveFile(name.getPath());
return fs.resolveFile(name);
}
}
Came across this question because I was having the same issue with the following
ftp://user:pass#host//home/user/file.txt
becoming... (note the single slash after 'home')
ftp://user:pass#host/home/user/file.txt
I did this to solve the issue...
// Setup some options, add as many as you need
FileSystemOptions opts = new FileSystemOptions( );
// This line tells VFS to treat the URI as the absolute path and not relative
FtpsFileSystemConfigBuilder.getInstance( ).setUserDirIsRoot( opts, false );
// Retrieve the file from the remote FTP server
FileObject realFileObject = fileSystemManager.resolveFile( fileSystemUri, opts );
I hope this can help someone, if not then provide a reference for the next time this stumps me.
My method is as such
public static Stream<Path> readFolder(String path) throws URISyntaxException, IOException
{
ClassLoader loader = FileUtil.class.getClassLoader();
URL dir = loader.getResource(path);
if (dir == null) return Stream.empty();
...
if (dir.getProtocol().equals("jar"))
{
FileSystem fileSystem = FileSystems.newFileSystem(dir.toURI(), Collections.<String, Object>emptyMap());
Stream<Path> results = Files.walk(fileSystem.getPath(path), 1).filter(file -> !file.toString().equals(dir.toString()));
fileSystem.close();
return results;
}
return Stream.empty();
}
Running the code from a jar fails. If I just run
Paths.get(path) the result is /WEB-INF/lib/Database-1.0.0.jar!/com/update/sql/ but fileSystem.getPath(path) returns com/update/sql/.
So I think the issue might be fileSystem is having issues reading the Database-1.0.0.jar inside my Application.jar!/WEB-INF/lib directory but maybe not since Paths.get(path) finds the correct file.
My error is:
java.nio.file.NoSuchFileException: com/update/sql/
at com.sun.nio.zipfs.ZipPath.getAttributes(ZipPath.java:666)
at com.sun.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:294)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
at java.nio.file.Files.walk(Files.java:3574)
at util.FileUtil.readFolder(FileUtil.java:36)
I've scanned other questions here on Stack but nothing has seemed to address this specific issue. Any help would be greatly appreciated.
We are developing one application which will read multiple files & write multiple files i.e. one output file for one input file (name of output file must be same as input file).
MultiResourceItemReader can read multiple files but not simultaneously, which is a performance bottleneck for us. Spring batch provides multithreading support for this but again many threads will read the same file & try to write it. Since output file name must be same as Input file name, we can't use that option too.
Now I am looking for one more possibility, if I can create 'n' threads to read & write 'n' files. But I am not sure how to integrate this logic with Spring Batch framework.
Advance thanks for any help.
Since MultiResourceItemReader doesn't meet your performance needs you may take a closer look at parallel processing, which you already mentioned is a desirable option. I don't think many threads will read the same file and try to write it when running multi-threaded, if configured correctly.
Rather than taking the typical chunk-oriented approach you could create a tasklet-orient step that is partitioned (multi-threaded). The tasklet class would be the main driver, delegating calls to a reader and a writer.
The general flow would be something like this:
Retrieve the names of all the files that need to be read in/written out (via some service class) and save them to the execution context within an implementation of Partitioner.
public class filePartitioner implements Partitioner {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, Path> filesToProcess = this.service.getFilesToProcess(directory); // this is just sudo-ish code but maybe you inject the directory you'll be targeting into this class
Map<String, ExecutionContext> execCtxs = new HashMap<>();
for(Entry<String, Path> entry : filesToProcess.entrySet()) {
ExecutionContext execCtx = new ExecutionContext();
execCtx.put("file", entry.getValue());
execCtxs.put(entry.getKey(), execCtx);
}
return execCtxs;
}
// injected
public void setServiceClass(ServiceClass service) {
this.service = service;
}
}
a. For the .getFilesToProcess() method you just need something that returns all of the files in the designated directory because you need to eventually know what is to be read and the name of the file that is to be written. Obviously there are several ways to go about this, such as...
public Map<String, Path> getFilesToProcess(String directory) {
Map<String, Path> filesToProcess = new HashMap<String, Path>();
File directoryFile = new File(directory); // where directory is where you intend to read from
this.generateFileList(filesToProcess, directoryFile, directory);
private void generateFileList(Map<String, Path> fileList, File node, String directory) {
// traverse directory and get files, adding to file list.
if(node.isFile()) {
String file = node.getAbsoluteFile().toString().substring(directory.length() + 1, node.toString().length());
fileList.put(file, directory);
}
if(node.isDirectory()) {
String[] files = node.list();
for(String filename : files) {
this.generateFileList(fileList, new File(node, filename), directory);
}
}
}
You'll need to create a tasklet, which will pull file names from the execution context and pass them to some injected class that will read in the file and write it out (custom ItemReaders and ItemWriters may be necessary).
The rest of the work would be in configuration, which should be fairly straight forward. It is in the configuration of the Partitioner where you can set your grid size, which could even be done dynamically using SpEL if you really intend to create n threads for n files. I would bet a fixed number of threads running across n files would show significant improvement in performance but you'll be able to determine that for yourself.
Hope this helps.
Basically i want to discover if a jar implements any number of interfaces wihtout activating or starting the bundle. Is it possible to read the meta data from the meta-inf from an API just like the container does but without activating a bundle ?
I want to use OSGi to support plugins of which numerous interfaces will be published and i would like to know which interfaces are implemented by a bundle when the user uploads without activating the bundle etc.
I do not think it is possible to discover what services a bundle is going to provide, because this can happen from inside the Java code, without any meta-data about it. Of course, if you use Declarative Services, there is a meta-data file. Also, the bundle needs to import (or provide) the service interface, which may give you a hint (but not more).
You can inspect what Java packages a bundles imports and exports without activating it.
If you are willing to install (not resolve, not activate) it, you can query it. The Felix or Equinox shells can list those packages after all.
Here is the relevant source from Felix' shell. It uses the PackageAdmin service:
public void execute(String s, PrintStream out, PrintStream err)
{
// Get package admin service.
ServiceReference ref = m_context.getServiceReference(
org.osgi.service.packageadmin.PackageAdmin.class.getName());
PackageAdmin pa = (ref == null) ? null :
(PackageAdmin) m_context.getService(ref);
// ...
Bundle bundle = m_context.getBundle( bundleId );
ExportedPackage[] exports = pa.getExportedPackages(bundle);
// ...
}
you may try something like below. Find the ".class" files in the exported packages using bundle.findResource(...) method.
BundleContext context = bundle.getBundleContext();
ServiceReference ref = context.getServiceReference(PackageAdmin.class.getName());
PackageAdmin packageAdmin = (PackageAdmin)context.getService(ref);
List<Class> agentClasses = new ArrayList<Class>();
ExportedPackage[] exportedPackages = packageAdmin.getExportedPackages(bundle);
for(ExportedPackage ePackage : exportedPackages){
String packageName = ePackage.getName();
String packagePath = "/"+packageName.replace('.', '/');
//find all the class files in current exported package
Enumeration clazzes = bundle.findEntries(packagePath, "*.class", false);
while(clazzes.hasMoreElements()){
URL url = (URL)clazzes.nextElement();
String path = url.getPath();
int index = path.lastIndexOf("/");
int endIndex = path.length()-6;//Strip ".class" substring
String className = path.substring(index+1, endIndex);
String fullClassName=packageName+"."+className;
try {
Class clazz = bundle.loadClass(fullClassName);
//check whether the class is annotated with Agent tag.
if(clazz.isAnnotationPresent(Agent.class))
agentClasses.add(clazz);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
}