as hbase is not available as osgi-ified bundle yet I managed to create the bundle with the maven felix plugin (hbase 0.92 and the corresponding hadoop-core 1.0.0), and both bundles are starting up in OSGi :)
also the hbase-default.xml is added to the resulting bundle. in the resulting osgi-jar, when I open it, the structure looks like this:
org/
META-INF
hbase-default.xml
This was achieved with <Include-Resource>#${pkgArtifactId}-${pkgVersion}.jar!/hbase-default.xml</Include-Resource>
The problem comes up when I actually want to connect to hbase. hbase-default.xml can not be found and thus I can not create any configuration file.
The hbase osgi bundle is used from within another osgi-bundle that should be used to get an hbase connection and query the database. This osgi-bundle is used by an RCP application.
My question is, where do I have to put my hbase-default.xml so that it will be found when the bundle is started? or why does it not realize that the file is existing?
Thank you for any hints.
-- edit
I found a decompiler so I could view the source where the loading of the configuration is executed (hadoop-core which does not provide any sources via maven) and I now see that the Threads contextClassLoader is used (and if not available the classLoader of the Configuration class itself), so it seems to me that it can't find the resource, but, it should, according to the description, also check the parents (but who is the parent in an OSGi environment?)?
I tested to get the resource from the OSGi-bundle that should use hbase, where I added hbase-default.xml to the created jar file (see above), and there I get a resource when I get the contextClassLoader of the thread. When I explored the code a bit more I realized that there is no way to set the classloader for the HBaseConfiguration (although it would be possible to set the classloader for a "simple" hadoop-Configuration, HBaseConfiguration inherits from, but the creation procedure of HBaseConfiguration does not allow it, as it simply creates a new object within the create() method.
I really hope you have some idea how to get this up and running :)
Thread.currentThread().setContextClassLoader(HBaseConfiguration.class.getClassLoader());
Make sure the HBaseConfiguration class loaded in your OSGI bundle.hbase will make use of the thread context classloader, in order to load resources (hbase-default.xml and hbase-site.xml). Setting the TCCL will allow you to load the defaults and override them later.
If hbase-default.xml is in the .jar file which is in the CLASSPATH, that file normally can be find by java program.
I have read the hbase mailing list.
check your pom.xml:
in 'process-resource' phase, hbase-default.xml's '###VERSION###' would be replaced with the actual version string. however, if this phase configuration is set to be 'target', not 'tasks', the replacement would not occur.
You could have a look at your pom.xml, ant correct the label to if so.
faced this issue, actually fixed it by putting hbase-site.xml in the bundle which I was calling hbase from, found advise here:
Using this component in OSGi: This component is fully functional in an OSGi environment however, it requires some actions from the user. Hadoop uses the thread context class loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.
Related
I am trying to do a log4j migration for a legacy application. I made the changes to the configuration files and tested it locally with Tomcat, where the logs were displayed correctly. However when I deploy on WAS, the logging stops.
I checked a lot of pages about the necessary dependencies and the possible problems, but now I feel I'm stuck. Here is what I already tried:
Added the log4j jars manually to the lib/ext directory in WAS (together with commons-logging, I read an article where it helped, although all these jar-s were already present in the ear)
Double checked, that the configuration file is on the classpath and that it's syntax is correct
Added logj42-web dependency
Added the necessary filters in web.xml (Spring version is 3.8, so also the config listener with the config name context-param)
Checked classloader in WAS - it's parent first
Am I missing something?
Any ideas are appreciated. Thank you in advance!
WAS includes its own commons-logging API (that does not utilize Log4j), so bringing your own logger requires a bit of extra class-loader-related config. Typically, that would go something like this:
Put your commons-logging and log4j jars, along with your logging properties files, in some directory (not WAS_HOME/lib/ext) readable by the user running the server.
Create a shared library on the server, with that directory as its class path, and select the "use an isolated class loader" option. Associate that shared library with your application or web module.
Ideally, that should be it. Isolated shared libraries search themselves before delegating to the server's loaders, so your application will "see" the commons-logging/log4j classes in the shared library instead of in the server, and likewise, it should pick up the configuration files from that directory instead of the ones found in the server.
You can also accomplish this same basic thing by leaving all the logging stuff in your WAR or EAR and setting its class loader to parent-last (which causes the class loader to search locally before delegating to the server-level loaders), but that is a bit riskier configuration - if your application includes APIs that are also provided by the server, parent-last class loading increases the possibility of ClassCastExceptions or LinkageErrors.
I'm creating a Liferay 7.1 OSGi bundle, which has some external dependencies in it. In consideration of time, we opted to embed the external JAR in our OSGi Bundle. I've managed to create a bnd file, which includes all of the ElasticSearch dependencies, and put them on the bundle classpath. I've used the source-code from github (https://github.com/liferay/liferay-portal/blob/master/modules/apps/portal-search-elasticsearch6/portal-search-elasticsearch6-impl/build.gradle) and the bnd.bnd file, to check what's imported.
When activating the bundle, an exception is thrown:
The activate method has thrown an exception
java.util.ServiceConfigurationError: org.elasticsearch.common.xcontent.XContentBuilderExtension: Provider org.elasticsearch.common.xcontent.XContentElasticsearchExtension not a subtype
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.elasticsearch.common.xcontent.XContentBuilder.<clinit>(XContentBuilder.java:118)
at org.elasticsearch.common.settings.Setting.arrayToParsableString(Setting.java:1257)
The XContentBuilderExtension is from the elasticsearch-x-content-6.5.0.jar,
the XContentElasticsearchExtension class, is included in the elasticsearch-6.5.0.jar. Both are Included Resources, and have been put on the classpath.
The Activate-method initializes a TransportClient in my other jar, hence it happens on activation ;).
Edit:
I've noticed that this error does NOT occur when installing this the first time, or when the portal restarts. So it only occurs when I uninstall and reinstall the bundle. (This is functionality I really prefer to have!). Maybe a stupid thought.. But could it be that there is some 'hanging thread'? That the bundle is not correctly installed, or that the TransportClient still is alive? I'm checking this out. Any hints are welcome!
Edit 2:
I'm fearing this is an incompatibility between SPI and OSGi? I've checked: The High Level Rest Client has the same issue. (But then with another Extension). I'm going to try the Low-Level Rest Client. This should work, as there are minimal dependencies, I'm guessing. I'm still very curious on why the incompatibility is there. I'm certainly no expert on OSGi, neither on SPI. (Time to learn new stuff!)
Seems like a case where OSGi uses your bundle to solve a dependency from another bundle, probably one that used your bundle to solve a package when the system started.
Looking at the symptoms: it does not occur when booting or restarts. Also it is not a subtype.
When OSGi uses that bundle to solve a dependency, it will keep a copy around, even when you remove it. When the bundle comes back a package that was previously used by another bundle may still be around and you can have the situation where a class used has two version of itself, from different classloaders, meaning they are not the same class and therefore, not a subtype.
Expose only the necessary to minimize the effects of this. Import only if needs importing. If you are using Liferay Gradle configuration to include the bundle inside, stop - it's a terrible way to include as it exposes a lot. If using the bnd file to include a resource and create an entry for the adicional classpath location, do not expose if not necessary. If you have several bundles using one as dependency, make sure about the version they use and if the exchange objects from the problematic class, if they do, than extra care is required.
PS: you can include attributes when exporting and/or importing in order to be more specific and avoid using packages from the wrong origin.
You can have 2 elastic search connections inside one Java app and Liferay is by default not exposing the connection that it holds.
A way around it is to rebuild the Liferay ES connector. It's not a big deal because you don't need to change the code only the OSGi descriptor to expose more services.
I did it in one POC project and worked fine. The tricky thing is to rebuild the Liferay jar but that was explained by Pettry by his google like search blog posts. https://community.liferay.com/blogs/-/blogs/creating-a-google-like-search (it is a series but it's kind of hard to navigate in the new Liferay blogs but Google will probably help) Either way it is all nicely documented here https://github.com/peerkar/liferay-gsearch
the only thing then what needs to be done is to add org.elasticsearch.* in the bnd.bnd file in the export section. You will then be able to work with the native elastic API.
Can anyone explain how Spring decides where to look for resources when one uses the ResourceLoader.getResource(...) method?
I am having a problem with a multi-module maven application built using Spring Boot whereby in my integration tests my code is able to find resources using resourceLoader.getResource("templates/") or even resourceLoader.getResource("classpath:templates/"). So far so good...
However, when the module is eventually packaged into the executable JAR and run with embedded Tomcat the resources can no longer be resolved. I also tried resourceLoader.getResource("classpath*:templates/") with no success.
What I find concerning is that when I add a logging statement to output the URL being used in the search i get a path to one of the other modules in the project (not the one that actually contains the resource in question). E.g: jar:file:/Users/david/exmaple/target/spring-boot-0.0.1-SNAPSHOT.jar!/lib/module1-0.0.1-SNAPSHOT.jar!/templates/ whereas I believe the resource is in jar:file:/Users/david/exmaple/target/spring-boot-0.0.1-SNAPSHOT.jar!/lib/module2-0.0.1-SNAPSHOT.jar!/templates/
The resource loader was obtained from an Autowired constructor param.
Thanks in advance for any hints.
Edit
Just in case it isn't clear or is of importance, my integration tests for the module in question aren't aware of the other module. I have module1, module2 and a spring-boot module which has dependencies on module1 & module2. Essentially, when I run the integration tests for module 2 the classpath isn't aware of module1 - so I suspect that this has something to do with why it works in the tests.
When you use classpath: or classpath*: prefix, internally, this essentially happens via a ClassLoader.getResources(…) call in spring.
The wildcard classpath relies on the getResources() method of the underlying classloader. As most application servers nowadays supply their own classloader implementation, the behavior might differ especially when dealing with jar files. A simple test to check if classpath* works is to use the classloader to load a file from within a jar on the classpath: getClass().getClassLoader().getResources("<someFileInsideTheJar>"). Try this test with files that have the same name but are placed inside two different locations. In case an inappropriate result is returned, check the application server documentation for settings that might affect the classloader behavior.
Do not use classpath: form as you have multiple classloader locations of templates/ .
Refer to: resources-classpath-wildcards
I'm having a problem I simply can't get my head around.
I'm creating a jsf application where I (as administrator) can upload a jar file, and that way around update the application.
Through this tutorial: http://docs.oracle.com/javase/tutorial/deployment/jar/jarclassloader.html I have managed to classload the jar file. The first issue is that I can only class load as "Object" and not cast to an other class type. I get a ClassCastException.
JarClassLoader jcl=new JarClassLoader(url);
Class cl= jcl.retreiveClass(jcl.getMainClassName());
Object ob=cl.newInstance(); //I would like to make this a RouteBuilder object instead of Object
Concretely I'm using Apache Camel, where I can add routes to an existing "core" (CamelContext). What's happening is that I have the core of apache camel running in my web app, and I can then add routes runtime. It's a route I want to package as a jar and load into the app runtime (I want to develop and test the route in my local environment, and then afterwords upload it to the production application). A concrete route is just a simple java class that extends RouteBuilder. I want to upload the jar, classLoad (URLClassLoader) the main class (maybe the whole jar? - does that make sense?), and convert it to a RouteBuilder. This seems to be impossible. I have chosen to upload the jar file to a folder outside my war, so that the routes do not get erased when I restart the webapp (is this smart or should this be done in an other way?). Is that a problem regarding name spaces? Furthermore, I haven't been able to find out whether I need to traverse my entire jar file and classload ever single class inside. Any comments on that?
To me it seems like I have some fundamental misconceptions on how extending / updating a java application is done. I simply can't find any good examples/explanations on how to solve my problem and therefore I conclude that I don't get this on a conceptual level.
Could someone elaborate on how to extend a running jsf application (I guess the same issues are relevant in native java apps), explain how to upload and class load a jar at runtime, and how to cast loaded classes to other class types than Object?
Thanks
Lasse
If you are interested in loading Routes without stopping your application you could consider using an OSGi container like Karaf. Karaf provides support for Apache Camel routes out-of-the-box: http://camel.apache.org/karaf.html
All class loading is managed by the OSGi container and you just need to run some commands to update things. I am not sure if this could work with your JSF application but it worths to take a look.
I am working on a custom session manager for Tomcat and I got it working with one caveat, I have to put the jar(and all the associated jars) in the CATALINA_HOME/lib directory or I get a noclassdef found exception, even though the classes are in the WEB-INF/lib directory.
Is there any way to force tomcat to look in the web apps lib directory when it is loading a session manager? Putting it in the lib directory is error-prone(as you have to remember to copy the jar over when upgrading and/or installing on a new system) and makes development of the jar more difficult.
As of Tomcat 7.0.27, the <Manager> element in your META-INF/context.xml file is processed by the commons-digester and the effective ClassLoader is set to that which loaded the ContextConfig class -- one which firmly resides in Tomcat's server ClassLoader, which is outside of the ClassLoading chain of anything with access to the webapp's classes.
So, without modifying the Tomcat source, there will be no way to load your Manager from within your webapp. Consider joining the Tomcat users' mailing list to discuss such a feature.
EDIT 2015-067-01
If you want to specify a ClassLoader for your web application, you'll need to use the <Loader> element within your <Context>: http://tomcat.apache.org/tomcat-8.0-doc/config/loader.html