Heritrix 3.2.0: Writing and Adding Extensions - spring

I am currently working with Heritrix and I have a standard installation (this one: http://builds.archive.org/maven2/org/archive/heritrix/heritrix/3.2.0/) and it works fine.
But now I want to write and add my own extensions e.g. change the priority of urls which should be crawled or just a simple extractor. I can inspect the java code of an existing extractor but how can I add it to the crawler?
I tried to export my java test project to a jar file and put this file in the lib folder of Heritrix (where the other libraries are). Furthermore I added a bean to my job's cxml file.
But after starting I got this error: 2014-11-07T19:51:40.296Z SEVERE Could not instantiate bean class [myModule.TestClass]: No default constructor found; nested exception is java.lang.NoSuchMethodException: myModule.TestClass.(); Can't create bean 'myModule.TestClass#0'
It is just the extractorHTML renamed and in a new project and exported to a jar file.
Any idea what is wrong? I read all the documentations but there are only explanations how to write extensions and not how to add it?
Greetings and thank you :-)

I think the issue is the class loader requires a default constructor (A constructor - which takes no arguments) Add a default constructor
public YourClass() { }
and the required getters and setters for setting the member variables.

Related

spring boot app cannot load bundle properties files

I am building an app that mostly provide REST services, nothing fancy. since my data consumed by the app can have multiple languages I thought about using the bundle files.
I created 3 files, one with the default file name and another two with specific languages. The files created using intellij IDE I am using.
I followed this guide https://www.baeldung.com/java-resourcebundle however on each run I am getting:
MissingResourceException: Can't find bundle for base name tp_app_strings, locale en_US
I tried numerous articles but none of them seems to resolve the issue.
One fun fact is that if I am using the #Value("classpath:tp_app_strings.properties") on a 'Resource' field I am able to get a reference to that file, so it spring is able to find it.
Additional thing that I tried was to create a WEB-INF directory and place the files there (read it in some article) but still no positive affect
The project structure is quite straight forward:
Spring boot version 2.2 running tomcat.
Any suggeestions would be highly appriciated
You can load the .properties file to the application context using #PropertySource annotation instead using #Value to load the .properties file to a org.springframework.core.io.Resource instance.
The usage;
#Configuration
#PropertySource("classpath:tp_app_strings.properties")
public class DefaultProperties {
#Value("${property1.name}") // Access properties in the above file here using SpringEL.
private String prop1;
#Value("${property2.name}")
private String prop2;
}
You wouldn't need java.util.ResourceBundle access properties this way. Use different or same class to load other .properties files as well.
Update 1:
In order to have the functionality of java.util.ResourceBundle, you can't just use org.springframework.core.io.Resource class. This class or non of it sub-classes don't provide functions to access properties by its name java.util.ResourceBundle whatsoever.
However, if you want a functionality like java.util.ResourceBundle, you could implement something custom like this using org.springframework.core.io.Resource;
#Configuration
public class PropertyConfig {
#Value("classpath:tp_app_strings.properties")
private Resource defaultProperties;
#Bean("default-lang")
public java.util.Properties getDefaultProperties() throws IOException {
Properties props = new Properties();
props.load(defaultProperties.getInputStream());
return props;
}
}
Make sure to follow correct naming convention when define the property file as java.util.Properties#load(InputStream) expect that.
Now you can #Autowire and use this java.util.Properties bean wherever you want just like with java.util.ResourceBundle using java.util.Properties#getProperty(String) or its overloaded counterpart.
I think it's problem of you properties file naming convention. use underline "_" for specifying locale of file like
filename_[languageCode]_[regionCode]
[languageCode] and [regionCode] are two letters standard code that [regionCode] section is optional
about code abbrivation standard take a look on this question
in your case change file name to tp_app_strings_en_US.properties

Deploying BEAN in OSGi plugin

I am currently deploying my custom controls as OSGi plugins and I wanted to do the same thing with my beans. I have tried putting them into the OSGi plugin and it works fine but the only problem I have is the faces-config.
It seems it has to be called faces-config in the OSGi plugin to work but that means i can't use beans in the NSF anymore because it seems to ignore the local faces-config.
Is there a way to change the name of the faces-config in the OSGi plugin?
Something like FEATURE-faces-config.xml?
In the class in your plugin that extends AbstractXspLibrary, you can override "getFacesConfigFiles", which should return an array of strings representing paths within the plugin to additional files of any name to load as faces-config additions. For example:
#Override
public String[] getFacesConfigFiles() {
return new String[] {
"com/example/config/beans.xml"
};
}
Then you can put the config file in that path within your Java source folder (or another folder that is included in build.properties) and it will be loaded in addition to your app's normal faces-config, beans and all.
The NSFs are running as separate, distinct Java applications. The OSGi plugin is running in the OSGi layer, above all those distinct Java applications, as a single code base. Consequently, the faces-config is only at that level.
It's possible to load them dynamically, by using an ImplicitObjectFactory, loaded from an XspContributor. That's what is done in OpenNTF Domino API for e.g. userScope (which is a bean stored in applicationScope of an NSF). See org.openntf.domino.xsp.helpers.OpenntfDominoImplicitObjectFactory, which is referenced in OpenntfDominoXspContributor, loaded via the extension point of type "com.ibm.xsp.library.Contributor".
A few caveats:
You have no control over what happens if you try to register your bean with a name the developer also uses for a different variable in that scope.
Unless you add code to check if the library is enabled, as we do, you'll be adding the bean to every database on the server.
You still need to add the library to the NSF. Unless you also provide a component that those databases will all use, there's no way you can programmatically add it, as far as I know.
It might be easier to skip the bean approach and just add an instance of the Java class in beforePageLoad, page controller class, or however you're managing the backing to the relevant XPage (if viewScope) or application (if sessionScope / applicationScope).

Look up a dynamic property at run-time in Spring from PropertySourcesPlaceholderConfigurer?

Not sure of the best approach to this. We've created a jar that could be used by different projects. The other projects relying on this jar need to provide certain properties defined in one of their spring properties files. (Our jar shouldn't care what they name those property files.)
Using #Value("${some.prop}") works great for most properties, however we now have the requirement that the name of the property to look up is dynamic. For example:
int val = getSomeVal();
String propNeeded = foo.getProperty("foo."+val+".dynamic.prop");
Not sure what "foo" should be to get my access. I looked into injecting Environment, however from all my googling it looks like that will not load from an xml property-placeholder definition (even if defined as a bean def for PropertySourcesPlaceholderConfigurer.) You seem to have to use #PropertySource, yet my main config is an XML file so not sure how to get Environment to work. (I can't really go 'old skool' and look up the property file as a class path Resource either since I'm not aware of the name of the file the users defined.)
I don't mind making this particular Service class ApplicationContextAware, but if I did that how could I get access to the underlying PropertySourcesPlaceholderConfigurer ? which I would 'seem?' to need in order to get access to a property dynamically?
The other option is that I force users of the jar to declare a bean by a name that I can look up
<util:properties id="appProps" location="classpath:application.properties" />
And I then inject appProps as Properties and look up from there. I don't like this approach though since it forces the users of the library to name an file by a common id. I would think the best solution is to just get a handle in some way to the underlying PropertySourcesPlaceholderConfigurer in my service class... I'm just not sure how to do it?
Why doesn't Spring simply allow PropertySource to be defined some how via your XML config and then I could just inject Environment?
Thanks for any suggestions how to accomplish what I want.
You could have a ReloadableResourceBundleMessageSource declared to read from the same source as the PropertySourcesPlaceholderConfigurer. This way you could just #Autowire MessageSource (or make your bean implement MessageSourceAware) and use that to retrieve your properties.
Main reason for using ReloadableResourceBundleMessageSource is to retrieve I18N messages, so that would kind of hacky...

Inject constructor argument Spring Resource file with Grails/Groovy

From our Grails/Groovy app we need to use a service from a legacy Java service class, the constructor of which has a parameter of type org.springframework.core.io.Resource, e.g.
public ServiceClass(Resource someResource)
We need to inject an instance of the service class into a Groovy class of our app using Spring DSL, with the Resource referring to an XML file within our /src/main/resources. I tried to create the Spring config for this purpose, but so far I couldn't find a working solution. The relevant part of the config file looks like this
beans = {
xmlns aop:"http://www.springframework.org/schema/aop",
sec:"http://www.springframework.org/schema/security",
context:"http://www.springframework.org/schema/context"
serviceClass(com.somepackage.ServiceClass) {
//here we need to refer to the constructor arg XML file some way
}
}
I have tried multiple syntaxes found in various tutorials, e.g. closure for beanDefinition.constructorArgs, but unfortunately without success so far. Although neither the app compilation (grails:war) nor the startup (grails:run-app) indicates any problems with the bean wiring, when the app is actually loaded into the browser, we receive a NPE stating that our Groovy class into which the service class is injected, is a null object. So it seems that the bean wiring was not successful after all. Any help is appreciated
After fixing various issues with the project setup itself and multiple cleanups/recompiles, it seems that the following two approaches are both OK
serviceClass(com.somepackage.ServiceClass, '/WEB-INF/constructor-arg-xml-file.xml') {}
and
serviceClass(com.somepackage.ServiceClass) { bean ->
bean.constructorArgs = [
'/WEB-INF/constructor-arg-xml-file.xml'
]
}

Spring cannot load java based configuration

I want to create a Solr data import handler using Spring as the Ioc. When I try to invoke the handler from Solr, I got below error
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Failed to load bean class: com.yoox.shanghai.AppConfig; nested exception is java.io.FileNotFoundException: class path resource **[com/my/app/AppConfig.class]** cannot be opened because it does not exist
Note the path printed in the message. It looks like it is using a relative path. I have no idea what is the cause.
I am using the Java based container configuration, and the compilation passes. And my code works with JUnit4.
appCtx = new AnnotationConfigApplicationContext(AppConfig.class);
Are you sure that you import AppConfig properly? Otherwise make sure that the class has really been deployed (check the deployment directory).
I googled for a long time, but could not find any answer about how spring resolve the class path by default. But I found people are trying to explicitly set the class loader, so I tried.
appCtx = new AnnotationConfigApplicationContext();
appCtx.setClassLoader(this.getClass().getClassLoader());
appCtx.register(AppConfig.class);
appCtx.refresh();
And it works :D
However I am not satisfied with this answer. I hope some one can point out what's wrong with my class loading logic.

Resources