My environment is ubuntu12.04+eclipse3.3.0+hadoop0.20.2
When I test on the System.serProperty, which would change the configuration defined in xml file. But I don't get the same effect when I test it. Here is my code snippet:
//cofiguration class test
public static void test() {
Configuration conf = new Configuration();
conf.addResource("raw/conf-1.xml");
System.out.println(conf.get("z"));
System.setProperty("z", "SystemProp_mz");
System.out.println(conf.get("z"));
}
conf-1.xml is as follows:
<configuration>
<property>
<name>z</name>
<value>mz</value>
</property>
</configuration>
the output is:
mz
mz
Could anyone give me some help? Thanks a lot!
The Configuration object isn't linked to the System properties - if you want to change the value of z in the configuration, then use conf.set('z', 'SystemProp_mz') instead of System.setProperty(..)
Update
The Configuration object can use variable expansion as outlined in http://hadoop.apache.org/docs/current/api/org/apache/hadoop/conf/Configuration.html, but this requires you to have defined an entry as follows:
<configuration>
<property>
<name>z</name>
<value>${z}</value>
</property>
</configuration>
If you have don't have the above entry, then just calling conf.get("z") will not resolve to system properties. The following unit test block demonstrates this:
#Test
public void testConfSystemProps() {
System.setProperty("sysProp", "value");
Configuration conf = new Configuration();
conf.set("prop", "${sysProp}");
Assert.assertNull(conf.get("sysProp"));
Assert.assertEquals(conf.get("prop"), "value");
}
Related
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<id>build-info</id>
<goals>
<goal>build-info</goal>
</goals>
</execution>
</executions>
</plugin>
this the main class
public class GetSpaOsmiumVersionClient implements CommandLineRunner{
#Autowired
BuildProperties buildProperties;
public static void main( String[] args ){
SpringApplication app = new SpringApplication(GetSpaOsmiumVersionClient.class);
app.setBannerMode(Banner.Mode.OFF);
app.run(args);
}
#Override
public void run(String... args) throws Exception {
Options options = new Options();
options.addOption("h", "help", false, "prints the help content");
options.addOption("v", "version", false, "version spa osmium");
try{
//Etape 2: Analyse de la ligne de commande
CommandLineParser parser = new DefaultParser();
CommandLine commandLine = parser.parse(options, args);
if(commandLine.hasOption("v")){
buildProperties.getVersion();
}else {
HelpFormatter formatter = new HelpFormatter();
formatter.printHelp( "App" , options );
System.exit(1);
}
Found this question in Google Search for a similar issue, so I'll leave this answer so other fellow developers can save some time.
The error message in the question title
Unsatisfied dependency expressed through field 'buildProperties'
is often caused by a misunderstanding on how the property BuildProperties works.
Basically, BuildProperties only works if we have executed the 'build-info' goal of Maven Spring Boot Plugin (aka, run the following command in cmd):
mvn spring-boot:build-info
The reason is that BuildProperties is NOT something built-in, but a product of that Maven goal. When that goal is executed, a file build-info.properties is generated - and the code will read from that file.
Usually Maven project is set up so that it will automatically execute that goal on pipeline (in plugins part, see picture below). However when we trigger a run on local IDE, that goal isn't automatically executed, hence the problem.
Detailed explanation on how it works can be found in this reference:
https://www.vojtechruzicka.com/spring-boot-version/
If you prefer to work with IDE (eg: IntelliJ) instead of command line, you might find Maven tool window. Your job is to "run" the "build-info" goal before starting the server.
Example for IntelliJ: https://www.jetbrains.com/help/idea/work-with-maven-goals.html
I have a MR program which runs perfectly on a bunch of SequenceFile's and output is as expected.
When I try to achieve the same via an Oozie WorkFlow for some reason the InputFormat class property is not recognized and I feel the input is considered as default TextInputFormat only.
Here is how the mapper is declared. SequenceFile key is LongWritable and value is Text.
public static class FeederCounterMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
// setup map function for stripping the feeder for a zone from the input
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
final int count = 1;
// convert input rec to string
String inRec = value.toString();
System.out.println("Feeder:" + inRec);
// strip out the feeder from record
String feeder = inRec.substring(3, 7);
// write the key+value as map output
context.write(new Text(feeder), new IntWritable(count));
}
}
The workflow layout for my application is as below
/{$namenode}/workflow.xml
/{$namenode}/lib/FeederCounterDriver.jar
The below is my workflow.xml. The $namenode, $jobtracker, $outputdir, $inputdir are defined in the job.properties file.
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>/flume/events/sincal*</value>
</property>
<property>
<name>mapred.output.dir</name>
<value>${outputDir}</value>
</property>
<property>
<name>mapred.input.format.class</name>
<value>org.apache.hadoop.mapred.SequenceFileInputFormat</value>
</property>
<property>
<name>mapred.output.format.class</name>
<value>org.apache.hadoop.mapred.TextOutputFormat</value>
</property>
<property>
<name>mapred.input.key.class</name>
<value>org.apache.hadoop.io.LongWritable</value>
</property>
<property>
<name>mapred.input.value.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapred.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapred.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.map.class</name>
<value>org.poc.hadoop121.gissincal.FeederCounterDriver$FeederCounterMapper</value>
</property>
<property>
<name>mapreduce.reduce.class</name>
<value>org.poc.hadoop121.gissincal.FeederCounterDriver$FeederCounterReducer</value>
</property>
<property>
<name>mapreduce.map.tasks</name>
<value>1</value>
</property>
</configuration>
</map-reduce>
A snippet of the stout(first 2 lines) when I run the MR job is
Feeder:00107371PA1700TEET67576 LKHS 5666LH 2.....
Feeder:00107231PA1300TXDS 8731TX 1FSHS 8731FH 1.....
A snippet of the output(first 3 lines) when I run using Ooozie work flow is
Feeder:SEQ!org.apache.hadoop.io.LongWritableorg.apache.hadoop.io.Text�������b'b��X�...
Feeder:��00105271PA1000FSHS 2255FH 1TXDS 2255TX 1.....
Feeder:��00103171PA1800LKHS 3192LH 2LKHS 2335LH 1.....
With the above output from the Oozie workflow I highly doubt the input format SequenceFileInputFormat mentioned in the workflow.xml is even considered, else I feel this is overridden.
Any inputs towards this would help. Thanks
Find the job.xml created for this mapreduce job in job tracker and see what is the input format class being set there. This will confirm whether it is a problem with input format or not.
I had a really similar problem and I got oozie to use the proper input format by setting my property like this
<property>
<name>mapreduce.inputformat.class</name>
<value>org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat</value>
</property>
So one dot to remove from the property name (check for your version) and the class to change too.
I use Spring with Mybatis. I have it configured to scan for mappers in my whole project and I assumed it determined a mapper because it found an XML file which has reference to a java interface.
But this is proven incorrect today because I had to add a new interface which is not a mapper class and Mybatis thinks it is, so it is causing problems in my app due to this error:
Mapped Statements collection does not contain value for com.blah.MyInterface.someMethod
com.blah.MyInterface is just a simple interface which I needed to be included in Spring context so I gave it the #Component tag. Is that the wrong tag to use? Is that where the confusion comes from?
I just needed to create this interface so that I can have a proxy wrap my database calls in one place where I can put a #Transactional tag, since Spring ignores it when it is in my Controller method.
Sample code
package com.blah.something;
#Component public interface MyInterface {
public void someMethod( SomeObject obj) throws Exception;
}
package com.blah.something;
public class MyImplementation implements MyInterface {
#Transactional
public void someMethod( SomeObject obj) throws Exception {
... do a whole bunch of stuff
}
}
I dont want this included in the MyBatis mappers!
Edit: added the mybatis config xml as requested:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration
PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
<settings>
<setting name="lazyLoadingEnabled" value="false" />
<setting name="defaultStatementTimeout" value="60"/>
</settings>
<typeAliases>
<typeAlias alias="StripTrailingZerosBigDecimalTypeHandler" type="com.blah.typehandlers.StripTrailingZerosBigDecimalTypeHandler"/>
</typeAliases>
</configuration>
This is the part of my spring xml config which calls the mybatis mapper scanner:
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
<property name="basePackage" value="com.blah" />
</bean>
So I set it to scan the whole project which includes my interface above but I can't imagine it just grabs every single interface and considers them all mappers!
In my debug log I see mybatis picking up my interface:
12/9/13 11:18:44 904 [org.mybatis.spring.mapper.MapperScannerConfigurer$Scanner.findCandidateComponents:4125] - Scanning file [D:\Weblogic\wls11\domains\ldapdomain\autodeploy\default\WEB-INF\classes\com\blah\MyInterface.class]
12/9/13 11:18:44 904 [org.mybatis.spring.mapper.MapperScannerConfigurer$Scanner.findCandidateComponents:4125] - Identified candidate component class: file [D:\Weblogic\wls11\domains\ldapdomain\autodeploy\default\WEB-INF\classes\com\blah\MyInterface.class]
12/9/13 11:18:44 904 [org.mybatis.spring.mapper.MapperScannerConfigurer$Scanner.findCandidateComponents:4125] - Scanning file [D:\Weblogic\wls11\domains\ldapdomain\autodeploy\default\WEB-INF\classes\com\blah\MyImplementation .class]
12/9/13 11:18:44 904 [org.mybatis.spring.mapper.MapperScannerConfigurer$Scanner.findCandidateComponents:4125] - Ignored because not a concrete top-level class: file [D:\Weblogic\wls11\domains\ldapdomain\autodeploy\default\WEB-INF\classes\com\blah\MyImplementation .class]
There is no XML for this interface, there is no mapper namespace for it, it's just a plain old regular interface and MyBatis should not be thinking it is a mapper service
Ok it looks like MyBAtis scanner does indeed take every interface, it does not have any "smarts" in it to identify mapper interfaces as I thought it would - based on finding matching XML or namespaces. I had to add a filter to the mapper configuration and then introduce a new annotation to annotate my mapper interfaces.
Issue: Spring Component Annotation scan not picking up the class annotated in the external jar which is not included in pom.xml. But i need to scan for classes with specific annotation from external jars. These external jars will be placed in the classpath but will not be known to my application during compile time.
1) We have a maven module(artifactId="metric_processor") which produces a jar file(metric_processor.jar) and has following classes
package com.metric;
#Target({ElementType.TYPE})
#Retention(RetentionPolicy.RUNTIME)
public #interface ProcessMetric {
String name();
}
package com.metric;
public interface MetricProcessor {
int computeMetric();
}
package com.metric;
#ProcessMetric(name="LATENCY")
#Component
public class LatencyMetricProcessor implements MetricProcessor {
.....
}
2) We have another maven module ("artifactId="metric_processor_external") which produces a jar(metric_processor_external.jar) and includes "metric_processor" module as compile time scope.
package com.metric;
#ProcessMetric(name="TEST_METRIC_EXTERNAL")
#Component
public class TestMetricProcessor implements MetricProcessor {
....
}
3) We have a third(main) maven module(artifactId="main_application") which is a stand alone application(uses spring) which includes module "metric_processor" in compile scope. (But does not include "metric_processor_external"). The build plugin for the third module is
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.main.TriggerMetricProcessor</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Application context xml for this module is
<beans>
<context:component-scan base-package="com.metric">
<context:include-filter type="annotation" expression="com.metric.ProcessMetric" />
</context:component-scan>
<bean id="triggerMetricProcessor" class="com.main.TriggerMetricProcessor" />
</beans>
I have the following class which is the starting point of the application
package com.main;
import ...
public class TriggerMetricProcessor {
public static void main(String[] args) throws Exception {
ApplicationContext context =
new ClassPathXmlApplicationContext("application-context.xml");
TriggerMetricProcessor triggerMetricProcessor = (TriggerMetricProcessor) context.getBean("triggerMetricProcessor");
triggerMetricProcessor.initMetricProcessor(context);
}
private void initMetricProcessor(ApplicationContext context) {
GenericBeanFactoryAccessor beanFactoryAccessor = new GenericBeanFactoryAccessor(context);
final Map<String, Object> metricProcessors = beanFactoryAccessor.getBeansWithAnnotation(ProcessMetric.class);
for (final Object metricProcessor : metricProcessors.values()) {
final Class<? extends MetricProcessor> metricProcessorClass = (Class<? extends MetricProcessor>)metricProcessor.getClass();
final ProcessMetric annotation = metricProcessorClass.getAnnotation(ProcessMetric.class);
System.out.println("Found MetricProcessor class: " + metricProcessorClass + ", with name: " + annotation.name());
}
}
}
we compile the third module as
maven clean install assembly:single
This produces the jar file "main_application-with-dependencies.jar"
Then we run its as
java -cp "metric_process_external.jar" -jar main_application-with-dependencies.jar
Now the application finds only "LatencyMetricProcessor" and does not find the "TestMetricProcessor".
Can someone please help?
When you use the -jar option to execute a jar file, the -cp option is ignored.
The Oracle Java docs for the -jar option say:
-jar
Execute a program encapsulated in a JAR file. The first argument is
the name of a JAR file instead of a startup class name. In order for
this option to work, the manifest of the JAR file must contain a line
of the form Main-Class: classname. Here, classname identifies the
class having the public static void main(String[] args) method that
serves as your application's starting point. See the Jar tool
reference page and the Jar trail of the Java Tutorial for information
about working with Jar files and Jar-file manifests.
When you use this option, the JAR file is the source of all user
classes, and other user class path settings are ignored.
Also check out this post: stackoverflow.com/questions/5879925/in-linux-how-to-execute-java-jar-file-with-external-jar-files
So you'll need to specify the metric_process_external.jar in your manifest file using a Class-Path: header. You should be able to get your Maven assembly plugin to do that.
If that's not practical, you'll need to run your application without the -jar flag:
java -cp "metric_process_external.jar:main_application-with-dependencies.jar" com.main.TriggerMetricProcessor
I'm trying to figure out how to get the values of a properties file into my Spring Environment properties.
The pre Spring 3.1 way of doing this would be something like:
<context:property-placeholder location="classpath:my.properties" />
<bean id="myBean" class="com.whatever.MyBean">
<property name="someValue" value="${myProps.value}" />
<!-- etc -->
</bean>
I could have also done this:
public class MyBean {
#Value(value = "#{myProps.value}")
private String someValue;
}
Now that I can ostensibly pull properties from the Environment class, this seems like a much cleaner way of getting properties than using the clunky #{myProps.value} syntax in either my xml or my bean itself.
I tried this in my XML:
<bean
class="org.springframework.context.support.PropertySourcesPlaceholderConfigurer">
<property name="location">
<value>classpath:my.properties</value>
</property>
</bean>
But the properties are not added to Environment.
I understand that I can use the PropertySource attribute, but I'm not doing my configuration with annotations.
So how can I set up my bean / xml so that variables setup in my props are available in Environment? Moreover, how can I inject those values into my bean without having to explicitly say "environmentInstance.getProperty("myProps.value")?
For the web applications
If you want to have Environment populated before the XML bean definitions are processed, you should implement a ApplicationContextInitializer(), as described in the Spring Source blog
<context-param>
<param-name>contextInitializerClasses</param-name>
<param-value>com.bank.MyInitializer</param-value>
</context-param>
Here is a slightly modified version of the example from the blog
public class MyInitializer implements
ApplicationContextInitializer<ConfigurableWebApplicationContext> {
public void initialize(ConfigurableWebApplicationContext ctx) {
ResourcePropertySource ps;
try {
ps = new ResourcePropertySource(new ClassPathResource(
"my.properties"));
} catch (IOException e) {
throw new AssertionError("Resources for my.properties not found.");
}
ctx.getEnvironment().getPropertySources().addFirst(ps);
}
}
For the standalone applications
Basically the same thing, you can modify the environment of the AbstractApplicationContext directly
ResourcePropertySource ps;
try {
ps = new ResourcePropertySource(new ClassPathResource(
"my.properties"));
}catch (IOException e) {
throw new AssertionError("Resources for my.properties not found.");
}
//assuming that ctx is an AbstractApplicationContext
ctx.getEnvironment().getPropertySources().addFirst(ps);
P.S. the earlier version of this answer showed an attempt to modify the Environment from the bean but it seems to be an antipattern spreading on SO, because for sure you would like to have Environment property sources list populated even before the XmlBeanDefinitionReader starts to process XML to make placeholders work inside the <import/> statement.
My understanding is also a little addled but this is what I could make out:
Environment is a way to indicate the currently active profiles (and profiles are a way to selectively create beans)
PropertySources can be associated to an environment and the way to do that is using #PropertySource annotation, there does not seem to be an xml equivalent to do this.
Your understanding actually should be the other way round, AFAIK: when you declare <context:property-placeholder the placeholder will resolve properties against the locally declared properties and can fallback on the property sources declared in the environment, environment itself does not get modified with the new properties. Again,the only way to add properties to the environment itself seems to be through the #PropertySource annotation