I have Map Reduce application and i want to use Spring Batch Yarn logic.
Like the one mentioned in below link.
https://spring.io/guides/gs/yarn-batch-restart/
But i want it to be really specific as normally we have in hadoop map reduce jobs.
Just looking for Spring Batch Yarn Class and configuration. Considering my hadoop map reduce logic is already inplace and working.
Thanks in Advance !!
Here is the ApplicationContext Configuration that u were looking for
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:batch="http://www.springframework.org/schema/batch"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-4.2.xsd">
<context:property-placeholder location="classpath:application.properties" />
<hdp:configuration namenode-principal="hdfs://xx.yy.com" rm-manager-uri="xx.yy.com"
security-method="kerb" user-keytab="location" rm-manager-principal="username"
user-principal="username">
fs.default.name=${fs.default.name}
mapred.job.tracker=${mapred.job.tracker}
</hdp:configuration>
<hdp:job id="wordCountJobId" input-path="${input.path}"
output-path="${output.path}" jar-by-class="com.xx.poc.Application"
mapper="com.xx.poc.Map" reducer="com.xx.poc.Reduce" />
<hdp:job-runner id="wordCountJobRunner" job-ref="wordCountJobId"
run-at-startup="true" />
</beans>
Related
I'm upgrading all our services to use the latest Fuse 7 / Karaf libs,
and I see that none of our Spring application contexts (<beans>...</beans> XML) are supported anymore. I need to switch everything to Blueprint's <blueprint>...</blueprint>.
For the most part, I can copy-paste. But I can't easily find the new Blueprint OSGi XSD/schema namespaces.
Is there a list somewhere, or an easier way than rummaging through github?
------------EDIT:
Here's an example.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
">
In <blueprint>, what would I change this to?
The Blueprint XML Schema is available at https://osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd.
I this XML Spring config using a variable from settings.xml
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">
<import resource="/spring-ds-${usingDb}.xml" />
...
</beans>
${usingDb} is from settings.xml maven file
I want to do the same in a Spring java config class
I think the only way for this is to set key in a properties files like this.
database.datasource=${key.datasource}
database.dialect=${key.database.dialect}
....
If any one has a better suggestion, you're welcome
When I validate aplicationContext.xml file I get this error:
Referenced file contains errors (project-aware://pm-master/org/springframework/beans/factory/xml/spring-tool-4.3.xsd).
and here is xml file:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:jpa="http://www.springframework.org/schema/data/jpa"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd
http://www.springframework.org/schema/data/jpa http://www.springframework.org/schema/data/jpa/spring-jpa.xsd">
<jpa:repositories base-package="com.jedlab.pm.dao" entity-manager-factory-ref="entityManagerFactory" transaction-manager-ref="transactionManager" />
</beans>
and spring data jpa version is 1.11.0.RELEASE.
When I remove the jpa:repositories line the error disappear.
How can I solve it?
If you are using eclipse then clearing cache from Windows-> Preferences -> Network Connections -> Cache will be helpful.
I know this has been asked many times before but my production tomcat server errors on the batch:job tag. The app runs fine in IntelliJ + the tomcat plugin. The proxy won't allow the server to download any of the schemas but that doesn't seem to affect the bean or tx tags
<tx:annotation-driven mode="aspectj" transaction-manager="transactionManager"/>
<batch:job id="createCsvForJob">
<batch:step id="createCsvFile">
<batch:tasklet ref="createCsvForTasklet" />
</batch:step>
</batch:job>
I can download the schemaLocation urls so I don't think there are any weird characters or typos in them. Is there anything obvious I'm missing?
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:task="http://www.springframework.org/schema/task"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
">
My fix was to include the version number in the schemaLocation, ie
http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
So I now have
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:task="http://www.springframework.org/schema/task"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
">
The reason was that because my server couldn't access the internet it needed to use the one that was included in the war file. To find the correct version number I command clicked the xmlns url in IntelliJ
xmlns:batch="http://www.springframework.org/schema/batch"
and just read the verison number.
Hope this helps someone else.
Eclipse/STS reports errors using Spring 3.2.1 with Spring-data-jpa together.
I have some XML configuration files with the following header:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:mvc="http://www.springframework.org/schema/mvc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:p="http://www.springframework.org/schema/p"
xmlns:jpa="http://www.springframework.org/schema/data/jpa"
xsi:schemaLocation="
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/mvc
http://www.springframework.org/schema/mvc/spring-mvc.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop.xsd
http://www.springframework.org/schema/data/jpa
http://www.springframework.org/schema/data/jpa/spring-jpa.xsd">
I am using Spring 3.2.1 with Spring-Data-JPA 1.3.0 and Eclipse/STS is reporting this:
Referenced file contains errors
(http://www.springframework.org/schema/data/jpa)
This happens in every single XML configuration file that contains the data-jpa schema.
When I remove JPA from the XML config, everything is just fine.
Is my configuration wrong or what happens here?
Thanks!
Paul
Today there is not a http://www.springframework.org/schema/data/jpa/spring-jpa.xml file so you have to use http://www.springframework.org/schema/data/jpa/spring-jpa-1.3.xsd, but this file use http://www.springframework.org/schema/data/repository/spring-repository.xsd that does not exist too so you have to add the location of this schema in your context file:
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/jdbc
http://www.springframework.org/schema/jdbc/spring-jdbc-3.2.xsd
http://www.springframework.org/schema/data/jpa
http://www.springframework.org/schema/data/jpa/spring-jpa-1.3.xsd
http://www.springframework.org/schema/data/repository
http://www.springframework.org/schema/data/repository/spring-repository-1.5.xsd
http://www.springframework.org/schema/jee
http://www.springframework.org/schema/jee/spring-jee-3.2.xsd"