JobInstanceAlreadyCompleteException when running a Batch-Task in Spring cloud Dataflow - spring

I have a Batch job working as a Task in spring cloud dataflow. When i try to execute the same task Definition a second time i get the exception:
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:793)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:335)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1234)
at com.tigerbooks.importer.ImportTask.main(ImportTask.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={-spring.cloud.task.executionid=3, -spring.datasource.username=dataflow_stage, -spring.datasource.url=jdbc:mysql://10.59.254.101:3306/dataflow_staging, -spring.datasource.driverClassName=org.mariadb.jdbc.Driver, -import.import-audio-content=false, -spring.datasource.password=bVs64CMlKvcTdkRLWL2zNPANYD3HMB, -import.syncfolder=/import-integrationtest, -spring.cloud.task.name=Integration-Test, -spring.profiles.active=ftp01,dev}. If you want to run this job again, change the parameters.
at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java
My Job configuration looks like this:
#EnableBatchProcessing
public class JobConfiguration {
#Bean
public Job importJob() {
return jobBuilderFactory.get("Import-Products").incrementer(new RunIdIncrementer())
.flow(step1())
.next(step2())
.end()
.build();
}
}
As far as i found out, a JobParametersIncrementor is supposed to fix this exception by adding a incrementing runId to the parameters. But in my case i can only run the job once on the server, then i have to clear the database (mysql) and the RunId is nowhere in the database.

As #Michael Minela already guessed my jobs where started with the wrong docker image-version. So the RunIdIncrementer was not present and it did not work.
After registering a new app with the correct version everything worked. Thanks.

Related

Springboot Application fails after database restart

We have developed a Springboot Java application and using 1.5.6 version. Using spring data jpa and provided custom implementation as well.
If I restart DB and execute some operation then it throws an below error for the services which are annotated with #Transactional annotation. Though, its working well for other DAO repository call which are not annotated with this annotation. Also, if I restart my spring boot application then it works fine.
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:133)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy155.findOne(Unknown Source)
at au.com.outware.swepad.tasks.WorkOrderManagerTask.taskFetchAndProcessWorkOrderDetails(WorkOrderManagerTask.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.persistence.PersistenceException: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
... 30 common frames omitted
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:206)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:724)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.setAutoCommit(SQLServerConnection.java:2615)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:79)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)
at com.sun.proxy.$Proxy102.setAutoCommit(Unknown Source)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
... 32 common frames omitted
2018-10-29 16:00:08,995 WARN pool-30-thread-1 org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
2018-10-29 16:00:08,996 ERROR pool-30-thread-1 org.hibernate.engine.jdbc.spi.SqlExceptionHelper - The connection is closed.
2018-10-29 16:30:00,017 INFO pool-13-threa
Herewith sharing the method which causing this issue:
#Transactional(propagation = Propagation.REQUIRES_NEW, readOnly = true, isolation = Isolation.READ_UNCOMMITTED, transactionManager = AppConstants.CHILD_TRANSACTION_MANAGER)
public Map<String, Object> getParentIdFromChildsId(String childIds, int userId, int projectID) {
jdbcTemplate.query("EXEC getParentIdFromChildsId?,?,?", new Object[] { childIds, userId, projectID },
(rs, rowNum) -> {
List<String> list = new ArrayList<>();
list.add(0, rs.getString("parent_id"));
list.add(1, rs.getString("box_id"));
list.add(2, rs.getString("permission"));
return list;
});
}
Also we are Using org.apache.tomcat.jdbc.pool.DataSource connection pool.
Let me know if other details required. Please help me on this.
This is likely because all the connections in your connection pool are broken after restarting the database. I can't really explain why it doesn't happen with other methods, but maybe it's just a timing issue, where the connections get closed and reopened after they failed once.
If my suspicion is correct setting testOnBorrow to true should solve the issue. This answer describes how to do that in Spring Boot:
spring.datasource.tomcat.testOnBorrow=true
spring.datasource.tomcat.validationQuery=SELECT 1

Error when create DataSource

I get a spring boot application and trying create a DataSource bean defined by Java configuration:
#Bean(name = "dsBD") #Primary
public DataSource dsBD() {
return DataSourceBuilder.create()
.url("jdbc:mysql://localhost:3306/‌​database?autoReconne‌​ct=true")
.username("‌​root")
.password("mysql")
.driverClassName("com.mysql.jdbc.Drive‌​r")
.build();
}
#Bean(name = "jdbcBD") #Autowired
public JdbcTemplate jdbcBD(#Qualifier("dsBD") DataSource dsBD) {
return new JdbcTemplate(dsBD);
}
But when starting application, I got the following error:
Caused by: java.lang.IllegalStateException: No supported DataSource type found
at org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder.getType(DataSourceBuilder.java:138) ~[spring-boot-autoconfigure-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder.build(DataSourceBuilder.java:69) ~[spring-boot-autoconfigure-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at com.package.DatabaseConfig.dsBD(DatabaseConfig.java:29) ~[classes/:na]
at com.package.DatabaseConfig$$EnhancerBySpringCGLIB$$ac56d75a.CGLIB$dsBD$0(<generated>) ~[classes/:na]
at com.package.DatabaseConfig$$EnhancerBySpringCGLIB$$ac56d75a$$FastClassBySpringCGLIB$$159befb3.invoke(<generated>) ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.package.DatabaseConfig$$EnhancerBySpringCGLIB$$ac56d75a.dsBD(<generated>) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) ~[spring-beans-4.3.9.RELEASE.jar:4.3.9.RELEASE]
... 46 common frames omitted
In terms to use DataSourceBuilder you need to have one of the following on your classpath:
COMMONS-DBCP
TOMCAT-JDBC
HIKARICP
Otherwise you will get an IllegalStateException for no supported type, as you get now.

Could not load TestContextBootstrapper - spring-boot integration testing error for non-web application

I am using spring-boot schedular based out of a gradle project. The scheduler runs fine when I run through the main class under src/main/java (either using debugger or by executing the spring-boot jar). The For integration testing - i created a new integrationTest directory parallel to java.
For integration test, the class that i created is as follows.
---------------
#RunWith(SpringJUnit4ClassRunner.class)
#SpringBootApplication
#ContextHierarchy({
#ContextConfiguration(name = "testContext", classes = ConfigCommons.class, loader = SpringApplicationContextLoader.class),
})
#TestPropertySource("classpath:integrationtest.properties")
//#IntegrationTest
public class IntegrationUseCase {
BeanA beana;
BeanB beanb;
#Before
public void setup() {
beana = new BeanA();
beanb = new BeanB();
}
#Test
public void testUseCase() {
// logic that uses beanA and beanB
}
}
---------------
Have following questions:
1. I have not leveraged autowired. Is there a way I can do that?
2. While buiding the project - Getting an error that TestContextBootstrapper cannot be loaded. I am using the following spring-boot packages-
"org.springframework.boot:spring-boot-starter-test:1.4.0.RELEASE". However, when I commented out this test class, the project buolds and runs fine. Does anyone know what is happening here?
The log is:
java.lang.IllegalStateException: Could not load TestContextBootstrapper [null]. Specify #BootstrapWith's 'value' attribute or make the default bootstrapper class available.
at org.springframework.test.context.BootstrapUtils.resolveTestContextBootstrapper(BootstrapUtils.java:143)
at org.springframework.test.context.TestContextManager.<init>(TestContextManager.java:105)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTestContextManager(SpringJUnit4ClassRunner.java:152)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.<init>(SpringJUnit4ClassRunner.java:143)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:88)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.springframework.core.annotation.AnnotatedElementUtils.findAllMergedAnnotations(Ljava/lang/reflect/AnnotatedElement;Ljava/lang/Class;)Ljava/util/Set;
at org.springframework.test.context.BootstrapUtils.resolveExplicitTestContextBootstrapper(BootstrapUtils.java:150)
at org.springframework.test.context.BootstrapUtils.resolveTestContextBootstrapper(BootstrapUtils.java:126)
... 39 more
I had a similar problem. There are problems with the dependencies in the pom.xml file. There could be many reasons for that: a conflict in versions, or conflict declaring the same dependency twice: in the current pom.xml and also in the parent pom.xml, like using different versions.

Issue when writing to elasticsearch using es-hadoop

Am getting this exception when I'm trying to write to Elasticsearch using mapreduce program with es-hadoop. Am trying to write to index=employee and type=basic which already exists in my Elasticsearch cluster.
My stack trace :-
Exception in thread "main"
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: No resource
['es.resource'] (index/query/location) specified at
org.elasticsearch.hadoop.util.Assert.hasText(Assert.java:30) at
org.elasticsearch.hadoop.mr.EsOutputFormat.init(EsOutputFormat.java:257)
at
org.elasticsearch.hadoop.mr.EsOutputFormat.checkOutputSpecs(EsOutputFormat.java:233)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at
com.mstack.mapreduce.DIGDriver.main(DIGDriver.java:22) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Driver class :-
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "es-hadoop");
job.setJarByClass(DIGDriver.class);
conf.set("es.nodes", "localhost:9200");
conf.set("es.port", "9200");
conf.set("es.resource", "employee/basic");
job.setNumReduceTasks(0);
job.setOutputFormatClass(EsOutputFormat.class);
job.setMapperClass(DIGMapper.class);
job.setMapOutputValueClass(MapWritable.class);
conf.setBoolean("mapreduce.map.speculative", false);
conf.setBoolean("mapreduce.reduce.speculative", false);
boolean status = job.waitForCompletion(true);
if (status) {
System.exit(0);
} else {
System.out.println("Job Failed : Some error!");
System.exit(1);
}
Resolved myself by changing the configs :-
conf.set("es.nodes", "localhost");
conf.set("es.port", "9200");

org.apache.hadoop.net.StandardSocketFactory not found

configuration = new Configuration();
configuration.set("fs.default.name",NAME_NODE_URL);
hdfs = FileSystem.get(configuration);
i am getting the below exception while using the code specified above,
java.lang.RuntimeException: Socket Factory class not found: java.lang.ClassNotFoundException: Class org.apache.hadoop.net.StandardSocketFactory not found
at org.apache.hadoop.net.NetUtils.getSocketFactoryFromProperty(NetUtils.java:142)
at org.apache.hadoop.net.NetUtils.getDefaultSocketFactory(NetUtils.java:122)
at org.apache.hadoop.net.NetUtils.getSocketFactory(NetUtils.java:100)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:477)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
at com.arista.cvp.commons.db.HdfsClient.copyfromLocaltoHdfs(HdfsClient.java:55)
at com.arista.cvp.services.hadoop.HDFSService.copyFromLocal(HDFSService.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
could anyone help in resolving the issue?
You definitely need either the hadoop-common-2.x jar on the classpath or the hadoop-core-1.x!

Resources