While running Kafka -> Apache Apex ->Hbase, it is saying following exception in Yarn tasks:
com.datatorrent.stram.StreamingAppMasterService: Application master, appId=4, clustertimestamp=1479188884109, attemptId=2
2016-11-15 11:59:51,068 INFO org.apache.hadoop.service.AbstractService: Service com.datatorrent.stram.StreamingAppMasterService failed in state INITED; cause: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:130)
at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:156)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:241)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:333)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:330)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:330)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:444)
And my DataTorrent log showing the following exception. I am running the app which communicates Kafka -> Apex -> Hbase streaming application.
Connecting to ResourceManager at hduser1/127.0.0.1:8032
16/11/15 17:47:38 WARN client.EventsAgent: Cannot read events for application_1479208737206_0008: java.io.FileNotFoundException: File does not exist: /user/hduser1/datatorrent/apps/application_1479208737206_0008/events/index.txt
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1893)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1834)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1814)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1786)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:552)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
Adding the code :
public void populateDAG(DAG dag, Configuration conf){
KafkaSinglePortInputOperator in
= dag.addOperator("kafkaIn", new KafkaSinglePortInputOperator());
in.setInitialOffset(AbstractKafkaInputOperator.InitialOffset.EARLIEST.name());
LineOutputOperator out = dag.addOperator("fileOut", new LineOutputOperator());
dag.addStream("data", in.outputPort, out.input);}
LineOutputOperator extends AbstractFileOutputOperator
private static final String NL = System.lineSeparator();
private static final Charset CS = StandardCharsets.UTF_8;
#NotNull
private String baseName;
#Override
public byte[] getBytesForTuple(byte[] t) {
String result = new String(t, CS) + NL;
return result.getBytes(CS);
}
#Override
protected String getFileName(byte[] tuple) {
return baseName;
}
public String getBaseName() { return baseName; }
public void setBaseName(String v) { baseName = v; }
How to resolve this problem?
Thanks.
Can you share some details about your environment like what version of hadoop and apex ? Also, which log does this exception appear in ?
Just as a simple sanity check, can you run the simple maven archetype generated application described at: http://docs.datatorrent.com/beginner/
If that works, try running the fileIO and kafka applications at:
https://github.com/DataTorrent/examples/tree/master/tutorials
If those work ok we can look at the details of your code.
I got the solution,
The problem related to expiry of my license, So reinstalled new one and works fine for actual code.
Related
HI I am using spring integration extensively in my project and in the current case dynamically creating my ftp, sftp adapters using spring dynamic flow registration. Also to provide session-factories I create them dynamically based on persisted configuration for each unique connection .
This works great but sometimes there are situations when I need to modify an existing session config dynamically and in this case I do require the session factory to refresh with a new session config . This can happen due to changing creds dynamically.
To do the same I am looking for two approches
remove the dynamic flows via flowcontext.remove(flowid). But this does not somehow kill the flow, I still see the old session factory and flow running.
If there is a way to associate a running adapter with a new Sessionfactory dynamically this would also work . But still have not find a way to accomplish this .
Please help
UPDATE
my dynamic registration code below
CachingSessionFactory<FTPFile> csf = cache.get(feed.getConnectionId());
IntegrationFlow flow = IntegrationFlows
.from(inboundAdapter(csf).preserveTimestamp(true)//
.remoteDirectory(feed.getRemoteDirectory())//
.regexFilter(feed.getRegexFilter())//
.deleteRemoteFiles(feed.getDeleteRemoteFiles())
.autoCreateLocalDirectory(feed.getAutoCreateLocalDirectory())
.localFilenameExpression(feed.getLocalFilenameExpression())//
.localFilter(localFileFilter)//
.localDirectory(new File(feed.getLocalDirectory())),
e -> e.id(inboundAdapter.get(feed.getId())).autoStartup(false)
.poller(Pollers//
.cron(feed.getPollingFreq())//
.maxMessagesPerPoll(1)//
.advice(retryAdvice)))
.enrichHeaders(s -> s.header(HEADER.feed.name(), feed))//
.filter(selector)//
.handle(fcHandler)//
.handle(fileValidationHandler)//
.channel(ftbSubscriber)//
.get();
this.flowContext.registration(flow).addBean(csf).//
id(inboundFlow.get(feed.getId())).//
autoStartup(false).register();
I am trying removing the same via
flowContext.remove(flowId);
on removing also the poller and adapter still look like they are active
java.lang.IllegalStateException: failed to create FTPClient
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:275)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:200)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:62)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:134)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:224)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.integration.handler.advice.AbstractRequestHandlerAdvice.invoke(AbstractRequestHandlerAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy188.call(Unknown Source)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:51)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:344)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
*POST Gary comments * changed the order of the chain and removing autostartup as defined in his example and now the polling adapter looks like getting removed .
changed order to match the one from Gary and remove autostartup from the flowcontext chain. Though looks like bug is still there if autstrtup is true .
this.flowContext.registration(flow).//
id(inboundFlow.get(feed.getId()))//
.addBean(sessionFactory.get(feed.getId()), csf)//
.register();
* researching more *
The standardIntegrationFlow.start does start all the components inside the flow irrespective of the autostartup status . I guess we do need to check the isAutostartup for these as well and only start them if autostartup is True when starting the IntegrationFlow. existing code below of standardIF . I there a way to override this or does this need a PR or fix .
if (!this.running) {
ListIterator<Object> iterator = this.integrationComponents.listIterator(this.integrationComponents.size());
this.lifecycles.clear();
while (iterator.hasPrevious()) {
Object component = iterator.previous();
if (component instanceof SmartLifecycle) {
this.lifecycles.add((SmartLifecycle) component);
((SmartLifecycle) component).start();
}
}
this.running = true;
}
remove() should shut everything down. If you are using CachingSessionFactory we need to destroy() it, so it closes the cached sessions.
The flow will automatically destroy() the bean if you add it to the registration (using addBean()).
If you can edit your question to show your dynamic registration code, I can take a look.
EDIT
Everything works fine for me...
#SpringBootApplication
public class So43916317Application implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(So43916317Application.class, args).close();
}
#Autowired
private IntegrationFlowContext context;
#Override
public void run(String... args) throws Exception {
CSF csf = new CSF(sf());
IntegrationFlow flow = IntegrationFlows.from(Ftp.inboundAdapter(csf)
.localDirectory(new File("/tmp/foo"))
.remoteDirectory("bar"), e -> e.poller(Pollers.fixedDelay(1_000)))
.handle(System.out::println)
.get();
this.context.registration(flow)
.id("foo")
.addBean(csf)
.register();
Thread.sleep(10_000);
System.out.println("removing flow");
this.context.remove("foo");
System.out.println("destroying csf");
csf.destroy();
Thread.sleep(10_000);
System.out.println("exiting");
Assert.state(csf.destroyCalled, "destroy not called");
}
#Bean
public DefaultFtpSessionFactory sf() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost("10.0.0.3");
sf.setUsername("ftptest");
sf.setPassword("ftptest");
return sf;
}
public static class CSF extends CachingSessionFactory<FTPFile> {
private boolean destroyCalled;
public CSF(SessionFactory<FTPFile> sessionFactory) {
super(sessionFactory);
}
#Override
public void destroy() {
this.destroyCalled = true;
super.destroy();
}
}
}
log...
16:15:38.898 [task-scheduler-5] DEBUG o.s.i.f.i.FtpInboundFileSynchronizer - 0 files transferred
16:15:38.898 [task-scheduler-5] DEBUG o.s.i.e.SourcePollingChannelAdapter - Received no Message during the poll, returning 'false'
16:15:39.900 [task-scheduler-3] DEBUG o.s.integration.util.SimplePool - Obtained org.springframework.integration.ftp.session.FtpSession#149a806 from pool.
16:15:39.903 [task-scheduler-3] DEBUG o.s.i.f.r.s.CachingSessionFactory - Releasing Session org.springframework.integration.ftp.session.FtpSession#149a806 back to the pool.
16:15:39.903 [task-scheduler-3] DEBUG o.s.integration.util.SimplePool - Releasing org.springframework.integration.ftp.session.FtpSession#149a806 back to the pool
16:15:39.903 [task-scheduler-3] DEBUG o.s.i.f.i.FtpInboundFileSynchronizer - 0 files transferred
16:15:39.903 [task-scheduler-3] DEBUG o.s.i.e.SourcePollingChannelAdapter - Received no Message during the poll, returning 'false'
removing flow
16:15:40.756 [main] INFO o.s.i.e.SourcePollingChannelAdapter - stopped org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0
16:15:40.757 [main] INFO o.s.i.channel.DirectChannel - Channel 'application.foo.channel#0' has 0 subscriber(s).
16:15:40.757 [main] INFO o.s.i.endpoint.EventDrivenConsumer - stopped org.springframework.integration.config.ConsumerEndpointFactoryBean#0
16:15:40.757 [main] DEBUG o.s.b.f.s.DefaultListableBeanFactory - Retrieved dependent beans for bean 'foo': [org.springframework.integration.ftp.inbound.FtpInboundFileSynchronizer#0, org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0, org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0.source, foo.channel#0, com.example.So43916317Application$$Lambda$12/962287291#0, org.springframework.integration.config.ConsumerEndpointFactoryBean#0, foocom.example.So43916317Application$CSF#0]
destroying csf
16:15:40.757 [main] DEBUG o.s.integration.util.SimplePool - Removing org.springframework.integration.ftp.session.FtpSession#149a806 from the pool
exiting
16:15:50.761 [main] TRACE o.s.c.a.AnnotationConfigApplicationContext - Publishing event in org.springframework.context.annotation.AnnotationConfigApplicationContext#27c86f2d: org.springframework.boot.context.event.ApplicationReadyEvent[source=org.springframework.boot.SpringApplication#5c18016b]
As you can see, the polling stops after the remove() and the session is closed by the destroy().
EDIT2
If you have auto start turned off you have to start via the registration...
IntegrationFlowRegistration registration = this.context.registration(flow)
.id("foo")
.addBean(csf)
.autoStartup(false)
.register();
...
registration.start();
I am trying to do map-reduce with kite-dataset api.
I have followed below urls to refer.
https://community.cloudera.com/t5/Kite-SDK-includes-Morphlines/Map-Reduce-with-Kite/td-p/22165
https://github.com/kite-sdk/kite/blob/master/kite-data/kite-data-mapreduce/src/test/java/org/kitesdk/data/mapreduce/TestMapReduce.java
My code snippet as below
public class MapReduce {
private static final String sourceDatasetURI = "dataset:hive:test_avro";
private static final String destinationDatasetURI = "dataset:hive:test_parquet";
private static class LineCountMapper
extends Mapper<GenericData.Record, Void, Text, IntWritable> {
#Override
protected void map(GenericData.Record record, Void value,
Context context)
throws IOException, InterruptedException {
System.out.println("Record is "+record);
context.write(new Text(record.get("index").toString()), new IntWritable(1));
}
}
private Job createJob() throws Exception {
System.out.println("Inside Create Job");
Job job = new Job();
job.setJarByClass(getClass());
Dataset<GenericData.Record> inputDataset = Datasets.load(sourceDatasetURI, GenericData.Record.class);
Dataset<GenericData.Record> outputDataset = Datasets.load(destinationDatasetURI, GenericData.Record.class);
DatasetKeyInputFormat.configure(job).readFrom(inputDataset).withType(GenericData.Record.class);
job.setMapperClass(LineCountMapper.class);
DatasetKeyOutputFormat.configure(job).writeTo(outputDataset).withType(GenericData.Record.class);
job.waitForCompletion(true);
return job;
}
public static void main(String[] args) throws Exception {
MapReduce httAvroToParquet = new MapReduce();
httAvroToParquet.createJob();
}
}
I am using HDP 2.3.2 box ,creating assembly jar and submitting my application through spark-submit.
I am getting below error when I submit my application.
2015-12-18 04:09:07,156 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2015-12-18 04:09:07,282 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2015-12-18 04:09:07,333 WARN [main] org.kitesdk.data.spi.Registration: Not loading URI patterns in org.kitesdk.data.spi.hive.Loader
2015-12-18 04:09:07,334 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.kitesdk.data.DatasetNotFoundException: Unknown dataset URI: hive://{}:9083/default/test_parquet. Check that JARs for hive datasets are on the classpath.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.kitesdk.data.DatasetNotFoundException: Unknown dataset URI: hive://{}:9083/default/test_parquet. Check that JARs for hive datasets are on the classpath.
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:478)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: org.kitesdk.data.DatasetNotFoundException: Unknown dataset URI: hive://{}:9083/default/test_parquet. Check that JARs for hive datasets are on the classpath.
at org.kitesdk.data.spi.Registration.lookupDatasetUri(Registration.java:109)
at org.kitesdk.data.Datasets.load(Datasets.java:103)
at org.kitesdk.data.Datasets.load(Datasets.java:165)
at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510)
at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476)
... 11 more
I am not getting what's wrong ? Is there any class-path problem ? If yes then where do I set it ?
You effectively have a classpath problem
Your project is missing org.kitesdk:kite-data-hive
You can
Add this jar to your fat jar before submitting it to Spark
Tells Spark to add it to your classpath when you submit
I have problem with calling remote ejb. I have successfully deployed remote EJB:
public interface IHelloWordlHome extends EJBHome {
mybeans.IHelloWordl create() throws RemoteException, javax.ejb.CreateException;
}
public interface IHelloWordl extends javax.ejb.EJBObject {
public String hello(String name) throws RemoteException;
}
#javax.ejb.Stateless(name = "HelloWordlEJB")
public class HelloWordlBean implements Serializable {
public HelloWordlBean() {
}
public String hello(String name) {
return "asd" + name;
}
public void ejbCreate() throws CreateException {
}
}
ejb-jar.xml:
<enterprise-beans>
<session>
<ejb-name>HelloWordlEJB</ejb-name>
<home>mybeans.IHelloWordlHome</home>
<remote>mybeans.IHelloWordl</remote>
<ejb-class>mybeans.HelloWordlBean</ejb-class>
<session-type>Stateless</session-type>
<transaction-type>Container</transaction-type>
</session>
</enterprise-beans>
and now I am trying to run standalone client. That means it's totaly different application which now runs on same machine as server (localhost) but later it will run on different machine. As the glassfish description (dont have the link atm) says I used InitialContext without parameters, in server log I found the JNDI name of my bean ("java:global/ear_ear_exploded/ejb/HelloWordlEJB!mybeans.HelloWordlBean") and trying to look it up. I use gl-client.jar lib and I have it on my classpath. Note that I didnt copy that .jar, I am using the .jar in glassfish installation folder (I know that could be problem becouse it links other .jars) I copied (ctrl+c & ctrl+v) the bean interface (IHelloWordl) from server to client.
client code:
public static void main(String[] args) throws NamingException, RemoteException {
IHelloWordl foo = (IHelloWordl) new InitialContext().lookup("java:global/ear_ear_exploded/ejb/HelloWordlEJB!mybeans.HelloWordlBean");
foo.hello("Martin");
}
This is what my IDE runs:
P:\Java\jdk1.8.0\bin\java -Didea.launcher.port=7534 "-Didea.launcher.bin.path=P:\IntelliJ IDEA 13.1.1\bin" -Dfile.encoding=UTF-8 -classpath "P:\Java\jdk1.8.0\jre\lib\charsets.jar;P:\Java\jdk1.8.0\jre\lib\deploy.jar;P:\Java\jdk1.8.0\jre\lib\javaws.jar;P:\Java\jdk1.8.0\jre\lib\jce.jar;P:\Java\jdk1.8.0\jre\lib\jfr.jar;P:\Java\jdk1.8.0\jre\lib\jfxswt.jar;P:\Java\jdk1.8.0\jre\lib\jsse.jar;P:\Java\jdk1.8.0\jre\lib\management-agent.jar;P:\Java\jdk1.8.0\jre\lib\plugin.jar;P:\Java\jdk1.8.0\jre\lib\resources.jar;P:\Java\jdk1.8.0\jre\lib\rt.jar;P:\Java\jdk1.8.0\jre\lib\ext\access-bridge.jar;P:\Java\jdk1.8.0\jre\lib\ext\cldrdata.jar;P:\Java\jdk1.8.0\jre\lib\ext\dnsns.jar;P:\Java\jdk1.8.0\jre\lib\ext\jaccess.jar;P:\Java\jdk1.8.0\jre\lib\ext\jfxrt.jar;P:\Java\jdk1.8.0\jre\lib\ext\localedata.jar;P:\Java\jdk1.8.0\jre\lib\ext\nashorn.jar;P:\Java\jdk1.8.0\jre\lib\ext\sunec.jar;P:\Java\jdk1.8.0\jre\lib\ext\sunjce_provider.jar;P:\Java\jdk1.8.0\jre\lib\ext\sunmscapi.jar;P:\Java\jdk1.8.0\jre\lib\ext\sunpkcs11.jar;P:\Java\jdk1.8.0\jre\lib\ext\zipfs.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\out\production\project-ejbclient;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.annotation.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.ejb.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.jms.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.transaction.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.persistence.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.servlet.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.resource.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.servlet.jsp.jar;D:\projects\self\dt-reservation-system-for-doctors\project-ejbclient\lib\javax.servlet.jsp.jstl.jar;P:\glassfish4\glassfish\lib\gf-client.jar;P:\glassfish4\glassfish\lib\appserv-rt.jar;P:\IntelliJ IDEA 13.1.1\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain mybeans.Main
When I run the client I am getting exception which I can't realy understand and found no help online:
Exception in thread "main" javax.naming.CommunicationException: Communication exception for SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.url.pkgs=com.sun.enterprise.naming, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl} [Root exception is java.rmi.MarshalException: CORBA BAD_PARAM 1398079494 Maybe; nested exception is:
java.io.NotSerializableException: ----------BEGIN server-side stack trace----------
org.omg.CORBA.BAD_PARAM: WARNING: 00100006: Class mybeans.__EJB31_Generated__HelloWordlBean__Intf____Bean__ is not Serializable vmcid: SUN minor code: 6 completed: Maybe
at com.sun.proxy.$Proxy153.notSerializable(Unknown Source)
at com.sun.corba.ee.impl.misc.ORBUtility.throwNotSerializableForCorba(ORBUtility.java:783)
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.writeAny(Util.java:360)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$10.write(DynamicMethodMarshallerImpl.java:306)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.writeResult(DynamicMethodMarshallerImpl.java:488)
at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:177)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1549)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:1425)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleInput(MessageMediatorImpl.java:930)
at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:213)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:694)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.dispatch(MessageMediatorImpl.java:496)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.doWork(MessageMediatorImpl.java:2222)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
----------END server-side stack trace----------]
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:513)
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:438)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at mybeans.Main.main(Main.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.rmi.MarshalException: CORBA BAD_PARAM 1398079494 Maybe; nested exception is:
java.io.NotSerializableException: ----------BEGIN server-side stack trace----------
org.omg.CORBA.BAD_PARAM: WARNING: 00100006: Class mybeans.__EJB31_Generated__HelloWordlBean__Intf____Bean__ is not Serializable vmcid: SUN minor code: 6 completed: Maybe
at com.sun.proxy.$Proxy153.notSerializable(Unknown Source)
at com.sun.corba.ee.impl.misc.ORBUtility.throwNotSerializableForCorba(ORBUtility.java:783)
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.writeAny(Util.java:360)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$10.write(DynamicMethodMarshallerImpl.java:306)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.writeResult(DynamicMethodMarshallerImpl.java:488)
at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:177)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1549)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:1425)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleInput(MessageMediatorImpl.java:930)
at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:213)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:694)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.dispatch(MessageMediatorImpl.java:496)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.doWork(MessageMediatorImpl.java:2222)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
----------END server-side stack trace----------
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:300)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:211)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:150)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:226)
at com.sun.enterprise.naming.impl._SerialContextProvider_DynamicStub.lookup(com/sun/enterprise/naming/impl/_SerialContextProvider_DynamicStub.java)
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:478)
... 8 more
Caused by: java.io.NotSerializableException: ----------BEGIN server-side stack trace----------
org.omg.CORBA.BAD_PARAM: WARNING: 00100006: Class mybeans.__EJB31_Generated__HelloWordlBean__Intf____Bean__ is not Serializable vmcid: SUN minor code: 6 completed: Maybe
at com.sun.proxy.$Proxy153.notSerializable(Unknown Source)
at com.sun.corba.ee.impl.misc.ORBUtility.throwNotSerializableForCorba(ORBUtility.java:783)
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.writeAny(Util.java:360)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$10.write(DynamicMethodMarshallerImpl.java:306)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.writeResult(DynamicMethodMarshallerImpl.java:488)
at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:177)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1549)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:1425)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleInput(MessageMediatorImpl.java:930)
at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:213)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:694)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.dispatch(MessageMediatorImpl.java:496)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.doWork(MessageMediatorImpl.java:2222)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
----------END server-side stack trace----------
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:292)
... 13 more
Caused by: org.omg.CORBA.BAD_PARAM: ----------BEGIN server-side stack trace----------
org.omg.CORBA.BAD_PARAM: WARNING: 00100006: Class mybeans.__EJB31_Generated__HelloWordlBean__Intf____Bean__ is not Serializable vmcid: SUN minor code: 6 completed: Maybe
at com.sun.proxy.$Proxy153.notSerializable(Unknown Source)
at com.sun.corba.ee.impl.misc.ORBUtility.throwNotSerializableForCorba(ORBUtility.java:783)
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.writeAny(Util.java:360)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$10.write(DynamicMethodMarshallerImpl.java:306)
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.writeResult(DynamicMethodMarshallerImpl.java:488)
at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:177)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528)
at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1549)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:1425)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleInput(MessageMediatorImpl.java:930)
at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:213)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequest(MessageMediatorImpl.java:694)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.dispatch(MessageMediatorImpl.java:496)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.doWork(MessageMediatorImpl.java:2222)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at com.sun.corba.ee.impl.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
----------END server-side stack trace---------- vmcid: SUN minor code: 6 completed: Maybe
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at com.sun.corba.ee.impl.protocol.giopmsgheaders.MessageBase.getSystemException(MessageBase.java:813)
at com.sun.corba.ee.impl.protocol.giopmsgheaders.ReplyMessage_1_2.getSystemException(ReplyMessage_1_2.java:131)
at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.getSystemExceptionReply(MessageMediatorImpl.java:594)
at com.sun.corba.ee.impl.protocol.ClientRequestDispatcherImpl.processResponse(ClientRequestDispatcherImpl.java:519)
at com.sun.corba.ee.impl.protocol.ClientRequestDispatcherImpl.marshalingComplete(ClientRequestDispatcherImpl.java:393)
at com.sun.corba.ee.impl.protocol.ClientDelegateImpl.invoke(ClientDelegateImpl.java:272)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:198)
... 12 more
I am despread :/ Can anyone help?
I'm not sure where you got this from but it looks like you mixed something up. In EJB 3 you don't have to extend EJBHome or EJBObject. You don't need the HomeInterface.
You should do it in this way:
import javax.ejb.Remote;
#Remote
public interface HelloWorldRemote {
public String hello(String name);
}
and:
#javax.ejb.Stateless(name = "HelloWorldEJB")
public class HelloWorldBean implements HelloWorldRemote {
public String hello(String name) {
return "asd" + name;
}
}
PS: There was a typo in your HelloWorld (HelloWordl).
You don't need any declaration in the ejb-jar.xml.
The client-code should look similar to this:
InitialContext con = new InitialContext();
HelloWorldBean foo = (HelloWorldBean) con.lookup("java:global/ear_ear_exploded/HelloWorldEJB");
See also:
EJB creating using SessionBean EJBObject and EJBHome interfaces
How to make EJB3 remote interface available to client?
EJB - Home/Remote and LocalHome/Local interfaces
Your class must be serializable, meaning your EJB needs implements Serializable
I'm facing ClassNotFoundException, when I run my job for the class org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat.
I tried to pass the additional jar files with -libjars, still I am facing the same issue. Any suggestions will be greatly helpful. Thanks in advance.
Below is the command I am using and exception I am facing!
hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hcatalog/rcfile/RCFileMapReduceOutputFormat
at com.cloudera.sa.omniture.mr.OmnitureToRCFileJob.run(OmnitureToRCFileJob.java:91)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.cloudera.sa.omniture.mr.OmnitureToRCFileJob.main(OmnitureToRCFileJob.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 8 more
I implemented ToolRunner as well, below is the code which confirms that!
public class OmnitureToRCFileJob extends Configured implements Tool {
public static void main(String[] args) throws Exception {
OmnitureToRCFileJob processor = new OmnitureToRCFileJob();
String[] otherArgs = new GenericOptionsParser(processor.getConf(), args).getRemainingArgs();
System.exit(ToolRunner.run(processor.getConf(), processor, otherArgs));
}
}
Did you try running by giving full path of "hcatalog-core-0.5.0-cdh4.4.0.jar" jar file in your below line.
hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir
or
Below configuration should also work for you
$ export LIBJARS= <fullpath>/hcatalog-core-0.5.0-cdh4.4.0.jar
$hadoop jar MyJob.jar MyDriver -libjars ${LIBJARS} inputDir OutputDir
If you look at hadoop command documentation, you can see that -libjars is a generic option. For parsing generic option, you got to override the ToolRunner.run() method in your driver class as follows :
public class TestDriver extends Configured implements Tool {
#Override
public int run(String[] args) throws Exception {
Configuration conf = getConf();
# Job configuration details
# Job submission
return 0;
}
}
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new TestDriver(), args);
System.exit(exitCode);
}
I'nk you are getting this exception from your driver code itself. Setting hcatalog-cor*.jar using -libjars option may not be available in client JVM(JVM in which driver code runs). Better you need to set this jar in HADOOP_CLASSPATH environment variable before executing the same using hadoop jar as follows
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:<PATH-TO-HCAT-LIB>/hcatalog-core-0.5.0-cdh4.4.0.jar;
hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir
Id had the same problem but found out the jar command doesn't accept the --libjars argument.
"Specify comma separated jar files to include in the classpath. Applies only to job." --> Hadoop Cli Generic Options
Instead you should use the env vars to add additional or replace jars.
export HADOOP_USER_CLASSPATH_FIRST=true
export HADOOP_CLASSPATH="./lib/*"
I am using Amazon EMR. I have a map / reduce job configured like this:
private static final String TEMP_PATH_PREFIX = System.getProperty("java.io.tmpdir") + "/dmp_processor_tmp";
...
private Job setupProcessorJobS3() throws IOException, DataGrinderException {
String inputFiles = System.getProperty(DGConfig.INPUT_FILES);
Job processorJob = new Job(getConf(), PROCESSOR_JOBNAME);
processorJob.setJarByClass(DgRunner.class);
processorJob.setMapperClass(EntityMapperS3.class);
processorJob.setReducerClass(SelectorReducer.class);
processorJob.setOutputKeyClass(Text.class);
processorJob.setOutputValueClass(Text.class);
FileOutputFormat.setOutputPath(processorJob, new Path(TEMP_PATH_PREFIX));
processorJob.setOutputFormatClass(TextOutputFormat.class);
processorJob.setInputFormatClass(NLineInputFormat.class);
FileInputFormat.setInputPaths(processorJob, inputFiles);
NLineInputFormat.setNumLinesPerSplit(processorJob, 10000);
return processorJob;
}
In my mapper class, I have:
private Text outkey = new Text();
private Text outvalue = new Text();
...
outkey.set(entity.getEntityId().toString());
outvalue.set(input.getId().toString());
printLog("context write");
context.write(outkey, outvalue);
This last line (context.write(outkey, outvalue);), causes this exception. Of course both outkey and outvalue are not null.
2013-10-24 05:48:48,422 INFO com.s1mbi0se.grinder.core.mapred.EntityMapperCassandra (main): Current Thread: Thread[main,5,main]Current timestamp: 1382593728422 context write
2013-10-24 05:48:48,422 ERROR com.s1mbi0se.grinder.core.mapred.EntityMapperCassandra (main): Error on entitymapper for input: 03a07858-4196-46dd-8a2c-23dd824d6e6e
java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1293)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1210)
at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
at org.apache.hadoop.io.WritableUtils.writeVLong(WritableUtils.java:264)
at org.apache.hadoop.io.WritableUtils.writeVInt(WritableUtils.java:244)
at org.apache.hadoop.io.Text.write(Text.java:281)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1077)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:698)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.s1mbi0se.grinder.core.mapred.EntityMapper.map(EntityMapper.java:78)
at com.s1mbi0se.grinder.core.mapred.EntityMapperS3.map(EntityMapperS3.java:34)
at com.s1mbi0se.grinder.core.mapred.EntityMapperS3.map(EntityMapperS3.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-10-24 05:48:48,422 INFO com.s1mbi0se.grinder.core.mapred.EntityMapperS3 (main): Current Thread: Thread[main,5,main]Current timestamp: 1382593728422 Entity Mapper end
The first records on each task are just processed ok. In some point of the task processing, I start to take this exception over and over, and then it doesn't process a single record anymore for that task.
I tried to set TEMP_PATH_PREFIX to "s3://mybucket/dmp_processor_tmp", but same thing happened.
Any idea why is this happening? What could be making hadoop not being able to write on it's output?