I have a Spring-Boot (1.4.0) application and I am using springframework.boot:spring-boot-starter-mail.
I have a method annotated with #Scheduled to check my inbox every a certain period of time.
This is how I get my inbox:
private static Folder getInbox() throws MessagingException {
final String protocol = "mail.store.protocol";
final String storeType = "imaps";
final String email = "email";
final String password = "password";
final String connect = "webmail.company.com";
final String folder = "INBOX";
final Properties props = new Properties();
props.setProperty(protocol, storeType);
final Session session = Session.getInstance(props, null);
final Store store = session.getStore();
store.connect(connect, email, password);
final Folder inbox = store.getFolder(folder);
inbox.open(Folder.READ_WRITE);
return inbox;
}
Then I have this:
#Scheduled(fixedRate = 10000)
#Override
public void checkEmailCreateCompanyAndSendCsv() throws MessagingException, IOException {
log.info("Checking e-mail...");
final Folder inbox = getInbox();
final Flags seen = new Flags(Flags.Flag.SEEN);
final FlagTerm unseenFlagTerm = new FlagTerm(seen, false);
inbox.getMessages();
final Message messages[] = inbox.search(unseenFlagTerm);
.....
.....
}
When the APP is running everything works great but after some time (around 7 to 8 e-mail checks) it starts to thrown an exception:
javax.mail.AuthenticationFailedException: AUTHENTICATE failed. at
com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:717) at
javax.mail.Service.connect(Service.java:366) at
javax.mail.Service.connect(Service.java:246) at
com.opessoftware.crs.selfcertification.services.EmailServiceBasic.getInbox(EmailServiceBasic.java:183)
at
com.opessoftware.crs.selfcertification.services.EmailServiceBasic.checkEmailCreateCompanyAndSendCsv(EmailServiceBasic.java:50)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
If I stop the application and run it again the error disappears and the cicle starts again.
Any suggestions?
I just solved it.
I was calling the connection too many times and I was exceeding the connection pool of my e-mail server. So what I did is to define the getInbox() and the getSession() as #Bean and Inject them to my EmailService.
Now it creates the connection only once. It is working fine.
Related
I am currently trying to test a simple post method in a normal Controller which returns a Mono to redirect to a different page or in this case the home page. I have tried all sorts of different aproaches mocking components but I always seem to be returning a null Mono in the test all works normally via form submission.
#Controller
public class AddNewEntryController {
private final EntryService service;
#PostMapping("/add-new-entry")
public Mono<String> addNewEntrySubmit(#ModelAttribute("timeEntry") Entry entry) {
return service.addTimeKeepingEntry(Flux.just(entry)).then(Mono.just("redirect:/"));
}
}
And the Service Class Code
public Mono<Void> addTimeKeepingEntry(Flux<Entry> entry) {
return entry.flatMap(entry -> Mono.when(repository.save(entry).log("Save to DB"))
.log("add entry when")).then().log("done");
}
And Test Code
#RunWith(SpringRunner.class)
#WebFluxTest(controllers = AddNewEntryController.class)
#Import({ThymeleafAutoConfiguration.class})
public class AddNewEntryControllerTest {
#Autowired
WebTestClient webTestClient;
#MockBean
EntryService service;
#Test
public void addNewEntrySubmit() {
MultiValueMap<String, String> formData = new LinkedMultiValueMap<>();
formData.add("month", month);
formData.add("dateOfMonth", Integer.toString(21));
formData.add("startTime", "09:00");
when(service.addEntry(Flux.just(entry1))).thenReturn(Mono.empty());
webTestClient.post().uri("/add-new-entry").body(BodyInserters.fromFormData(formData)).exchange().expectStatus().isSeeOther().expectHeader().valueEquals(HttpHeaders.LOCATION, "/");
Whenever I run the test I am always getting a Null Pointer and after debugging it is pointing to Mono as being Null. Problem is I am not sure which Mono or at which step.
The StackTrace I get is as follows.
java.lang.NullPointerException: null
at com.dbeer.timekeeping.UI.AddNewEntryController.addNewEntrySubmit(AddNewEntryController.java:47) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.web.reactive.result.method.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:243) ~[spring-webflux-5.0.9.RELEASE.jar:5.0.9.RELEASE]
at org.springframework.web.reactive.result.method.InvocableHandlerMethod.lambda$invoke$0(InvocableHandlerMethod.java:138) ~[spring-webflux-5.0.9.RELEASE.jar:5.0.9.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118) [reactor-core-3.1.9.RELEASE.jar:3.1.9.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.9.RELEASE.jar:3.1.9.RELEASE]
at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) ~[reactor-core-3.1.9.RELEASE.jar:3.1.9.RELEASE]
at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) ~[reactor-core-3.1.9.RELEASE.jar:3.1.9.RELEASE]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:185) ~[reactor-core-3.1.9.RELEASE.jar:3.1.9.RELEASE]
after looking at your project, the issue seems to be the consistency of naming of controllers and the html page.
e.g. in header.html you had a url to link add-entry but your controller had add-new-entry if you change the url in header to add-new-entry it works.
As a clean up, you should use thmyeleaf to generate the URL not a normal href, as if you ever add security later, thymeleaf will add the session id to the URL etc
***********Edit pulled the branch and could reproduce *******
The line
given(service.addTimeKeepingEntry(Flux.just(new TimeKeepingEntry(month, 21, "Tuesday", "09:00", "30", "17:00", "7.5", false)))).willReturn(Mono.empty());
is the issue, since Mockito matches on Object.equals here and you have not defined what equals means to your object.
another way is to capture the object passed into the mock
e.g.
#Captor
private ArgumentCaptor<Flux<TimeKeepingEntry>> captor;
#Test
public void addNewEntrySubmit() {
MultiValueMap<String, String> formData = new LinkedMultiValueMap<>();
formData.add("month", month);
formData.add("dateOfMonth", Integer.toString(21));
formData.add("day", "Tuesday");
formData.add("startTime", "09:00");
formData.add("endTime", "17:00");
formData.add("breakLength", "30");
given(service.addTimeKeepingEntry(any())).willReturn(Mono.empty());
webTestClient.post().uri("/add-new-entry")
.body(BodyInserters.fromFormData(formData)).exchange().expectStatus().isSeeOther().expectHeader().valueEquals(HttpHeaders.LOCATION, "/");
verify(service).addTimeKeepingEntry(captor.capture());
TimeKeepingEntry timeKeepingEntry = captor.getValue().blockFirst();
assertThat(timeKeepingEntry.getMonth()).isEqualTo(month);
//and whatever else you want to test
}
I use kafkaEmbedded in integration test and I get FileNotFoundException :
java.io.FileNotFoundException: /tmp/kafka-7785736914220873149/replication-offset-checkpoint.tmp
at java.io.FileOutputStream.open0(Native Method) ~[na:1.8.0_141]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[na:1.8.0_141]
at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[na:1.8.0_141]
at java.io.FileOutputStream.<init>(FileOutputStream.java:162) ~[na:1.8.0_141]
at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:43) ~[kafka_2.11-0.11.0.0.jar:na]
at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:58) ~[kafka_2.11-0.11.0.0.jar:na]
at kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:1118) [kafka_2.11-0.11.0.0.jar:na]
at kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:1115) [kafka_2.11-0.11.0.0.jar:na]
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) [scala-library-2.11.11.jar:na]
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116) [scala-library-2.11.11.jar:na]
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) [scala-library-2.11.11.jar:na]
at kafka.server.ReplicaManager.checkpointHighWatermarks(ReplicaManager.scala:1115) [kafka_2.11-0.11.0.0.jar:na]
at kafka.server.ReplicaManager$$anonfun$1.apply$mcV$sp(ReplicaManager.scala:211) [kafka_2.11-0.11.0.0.jar:na]
at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) [kafka_2.11-0.11.0.0.jar:na]
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57) [kafka_2.11-0.11.0.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_141]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_141]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_141]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
My tests pass with success but I get this error in the end of my build
After many hours of research I found this :
kafka TestUtils.tempDirectory method is used to create temporary directory for embedded kafka broker. It also registers shutdown hook which deletes this directory when JVM exits.
when unit test finishes execution it calls System.exit, which in turn executes all registered shutdown hooks
If kafka broker runs at the end of unit test it will attempt to write/read data in a dir which is deleted and produces different FileNotFound exceptions.
My config class :
#Configuration
public class KafkaEmbeddedConfiguration {
private final KafkaEmbedded kafkaEmbedded;
public KafkaEmbeddedListenerConfigurationIT() throws Exception {
kafkaEmbedded = new KafkaEmbedded(1, true, "topic1");
kafkaEmbedded.before();
}
#Bean
public KafkaTemplate<String, Message> sender(ProtobufSerializer protobufSerializer,
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry) throws Exception {
KafkaTemplate<String, Message> sender = KafkaTestUtils.newTemplate(kafkaEmbedded, new StringSerializer(),
protobufSerializer);
for (MessageListenerContainer listenerContainer :
registry.getListenerContainers()) {
ContainerTestUtils.waitForAssignment(listenerContainer,
kafkaEmbedded.getPartitionsPerTopic());
}
return sender;
}
Test class :
#RunWith(SpringRunner.class)
public class DeviceEnergyKafkaListenerIT {
...
#Autowired
private KafkaTemplate<String, Message> sender;
#Test
public void test (){
...
sender.send(topic, msg);
sender.flush();
}
Any ideas how to resolve this please ?
With a #ClassRule broker, add an #AfterClass method...
#AfterClass
public static void tearDown() {
embeddedKafka.getKafkaServers().forEach(b -> b.shutdown());
embeddedKafka.getKafkaServers().forEach(b -> b.awaitShutdown());
}
For a #Rule or bean, use an #After method.
final KafkaServer server =
embeddedKafka.getKafkaServers().stream().findFirst().orElse(null);
if(server != null) {
server.replicaManager().shutdown(false);
final Field replicaManagerField = server.getClass().getDeclaredField("replicaManager");
if(replicaManagerField != null) {
replicaManagerField.setAccessible(true);
replicaManagerField.set(server, null);
}
}
embeddedKafka.after();
For a more detail discussion you can refer this thread
Embedded kafka issue with multiple tests using the same context
The following solution provided by mhyeon-lee has worked for me:
import org.apache.kafka.common.utils.Exit
class SomeTest {
static {
Exit.setHaltProcedure((statusCode, message) -> {
if (statusCode != 1) {
Runtime.getRuntime().halt(statusCode);
}
});
}
#Test
void test1() {
}
#Test
void test2() {
}
}
When JVM shutdown Hook is running, kafka log file is deleted and
Exit.halt (1) is called when other shutdown hook accesses kafka log
file at the same time.
Since halt is called here and status is 1, i only defend against 1.
https://github.com/a0x8o/kafka/blob/master/core/src/main/scala/kafka/log/LogManager.scala#L193
If you encounter a situation where the test fails with a different
status value, you can add defense code.
An error log may occur, but the test will not fail because the command
is not propagated to Runtime.halt.
References:
https://github.com/spring-projects/spring-kafka/issues/194#issuecomment-612875646
https://github.com/spring-projects/spring-kafka/issues/194#issuecomment-613548108
I'm trying to create an embedded ElasticSearch node for integration testing.
Here is the code creation
private static final String THREAD_NAME = "ES-THREAD";
private static final String CLUSTER_NAME = "ES-INTEGRATION-TEST";
private static final String ES_HOME_PATH = "elastic-search-home";
private static final String ES_DATA_PATH = "elastic-search-data";
private static final String DATA_PORTS = "9500-9599";
private static final String TRANSPORT_PORTS = "9600-9699";
public void before() throws Throwable {
try {
homeDir = Files.createTempDirectory(ES_HOME_PATH);
dataDir = Files.createTempDirectory(ES_DATA_PATH);
log.info("Created temp directory {} and {}", homeDir, dataDir);
} catch (IOException ex) {
throw new IllegalStateException("Temp Elastic Search directory not created", ex);
}
Properties props = new Properties();
props.setProperty("name", THREAD_NAME);
props.setProperty("path.home", homeDir.toString());
props.setProperty("path.data", dataDir.toString());
props.setProperty("http.port", DATA_PORTS);
props.setProperty("transport.tcp.port", TRANSPORT_PORTS);
props.setProperty("node.local", "true");
props.setProperty("script.groovy.sandbox.enabled", "true");
props.setProperty("script.engine.groovy.inline.aggs", "true");
props.setProperty("script.engine.groovy.inline.search", "true");
props.setProperty("script.engine.groovy.inline.update", "true");
props.setProperty("script.engine.groovy.inline.mapping", "true");
esNode = NodeBuilder.nodeBuilder().local(false).client(false)
.settings(Settings.settingsBuilder().put(props).build()).clusterName(CLUSTER_NAME).build();
esNode.start();
}
In the code tested there is the following method which creates transport connection to ElasticSearch
private Client createClient() throws UnknownHostException {
Settings.Builder builder = Settings.builder();
builder.put("cluster.name", clusterName);
builder.put("client.transport.ignore_cluster_name", true);
Settings settings = builder.build();
return TransportClient.builder().settings(settings).build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(hostname), port));
}
when I run the test i receive excepetion
java.net.BindException: Can't assign requested address
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.jboss.netty.channel.Channels.connect(Channels.java:634)
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:216)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:913)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:880)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:852)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:250)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:354)
at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:300)
at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:333)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The values I send for the tested code as hostname is localhost and I send the port I pull from the embedded ElasticSearch by
NodeInfo nodeInfo = esNode.client().admin().cluster().prepareNodesInfo(localNodeId).get().iterator().next();
transportAddress = nodeInfo.getTransport().address().publishAddress().getAddress();
I saw that the transport port is always 0 and when I evaluate nodeInfo.getTransport().address() its value is local[1].
What in the node creation is wrong?
Is there another configuration I need to add?
Thanks,
Daniela
when I changed props.setProperty("node.local", "true"); to props.setProperty("node.local", "false"); the transport port was created.
This is how I create "AbstractInputStreamContent" from inputstream of file:
final Long length = Long.valueOf(filesData.get(uploadedFileName).get("size")).longValue();
final InputStream fileStream = item.openStream(); //FileItemStream item
AbstractInputStreamContent fileContent = new AbstractInputStreamContent(uploadedFileMimeType) {
#Override
public boolean retrySupported() {
return false;
}
#Override
public long getLength() throws IOException {
return length;
}
#Override
public InputStream getInputStream() throws IOException {
return fileStream;
}
};
And "InputStreamContent" as:
InputStreamContent fileContent = new InputStreamContent(uploadedFileMimeType, item.openStream());
fileContent.setLength(Long.valueOf(filesData.get(uploadedFileName).get("size")).longValue());
To replace old file with new file I use(both files are of .docx format):
Drive.Files.Update update = driveService.files().update(fileIdOfFileToReplace,fileMeta,fileContent);
update.set("uploadType", "resumable");
update.getMediaHttpUploader().setDirectUploadEnabled(false);
update.getMediaHttpUploader().setChunkSize(MediaHttpUploader.DEFAULT_CHUNK_SIZE);
File updatedFile = update.execute();
Uploading a new file works fine whether I use InputStreamContent or AbstractInputStreamContent. But update gives "java.lang.IllegalArgumentException" with both
java.lang.IllegalArgumentException
at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:111)
at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:37)
at com.google.api.client.googleapis.media.MediaHttpUploader.setInitiationRequestMethod(MediaHttpUploader.java:872)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.initializeMediaUpload(AbstractGoogleClientRequest.java:237)
at com.google.api.services.drive.Drive$Files$Update.<init>(Drive.java:3163)
at com.google.api.services.drive.Drive$Files.update(Drive.java:3113)
at com.util.DocumentsUtil.updateFile(DocumentsUtil.java:22)
at com.controllers.collab.documents.Documents.fileUpload(Documents.java:165)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:44)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
Solved it by giving exclusion of google-api-client from each google api maven dependency. Later added google-api-client-1.22.0-SNAPSHOT dependency from sonatype repo. It works fine
I want to use Kryo serialization in Spark job.
public class SerializeTest {
public static class Toto implements Serializable {
private static final long serialVersionUID = 6369241181075151871L;
private String a;
public String getA() {
return a;
}
public void setA(String a) {
this.a = a;
}
}
private static final PairFunction<Toto, Toto, Integer> WRITABLE_CONVERTOR = new PairFunction<Toto, Toto, Integer>() {
private static final long serialVersionUID = -7119334882912691587L;
#Override
public Tuple2<Toto, Integer> call(Toto input) throws Exception {
return new Tuple2<Toto, Integer>(input, 1);
}
};
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("SerializeTest");
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.registerKryoClasses(new Class<?>[]{Toto[].class});
JavaSparkContext context = new JavaSparkContext(conf);
List<Toto> list = new ArrayList<Toto>();
list.add(new Toto());
JavaRDD<Toto> cursor = context.parallelize(list, list.size());
JavaPairRDD<Toto, Integer> writable = cursor.mapToPair(WRITABLE_CONVERTOR);
writable.saveAsHadoopFile(args[0], Toto.class, Integer.class, SequenceFileOutputFormat.class);
context.close();
}
}
But i have this error :
java.io.IOException: Could not find a serializer for the Key class: 'com.test.SerializeTest.Toto'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization.
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1179)
at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1094)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapred.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:63)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/09/21 17:49:14 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: Could not find a serializer for the Key class: 'com.test.SerializeTest.Toto'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization.
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1179)
at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1094)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapred.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:63)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks.
This error is related neither to Spark nor Kryo.
When using Hadoop output formats you need to make sure your key and value are instances of Writable. Hadoop doesn't use Java serialization by default (and you don't want to use it either, because it's very ineffective)
You can check your io.serializations property in configuration and you'll see list of used serializers including org.apache.hadoop.io.serializer.WritableSerialization
To fix this issue your Toto class must implement Writable. The same issue is with Integer, use rather IntWritable.