same thread name for ThreadPoolTaskExecutor - spring

Below is my config for Executor
#Configuration
#EnableAsync
public class AsyncServiceExecutorConfig extends AsyncConfigurerSupport {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(xxx.getCorePoolSize());
executor.setMaxPoolSize(xxx.getMaxPoolSize());
executor.setQueueCapacity(xxx.getQueueCapacity());
executor.setThreadNamePrefix("MyAsync-");
executor.initialize();
return executor;
}
}
That is how I call it.
Executor executor = asyncExecutorConfig.getAsyncExecutor();
CompletableFuture.runAsync(() -> {
}, executor);
When I print out the thread name, they all end with MyAsync-1
I set Thread.sleep in one MyAsync-1. The next new thread still shows as MyAsync-1
Did I do something wrong? Why it doesn't give me MyAsync-2, MyAsync-2, etc
Updated:
I tried to print out more details about threads. 10 threads are being created. They all have same thread name MyAsync-1
#Configuration
#EnableAsync
public class MyAsyncServiceExecutorConfig extends AsyncConfigurerSupport {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(0);
executor.setMaxPoolSize(3);
executor.setQueueCapacity(2);
executor.setThreadNamePrefix("MyAsync-");
executor.initialize();
return executor;
}
}
for(int i = 0; i < 10; i++) {
Executor executor = myAsyncServiceExecutorConfig.getAsyncExecutor();
CompletableFuture.runAsync(() -> {
try {
log.info("Thread.currentThread().getName() = " + Thread.currentThread().getName());
Thread.sleep(20000);
} catch (Exception e) {
e.printStackTrace();
}
}, executor);
}
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
threadSet.stream()
.filter(e -> e.getName().startsWith("MyAsync-"))
.forEach(e -> System.out.println(e.getName() + " id = " + e.getId()));
Below is the result:
...
2023-02-08 14:04:31.107 INFO 7832 --- [ MyAsync-1] c.h.w.controller.MyController : Thread.currentThread().getName() = MyAsync-1
2023-02-08 14:04:31.107 INFO 7832 --- [ MyAsync-1] c.h.w.controller.MyController : Thread.currentThread().getName() = MyAsync-1
2023-02-08 14:04:31.107 INFO 7832 --- [ MyAsync-1] c.h.w.controller.MyController : Thread.currentThread().getName() = MyAsync-1
MyAsync-1 id = 76
MyAsync-1 id = 70
MyAsync-1 id = 69
MyAsync-1 id = 73
MyAsync-1 id = 71
MyAsync-1 id = 77
MyAsync-1 id = 78
MyAsync-1 id = 74
MyAsync-1 id = 72
MyAsync-1 id = 75

Related

can anyone help me Spring Batch Issue? (Unintended schedule Spring Batch)

The implemented function is to send LMS to the user at the alarm time.
Send a total of 4 alarms (9:00, 13:00, 19:00, 21:00).
Log was recorded regardless of success.
It was not recorded in the Log, but when I looked at the batch data in the DB, I found an unintended COMPLETED.
Issue>
Batch was successfully executed at 9 and 13 on the 18th.
But at 13:37 it's not even a schedule, but it's executed. (and FAILED)
Subsequently, 13:38, 40, 42, 44 minutes executed. (all COMPLETED)
Q1. Why was it executed when it wasn't even the batch execution time?
Q2. I save the log even when executing batch and sending SMS. Log was printed normally at 9 and 13 o'clock.
But Log is not saved for non-schedule(13:37, 38, 40, 42, 44).
Check spring boot service and tomcat service with one
server CPU, memory usage is normal
Batch Problem
Spring Boot (2.2.6 RELEASE)
Spring Boot - Embedded Tomcat
===== Start Scheduler =====
#Component
public class DosageAlarmScheduler {
public static final int MORNING_HOUR = 9;
public static final int LUNCH_HOUR = 13;
public static final int DINNER_HOUR = 19;
public static final int BEFORE_SLEEP_HOUR = 21;
#Scheduled(cron = "'0 0 */1 * * *") // every hour
public void executeDosageAlarmJob() {
LocalDateTime nowDateTime = LocalDateTime.now();
try {
if(isExecuteTime(nowDateTime)) {
log.info("[Send LMS], {}", nowDateTime);
EatFixCd eatFixCd = currentEatFixCd(nowDateTime);
jobLauncher.run(
alarmJob,
new JobParametersBuilder()
.addString("currentDate", nowDateTime.toString())
.addString("eatFixCodeValue", eatFixCd.getCodeValue())
.toJobParameters()
);
} else {
log.info("[Not Send LMS], {}", nowDateTime);
}
} catch (JobExecutionAlreadyRunningException e) {
log.error("[JobExecutionAlreadyRunningException]", e);
} catch (JobRestartException e) {
log.error("[JobRestartException]", e);
} catch (JobInstanceAlreadyCompleteException e) {
log.error("[JobInstanceAlreadyCompleteException]", e);
} catch (JobParametersInvalidException e) {
log.error("[JobParametersInvalidException]", e);
} catch(Exception e) {
log.error("[Exception]", e);
}
/* Start private method */
private boolean isExecuteTime(LocalDateTime nowDateTime) {
return nowDateTime.getHour() == MORNING_TIME.getHour()
|| nowDateTime.getHour() == LUNCH_TIME.getHour()
|| nowDateTime.getHour() == DINNER_TIME.getHour()
|| nowDateTime.getHour() == BEFORE_SLEEP_TIME.getHour();
}
private EatFixCd currentEatFixCd(LocalDateTime nowDateTime) {
switch(nowDateTime.getHour()) {
case MORNING_HOUR:
return EatFixCd.MORNING;
case LUNCH_HOUR:
return EatFixCd.LUNCH;
case DINNER_HOUR:
return EatFixCd.DINNER;
case BEFORE_SLEEP_HOUR:
return EatFixCd.BEFORE_SLEEP;
default:
throw new RuntimeException("Not Dosage Time");
}
}
/* End private method */
}
}
===== End Scheduler =====
===== Start Job =====
#Configuration
public class DosageAlarmConfiguration {
private final int chunkSize = 20;
private final JobBuilderFactory jobBuilderFactory;
private final StepBuilderFactory stepBuilderFactory;
private final EntityManagerFactory entityManagerFactory;
#Bean
public Job dosageAlarmJob() {
log.info("[dosageAlarmJob excute]");
return jobBuilderFactory.get("dosageAlarmJob")
.start(dosageAlarmStep(null, null)).build();
}
#Bean
#JobScope
public Step dosageAlarmStep(
#Value("#{jobParameters[currentDate]}") String currentDate,
#Value("#{jobParameters[eatFixCodeValue]}") String eatFixCodeValue
) {
log.info("[dosageAlarm Step excute]");
return stepBuilderFactory.get("dosageAlarmStep")
.<Object[], DosageReceiverInfoDto>chunk(chunkSize)
.reader(dosageAlarmReader(currentDate, eatFixCodeValue))
.processor(dosageAlarmProcessor(currentDate, eatFixCodeValue))
.writer(dosageAlarmWriter(currentDate, eatFixCodeValue))
.build();
}
#Bean
#StepScope
public JpaPagingItemReader<Object[]> dosageAlarmReader(
#Value("#{jobParameters[currentDate]}") String currentDate,
#Value("#{jobParameters[eatFixCodeValue]}") String eatFixCodeValue
) {
log.info("[dosageAlarm Reader excute : {}, {}]", currentDate, eatFixCodeValue);
if(currentDate == null) {
return null;
} else {
JpaPagingItemReader<Object[]> jpaPagingItemReader = new JpaPagingItemReader<>();
jpaPagingItemReader.setName("dosageAlarmReader");
jpaPagingItemReader.setEntityManagerFactory(entityManagerFactory);
jpaPagingItemReader.setPageSize(chunkSize);
jpaPagingItemReader.setQueryString("select das from DosageAlarm das where :currentDate between das.startDate and das.endDate ");
HashMap<String, Object> parameterValues = new HashMap<>();
parameterValues.put("currentDate", LocalDateTime.parse(currentDate).toLocalDate());
jpaPagingItemReader.setParameterValues(parameterValues);
return jpaPagingItemReader;
}
}
#Bean
#StepScope
public ItemProcessor<Object[], DosageReceiverInfoDto> dosageAlarmProcessor(
#Value("#{jobParameters[currentDate]}") String currentDate,
#Value("#{jobParameters[eatFixCodeValue]}") String eatFixCodeValue
) {
log.info("[dosageAlarm Processor excute : {}, {}]", currentDate, eatFixCodeValue);
...
convert to DosageReceiverInfoDto
...
}
#Bean
#StepScope
public ItemWriter<DosageReceiverInfoDto> dosageAlarmWriter(
#Value("#{jobParameters[currentDate]}") String currentDate,
#Value("#{jobParameters[eatFixCodeValue]}") String eatFixCodeValue
) {
log.info("[dosageAlarm Writer excute : {}, {}]", currentDate, eatFixCodeValue);
...
make List
...
if(reqMessageDtoList != null) {
sendMessages(reqMessageDtoList);
} else {
log.info("[reqMessageDtoList not Exist]");
}
}
public SmsExternalSendResDto sendMessages(List<reqMessagesDto> reqMessageDtoList) {
log.info("[receiveList] smsTypeCd : {}, contentTypeCd : {}, messages : {}", smsTypeCd.LMS, contentTypeCd.COMM, reqMessageDtoList);
...
send Messages
}
}
===== End Job =====
Thank U.
i want to fix my problem and i hope this question is hepled another people.

How can i use #autowire in runnable spring boot

I have few MongoTemplate and Repos and i need to call them using #Autowire in my runnable class that is being executed by exceutor class using multi threading, now the problem is that when i run the application my AutoWire for mongoTempelate and Repos returns null pointer exception.
Executor class:
#Component
public class MessageConsumer implements ConsumerSeekAware {
#Autowired
AlarmDataRepository alarmDataRepository;
int assignableCores = ((Runtime.getRuntime().availableProcessors()));
ExecutorService executor = Executors.newFixedThreadPool(
assignableCores > 1 ? assignableCores : 1
);
int counter = 0;
List<String> uniqueRecords = new ArrayList<String>();
#KafkaListener(topics = "teltonikaTest", groupId = "xyz")
public void processMessages(#Payload List<String> payload, #Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions, #Header(KafkaHeaders.OFFSET) List<Long> offsets) throws UnsupportedEncodingException, DecodeException {
System.out.println("assignable resources are: " + assignableCores);
log.info("Batch Size is: {}", payload.size());
if(counter==0){
log.info("Teletonica Packets Received!");
}
for (int i = 0; i < payload.size(); i++) {
log.info("processing message='{}' with partition off-set='{}'", payload.get(i), partitions.get(i) + " _" + offsets.get(i));
}
uniqueRecords = payload.stream().distinct().collect(Collectors.toList());
Runnable worker = new TeltonikaWorkerThread(uniqueRecords);
executor.execute(worker);
counter++;
}
}
public class TeltonikaWorkerThread implements Runnable{
List<String> records;
List<CurrentDevice> currentDevices = new ArrayList<>();
#Autowired
CurrentDeviceRepository currentDeviceRepository;
#Autowired
MongoTemplate mongoTemplate;
public TeltonikaWorkerThread(List<String> records) {
this.records = records;
}
public void run() {
try {
processMessage();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (DecodeException e) {
e.printStackTrace();
}
}
public void processMessage() throws UnsupportedEncodingException,DecodeException {
for(Object record : records){
if(record!="0"){
try{
int IMEILength = record.toString().indexOf("FF");
String IMEI = record.toString().substring(0,IMEILength);
}
catch (Exception e){
e.printStackTrace();
}
}
}
}
}
If I understand correctly, your problem is about multiple beans and Spring doesn't know which one should be injected. There are several options here.
For example, you can use #Qualifier annotation based on the bean name or #Primary annotation.
If your problem is something else, please add an example to your question.

Drools decision table - rules not matching

I have a hello-world type spring/drools setup. The issue is no rules fire when in theory they should.
Decision Table:
Console output - server startup:
package com.example.drools;
//generated from Decision Table
import com.example.drools.TestRules;
// rule values at B9, header at B4
rule "_9"
when
$test:TestRules(number1 == 10)
then
$test.add("10");
end
Drools Config:
#Configuration
public class DroolsConfiguration
{
private final static String VALIDATION_RULES = "validation-rules.xls";
#Bean
public KieContainer validationRulesKieContainer() {
KieServices kieServices = KieServices.Factory.get();
Resource rules = ResourceFactory.newClassPathResource(VALIDATION_RULES);
compileXlsToDrl(rules);
KieFileSystem kieFileSystem = kieServices.newKieFileSystem().write(rules);
KieBuilder kieBuilder = kieServices.newKieBuilder(kieFileSystem);
KieBuilder builder = kieBuilder.buildAll();
KieModule kieModule = kieBuilder.getKieModule();
return kieServices.newKieContainer(kieModule.getReleaseId());
}
private static void compileXlsToDrl(Resource resource) {
try {
InputStream is = resource.getInputStream();
SpreadsheetCompiler compiler = new SpreadsheetCompiler();
String drl = compiler.compile(is, InputType.XLS);
System.out.println(drl);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Service:
#Service
public class ValidationRulesEngine
{
#Autowired
#Qualifier("validationRulesKieContainer")
private KieContainer validationKieContainer;
public void validate() {
KieSession kieSession = validationKieContainer.newKieSession();
kieSession.addEventListener(new DebugAgendaEventListener());
kieSession.addEventListener(new DebugRuleRuntimeEventListener());
TestRules tr = new TestRules(10, 20, 30);
kieSession.insert(tr);
int noOfRulesFired = kieSession.fireAllRules();
System.out.println("noOfRulesFired: " + noOfRulesFired);
System.out.println(tr);
System.out.println(tr.getRule());
}
}
TestRule - Fact:
public class TestRules
{
public int number1;
public int number2;
public int number3;
public List<String> rules = new ArrayList<String>();
public TestRules() {}
public TestRules(int number1, int number2, int number3)
{
super();
this.number1 = number1;
this.number2 = number2;
this.number3 = number3;
}
public void add(String rule) {
rules.add(rule);
}
public String getRule() {
return this.rules.size() > 0 ? this.rules.get(0) : "";
}
#Override
public String toString()
{
return "TestRules [number1=" + number1 + ", number2=" + number2 + ", number3=" + number3 + ", rules=" +
rules.stream().map(s -> s.toString()).collect(Collectors.joining(",")) + "]";
}
}
Console output - result:
2021-07-20 17:02:27.549 ERROR 20212 --- [nio-9016-exec-1] c.g.i.e.p.c.OfficeController : --> Rules Engine
==>[ObjectInsertedEventImpl: getFactHandle()=[fact 0:1:1539328290:1539328290:1:DEFAULT:NON_TRAIT:com.example.drools.TestRules:TestRules [number1=10, number2=20, number3=30, rules=]], getObject()=TestRules [number1=10, number2=20, number3=30, rules=], getKnowledgeRuntime()=KieSession[0], getPropagationContext()=PhreakPropagationContext [entryPoint=EntryPoint::DEFAULT, factHandle=[fact 0:1:1539328290:1539328290:1:DEFAULT:NON_TRAIT:com.example.drools.TestRules:TestRules [number1=10, number2=20, number3=30, rules=]], leftTuple=null, originOffset=-1, propagationNumber=2, rule=null, type=INSERTION]]
noOfRulesFired: 0
TestRules [number1=10, number2=20, number3=30, rules=]
2021-07-20 17:02:28.454 ERROR 20212 --- [nio-9016-exec-1] c.g.i.e.p.c.OfficeController : <-- Rules Engine
What am I missing?
This is no good:
$test:TestRules($test.number1 == 10, $test.number2 == 20)
You can't refer to $test before you declare it. The correct syntax is:
$test: TestRules( number1 == 10, number2 == 20 )
Fix your decision table from $test.number1 == $param to instead be number1 == $param. (And do the same for number2 adjacent.)
The rest looks fine, though I would suggest using a try-with-resources instead of a try-catch in your XLSX parsing method.

Partitions and JdbcPagingItemReader doesn't gives correct values

I am working on Spring Batch and Partition using the JdbcPagingItemReader, but I am only getting half records.
If I am expecting 100 thousand records instead getting only 50 thousand. What's wrong is happening
How to use Nested or Inner Query in OraclePagingQueryProvider ?
My Original Query
SELECT q.*
FROM (SELECT DEPT.ID id,
DEPT.CREATOR createdby,
DEPT.CREATE_DATE createddate,
DEPT.UPDATED_BY updatedby,
DEPT.LAST_UPDATE_DATE updateddate,
DEPT.NAME name,
DEPT.STATUS status,
statusT.DESCR statusdesc,
REL.ROWID_DEPT1 rowidDEPT1,
REL.ROWID_DEPT2 rowidDEPT2,
DEPT2.DEPT_FROM_VAL parentcid,
DEPT2.NAME parentname,
ROW_NUMBER() OVER (PARTITION BY DEPT.CREATE_DATE ORDER BY DEPT.ID) AS rn
FROM TEST.DEPT_TABLE DEPT
LEFT JOIN TEST.STATUS_TABLE statusT
ON DEPT.STATUS = statusT.STATUS
LEFT JOIN TEST.C_REL_DEPT rel
ON DEPT.ID = REL.ROWID_DEPT2
LEFT JOIN TEST.DEPT_TABLE DEPT2
ON REL.ROWID_DEPT1 = DEPT2.ID) q
WHERE rn BETWEEN ? AND ?; // ? will be fromValue to toValue
Code:
#Configuration
public class CustomerJob2 {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public CustomerPartitioner customerPartitioner() {
return new CustomerPartitioner();
}
#Bean("readCustomerJob")
#Primary
public Job readCustomerJob() throws Exception {
return jobBuilderFactory.get("readCustomerJob")
.incrementer(new RunIdIncrementer())
.start(customerStepOne())
.build();
}
#Bean
public Step customerStepOne() throws Exception {
return stepBuilderFactory.get("customerStepOne")
.partitioner(slaveStep().getName(), customerPartitioner())
.step(slaveStep())
.gridSize(5)
.taskExecutor(new SimpleAsyncTaskExecutor())
.build();
}
// slave step
#Bean
public Step slaveStep() throws Exception {
return stepBuilderFactory.get("slaveStep")
.<Customer, Customer>chunk(3000)
.reader(pagingItemReader(null, null))
.writer(customerWriter())
.listener(customerStepOneExecutionListener())
.build();
}
// Reader
#Bean(destroyMethod = "")
#StepScope
public JdbcPagingItemReader<Customer> pagingItemReader(
#Value("#{stepExecutionContext['fromValue']}") Long fromValue,
#Value("#{stepExecutionContext['toValue']}") Long toValue) throws Exception {
System.out.println(" FROM = "+ fromValue + " TO VALUE ="+ toValue);
JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>();
reader.setDataSource(this.dataSource);
reader.setRowMapper(new CustomerRowMapper());
reader.setSaveState(false);
reader.setPageSize(3000);
// Sort Keys
Map<String, Order> sortKeys = new HashMap<>();
sortKeys.put("id", Order.ASCENDING);
OraclePagingQueryProvider queryProvider = new OraclePagingQueryProvider();
queryProvider.setSelectClause("q.* FROM ( SELECT Row_Number() OVER (ORDER BY party.ROWID_OBJECT) MyRow, "
+ " OTHER coumns in the Query");
queryProvider.setFromClause("**** "
+ "LEFT JOIN ********* "
+ "LEFT JOIN ********* "
+ "LEFT JOIN ********* ) q ");
queryProvider.setWhereClause("MyRow BETWEEN "+ fromValue + " AND "+ toValue);
queryProvider.setSortKeys(sortKeys);
reader.setQueryProvider(queryProvider);
reader.afterPropertiesSet();
return reader;
}
#Bean
public CustomerWriter customerWriter() {
return new CustomerWriter();
}
}
Partition Logic
public class CustomerPartitioner implements Partitioner{
private static final String CUSTOMER_CNT = "SELECT COUNT(party.IS) ***** COMPLEX JOIN";
#Autowired
#Qualifier("edrJdbcTemplate")
private JdbcTemplate jdbcTemplate;
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Long custCnt = jdbcTemplate.queryForObject(CUSTOMER_CNT, Long.class);
int toValue = 0;
int fromValue = 0;
int increment = 3000;
int counter = 0;
int temp = 0;
Map<String, ExecutionContext> partitionMap = new HashMap<>();
for (int i = 0; i < custCnt; i += increment) { // custCnt fives 100 thousand
counter++;
temp = i;
if(i == 0) {
fromValue = temp;
toValue = increment;
}else {
fromValue = toValue + 1;
toValue = fromValue + increment - 1;
}
ExecutionContext context = new ExecutionContext();
context.put("fromValue", fromValue);
context.put("toValue", toValue);
partitionMap.put("Thread--" + counter, context);
}
return partitionMap;
}
}
Here are the logs -
2020-06-22 22:44:14.750 INFO 15752 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=readCustomerJob]] launched with th
e following parameters: [{JobID=1592846054670, date=1592846054670}]
2020-06-22 22:44:14.790 INFO 15752 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [customerStepOne]
Cust Count = 1035483
FROM = 6001 TO VALUE =9000
FROM = 0 TO VALUE =3000
FROM = 3001 TO VALUE =6000
FROM = 9001 TO VALUE =12000
2020-06-22 22:44:15.874 DEBUG 15752 --- [cTaskExecutor-4] o.s.b.i.database.JdbcPagingItemReader : Reading page 0
2020-06-22 22:44:15.874 DEBUG 15752 --- [cTaskExecutor-1] o.s.b.i.database.JdbcPagingItemReader : Reading page 0
2020-06-22 22:44:15.874 DEBUG 15752 --- [cTaskExecutor-2] o.s.b.i.database.JdbcPagingItemReader : Reading page 0
2020-06-22 22:44:15.874 DEBUG 15752 --- [cTaskExecutor-3] o.s.b.i.database.JdbcPagingItemReader : Reading page 0

Reduce doesn't run but job is successfully completed

Firstly, I am a newbie at Hadoop MapReduce. My reducer does not run but shows that the job is successfully completed. Below is my console output :
INFO mapreduce.Job: Running job: job_1418240815217_0015
INFO mapreduce.Job: Job job_1418240815217_0015 running in uber mode : false
INFO mapreduce.Job: map 0% reduce 0%
INFO mapreduce.Job: map 100% reduce 0%
INFO mapreduce.Job: Job job_1418240815217_0015 completed successfully
INFO mapreduce.Job: Counters: 30
The main class is :
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
#SuppressWarnings("deprecation")
Job job = new Job(conf,"NPhase2");
job.setJarByClass(NPhase2.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(NPhase2Value.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(MapClass.class);
job.setReducerClass(Reduce.class);
int numberOfPartition = 0;
List<String> other_args = new ArrayList<String>();
for(int i = 0; i < args.length; ++i)
{
try {
if ("-m".equals(args[i])) {
//conf.setNumMapTasks(Integer.parseInt(args[++i]));
++i;
} else if ("-r".equals(args[i])) {
job.setNumReduceTasks(Integer.parseInt(args[++i]));
} else if ("-k".equals(args[i])) {
int knn = Integer.parseInt(args[++i]);
conf.setInt("knn", knn);
System.out.println(knn);
} else {
other_args.add(args[i]);
}
job.setNumReduceTasks(numberOfPartition * numberOfPartition);
//conf.setNumReduceTasks(1);
} catch (NumberFormatException except) {
System.out.println("ERROR: Integer expected instead of " + args[i]);
} catch (ArrayIndexOutOfBoundsException except) {
System.out.println("ERROR: Required parameter missing from " + args[i-1]);
}
}
// Make sure there are exactly 2 parameters left.
if (other_args.size() != 2) {
System.out.println("ERROR: Wrong number of parameters: " +
other_args.size() + " instead of 2.");
}
FileInputFormat.setInputPaths(job, other_args.get(0));
FileOutputFormat.setOutputPath(job, new Path(other_args.get(1)));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
My mapper is :
public static class MapClass extends Mapper
{
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException
{
String line = value.toString();
String[] parts = line.split("\\s+");
// key format <rid1>
IntWritable mapKey = new IntWritable(Integer.valueOf(parts[0]));
// value format <rid2, dist>
NPhase2Value np2v = new NPhase2Value(Integer.valueOf(parts[1]), Float.valueOf(parts[2]));
context.write(mapKey, np2v);
}
}
My reducer class is :
public static class Reduce extends Reducer<IntWritable, NPhase2Value, NullWritable, Text>
{
int numberOfPartition;
int knn;
class Record
{
public int id2;
public float dist;
Record(int id2, float dist)
{
this.id2 = id2;
this.dist = dist;
}
public String toString()
{
return Integer.toString(id2) + " " + Float.toString(dist);
}
}
class RecordComparator implements Comparator<Record>
{
public int compare(Record o1, Record o2)
{
int ret = 0;
float dist = o1.dist - o2.dist;
if (Math.abs(dist) < 1E-6)
ret = o1.id2 - o2.id2;
else if (dist > 0)
ret = 1;
else
ret = -1;
return -ret;
}
}
public void setup(Context context)
{
Configuration conf = new Configuration();
conf = context.getConfiguration();
numberOfPartition = conf.getInt("numberOfPartition", 2);
knn = conf.getInt("knn", 3);
}
public void reduce(IntWritable key, Iterator<NPhase2Value> values, Context context) throws IOException, InterruptedException
{
//initialize the pq
RecordComparator rc = new RecordComparator();
PriorityQueue<Record> pq = new PriorityQueue<Record>(knn + 1, rc);
// For each record we have a reduce task
// value format <rid1, rid2, dist>
while (values.hasNext())
{
NPhase2Value np2v = values.next();
int id2 = np2v.getFirst().get();
float dist = np2v.getSecond().get();
Record record = new Record(id2, dist);
pq.add(record);
if (pq.size() > knn)
pq.poll();
}
while(pq.size() > 0)
{
context.write(NullWritable.get(), new Text(key.toString() + " " + pq.poll().toString()));
//break; // only ouput the first record
}
} // reduce
}
This is my helper class :
public class NPhase2Value implements WritableComparable {
private IntWritable first;
private FloatWritable second;
public NPhase2Value() {
set(new IntWritable(), new FloatWritable());
}
public NPhase2Value(int first, float second) {
set(new IntWritable(first), new FloatWritable(second));
}
public void set(IntWritable first, FloatWritable second) {
this.first = first;
this.second = second;
}
public IntWritable getFirst() {
return first;
}
public FloatWritable getSecond() {
return second;
}
#Override
public void write(DataOutput out) throws IOException {
first.write(out);
second.write(out);
}
#Override
public void readFields(DataInput in) throws IOException {
first.readFields(in);
second.readFields(in);
}
#Override
public boolean equals(Object o) {
if (o instanceof NPhase2Value) {
NPhase2Value np2v = (NPhase2Value) o;
return first.equals(np2v.first) && second.equals(np2v.second);
}
return false;
}
#Override
public String toString() {
return first.toString() + " " + second.toString();
}
#Override
public int compareTo(NPhase2Value np2v) {
return 1;
}
}
The command line command I use is :
hadoop jar knn.jar NPhase2 -m 1 -r 3 -k 4 phase1out phase2out
I am trying hard to figure out the error but still not able to come up with solution. Please help me in this regards as I am running on a tight schedule.
Because you have set the number of reducer task as 0. See this:
int numberOfPartition = 0;
//.......
job.setNumReduceTasks(numberOfPartition * numberOfPartition);
I dont see you have resetted numberOfPartition anywhere in your code. I thins you should set it where you are parsing -r option or remove call to setNumReduceTasks method as above completely as you are setting it already while parsing -r option.

Resources