I have a case that I use Hadoop to listen/receive message from JMS queue. If the queue have the message then trigger map/reduce program, so we don't want the map reduce die we need loop execute map/reduce code many many times.
My problem is:
public boolean nextKeyValue() throws IOException Using this method we return the key and value every times. If I return false the map reduce code will run to finished. If I return true the map/ reduce code will wait for the next key value rather than to call reduce method. so is there any way to run reduce method after map method soon and the nextKeyValue return true to wait the JMS queue next message?.
Or anybody has good ideas for Hadoop read continue datasource then to do map/reduce in parallel, the same function with Hstreaming?
Related
I have a mongoDB that contains a list of "task" and two istance of executors. This 2 executors have to read a task from the DB, save it in the state "IN_EXECUTION" and execute the task. Of course I do not want that my 2 executors execute the same task and this is my problem.
I use the transaction query. In this way when An executor try to change state of the task it get "write exception" and have to start again and read a new task. The problem of this approach is that sometimes an Executor get a lot of errors before it can save the change of task state correctly and execute a new task. So it is like I have only one exector.
Note:
- I do not want to block my entire DB on read/write becouse in this way I will slow down the entire process.
- I think it is necessay to save the state of the task because it could be a long task.
I asked if it is possible to lock only certain record and execute a query on the "not-locked" records but each advices that solves my problem will be really appriciated.
Thanks in advance.
EDIT1:
Sorry, I simplified the concept in the question above. Actually I extract n messages that I have to send. I have to send this messages in block of 100 messages so my executors will split the messages extracted in block of 100 and pass them to others executors basically.
Each executor extract the messages and then update them with the new state. I hope this is more clear now.
#Transactional(readOnly = false, propagation = Propagation.REQUIRED)
public List<PushMessageDB> assignPendingMessages(int limitQuery, boolean sortByClientPriority,
LocalDateTime now, String senderId) {
final List<PushMessageDB> messages = repositoryMessage.findByNotSendendAndSpecificError(limitQuery, sortByClientPriority, now);
long count = repositoryMessage.updateStateAndSenderId(messages, senderId, MessageState.IN_EXECUTION);
return messages;
}
DB update:
public long updateStateAndSenderId(List<String> ids, String senderId, MessageState messageState) {
Query query = new Query(Criteria.where(INTERNAL_ID).in(ids));
Update update = new Update().set(MESSAGE_STATE, messageState).set(SENDER_ID, senderId);
return mongoTemplate.updateMulti(query, update, PushMessageDB.class).getModifiedCount();
}
You will have to do the locking one-by-one.
Trying to lock 100 records at once and at the same time have a second process also lock 100 records (without any coordination between the two) will almost certainly result in an overlapping set unless you have a huge selection of available records.
Depending on your application, having all work done by one thread (and the other being just a "hot standby") may also be acceptable as long as that single worker does not get overloaded.
I am running an Apache Storm benchmark on a local machine.
However, I am seeing a weird behavior. One of the benchmarks i.e., SOL (Speed of light) test, uses a RandomMessageSpout to generate random tuples as source. Here is the nextTuple() code for that spout:
public void nextTuple() {
final String message = messages[rand.nextInt(messages.length)];
if(ackEnabled) {
collector.emit(new Values(message), messageCount);
messageCount++;
} else {
collector.emit(new Values(message));
}
}
When I run this benchmark and profile it using a Java profiler (Yourkit in my case). The spout thread shows sleep intervals in accordance with the SleepSpoutWaitStrategy.emptyEmit(). As my understanding goes, this function is called when nextTuple() has no tuples to emit and thus the spout thread goes to sleep for a configurable amount of time, as shown in the screenshot.
I do not understand why this function would be called given this particular nextTuple() implementation that will always return a tuple. What I am misunderstanding here?
Empty emit is also called in following situations
If the number of unacked messages reach the max spout pending.
If the executor send queue as well as the overflow buffer of spout is full.
I have written cascade flow which executes MapReduce flow containg both Mapper and Reducer.
In reduce() method, it throwsIllegalArgumentException. How to handle this exception ?
I have written catch block in class where I created JobConf for the same and added it into MapReduceFlow constructor.
In your Cascading job, you can use failure traps to do this. From the example on the link, it's something like:
flowDef.addTrap( "assertions", trap );
We use Elastic Map Reduce quite extensively, and are processing more and more data with it. Sometimes our jobs fail because the data is malformed. We've constantly revised our map scripts to handle all sorts of exceptions but sometimes there is still some malformed data that manages to break our scripts.
Is it possible to specify Elastic Map Reduce to "continue on error" even when some of the map or reduce jobs fail?
At least, is it possible to increase the minimum number of failed tasks under which the entire cluster fails (sometimes, we have only 1 failed job out of 500 or so jobs, and we would like to obtain at least those results and have the cluster continue running.)
In addition, while we can revise our map script to handle new exceptions, we use the default Hadoop "aggregate" reducer, and when that fails, we have no way to catch an exception. Is there any special way to handle errors in the "aggregate" reducer, or do we have to work with anything available to us in question #2 above (increasing minimum number of failed tasks.)
You may catch Exception in both mapper and reducer and inside the catch block have a counter like the following:
catch (Exception ex){
context.getCounter("CUSTOM_COUNTER", ex.getMessage()).increment(1);
System.err.println(GENERIC_INPUT_ERROR_MESSAGE + key + "," + value); // also log the payoad which resulted in the exception
ex.printStackTrace();
}
If the exception message is something you would have expected and also the counter's value is acceptable then you can very well go ahead with the results or else investigate the logs. I know catching Exception isn't advised but if you want to "continue on error", then it's pretty much the same thing. Since here cost of clusters are at stake, I think we are better off catching Excpetion instead of specific exceptions.
Though, there may be side effects to it, such as your code might be run on entirely wrong input and but for the catch it would have failed much earlier. But chances of something like this happening is very less.
EDIT:
For your point #2, you may set max number of allowed failures per tracker by using the following:
conf.setMaxTaskFailuresPerTracker(noFailures);
OR
The config which you must set is mapred.max.tracker.failures. As you may know the default is 4. For all other mapred configurations see here.
If I am reading your question right, you can have your cluster continue on failure to the next step defined in the elastic-mapreduce call in the ruby based command line tool for emr
--jar s3://elasticmapreduce/libs/script-runner/script-runner.jar --args "s3://bucket/scripts/script.sh" --step-name "do something using bash" --step-action CONTINUE \
I ran into an interesting situation, and now am looking for how to do it intentionally. On my local single node setup, I ran 2 jobs simultaneously from the terminal screen. My both jobs use same reducer, they only have difference in map function (aggregation key - the group by), the output of both jobs was written to the output of first job (though second job did created its own folder, but it was empty). What I am working on is providing rollup aggregations across various levels, and this behavior is fascinating for me, that the aggregation output from two different levels are available to me in one single file (also perfectly sorted).
My question is how to achieve the same in real Hadoop cluster, where we have multiple data nodes i.e. I programmatically initiate multiple jobs, all accessing same input file, mapping the data differently, but using the same reducer, and the output is available in one single file, and not in 5 different output files.
Please advise.
I was taking a look at merge output files after reduce phase before I decided to ask my question.
Thanks and Kind regards,
Moiz Ahmed.
When different Mappers consume the same input file, with other words the same data structure, then source code for all these different mappers can be placed into separate methods of a single Mapper implementation and use a parameter from the context to decide which map functions to invoke. On the pluss side you need to start only one Map Reduce Job. Example is pseudo code:
class ComplexMapper extends Mapper {
protected BitSet mappingBitmap = new BitSet();
protected void setup(Context context) ... {
{
String params = context.getConfiguration().get("params");
---analyze params and set bits into the mappingBitmap
}
protected void mapA(Object key, Object value, Context context){
.....
context.write(keyA, value);
}
protected void mapB(Object key, Object value, Context context){
.....
context.write(keyA, value);
}
protected void mapB(Object key, Object value, Context context){
.....
context.write(keyB, value);
}
public void map(Object key, Object value, Context context) ..... {
if (mappingBitmap.get(1)) {
mapA(key, value, context);
}
if (mappingBitmap.get(2)) {
mapB(key, value, context);
}
if (mappingBitmap.get(3)) {
mapC(key, value, context);
}
}
Of cause it can be implemented more elegantly with interfaces etc.
In the job setup just add:
Configuration conf = new Configuration();
conf.set("params", "AB");
Job job = new Job(conf);
As Praveen Sripati mentioned, having a single output file will force you into having just one Reducer which might be bad for performance. You can always concatenate the part** files when you download them from the hdfs. Example:
hadoop fs -text /output_dir/part* > wholefile.txt
Usually each reducer task produces a separate file in HDFS, so that the reduce tasks can operate in parallel. If the requirement is to have one o/p file from the reduce task then configure the job to have one reducer task. The number of reducers can be configure using the mapred.reduce.tasks property which is defaulted to 1. The con of this approach is there is only one reducer which might be a bottle neck for the job to complete.
Another option is to use some other output format which allows multiple reducers to write to the same sink simultaneously like DBOuputFormat. Once the Job processing is complete, the results from the DB can be exported into a flat file. This approach will enable multiple reduce tasks to run in parallel.
Another options is to merge the o/p files as mentioned in the OP. So, based on the pros and cons of each of the approach and the volume of the data to be processed the one of the approach can be chosen.