I am running a MR code using Multioutputformat class. part**** is getting appended in the end of my output file. How can i avoid that?
public class MR_reducer extends
Reducer {
private MultipleOutputs multipleOutputs;
#Override
protected void setup(Context context) throws IOException,
InterruptedException {
multipleOutputs = new MultipleOutputs(context);
}
#Override
protected void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
for (Text value : values) {
multipleOutputs.write(value, new Text(""), key.toString());
}
}
#Override
protected void cleanup(Context context) throws IOException,
InterruptedException {
multipleOutputs.close();
}
}
This code snippet is working from me. You have few differences:
public static class Reduce extends Reducer<Text, Text, NullWritable, Text> {
private MultipleOutputs<NullWritable, Text> multipleOutputs;
protected void setup(Context context) throws IOException, InterruptedException {
multipleOutputs = new MultipleOutputs<NullWritable, Text>(context);
}
public void reduce(Text key, Iterable<Text> values, Context output) throws IOException, InterruptedException {
while (values.iterator().hasNext()) {
multipleOutputs.write(NullWritable.get(), values.iterator().next(), key.toString());
}
}
protected void cleanup(Context context) throws IOException, InterruptedException {
multipleOutputs.close();
}
}
Related
I was trying to write a custom implementation for serializing an object like explained in here: Jackson custom serialization
Where I have:
#Override
public void serialize(
Item value, JsonGenerator jgen, SerializerProvider provider)
throws IOException, JsonProcessingException {
jgen.writeStartObject();
jgen.writeNumberField("id", value.id);
...
}
But, I noticed that I could also use the SerializerProvider to write the same thing in this other way:
#Override
public void serialize(
Item value, JsonGenerator jgen, SerializerProvider provider)
throws IOException, JsonProcessingException {
jgen.writeStartObject();
provider.defaultSerializeField("id", value.id, jsonGenerator);
...
}
So I was wondering: what is the difference between:
jgen.writeNumberField("id", value.id);
and
provider.defaultSerializeField("id", value.id, jsonGenerator);
I have defined 2 filters which should run on every request, but only after SecurityContextHolder's context is set by spring boot.
However, i always get SecurityContextHolder.getContext().getAuthentication() as null.
Here is my filter configuration:
#Bean
public FilterRegistrationBean SecurityContextHystrixRequestVariableSetterBean() throws Exception {
FilterRegistrationBean registration = new FilterRegistrationBean();
registration.setFilter(securityContextHystrixRequestVariableSetterFilter());
registration.setOrder(Ordered.LOWEST_PRECEDENCE);
return registration;
}
#Bean
public FilterRegistrationBean HystrixRequestContextEnablerFilterBean() throws Exception {
FilterRegistrationBean registration = new FilterRegistrationBean();
registration.setFilter(hystrixRequestContextEnablerFilter());
registration.setOrder(Ordered.LOWEST_PRECEDENCE);
return registration;
}
Filter details:
public class SecurityContextHystrixRequestVariableSetterFilter implements Filter {
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
SecurityContextHystrixRequestVariable.getInstance().set(SecurityContextHolder.getContext());
chain.doFilter(request, response);
}
#Override
public void init(FilterConfig filterConfig) throws ServletException {
}
#Override
public void destroy() {
}
}
public class HystrixRequestContextEnablerFilter implements Filter {
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
HystrixRequestContext context = HystrixRequestContext.initializeContext();
try {
chain.doFilter(request, response);
} finally {
context.shutdown();
}
}
#Override
public void init(FilterConfig filterConfig) throws ServletException {
}
#Override
public void destroy() {
}
}
You can use OncePerRequestFilter:
public class CustomFilter extends OncePerRequestFilter {
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain) {
//do
chain.doFilter(request, response);
}
}
#Configuration
public class CustomConfig extends WebSecurityConfigurerAdapter {
#Override
protected void configure(HttpSecurity http) throws Exception {
http
.addFilterAfter(new SecurityFilter(authenticationManager()), AnonymousAuthenticationFilter.class)
}
}
I have created onse serializer class which should restrict Bigdecimal to tow digits after decimal. But its removing zeros too. For ex: if value is 95.50, its truncating zero and output as 95.5 in json.
public class PriceJsonSerializer extends JsonSerializer {
#Override
public void serialize(BigDecimal value, JsonGenerator jgen,
SerializerProvider provider) throws IOException, JsonProcessingException
{
jgen.writeNumber(value.setScale(2,
BigDecimal.ROUND_HALF_UP).toString());
}
}
To force the serializer to preserve your rounding without writing the value as a string, try "writeRawValue".
public class PriceJsonSerializer extends JsonSerializer {
#Override
public void serialize(BigDecimal value, JsonGenerator jgen,
SerializerProvider provider) throws IOException, JsonProcessingException
{
jgen.writeRawValue(value.setScale(2,
BigDecimal.ROUND_HALF_UP).toString());
}
}
The issue is that it seems like interceptor is not called.
#Component
public class LoggingInterceptor extends HandlerInterceptorAdapter {
private static final Logger LOGGER = LogManager.getLogger(MethodHandles.lookup().lookupClass());
#Overridegre
public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object object, Exception arg3)
throws Exception {
LOGGER.info("Request Completed!");
}
#Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object object, ModelAndView model)
throws Exception {
LOGGER.info("Method executed");
}
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object object) throws Exception {
LOGGER.info("Before process request");
return true;
}
}
#Configuration
public class WebConfig extends WebMvcConfigurerAdapter {
#Autowired
LoggingInterceptor loggingInterceptor;
#Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(loggingInterceptor);
}
}
I've found examples but they are not working !
I'm trying to add start and end log and also performance log. Any idea please ?
#SpringBootApplication(scanBasePackages = { "com.sofelite.proj.controllers" })
public class ProjApplication extends SpringBootServletInitializer {
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
return application.sources(GrentyApplication.class);
}
public static void main(String[] args) throws Exception {
SpringApplication.run(GrentyApplication.class, args);
}
}
This is the Application class
Please note that in com.sofelite.proj I have all application packages such as controllers and interceptors.
Mine is working:
#Configuration
public class LoggingConfiguration extends WebMvcConfigurerAdapter {
#Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(new LoggingInterceptor());
}
}
and the LoggingInterceptor class:
#Component
public class LoggingInterceptor extends HandlerInterceptorAdapter {
private static final Logger LOGGER =
LoggerFactory.getLogger(LoggingInterceptor.class);
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response,
Object handler)
throws Exception {
long startTime = System.currentTimeMillis();
LOGGER.info("Request URL: " + request.getRequestURL());
LOGGER.info("Start Time: " + System.currentTimeMillis());
request.setAttribute("startTime", startTime);
return true;
}
#Override
public void postHandle(HttpServletRequest request, HttpServletResponse response,
Object handler, ModelAndView modelAndView) throws Exception {
StringBuilder sb = new StringBuilder();
sb.append("!Status: "+response.getStatus()+"\n");
sb.append("!URL: "+ request.getRequestURL());
sb.append("!Method: " + request.getMethod() + "\n");
LOGGER.info(sb.toString());
}
#Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response,
long endTime = System.currentTimeMillis();
System.out.println("URL Request Completed. End Time: "+ endTime);
}
}
My Original Question
since only feasible ans to that question is found in This Question
And the answer is in old API . So that force me to put a stupid question of translating these line to New API :
private long mapperCounter;
#Override
public void configure(JobConf conf) {
JobClient client = new JobClient(conf);
RunningJob parentJob =
client.getJob(JobID.forName( conf.get("mapred.job.id") ));
mapperCounter = parentJob.getCounters().getCounter(MAP_COUNTER_NAME);
}
Note : I want this code in reducer so inherited functions available are :
#Override
protected void setup(Context context) throws IOException,
InterruptedException {
}
#Override
protected void cleanup(Context context) throws IOException,
InterruptedException {
}
#Override
public void run(Context context) throws IOException,
InterruptedException {
Job job=new Job(context.getConfiguration());
}
#Override
public void reduce(ImmutableBytesWritable key,Iterable<ImmutableBytesWritable> result,Context context )
{
}
Thanks :) :)
This presentation summarizes the changes between the old and new API.