Geting exception while using DistributedCache in Hadoop - hadoop

I have used below mentioned code for Distributed Caching in main class.
job.addCacheFile(new URI(args[2]));
Below mentioned code is in reduce class.
#Override
protected void setup(Reducer<LongWritable, Text, Text, Text>.Context context) throws IOException, InterruptedException {
super.setup(context);
URI[] paths = context.getCacheFiles();
if (paths.length > 0) {
loadDeliveryStatusCodes(paths[0].toString());
}
}
private void loadDeliveryStatusCodes(String file) {
String strRead;
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader("./some"));
while ((strRead = reader.readLine()) != null) {
String splitarray[] = strRead.split(",");
deliveryCodesMap.put(splitarray[0].trim(), splitarray[1].trim());
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I am getting below mentioned exception.
########/user/DeliveryStatusCodes.txt 1
java.io.FileNotFoundException: ./some (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at com.hadoop.intellipaat.UserSMSDeliveryJob$USERNameSMSStatusCodeMapper.loadDeliveryStatusCodes(UserSMSDeliveryJob.java:95)
Your help will save my day.
Thanks.

It looks like you're missing the #some from end of your args[2].At the moment you have /user/DeliveryStatusCodes.txt, but you should have
/user/DeliveryStatusCodes.txt#some
Without it, FileReader is literally looking for ./some, which of course won't exist.
Or you can skip the alias (it's optional) and write
new FileReader("/user/DeliveryStatusCodes.txt")

Related

finally block - variable cannot be resolved

Java 8
import java.util.zip.GZIPOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
private void createFile(final String json) throws IOException {
final String fileName = getConfigFileName(this.getSomePath());
GZIPOutputStream out = null;
try {
out = new GZIPOutputStream(new FileOutputStream(fileName + ".gz"));
out.write(json.getBytes());
} catch (IOException e) {
throw e;
} finally {
try {
if (out != null) {
out.finish();
out.close();
}
} catch (IOException e) {
LOGGER.error("createFile: IOException while closing resources", e);
}
}
}
Nice. This work fine.
Now I want to use try-with-resource
private void createFile(final String json) throws IOException {
final String fileName = getConfigFileName(this.getSomeFile());
try (GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(fileName + ".gz"))) {
out.write(json.getBytes());
} catch (IOException e) {
throw e;
} finally {
try {
if (out != null) {
out.finish();
}
} catch (IOException e) {
LOGGER.error("createFile: IOException while closing resources", e);
}
}
}
But now I get error in this line:
if (out != null) {
Error is:
out cannot be resolved
I know this error is rise because variable out is on finally section.
But how I can use try-with-resources and execute method out.finish ?
From a technical perspective - a variable declared in the try argument isn't available in the finally clause, as you've seen. The good news here is that from a function perspective - finish() shouldn't be in the finally block anyway. finish is part of the positive (a.k.a "happy") flow, and should only be called when you're done writing to the stream. In other words, if the write operation failed and an exception was thrown, you shouldn't call finish anyway.
To make a long story short - move the finish call inside the try block:
Side note: Since your method throws an IOException, there's no reason to catch the exception and rethrow it. You can clean up the code by allowing it to be thrown from the method call directly:
private void createFile(final String json) throws IOException {
final String fileName = getConfigFileName(this.getSomeFile());
try (GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(fileName + ".gz"))) {
out.write(json.getBytes());
out.finish();
}
}

Cloning javax.mail.Message and Cloning javax.mail.Multipart, Java 8

I'm implementing a mail Sender, near 1'6000.000 mails (with images and PDF) in one day per month (closing month extract), the mails are about 12 products...
I need to fill a Message Scratch per product... in order to not read (per email) else per product.
I'm trying to implement cloning javax.mail.Message and javax.mail.Multipart in order to be faster.
AddContent to Multipart
public static void addContent(final Multipart multipart, String contenidoCorreo) throws Exception {
MimeBodyPart mimeBodyPart = new PreencodedMimeBodyPart("8bit");
mimeBodyPart.setText(contenidoCorreo, "iso-8859-1", "html");
multipart.addBodyPart(mimeBodyPart, 0);
}
Add Image per Bytes
public static void addImageToMultipart(final Multipart multipart, byte[] contenidoImagen, String nombreImagen) throws Exception {
MimeBodyPart imagenMimeBodyPart = new MimeBodyPart();
try {
ByteArrayDataSource byteArrayDataSource = new ByteArrayDataSource(contenidoImagen, "image/*");
imagenMimeBodyPart.setDataHandler(new DataHandler(byteArrayDataSource));
imagenMimeBodyPart.setFileName(nombreImagen);
imagenMimeBodyPart.setContentID("<" + nombreImagen + ">");
imagenMimeBodyPart.setDisposition(MimeBodyPart.INLINE);
multipart.addBodyPart(imagenMimeBodyPart);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e.getMessage());
}
}
AddPDF per File
public static void addPDF(final Multipart multipart, String ruta, String nombre) throws Exception {
Path path = Paths.get(ruta, nombre);
if (path.toFile().exists()) {
MimeBodyPart preencodedMimeBodyPart = new PreencodedMimeBodyPart("base64");
preencodedMimeBodyPart.attachFile(path.toFile());
preencodedMimeBodyPart.setFileName(nombre);
preencodedMimeBodyPart.setHeader("Content-Type", "application/pdf");
preencodedMimeBodyPart.setDisposition(MimeBodyPart.ATTACHMENT);
multipart.addBodyPart(preencodedMimeBodyPart);
MimeBodyPart pdfMimeBodyPart = new MimeBodyPart();
}
My Cloning Message
public static Message cloneMessage(Message source) {
//Multiple and Separated Exceptions because maybe not all properties are defined in some time.
Message target = new MimeMessage(source.getSession());
try {
if (source.getFrom() != null && source.getFrom().length > 0) {
Address address = (source.getFrom()[0]);
target.setFrom(new InternetAddress(((InternetAddress) address).getAddress(), ((InternetAddress) address).getPersonal()));
}
} catch (Exception ex) {
//Handle Exception
}
try {
target.setSentDate((Date) (source.getSentDate().clone()));
} catch (MessagingException ex) {
//Handle Exception
}
try {
target.setRecipients(Message.RecipientType.TO, target.getRecipients(Message.RecipientType.TO).clone());
} catch (MessagingException ex) {
//Handle Exception
}
try {
Enumeration numerationHeaders = source.getAllHeaders();
while (numerationHeaders.hasMoreElements()) {
Header header = (Header) numerationHeaders.nextElement();
target.addHeader(header.getName(), header.getValue());
}
} catch (MessagingException ex) {
//Handle Exception
}
try {
target.setSubject(new String(source.getSubject()));
} catch (MessagingException ex) {
//Handle Exception
}
try {
target.setContent(cloneMultipart((Multipart)(source.getContent())));
} catch (Exception ex) {
//Handle Exception
}
return target;
}
Cloning Multipart
public static Multipart cloneMultipart(Multipart source) {
MimeMultipart target = new MimeMultipart();
try {
for (int i = 0; i < source.getCount(); i++) {
MimeBodyPart mimeBodyPart = (MimeBodyPart)source.getBodyPart(i);
mimeBodyPart //?????
}
} catch (MessagingException ex) {
//Handle Exception
}
return target;
}
How cloning Multipart?
some advice to clone Message?
How detect the Content (the used with addContent method) has been added?

download csv using spring boot and apache commons

I have this below code for downloading CSV as an ajax button click, But the file is not downloading. Only showing the black new tab on the browser.
#RequestMapping(value = "/batch/download", method = RequestMethod.POST, produces = "text/csv")
#ResponseBody
public void downloadNGIBatchSelected(HttpServletResponse response) throws IOException {
List<String> ids = Arrays.asList("1312321","312313");
generateNewCustomerCSV(response.getWriter(),ids);
}
private void generateNewCustomerCSV(PrintWriter writer, List<String> ids){
String NEW_LINE_SEPARATOR = "\n";
//CSV file header
Object[] FILE_HEADER = {"Token Number",
"Token Expiry Date",
};
CSVPrinter csvPrinter = null;
try {
csvPrinter = new CSVPrinter(new BufferedWriter(writer), CSVFormat.DEFAULT.withRecordSeparator(NEW_LINE_SEPARATOR));
//Create CSV file header
csvPrinter.printRecord(FILE_HEADER);
for (PolicyMap PolicyMap : policyMaps) {
List customerCSV = new ArrayList();
customerCSV.add(PolicyMap.getInsurancePolicy().getTokenNo());
try {
csvPrinter.printRecord(customerCSV);
} catch (IOException e) {
e.printStackTrace();
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
writer.flush();
writer.close();
csvPrinter.close();
} catch (IOException e) {
System.out.println("Error while flushing/closing fileWriter/csvPrinter !!!");
e.printStackTrace();
}
}
}
You have set the content type in #RequestMapping annotation. But it is not going to work in the case when response is being written using HttpServletResponse. In this case, instead of spring, HttpServletResponse is writing the response that's why you have to set the response type in the response before getting the writer.
response.setContentType ("application/csv");
response.setHeader ("Content-Disposition", "attachment; filename=\"nishith.csv\"");

How to call shell script from Toad

I tried with the below linked steps:
https://slobaray.com/tag/execute-shell-script-from-plsql/
It consists in
creating a java object stored as BASH_OS function with
CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "BASH_OS"..
Then to execute the java with
CREATE OR REPLACE PROCEDURE unix_command (p_command IN VARCHAR2)
AS LANGUAGE JAVA
NAME 'BASH_OS.executeCommand (java.lang.String)';
But It shows that
the object BASH_OS does not exist.
Can any one help me on how we can execute the shell script from the Toad with this method?
or Is there any other method to call shell script from Toad?
Referring your link, I found that the class name in line 13 is wrong for creating BASH_OS.
The class name should be "BASH_OS" instead of "Host". Since there is no comment feature on the blog, so i copy the fixed code here.
CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "BASH_OS" AS
/******************************************************************************
NAME: BASH_OS
PURPOSE: To perform the shell command using Java class
REVISIONS:
Ver Date Author Description
--------- ---------- ------ ---------------------------------------------
0.1 <<>> S.Ray Initial Version
******************************************************************************/
import java.io.*;
/* public class Host { <-- THIS IS WRONG */
public class BASH_OS {
public static void executeCommand(String command) {
try {
String[] finalCommand;
{
finalCommand = new String[3];
finalCommand[0] = "/bin/sh";
finalCommand[1] = "-c";
finalCommand[2] = command;
}
final Process pr = Runtime.getRuntime().exec(finalCommand);
pr.waitFor();
new Thread(new Runnable(){
public void run() {
BufferedReader br_in = null;
try {
br_in = new BufferedReader(new InputStreamReader(pr.getInputStream()));
String buff = null;
while ((buff = br_in.readLine()) != null) {
System.out.println("Process out :" + buff);
try {Thread.sleep(100); } catch(Exception e) {}
}
br_in.close();
}
catch (IOException ioe) {
System.out.println("Exception caught printing process output.");
ioe.printStackTrace();
}
finally {
try {
br_in.close();
} catch (Exception ex) {}
}
}
}).start();
new Thread(new Runnable(){
public void run() {
BufferedReader br_err = null;
try {
br_err = new BufferedReader(new InputStreamReader(pr.getErrorStream()));
String buff = null;
while ((buff = br_err.readLine()) != null) {
System.out.println("Process err :" + buff);
try {Thread.sleep(100); } catch(Exception e) {}
}
br_err.close();
}
catch (IOException ioe) {
System.out.println("Exception caught printing process error.");
ioe.printStackTrace();
}
finally {
try {
br_err.close();
} catch (Exception ex) {}
}
}
}).start();
}
catch (Exception ex) {
System.out.println(ex.getLocalizedMessage());
}
}
};

Posting HTTPS form results in html 404 status code

I keep getting a HTML 404 reply from the server when I try to login via a httppost (https). Not sure if this is a cookie problem or something else. The code should be good as I have copied it from another activity. I need some help.
This is my current code:
public int postData(String usernamne, String password) {
String url = "https://domainname.com/nclogin.submit";
HttpPost httppost = new HttpPost(url);
try {
KeyStore trusted = null;
try {
trusted = KeyStore.getInstance("BKS");
} catch (KeyStoreException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
trusted.load(null, "".toCharArray());
MySSLSocketFactory sslf = null;
try {
sslf = new MySSLSocketFactory(trusted);
} catch (KeyManagementException e) {
Log.d(TAG, "Exception " + e);
// TODO Auto-generated catch block
e.printStackTrace();
} catch (UnrecoverableKeyException e) {
Log.d(TAG, "Exception " + e);
// TODO Auto-generated catch block
e.printStackTrace();
} catch (KeyStoreException e) {
Log.d(TAG, "Exception " + e);
// TODO Auto-generated catch block
e.printStackTrace();
}
sslf.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add(new BasicNameValuePair("f_username", "myemail#address.com"));
nameValuePairs.add(new BasicNameValuePair("f_passwd", "mypassword"));
nameValuePairs.add(new BasicNameValuePair("f_method", "LOGIN"));
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(new Scheme("https", sslf, 443));
SingleClientConnManager cm = new SingleClientConnManager(httppost.getParams(), schemeRegistry);
// NEW API WONT ALLOW THIS IN THE MAIN THREAD! hence ASYNC
DefaultHttpClient client = new DefaultHttpClient(cm, httppost.getParams());
HttpResponse result = client.execute(httppost);
// Check if server response is valid
StatusLine status = result.getStatusLine();
Log.d(TAG, "STatus" + result.getStatusLine());
if (status.getStatusCode() != 200) {
throw new IOException("Invalid response from server: " + status.toString());
}
HttpEntity entity = result.getEntity();
InputStream is = entity.getContent();
if (is != null) {
is.close(); // release connection
}
String phpsessid = "";
// cookies from another blog
// http://stackoverflow.com/questions/4224913/android-session-management
List cookies = client.getCookieStore().getCookies();
if (cookies.isEmpty()) {
Log.d(TAG, "no cookies received");
} else {
for (int i = 0; i < cookies.size(); i++) {
// Log.d(TAG, "COOKIE-" + i + " " +
// cookies.get(i).toString());
if (cookies.get(i).toString().contains("PHPSESSID")) {
phpsessid = cookies.get(i).toString();
Log.d(TAG, "COOKIE FOR PHPSESSID - " + phpsessid);
}
}
} // end of blog
entity.consumeContent();
client.getConnectionManager().shutdown();
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
} catch (IOException e) {
// TODO Auto-generated catch block
} catch (NoSuchAlgorithmException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (CertificateException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
return 1;
} // end of postData()
public class MySSLSocketFactory extends SSLSocketFactory {
SSLContext sslContext = SSLContext.getInstance("TLS");
public MySSLSocketFactory(KeyStore truststore)
throws NoSuchAlgorithmException, KeyManagementException,
KeyStoreException, UnrecoverableKeyException {
super(truststore);
TrustManager tm = new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] chain,
String authType) throws CertificateException {
}
public void checkServerTrusted(X509Certificate[] chain,
String authType) throws CertificateException {
}
public X509Certificate[] getAcceptedIssuers() {
return null;
}
};
sslContext.init(null, new TrustManager[] { tm }, null);
}
#Override
public Socket createSocket(Socket socket, String host, int port,
boolean autoClose) throws IOException, UnknownHostException {
return sslContext.getSocketFactory().createSocket(socket, host,
port, autoClose);
}
#Override
public Socket createSocket() throws IOException {
return sslContext.getSocketFactory().createSocket();
}
I know the url is correct, as are the name value pairs, as I can login via a query string via a browser or via wget:
https://domainname.com/nclogin.submit?f_username=myemail#email.com&f_passwd=password&f_method=LOGIN
This results in a connection established and a redirect to my dashboard page.
The HTML code (source) from the login page can be viewed
here

Resources