I've been working on a program written in Java running on Linux (x86) for years, now porting it to work on Linux (ARM). I am facing an issue and not sure where it comes from.
When running the example on Slackware 12.1 (x86) with
# java -version
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)
it is running perfectly fine. When running on Slackware 14.1 (ARM) with
# java -version
java version "1.7.0_13"
OpenJDK Runtime Environment (IcedTea7 2.3.6) (Slackware)
OpenJDK Zero VM (build 22.0-b10, mixed mode)
I get an exception :
# java -jar SpectoTechnologies_TestHTTP.jar
java.security.ProviderException: Error parsing configuration
at sun.security.pkcs11.Config.getConfig(Config.java:88)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:128)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:103)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:224)
at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
at sun.security.jca.ProviderList.getProvider(ProviderList.java:232)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:433)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:170)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:97)
at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:121)
at javax.net.ssl.HttpsURLConnection.getDefaultSSLSocketFactory(HttpsURLConnection.java:333)
at javax.net.ssl.HttpsURLConnection.<init>(HttpsURLConnection.java:291)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:85)
at sun.net.www.protocol.https.Handler.openConnection(Handler.java:62)
at java.net.URL.openConnection(URL.java:1020)
at com.spectotechnologies.tests.HTTP.main(HTTP.java:30)
Caused by: sun.security.pkcs11.ConfigurationException: name must be specified
at sun.security.pkcs11.Config.parse(Config.java:465)
at sun.security.pkcs11.Config.<init>(Config.java:216)
at sun.security.pkcs11.Config.getConfig(Config.java:84)
... 25 more
It is most likely the problem has something to do with code incompatibility with Oracle/Sun JRE vs OpenJDK... or something missing on Linux (SSL package or something), but find no answers about something similar.
I've made a SSCCE to reproduce the problem :
package com.spectotechnologies.tests;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.StringWriter;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.Proxy;
import java.net.URL;
/**
*
* #author Alexandre Lavoie
*/
public class HTTP
{
/**
* #param args the command line arguments
*/
public static void main(String[] args)
{
try
{
InputStream oInput;
URL oURL = new URL("https://google.com");
HttpURLConnection oConnection = (HttpURLConnection)oURL.openConnection(Proxy.NO_PROXY);
oConnection.setDoOutput(true);
oConnection.setDoInput(true);
oConnection.setRequestMethod("GET");
if(oConnection.getResponseCode() != 500)
{
oInput = oConnection.getInputStream();
}
else
{
oInput = oConnection.getErrorStream();
}
String sResult = convertStreamToString(oInput);
System.out.println(sResult);
}
catch(Exception e)
{
e.printStackTrace();
}
}
public static String convertStreamToString(java.io.InputStream is) throws IOException
{
if(is != null)
{
Writer writer = new StringWriter();
char[] buffer = new char[1024];
try
{
Reader reader = new BufferedReader(new InputStreamReader(is,"utf-8"));
int n;
while ((n = reader.read(buffer)) != -1)
{
writer.write(buffer, 0, n);
}
}
finally
{
is.close();
}
return writer.toString();
}
else
{
return "";
}
}
}
The error is raised when parsing the configuration file $JAVA_HOME/lib/security/java.security. Several bugs are related to this.
java.security.ProviderException: Error parsing configuration with space
sun.security.pkcs11.ConfigurationException: Unexpected value Token ['(']
My guess is that you have special characters or spaces in your configuration file which are correctly interpreted in one version of the jvm and not in the other version.
Related
Objective
I'm trying to write to Oracle's ADWC (basically oracle database) from a Spark application running on Yarn. The only way to connect to this database is by using an Oracle Wallet file, which is basically a Java keystore.
Problem
The problem arises when the JDBC driver tries to read the wallet from HDFS. If I include the hdfs:// prefix the parser in the JDBC driver throws an error and if I don't then it cannot find the file.
Previous Attempts
including the directory in the connect string (prefixed and non) jdbc:oracle:thin:#luigi_low?TNS_ADMIN=/user/spark/wallet_LUIGI
including the directory as an spark.driver.extraJavaOptions with -Doracle.net.tns_admin and -Doracle.net.wallet_location
All the code is on GitHub, and specifically, the error messages are here https://github.com/sblack4/kafka-scala-jdbc/blob/master/ERROR.md
I've got a working example of the same connection here https://github.com/sblack4/scala-jdbc-adwc
help me StackOverflow. you are my only hope
If you need any more clarification don't hesitate :)
update (SparkFiles attempt)
the code is on a separate branch of the same repository, https://github.com/sblack4/kafka-scala-jdbc/tree/sparkfiles
This error message mystifies me as it seems my JDBC library has stopped trying to read the wallet files. It may be unrelated to the previous problem
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: Invalid connection string format, a valid format is: "host:port:sid"
I've deleted the other JDBC libraries from my classpath through Ambari as this error could be related to spark picking up an older version of my JDBC library
Here's some code that will help diagnose what the issues is.
It checks and configures everything required to connect.
JDBC Driver version
JCE Installed
Classpath dependencies
Configures
tns_admin
ssl settings
trust/key stores
This is a slimmed down version of what's in sqldev/sqlcl
import java.security.NoSuchAlgorithmException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Collections;
import java.util.Properties;
import javax.crypto.Cipher;
import oracle.jdbc.OracleConnection;
public class JDBCTest {
public static void fail(String msg){
System.err.println(String.join("", Collections.nCopies(20, "*")));
System.err.println(msg);
System.err.println(String.join("", Collections.nCopies(20, "*")));
System.exit(1);
}
public static void main(String[] args) throws SQLException {
System.out.println("JDBC Driver Version:" + oracle.jdbc.OracleDriver.getDriverVersion());
// Check JDBC Driver Version
if (!oracle.jdbc.OracleDriver.getDriverVersion().startsWith("18.")) {
fail(" DRIVER TOOO OLD!!!");
}
// Check JCE Installed
int maxKeySize = 0;
try {
maxKeySize = Cipher.getMaxAllowedKeyLength("AES");
} catch (NoSuchAlgorithmException e) {
}
if (maxKeySize < 129 ) {
fail(" JCE Policy not unlimited!!!");
}
// Check Classpath
String cp = System.getProperty("java.class.path");
String[] cpFiles = {"ojdbc8.jar","oraclepki.jar","osdt_cert.jar","osdt_core.jar"};
for (String file:cpFiles){
if ( cp.indexOf(file) == -1 ){
fail("CLASSPATH Missing:" + file);
}
}
// Wallet unziped location
String unzippedWalletLocation = "/Users/klrice/workspace/12.2JDBC/wallet";
String conString = "jdbc:oracle:thin:#sqldev_medium";
Properties props = new Properties();
props.setProperty("oracle.net.wallet_location",unzippedWalletLocation);
props.setProperty(OracleConnection.CONNECTION_PROPERTY_THIN_NET_CONNECT_TIMEOUT, "2000");
// unzipped includes a tnsnames.ora
props.setProperty("oracle.net.tns_admin",unzippedWalletLocation);
props.setProperty("javax.net.ssl.trustStore","truststore.jks");
props.setProperty("javax.net.ssl.trustStorePassword","<password>");
props.setProperty("javax.net.ssl.keyStore","keystore.jks");
props.setProperty("javax.net.ssl.keyStorePassword","<password>");
props.setProperty("oracle.net.ssl_server_dn_match","true");
props.setProperty("oracle.net.ssl_version","1.2");
props.setProperty("user", "ADMIN");
props.setProperty("password", "<password>");
try {
// now Connect
Connection conn = DriverManager.getConnection(conString,props);
} catch (Exception e){
e.printStackTrace();
fail(e.getLocalizedMessage());
}
System.out.println("SUCCESS!!");
}
}
Are you using 18.3 JDBC drivers? Passing TNS_ADMIN as part of the connection URL requires 18.3 JDBC driver. Also, are you attempting to connect within the corporate network. In that case, you will need to pass HTTPS_PROXY and HTTPS_PROXY_PORT in the connection URL. Let us know. Happy to help with the problem.
This question already has answers here:
How to add JAR libraries to WAR project without facing java.lang.ClassNotFoundException? Classpath vs Build Path vs /WEB-INF/lib
(5 answers)
Closed 7 years ago.
The code below fails on the line:
Class.forName("oracle.jdbc.driver.OracleDriver");
with the error:
java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
The two printlns print:
Wed_Jun_22_11:18:51_PDT_2005
false
This makes me think the class exists and can be found. Also this exact same class works in an a non-servlet application.
I have rebooted everything multiple times and regenerated the application/servlet multiple times. All values have been hard coded to make it simple and short.
private static Connection getDBConnection() throws Exception {
System.out.println(oracle.jdbc.driver.OracleDriver.BUILD_DATE);
System.out.println(Class.class.desiredAssertionStatus());
//load the driver
Class.forName("oracle.jdbc.driver.OracleDriver");
return DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521:orcl", "SYSTEM", "pass");
}
full servlet that fails:
package servletClass_3;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* Servlet implementation class OneMoreBookStore
*/
#WebServlet("/OneMoreBookStore")
public class OneMoreBookStore extends HttpServlet {
private static final long serialVersionUID = 1L;
/**
* #see HttpServlet#HttpServlet()
*/
private static Connection getDBConnection() throws Exception {
System.out.println(oracle.jdbc.driver.OracleDriver.BUILD_DATE);
System.out.println(Class.class.desiredAssertionStatus());
//load the driver
Class.forName("oracle.jdbc.driver.OracleDriver");
return DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521:orcl", "SYSTEM", "pass");
}
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
try
{
Connection con = getDBConnection();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
This application works:
package servletClass_3;
import java.sql.Connection;
import java.sql.DriverManager;
public class DBConnect {
private static Connection getDBConnection() throws Exception {
System.out.println(oracle.jdbc.driver.OracleDriver.BUILD_DATE);
System.out.println(Class.class.desiredAssertionStatus());
//load the driver
Class.forName("oracle.jdbc.driver.OracleDriver");
return DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521:orcl", "SYSTEM", "pass");
}
public static void main(String[] args) {
try
{
Connection con = getDBConnection();
System.out.println("connection worked");
con.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
I'm using:
Eclipse JavaEE 1.4.2
Tomcat 7
jdk1.7
Oracle 11g R2
Windows 7 64bit
Probably you aren't deploying the oracle driver with your application.
You have several options:
You can place the driver jars in your WEB-INF/lib folder
You export it with your application. -> Right Click on Project -> Build Path-> Configure Build Path... -> Order and Export -> Check the drivers.
Place the driver jars in a shared or library extension folder of your application server. (You should go with option one or two though.)
You must include the ojdbc6.jar file in the Deployment Assembly of the Project...
select the web project which contains the jsp file...
select Project tab in the menu bar in Eclipse
select properties in the drop down menu
select Deployment Assembly
Add your ojdbc6.jar file in it.
Try this, change the oracle.jdbc.driver.OracleTypes to oracle.jdbc.OracleTypes
I have been trying to get a grip on the spring boot config server that is located Here: https://github.com/spring-cloud/spring-cloud-config and after reading the documentation more thoroughly I was able to work through most of my issues. I did however have to write an additional class for a file based PropertySourceLocator
/*
* Copyright 2013-2014 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.config.client;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Iterator;
import java.util.Properties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.core.env.PropertiesPropertySource;
import org.springframework.core.env.PropertySource;
import org.springframework.util.StringUtils;
/**
* #author Al Dispennette
*
*/
#ConfigurationProperties("spring.cloud.config")
public class ConfigServiceFilePropertySourceLocator implements PropertySourceLocator {
private Logger logger = LoggerFactory.getLogger(ConfigServiceFilePropertySourceLocator.class);
private String env = "default";
#Value("${spring.application.name:'application'}")
private String name;
private String label = name;
private String basedir = System.getProperty("user.home");
#Override
public PropertySource<?> locate() {
try {
return getPropertySource();
} catch (IOException e) {
logger.error("An error ocurred while loading the properties.",e);
}
return null;
}
/**
* #throws IOException
*/
private PropertySource getPropertySource() throws IOException {
Properties source = new Properties();
Path path = Paths.get(getUri());
if(Files.isDirectory(path)){
Iterator<Path> itr = Files.newDirectoryStream(path).iterator();
String fileName = null!=label||StringUtils.hasText(label)?label:name+".properties";
logger.info("Searching for {}",fileName);
while(itr.hasNext()){
Path tmpPath = itr.next();
if(tmpPath.getFileName().getName(0).toString().equals(fileName)){
logger.info("Found file: {}",fileName);
source.load(Files.newInputStream(tmpPath));
}
}
}
return new PropertiesPropertySource("configService",source);
}
public String getUri() {
StringBuilder bldr = new StringBuilder(basedir)
.append(File.separator)
.append(env)
.append(File.separator)
.append(name);
logger.info("loading properties directory: {}",bldr.toString());
return bldr.toString();
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getEnv() {
return env;
}
public void setEnv(String env) {
this.env = env;
}
public String getLabel() {
return label;
}
public void setLabel(String label) {
this.label = label;
}
public String getBasedir() {
return basedir;
}
public void setBasedir(String basedir) {
this.basedir = basedir;
}
}
Then I added this to the ConfigServiceBootstrapConfiguration.java
#Bean
public PropertySourceLocator configServiceFilePropertySource(
ConfigurableEnvironment environment) {
ConfigServiceFilePropertySourceLocator locator = new ConfigServiceFilePropertySourceLocator();
String[] profiles = environment.getActiveProfiles();
if (profiles.length==0) {
profiles = environment.getDefaultProfiles();
}
locator.setEnv(StringUtils.arrayToCommaDelimitedString(profiles));
return locator;
}
In the end this did what I wanted.
Now I'm curious to know if this is what I should have done or if I am still missing something and this was already handled and I just missed it.
*****Edit for info asked for by Dave******
If I take out the file property source loader and update the bootstrap.yml with
uri: file://${user.home}/resources
the sample application throws the following error on start up:
ConfigServiceBootstrapConfiguration : Could not locate PropertySource: Object of class [sun.net.www.protocol.file.FileURLConnection] must be an instance of class java.net.HttpURLConnection
This is why I thought the additional class would be needed. As far as the test case goes I believe you are talking about the SpringApplicationEnvironmentRepositoryTests.java and I agree creating the environment works but as a whole the application does not seem to be opertaing as expected when the uri protocol is 'file'.
******Additional Edits*******
This is how I understanding this is working:
The sample project has a dependency on the spring-cloud-config-client artifact so therefore has a transitive dependency on the spring-cloud-config-server artifact.
The ConfigServiceBootstrapConfiguration.java in the client artifact creates a property source locator bean of type ConfigServicePropertySourceLocator.
The ConfigServicePropertySourceLocator.java in the config client artifact has the annotation #ConfigurationProperties("spring.cloud.config")
And the property uri exists in said class, hence the setting of spring.cloud.config.uri in the bootstrap.yml file.
I believe this is reenforced up by the following statement in the quickstart.adoc:
When it runs it will pick up the external configuration from the
default local config server on port 8888 if it is running. To modify
the startup behaviour you can change the location of the config server
using bootstrap.properties (like application.properties but for
the bootstrap phase of an application context), e.g.
---- spring.cloud.config.uri: http://myconfigserver.com
At this point, some how the JGitEnvironmentRepository bean is getting used and looking for a connection to github.
I assumed that since uri was the property being set in the ConfigServicePropertySourceLocator then any valid uri protocol would work for pointing to a location.
That is why I used the 'file://' protocol thinking that the server would pick up the NativeEnvironmentRepository.
So at this point I'm sure I'm either missing some step or the file system property source locator needs to be added.
I hope that is a little clearer.
the Full Stack:
java.lang.IllegalArgumentException: Object of class [sun.net.www.protocol.file.FileURLConnection] must be an instance of class java.net.HttpURLConnection
at org.springframework.util.Assert.isInstanceOf(Assert.java:339)
at org.springframework.util.Assert.isInstanceOf(Assert.java:319)
at org.springframework.http.client.SimpleClientHttpRequestFactory.openConnection(SimpleClientHttpRequestFactory.java:182)
at org.springframework.http.client.SimpleClientHttpRequestFactory.createRequest(SimpleClientHttpRequestFactory.java:140)
at org.springframework.http.client.support.HttpAccessor.createRequest(HttpAccessor.java:76)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:541)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:506)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:448)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:68)
at org.springframework.cloud.bootstrap.config.ConfigServiceBootstrapConfiguration.initialize(ConfigServiceBootstrapConfiguration.java:70)
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:572)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:952)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:941)
at sample.Application.main(Application.java:20)
I read this thread yesterday and it was missing a vital piece of information
If you don't want to use git as a repository, then you need to configure the spring cloud server to have spring.profiles.active=native
Checkout the spring-config-server code to understand it
org.springframework.cloud.config.server.NativeEnvironmentRepository
spring:
application:
name: configserver
jmx:
default_domain: cloud.config.server
profiles:
active: native
cloud:
config:
server:
file :
url : <path to config files>
I just came cross same issue. I want the configuration server load properties from local file system instead of git repository. The following configuration works for me on windows.
spring:
profiles:
active: native
cloud:
config:
server:
native:
searchLocations: file:C:/springbootapp/properties/
Suppose the property file is under C:/springbootapp/properties/
For more information please refer to Spring Cloud Documentation and Configuring It All Out
I think I have the final solution based on your last comments
In the configserver.yml I added
spring.profiles.active: file
spring.cloud.config.server.uri: file://${user.home}/resources
In the ConfigServerConfiguration.java I added
#Configuration
#Profile("file")
protected static class SpringApplicationConfiguration {
#Value("${spring.cloud.config.server.uri}")
String locations;
#Bean
public SpringApplicationEnvironmentRepository repository() {
SpringApplicationEnvironmentRepository repo = new SpringApplicationEnvironmentRepository();
repo.setSearchLocations(locations);
return repo;
}
}
And I was able to view the properties with:
curl localhost:8888/bar/default
curl localhost:8888/foo/development
I have a simple java program called putmerge that I am trying to execute. I have been at it for like 6hrs, researched many places on the web but could not find solution. Basically I try to build the jar with all class libraries with the following command:
javac -classpath *:lib/* -d playground/classes playground/src/PutMerge.java
And then I build the jar with the following command.
jar -cvf playground/putmerge.jar -C playground/classes/ .
And then I try to execute it with the following command:
bin/hadoop jar playground/putmerge.jar org.scd.putmerge "..inputPath.." "..outPath"
..
Exception in thread "main" java.lang.ClassNotFoundException: com.scd.putmerge
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
I tried every permutation/combination to run this simple jar, however I always get some kind of exception as shown above.
My source code:
package org.scd.putmerge;
import java.io.IOException;
import java.util.Scanner;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
/**
*
* #author Anup V. Saumithri
*
*/
public class PutMerge
{
public static void main(String[] args) throws IOException
{
Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
FileSystem local = FileSystem.getLocal(conf);
Path inputDir = new Path(args[0]);
Path hdfsFile = new Path(args[1]);
try
{
FileStatus[] inputFiles = local.listStatus(inputDir);
FSDataOutputStream out = hdfs.create(hdfsFile);
for(int i=0; i<inputFiles.length; i++)
{
System.out.println(inputFiles[i].getPath().getName());
FSDataInputStream in = local.open(inputFiles[i].getPath());
byte buffer[] = new byte[256];
int bytesRead = 0;
while((bytesRead = in.read(buffer)) > 0)
{
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
}
catch(IOException ex)
{
ex.printStackTrace();
}
}
}
The way you are putting your PutMerge class inside the jar may be a little incorrect.
If you do a jar tf putmerge.jar, you must see the PutMerge class inside the path mentioned in your package (org.scd.putmerge) in your code (i.e. org/scd/putmerge).
If not try doing the following to achieve that. Make sure you have copied PutMerge.class inside org/scd/putmerge/ directory.
jar -cvf playground/putmerge.jar org/scd/putmerge/PutMerge.class
Next, verify again with jar tf putmerge.jar to check if now see org/scd/putmerge/PutMerge.class in the output.
If everything's fine, you can try to run the hadoop jar again. But looking at the errors, I see that you haven't actually included the PutMerge class with the package. You should use org.scd.putmerge.PutMerge. So, the correct way should be something like --
bin/hadoop jar playground/putmerge.jar org.scd.putmerge.PutMerge "..inputPath.." "..outPath"
I want to store the data in hdfs which is emitted by Storm Spout. I have added hadoop FS API code in Bolt Class, but It is throwing compilation error with storm.
Following is the Storm bolt Class :
package bolts;
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
public class DataNormalizer extends BaseBasicBolt {
public void execute(Tuple input, BasicOutputCollector collector) {
String sentence = input.getString(0);
String[] process = sentence.split(" ");
int n = 1;
String rec = "";
try {
String filepath = "/root/data/top_output.csv";
String dest = "hdfs://localhost:9000/user/root/nishu/top_output/top_output_1.csv";
Configuration conf = new Configuration();
FileSystem fileSystem = FileSystem.get(conf);
System.out.println(fileSystem);
Path srcPath = new Path(source);
Path dstPath = new Path(dest);
String filename = source.substring(source.lastIndexOf('/') + 1,
source.length());
try {
if (!(fileSystem.exists(dstPath))) {
FSDataOutputStream out = fileSystem.create(dstPath, true);
InputStream in = new BufferedInputStream(
new FileInputStream(new File(source)));
byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) > 0) {
out.write(b, 0, numBytes);
}
in.close();
out.close();
} else {
fileSystem.copyFromLocalFile(srcPath, dstPath);
}
} catch (Exception e) {
System.err.println("Exception caught! :" + e);
System.exit(1);
} finally {
fileSystem.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
I have added hadoop jars in CLASSPATH also..
Following is the value of classpath :
$STORM_HOME/storm-0.8.1.jar:$JAVA_HOME/lib/:$HADOOP_HOME/hadoop-core-1.0.4.jar:$HADOOP_HOME/lib/:$STORM_HOME/lib/
Also copied hadoop libraries : hadoop-cor-1.0.4.jar, commons-collection-3.2.1.jar and commons-cli-1.2.jar in Storm/lib directory.
When I am building this project, It is throwing following error :
3006 [Thread-16] ERROR backtype.storm.daemon.executor -
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:184)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:236)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:466)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
at bolts.DataNormalizer.execute(DataNormalizer.java:67)
at backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:32)
......................
The error message tells you that Apache commons configuration is missing. You have to add it to the classpath.
More generally, you should add all Hadoop dependencies to your classpath. You can find them using a dependency manager (Maven, Ivy, Gradle etc.) or look into /usr/lib/hadoop/lib on a machine on which Hadoop is installed.