I'm a new on Spark. When I install the Spark on Cloudera, it appears this error:
Error: Could not find or load main class org.apache.spark.deploy.history.HistoryServer
The last lines of the log file are:
/user/spark/applicationHistory
Thu Sep 10 23:43:50 CST 2015
Thu Sep 10 23:43:50 CST 2015: Detected CDH_VERSION of [5]
Thu Sep 10 23:43:50 CST 2015: Starting Spark History Server
Error: Could not find or load main class org.apache.spark.deploy.history.HistoryServer
My cloudera manager version is 5.2.5 and CDH version is 5.0.0.
I have installed Yarn, HDFS and zookeeper, and they works well so far.
And there is a spark jar (spark-assembly_2.10-0.9.0-cdh5.0.0-hadoop2.3.0-cdh5.0.0.jar) at:
/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/spark/assembly/lib/
Thanks!
Related
I am facing an issue from past few days,
If I use Hadoop 2.7.4, I am not able to start nodemanager's at slaves with this version, because of this I can't run any mapred jobs.
When I use Hadoop 2.8.2, everything is starting well(start-dfs.sh and start-yarn.sh) and I am able to see the content, nodes activity at http://:50070, but while running my programs that use data from HDFS, it keeps on displaying
17/11/15 12:51:46 WARN hdfs.DFSClient: zero
but the job runs well. I am not aware of this issue and how it is effected. So, I tried 2.7.3
When I tried 2.7.3, I am not facing the above error, but I see that the yarn are not starting completing and when I start yarn, it throws more line on screen that usual way and it starts. The main issue, in this case, I am not able to watch watch going on in hadoop from web url as here it doesn't display anything on web, except for those files present in HDFS(So, I am only able see the data from Utilities --> Browse files )
The error is similar to this:
starting yarn daemons
starting resourcemanager, logging to
/usr/local/hadoop/logs/yarn--resourcemanager-hadoop-master.out
Nov 20, 2017 8:01:28 AM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
register
INFO: Registering
org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver
as a provider class
Nov 20, 2017 8:01:28 AM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
register
INFO: Registering
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices as
a root resource class
Nov 20, 2017 8:01:28 AM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
register
INFO: Registering
org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider
class
Nov 20, 2017 8:01:28 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011
11:17 AM'
Nov 20, 2017 8:01:28 AM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver
to GuiceManagedComponentProvider with the scope "Singleton"
hadoop-slave1: Warning: Permanently added 'hadoop-slave1,10.40.0.0'
(ECDSA) to the list of known hosts.
hadoop-slave1: starting nodemanager, logging to
/usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-slave1.weave.local.out
Any idea how to set it up completely without any issue?
I'm using EAServer 5.5 on Windows7 and the server is running with JDK 1.5
Now I want to UpGrade JDK1.5 to 1.8. Can you please some one help me.
I am getting the following error.
DynamicLibrary::lookup: af_dll_lookup (libjjdk12.dll, new_JagComponent) failed (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\DynamicLibrary.cc#80)
000095 Tue Jul 21 12:50:32 2015 E [018286] CTS_JagComponent::load: failed to load component model library (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_JagComponent.cc#841)
000096 Tue Jul 21 12:50:32 2015 E [018918] CTS_Component::load: failed for component 'Jaguar/Repository' (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_Component.cc#1993)
000097 Tue Jul 21 12:50:32 2015 A [099999] SystemException: OBJECT_NOT_EXIST (CosNaming/NamingContext/resolve - #0.0.0.0) (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_Object.cc#3995)
000098 Tue Jul 21 12:50:32 2015 E [018532] Exception 'CORBA::OBJECT_NOT_EXIST' in Session::create for component 'Jaguar/Repository' (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\Session.cc#1029)
000099 Tue Jul 21 12:50:32 2015 A [099999] SystemException: OBJECT_NOT_EXIST (Session/create - #0.0.0.0) (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_Object.cc#3995)
000100 Tue Jul 21 12:50:32 2015 A [099999] new_JagObjectRef: Bad URL - Jaguar/Repository (libjdispatch.iiop) (Y:\calm\conn\itg\jagsrv\generic\cc\JagORB.c#1074)
000101 Tue Jul 21 12:50:32 2015 E [018480] SEVERE ERROR - failed to access Jaguar Repository (libjdispatch.repository) (Y:\calm\conn\itg\jagsrv\generic\dispatch\Repository.cc#114)
000102 Tue Jul 21 12:50:32 2015 E [018488] Warning: failed to lookup properties for Package'CosNaming'(CORBA::INV_OBJREF) (libjdispatch.repository) (Y:\calm\conn\itg\jagsrv\generic\dispatch\Repository.cc#187)
000103 Tue Jul 21 12:50:32 2015 E [018477] Missing value for Component 'CosNaming/NamingContext' property 'com.sybase.jaguar.component.type' (sybase.eas.global) (Y:\calm\conn\itg\jagsrv\generic\dispatch\Properties.cc#151)
000104 Tue Jul 21 12:50:32 2015 E [018918] CTS_Component::load: failed for component 'CosNaming/NamingContext' (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_Component.cc#1993)
000105 Tue Jul 21 12:50:32 2015 A [099999] SystemException: OBJECT_NOT_EXIST (CosNaming/NamingContext/resolve - #0.0.0.0) (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_Object.cc#3995)
000106 Tue Jul 21 12:50:32 2015 E [018532] Exception 'CORBA::OBJECT_NOT_EXIST' in Session::create for component 'Jaguar/JavaInit' (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\Session.cc#1029)
000107 Tue Jul 21 12:50:32 2015 A [099999] SystemException: OBJECT_NOT_EXIST (Session/create - #0.0.0.0) (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\CTS_Object.cc#3995)
000108 Tue Jul 21 12:50:32 2015 A [099999] new_JagObjectRef: Bad URL - Jaguar/JavaInit (libjdispatch.iiop) (Y:\calm\conn\itg\jagsrv\generic\cc\JagORB.c#1074)
000109 Tue Jul 21 12:50:32 2015 F [018506] SEVERE ERROR - failed to access Jaguar/JavaInit (libjdispatch) (Y:\calm\conn\itg\jagsrv\generic\dispatch\Server.cc#418)
I don't think this is possible or recommended.
OP was asking for upgrading Java (inside) used by EAServer (app server like JBoss, but for PowerBuilder+Java). Most of the Java classes inside EAServer were compiled to Java 1.2 and support only up to Java 1.5. I recently tried to upgrade my EAS 5.5 to JDK 1.6. This caused above issues you mentioned.
Several things are at play here:
Deprecated Java classes, methods, models.
JDBC interface itself is different now. For Oracle, I had to use OJBC6 and that breaks with older databases.
In the end, decided to simply upgrade JDK to 1.5.0_22, max build for 1.5 (Sybase's own limit was 1.5.0_03).
That said, if you still want to try, you will have to update a few batch files (I am assuming you are on windows) in \bin.
You may have to install/copy the JDK inside Sybase\Shared
folder, so you can customize, if needed.
You will need to add new JDK entries into setenv.bat and
serverstart.bat. Basically, current batches are agnostic about the
new JDK versions you are introducing to the EAServer.
You may have to update user_setenv.bat to customize JDBC/classpaths.
You may also have to generate batch file for Jaguar Manager and
change Java directory used to launch it.
It didn't work for me. But, if you are able to make it to work, can you please post back with your suggestions here?
Sorry for the late reply. Been busy with a project. Are you still having the problem? That essentially says the service did not start. Did you look at the Jaguar.log?
I suggest, you start it using the batch file first -
\EAServer\bin\serverstart.bat -jdk18
I hope you have added jdk18 options in the batch files. And watch the below logs:
Jaguarout.log, Jaguar_performance.log (if found), Jaguar.log
Once you get it to work, then you can use Serverstart with -install option to start as a service.
I just want to reiterate, that even if you get it to start, you may have issues with connection caches (particularly JDBC) and other interfaces as the specifications changed a lot since Java 1.4. If you are getting issues there, you will have to update the OJDBC driver etc. Good luck!
I want to run WordCount Example.
In eclipse it run correctly. In output folder the output file is present.
I made a jar file of WordCount and want to run it through command
hadoop jar WordCount.jar /Projects/input /Projects/output
it gives me error
Not a valid JAR: /Projects/WordCount.jar
result of hdfs dfs -ls /Projects
Found 3 items
-rw-r--r-- 1 hduser supergroup 3418 2014-11-02 15:38 /Projects/WordCount.jar
drwxr-xr-x - hduser supergroup 0 2014-11-02 14:13 /Projects/input
drwxr-xr-x - hduser supergroup 0 2014-11-02 14:16 /Projects/output
it gives me same error on this also
hadoop jar /Projects/WordCount.jar wordPackage.WordCount /Projects/input /Projects/output
Not a valid JAR: /Projects/WordCount.jar
how to solve this error.
I have run tvf command it gives this output
jar -tvf /home/hduser/Desktop/Files/WordCount.jar
60 Sun Nov 02 16:10:10 PKT 2014 META-INF/MANIFEST.MF
1895 Sun Nov 02 14:02:38 PKT 2014 wordPackage/WordCount.class
1295 Sun Nov 02 14:02:38 PKT 2014 wordPackage/WordCount.java
2388 Sun Nov 02 14:02:06 PKT 2014 wordPackage/WordReducer.class
707 Sun Nov 02 14:02:06 PKT 2014 wordPackage/WordReducer.java
2203 Sun Nov 02 14:02:08 PKT 2014 wordPackage/WordMapper.class
713 Sun Nov 02 14:02:06 PKT 2014 wordPackage/WordMapper.java
16424 Sun Nov 02 13:50:00 PKT 2014 .classpath
420 Sun Nov 02 13:50:00 PKT 2014 .project
You cannot keep the jar in HDFS when executing the same using hadoop command, Jar should be available in the local path
If the jar is not runnable try the following (Need to specify the package.mainclass)
hadoop jar /home/hduser/Desktop/Files/WordCount.jar wordPackage.WordCount /Projects/input /Projects/output
If the jar is runnable following can be used
hadoop jar /home/hduser/Desktop/Files/WordCount.jar /Projects/input /Projects/output
If the issue still persists, you need to rebuild this jar(WordCount.jar) in eclipse again
in spite of rebuilding your jar again if issue still persists, Please make sure you have given +x(execution chmod 755) privileges before you run the command. in my case this was the reason for the issue.
command help:
chmod +x jarname.jar
I faced the same issue. But in my case what I did was I wrote the java code referring to hadoop 1.x libraries and tried to execute it using 2.x. Initially, I got the same error in the terminal. Then I tried navigating to the path where I had my jar file. It worked.
May be you can actually try the following:(after navigating to the jar files path)
hadoop jar WordCount.jar wordPackage.WordCount /Projects/input /Projects/output
I'm installing TeamCity in EC2, starting with the Server then moving on the agents. I'm starting with the Amazon Linux AMI, running on a micro instance. Then I did:
sudo yum update
wget http://download.jetbrains.com/teamcity/TeamCity-7.1.1.tar.gz
tar -xvzf TeamCity-7.1.1.tar.gz
cd TeamCity
bin/teamcity-server.sh start
When I start it using bin/teamcity-server.sh start, things happen. I can connect using a web browser which shows the 'TeamCity is starting' page. The teamcity-server.log shows a bunch of activity, unzipping plugins etc.
But then suddently, the server process just disappears. The port's no longer listened to, ps shows no java process running, and the browser can't connect.
There's no error messages in the catalina or teamcity logs. After much trial and error though, I ran bin/teamcity-server.sh run (instead of start) to get console output, and got the following:
Using CATALINA_BASE: /home/ec2-user/TeamCity
Using CATALINA_HOME: /home/ec2-user/TeamCity
Using CATALINA_TMPDIR: /home/ec2-user/TeamCity/temp
Using JRE_HOME: /usr/lib/jvm/jre
Using CLASSPATH: /home/ec2-user/TeamCity/bin/bootstrap.jar:/home/ec2-user/TeamCity/bin/tomcat-juli.jar
Nov 1, 2012 7:22:25 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Nov 1, 2012 7:22:26 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8111"]
Nov 1, 2012 7:22:26 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2742 ms
Nov 1, 2012 7:22:26 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Nov 1, 2012 7:22:26 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.23
Nov 1, 2012 7:22:26 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /home/ec2-user/TeamCity/webapps/ROOT
Log4J configuration file /home/ec2-user/TeamCity/bin/../conf/teamcity-server-log4j.xml will be monitored with interval 10 seconds.
Nov 1, 2012 7:22:30 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8111"]
Nov 1, 2012 7:22:30 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3786 ms
=======================================================================
TeamCity 7.1.1 (build 24074) initialized, OS: Linux, JRE: 1.6.0_24-b24
TeamCity is running in professional mode
bin/teamcity-server.sh: line 18: 4231 Killed ./catalina.sh $1
I promise that I did not kill the process! I can find my way around in Linux well enough, but I'm not at all sure where to go next to find out why or what killed the process. Can anyone help?
After some further scanning of .sh files to see how TeamCity was starting itself up, I noticed that it was grabbing a fair amount of memory for it's java process (either 512m or 750m depending on which line you use).
The EC2 micro instance only has 613m of RAM total. When I realized this, I tried the whole process again with a larger instance, and things worked fine.
I'm still curious if there's a better way I could've known what was causing catalina to die, so if anyone wants to answer with that information...
I installed Xampp on my Windows Server 2008. It worked fine, untill I decided to install some updates.
Now Apache doesn't start any more and I get these errors;
[Wed Aug 29 23:31:20.328125 2012] [core:warn] [pid 1540:tid 312] AH00098: pid file
C:/xampp/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Aug 29 23:31:20.968750 2012] [ssl:warn] [pid 1540:tid 312] AH01873: Init:Session Cache is not configured [hint: SSLSessionCache]
I'm trying to install Cacti on the server to monitor everything... Don't think it's relevant, but just saying
If you still need it:
http://httpd.apache.org/docs/2.4/upgrading.html#post_234
Apache 2.4 will not start with added SSL support on windows until additional module (compare to Apache 2.2 configuration) was enabled: socache_shmcb_module
This additional info is at the beginning of the httpd-ssl.conf, but it takes time to read it :)
# Required modules: mod_log_config, mod_setenvif, mod_ssl, socache_shmcb_module (for default value of SSLSessionCache)