I'm able to submit a topology job in the multi-tenant cluster. The job is running. However, the logviewer page is not available. Is there any way to solve this issue?
you need to start the logviewer before you click on topology port to see logviewer.
To start logviewer run:
$ storm logviewer same as you run $ storm list
I faced the same issue for logviewer's home page, but directly navigating to a particular log file that exists in the logs folder works. Try this:
MachineIP:8000/log?file=worker-6700.log
Related
I'm running a custom Yarn Application using Apache Twill in HDP 2.5 cluster, but I'm not able to see my own container logs (syslog, stderr and stdout) when I go to my container web page:
Also the login changes from my kerberos to "dr.who" when I navigate to this page.
But I can see the logs of map-reduce jobs. Hadoop version is 2.7.3 and the cluster is yarn acl enabled.
i had this issue with hadoop ui. I found in this doc, that the hadoop.http.staticuser.user is set to dr.who by default and you need include it in the related setting file (in my issue is core-site.xml file).
so late but hope useful.
in the mapreduce webconsole for each application there is a tracking ui link which points to xx:19888/jobhistory/, but how to start the service on 19888 (i have started 4 services: yarn-resource-manager, yarn-node-manager, hdfs-name-node, hdfs-data-node, what i have missed?)
is the jobtracker removed in 2.5.2
I want to check the job.xml generated for my job, where can i find it. I have specified "mapreduce.jobtracker.jobhistory.location" but nothing is there
Thank you.
To access the JobHistory server's web interface then you have start the hadoop-mapreduce-historyserver service, which will bind to 19888 by default.
If your are running YARN in the cluster then you don't need jobtracker anymore, the work done jobtracker is offloaded to ResourceManager, NodeManager's & ApplicationMaster's. But, you could still install just MRv1 in that case you will install JobTracker and TaskTracker's (which is not recommended).
You could check the job.xml from the ResourceManager's UI by navigating to http://RESOURCEMANAGER_HOST:8088/cluster -> select your application's Tracking UI -> Select your Job ID -> On the left tab you'll be able to see Configuration. Or simply if you already know you'r job's id then visit this link: http://JOBHISTORY_SERVER:19888/jobhistory/conf/YOUR_JOB_ID.
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh --config $HADOOP_HOME/etc/hadoop start historyserver
then run jps to see if it is running.
Please try 'jps' command to verify all the services are running.
try //jhs_host:port/ Default HTTP port is 19888."
Make sure that your hadoop services are running.
I installed hadoop 2.5.2, and I can run the wordcount sample successfully. However, when I want to see the application running on yarn (job running), I cannot as all applictaions interface is always empty (shown in the following screen).
Is there anyway to make the jobs visible?
Please try localhost:19888 or check value of the the property for web url for job history (mapreduce.jobhistory.webapp.address) configured in you yarn config file.
I followed the tutorialshttp://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/SingleCluster.html and installed hadoop 2.4.1 as pseudo distributed cluster. I created a ubuntu VM using OracleVM and installed hadoop as mentioned in the link. It was setup fine and able to run the examples. However the job tracker URL is not working. :50030 gives page not found. I also tried netstat on the server and there is no process waiting on 50030 port . Do i need to start any other service ? What are the possible reasons ?
You need to execute this:
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
Or JobTracker won't start.
(In my case, $HADOOP_HOME is in /usr/local/hadoop)
Check the value of mapred.job.tracker.http.address in mapred-site.xml
If the port is different, use that.
Also check if jobtracker is running. Check the jobtracker logs.
You need to enter the following command
http://localhost:50030/
Job Tracker web UI.
I have configured Storm on my machine. Zookeeper, Nimbus and Supervisor are running properly.
Now I want to submit a topology to this storm.
I am trying to use storm jar.
but I am not able to submit it.
Can anybody please give an example for this.
It will be very helpful.
Thanks in advance:)
The answer is in the official documentation, and it is clear enough. Run storm jar path/to/allmycode.jar org.me.MyTopology arg1 arg2 arg3 (replace with your project name and arguments if any). Make sure you are using StormSubmitter object instead of LocalCluster.
Unfortunately, almost all the examples on the internet show the word counter example, and do not mention the steps required in a simple way:
All you need to do is this:
1. Navigate to your storm bin folder:
cd /Users/nav/programming/apache-storm-1.0.1/bin
2. Start nimbus
./storm nimbus
3. Start supervisor
./storm supervisor
4. Start the ui program
./storm ui
5. Make sure you build your jar file with the storm jar excluded from it.
6. Make sure your /Users/nav/programming/apache-storm-1.0.1/conf/storm.yaml file is valid (this should've been step 2).
7. Make sure that in your code, you are submitting the topology using StormSubmitter.submitTopology
8. Navigate to the storm bin folder again
cd /Users/nav/programming/apache-storm-1.0.1/bin
9. Submit your jar file to storm
./storm jar /Users/nav/myworkspace/StormTrial/build/libs/StormTrial.jar com.abc.stormtrial.StormTrial
The above command is basically just this:
stormExecutable jarOption pathToYourJarFile theClassContainingYourMainFile
If you want to pass commandline arguments to your program, add it at the end:
stormExecutable jarOption pathToYourJarFile theClassContainingYourMainFile commandlineArguments
Here, com.abc.stormtrial is the full package name and .StormTrial is the name of the class that contains your main function.
Now open up your browser and type http://127.0.0.1:8080 and you'll see your topology running via Storm's UI.