When I am trying to open the history of any of the spark job I am facing this issue "Application_id: Application application_xyz not found",
NOTE:
previously I figured out this as one of the spark history folder was full so this error occurred but now I don't remember how to do it?
Any Help is much appreciated.
Tobe able to access Spark UI after application has finished you need a separate history server.
Please start the server
$SPARK_HOME/sbin/start-history-server.sh
and follow configuration notes.
Related
I am unable to create a new Common Data Service Database in my Power Apps default environment. Please see the error text below.
It looks like you don't have permission to use the Common Data Service
in this environment. Switch to a different environment, or create your
own.
Which as I understand I should be able to create after the Microsoft Business Application October 2018 update as listed in the article available at following link.
https://community.dynamics.com/365/b/dynamicscitizendeveloper/archive/2018/10/17/demystifying-dynamics-365-and-powerapps-environments-part-1
Also when I try to create a Common Data Service app in my default environment, I encounter following error.
The data did not load correctly. Please try again.
The environment 'Default-57e1485d-1197-4afd-b792-5c423ab508d9' is not
linked to a new CDS 2.0 instance. The operation 'ListInstanceMetadata'
is forbidden for unlinked environments
Moreover I am unable to see the default environment on https://admin.powerapps.com/environments, I can only see the Sandbox environment there.
Any ideas what I am missing here?
Thank you.
Someone else faced a similar issue and I read in one of the threads about deleting the browser cache and trying it again or trying it in a different browser resolved the issue. Could you try these first level steps and check if you still have these issues?
Ref: https://powerusers.microsoft.com/t5/Common-Data-Service-for-Apps/Default-Environment-Error-on-CDS/m-p/233582#M1281
Also, for your permission error ref: https://powerusers.microsoft.com/t5/Common-Data-Service-for-Apps/Common-Data-Service-Business-Flows/td-p/142053
I have not validated these findings. But as these answers are from MS and PowerApps team, hope it helps!
I recently upgraded HDP from 2.6.5 to 3.1.0, which runs YARN 3.1.0, and I can no longer kill applications from the YARN ResourceManager UI, using either the old (:8088/cluster/apps) or new (:8088/ui2/index.html#/yarn-apps/apps) version. I can still kill them using the shell in RHEL 7 with yarn app -kill {app-id}
These applications are submitted via Livy. Here is my workflow:
Open the ResourceManagerUI, open the Application, click Settings and choose Kill Application. Notice, the 'Logged in as:' is set to UNKNOWN_USER:
Confirm that I want to kill the Application:
I get the following error in the UI:
Opening the console in Chrome, I see a 401 (Unauthorized) error.
If I try this from the old UI I am able to expand the error message and it shows the following:
{"RemoteException":{"exception":"AuthorizationException","message":"Unable to obtain user name, user not authenticated","javaClassName":"org.apache.hadoop.security.authorize.AuthorizationException"}}
I've read lots of posts, verified and changed several settings to try to fix this with no luck. Here are some of the settings I checked or changed as a result of my research:
hadoop.http.filter.initializers=org.apache.hadoop.security.HttpCrossOriginFilterInitializer,org.apache.hadoop.http.lib.StaticUserWebFilter
hbase.security.authentication=simple
hbase.security.authorization=false
yarn.nodemanager.webapp.cross-origin.enabled=true
yarn.resourcemanager.webapp.cross-origin.enabled=true
yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled=false
yarn.resourcemanager.webapp.ui-actions.enabled=true
yarn.timeline-service.http-authentication.simple.anonymous.allowed=true
yarn.timeline-service.http-authentication.type=simple
yarn.webapp.api-service.enable=true
yarn.webapp.ui2.enable=true
ranger.add-yarn-authorization=false
Some of these seem way off base to me, like the hbase stuff, since I don't think that has anything to do with what I'm seeing. However, some users, in other situations, had it work for them so I wanted to try it.
Digging through the documentation it seems like you need to be authenticated before you can call the API. However, that same language was in the documentation for 2.6.5, which is the version of YARN I was running before where this worked.
Hopefully someone can point me to documentation that more clearly outlines what I can do to resolve the issue.
Thanks in advance.
Hey I know this isn't solution (I'm experiencing the same issue post-upgrade), but I found adding "?user.name=" at the end of the url to the old resource manager url will log you in as that user on both pages. I've found the old RM page as the only way to kill jobs though.
I have an existing spark-job, the functionality of this spark-job is to connect kafka-server get the data and then storing the data into cassandra tables, now this spark-job is running on server inside spark-2.1.1-bin-hadoop2.7/bin but whenever I am trying to run this spark-job from other location, Its not running, this spark-job contains some JavaRDD related code.
Is there any chance, I can run this spark-job from outside also by adding any dependency in pom or something else?
whenever I am trying to run this spark-job from other location, Its not running
spark-job is a custom launcher script for a Spark application, perhaps with some additional command-line options and packages. Open it, review the content and fix the issue.
If it's too hard to figure out what spark-job does and there's no one nearby to help you out, it's likely time to throw it away and replace with the good ol' spark-submit.
Why don't you use it in the first place?!
Read up on spark-submit in Submitting Applications.
For the past three to four months, we have our application live and running, we haven't deployed any new fixes / changes on Live. However ever unfortunately, we noticed that application has stopped running.
Following is the issue we observed from our logs :
"Can't create/write to file '/var/tmp/#sql_2f6_0.MYI" .
It would be really appreciable if anyone of you can extend your help.
Check the services and the User for which your Mysql is giving you this error. It is very much possible that any of the services might be down, or the User by which you are using the DB is not getting authenticated.
You or the user that handles your SQL service doesn't have permission to /var/tmp/. You can fix this by using chmod or Security permissions, depending on which platform you're on.
I encountered an error while adding a new service (Service Type = HDFS) using Cloudera Manager (Free Edition). The error message says as follows:
Could not create process: com.cloudera.cmf.service.config.ConfigFileSpec$GenerateException: Unable to process template:couldn't find the template hadoop/topology.py.vm
I checked /var/log/cloudera-scm-server/cloudera-scm-server.log and found a line like below.
org.apache.velocity.exception.ResourceNotFoundException: Unable to find resource '/WEB-INF/templates/hadoop/topology.py.vm'
I guess that a certain war file does not contain hadoop-metrics.properties.vm (Velocity template file?) although it should do and that this might be related to WHIRR-370.
Could you help me to solve this problem, please?
May I ask which version of Cloudera Manager is being used? Does this error occurred just after you try to add add the service of after some time when the service is added?
Based on the error, it seems some of the configuration is missing that why service addition failed. So I would like to know how did you install Hadoop on this cluster?
If you download the virtual machine and compare from your installation, you can compare the folder for completeness and missing content. It does work for me always.