I have built a streaming pipeline with spark autoloader.
Source Folder is a azure blob container.
We encountered a rare issue (could not replicate it). Below is the exception Message:
java.lang.IllegalStateException: dbfs:/mnt/delta_checkpoints/offsets/5 exists before writing
Please help on this with a resolution, as this looks like some known platform issue.
Please let know if I need to attach the entire stacktrace.
Related
i am working on poc where am creating adf pipeline which needs to pickup file from source sftp and move to target sftp.as this is poc i am using azure blob storage sftp feature which is in preview for now. But when i am creating linked service using sftp connection string i am getting error as below
Error code: 9978
Details: Access 'storageacc.container.user#acc.blib.core.windows.net' is not allowed right on azure integration runtime.
I searched a lot but couldn't find any solution,if you know the issue plz help me. Thanks in advance.
Your Host control can't accept 'storageacc.container.user#acc.blib.core.windows.net'. It have to be divided into two parts:
acc.blib.core.windows.net - for [Host field] control
storageacc.container.user - for [User name] control
My current Connection:
Data factory Copy activity fails when copy the delta table from databricks to storage account gen2
Details
ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key
Caused by: Invalid configuration value detected for fs.azure.account.key.
Appreciate your help.
The above error mainly happens because the staging is not enabled. We need to enable staging to copy data from delta Lake.
Go to Azure Databricks inside cluster -> advance option and edit spark config as per the below format.
spark.hadoop.fs.azure.account.key.<storage_account_name>.blob.core.windows.net <Access Key>
After that you can follow this official document it has detail explanation about copy activity with delta lake.
you can refer this Article by RishShah-4592
Edit the cluster,
fs.azure.account.key..dfs.core.windows.net{{secrets//}}
Its working fine now...Able to copy data from delta lake table to adls gen2
I think so you can pass the secret as below:
spark.hadoop.fs.azure.account.key.<storage_account_name>.blob.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}
I have been attempting to install oracle db 12c (both 12.2.0.1 as well as 12.1.0.2) But keep getting errors that I can't figure out. I have gone through AWS's own cloudformation troubleshooting guide along with several forums.
I am getting the following messages in the events log. I have included text below as well as a screen shot of the events for the root stack:
The following resource(s) failed to create: [OracleStack].
Embedded stack arn:aws:cloudformation:us-west-2:007849015507:stack/Oracle-Database-OracleStack-1W5U4RT91JZXT/a1f67bb0-be05-11e9-8c50-02ee71065ed8 was not successfully created: The following resource(s) failed to create: [PrimaryInstance].
I have come accross some info on cloudformation helper scripts and timing out of parameters, but no topic addresses how to correct the problem.
Help is appreciated
I am unable to create a new Common Data Service Database in my Power Apps default environment. Please see the error text below.
It looks like you don't have permission to use the Common Data Service
in this environment. Switch to a different environment, or create your
own.
Which as I understand I should be able to create after the Microsoft Business Application October 2018 update as listed in the article available at following link.
https://community.dynamics.com/365/b/dynamicscitizendeveloper/archive/2018/10/17/demystifying-dynamics-365-and-powerapps-environments-part-1
Also when I try to create a Common Data Service app in my default environment, I encounter following error.
The data did not load correctly. Please try again.
The environment 'Default-57e1485d-1197-4afd-b792-5c423ab508d9' is not
linked to a new CDS 2.0 instance. The operation 'ListInstanceMetadata'
is forbidden for unlinked environments
Moreover I am unable to see the default environment on https://admin.powerapps.com/environments, I can only see the Sandbox environment there.
Any ideas what I am missing here?
Thank you.
Someone else faced a similar issue and I read in one of the threads about deleting the browser cache and trying it again or trying it in a different browser resolved the issue. Could you try these first level steps and check if you still have these issues?
Ref: https://powerusers.microsoft.com/t5/Common-Data-Service-for-Apps/Default-Environment-Error-on-CDS/m-p/233582#M1281
Also, for your permission error ref: https://powerusers.microsoft.com/t5/Common-Data-Service-for-Apps/Common-Data-Service-Business-Flows/td-p/142053
I have not validated these findings. But as these answers are from MS and PowerApps team, hope it helps!
I encountered an error while adding a new service (Service Type = HDFS) using Cloudera Manager (Free Edition). The error message says as follows:
Could not create process: com.cloudera.cmf.service.config.ConfigFileSpec$GenerateException: Unable to process template:couldn't find the template hadoop/topology.py.vm
I checked /var/log/cloudera-scm-server/cloudera-scm-server.log and found a line like below.
org.apache.velocity.exception.ResourceNotFoundException: Unable to find resource '/WEB-INF/templates/hadoop/topology.py.vm'
I guess that a certain war file does not contain hadoop-metrics.properties.vm (Velocity template file?) although it should do and that this might be related to WHIRR-370.
Could you help me to solve this problem, please?
May I ask which version of Cloudera Manager is being used? Does this error occurred just after you try to add add the service of after some time when the service is added?
Based on the error, it seems some of the configuration is missing that why service addition failed. So I would like to know how did you install Hadoop on this cluster?
If you download the virtual machine and compare from your installation, you can compare the folder for completeness and missing content. It does work for me always.