I am trying to create external data source in Azure Synapse Analytics (Azure SQL Data warehouse) to Oracle external database. I am using the following code in SSMS to do that:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'myPassword';
CREATE DATABASE SCOPED CREDENTIAL MyCred WITH IDENTITY = 'myUserName', Secret = 'Mypassword';
CREATE EXTERNAL DATA SOURCE MyEXTSource
WITH (
LOCATION = 'oracle://<myIPAddress>:1521',
CREDENTIAL = MyCred
)
I am getting the following error:
CREATE EXTERNAL DATA SOURCE statement failed because the 'TYPE' option is not specified. Specify a value for the 'TYPE' option and try again.
I understand from the below that TYPE is not a required option for Oracle databases.
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-data-source-transact-sql?view=azure-sqldw-latest
Not sure how what the problem is here, is this feature still not supported in Azure Synapse Analytics (Azure DW) when it is already available in MS SQL Server 2019? Any ideas are welcome.
Polybase has different versions across the different products with different capabilities. Most of these are described here:
The ability to connect to Oracle is only present in the SQL Server versions, currently 2019. The documentation is quite clear that is only applies to SQL Server and not to Azure Synapse Analytics (formerly Azure SQL Data Warehouse):
https://learn.microsoft.com/en-us/sql/relational-databases/polybase/polybase-configure-oracle?view=sql-server-ver15
In summary, Azure Synapse Analytics and its version of Polybase does not currently support to access external Oracle tables at this time.
Related
I have Azure Databricks setup with External hive metaStore (with Azure SQL) and the database connection URL is setup in Databricks Cluster's advanced Settings. In this way I am able to see/access the database tables from delta lake (which is on azure storage account adls) in Databricks's Data section.
Now, I want my users to access these tables through Databricks's 'SQL workspace'. I have configured the 'Data access' using service principle in 'SQL warehouse' section.
Per databricks SQL documentation, I am supposed to see the delta lake tables which I can see through 'Data Science and Engineering' section. But I cant see schema or tables in meta store.
Problem, I am not able to see the tables through the 'SQL workspace' > Data. And I am puzzled how it will know where is my External metaStore and what are schema definition?
In SQL workspace should be setup to indicate the hive metastore connection. But I am not sure as databricks documentation is not very clear on this point.
Please suggest
Below are 'data access' details for service principal:
spark.hadoop.fs.azure.account.auth.type.<adlsContainer>.dfs.core.windows.net OAuth
spark.hadoop.fs.azure.account.oauth.provider.type.<adlsContainer>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
spark.hadoop.fs.azure.account.oauth2.client.id.<adlsContainer>.dfs.core.windows.net <CLIENT_ID>
spark.hadoop.fs.azure.account.oauth2.client.secret.<adlsContainer>.dfs.core.windows.net <CLIENT_SECRET>
spark.hadoop.fs.azure.account.oauth2.client.endpoint.<adlsContainer>.dfs.core.windows.net https://login.microsoftonline.com/<TENANT_ID>/oauth2/token
As mentioned in the data access documentation you need to configure the same Spark configuration properties for external hive metastore as for "normal" Spark clusters - see documentation on this topic.
I am trying to configure an external hive metastore for my azure synapse spark pool. The rational behind using external meta store is to share table definitions across databricks and synapse workspaces.
However, I am wondering if its possible to access backend data via the metastore. For example, can clients like PowerBI,tableau connect to external metastore and retrieve not just the metadata, but also the business data in the underlying tables?
Also what additional value does an external metastore provides ?
You can configure the external Hive Metadata in Synapse by creating a Linked Service for that external source and then query it in Synapse Serverless Pool.
Follow the below steps to connect with External Hive Metastore.
In Synapse Portal, go to the Manage symbol on the left side of the of the page. Click on it and then click on Linked Services. To create the new Linked Service, click on + New.
Search for Azure SQL Database or Azure Database for MySQL for the external Hive Metastore. Synapse supports these two Hive external metastore. Select and Continue.
Fill in all the required details like Name, Subscription, Server name, Database name, Username and Password and Test the connection.
You can test the connection with the Hive metadata using below code.
%%spark
import java.sql.DriverManager
/** this JDBC url could be copied from Azure portal > Azure SQL database > Connection strings > JDBC **/
val url = s"jdbc:sqlserver://<servername>.database.windows.net:1433;database=<databasename>;user=utkarsh;password=<password>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
try {
val connection = DriverManager.getConnection(url)
val result = connection.createStatement().executeQuery("select * from dbo.persons")
result.next();
println(s"Successful to test connection. Hive Metastore version is ${result.getString(1)}")
} catch {
case ex: Throwable => println(s"Failed to establish connection:\n $ex")
}
Check the same in below snippet for your reference.
can clients like PowerBI,tableau connect to external metastore and retrieve not just the metadata, but also the business data in the underlying tables?
Yes, Power BI allows us to connect with Azure SQL Database using in-built connector.
In Power BI Desktop, go to Get Data, click on Azure and select Azure SQL Database. Click connect.
In the next step, go give the Server name in this format <utkarsh.database.windows.net>, database name, Username and Password and you can now access data in Power BI. Refer below image.
I want to explore Oracle data integrator , i am not able to understand what does 'Use credential File' option in Data server does in Oracle data integrator. If anyone can explain it would be helpful and i want to improve performance of my oracle data integrator script as well, any ideas on that as well.
Ok, now I think that I understood. You run ODI in Cloud.
You will need a credential File in order to connect to your database.
The way you obtain that credential file, is:
Credential files are downloaded from the ADW console to the ODI host in the Oracle Cloud Infrastructure (OCI).
Note: When ODI is deployed from the Marketplace, client credential folders are downloaded from autonomous databases that exist in the OCI compartment containing ODI.
If ADW is in a different compartment than ODI follow the steps below.
Download the Credentials
Connect to the ODI host using VNC. Refer to the Deployment blog above
for details.
Launch Firefox from the Applications>Favorites list.
Follow the steps in Downloading Autonomous Data Warehouse Credentials
to obtain the client credentials compressed folder containing the
wallet and network configuration files used by ODI to make the
connections.
The entire way of connecting is described here.
I have a subscription to Oracle Cloud Database Schema Services ( Schema not full Database ) , I am trying to access the database instance using Oracle SQL Developer , I followed the below to make connection in SQL developer :
1 - From "Database schema Service connection"
2 - New Cloud Connection and enter my username/password/instance URL
3- The database connected and I can list all tables inside the schema
But When I tried to open SQL worksheet and write some select statements , I can't achieve this.
I know that this task can be done through APEX console , but is there anyway to do it by using SQL Developer?
This is not currently supported.
The SCHEMA service is only reachable via HTTPS. We have built several REST Services that allow SQL Developer to do what you see currently today, which includes browsing the schema and uploading data via the CART.
We have just built a SQL 'REST Service' feature into Oracle REST Data Services which allow us..or any authenticated user, to run an ad-hoc SQL or PL/SQL block statement via a POST. This would allow us to add what you're looking for - the ability to do a SQL Worksheet for your service.
However, instead of building this into the SQL Developer desktop, we're looking at releasing 'SQL Developer Web' which will be available in your Oracle Cloud Database Services' consoles.
I can't tell you if/when that will be available for you in your Schema Service, but it's on the road map.
In the mean time, the APEX UI and its SQL Workshop is the way to go.
I have a Sql database and I have to put it in my Azure Storage Emulator (In Storage > Development > Tables); how can I do it? In Visual Studio I don't find any possibility to import, for example, a .bacpac file.
UPDATE:
I discovered that the database should not be put in there, but then how do I properly simulate Azure storage and my Sql database as well?
Just to clarify: Azure Storage (Tables, Blobs, Queues) is completely different from SQL Azure databases.
If you use the latest version of SQL Express it's very very similar to SQL Azure and fine for most development needs.
There isn't a SQL Azure simulator that I know of - but it's really easy to get a FREE 30 day Azure account and I would strongly suggest you do some development and testing against that: SQL Azure has far more connection issues (no connection, handshake problems, lags) than a database that is local to the app, and you need to be prepared for this (investigate Transient Fault Handling strategies).
You can work locally and connect to your SQL Azure instance = a 'real' SQL Azure simulation.
When developing locally for Azure SQL Database, please use SQL Server Express - it is the relational database equivalent for Azure Storage Emulator (Azure Blob, Queue, and Table).