Azure Data Tables nuget package to get total requests (RUs) for Cosmos db using table API
Related
I am trying to configure an external hive metastore for my azure synapse spark pool. The rational behind using external meta store is to share table definitions across databricks and synapse workspaces.
However, I am wondering if its possible to access backend data via the metastore. For example, can clients like PowerBI,tableau connect to external metastore and retrieve not just the metadata, but also the business data in the underlying tables?
Also what additional value does an external metastore provides ?
You can configure the external Hive Metadata in Synapse by creating a Linked Service for that external source and then query it in Synapse Serverless Pool.
Follow the below steps to connect with External Hive Metastore.
In Synapse Portal, go to the Manage symbol on the left side of the of the page. Click on it and then click on Linked Services. To create the new Linked Service, click on + New.
Search for Azure SQL Database or Azure Database for MySQL for the external Hive Metastore. Synapse supports these two Hive external metastore. Select and Continue.
Fill in all the required details like Name, Subscription, Server name, Database name, Username and Password and Test the connection.
You can test the connection with the Hive metadata using below code.
%%spark
import java.sql.DriverManager
/** this JDBC url could be copied from Azure portal > Azure SQL database > Connection strings > JDBC **/
val url = s"jdbc:sqlserver://<servername>.database.windows.net:1433;database=<databasename>;user=utkarsh;password=<password>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
try {
val connection = DriverManager.getConnection(url)
val result = connection.createStatement().executeQuery("select * from dbo.persons")
result.next();
println(s"Successful to test connection. Hive Metastore version is ${result.getString(1)}")
} catch {
case ex: Throwable => println(s"Failed to establish connection:\n $ex")
}
Check the same in below snippet for your reference.
can clients like PowerBI,tableau connect to external metastore and retrieve not just the metadata, but also the business data in the underlying tables?
Yes, Power BI allows us to connect with Azure SQL Database using in-built connector.
In Power BI Desktop, go to Get Data, click on Azure and select Azure SQL Database. Click connect.
In the next step, go give the Server name in this format <utkarsh.database.windows.net>, database name, Username and Password and you can now access data in Power BI. Refer below image.
I am working on IoT solution, where there are multiple sensors which are sending data. I have one job which listen to Event hub, get the IoT sensor data and store in in Delta lake table. (Underlying Azure ADLS Gen-2 Storage in parquet file format)
I have to display the sensor data on UI (custom UI - developed in React). For that, I have API layer developed in .NET core API / Node JS API.
So finally I have to query Delta table created in data bricks to retrieve the sensor data - using Node JS / .NET CORE and display it on UI. How I can query the data from C# / Node JS API from delta lake table?
You may be better off using query compute that already is capable of reading the delta formatted data. A low cost option would be to create an Azure Synapse Analytics workspace and use a serverless SQL pool to query the delta content. A serverless pool exposes itself as an Azure SQL DB. So, any tool that can query an Azure SQL DB (not a problem for either C# or node.js) can then query those delta tables.
The SQL syntax is a little different looking as it uses an OPENROWSET like follows:
SELECT
*
FROM
OPENROWSET(
BULK 'https://<my-storage-account>.dfs.core.windows.net/<my-container>/<path-to-delta-folder>/',
FORMAT='DELTA'
) AS [recordset];
Alternatively, you can create a logical database in the serverless pool and create external tables for each of your delta folders. Doing this would make it seems a little closer to a traditional relational database.
Since this is a serverless instance there is no provisioning costs. You'd only pay for the consumption (e.g. the actual query). You can find the current pay-as-you go pricing here: Azure Synapse Analytics pricing
I want to read 5+ million events from an Azure SQL DB table and perform a BULK INSERT to Cassandra.
The table has 2 columns.
I see the SQL component available for reading from Azure SQL DB. https://camel.apache.org/components/3.7.x/sql-component.html
Question: Consuming from Azure SQL DB
Is there a better way to read all the rows and store in a map considering 5M records ?
Is there a possibility to read messages in batches ?
There is a cql component available for Cassandra
https://camel.apache.org/components/3.7.x/cql-component.html
Question: Producing to Cassandra
Can we INSERT in batches ?
Can I use camel for this use case ?
For each table in your Azure database, we recommend that you export the data into a CSV file. There are lots of tools and methods available that would allow you to do that. For example, have a look at Import and export data from Azure SQL Database.
Once you have exported your data to CSV, you can use the DataStax Bulk Loader tool (DSBulk) to bulk load it to a Cassandra table.
Here are some references with examples to help you get started quickly:
Blog - DSBulk Intro + Loading data
Blog - More DSBulk Loading examples
Blog - Counting records with DSBulk
Docs - Loading data examples
Answered questions - DS Community
DSBulk is open-source so it's free to use. Cheers!
I am trying to create external data source in Azure Synapse Analytics (Azure SQL Data warehouse) to Oracle external database. I am using the following code in SSMS to do that:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'myPassword';
CREATE DATABASE SCOPED CREDENTIAL MyCred WITH IDENTITY = 'myUserName', Secret = 'Mypassword';
CREATE EXTERNAL DATA SOURCE MyEXTSource
WITH (
LOCATION = 'oracle://<myIPAddress>:1521',
CREDENTIAL = MyCred
)
I am getting the following error:
CREATE EXTERNAL DATA SOURCE statement failed because the 'TYPE' option is not specified. Specify a value for the 'TYPE' option and try again.
I understand from the below that TYPE is not a required option for Oracle databases.
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-data-source-transact-sql?view=azure-sqldw-latest
Not sure how what the problem is here, is this feature still not supported in Azure Synapse Analytics (Azure DW) when it is already available in MS SQL Server 2019? Any ideas are welcome.
Polybase has different versions across the different products with different capabilities. Most of these are described here:
The ability to connect to Oracle is only present in the SQL Server versions, currently 2019. The documentation is quite clear that is only applies to SQL Server and not to Azure Synapse Analytics (formerly Azure SQL Data Warehouse):
https://learn.microsoft.com/en-us/sql/relational-databases/polybase/polybase-configure-oracle?view=sql-server-ver15
In summary, Azure Synapse Analytics and its version of Polybase does not currently support to access external Oracle tables at this time.
I've been task with figuring out a way to get data from Oracle and store it in a SQLite database. The back story is we currently use SQLite for our local storage on a mobile application and we currently populate that data via a file download, because the data is a large amount it could take up to 5 minutes to populate the database. An easy solution for us would be to build the table on the server and download it via http. The data is currently stored in a Oracle database on the server. My question is is it possible to create a DBLink from Oracle to SQLite to insert the data into the SQLite database on the server? If this is not possible are there any other solutions that would achieve this?
Thanks