I have data in an Oracle Siebel data base and I want to move the data to the Azure Cloud over a VPN.
Is it possible to copy the database and then apply daily snapshots until I am ready to take the new Azure application live.
This reduces the risk of relying on a single big migration on the cutover weekend.
The destination DB is not Siebel, I will have to put the data into an Azure DB.
Just gathering ideas at the moment
Related
I have a following scenario. (A rather common one but I am not entirely sure where to start)
I have data incoming into blob storage container (our raw zone). The files get dropped in raw zone everyday(by someone sitting somewhere). Each day as the new files come in, the old files are overwritten, but the number of records increases.
Suppose a customer file from yesterday may have 100 records, today's file might have 150 records. (100 from yesterday and 50 from today).
Now, what is the best way to do an incremental load (or other solutions welcome) for moving latest number of records into the azure table storage.
I have worked with using watermarks etc when loading data from or into sql, but don't have so much experience with Azure table. Would appreciate if I can get a lead.
Thanks in advance.
You can use ADF to do incremental load into Azure Table Storage using watermarks. Refer to below links and you might need to tweak the implementation a little based on requirements.
Incrementally load data from Azure SQL Database to Azure Blob storage using the Azure portal
Copy data to and from Azure Table storage using Azure Data Factory or Synapse Analytics
I'm working on a project to create hybrid SQL database per tenant model. While we were able to replicate the on-premise databases to databases in Azure. I'm not able to find a way to continuously sync both on-prem DB and cloud DB (We cannot use data sync agent or transaction replication). We are looking for any other alternatives that we can try to achieve our purpose.
Also, how does synchronization works when the internet is down and cannot sync with cloud?
Sorry for my ignorance since I'm new to this field.
Thanks.
why can't you use data sync agent? It can be installed in other machine that has access to the database and internet.
I am new to AWS DMS service. Plans are to migrate on-prem Oracle to Redshift. Before going into production environment, currently trying out a test Oracle RDS in AWS which is a small subset of actual database as source. So far have been successful in the bulk load and incremental migration from RDS to Redshift.
When it comes to on-prem oracle , particularly for the incremental load
1) As per document : http://docs.aws.amazon.com/dms/latest/sbs/CHAP_On-PremOracle2Aurora.Steps.ConfigureOracle.html, the on-prem needs to be enabled with supplemental logging. Plans are to use the following two commands.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
The production database has multiple logging locations. Is there any other log settings other than the above two that I should be looking into for DMS to pick up multiple log locations?
2) In the same link given, point 4 says 'Create or configure a database account to be used by AWS DMS.'
Where should I create this user? on-prem oracle or AWS?
How do I configure DMS to use this user?
You need to read this documentation:
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html
For your second question; You need to create a user in the Oracle source database, the section 'Working with a Self-Managed Oracle Database as a Source for AWS DMS' tells you all of the grants you need to give.
For your first question, if you look at the SQL Server documentation;
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html
It specifies the limitation of; 'SQL Server backup to multiple disks isn't supported. If the backup is defined to write the database backup to multiple files over different disks, AWS DMS can't read the data and the AWS DMS task fails.'
I can't see a similar stipulation in the oracle documentation, first link, I would hazard a guess that DMS is able, in the case of oracle, to determine and cope with multiple logging locations from a configuration value inside the database.
I'm looking for the best way to upload files to a Azure SQL Database.
We have to use Azure Data Factory as at this moment we are not allowed to use Azure VM's with SSIS.
Each day we are upload 1,5Gb of XML files.
Currently we we are uploading the to a Blob storage and with a Copy activity we are uploading them into the DB.
But this takes up to 2,5 hours.
What would be a better/faster concept to do this ?
Any Suggestions ?
You can use the bcp utility to import your data into an instance of SQL Server as explained in this document. From there, you can use the Azure Data Synchronization tool to synchronize your Azure SQL Database with your SQL Server Database. This may provide a faster execution time.
Finally it dropped to 35 minutes.
Just by using several storage accounts (5) and split the data over the 5 accounts.
5 ADF pipelines uploding all into the same staging table.
Some where huge files, but we had over 100.000 small files from 2 up to 100K.
This worked out fine for us.
We noticed that the DTU's never went up to their limit so we thought that the DB was not the problem, by firts splitting into 2 we saw DTU rising a bit more. we continued on that path....
It is recommended that we store document information in blob storage. In our case the blob storage is related to the SQL Azure data, is the facility available to back up the blob storage in sync with the SQL Azure data ? What I don't want to see is a point in time restore of the SQL Azure data only to find we don't have the same snapshot of the blob data at that time :(
Does anyone know what is available
Interesting issue you have to solve. But there is no automated way to keep in sync BLOB and Azure SQL Database Data. You have to do manage this yourself. And here is not just about blob snapshots. How about your updated DB record refers to a new blob. What happens with the old one ?! This is all business rules to apply at application level. And you have to question yourself to what degree you want that backup of Blobs.
Here is an interesting blog post on Azure SQL and Storage backup. But again - there is no service that will keep for you in sync data between SQL DB and Azure Storage.