We consider switching our existing MySql-cluster database into AWS Aurora.
Does Aurora support Auto-increment column?
Suppose the answer is YES -
Does Aurora's Multi-Master support Auto-increment?
Aurora is "MySQL compatible". From the FAQ:
https://aws.amazon.com/rds/aurora/faqs/
"most of the code, applications, drivers and tools you already use today with your MySQL databases can be used with Aurora with little or no change."
That should include autoincrement columns unless otherwise specified, and that is not mentioned in list of MySQL features not supported in Aurora:
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMySQL.Overview.html#Aurora.AuroraMySQL.CompareReleases
Related
I have a couple of questions on AWS RDS service for migrating an oracle 19c server on RHEL with 3 standalone instances and databases:
Can RDS instance support multiple oracle standalone instances/databases or only one instance?
If I have an existing RDS service running in AWS, can I migrate another on-premise oracle database to the RDS database as another oracle schema?
Have not tried it yet.
RDS for Oracle limits one instance to a single database. However you can have multiple schemas in one database.
An account can have up to 40 BYOL type Oracle RDS instances, or up to 10 instances where license is included. You can also increase these limits by contacting AWS support.
See here for more details.
Each RDS instance equals one database. However, you can set up multiple RDS instances, if you want to.
For migrations, please have a look at the Database Migration Service (DMS). Regarding your schema in particular, please check the Schema Conversion Tool which is a part of the DMS.
Sorry I can’t be more specific as the questions are rather vague themselves.
I'd like to connect Database Note Taker to Redshift, but the standard drivers dont do it. What do I have to do in order to connect?
From the Database Note Taker FAQ:
Can Database Note Taker connect to other database platforms?
Yes and we will add support for other platforms in the future. So long as it's a relational database and there are freely available connectors, then Database Note Taker will be able to work with it. Out of the box you can connect to Microsoft Sql Server, and MySql via ODBC.
Given that Amazon Redshift is based on PostgreSQL, it is highly unlikely that Database Note Taker will work with Amazon Redshift.
I also note that the the website is dated 2014 and the program was last updated in 2016.
I also notice that there is a 'Contact Us' page on that website, so you should probably ask them.
In my company we are evaluating the use of Airflow mainly to run ML model.
In the documentation they suggest to use Postgres or MySQL, but we prefer to stick with our tools, in this particular scenario we'd like to give to Airflow a dedicated schema in Oracle Database Enteprise Edition 19.
Is it possible to have oracle as a backend db? Are there any drawback?
It seams that Oracle is not supported as a backend for Airflow corresponding to the Jira ticket: AIRFLOW-3994
From this link, it seems that you can use Oracle with airflow.
Airflow can run with any DB supported by SqlAlchemy. For example:
PostgreSQL
MySQL
SQLite
Oracle
Microsoft SQL Server
List of supported dialects: https://docs.sqlalchemy.org/en/13/dialects/
Although from my experience I recommend using Postgres from my experience and for performance reasons.
The question sums it up basically.
I have an external (not in AWS) MySQL v5.7 database that is 40Gb (it has one table of 16Gb). This database will be imported to RDS MySQL by another team.
I will then need to have a copy of it in my microservice but not the whole DB, just some tables (among them, the 16Gb one).
The optimal solution would be a "read replica" of just some columns of various tables.
It will need to be in constant sync with the "Master" table.
I would also like to use Aurora instead of MySQL (reason: speed & cost).
I googled for information but couldn't find anything helpful besides
AWS Data migration service.
Anyone here has experience with this? And what would you suggest?
Yes you can definetly use DMS service. You can use transformation rule and put db name and table name , which will help in selective migration.
Note :- Binlog_format has to be set true for any kind of source of RDS . They should also have backup enabled.
Is it possible to establish a connection from Airflow to Greenplum?Keeping in mind that Greenplum is based on PostgreSQL, would it be possible to establish a connection to the Greenplum master server?
Andrea,
I think you can use Airflow to run ETLs on your analytic data within Greenplum.
The "no" answer that Jon provided was apparently in regard to using Greenplum as your backend metadata store, used internally by Airflow for keeping track of its DAGs and tasks. The code that Jon used as an example is how Airflow creates tables it uses for its backend metadata store, which has nothing to do with the contents of your Greenplum data warehouse you want to manage.
I suspect you are instead interested in Greenplum for your high-volume analytic data, not for the Airflow backend. So the answer is almost certainly yes!
You might even get by using the standard PostgreSQL hook and operator:
I say this since it appears that Greenplum can use the standard PostgreSQL Python API:
https://gpdb.docs.pivotal.io/4330/admin_guide/managing/access_db.html
If Airflow's standard PostgreSQL hook & operator do not work for you, it is easy to create your own. See for example my work with the Snowflake data warehouse; it was a simple matter to integrate the Snowflake Python connector into airflow.
https://github.com/aberdave/airflow-snowflake
Failing all that, you should be able to use ODBC or JDBC to manage date in Greenplum via Airflow. I went with Python for my work with Snowflake, since it was so easy to customize their Python connector.
No. A quick look at the Airflow github repo shows that they are using primary key constraints plus an additional column with a unique constraint which isn't supported in Greenplum.
For example:
op.create_table(
'user',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('username', sa.String(length=250), nullable=True),
sa.Column('email', sa.String(length=500), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('username')
)
You can't have a primary key on (id) and another unique constraint on (username) in Greenplum.
Their github repo also doesn't have any mention of other MPP database platforms like Netezza and Teradata. Maybe Airflow is for small data, data science but that sounds like an oxymoron.