I would like to compare two clickhouse databases and generate an upgrade script which I can apply for out of date databases.
For SQL server I know tool called Red Gate toolbelt which does this job.
Is there any product which can do exactly same for clickhouse database
all-in-one solution is not available yet,
you could try to use https://github.com/dbeaver/dbeaver/ and https://www.jetbrains.com/datagrip/
Related
In my company we are evaluating the use of Airflow mainly to run ML model.
In the documentation they suggest to use Postgres or MySQL, but we prefer to stick with our tools, in this particular scenario we'd like to give to Airflow a dedicated schema in Oracle Database Enteprise Edition 19.
Is it possible to have oracle as a backend db? Are there any drawback?
It seams that Oracle is not supported as a backend for Airflow corresponding to the Jira ticket: AIRFLOW-3994
From this link, it seems that you can use Oracle with airflow.
Airflow can run with any DB supported by SqlAlchemy. For example:
PostgreSQL
MySQL
SQLite
Oracle
Microsoft SQL Server
List of supported dialects: https://docs.sqlalchemy.org/en/13/dialects/
Although from my experience I recommend using Postgres from my experience and for performance reasons.
Is it possible to establish a connection from Airflow to Greenplum?Keeping in mind that Greenplum is based on PostgreSQL, would it be possible to establish a connection to the Greenplum master server?
Andrea,
I think you can use Airflow to run ETLs on your analytic data within Greenplum.
The "no" answer that Jon provided was apparently in regard to using Greenplum as your backend metadata store, used internally by Airflow for keeping track of its DAGs and tasks. The code that Jon used as an example is how Airflow creates tables it uses for its backend metadata store, which has nothing to do with the contents of your Greenplum data warehouse you want to manage.
I suspect you are instead interested in Greenplum for your high-volume analytic data, not for the Airflow backend. So the answer is almost certainly yes!
You might even get by using the standard PostgreSQL hook and operator:
I say this since it appears that Greenplum can use the standard PostgreSQL Python API:
https://gpdb.docs.pivotal.io/4330/admin_guide/managing/access_db.html
If Airflow's standard PostgreSQL hook & operator do not work for you, it is easy to create your own. See for example my work with the Snowflake data warehouse; it was a simple matter to integrate the Snowflake Python connector into airflow.
https://github.com/aberdave/airflow-snowflake
Failing all that, you should be able to use ODBC or JDBC to manage date in Greenplum via Airflow. I went with Python for my work with Snowflake, since it was so easy to customize their Python connector.
No. A quick look at the Airflow github repo shows that they are using primary key constraints plus an additional column with a unique constraint which isn't supported in Greenplum.
For example:
op.create_table(
'user',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('username', sa.String(length=250), nullable=True),
sa.Column('email', sa.String(length=500), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('username')
)
You can't have a primary key on (id) and another unique constraint on (username) in Greenplum.
Their github repo also doesn't have any mention of other MPP database platforms like Netezza and Teradata. Maybe Airflow is for small data, data science but that sounds like an oxymoron.
My team does not have direct, command-line access to the oracle boxes but we have need to, from time to time, spin up dev instances of the DB.
We do not need the data, but we do need the layout. Are there any tools that would interrogate an oracle instance and build a new instance?
Take a look at this tool.
http://www.red-gate.com/products/oracle-development/schema-compare-for-oracle/
We use the version for Sql Server in my shop and are very happy with it.
(I'm not affiliated with Red Gate, just a satisfied customer)
There are lots of ways...
You can do a schema export in TOAD and most other decent tools or you can write something using data pump
http://www.oracle-base.com/articles/10g/oracle-data-pump-10g.php
for starters
My company has an existing Database with dozens of DB tables with various relationships.
Is there any tool available that will generate a relationship diagram?
I primarily use SQL Developer 3.0.4 on an Ubuntu 11.10 machine by the way.
SQL Developer Data Modeller can reverse engineer a database schema & make a start on an ERD using the foreign keys. Obviously some manual work will be needed once the initial import has been done in order to generate a nice looking meaningful diagram.
Since you already have SQL Developer installed, you can make a start with File->Data Modeller->Import->Data Dictionary.
Does the SQL syntax differ in any way for SQL Plus and apex.oracle.com
From this article I can assume that it doesn't, but I want to be sure. Is SQL Plus only an environment that is able to connect to an Oracle server ?
I'm asking this because I just started learning Oracle's SQL syntax and I don't have access to my faculty's server from home. The only solution I found so far is Apex since I'm not planning to install a server on my laptop.
Is SQL Plus only an environment that
is able to connect to an Oracle server?
Yes, SQL*Plus is not very useful unless you can connect to an Oracle database server.
I'm asking this because I just started
learning Oracle's SQL syntax and I
don't have access to my faculty's
server from home. The only solution I
found so far is Apex since I'm not
planning to install a server on my
laptop.
If you cannot connect to your faculty's database server from home, you have no choice but to install your own server somewhere. Apex only runs on an Oracle database server.
However, it's not too difficult nowadays, as long as you follow the installation instructions carefully.
Another option is to get a free account at apex.oracle.com. You'll be able to use Apex's SQL interface. I don't think you can connect to it using SQL Developer, though.
Alternatively, you can start up a simple pre-prepared Apex instance on Amazon's EC2. With this option, if you open the relevant port (1521) you'll be able to connect to it using SQL Developer.
The SQL commands are identical. They are all passed directly to the database engine for execution.
SQL*Plus has a bunch of extra commands (mostly for formatting output, but also stuff like dealing with variables and database startup/shutdown).
[Very old versions of SQL*Plus from the Oracle 7 era don't recognize the MERGE command as SQL and would refuse to do anything with it. But I'd hope those versions are all dead and buried by now.]
SQLPlus commands are to manage the SQLPLus environment and have really nothing to do with SQL. They are mostly for formatting output or taking in data that is used with SQL to send to the server.
SQLPlus is a proprietary Oracle product for connecting to Oracle server and issuiung commands. It is being replaced (officially) by SQL Developer.
I strongly suggest you download the free SQL Developer program from oracle:
http://www.oracle.com/technetwork/developer-tools/sql-developer/downloads/index.html
it's a great tool for working with sql, pretty easy to get started (install, create a new db connection and start typing sql_ plus does fill in for columns once it know the tablenames.
Also, it can run most sqlplus commands if you want to format output, etc.
A nice visual intro to the oracle schema and useful if you want to write plsql.