I need to start a long-term project in mapping out data tables so that we can get a high-level view of what information we store in our Oracle database and how the tables are linked to each other. This is largely for GDPR preparation.
Since our organization has been around for a number of decades, its database is massive. With TOAD for Oracle, I'm able to see all columns in our tables easily, so I started looking at different database mapping tools (ER/ONE, DDM, Astah) but they all look like I need to manually create all the tables and columns and draw their relationships out by hand.
I'm hoping to minimize as much manual labor as possible and am wondering if using TOAD data modeler would help since I'm using TOAD for Oracle anyways. Could I somehow automate the table, column, and relationship creation process?
Our organization only has Oracle's base version unfortunately (I think the premium bundle has data mapper included in it maybe... not sure.) Any thoughts on the options I have?
-
Bundle: Toad for Oracle Base (64-bit), Add-Ons: <-none->
Our organization only has Oracle's base version
Note: TOAD is not an Oracle product, it is owned and developed by Quest.
they all look like I need to manually create all the tables and columns and draw their relationships out by hand
Any decent data modelling tool supports reverse engineer a physical data model from an existing schema. How good the derived model is will depend on how good your schema is (my bet: decades of development without an existing data modelling tool? not good). For instance, if your schema has foreign keys the reverse engineering process will use them to draw the relationships between tables (even if they are disabled). But if there are no foreign keys then you're on your own.
As you're using already TOAD you are right to want the TOAD modelling extension. You can buy it as a standalone purchase. But if your company won't spring for the extra licenses you should check out Oracle SQL Developer Data Modeler. It's free and it has the most comprehensive support for idiomatic Oracle. (I'm not saying it's the best DM tool of them all but it's very good for something which is free). Find out more.
Related
Let's imagine a standard situation.
Having the current DB schema in a working state, I would like to create a snapshot of this state of schema objects, name it SNAP_1.
Then if updated schema and got problems (bugs or unstable work of new code) it would be good to switch quickly - in one command - the whole schema code back to SNAP_1.
I'm wondering is there any built-in feature of Oracle DBMS for versioning:
PL/SQL code (schema objects)
Data (for example, within configuration tables)
Does Oracle DBMS give native tools for versioning at least one of these two?
The answer is no. But Oracle 11.2+ has something called "Editions".
This method has many restrictions. For example, data and table structure cannot be versioned.
Cool thing is that separate sessions can use a different version of the DB objects simultaneously. (package before fix and after fix)
Here is oracle's documentation.EDITION and
Examples of editions
We are developing a large data migration from Oracle DB (12c) to another system with SSIS. The developers are using a production copy database but the problem is that, due to the complexity of the data transformation, we have to do things in stages by preprocessing data into intermediate helper tables which are then used further downstream. The problem is that all developers are using the same database and screw each other up by running things simultaneously. Does Oracle DB offer anything in terms of developer sandboxing? We could build a mechanism to handle this (e.g. have dev ID in the helper tables, then query views that map to the dev), but I'd much rather use built-in functionality. Could I use Oracle Multitenant for this?
We ended up producing a master subset database of select schemas/tables through some fairly elaborate PL/SQL, then made several copies of this master schema so each dev has his/her own sandbox (as Alex suggested). We could have used Oracle Data Masking and Subsetting but it's too expensive. Another option for creating the subset database wouldn have been to use Jailer. I should note that we didn't have a need to mask any sensitive data.
Note. I would think this a fairly common problem so if new tools and solutions arise, please post them here as answers.
I know this question is similar to another one, but I have now tried both the Oracle data provider and the Devart (dotConnect) provider, and cannot get either combination to work well with our Oracle 11g database and the Power Tools "reverse engineer code first" feature.
With the Oracle provider, there was no way to specify a schema or subset of the database, and it cranked out almost 3,000 entity classes, even though the connection was specific to the target application. It grabbed all the Oracle system tables, etc., along with the application data tables.
With the Devart provider, I still could not specify WHAT to generate, and ended up having to kill it off after running for almost a day.
Anyone know how to get this stuff working with Oracle?
We have answered you about dotConnect for Oracle (Entity Developer) at http://forums.devart.com/viewtopic.php?f=1&t=27520.
We are using liquibase as evolutionary DB change management tool in our applications, it works great when we use it in "common" database schemas.
But we also work with GIS applications using esri arcSDE 9.3 platform over Oracle and in this case, all (or almost all) tables (both GIS and 'alphanumeric' tables) in the schema are managed (create table, grants, etc.) through arcSDE. So when we want to create new feature classe now we use arcCatalog, and this way, it's not possible to manage the feature classes’ changes directly through SQL using liquibase or other automate refactoring tool.
So if we cannot use liquibase to manage changes, at least we want to execute the management operations over our features through command line.
We’ve started looking for tools that avoid the use of arcCatalog, and then try to automate the changes using scripts, we are investigating these possibilities:
Try to capture the SQL that arcCatalog/arcSDE is executing each time we make a change in one Feature Class monitoring the oracle connection. It outcomes us a too complex set of SQL instructions that involves indexes, versioning tables, etc. so we give up this way.
Use the sdelayer and sdetable admin commands installed on arcSDE server.
Use the data management tool: a python based library to manage the feature classes, but it has to be executed from a machine with the desktop version installed.
These last two options will provide a way to manage features from command-line, but our target is to find/develop a tool to manage changes similar to the way liquibase do. But with these tools we will have to find a tool that let us map each SQL DDL operation to an arcSDE command, and currently no db refactoring tool provides this (currently we have check liquibase, dbdeploy, flyway).
Anybody had solved this problem of evolutionary change management with arcSDE? Any insight of another way to face this problem?
I'll take a stab at this, although I'm unfamiliar with one of the products you mention (liquibase specifically - I have used Oracle and I'm very familiar with ArcGIS (ArcMap & ArcCatalog).
Here's just some additional information that may help and my interpretation of your question.
My interpretation - "What's a simple way to manage or enable us to automate the management of our tables of GIS data in our Oracle database without having to use ArcCatalog all the time?"
So - I'll throw this concept back in the ring - I know SQL Server has spatial datatypes "geometry", etc. and that you can bypass SDE and let ArcGIS directly connect and interpret this data without even installing SDE. I also know Oracle has compatible spatial types. So I would possibly consider migrating my data from the managed FC's that ArcCatalog creates and push them into oracle-native geometry based tables. This way you can treat them like regular tables, cut ESRI out of the solution, and manage them with liquibase, etc. Hopefully that helps.
I would also consider upgrading to 10.1 or at least 10.0 (I promise I'm not an undercover salesman), although that will require your users to come with you on the client side (http://resources.arcgis.com/en/help/main/10.1/index.html#//002q000000n8000000) because the newer python APIs are much easier and faster to use (arcpy vs. the GP model), if you do choose to use Python to manage your stuff. (Regardless, either API isn't very well developed and isn't intuitive to code in or fast.)
Good luck.
My company has an existing Database with dozens of DB tables with various relationships.
Is there any tool available that will generate a relationship diagram?
I primarily use SQL Developer 3.0.4 on an Ubuntu 11.10 machine by the way.
SQL Developer Data Modeller can reverse engineer a database schema & make a start on an ERD using the foreign keys. Obviously some manual work will be needed once the initial import has been done in order to generate a nice looking meaningful diagram.
Since you already have SQL Developer installed, you can make a start with File->Data Modeller->Import->Data Dictionary.