Progress/OpenEdge is not a SQL database and although it does have support for SQL (through the ODBC interface) that is not the natural way to access OpenEdge data. Progress applications are built using "ABL" (formerly known as "Progress 4gl") and some SQLish things like referential integrity constraints are implemented through procedural logic in 4gl code. Which makes things difficult if you are not really interested in learning ABL.
Has anybody successfully managed to use Apache Drill (https://drill.apache.org) to access Progress data? Are there any pointers particular to the combination of Progress and Apache Drill to share?
Short answer: No, this is impossible.
Longer answer: my admittedly cursory read of what Apache/Drill does lead to the mistaken conclusion that you can just access Progress (or anything else) via ODBC and then use Drill on top of it.
Apache/Drill is exposing itself via ODBC but currently the only datasources it can use are:
HBase
Hive
MapR-DB
Avro
Parquet
File system:
CSV (Comma-Separated-Values)
TSV (Tab-Separated-Values)
PSV (Pipe-Separated-Values)
Therefore at the time being Progress cannot be accessed by Apache/Drill.
Related
Is it possible to use the saiku-ui component with a different jolap provider than mondrian, or with a different server backend than the saiku-server component?
I have been looking but I have not found an architecture description of how these pieces fit together and what interfaces they use to communicate. Can anyone point me towards an understanding of what the saiku-ui wants to speak with and what the saiku-server is providing?
The reason for my interest is that I have a set of data spread across hundreds of csv files that I would like to query with a pivot and charting tool. It looks like the standard way to use this with saiku would be to have an ETL process to load in to an RDBMS. However, this would not be a simple process because the files and content and the way the files relate to each other vary, so the ETL would have to do a lot of inspection of the data sources to figure it out.
Given this it seems to me that I would have three options in how to use saiku:
1) write a complex ETL to load in to a rdbms, and then use a standard jdbc driver to provide the data to modrian. A side function of the ETL would be to analyze the inputs and write the mondrian schema file describing the cubes.
2) write a jdbc driver to access the data natively. This driver would parse sql and provide access to the underlying tables. Essentially this would be a custom r/o dbms written on top of the csv files. The jdbc connection would be used by mondrian to access the data. A side function of this custom dbms would be to produce the mondrian schema file.
3) write a tool that provides a jolap interface to the native data (accepts discovery and mdx queries). This would bypass mondrian entirely and interface with the ui.
I may be a bit naive here but I consider each of the three options to be feasible. Option #1 is my least preferred because of the likelihood of the data in the rdbms becoming out of sync with the cvs files. Option #3 is most preferred because the data are simple, so not much aggregating required and I suspect that mdx will be easier to parse than sql.
So, if i could produce my own jolap data source would it be possible to hook the saiku-ui tools up to it? Where would I look to find out the interface configuration details?
Many years ago, #ronaldbouman created xmondrian - the set of tools with the olap server, and web ui tools for xmla browsing and visualusation. But that project was not updating, and has no source code.
I just updated olap server and libraries to the latest versions.
You may get it here and build:.
https://github.com/Muritiku/xmondrian-build.
You may use web package as the example. The mondrian server works with the saiku-ui.
IMHO,
I would not be as confident as your are, because it took Julian Hyde more than a decade to build Mondrian (MDX->SQL) and Calcite (SQL), fulfilling your last two proposals.
You might simply consider using Calcite, or even better Dremio. Dremio has a JDBC interface, and can query directories of CSV files in SQL. I tested Saiku over Dremio successfully (with a schema based on two separate RDBMS). Just be careful to setup tables' schema accordingly in the Mondrian v4 schema.
Best regards,
Fabrice Etanchaud
Dremio
I have been tasked with creating some backups for some Oracle Apex apps (Application Express v4.1.1.00.23). The request is to back up both the applications & referenced db objects (not sure if this means just structure or structure & data).
On the one hand, I would have expected standard db backups to handle most or all of this but I'm very new to Apex so it's all a learning curve.
I'm currently exporting the application from apex and then exporting (using SQL Developer) all the database object dependencies that Apex gives me - although I see that the list doesn't include functions that are used for auth.
This seems a really clunky process that's very prone to mistakes (miss an object, save something to the wrong place, no guarantees of consistency etc).
Does Apex (my version!) offer something to do the job or is there something else I could be doing? I've had a good google but nothing has stood out.
UPDATE: I realise now that I should have included some extra info. I'm currently at a large organisation & I believe our db backups (which I guess/hope are done using rman) are done by a different department. I think the motivation for the request is so that we have some local, easily accessible backup so that if one of the developers messes something up we don't have to go through multiple layers of organisation (& undoubtedly a lot of time) to sort ourselves out. I suspect that some kind of source control would be a great starting point but I'm not sure how far I'll get with that idea - especially as we seem to have little in the way of autonomy over things like servers.
RMAN is the way to go to backup an oracle database:
https://docs.oracle.com/database/121/BRADV/toc.htm
there are tons of material on the hows and whys online; just google "oracle rman" and you'll find what you need (the documentation should cover you as well of course).
cheers
Standard DB backups will include everything you need.
The Apex applications I develop are static, meaning the end users make no changes to the Apex application, and there is no need to make a specific backup other than to store the original apex application .sql installation files in a safe place.
If you must, you can make an export of the database schemas the application uses. For example with the expdp utility.
IN apex you need to take 2 backups one in the workspace
Second your application
Third is while using export import from the database it tends to loose & character in procedure ..So beter use rman and take a complete backup.
I have found this Oracle white paper Life Cycle Management with Oracle Application Express (Revision 2) which does what it says on the tin - including various strategies for exporting, backing up & managing 'lost application development'. It's a really good read and I'll be using it as a template for suggestions of how we can manage our process in future.
Does Cloudera Imala- Does it support Stored procdudre/ Function or SQL Blocks like PL/SQL?
I have some logic to be built which will need multiple sql statements. Is there another way to achieve this?
Impala only provides SQL.
You need to use some programming language or tools to build applications - SQL it self cannot be used. PL/SQL is traditional database programming language which runs in PL/SQL engine embedded in database. It reduces overhead between application server and database server, as the PL/SQL engine is embedded in the database. However, storage is network mounted and still data have to come to database server.
In case of big data technologies, there is not much beneficial of doing so. Even if you use Java, it will be running on all the nodes in the cluster and code goes to data. To build applications using Impala, you can just use Java with JDBC or you can even connect third party applications. Hence there is very little advantage of having programming language like PL/SQL.
We are using liquibase as evolutionary DB change management tool in our applications, it works great when we use it in "common" database schemas.
But we also work with GIS applications using esri arcSDE 9.3 platform over Oracle and in this case, all (or almost all) tables (both GIS and 'alphanumeric' tables) in the schema are managed (create table, grants, etc.) through arcSDE. So when we want to create new feature classe now we use arcCatalog, and this way, it's not possible to manage the feature classes’ changes directly through SQL using liquibase or other automate refactoring tool.
So if we cannot use liquibase to manage changes, at least we want to execute the management operations over our features through command line.
We’ve started looking for tools that avoid the use of arcCatalog, and then try to automate the changes using scripts, we are investigating these possibilities:
Try to capture the SQL that arcCatalog/arcSDE is executing each time we make a change in one Feature Class monitoring the oracle connection. It outcomes us a too complex set of SQL instructions that involves indexes, versioning tables, etc. so we give up this way.
Use the sdelayer and sdetable admin commands installed on arcSDE server.
Use the data management tool: a python based library to manage the feature classes, but it has to be executed from a machine with the desktop version installed.
These last two options will provide a way to manage features from command-line, but our target is to find/develop a tool to manage changes similar to the way liquibase do. But with these tools we will have to find a tool that let us map each SQL DDL operation to an arcSDE command, and currently no db refactoring tool provides this (currently we have check liquibase, dbdeploy, flyway).
Anybody had solved this problem of evolutionary change management with arcSDE? Any insight of another way to face this problem?
I'll take a stab at this, although I'm unfamiliar with one of the products you mention (liquibase specifically - I have used Oracle and I'm very familiar with ArcGIS (ArcMap & ArcCatalog).
Here's just some additional information that may help and my interpretation of your question.
My interpretation - "What's a simple way to manage or enable us to automate the management of our tables of GIS data in our Oracle database without having to use ArcCatalog all the time?"
So - I'll throw this concept back in the ring - I know SQL Server has spatial datatypes "geometry", etc. and that you can bypass SDE and let ArcGIS directly connect and interpret this data without even installing SDE. I also know Oracle has compatible spatial types. So I would possibly consider migrating my data from the managed FC's that ArcCatalog creates and push them into oracle-native geometry based tables. This way you can treat them like regular tables, cut ESRI out of the solution, and manage them with liquibase, etc. Hopefully that helps.
I would also consider upgrading to 10.1 or at least 10.0 (I promise I'm not an undercover salesman), although that will require your users to come with you on the client side (http://resources.arcgis.com/en/help/main/10.1/index.html#//002q000000n8000000) because the newer python APIs are much easier and faster to use (arcpy vs. the GP model), if you do choose to use Python to manage your stuff. (Regardless, either API isn't very well developed and isn't intuitive to code in or fast.)
Good luck.
I'm trying to move snapshots of data from our MongoDB into our Oracle BI data store.
From the BI team I've been asked to make the data available for ODI, but I haven't been able to find an example of that being done.
Is it possible and what do I need to implement it?
If there is a more generic way of getting MongoDB data into Oracle then I'm happy to propose that as well.
Versions
MongoDB: 2.0.1
ODI: 11.1.1.5
Oracle: 11.2g
Edit:
This is something that will be queried once a day, maybe twice but at this stage the BI report granularity is daily
In ODI, under the Topology tab and Physical Architecture sub-tab, you can see all technologies that are supported out of the box. MongoDB is not one of them. There are also no Knowledge Modules available for importing/exporting from/to MongoDB.
ODI supports implementing your own technologies and your own Knowledge Modules.
This manual will get you started with developing your won Knowledge module, and in one of the other manuals i'm sure you can find an explanation on how to implement your own technologies. (Ctrl-F for "Data integrator")
If you're lucky, you might find someone else who has already implemented it. Your best places to look would be The Oracle Technology Network Forum, or a forum related to MongoDB.
Instead of creating a direct link, you could also take an easier workaround. Export the data from the MongoDB to a format that ODI supports, and MongoDB can extract to. CSV or XML maybe? Then load the data trough ODI into the oracle database. I think... that will be the best option, unless you have to do this frequently...
Look at the blog post below for an option;
https://blogs.oracle.com/dataintegration/entry/odi_mongodb_and_a_java
Cheers
David