GoodData: "CloudConnect" or another tool for ETL development - etl

We are GoodData customers who are beginning the process of evaluating ETL tools other than CloudConnect. I'd like some recommendations from other GoodData customers who do their own ETL/LDM development with a tool other than CloudConnect. What has been your experience with these other tools? How do they compare with CloudConnect? I have another conversation going on LinkedIn (https://www.linkedin.com/groups/Model-ETL-Development-CloudConnect-vs-6616061.S.5897711443083538433?qid=fbab6f85-4bd2-4515-8737-98a365bf9208&trk=groups_most_popular-0-b-ttl&goback=%2Egmp_6616061). From this conversation I have learned a lot about Keboola but I would like to hear others' experiences with other tools.

Other option is to use our "BI Automation Framework" that is being developed on the top of our Ruby SDK and it is great fit if you are more "Developer/Coder". It will be integrated with our Agile DataWarehouse Service (ADS) where you have option to manage your data transformation process using the Vertica database with SQL. We are moving forward quickly in this space.
Another option you can use is to use the ADS + CloudConnect as orchestration tool. Again, this helps you when doing SQL transformation is more comfortable for you. If you want to start testing those tools, let me know.
JT

Related

Business Intelligence Sample Data or Database

I don't know I should ask this question over here. I know it can be marked too broad or marked as close.
I have started learning BI and data analysis. I search on internet I found some good tools which I am interested QlikView & PowerBI.
I like some help from you guys like Sample data where I can play around and some scenarios. If I get some website or some tips to learn.
I will very much appreciate for some help.
Microsoft Power BI:
https://powerbi.microsoft.com/it-it/desktop/
Sign up (free) and Microsoft will give you sample data to use and see on Power BI.
If you are interested in the single component:
Excel 2013 Power Pivot: https://technet.microsoft.com/en-us/library/gg399078(v=sql.110).aspx or https://support.office.com/en-us/article/Tutorial-Extend-Data-Model-relationships-using-Excel-Power-Pivot-and-DAX-cf7197d3-1938-490e-93fb-20371e8dd67a (I prefer this one)
Excel 2013 Power Query:https://support.office.com/en-us/article/Power-Query-101-008b3f46-5b14-4f8b-9a07-d3da689091b5 or https://support.office.com/en-us/article/Microsoft-Power-Query-for-Excel-Help-2b433a85-ddfb-420b-9cda-fe0e60b82a94
Excel 2013 Power View: https://support.office.com/en-us/article/Tutorial-Create-Amazing-Power-View-Reports-Part-1-e2842c8f-585f-4a07-bcbd-5bf8ff2243a7
Power BI Desktop: https://powerbi.microsoft.com/en-us/documentation/powerbi-desktop-getting-started/
Try the Lahman baseball database. A number of R scrips exist w/ output, so that you can compare your analysis from whatever tool you used.
Once you sign up at www.qlik.com there is a documentation and tutorial download in their download section for each of their products. The complete version of the tutorial includes a data set that will get you going
Qlik sense has a desktop version which is free to try and comes with sample data. Its easy to get started with! http://www.qlik.com/products/qlik-sense/desktop
Best of luck!
For both Qlikview and Qlik Sense you can download their free desktop app.
Afterwards you can install the Google Bigquery ODBC driver and access the Bigquery public dataset. In fact it is beneficial to simply learn BI using Google Bigquery.
Follow this tutorial to see how you can retrieve/manipulate the data.
You can find QlikView and Qlik Sense demos in this link:
http://eu-a.demo.qlik.com/
Both applications are fully free to use for personal use and I would suggest you start your Qlik journey with Qlik Sense. It is their latest product where you can create applications and even share them with your friends through Qlik Cloud.
Definitely worth a go.

BI Tool(information builders vs oracle BI

we are kind of in a ‘decision making situation’ to make a road map of our BI system.
I would like to hear experts opinion on Information Builders when compared to Oracle BI. I am working in Oracle BI but I dont have knowledge on Webfocus offered by Information Builders, so problably users who has knowledge on both sides, has a good overview on pros and cons.
Like to hear any opinion or suggestion.
Thanks in advance.
I currently work with IBI products and their reporting language WebFocus 8. This along with their new AppStudio development tool and BI Portal deployment platform. If you like being forced to use a GUI to build your user-facing components, having no good access to the HTML, then go with IBI and WF. Other than their having forced the developer to lose access to edit HTML, and tool generated code being horrid, it's a fairly good product. Their new push for responsive design is still early in its implementation. Lots of restrictions and good portions of their products rely on being both compatible with IE8 or used in IE. Not much of a fan of IBI.

Where does Oracle ADF 11g stands among Java EE Frameworks?

This is a two part question:
I would like to know where does Oracle ADF 11g stand as a framework to develop / deploy Web / Java EE Applications as compared to other frameworks.
How much is Oracle ADF being used as a framework to Develop Web Applications.
I am asking from a perspective that whether or not there are good job opportunities as an Oracle ADF Developer and what the future looks like for those holding Oracle ADF skills.
Also as Oracle plans to use ADF as core technology for Fusion Applications, Is it good time to build Oracle ADF skills as a siebel developer? As the future Oracle CRM Applications like ERP, SCM will be based on ADF should developers start building ADF skills. Around when will customers start implementing those applications based on ADF. Will these be available On Premise or mostly On Demand in SaaS way.
We have chosen Oracle ADF in our company for one of our projects. Sadly, this proved to be a big mistake. I personally have used Oracle products, in particular their DB, but as far as fusion middleware is concerned, I would advise you to stay away from it. ADF was by far the worst framework I have ever used as an architect. Some of its features I noticed are: very complex, Oracle just "invents" or forces some development approaches that have been proven as a bad practice by the Java community years ago. ADF is very slow compared to other frameworks. With being slow I mean ADF pages run slow. The reason for this is extremely complex generated html and javascript. You can check this by opening up Firebug and inspecting the generated html...it looks like something from the 90s...Terrible. Not to be overly pedantic, but ADF pages have huge amounts of validation errors causing problems in making them run on all browsers. The architecture of ADF is, in my view, messy. Struts2 for example is a much cleaner framework which makes it very easy to integrate it with 3rd party libraries such jQuery, etc. Oracle advocates ADF as an MVC framework but frankly I failed to see the actual MVC architecture there.
ADF uses its own javascript libraries, which are huge, practically impossible to modify, unoptimized and slow in comparison to others such as jQuery or Prototype. The emerging trends in J2EE are lightweight pluggable frameworks that are easily integrated with other tools such a Spring for managing dependencies, jQuery for scripting, CC for continuous integration. ADF is a heavy-weight tightly-integrated framework with other ORacle tools, which makes it very difficult to use it any other way than Oracle devised.
Not to mention the development tools that you are forced to used when developing ADF - the JDeveloper which is very buggy and crashes regularly. Developing in Eclipse is problematic, because ADF has numerous "specific" files which JDeveloper generates automatically.
All in all...ADF was a huge disappointment for us. We spent months messing with it, when finally our client decided the application was too slow and difficult for them to support it and the cancelled it altogether.
Based on my experience i can only say stay away from it. Choose one of the more supported and architecturally clean solutions such as Struts2 or Spring.
I have used JDeveloper 10 and 11, both with ADF, on two enterprise ADF projects for a prominent defense contractor. I agree that ADF is complex, but disagree that JDev crashes frequently, or that the pages are slow to render.
Now... why ADF? Read Oracle's synopsis below, stating that JDeveloper and ADF make Java EE development more accessible to "business" programmers. While this does not guarantee elegant programs, it does almost certainly guarantee that, when these developers get into trouble with ADF, they will probably be utterly lost and will have little idea how to open the hood and reach inside. Conversely, they would be lost even sooner trying to integrate the current J2EE open-source cornucopia.
THE SERMON
Source: ADF Developer's guide (paraphrased)
"Since the early Java days in the late 1990's , the Jave enterprise platform has grown massively and today is used by a large community of developers. However, the developer community is not homogenous and includes developers who are not expert Java programmers, but are business developers who have core competencies in their industries. As the Java EE platform and community grows, the average programmer skill level is declining.
It's hard to imagine that there is a single developer who understands all aspects of the Java EE platform in depth. The problem to address in Java EE is to provide a technology that empowers developers to build cutting edge web and SOA app's without limiting their agility.
Existing Java frameworks such as Struts, Spring, JSF, EJB, GWT, etc are blankets pulled over areas of complexity that expose simpler controls to work with. Using frameworks, developers no longer work with the java core API's but instead interact with interfaces and services exposed by the framework.
Blah, blah, blah...
What seems a small problem for experienced java developers is another hurdle for business developers who are new to java and EE development. This type of developer may turn to a a single dev environment promising integration of all aspects of app development in a single technology chouice. Developers from a 4GL desktop dev background might be attracted by PLSQL, .NET, Adobe Flash.
An experienced Java developer might see this a a big mistake, but how would those developers know, unless Java too provides a single solution for them? The solution is an end-to-end framework which pulls another blanket over the technologies, one that combines different framewokrs and technologies into a single product that exposes visual and declarative development gestures as it's predominant programming methodology... ADF."
I have been working with ADF since last 6 months, and i realise that ADF is made particularly large application with strong database integration. I haven't seen any other Framework which can provide you with so fast and easy development of database rich applications.
Yes i agree that it have some bugs, but still it works well if you know how to work with it.
My perception of and experience with ADF is that it is a very solid and robust framework (unfortunately, the development tools are not.) I'm not sure there's much competition to it, at least, not from a single framework. ADF itself is really several different frameworks combined over the years into a mostly cohesive product. Now that Oracle owns Java, i really expect (hope) parts/much of ADF finds its way into Java EE-proper so we can maybe see alternative implementations and better development tools. If that were to happen, it'd pretty well solidify ADF's future (as long as Oracle doesn't screw the Java community in the meantime and push them to alternative VM platforms).
If you are already familiar with or want to work with Oracle(-related) products, ADF is what you want. Given the market penetration of Oracle, it's probably a reasonable bet that they'll remain dominant and your skills will remain marketable for years to come. Oracle is spending lots of money and acquiring lots of companies to try to own and retain the enterprise.
The SaaS question, to me, is a little harder to answer. While it does seem to be moving in that direction, it's not clear to me if it's mostly marketing's attempt to give it that push or if business-need is driving it. And there are lots of questions i don't really track such as business trust of the cloud for critical data and reliability.
Short answer:
You developing Enterpise solution, your product vendor is Oracle - ADF is a best choice
You indie developer, your solution is not for enterprise, you not going to use other products in the Oracle stack - ADF is not for you, look for another framework.
Long answer:
If you going to Enterprise, if your general product's stack vendor is Oracle. Your best choise is to go along with ADF. ADF becomes better with each major release. I've started using ADF since 11.1.1.2, had a look at 10.x and now using 12.1.3, while still supporting large enterprise applications on 11.1.1.6.
I see a huge improvement and its really hard to find another product that provide better integration and support for whole Oracle stack.
However, if you quite familiar with other framework or/and not going to use other Oracle products and/or your customers can't afford to pay quite a lot for ADF/Weblogic/DB/ECM etc.. - your best choise is to avoid ADF. Its good for large enterprise solutions.
Yes, in fact actually you can develop using free and limited version of ADF - Essentials, use other app server as tomcat or glassfish, but you'll be better off with the other framework. ADF Essentials is only kickstart to large enterprise world, not a solution for small lightweight application.
Development in ADF is a pain in the heart. For a newcomer, the learning curve is too steep. I wish it could be more like learning dot NET.
From my perspective, learning and constructing a functionality in dot NET takes a quarter of the time to do the same in ADF. Of course, it imbalance reduces as a person learns along, but that's pretty much the scenario for beginners. It's frustrating.
I have been working with ADF since last 3 months, and i can tell you that ADF is a really suitable for situations which is essential to provide fast development process for large enterprise applications with strong database integration.
Compairing to other JavaEE framworks like Spring,
It is extremely complex when fixing bugs becasue of the architechtur of the ADF and lack of dev community relative to Spring. Spring has large development community
around the framework.
It is hard to impliment MVC architecture with the ADF inter component integrations.
Spring Developers have much more reputation and demand compared to ADF developers.
ADF is very slow, heavy-weight, tightly-coupled compared to Spring and integrated framework with Oracle tools.
Hope this will help you to solve your problem and gain your knowledge. Happy coding!!!

Which MS technologies would be suited for a data intensive application?

I'm a junior VB.net developer with little application design knowledge. I've been reading a lot of material online regarding different design patterns, frameworks, and methodologies. It's become a bit confusing for me.
Right now I'm trying to decide on what language would be best suited to convert an existing VB6 application (with SQL server backend.) I need to update the UI and add more user functionality and reporting capabilities. Initially I was thinking of using WPF and attempting the MVVM model for this big project. Reports would be generated from SSRS.
A peer suggested using ASP.net and I don't have enough experience to determine what would be better. The senior programmers here are stuck on using VB6 and don't have any input on what to use. They are encouraging me to use the latest technologies.
This application would be for ~20 users in a central location. Ideally I would stick to a Microsoft .net language. Current interface is similar to a datagrid table where the user would click in to see the detail of each record. They would need to have multiple records open at any given time.
I look forward to all the advice I can get.
EDIT 2010/04/22 2:47 PM EST
What is your audience? Internal clients within an intranet
How complex are the interactions you expect to implement? not very... displaying data from SQL server to UI. Allow user updates to said data. Typically just one user modifying a record.
Do you require near real-time data updates? no
How often do you expect to update the application after the first release? twice/year
Do you expect a well-defined set of client platforms? Yes, windows xp environment, potentially upgrading to Win7. Currently in IE.6 moving to IE7 or 8 within a couple of months.
Do users need access from anywhere? No, just from their PC.
What would be wrong about building a simple ASP.Net application in VB.Net using Gridviews for allowing the data access and manipulation? Seems like a simple ADO.Net trial application if you aren't familiar with it in the beginning you will be by the end. CRUD applications are pretty common so it shouldn't be too hard to build it and then refine it as more requirements become apparent.
Sounds like you need to use a web-based solution--this eliminates alot of your potential distribution woes with multiple users. You could use silverlight, but if you are locked into SSRS, this might not be the way to go.

Microsoft Team System value within only a Dev Team

Microsoft Team System appears to be a great platform for process-oriented systems implementation, however if you strip out access for the BAs, PM and Business Users and just purely use it within a Dev team does it have any more value than just using Visual Studio Professional, SourceSafe, a Defect Tracking Tool and a continuous integration server like CruiseControl or TeamCity?
Yes. Every replacement technology you've mentioned is something that is supported by the Team System package (either in this release or the next). All of these components are designed to integrate and work with each other in TFS. This is a high priority of the TFS team for all components. The result is a set of features which in most cases seamlessly integrate with each other.
I'm not familiar with several of the other projects you mentioned but it's unlikely that they integrate as well with each other as the corresponding TFS components. This is not to say they have no integration or perform poorly as products. Just that they are not designed ground up to work with each other. Hence the interaction will not be as crisp as the TFS components.
Is this valuable enough to continue using TFS? Don't know because it would be highly dependent on how much you value this integration.
One of the big selling points for TFS for my team is the coherency it provides to our overall product life cycle. We do allow BAs, PMs and Business Users to have certain levels of access to TFS, but even if we did not, the product would still be of great value to use. The ability to manage our workflows within TFS and enforce consistency across the development team is great.
Some of the features that TFS provides that we use: security, reporting, work flow management, integrated builds, email alerts, branching / merging.
Could you pull it off with a hodge-podge of other tools? Probably, but it wouldn't be as easy to manage and maintain and you probably wouldn't be able to pull out the kind of data necessary for reporting and tracking the way you can with TFS.
On a sidenote, if your counting on Visual SourceSafe as your repository I would highly suggest looking elsewhere. From personal and business experience, I can attest that it cannot be counted on as a stable/robust repository.
My thoughts.
Sure it has value. There are a ton of client features only in the Team SKUs (don't let the name fool you -- they are primarily just the new "super premium" kitchen-sink versions, that also have the nice bonus of including a server CAL for TFS.) Exact specs available here: http://www.microsoft.com/visualstudio/en-us/products/teamsystem/default.mspx
Looking specifically at the collaboration features, again there's clear value in a system whose components were design to "just work" with each other. The setup is streamlined (though it has a ways to go); the UIs are consistent and accessible from each other; the backend feeds a unified reporting/analysis service. If you have a large team, the overall perf/scalability also far exceeds what the typical OSS suite is capable of at the moment.
The question is whether it's worth the $$ to you. Why use Visual Studio Professional instead of SharpDevelop? Why SourceSafe instead of Git? Why not Notepad and specially labeled folders?
All of the commercial products are commercial for a reason (ok, maybe not SourceSafe!). If you want something with a broad feature set, tight integration, well-defined support & testing lifecycle, good fit & finish, etc then it's usually worthwhile to spend the $$ and let your development staff get on with their work. If you don't mind doing setup & troubleshooting yourself, switching between several applications as part of the development workflow, losing the ability to query & report on team statistics as a whole, etc then by all means go open source -- many OSS dev tools are very solid nowadays.

Resources