Are there any concerns with using Snowflake as the data repository for a web API from an enterprise architecture perspective?
I think the question to be asked is how are you going to use the data. It is not clear what you mean by web API data repository. If you are talking about the API interaction data, then Snowflake is not the right choice for that. You should look for a transactional data store for such use cases. However, from that data , if you want to derive insights and analytics you can ingest the transactional data to Snowflake and build your analytics layer on top of it. But the question will be why would you like to do that, most of the API products have that analytics engine already built in their product.
Related
This is for a project on Bot Framework Composer (not SDK, so i'm using built in telemetry export settings).
I am looking for the best way to store event logs from bot conversations for analysis. From what I've researched, the method recommended is going through Application Insights, which I activated and tested. The data I require seems to be all captured in table customEvents.
The issue is I need to be able manipulate the data for analysis. But in Application Insights it's read only (and possibly purge via API). I need to be able to add tables, edit text, etc. I have a lot of experice with postgreSQL so that's my first choice for bot log storage.
So my question is, what is the efficient way to get the customEvents data table that is in application insights to a postgres database? From what I see, application insights only exports to azure storage? But that does not have a database option. And if I understand some of the pipelines suggested, they copy data to storage, and then copy to a database. Isn't that a lot of storage cost, as same data will be in application insights, storageBlobs AND postgres?
What is the best pipeline? The goal is to have non-redundant pipeline that transfers event data that is in 'customEvents' to a postgres table with same columns.
(If there is a way to redirect data that goes to customEvents in application isights directly to postgres table that would be perfect too. )
There is no such to redirect data from application insights directly into postgres table.
The first solution is continuous export to azure storage as you know. Storage blob does not cost very much and you can clear the old data periodically to reduce the cost.
Another way is to use the application insights query api. To do that, you need to write your own logic to query the custom events from application insights, then insert them into your DB by your code.
Context: it is my first use of FireStore. I want to use it to push notification status to our Mobile Application. I can see that there is Google Firestore Dashboard under Analytics umbrella. In our company we use mainly three tools for monitoring our applications: Zabbix, Dynatrace and certain internal solution based on Elasticsearch. I need to ntegrate our internal monitoring systems with metrics resulted from our first Firestore project.
What I am looking for: based on personal assumptions:
1) Maybe there might exist either some GET endpoints that a I can connect and poll for information let's say each minute
2) Maybe, following the idea of Database Realtime pushing events accross a long time connection, I can code a Spring Boot application that import Firebase SDK and every day I connect to some specific Firestore endpoint which will push any interested events (eg. delay based on custom logic or dead service)
3) Maybe some plugin I can connect straight to a Kafka hosted in our internal Datacent
4) Some plugin to connect from Firestore/Firebase to either third tools (eg. Zabbix or Dynatrace or Elasticsearch)
5) Some dependency I could import in google-cloud-funtions thiggered from Firestore Healcheck engine in orther to consume some internal end-point posting data
Perhaps there is already some approach universally used for a scenario when you have to connect Firestore to internal monitoring system. I will be highly appreciated if tell me that than I can narrow my googling searchs because I am not finding anything usefull.
Please, it is not part of this question comparing Monitoring approach. It is a very solid fact in our company use internal Dashboards and some custom alerts trigger. I just mentioned the names above to clarify what I mean by internal monitoring tools. The focus on this question is HOW IMPORT/INTEGRATE/OBSERVE/CONSUME Firestore monitoring data. Our internal stack is beyond this question.
Here is the Official Documentation for Cloud Monitoring using which you can collect metrics, events, and metadata from Google Cloud Platform products that you can use to create dashboards, charts, and alerts.
Please let me know if you have further questions.
I'm a hobby developer that use Parse.com as my database and website host, as Facebook is going to shutdown Parse.com, I'm now looking for alternative Parse server.
I use Parse's Cloud Code Hosting to build Dynamic Web App, and Parse itself to store data collected from the website I've build, with custom Cloud Code to help getting and managing data. I've build a Windows Phone app for myself to manage the data I've collected too.
Is there any alternative server that has my requirement?
Dynamic Website
Database host
Custom Cloud Code (with BeforeSave and AfterSave trigger)
with Windows Phone SDK (or REST API if doesn't have)
Very thank you for helping me!
Try out Hasura.
Hasura (http://www.hasura.io): Hasura is a neat PaaS + BaaS solution. It is now competing with Firebase, Kinvey, Heroku et al. There is a full comparison page here: Compare | Hasura (https://compare.beta.hasura.io) . The difference majorly lies in infra ownership as well as no tech lock-in due to open-source components(like docker, kubernetes,postgres) building the major chunk of the platform. Check it out. There is also an option to explore (https://explore.beta.hasura.io/) Hasura by building your own blog web app and a todo app in under 15 mins.
Hasura should fit in perfectly for your needs.
DISCLAIMER : Hasura Engineer here.
I'm using Simbla website application development. It doesn't support all of your requirments but it has great UI builder with a backend parse database.
You can try using the parse open source server it has cloud code and you can use a custom database with it.
I am very new to the MVC frame work and ASP.NET WEB API
I have been Building Web Application using ASP.NET AJAX with n-tier architecture with data as Stored Procedure Only for quite some time.
We are trying to upgrade one of our Products to develop using HTML5 ASP.NET WEB API and We would like to keep our DAL and stored Procedures intact and add a service Layer using ASP.NET WEB API or WCF Data Services on top of the DAL and HTML5 Presentation layer will hit the service layer for the data.
can you suggest if this is a possible scenario where we want to keep the database stored Procedures and DAL intact?
As I have noticed the support for Stored Procedures in EF5 needs a lot more maturity to support some of our stored Procedures with multi table datasets I know there are workarounds on this. I have seen EF 6 Alpha Specifications and I am excited about the features.
Does any one have a link to a tutorial or a sample of ASP.NET WEB API Service layer on top of a Data Access Layer? OR can you make some suggestions or point me to right direction please
If I have to make a wild guess on how I want to implement a solution for our current problem. I would say.
Present DAL gives us Dataset and wrapper for DataTable in the dataset to Convert to IQueryable and use them in ServiceLayer skip the whole EF workarounds.
thanks in advance
In reality you could to this. You could place your service logic into Web API, however, I wouldn't do this. I would rather add one more layer of abstraction in order to keep the API as lightweight and as simple as possible. According to your scenario you have something like this:
Back-end DB Server with Stored Procedures
Data Access Layer component for working with number 1, back-end DB.
Now you want to build API on top of 2. My suggestion to you would be to add Service layer (aka Business Logic layer) where you would put additional logic like calculations, if necessary additional validation, messaging services, etc.
And then on top of the service layer I would add Web API. So at the end your layering could be something like this:
Back-end DB Server with Stored Procedures
Data Access Layer component for working with number 1, back-end DB.
Service Layer (Business Logic)
ASP.NET Web API
HTML5 Client
The idea behind this is that sometime again in the future, when you need to add additional features to your product, you would do it in the service layer. Don't go on and add complexity to your Web API. Think about maintenance, testing and future expansion.
I am trying to find the necessary tables in Websphere Process Server to do some basic productivity reporting. I an completely UN-familiar with WPS, but I assume that there has to be a core set of tables that capture state transitions of workflows.
WPS flow data is managed internally by a DB2 Database, but i haven't seen any application that access directly to that data. WPS exposes a series of API's (EJB's, Web Services and RESTful Services) that allows applications to query WPS flow information but i don't think that approach is suitable for Reporting. More of that here: http://publib.boulder.ibm.com/infocenter/dmndhelp/v7r0mx/index.jsp?topic=%2Fcom.ibm.bspace.help.api.rest.doc%2Frest%2Fwsprocessserver%2Findex.htm
What several companies does for Reporting and Monitoring is query WPS through IBM Websphere Business Monitor: it's a product designed to make reports about process instances in a very fancy way. It's a great product but it's also very expensive.
What are you trying to achieve by looking at the WPS database tables?
WPS tables should ideally be left alone. As suggested by Carlos look at the APIs exposed by WPS and use them for your purposes.
The database - "how it is used internally to store information" is not publicly exposed so you won't find a great deal of information. There is information available on the database views which you can query on:
http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic/com.ibm.websphere.bpc.620.doc/doc/bpc/r6bpc_dbviews.html
I don't know the version of WPS that you are using. I am pointing you to the 6.2 docs. You should be able to locate them in 7.0 or what ever version you are using in your environment.
WB Monitor is a BAM tool that can be used for your BAM needs.