Is it possible to load/import data through third party API to Parse.com daily?
To be more specific, I want to load data from https://api.sample.com/event.json?key=1234 to Parse.com daily. And then sort those data on Parse. And then iOS users pull data from Parse.
Of course. Parse provide a REST API for all collections (db tables) or you could write your own custom code to load the data if you want to add a business rules layer.
You just need to watch your volume of JSON data that you are posting.
I should also add that since you want to do this daily you will need run a daily batch job to do this. Parse has this ability or you can initiative this process manually yourself. It depends how automated this needs to be.
Related
This is for a project on Bot Framework Composer (not SDK, so i'm using built in telemetry export settings).
I am looking for the best way to store event logs from bot conversations for analysis. From what I've researched, the method recommended is going through Application Insights, which I activated and tested. The data I require seems to be all captured in table customEvents.
The issue is I need to be able manipulate the data for analysis. But in Application Insights it's read only (and possibly purge via API). I need to be able to add tables, edit text, etc. I have a lot of experice with postgreSQL so that's my first choice for bot log storage.
So my question is, what is the efficient way to get the customEvents data table that is in application insights to a postgres database? From what I see, application insights only exports to azure storage? But that does not have a database option. And if I understand some of the pipelines suggested, they copy data to storage, and then copy to a database. Isn't that a lot of storage cost, as same data will be in application insights, storageBlobs AND postgres?
What is the best pipeline? The goal is to have non-redundant pipeline that transfers event data that is in 'customEvents' to a postgres table with same columns.
(If there is a way to redirect data that goes to customEvents in application isights directly to postgres table that would be perfect too. )
There is no such to redirect data from application insights directly into postgres table.
The first solution is continuous export to azure storage as you know. Storage blob does not cost very much and you can clear the old data periodically to reduce the cost.
Another way is to use the application insights query api. To do that, you need to write your own logic to query the custom events from application insights, then insert them into your DB by your code.
Are there any concerns with using Snowflake as the data repository for a web API from an enterprise architecture perspective?
I think the question to be asked is how are you going to use the data. It is not clear what you mean by web API data repository. If you are talking about the API interaction data, then Snowflake is not the right choice for that. You should look for a transactional data store for such use cases. However, from that data , if you want to derive insights and analytics you can ingest the transactional data to Snowflake and build your analytics layer on top of it. But the question will be why would you like to do that, most of the API products have that analytics engine already built in their product.
I have an android app which gets its data from firebase realtime database. For updating the realtime database automatically, I've written a python script which crawls data from a website and processes it. Then it sends the data to my firebase realtime database using the admin sdk. I am willing to store and execute the script on my server, so that it is executed automatically twice a day. Is it safe to upload my serviceAccountKey.json along with it? If it is not, then how can I achieve my desired functionality?
Yes, it is fine to store the service account JSON file in your own server. That's the intended use case. Just make sure it's not exposed to users in anyway.
So I want to create a live dashboard (probably a node based app, with react front end). This dashboard will display performance data from a series of websites from the data gathered using Googles Lighthouse Performance audit tool.
The Lighthouse tool published a JSON file with a bunch of keys and values for performance analytics.
I will using something like d3 or chart.js to eventually render this data.
My issue is with how to provide this "live" data to the web front end.
here is my idea so far (just need to know if it is viable.)
A Jenkins job will run my dockerised scrit which uses the lighthouse adk to give it a site and return a json performance report.
The jenkins job will put the json file into an S3 bucket.
A lambda will be triggered each time an item is added to the S3 bucket
The lambda will extract the desired values from the json report and write these to dynamo db
Dynamo DB stream will be used to get the latest values from the dynamo table.
The web front end will query the dynamo DB streams and render the data into chars and graphs.
Can you see this process working? would this give me a sort of "live" data feed? the idea is that the performance reports will be created multiple times during the day
I don't think the DynamoDB stream will work the way you think, unless I'm totally misunderstanding something about DynamoDB streams. How would DynamoDB push streaming data to a web browser?
I would recommend having the Lambda function add a timestamp to each record it inserts into DynamoDB. Have the timestamp field be the sort key for the primary index of the table.
Next have another Lambda function that queries the DynamoDB table for the latest record(s) using the timestamp field. Expose that Lambda function via API Gateway.
Finally have the web front-end make API calls to the endpoint you created in API Gateway to retrieve the latest performance data.
"live" can mean different things to different people and for infrequently changing data (a few times a day is not frequent compared to an interactive chat) the overhead of managing sockets, etc. might not be worth it compared to simply refreshing the page.
I don't see why you need Dynamo here; you can just read from S3 directly and perhaps use versioning on objects to track the different stats for each run.
If you genuinely want browser-based notifications you can look in to AWS IoT, and have a Lambda subscribed to the S3 bucket where the results are run that extracts the values and publishes them to IoT, which can expose a web socket for your browser based app.
I am a relatively new Parse.Com user. All of the examples involve using the credentials of one of your apps to interact with Parse. I am writing an OSX application to use as sort of a console, and I need to query Parse.Com for a list of all of my registered apps along with their Client IDs so that my console can then interact with Parse on an individual app level.
You can create a master Parse table that has a list of all of your apps with their client IDs and any other info you may need. Then you just need to query that one table to get the rest of the data. This can then be modified on the server.