So I want to create a live dashboard (probably a node based app, with react front end). This dashboard will display performance data from a series of websites from the data gathered using Googles Lighthouse Performance audit tool.
The Lighthouse tool published a JSON file with a bunch of keys and values for performance analytics.
I will using something like d3 or chart.js to eventually render this data.
My issue is with how to provide this "live" data to the web front end.
here is my idea so far (just need to know if it is viable.)
A Jenkins job will run my dockerised scrit which uses the lighthouse adk to give it a site and return a json performance report.
The jenkins job will put the json file into an S3 bucket.
A lambda will be triggered each time an item is added to the S3 bucket
The lambda will extract the desired values from the json report and write these to dynamo db
Dynamo DB stream will be used to get the latest values from the dynamo table.
The web front end will query the dynamo DB streams and render the data into chars and graphs.
Can you see this process working? would this give me a sort of "live" data feed? the idea is that the performance reports will be created multiple times during the day
I don't think the DynamoDB stream will work the way you think, unless I'm totally misunderstanding something about DynamoDB streams. How would DynamoDB push streaming data to a web browser?
I would recommend having the Lambda function add a timestamp to each record it inserts into DynamoDB. Have the timestamp field be the sort key for the primary index of the table.
Next have another Lambda function that queries the DynamoDB table for the latest record(s) using the timestamp field. Expose that Lambda function via API Gateway.
Finally have the web front-end make API calls to the endpoint you created in API Gateway to retrieve the latest performance data.
"live" can mean different things to different people and for infrequently changing data (a few times a day is not frequent compared to an interactive chat) the overhead of managing sockets, etc. might not be worth it compared to simply refreshing the page.
I don't see why you need Dynamo here; you can just read from S3 directly and perhaps use versioning on objects to track the different stats for each run.
If you genuinely want browser-based notifications you can look in to AWS IoT, and have a Lambda subscribed to the S3 bucket where the results are run that extracts the values and publishes them to IoT, which can expose a web socket for your browser based app.
Related
This is for a project on Bot Framework Composer (not SDK, so i'm using built in telemetry export settings).
I am looking for the best way to store event logs from bot conversations for analysis. From what I've researched, the method recommended is going through Application Insights, which I activated and tested. The data I require seems to be all captured in table customEvents.
The issue is I need to be able manipulate the data for analysis. But in Application Insights it's read only (and possibly purge via API). I need to be able to add tables, edit text, etc. I have a lot of experice with postgreSQL so that's my first choice for bot log storage.
So my question is, what is the efficient way to get the customEvents data table that is in application insights to a postgres database? From what I see, application insights only exports to azure storage? But that does not have a database option. And if I understand some of the pipelines suggested, they copy data to storage, and then copy to a database. Isn't that a lot of storage cost, as same data will be in application insights, storageBlobs AND postgres?
What is the best pipeline? The goal is to have non-redundant pipeline that transfers event data that is in 'customEvents' to a postgres table with same columns.
(If there is a way to redirect data that goes to customEvents in application isights directly to postgres table that would be perfect too. )
There is no such to redirect data from application insights directly into postgres table.
The first solution is continuous export to azure storage as you know. Storage blob does not cost very much and you can clear the old data periodically to reduce the cost.
Another way is to use the application insights query api. To do that, you need to write your own logic to query the custom events from application insights, then insert them into your DB by your code.
I have a mobile application (Native iOS and Android) that I'd like to have receive realtime state information on whether or not to display certain things in the UI.
Our current backend runs Sails and has a MySQL Database. Sails is deployed over EC2 and the MySQL DB uses AWS RDS. I've been reading about AWS AppSync and believe it's a viable solution to pushing realtime state information from the MySQL DB up to the frontends.
My question is, is it possible to connect AppSync to subscribe to the RDS instance, and if so, does the solution require a Lambda function to run the actual querying against the DB? The entire concept is still new to me & today's the first time I've heard of GraphQL, so any and all information on how AppSync works and how this could be implemented would be great!
Currently I believe I'd create a GraphQL Schema that matches the state information I need, AppSync would expose an API for me to subscribe to with their mobile SDKs?, and I could use my Sails backend to send GraphQL mutations to AppSync which would then (somehow?) update the mySQL DB? Is this a proper flow and could somebody please fill in the gaps here :)
Are you trying to update the database directly/bypassing AppSync?
If yes, the lambda solution you mentioned maybe useful.
If no, by setting up subscriptions and mutations to the database via AppSync is sufficient.
Is it possible to load/import data through third party API to Parse.com daily?
To be more specific, I want to load data from https://api.sample.com/event.json?key=1234 to Parse.com daily. And then sort those data on Parse. And then iOS users pull data from Parse.
Of course. Parse provide a REST API for all collections (db tables) or you could write your own custom code to load the data if you want to add a business rules layer.
You just need to watch your volume of JSON data that you are posting.
I should also add that since you want to do this daily you will need run a daily batch job to do this. Parse has this ability or you can initiative this process manually yourself. It depends how automated this needs to be.
I am a relatively new Parse.Com user. All of the examples involve using the credentials of one of your apps to interact with Parse. I am writing an OSX application to use as sort of a console, and I need to query Parse.Com for a list of all of my registered apps along with their Client IDs so that my console can then interact with Parse on an individual app level.
You can create a master Parse table that has a list of all of your apps with their client IDs and any other info you may need. Then you just need to query that one table to get the rest of the data. This can then be modified on the server.
I am looking for some recommendations on a good data store for activity feeds. The goal is to have a Twitter/Facebook type feed log consisting of various activities users can do throughout our website. The "wall" or "feed" would updated via AJAX showing what the users of the website are currently doing. It will be written to often and then the most recent will be displayed on the site.
(e.g. John Smith recommended Jane Smith's article 2 seconds ago)
We currently are storing the feeds in MySQL but performance has been poor and I'm concerned with hindering performance throughout the rest of the website if we are constantly hitting the database to grab the most recent user activity as well as writing the feeds.
Any recommendations would be greatly appreciated!
Make use of the best caching solutions like memcache to increase performance. Other than scaling, there are no performance-increasing possibilities for an activity feed.
I would vote for using http://redis.io/ or http://www.mongodb.org/ as an alternative to MySQL for short-term, almost live activity feeds across a site. And a cron job to dump history of activities into MySQL for record keeping.
A look at tumblr's or twitters architectures can push you to the right direction as well.
You should take the microservices approach to separate between the datastore that stores the users' actions to the one that store the actual data.
Pub/Sub is the right approach to handle the big stream of users' actions.
Use Kafka or Google Pub/Sub cloud service for a scalable data pipeline. They can take the load with its scalable architecture.
Independently consume the messages from Kafka to some database such as MySQL or Google BigQuery for analytics purposes you must have.