I am developing a Django app that functions basically as a data entry tool for websites. The use case has a trusted user or paid technician browsing the web. As they browse they enter data into an overlaid bar similar to what you see on many proxy websites, but containing a form that allows user to write metadata about the website (in this case, training classification data for an ML algorithm) and submit it to my app.
See http://hidemyass.com/proxy/ for an example of a proxy website that inserts an overlay into browsed sites.
I have heard conflicting suggestions on how to approach this.
Serve Websites as Proxy
Pipe all url requests through the django app with something like http://httpproxy.yvandermeer.net/, and rewrite the responses to include the header.
Pros
I can process the responses with sexy scientific libraries like the NLTK
AJAX-free failover. Users can submit human data (albeit with more of a hassle) without the need to submit computed data.
Cons
Greatly increased traffic. Now my webapp has to retrieve all websites and upload them to the user.
Some websites might block proxy requests. My intention is to deploy this on Heroku, but they might frown on an app that generates so many requests.
User Browses in an iFrame
The overlay is separated from the content by an iFrame, and I use javascript to inform the overlay on the page that is currently being browsed
Pros
Distributed Computing. User machines are used to make requests and do any necessary computations. The server is no longer a bottleneck.
Tighter Ajax integration. I can just post a JSON object representative of my entire Model.
Cons
iframes weren't really designed for full-scale browsing. Some websites force themselves out of iframes, and I worry that it won't be a reliable method of browsing.
I don't get to use all those sexy python libraries. My language processing will have to be done in javascript.
Question
I've never done anything like this before. I'm pretty new to all the tools involved, and seriously having trouble choosing between the two very different approaches.
Which method would you suggest? Why? Are there any considerations I have missed?
OKFN's annotator provides imho a good basis for what you are trying to accomplish http://okfn.github.com/annotator/
Related
Can anyone help me out how to implement the following scenario using Jmeter, in this scenario N number of users should be able to log in at once and implement the same scenario at the same time?
I have already tried using recording option but it is not updating in the database.
To some HTTP requests, it is showing SSO issue.
Steps:
1. Login to the web application.
2. A landing page is displayed.
3. Two options displayed.
4. Selecting one option, which will redirect to the specific landing page.
5. From home page, we will search for the products.
6. Selecting one product and it redirects to the customization page.
7. In the customization page, we are inputting the values, updating the changes to the order.
8. Finally completing the order with update changes in it.
Where the customize order will ask us to move into the cart or it will ask us to submit for approval.
The solution basically depends on what you exactly testing.
Let's consider the application consists of single HTTP server (at least, in terms of endpoint) + single HTTP client (which doesn't reach different sources for data/services, especially with modern protocols like websocket).
Then each of your scenarios turns out to be a series of HTTP request/response interactions, whatever happens under the hood.
So the best way to lay it out for JMeter would be to record sample(s) of this interaction & parametrize it.
You can do it with JMeter itself (check, for example, this guideline or "official" one), or use other tools (like Fiddler) then implement the sequence in the JMeter.
But that is for more, say, "classic" web app, where client (browser) are pretty lightweight and mostly in charge of sending requests & rendering responses - while most of the duties lays on server, and this is that you going to test for performance.
For modern, web apps with responsibilities blurred & spread between server AND client (not to mention asyncronous interactions) that way may be good enough too - but may be NOT.
Then you'd may want to evaluate performance for app as a whole, in assembly.
One of the way to achieve that in JMeter is to use WebDriver plugin for JMeter, check this tutorial.
My single page app is hosted on Google's cloudstore. I love that I don't have to worry about a server. The app is, naturally, javascript heavy.
Now I would like to add a feature where users can store some data, generate a link to be shared with others and retrieve stored data. Think of a pastebin where some snippet of text is saved and a unique link is generated to be shared with others.
In fact, if it helps, think of this as my attempt to create a pastebin without having to setup a server.
It looks like Google's cloud datastore nosql solution is what I want. Given a key, it will return a snippet of text. However, all the examples on the documentation page imply that I have to setup a back end service using python, node, etc.
Questions:
Can't I just read and write from a web page, perhaps using ajax style http call (since I need to get and put text snippets once data has already been loaded)? I believe I can take care of cross-origin issues by changing some configs in the cloudstore static website server.
Obviously I don't want to serve any encryption keys from the web page. I'm hoping that since my site is served from Google as well, I can configure the nosql service handle permissions intelligently for this scenario.
Is there any documentation which shows how to do this correctly?
Google Datastore is not supposed to be used from client side, it's a served side database. You cannot do that w/o having server side code to authenticate, authorize and validate db related requests.
But there're an alternative. Firebase is a ready to use backend for client side applications, including Javascript apps. It's a separate project, that belongs to Google but not (yet?) part of Google Cloud. Take a look - https://www.firebase.com/
Although the API Rest is still beta, it is possible now to connect from a web client or anything RESTful capabilities. https://cloud.google.com/datastore/reference/rest/
So, I'm starting out building a social network web app. I'm looking into how to fit the parts of my stack together and I'm looking for some guidance about what various frameworks will allow me to do. My current stack idea is to have:
Firebase JSON API: serving user, post, comment, and all the other data
EmberFire: to plug that API into EmberJS
EmberJS: my front-end MVC (because I'm new to MVC and Ember seems the most accessible)
What I'm stumbling on at the moment is how I'm going to implement users with this stack. I've looked at basic authentication stuff but I haven't found anything that would allow me to allow certain actions and views for certain users and not others - the basics of a social network really.
Is it sensible to be doing this stuff in front-end MVC? If so what should I be using to do authentication/personalisation? If not, should I just be doing a PHP/SQL setup? I'd rather avoid that because my skills are all front-end.
If you are just getting started, Firebase is a great service to learn on due to their 'back end as a service' model - you will spend more time building/modeling your data and less time running/installing. Not that you won't want to learn more about that later, but it lets you focus on one piece at a time.
From an access perspective, JS/NoSQL vs PHP/MySQL isn't going to be the issue. They each have their own security requirements - it's more that PHP/MySQL has had more time to establish those rules. Additionally, Firebase being a hosted service has it's own set of requirements.
Firebase security rules are a little weird when you first look at them, but they begin to make some sense after you sit with them for a bit. The Firebase docs are actually a pretty great resource. https://www.firebase.com/docs/security/
Basically, if you use a Firebase 'login provider' it makes Firebase act as both a database and a authentication manager, and the combo helps keep users 'fenced' to where you want them. You can use data from other paths, variables, validation rules, etc. You can even make a 'custom login provider' if you need to integrate with an existing one.
Finally, on the client end, your view can respond to whatever Firebase returns - if a user does 'hack' their way through to a page they shouldn't be on client side, no data is returned anyways and no submitted information would be allowed because of the rules.
We are developing a free, open source Google Reader alternative at http://reader.pykih.com and the code is at http://github.com/pykih/reader
When a user signs up or adds a RSS feed, we add the feed url to the database and then ask a DelayedJob to fetch articles from that RSS url. This typically takes few seconds to minute or two, depending on the DelayedJob queue. Many users have complained that RSS feed is not being fetched at all when in reality it is being fetched. All that the user has to do is refresh his own page. We wrote a message there, yet users are complaining.
Can anyone point us in direction towards - what is the best way to design a Facebook or Google style "Loading" (icon in yellow) functionality and once loaded it automatically adds the entries to the screen without page refresh.
Thank you in advance
If I understand correctly, you basically want to update the user's view of the page while it's still open, in real time. (At any rate, that's what Facebook and most Google products do nowadays). This technique is usually called server push - information is pushed from the server to the client, instead of having the client request (pull) information from the server.
There are multiple ways to implement server push.
You could use AJAX to 'reload' the page every ten seconds or so. This is very easy to implement, but not realtime at all, and could cause unneeded load on your server. It works with all browsers.
You could use EventSource, a relatively new format supported by Firefox, Chrome, Safari, Opera and others (but not IE). It's a very simple format and easy to implement. EventSource is one-way communication only: it sends events from the server to the client, not in reverse.
You could use WebSockets, probably using a library like EventMachine and a WebSocket library. WebSockets allow fast bidirectional communication, but it's more complex than EventSource and only the newest IE versions support it.
You could use a commercial service like Pusher. Pusher is easy to integrate and fast, but not free. Browser compatibility is great, though.
The options differ primarily in the amount of client support (do you need IE support?) and the amount of Ruby integration you get.
What is the difference between all three? They seem to do the exact same thing. Why and when would you choose to use one method over the other?
AJAX and websockets do similar tasks — they both establish a communication channel to a server. Web workers have nothing to do with either of them, they are just separate threads of JS execution.
AJAX is more mature than websockets — it has been around much longer and has a much wider browser support. AJAX is request-oriented — you make a request to the server, the server responds, and the connection is closed. Websockets on the other hand establish a persistent connection to the server, over which you exchange multiple messages in both directions.
Webworkers are useful if you want to perform a processor intensive task without blocking the browser interface.
They are not same. But one can use them together to build advanced application.
Ajax: As abbrevation States is asynchronous javascript and xml.. is used to load the content dynamically from the server upon called.
Websockets : Websockets is the feature defined in HTML5 . As wikipedia states "WebSocket is a protocol providing full-duplex communications channels over a single TCP connection." so this is mainly used for real time communication such as video call, live chat etc..
WebWorkers : this feature is also defined in HTML5. This is basically used to make bring multi threading feature in Javascript. Since javascript is a single threaded programming language , it breaks or pause whenever i.e heavy calculation tasks are done using it. to overcome this breakage , Webworkers are added to javascript.
You can perform Ajax and Websockets inside Webworkers . however you cannot manipulate DOM using webworkers due to security reasons.
They are not the same.
Ajax: It is a way of interacting with a web server asynchronously from a UI renderer
Web Sockets: An HTML5 feature using which you can interact with any Socket server extending the reach of the browser
Web Workers : Another HTML5 feature that helps you do multi-threaded programming from a web browser using Java Script
Ajax & Websockers are siblings.
Webworkers are completely different.
AJAX
The best example of AJAX is Google's search bar - suggestions appear as you type, but the current webpage is not redirected or refreshed! (10 years ago this was amazing, not so much anymore). This is AJAX in action.
AJAX uses what's called a "request" and "response" model: you ask a question, and you receive an answer from the 'server'.
AJAX allows webpages to talk to "servers" behind the scenes, allowing you to update a webpage without navigating away from your URL. Back in the old days of the web, if you wanted to show different content on a webpage, users would have to navigate to a different URL: not any more. This concept has been taken to the next level with single page apps and applications (like React, Vue, Elm etc.).
Websockets:
With websockets, your web-page talks to your server (as with Ajax), and your server responds - except you do so like you're talking on the phone. There is a "connection" between your users and your server. This "connection" is not there with AJAX: in that case, you have a simple request and a response coming back from the server.
In other words, if you wanted to stream stock market data, constantly updating it to your users: it would probably be better to use websockets, than AJAX.
Web-workers:
Use When you need intensive calculations - if you were to ask a web-page to calculate Pi to 100000 decimal places: that might take a while. The web-page might freeze, and you might lose $$. The intensive calculations can be done in the background, without freezing your webpage. People using your site can do other things - e.g. click around, while waiting for the result in the meantime.