Is there a way to get ALL tweets(or at least 3000) of a certian user using the Twitter Gem?
Twitter.user_timeline("BarackObama")
Doesn't return as many tweets as expected...
By default, user_timeline only returns the latest (up to) 200 tweets 1 with a single request. You could use multiple requests (getting the ID of the last tweet of each request and using the max_id parameter to request older tweets) but you'll want to read up on rate limiting to ensure that you don't get blocked for excessive requests. I expect your application will need a caching solution of some form to prevent this from happening.
Related
We currently use our applications to download some pins information for analytics purposes
Currently we hit the API on hourly basis but we have some problems with rate limiting, so we need the following questions to be addresses:
The documentation states that the rate limit is 1000 calls per hour per user token per endpoint. By looking at the HTTP headers apparently the actual limit is 200. So, which is the correct one?
Are GET /v1/pins/1234 and GET /v1/pins/5678 considered to be a call to the same endpoint?
Are there any application level (so, despite of the used user token) rate-limit in place?
Are there any IP level rate limits?
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Thanks in advance.
PS: I tried to contact them via support form, but it returns a 422. Which is awesome!
For apps that is still in development stage the limit is 200. You will get 1000 if your apps passed the review.
The documentation states that the rate limit is 1000 calls per hour
per user token per endpoint. By looking at the HTTP headers
apparently the actual limit is 200. So, which is the correct one?
Ans: Its 10 per token per app id per hour for app in development(tested), 1000 per token per hour for production app.
Are there any application level (so, despite of the used user token)
rate-limit in place?
Ans: No, rate limit will be 10(app in development) or 1000(approved app)/hour for each tokens you have
Are there any IP level rate limits?
Ans: No
Since we use concurrent requests, are there any limitation on the number of CONCURRENT request? (which means: make another call from the same IP and using the same token, even if the previous one hasn't ended yet)
Ans: No there is no such limit but they will probably check for bot activity and you may end up with banned account, so please do maintain some delay in between requests for same token IMO 12 seconds should be enough
I need to simulate a test scenario where my application sends a request with 100s of queries. On the back-end, this request is broken down into requests containing a single query each. So a request from Jmeter with 100 queries will become 100 requests on the back-end. Now - the response from the back-end could either contain the requested data for each of those queries OR contain a unique queryID. Sending back a queryID is server's way of telling that this query is still running. For example, if Jmeter sends a request with 100 queries, it might get back data for 80 and 20 unique queryIDs. So my application under test makes a callback request with those 20 queryIDs every 15 seconds until it gets back the requested data or timeout.
Here is what I have implemented so far.
-main_request_with_100_queries
--XPath_extractor_to_extract_any_queryIDs_found
-if_controller_to_check_if_queryID_MatchNr_is_greater_than_0
--15_second_pause
--beanshell_preprocessor_to_create_the_request_body_with_all_queryIDs
--callback_request_with_queryIDs
What I want to implement is to have another XPath extractor for my callback_request and if any queryIDs are found, then go back to the if_controller
I'm trying to make this work by using a module_controller but so far no luck. Has anyone ever implemented something like this? Can anyone suggest some ideas?
You can use While Controller to keep making the request until there is a queryID in the response.
While Controller [ "${querid.present}"=="true" ]
HTTP Request
Pre Processor [to_create_the_request_body_with_all_queryIDs]
Post Processor [to check for query ID. if no query id - change querid.present to false ]
If possible, try to use Regular Expression Extractor. xpath is very slow and might affect your performance of the script. Check here for more details.
Creating modular test script in JMeter.
I'm building a web-based reporting tool that queries but does not change large amounts of data.
In order to verify the reporting query, I am using a form for input validation.
I know the following about HTTP GET:
It should be used for idempotent requests
Repeated requests may be cached by the browser
What about the following situations?
The data being reported changes every minute and must not be cached?
The query string is very large and greater than the 2000 character URL limit?
I know I can easily just use POST and "break the rules", but are there definitive situations in which POST is recommended for idempotent requests?
Also, I'm submitting the form via AJAX and the framework is Python/Django, but I don't think that should change anything.
I think that using POST for this sort situation is acceptable. Citing the HTTP 1.1 RFC
The action performed by the POST method might not result in a
resource that can be identified by a URI. In this case, either 200
(OK) or 204 (No Content) is the appropriate response status,
depending on whether or not the response includes an entity that
describes the result.
In your case a "search result" resource is created on the server which adheres to the HTTP POST request specification. You can either opt to return the result resource as the response or as a separate URI to the just created resource and may be deleted as the result resource is no longer necessary after one minute's time(i.e as you said data changes every one minute).
The data being reported changes every minute
Every time you make a request, it is going to create a new resource based on your above statement.
Additionally you can return 201 status and a URL to retrieve the search result resource but I m not sure if you want this sort of behavior but I just provided as a side note.
Second part of your first question says results must not be cached. Well this is something you configure on the server to return necessary HTTP headers to force intermediary proxies and clients to not cache the result, for example, with If-Modified-Since, Cache-control etc.
Your second question is already answered as you have to use POST request instead of GET request due to the URL character limit.
What are the strengths of GET over POST and vice versa when creating an ajax request? How do I know which I should use at any given time? Is it a security-minded decision?
Also, what is the difference in how they are actually sent?
GETs should be used for idempotent operations, that is operations that can be safely repeated more than once without changing anything. Browsers will cache GET requests (for normal and AJAX requests)
POSTs should be generally be used for non-idenpotent operations, like saving something. Although you can use them for other operations if you want.
Data for GETs is sent over the URL query string. Data for POSTs is sent separately. Some browsers have a maximum URL length (I think Internet Explorer is 2048 characters), and if the query string becomes too long you'll get an error.
You should use GET and POST requests in AJAX calls just as you would use GET and POST requests in normal calls. Basic rule of thumb:
Will the request modify anything in your Model?
YES: The request will modify (add/update/delete) data from your data store,
or in some other way change the state of the server (cause creation of
a file, for example). Use POST.
NO: The request will not affect the state of anything (database, file system,
sessions, ...) on the server, but merely retrieve information. Use GET.
POST requests are requests that you do not want to accidentally happen. GET requests are requests you are OK with happening by a user pointing a browser to via a URL.
GET requests can be repeated quite simply since their data is based in the URL itself.
You should think about AJAX requests like you think about regular form requests (and their GET and POST)
The Yahoo! Mail team found that when using XMLHttpRequest, POST is implemented in the browsers as a two-step process: sending the headers first, then sending data. So it's best to use GET, which only takes one TCP packet to send (unless you have a lot of cookies). The maximum URL length in IE is 2K, so if you send more than 2K data you might not be able to use GET.
http://developer.yahoo.com/performance/rules.html#ajax_get
Several of my ajax applications in the past have used GET request but now I'm starting to use POST request instead. POST requests seem to be slightly more secure and definitely more url friendly/pretty. Thus, i'm wondering if there is any reason why I should use GET request at all.
I generally set up the question as thus: Does anything important change after the request? (Logging and the like notwithstanding). If it does, it should be a POST request, if it doesn't, it should be a GET request.
I'm glad that you call POST requests "slightly" more secure, because that's pretty much what they are; it's trivial to fake a POST request by a user to a page. Making it a POST request, however, prevents web accelerators or reloads from re-triggering the action accidentally.
As AJAX, there is one more consideration: if you are returning JSON with callback support, be very careful not to put any sensitive data that you don't want other websites to be able to see in there. Wikipedia had a vulnerability along these lines where the user anti-CSRF token was revealed via their JSON API.
All good points, however, in answer to the question, GET requests are more useful in certain scenarios over POST requests:
They can be bookmarked
They can be cached
They're faster
They have known consequences (assuming they don't change data), so visiting them multiple
times is not a problem.
For the sake of posterity, updating this comment with the blog notes re: point #3 here, all credit to Omar AL Zabir (the author of the referenced blog post):
"Atlas by default makes HTTP POST for all AJAX calls. Http POST is
more expensive than Http GET. It transmits more bytes over the wire,
thus taking precious network time and it also makes ASP.NET do extra
processing on the server end. So, you should use Http Get as much as
possible. However, Http Get does not allow you to pass objects as
parameters. You can pass numeric, string and date only. When you make
a Http Get call, Atlas builds an encoded url and makes a hit to that
url. So, you must not pass too much content which makes the url become
larger than 2048 chars. As far as I know, that’s what is the max
length of any url.
Another evil thing about http post is, it’s actually 2 calls. First
browser sends the http post headers and server replies with “HTTP 100
Continue”. When browser receives this, it sends the actual body."
You should use GET where you're doing a request which has no side effects, e.g. just fetching some info. This request can:
Be repeated without any problem - if the browser detects an error it can silently retry
Have its result cached by the browser
Be cached by a proxy
These things are all good. Anything which is only retrieving data (particularly public data) should really be a GET. The server should send sensible Last-Modified: and Expires: headers to allow caching if required.
There is one other difference not mentioned by anyone.
GET requests are passed in the URL string and are therefore subject to a length limit usually dependent on the browser. It seems that most are around 2000 chars.
POST requests can be much much larger - in fact not limited really. So if you're needing to request data from a web server and you're passing in lots of parameter information then a POST request might be the only option.
So, as mentioned before really a GET request is for requesting data (no side effects) while a POST request is generally used for transmitting data back to the server to be stored (with side effects). e.g. Use POST to upload a file. GET to retrieve a file.
There was a time when IE I believe had a very short GET URL string. Some applications like Lotus notes use large numbers of random characters to represent document id's. I had the displeasure of using another product that generated random strings so the page URL was unique each time. The random string was HUGE... and it didn't always work with IE6 from memory.
This might help you to decide where to use GET and where to use POST:
URIs, Addressability, and the use of HTTP GET and POST.
POST requests are just as insecure as GETs. The main difference is that POST is used to modify the state of the server application, while GET only requests data from it.
The difference matters when you use clean, "restful" URLs, where the URL itself specifies the resource, and the different methods trigger different actions on the server side.
Perhaps most importantly, GET is book-markable / viewable in url history, and searchable with Google.
POST is important where you don't want the event to be bookmarkable or able to be typed in as a URL - otherwise you (or Google crawling your URLS) could end up accidentally doing things like deleting users from your system, for example.
GET
POST
In GET method, values are visible in the URL
In POST method, values are not visible in the URL.
GET has a limitation on the length of the values, generally 255 characters.
POST has no limitation on the length of the values since they are submitted via the body of HTTP.
GET performs are better compared to POST because of the simple nature of appending the values in the URL.
It has lower performance as compared to GET method because of time spent in including POST values in the HTTP body
This method supports only string data types.
This method supports different data types, such as string, numeric, binary, etc.
GET results can be bookmarked.
POST results cannot be bookmarked.
GET request is often cacheable.
The POST request is hardly cacheable.
GET Parameters remain in web browser history.
Parameters are not saved in web browser history.
Source and more in depth analysis: https://www.guru99.com/difference-get-post-http.html