Continuing get request from where it left off after hitting rate limit? - ruby

Question

I found a script that I can use to create a list of all the people a twitter user is following (in this case "#kingjames"). There used to be a private method next_cursor that was somehow public in the Twitter API, but now it's really private in that I can't call it and it throws an error if I try.
Removing all references to next_cursor in the script below gets rid of the error message but the script doesn't have a way to start adding users. The script has to pause for 15 minutes due to rate limits from where it left off, so it's adding the same users over and over again.
Can you suggest a way to change the script to make it continue from where it left off so the same users aren't added over and over again?
#!/usr/bin/env ruby
require 'rubygems'
require 'twitter'
def fetch_all_friends(twitter_username, max_attempts = 100)
# in theory, one failed attempt will occur every 15 minutes, so this could be long-running
# with a long list of friends
num_attempts = 0
client = client = Twitter::REST::Client.new do |config|
config.consumer_key = "8nwjpoIsqag..."
config.consumer_secret = "Wj20rZEfPsyHd0KnW..."
config.access_token = "363090951-n5NdXfp5wWCkNU5eY..."
config.access_token_secret = "7eydU2nQHMsSVB8W76Z2PKH1P...."
end
myfile = File.new("#{twitter_username}_friends_list.txt", "w")
running_count = 0
cursor = -1
while (cursor != 0) do
begin
num_attempts += 1
# 200 is max, see https://dev.twitter.com/docs/api/1.1/get/friends/list
friends = client.friends(twitter_username, {:cursor => cursor, :count => 200} )
# friends = client.friends(twitter_username ).take(200) //seems like another way to get 200 users
friends.each do |f|
running_count += 1
myfile.puts "\"#{running_count}\",\"#{f.name.gsub('"','\"')}\",\"#{f.screen_name}\",\"#{f.id}\""
end
puts "#{running_count} done"
# cursor = friends.next_cursor
# break if cursor == 0
rescue Twitter::Error::TooManyRequests => error
if num_attempts <= max_attempts
# cursor = friends.next_cursor if friends && friends.next_cursor
puts "#{running_count} done from rescue block..."
puts "Hit rate limit, sleeping for #{error.rate_limit.reset_in}..."
sleep error.rate_limit.reset_in
retry
else
raise
end
end
end
end
fetch_all_friends("kingjames")

Answer 1

There used to be a private method next_cursor that was somehow public
in the Twitter api but now it's really private in that you can't call
it and it throws an error if you try.
There is no restriction in ruby against calling private methods. The only restriction is that you can't explicitly specify the receiver when you call a private method. Here is an example:
class Dog
private
def next_cursor
puts "I'm the next cursor."
end
end
friends = Dog.new
friends.next_cursor
--output:--
1.rb:10:in `<main>': private method `next_cursor' called for #<Dog:0x00000100a40fc8> (NoMethodError)
friends.instance_eval do
next_cursor #You can't explicitly specify a receiver for a private method,
#and ruby uses whatever object is self as the receiver
end
--output:--
I'm the next cursor.
If you don't know the ins and outs of Ruby, then that code will seem pretty confusing. But Ruby also provides a method named send(), which allows you to call any method:
friends.send(:next_cursor)
--output:--
I'm the next cursor.