Twitter API Hashtag Search Results don't contain images - ruby

I am not seeing image entities in my twitter API search results when I search for a hashtag, but if I use the api endpoint for that specific tweet I do.
Using this tweet for an example: https://twitter.com/mrbuddylee/status/733407581788463104
results = #twitter_client.search('#BecauseSummer', { include_entities: true, count: 200 })
result = results.first # not actually the first result, but just to illustrate.
result.to_h[:entities)
=> {:hashtags=>[{:text=>"BecauseSummer", :indices=>[7, 21]}],
:symbols=>[], :user_mentions=>[],
:urls=>[{:url=>"TWITTER_SHORTENED_URL", :expanded_url=>"http://twitter.com/mrbuddylee/status/733407581788463104/photo/1", :display_url=>"pic.twitter.com/FAAY00SYQH", :indices=>[22, 45]}]}
but If I search for the tweet directly:
#twitterclient.status(733407581788463104).to_h[:entities]
=> {:hashtags=>[{:text=>"BecauseSummer", :indices=>[7, 21]}],
:symbols=>[], :user_mentions=>[], :urls=>[],
:media=>[{:id=>733407573345341441, :id_str=>"733407573345341441", :indices=>[22, 45], :media_url=>"http://pbs.twimg.com/tweet_video_thumb/Ci2WTVzUkAE_gGD.jpg", :media_url_https=>"https://pbs.twimg.com/tweet_video_thumb/Ci2WTVzUkAE_gGD.jpg", :url=>"TWITTER_SHORTENED_URL", :display_url=>"pic.twitter.com/FAAY00SYQH", :expanded_url=>"http://twitter.com/mrbuddylee/status/733407581788463104/photo/1", :type=>"photo", :sizes=>{:small=>{:w=>340, :h=>173, :resize=>"fit"}, :thumb=>{:w=>150, :h=>150, :resize=>"crop"}, :medium=>{:w=>392, :h=>200, :resize=>"fit"}, :large=>{:w=>392, :h=>200, :resize=>"fit"}}}]}
Notice the media hash on the second result.
Why is this? Is it possible to get the media url on the initial search request?

I ended up doing a double query to make sure the images are returned:
results = #twitter_client.search('#BecauseSummer', { include_entities: true, count: 100 })
ids = .first(100).map(&:id)
results = #client.statuses(ids)

Related

Can GA4-API fetch the data from requests made with a combination of minute and region and sessions?

Problem
With UA, I was able to get the number of sessions per region per minute (a combination of minute, region, and sessions), but is this not possible with GA4?
If not, is there any plan to support this in the future?
Detail
I ran GA4 Query Explorer with date, hour, minute, region in Dimensions and sessions in Metrics.
But I got an incompatibility error.
What I tried
I have checked with GA4 Dimensions & Metrics Explorer and confirmed that the combination of minute and region is not possible. (see image below).
(updated 2022/05/16 15:35)Checked by Code Execution
I ran it with ruby.
require "google/analytics/data/v1beta/analytics_data"
require 'pp'
require 'json'
ENV['GOOGLE_APPLICATION_CREDENTIALS'] = '' # service acount file path
client = ::Google::Analytics::Data::V1beta::AnalyticsData::Client.new
LIMIT_SIZE = 1000
offset = 0
loop do
request = Google::Analytics::Data::V1beta::RunReportRequest.new(
property: "properties/xxxxxxxxx",
date_ranges: [
{ start_date: '2022-04-01', end_date: '2022-04-30'}
],
dimensions: %w(date hour minute region).map { |d| { name: d } },
metrics: %w(sessions).map { |m| { name: m } },
keep_empty_rows: false,
offset: offset,
limit: LIMIT_SIZE
)
ret = client.run_report(request)
dimension_headers = ret.dimension_headers.map(&:name)
metric_headers = ret.metric_headers.map(&:name)
puts (dimension_headers + metric_headers).join(',')
ret.rows.each do |row|
puts (row.dimension_values.map(&:value) + row.metric_values.map(&:value)).join(',')
end
offset += LIMIT_SIZE
break if ret.row_count <= offset
end
The result was an error.
3:The dimensions and metrics are incompatible.. debug_error_string:{"created":"#1652681913.393028000","description":"Error received from peer ipv4:172.217.175.234:443","file":"src/core/lib/surface/call.cc","file_line":953,"grpc_message":"The dimensions and metrics are incompatible.","grpc_status":3}
Error in your code, Make sure you use the actual dimension name and not the UI name. The correct name of that dimension is dateHourMinute not Date hour and minute
dimensions: %w(dateHourMinute).map { |d| { name: d } },
The query explore returns this request just fine
results
Limited use for region dimension
The as for region. As the error message states the dimensions and metrics are incompatible. The issue being that dateHourMinute can not be used with region. Switch to date or datehour
at the time of writing this is a beta api. I have sent a message off to google to find out if this is working as intended or if it may be changed.

Google Sheets API inserting records twice (duplicated records)

I'm currently working on a Ruby script application using Google::Apis::Sheets. I'm encountering an issue that I'm trying to figure out where is being generated from. Basically when data is append into the Google sheet the data is being inserted twice. So for example, if 50 records are pass to be appended into the Google Sheet, 100 records are created.
# data should be an array of arrays
class Sheets
#some code here
def append(data)
# Initialize the API
service = Google::Apis::SheetsV4::SheetsService.new
service.client_options.application_name = APPLICATION_NAME
service.authorization = authorize
# Prints the names and majors of students in a sample spreadsheet:
# https://docs.google.com/spreadsheets/d/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/edit
spreadsheet_id = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
range = "Sheet1!A:C"
request_body = Google::Apis::SheetsV4::ValueRange.new
request_body.values = data
service.append_spreadsheet_value(spreadsheet_id, range, request_body, value_input_option: 'USER_ENTERED')
end
end
The above method append_spreadsheet_value appends the values into the spreadsheet. I'm trying to figure out if the error is coming from the range or request_body when data is passed into the request_body.values.
The method append is being called from another file reporting.rb which is hosted on AWS Lambda. The method has the following script.
def self.process(event:, context:, db: MySQL.new)
FileUtils.cp('./token.yaml', '/tmp/token.yaml')
last_day = db.get_last_day()
sheets = Sheets.new
data = []
last_day.each do |row|
data.push([row["created_at"].strftime("%Y-%m-%d"), row["has_email"], row["type"]])
end
sheets.append(data)
api_gateway_resp(statusCode: 204)
end
Basically in the method last_day I'm retrieving some records from a DB via a MySQL2 client. I then iterate over last_day and I add each rwo into data. So basically data is an array of arrays holding the records in the following format.
data = [
[2020-08-06, 1, QUARANTINE],
[2020-08-06, 1, QUARANTINE],
[2020-08-06, 1, POSITIVE],
[2020-08-06, 1, POSITIVE],
[2020-08-06, 1, POSITIVE],
[2020-08-06, 1, QUARANTINE],
[2020-08-06, 0, POSITIVE],
[2020-08-06, 1, QUARANTINE]
]
So if data has 10 records when sheets.append(data) data is append to the Google sheep 20 records are created.
Since my other post got nuked, maybe try replacing
request_body = Google::Apis::SheetsV4::ValueRange.new
with
request_body={
"range":range,
"majorDimension":"ROWS",
"values":values
}
The issue was due to the configuration of the Lambda function trigger being set to run every 10 hours.

multisearch give diffrent total hits in different runs

I'm using Elasticsearch 6.8 and python 3.
I'm running on my laptop with 1 node, and there are no threads/processes that insert/update/delete docs into the index while I'm running the multi search
I'm running the following multi search command:
es = Elasticsearch()
search_arr = []
# search-1
search_arr.append({'index': 'test1', 'type': 'type1'})
search_arr.append({"query": {"term": {"confidence": "1"}}})
# search-2
search_arr.append({'index': 'test1', 'type': 'type1'})
search_arr.append({"query": {"match_all": {}}, 'from': 0, 'size': 2})
request = ''
for each in search_arr:
request += '%s \n' % json.dumps(each)
res = es.msearch(body=request)
print("First Query, num of results = ", res['responses'][0]['hits']['total'])
print("Second Query, num of results = ", res['responses'][1]['hits']['total'])
Each time I run this code, I'm getting different results (as I wrote before, there are no processes that insert/delete/update documents)
Why I'm getting different result each time ?
And what I need to do in order to fix it and get consistent results ?
I found the problem cause.
I needed to add refresh = True after the first time I added new data.
(I bulked thousands of new documents and right after use the multi search).

Google Search Console API not returning enough data

https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query
Google Analytics tells me ~20k pages were visited from Google search, but the Google Search Console API returns just under 5k urls. Changing startRow doesn't help.
What's really odd is I've connected Google Search Console to Google Analytics, and when viewing GSC data in GA (Acquisition -> Search Console -> Landing Pages) the GSC data there also gives me ~20k rows.
How do I get all ~20k rows out of the Google Search Console API?
date_str = '2017-12-20'
start_index = 0
row_limit = 5000
next_index = 5000
rows = []
while next_index == row_limit:
req = webmasters_service.searchanalytics().query(
siteUrl='https://tenor.com/',
fields='responseAggregationType,rows',
body={
"startDate": date_str,
"endDate": date_str,
"searchType": search_type,
"dimensions": [
"page",
],
"rowLimit": row_limit,
"startRow": start_index,
},
)
try:
resp = req.execute()
next_rows = resp['rows']
rows += next_rows
next_index = len(next_rows)
start_index += next_index
except Exception as e:
print(e)
break
return rows
For anyone else viewing this post:
When I view 'Search Results' under 'Performance' in the sidebar of the google search console webpage, at the end of the url there is a variable 'resource_id=sc-domain%3example.com'. I recommend using this resource_id variable as your siteURL.

Working with nested hashes in Rails 3

I'm working with the Koala gem and the Facebook Graph API, and I want to break down the results I get for a users feed into separate variables for inserting into a mySQL database, probably using Active Record. Here is the code I have so far:
#token = Service.where(:provider => 'facebook', :user_id => session[:user_id]).first.token
#graph = Koala::Facebook::GraphAPI.new(#token)
#feeds = params[:page] ? #graph.get_page(params[:page]) : #graph.get_connections("me", "home")
And here is what #feeds looks like:
[{"id"=>"1519989351_1799856285747", "from"=>{"name"=>"April Daggett Swayne", "id"=>"1519989351"},
"picture"=>"http://photos-d.ak.fbcdn.net/hphotos-ak-ash4/270060_1799856805760_1519989351_31482916_3866652_s.jpg",
"link"=>"http://www.facebook.com/photo.php?fbid=1799856805760&set=a.1493877356465.2064294.1519989351&type=1", "name"=>"Mobile Uploads",
"icon"=>"http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/og8V99JVf8G.gif", "type"=>"photo", "object_id"=>"1799856805760", "application"=>{"name"=>"Facebook for Android",
"id"=>"350685531728"}, "created_time"=>"2011-07-03T03:14:04+0000", "updated_time"=>"2011-07-03T03:14:04+0000"}, {"id"=>"2733058_10100271380562998", "from"=>{"name"=>"Joshua Ramirez",
"id"=>"2733058"}, "message"=>"Just posted a photo",
"picture"=>"http://platform.ak.fbcdn.net/www/app_full_proxy.php?app=124024574287414&v=1&size=z&cksum=228788edbab39cb34861aecd197ff458&src=http%3A%2F%2Fimages.instagram.com%2Fmedia%2F2011%2F07%2F02%2F2ad9768378cf405fad404b63bf5e2053_7.jpg",
"link"=>"http://instagr.am/p/G1tp8/", "name"=>"jtrainexpress's photo", "caption"=>"instagr.am",
"icon"=>"http://photos-e.ak.fbcdn.net/photos-ak-snc1/v27562/10/124024574287414/app_2_124024574287414_6936.gif", "actions"=>[{"name"=>"Comment",
"link"=>"http://www.facebook.com/2733058/posts/10100271380562998"}, {"name"=>"Like", "link"=>"http://www.facebook.com/2733058/posts/10100271380562998"}], "type"=>"link",
"application"=>{"name"=>"Instagram", "id"=>"124024574287414"}, "created_time"=>"2011-07-03T02:07:37+0000", "updated_time"=>"2011-07-03T02:07:37+0000"},
{"id"=>"588368718_10150230423643719", "from"=>{"name"=>"Eric Bailey", "id"=>"588368718"}, "link"=>"http://www.facebook.com/pages/Martis-Camp/105474549513998", "name"=>"Martis Camp",
"caption"=>"Eric checked in at Martis Camp.", "description"=>"Rockin the pool", "icon"=>"http://www.facebook.com/images/icons/place.png", "actions"=>[{"name"=>"Comment",
"link"=>"http://www.facebook.com/588368718/posts/10150230423643719"}, {"name"=>"Like", "link"=>"http://www.facebook.com/588368718/posts/10150230423643719"}],
"place"=>{"id"=>"105474549513998", "name"=>"Martis Camp", "location"=>{"city"=>"Truckee", "state"=>"CA", "country"=>"United States", "latitude"=>39.282813917575,
"longitude"=>-120.16736760768}}, "type"=>"checkin", "application"=>{"name"=>"Facebook for iPhone", "id"=>"6628568379"}, "created_time"=>"2011-07-03T01:58:32+0000",
"updated_time"=>"2011-07-03T01:58:32+0000", "likes"=>{"data"=>[{"name"=>"Mike Janes", "id"=>"725535294"}], "count"=>1}}]
I have looked around for clues on this, and haven't found it yet (but I'm still working on my stackoverflow-foo). Any help would be greatly appreciated.
That isn't a Ruby Hash, that's a fragment of a JSON string. First you need to decode into a Ruby data structure:
# If your JSON string is in json...
h = ActiveSupport::JSON.decode(json) # Or your favorite JSON decoder.
Now you'll have a Hash in h so you can access it like any other Hash:
array = h['data']
puts array[0]['id']
# prints out 1111111111_0000000000000
puts array[0]['from']['name']
# prints Jane Done

Resources