How to get athleteId from first/last name using ESPN APIs? - espn

Using ESPN's developer API's, you can query for news on a specific athlete using the 'athleteId'. See the Methods section of their docs:
http://developer.espn.com/docs/headlines#parameters
If you look up a player profile you can find the athlete id in the url of the page, but my question is how can we find this id using solely the APIs? With just the athlete's name?

If you go to a team's roster page, use inspect element, or whatever equivalent for the browser you use, and look at the table element of all the players on the team. Within the 'td' tags, you'll find a player ID. You could scrape the screen, store all of the player ids locally, and then use them as parameters in your api calls.
This is merely a suggestion, but it should work, if the id that is used on the page is the same as the id needed to get that player's information in api.

As stated, it's all available in the api:
import requests
import pandas as pd
url = 'https://sports.core.api.espn.com/v3/sports/football/nfl/athletes?limit=18000'
jsonData = requests.get(url).json()
players = pd.DataFrame(jsonData['items']).dropna(subset='firstName')
players = players[['id', 'fullName']].dropna()
Output:
print(players)
id fullName
6 14856 Isaako Aaitui
7 3058033 Manny Abad
8 16836 Jared Abbrederis
9 2576467 Mehdi Abdesmad
10 14466 Isa Abdul-Quddus
... ...
16920 16424 Mike Zupancic
16921 15462 Markus Zusevics
16922 11317 Jeremy Zuttah
16923 4294520 Brandon Zylstra
16924 4608362 Shane Zylstra
[16919 rows x 2 columns]

All NFL player ID can be found on this page
https://sports.core.api.espn.com/v3/sports/football/nfl/athletes?limit=18000
This will return all NFL players ID , name and DOB
function GetESPNIds() {
var url = 'https://sports.core.api.espn.com/v3/sports/football/nfl/athletes?limit=20000';
$.ajax({
type: "GET",
url: url,
success: function (data) {
for (var i=0;i<data.items.length;i++) {
if(data.items[i].hasOwnProperty("lastName") && data.items[i].hasOwnProperty("firstName") && data.items[i].hasOwnProperty("dateOfBirth")) {
console.log("Name: "+data.items[i].lastName+","+data.items[i].firstName +" ID: " +data.items[i].id +" DOB: " +data.items[i].dateOfBirth);
}
}
}
});
}
GetESPNIds();

Related

Getting vehicle ID

I am trying to get vehicle id as follow:
mobility = TraCIMobilityAccess().get(getParentModule());
assert(mobility);
traci = mobility->getCommandInterface();
traciVehicle = mobility->getVehicleCommandInterface();
cout<< mobility->getExternalId();
But it returns an invalid vehicle id. What is wrong?
Please help me to solve this problem. Thanks.
What do you mean by an invalid vehicle id? The way you are getting the identifier is the one used by sumo. If that is the case, can you specify what do you expect as an identifier? (that of omnet which starts from [1]?)
As the id of SUMO and that one apparent in omnet are not the same (the order of creation), you may add the following to get your own id (that matches the one of omnet) :
in the ".h" file of your TraCIDemo11p, add your id:
protected:
int your_id;//added
in the ".c" file of your TraCIDemo11p, affect the index in your id:
if (stage == 0) {
...
your_id = getParentModule()->getIndex();//added
...
next, in the place you want to verify a statement, add this:
EV << "My SUMO id = " << mobility->getExternalId() << endl;
EV << "My VEINS id = " << your_id /*or just : getParentModule()->getIndex()*/<< endl;
I hope this helps.
You can try the FindModlue::findHost() in DemoBaseApplLayer::initialize(int stage) in DemoBaseApplLayer.cc:
EV << FindModule<BaseMobility*>::findHost(getParentModule())->getId() << endl;
It will first return the host module and then use the getId() function to get its id.
For better understanding:
Firstly, you can run the simulation to see the indexing of the whole simulation and it would be like this:
Simulation information in veins
As read from the figure, each objects are assigned to a number, e.g. node[0]has the id 7, besides that, each sub-modules are also assigned with id numbers, e.g.
node[0] id = 7
appl id = 8
nic id = 9
veinsmobility id = 10
All of this ids (7,8,9,10) point to the node[0], which means you can use thoese ids to identify a specific car.
In the default DemoBaseApplLayer.cc, you can find
mac = FindModule<DemoBaseApplLayerToMac1609_4Interface*>::findSubModule(getParentModule());
and
myId = mac->getMACAddress();
in the initialization function void DemoBaseApplLayer::initialize(int stage).
Therefore, you can already use the myId as the vehicle id.
By the way, the reason that you get the 18 and 20 for vehicle id, is that the returned module might just be the host module and the sub-module, e.g. 18 is for the node[*] and the 20 is for its nic sub-module.

`*': negative argument (ArgumentError)

I'm trying to sort in descending order an array of photo objects from Flickr API based on the number of comments(count_comments) of each photo. I'm using the following code.
def rank_photos(photos)
photos.sort_by { |photo| photo.count_comments * -1 }
end
However I get the following error message.
*': negative argument (ArgumentError)
Here is what the Array looks like
[{"id"=>"38280904752", "owner"=>"131718287#N07",
"secret"=>"abe0b93180", "server"=>"4583", "farm"=>5,
"title"=>"IMG_3640", "ispublic"=>1, "isfriend"=>0, "isfamily"=>0,
"count_comments"=>"0", "tags"=>"washington post dc web women codeher17
dctech tech technology",
"url_m"=>"https://farm5.staticflickr.com/4583/38280904752_abe0b93180.jpg", "height_m"=>"333", "width_m"=>"500"}, {"id"=>"38312540901",
"owner"=>"131718287#N07", "secret"=>"7b6e6805d4", "server"=>"4568",
"farm"=>5, "title"=>"IMG_3458", "ispublic"=>1, "isfriend"=>0,
"isfamily"=>0, "count_comments"=>"0", "tags"=>"washington post dc web
women codeher17 dctech tech technology",
"url_m"=>"https://farm5.staticflickr.com/4568/38312540901_7b6e6805d4.jpg", "height_m"=>"500", "width_m"=>"333"}, {"id"=>"38281453252",
"owner"=>"131718287#N07", "secret"=>"438293cffd", "server"=>"4539",
"farm"=>5, "title"=>"IMG_3460", "ispublic"=>1, "isfriend"=>0,
"isfamily"=>0, "count_comments"=>"0", "tags"=>"washington post dc web
women codeher17 dctech tech technology",
"url_m"=>"https://farm5.staticflickr.com/4539/38281453252_438293cffd.jpg", "height_m"=>"333", "width_m"=>"500"}
Why is throwing this error?
count_comments is a string, so you should convert it to a number first. In the process you can also eliminate the multiplication altogether.
def rank_photos(photos)
photos.sort_by { |photo| -photo.count_comments.to_i }
end

Issue in Facebook Replies download from post comments

I am trying to download public comments and replies from the FACEBOOK public post by page.
my code is working until 5 Feb'18, Now it is showing below error for the "Replies".
Error in data.frame(from_id = json$from$id, from_name = json$from$name, :
arguments imply differing number of rows: 0, 1
Called from: data.frame(from_id = json$from$id, from_name = json$from$name,
message = ifelse(!is.null(json$message), json$message, NA),
created_time = json$created_time, likes_count = json$like_count,
comments_count = json$comment_count, id = json$id, stringsAsFactors = F)
please refer below code I am using.
data_fun=function(II,JJ,page,my_oauth){
test <- list()
test.reply<- list()
for (i in II:length(page$id)){
test[[i]] <- getPost(post=page$id[i], token = my_oauth,n= 100000, comments = TRUE, likes = FALSE)
if (nrow(test[[i]][["comments"]]) > 0) {
write.csv(test[[i]], file = paste0(page$from_name[2],"_comments_", i, ".csv"), row.names = F)
for (j in JJ:length(test[[i]]$comments$id)){
test.reply[[j]] <-getCommentReplies(comment_id=test[[i]]$comments$id[j],token=my_oauth,n = 100000, replies = TRUE,likes = FALSE)
if (nrow(test.reply[[j]][["replies"]]) > 0) {
write.csv(test.reply[[j]], file = paste0(page$from_name[2],"_replies_",i,"_and_", j, ".csv"), row.names = F)
}}}
}
Sys.sleep(10)}
Thanks For Your support In advance.
I had the very same problem as Facebook changed the api rules at the end of January. If you update your package with 'devtools' from Pablo Barbera's github, it should work for you.
I have amended my code (a little) and it works fine now for replies to comments.There is one frustrating thing though, is that Facebook dont appear to allow one to extract the user name. I have a pool of data already so I am now using that to train and predict gender.
If you have any questions and want to make contact - drop me an email at 'robert.chestnutt2#mail.dcu.ie'
By the way - it may not be an issue for you, but I have had challenges in the past writing the Rfacebook output to a csv. Saving output as an .RData file maintains the form a lot better

crawl realtime google finance price

I want to create a small excel sheet which sort of like Bloomberg's launchpad for me to monitor live stock market price. So far, out of all the available free data source, I only found Google finance provides real time price for a list of exchanges I need. The issue with Google finance is they have already closed down their finance API. I am looking for a way to help me to programmatically retrieve the real price that I circled in chart below to have it update live in my excel.
I have been searching around and to no avail as of now. I read some post here:
How does Google Finance update stock prices? but the method suggested in the answer points to retrieving a time series of data in the chart, instead of the live updating price part I need. I have been examining the network communication of the web page in chrome's inspection and didn't find any request that returns the part of real time price I need. Any help is greatly appreciated. some sample codes (can be in other languages other than VBA) would be very beneficial. Thanks everyone !
There are so many way ways to do this: VBA, VB, C# R, Python, etc. Below is a way to download statistics from Yahoo finance.
Sub DownloadData()
Set ie = CreateObject("InternetExplorer.application")
With ie
.Visible = True
.navigate "https://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL"
' Wait for the page to fully load; you can't do anything if the page is not fully loaded
Do While .Busy Or _
.readyState <> 4
DoEvents
Loop
' Set a reference to the data elements that will be downloaded. We can download either 'td' data elements or 'tr' data elements. This site happens to use 'tr' data elements.
Set Links = ie.document.getElementsByTagName("tr")
RowCount = 1
' Scrape out the innertext of each 'tr' element.
With Sheets("DataSheet")
For Each lnk In Links
.Range("A" & RowCount) = lnk.innerText
RowCount = RowCount + 1
Next
End With
End With
MsgBox ("Done!!")
End Sub
I will leave it up to you to find other technologies that do the same. Thing, for instance, R and Prthon can do exactly the same thing, although, the scripts will be a bit different than the VBA scripts that do this kind of work.
It's fairly easy to make it work in Python. You'll need a few libraries:
Library
Purpose
requests
to make a request to Google Finance and then return HTML.
bs4
to process returned HTML.
pandas
to easily save to CSV/Excel.
Code and full example in the online IDE:
from bs4 import BeautifulSoup
import requests, lxml, json
from itertools import zip_longest
def scrape_google_finance(ticker: str):
# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
"hl": "en", # language
}
# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
# https://www.whatismybrowser.com/detect/what-is-my-user-agent
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36",
}
html = requests.get(f"https://www.google.com/finance/quote/{ticker}", params=params, headers=headers, timeout=30)
soup = BeautifulSoup(html.text, "lxml")
ticker_data = {"right_panel_data": {},
"ticker_info": {}}
ticker_data["ticker_info"]["title"] = soup.select_one(".zzDege").text
ticker_data["ticker_info"]["current_price"] = soup.select_one(".AHmHk .fxKbKc").text
right_panel_keys = soup.select(".gyFHrc .mfs7Fc")
right_panel_values = soup.select(".gyFHrc .P6K39c")
for key, value in zip_longest(right_panel_keys, right_panel_values):
key_value = key.text.lower().replace(" ", "_")
ticker_data["right_panel_data"][key_value] = value.text
return ticker_data
# tickers to iterate over
tickers = ["DIS:NYSE", "TSLA:NASDAQ", "AAPL:NASDAQ", "AMZN:NASDAQ", "NFLX:NASDAQ"]
# temporary store the data before saving to the file
tickers_prices = []
for ticker in tickers:
# extract ticker data
ticker_data = scrape_google_finance(ticker=ticker)
# append to temporary list
tickers_prices.append({
"ticker": ticker_data["ticker_info"]["title"],
"price": ticker_data["ticker_info"]["current_price"]
})
# create dataframe and save to csv/excel
df = pd.DataFrame(data=tickers_prices)
# to save to excel use to_excel()
df.to_csv("google_finance_live_stock.csv", index=False)
Outputs:
ticker,price
Walt Disney Co,$137.06
Tesla Inc,"$1,131.21"
Apple Inc,$176.99
"Amazon.com, Inc.","$3,321.61"
Netflix Inc,$384.93
Returned data from ticker_data
{
"right_panel_data": {
"previous_close": "$138.61",
"day_range": "$136.66 - $139.20",
"year_range": "$128.38 - $191.67",
"market_cap": "248.81B USD",
"volume": "9.98M",
"p/e_ratio": "81.10",
"dividend_yield": "-",
"primary_exchange": "NYSE",
"ceo": "Bob Chapek",
"founded": "Oct 16, 1923",
"headquarters": "Burbank, CaliforniaUnited States",
"website": "thewaltdisneycompany.com",
"employees": "166,250"
},
"ticker_info": {
"title": "Walt Disney Co",
"current_price": "$136.66"
}
}
If you want to scrape more data with a line-by-line explanation, there's a Scrape Google Finance Ticker Quote Data in Python blog post of mine that also covers scraping time-series chart data.

Twitter API Hashtag Search Results don't contain images

I am not seeing image entities in my twitter API search results when I search for a hashtag, but if I use the api endpoint for that specific tweet I do.
Using this tweet for an example: https://twitter.com/mrbuddylee/status/733407581788463104
results = #twitter_client.search('#BecauseSummer', { include_entities: true, count: 200 })
result = results.first # not actually the first result, but just to illustrate.
result.to_h[:entities)
=> {:hashtags=>[{:text=>"BecauseSummer", :indices=>[7, 21]}],
:symbols=>[], :user_mentions=>[],
:urls=>[{:url=>"TWITTER_SHORTENED_URL", :expanded_url=>"http://twitter.com/mrbuddylee/status/733407581788463104/photo/1", :display_url=>"pic.twitter.com/FAAY00SYQH", :indices=>[22, 45]}]}
but If I search for the tweet directly:
#twitterclient.status(733407581788463104).to_h[:entities]
=> {:hashtags=>[{:text=>"BecauseSummer", :indices=>[7, 21]}],
:symbols=>[], :user_mentions=>[], :urls=>[],
:media=>[{:id=>733407573345341441, :id_str=>"733407573345341441", :indices=>[22, 45], :media_url=>"http://pbs.twimg.com/tweet_video_thumb/Ci2WTVzUkAE_gGD.jpg", :media_url_https=>"https://pbs.twimg.com/tweet_video_thumb/Ci2WTVzUkAE_gGD.jpg", :url=>"TWITTER_SHORTENED_URL", :display_url=>"pic.twitter.com/FAAY00SYQH", :expanded_url=>"http://twitter.com/mrbuddylee/status/733407581788463104/photo/1", :type=>"photo", :sizes=>{:small=>{:w=>340, :h=>173, :resize=>"fit"}, :thumb=>{:w=>150, :h=>150, :resize=>"crop"}, :medium=>{:w=>392, :h=>200, :resize=>"fit"}, :large=>{:w=>392, :h=>200, :resize=>"fit"}}}]}
Notice the media hash on the second result.
Why is this? Is it possible to get the media url on the initial search request?
I ended up doing a double query to make sure the images are returned:
results = #twitter_client.search('#BecauseSummer', { include_entities: true, count: 100 })
ids = .first(100).map(&:id)
results = #client.statuses(ids)

Resources