I'm still learning elastic and a lot of things are unclear to me, including this example:
Suppose I have marketplace like amazon/any (many products with many sub options and availability by cities). And I want use elastic for searching only by string field.
For example I want to search "lord of the rings compilation in dublin" and elastic should return only books compilation on lord of the rings which availability in dublin.
Into elastic I can put documents with any schema (using only for searching).
So now I have this schema for elastic (data compilation from prod database):
[
{
"name": "lord of rings",
"seller": "Home Production",
"availability": [
{
"city": "dublin",
"category": "book",
"types": [
"one book",
"compilation"
]
},
{
"city": "london",
"category": "book",
"types": [
"one book",
"compilation"
]
}
]
},
{
"name": "lord of rings",
"seller": "Some",
"availability": [
{
"city": "dublin",
"category": "book",
"types": [
"one book",
"compilation"
]
},
{
"city": "london",
"category": "book",
"types": [
"one book",
"compilation"
]
},
{
"city": "dublin",
"category": "dvd",
"types": [
"disk"
]
}
]
}
]
This is a very abstract example. We can format the data schema in any way for ease of searching. The search city is always known (it is not part of the text query).
The difficulty is that one seller, for one product, has many cities of availability and in each city we know the "options" of availability (for example, one book or a whole collection)
I don't know how to describe it in more detail or how to find it in Google correctly.
I tried multi_match but it gives wrong answers if i want 'lord of rings dvd in dublin'.
He suggests that the first document is more relevant to me, although in fact the second document is the correct answer.
Relevance issues are not easy to solve, sometimes you could get better results if you boost the city field using multi-match.
Anyway, you need to study more to understand your scenario and make the documents you want more relevant.
I recommend that you read the book Relevant Search that will help you a lot to understand why some results are not relevant as you want.
For example,
on a video called "B" created by a channel called "A",
is it possible to get a specific comment that "A" has liked?
Currently, I think I can only get the total number of likes for a particular comment on a video called "B" created by a channel called "A".
One more time YouTube Data API v3 doesn't provide a basic feature.
I recommend you to try out my open-source YouTube operational API. Indeed by fetching https://yt.lemnoslife.com/commentThreads?part=snippet,replies&videoId=VIDEO_ID&order=ORDER (ORDER can be either time or relevance, but note that this latter returns less comments), you will get the video comments with the isHearted you are looking for in item["snippet"]["topLevelComment"]["snippet"]["isHearted"].
The YouTube video mWdFMNQBcjs is a good video to test this feature of my API because it doesn't have too many comments while having both pagination mechanism for top level comments and answers, in addition to have its first comment hearted and pinned. With this video id you would get:
{
"kind": "youtube#commentThreadListResponse",
"etag": "NotImplemented",
"pageInfo": {
"totalResults": 67,
"resultsPerPage": 20
},
"nextPageToken": "Eg0SC21XZEZNTlFCY2pzGAYy3gIKtAJnZXRfcmFua2VkX3N0cmVhbXMtLUNxVUJDSUFFRlJlMzBUZ2FtZ0VLbFFFSTJGOFFnQVFZQnlLS0FkdEtUa2V3RXhLY2Q3U3NodkZrZnBjbHlYVEZCTUxWV1RmVUdiUWtGbjFoa3FpNUlucVE2ZmMyRVptc1BGbG15UmhCNXo1LWFhTmNWelc0UGFYdGRMYzd0SUVmRlZmYzlmbUN4SktkVXY0MS1FcWZadXNrQnhrc2g4ZjdnUHdpZHpXM2M4SFA3RGJjVjdCZWtLTWwtMHJXa0phVU0zRW9NU05wS0xjZ1o0aHFpRWRabWNENDE0bVFQQkFVRWdVSWhpQVlBQklGQ0lnZ0dBQVNCUWlYSUJnQUVnVUlpU0FZQUJJSENJY2dFQUVZQUJJSENJVWdFQlFZQVJnQiIRIgttV2RGTU5RQmNqczAAeAEoFEIQY29tbWVudHMtc2VjdGlvbg==",
"items": [
{
"kind": "youtube#commentThread",
"etag": "NotImplemented",
"id": "UgzT9BA9uQhXw05Q2Ip4AaABAg",
"snippet": {
"topLevelComment": {
"kind": "youtube#comment",
"etag": "NotImplemented",
"id": "UgzT9BA9uQhXw05Q2Ip4AaABAg",
"snippet": {
"textOriginal": "Twenty-three",
"isHearted": true,
"authorDisplayName": null,
"authorHandle": "#user-wl1ce4xg4j",
"authorProfileImageUrls": [
{
"url": "https:\/\/yt3.ggpht.com\/ytc\/AMLnZu_3pI88H9gEgJtksqziDFfJj3PSO1E-_7Z5Tn_GJtTMyp7wglMOOAuYSEs_u0LR=s48-c-k-c0x00ffffff-no-rj",
"width": 48,
"height": 48
},
...
],
"authorChannelId": {
"value": "UCv_LqFI-0vMVYgNR3TeB3zQ"
},
"likeCount": "",
"publishedAt": "3 weeks ago",
"wasEdited": false,
"isPinned": true,
"authorIsChannelOwner": true,
"videoCreatorHasReplied": true,
"totalReplyCount": 23,
"nextPageToken": "Eg0SC21XZEZNTlFCY2pzGAYygwEaUBIaVWd6VDlCQTl1UWhYdzA1UTJJcDRBYUFCQWciAggAKhhVQ3ZfTHFGSS0wdk1WWWdOUjNUZUIzelEyC21XZEZNTlFCY2pzQAFICoIBAggBQi9jb21tZW50LXJlcGxpZXMtaXRlbS1VZ3pUOUJBOXVRaFh3MDVRMklwNEFhQUJBZw=="
}
}
}
},
...
]
}
As YouTube Data API v3 CommentThreads: list and Comments: list there are two pagination mechanisms:
the top level pagination (using here
Eg0SC21XZEZNTlFCY2pzGAYyiwEKT0FEU0pfaTJYUjJDTWhJQ2ZGWWdoV2NGUkpqWXRDX1FMQVZYVE9JOU1hUjZmQ0tEUDBkcWpacmQzOG5Yc0kwYUFGUnJFTkdkMmpINlZaMlEiESILbVdkRk1OUUJjanMwAXgBKBQwAUIhZW5nYWdlbWVudC1wYW5lbC1jb21tZW50cy1zZWN0aW9u
for the nextPageToken) browse top level comments
the answers pagination (using here
Eg0SC21XZEZNTlFCY2pzGAYygwEaUBIaVWd6VDlCQTl1UWhYdzA1UTJJcDRBYUFCQWciAggAKhhVQ3ZfTHFGSS0wdk1WWWdOUjNUZUIzelEyC21XZEZNTlFCY2pzQAFICoIBAggCQi9jb21tZW50LXJlcGxpZXMtaXRlbS1VZ3pUOUJBOXVRaFh3MDVRMklwNEFhQUJBZw==
for the nextPageToken) browse answers to a given comment
result = requests.get('http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=tesla&callback=YAHOO.Finance.SymbolSuggest.ssCallback').json()
result
When I run the python code above, I get <Response [404]>, does anyone know why that might be? I am worried this API no longer works even though I saw posts from just a year ago talking about it working.
If it's not documented, you can't rely on it working. The only (relatively) sure bet is to use some official API instead (which usually comes at a cost).
That said, if you want to continue using undocumented stuff (with the same risk of it getting shut down or you getting blocked any day), give this a try:
https://query2.finance.yahoo.com/v1/finance/search?q=tesla
(I looked at what https://finance.yahoo.com/ uses.)
This delivers results like these:
{
"explains": [],
"count": 15,
"quotes": [
{
"exchange": "NMS",
"shortname": "Tesla, Inc.",
"quoteType": "EQUITY",
"symbol": "TSLA",
"index": "quotes",
"score": 2048451,
"typeDisp": "Equity",
"longname": "Tesla, Inc.",
"exchDisp": "NASDAQ",
"isYahooFinance": true
},
{
"exchange": "NEO",
"shortname": "TESLA, INC. CDR (CAD HEDGED)",
"quoteType": "EQUITY",
"symbol": "TSLA.NE",
"index": "quotes",
"score": 24083,
"typeDisp": "Equity",
"longname": "Tesla, Inc.",
"exchDisp": "NEO",
"isYahooFinance": true
},
// ...
],
"news": [
// Also delivers news here...
],
// Some meta stuff here
}
Use at your own risk.
I'm currently implementing Rich Card Snippets for the Services we provide on our site, so that there's a chance that the products may show higher on results thanks to these rich cards but after a look around there's no option that suits the packages which require a bespoke solution for each instance.
Due to the complexity of registering RTM’s we do not have set costs or
set packages. Please contact us directly to discuss your
requirements, where we will be able to give you an estimate and time
scale.
<script type="application/ld+json">
{
"#context": "http://schema.org/",
"#type": "Product",
"name": "Limited Liability Partnership",
"alternateName":"LLP",
"image": "https://pbs.twimg.com/profile_images/378800000561746118/2bf53fff069b75bfa53e33bd79cd3230.png",
"description": "RTM companies were introduced under the Commonhold and Lease Reform Act 2002. They are private companies Limited by Guarantee, enabling leaseholders in blocks of apartments to take over the management of their building. There are constitutional rules which apply to this type of company. Due to the complexity of registering RTM’s we do not have set costs or set packages. ",
"brand": {
"#type": "Service",
"name": "Company Formation"
},
"offers": {
"#type": "Offer",
"priceCurrency": "GB",
"price": "0",
"itemOffered": "Service",
"availability": "http://schema.org/InStock",
"seller": {
"#type": "LocalBusiness",
"name": "Murphy Thompson Moore LLP"
}
}
}
</script>
What's my best option here;
Don't include the price field? (and have Google Search Console complain of errors. And also potentially confuse users.)
Put Price as Zero? (which causes Structured Data to put price as free, which may appear misleading.)
Don't include the products which require bespoke solutions at all?
Some other option which I'm not aware of?
If you use Google Finance, you see the stock price update in real time. However, I do not see ajax requests for each price change. How exactly is it receiving it's data? The ajax calls do happen every so often but not for every price change.
https://www.google.com/finance?q=aapl
It is true that the graph is supplied by flash, but the price of the stock is in HTML and updated in real time.
By looking at the connections, you'll notice that one of the connection stays open. My guess is that it provides the data:
https://www.google.com/finance/qs/channel?VER=8&RID=rpc&SID=628A5B4C83473350&CI=0&AID=0&TYPE=xmlhttp&zx=dcad1i8o61q8&t=1
The graph is supplied by Adobe Flash Player so I don't think you'd see any traffic via inspector for that. The rest though does generate calls:
https://www.google.com/finance/getprices?q=AAPL&x=NASD&i=120&p=25m&f=d,c,v,o,h,l&df=cpct&auto=1&ts=1368199239667&ei=jw-NUcivEsmrqQGT7wE
would be an example of what I'm seeing.
Here's is the response from the server:
EXCHANGE%3DNASDAQ
MARKET_OPEN_MINUTE=570
MARKET_CLOSE_MINUTE=960
INTERVAL=120
COLUMNS=DATE,CLOSE,HIGH,LOW,OPEN,VOLUME
DATA=
TIMEZONE_OFFSET=-240
a1368197880,452.2847,452.29,451.6,452.0101,35458
1,451.81,452.3,451.661,452.24,91267
2,452.2,452.3,451.8314,451.84,35767
3,451.91,452.3,451.8,452.3,89882
4,452.4575,452.61,451.88,451.88,45130
5,452.77,452.88,452.45,452.48,35866
6,453.626,453.9,453.27,453.3571,0
7,453.05,453.6599,453,453.61,0
8,452.9,453.17,452.83,453.09,0
9,452.58,453.15,452.4377,452.865,0
10,452.7568,453.068,452.45,452.475,0
11,452.53,452.8,452.302,452.75,0
The other call is makes is to:
https://www.google.com/finance/qs/channel?VER=8&SID=6C5BE78E1E7D9597&RID=82123&AID=500&zx=p0b0io9ifynn&t=1
which responds with something like:
10
[1,500,0]
This is an old answer but pops up on the first Google results. Since quite a few years passed since the question was asked, Google has changed quite a few things on its website.
Currently, the ticker price can be extracted with the CSS selector, which in this case is called descendant combinator selector:
In the screenshot above I'm using SelectorGadget Chrome extension which let's pick CSS selectors by clicking on the desired element in the browser, and returns a selector afterward.
Code and example in the online IDE:
from parsel import Selector
import requests
# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
"hl": "en" # displayed language
}
# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
# https://www.whatismybrowser.com/detect/what-is-my-user-agent
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
}
html = requests.get(f"https://www.google.com/finance/quote/{ticker}", params=params, headers=headers, timeout=30)
selector = Selector(text=html.text)
current_price = selector.css(".AHmHk .fxKbKc::text").get()
If you need to extract more data, there's a Scrape Google Finance Ticker Data in Python blog post of mine that shows you how to get output like this one:
{
"ticker_data": {
"current_price": "$138.98",
"quote": "AAPL:NASDAQ",
"title": "Apple Inc"
},
"about_panel": {
"previous_close": "$138.98",
"year_range": "$129.04 - $182.94",
"market_cap": "2.23T USD",
"avg_volume": "95.19M",
"p/e_ratio": "22.96",
"dividend_yield": "0.66%",
"primary_exchange": "NASDAQ",
"ceo": "Tim Cook",
"founded": "Apr 1, 1976",
"headquarters": "Cupertino, CaliforniaUnited States",
"website": "apple.com",
"employees": "154,000",
"description": "Apple Inc. is an American multinational technology company that specializes in consumer electronics, software and online services headquartered in Cupertino, California, United States. Apple is the largest technology company by revenue and, as of June 2022, is the world's biggest company by market capitalization, the fourth-largest personal computer vendor by unit sales and second-largest mobile phone manufacturer. It is one of the Big Five American information technology companies, alongside Alphabet, Amazon, Meta, and Microsoft.\nApple was founded as Apple Computer Company on April 1, 1976, by Steve Jobs, Steve Wozniak and Ronald Wayne to develop and sell Wozniak's Apple I personal computer. It was incorporated by Jobs and Wozniak as Apple Computer, Inc. in 1977 and the company's next computer, the Apple II, became a best seller and one of the first mass-produced microcomputers. Apple went public in 1980 to instant financial success. The company developed computers featuring innovative graphical user interfaces, including the 1984 original Macintosh, announced that year in a critically acclaimed advertisement. ",
"extensions": [
"Most active",
"Stock",
"US listed security",
"US headquartered"
]
},
"news": {
"items": [
{
"position": 1,
"title": "Meta's new VR headset will cost $1,500 as Zuckerberg sets up battle with \nApple",
"link": "https://www.marketwatch.com/story/metas-new-vr-headset-will-cost-1-500-as-zuckerberg-sets-up-battle-with-apple-11665511178",
"source": "MarketWatch",
"published": "18 hours ago",
"thumbnail": "https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRjGeMvNEiJhFMgpCM3ewmD0zEzUDJfXIqR8I5EXy3eEH0F7Jo75O0Gqe6N5S8"
},
{
"position": 2,
"title": "Apple (AAPL) in Focus for US Stocks Facing Brutal Earnings Season",
"link": "https://www.bloomberg.com/news/articles/2022-10-09/apple-aapl-in-focus-for-us-stocks-facing-brutal-earnings-season",
"source": "Bloomberg.com",
"published": "2 days ago",
"thumbnail": "https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcSIZYapO5l-u2YS_D0x3T8WaC5YHibvnDgA-SNY6Csyf16R4nFTbgbkdpC861A"
}
]
},
"finance_perfomance": {
"table": []
},
"people_also_search_for": {
"items": [
{
"position": 1,
"ticker": "AMZN",
"ticker_link": "https://www.google.com/finance/quote/AMZN:NASDAQ",
"title": "Amazon.com, Inc.",
"price": "$112.21",
"price_change": "Changed by 0.00%",
"price_change_formatted": "0.00%"
}, ...
{
"position": 18,
"ticker": "RIVN",
"ticker_link": "https://www.google.com/finance/quote/RIVN:NASDAQ",
"title": "Rivian Automotive Inc",
"price": "$32.33",
"price_change": "Changed by 0.00%",
"price_change_formatted": "0.00%"
}
]
},
"interested_in": {
"items": [
{
"position": 1,
"ticker": "Index",
"ticker_link": "https://www.google.com/finance/quote/.DJI:INDEXDJX",
"title": "Dow Jones Industrial Average",
"price": "29,239.19",
"price_change": "Changed by 0.00%",
"price_change_formatted": "0.00%"
}, ...
{
"position": 18,
"ticker": "NFLX",
"ticker_link": "https://www.google.com/finance/quote/NFLX:NASDAQ",
"title": "Netflix Inc",
"price": "$214.29",
"price_change": "Changed by 0.00%",
"price_change_formatted": "0.00%"
}
]
}
}