I am currently consuming data from the G Suite API.
An inconvenience I have found is that for some of the APIs the number of resources available might be quite large.
For instance, when I consume the Users:list API (https://www.googleapis.com/admin/directory/v1/users), given the number of resources and the maximum number of results per query I need to perform a significant number of queries. Find below an example JSON response:
{
"kind": "admin#directory#users",
"etag": "\"WqpSTs-zelqnIvn63V............................/v3ENarMfXkTh9ijs3OVkQRoUSVU\"",
"users": [
{
"kind": "admin#directory#user",
"id": "7720745322191632224007",
"etag": "\"WqpSTs-zelqnIvn63V........................PfcSmik3zEJwHAl1UbgSk\"",
"primaryEmail": ...,
...
},
{
"kind": "admin#directory#user",
"id": "227945583287518253104",
"etag": "\"WqpSTs-zelqnIvn63V..........-zY30eInIGOmLI\"",
"primaryEmail": ...,
...
},
...
N-users
...
]
}
I am running this query several times a day.
Ideally I would only retrieve the resources that have changed and the new ones, excluding from the response the ones that have not changed.
Is it possible to do that? If so, how?
Thank you in advance for your answers.
You could create custom attributes for your users, and then filter your requests using the query parameter according to your custom attribute.
Or define exactly what you mean by "changed" or "not changed" as the user properties will change on every login to update the last login attribute.
Update:
You can watch for changes on the list of users in your domain by supplying an address to receive notifications in a POST request to the watch endpoint:
https://www.googleapis.com/admin/directory/v1/users/watch
References:
Users.watch
Custom User Fields
Query string for User fields
Related
How i can get list of popular youtube channels per each country?
this service find solution
https://www.channelcrawler.com/eng/results2/281574
I tried youtube api but i didn't find, anyone know solution?
As mentioned, is not possible to get this information using the YouTube API directly, even in the FAQ of the page you shared says:
The Channel Crawler was made to discover new YouTube channels, based on your search criteria.
and:
The Channel Crawler uses advanced data collection methods in order to collect channel information from YouTube and store it in the database. Basically, it just checks the liked videos and comment sections of channels that are already in the database, in order to add more channels to it. You can also manually add a channel.
Following the highlighted information, I have this idea and you can try it too:
Use the search endpoint for search channels in a specific country and in a specific videoCategory, then, with the channelId returned in the results of the search, use the channel endpoint for get their country1.
1 take into account that some channels doesn't have the country value; in this case, you have to set another criteria(s) for determine whether the channel matches with your requirements.
Example:
Use the search endpoint for search channels in country/region Pakistan and in the videoCategory Sports - test it here:
URL:
https://youtube.googleapis.com/youtube/v3/search?part=id%2Csnippet&order=videoCount&q=Sports®ionCode=PK&key=[YOUR_API_KEY]
The results of this request are as follows:
{
"kind": "youtube#searchListResponse",
"etag": "0C7hSI3oNXJt66PsERuviVQcLCo",
"nextPageToken": "CAUQAA", // Remember the nextPageToken for get the next results.
"regionCode": "PK", // Region queried.
"pageInfo": {
"totalResults": 1000000, // Look the amount of data you have to check, filter and debug.
"resultsPerPage": 5 // Set "maxResults" parameter to "50" for get more results per page.
},
"items": [
{
"kind": "youtube#searchResult",
"etag": "iSwEnBs_yV6lOIBubmRXVwjjujQ",
"id": {
"kind": "youtube#channel", // Make sure that this item is actually an channel.
"channelId": "UCo2TvjBHS1BtyIkeGGTMe6w"
},
"snippet": {
"publishedAt": "2018-07-28T18:34:04Z",
"channelId": "UCo2TvjBHS1BtyIkeGGTMe6w", // Use this value for the "channel" endpoint.
"title": "ONTime Sports",
"description": "قناة اون تايم سبورت واحدة من مجم...",
"thumbnails": { [thumbnails here...] },
"channelTitle": "ONTime Sports",
"liveBroadcastContent": "upcoming",
"publishTime": "2018-07-28T18:34:04Z"
}
},
[other results here...]
]
}
Use channel endpoint for get the channel detailed information - in this case, their country - test it here:
URL - using the channelId UCo2TvjBHS1BtyIkeGGTMe6w - obtained from the previous search results:
https://youtube.googleapis.com/youtube/v3/channels?part=id%2Csnippet&id=UCo2TvjBHS1BtyIkeGGTMe6w&key=[YOUR_API_KEY]
The results of this request are as follows:
{
"kind": "youtube#channelListResponse",
"etag": "8BfUXxlAEBLe7lBmih1JXUwZ394",
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 5
},
"items": [
{
"kind": "youtube#channel",
"etag": "AFk5NCl9393ui58WyRf7WljoatE",
"id": "UCo2TvjBHS1BtyIkeGGTMe6w",
"snippet": {
"title": "ONTime Sports",
"description": "large description here...",
"customUrl": "ontimesportseg",
"publishedAt": "2018-07-28T18:34:04Z",
"thumbnails": {[thumbnails here]},
"localized": {
"title": "ONTime Sports",
"description": "large description here..."
},
"country": "EG" // This is the regionCode of the country this channels has provided.
}
}
]
}
Here, you can see that the value "country" for this channel is "EG" = Egypt2.
Then, repeat these steps with all countries and videoCategory for each country.
Considerations:
As I tested, the type parameter in search is not working as one might expect, in this case (for get channels only), use the order=viewCount combination of parameter=value. Even with this combination, make sure to check that the value of the kind attribute is: youtube#channel. Also, I search the videoCategory using its name, no its id - as it should be -, but, the API is not perfect, that's why I used the name of the videoCategory.
Even specifiyng the region parameter with a valid country, you might get results that are either from another countries or doesn't have the country attribute and value at all = and that's due the channel's popularity in the given country and public channel information. A simple example could be: ESPN, they probably have a YouTube channel for each country, but, their main channel is the most popular in all countries, so, in this case, you'll get the ESPN channel in english and that might differ from your expected results. You have to work with what YouTube provides.
As you notice, there are too many channels/results to debug, so, make sure to provide more filters and set your queries/search criteria and (once you get the desired results), store the valid results in a database or similar.
If you know specific channels that you know are popular in a given country, but, when you query that channel using the YouTube Data API, it doesn't bring the country value, you have to save that channel manually in your database and/or collect more information for automatically set if a channel is from a given country - this point is very related to my point # 2.
During developing pipeline which will use Elasticsearch as a source I faced with issue related paging. I am using SQL Elasticsearch API. Basically, I've started to do request in postman and it works well. The body of request looks following:
{
"query":"SELECT Id,name,ownership,modifiedDate FROM \"core\" ORDER BY Id",
"fetch_size": 20,
"cursor" : ""
}
After first run in response body it contains cursor string which is pointer to next page. If in postman I send the request and provide cursor value from previous request it return data for second page and so on. I am trying to archive the same result in Azure Data Factory. For this I using copy activity, which store response to Azure blob. Setup for source is following.
copy activity source configuration
This is expression for body
{
"query": "SELECT Id,name,ownership,modifiedDate FROM \"#{variables('TableName')}\" WHERE ORDER BY Id","fetch_size": #{variables('Rows')}, "cursor": ""
}
I have no idea how to correctly setup pagination rule. The pipeline works properly but only for the first request. I've tried to setup Headers.cursor and expression $.cursor but this setup leads to an infinite loop and pipeline fails with the Elasticsearch restriction.
I've also tried to read document at https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support but it seems pretty limited in terms of usage examples and difficult for understanding.
Could somebody help me understand how to build the pipeline with paging abilities utilization?
Responce with the cursor looks like:
{
"columns": [
{
"name": "companyId",
"type": "integer"
},
{
"name": "name",
"type": "text"
},
{
"name": "ownership",
"type": "keyword"
},
{
"name": "modifiedDate",
"type": "datetime"
}
],
"rows": [
[
2,
"mic Inc.",
"manufacture",
"2021-03-31T12:57:51.000Z"
]
],
"cursor": "g/WuAwFaAXNoRG5GMVpYSjVWR2hsYmtabGRHTm9BZ0FBQUFBRUp6VGxGbUpIZWxWaVMzcGhVWEJITUhkbmJsRlhlUzFtWjNjQUFBQUFCQ2MwNWhaaVIzcFZZa3Q2WVZGd1J6QjNaMjVSVjNrdFptZDP/////DwQBZgljb21wYW55SWQBCWNvbXBhbnlJZAEHaW50ZWdlcgAAAAFmBG5hbWUBBG5hbWUBBHRleHQAAAABZglvd25lcnNoaXABCW93bmVyc2hpcAEHa2V5d29yZAEAAAFmDG1vZGlmaWVkRGF0ZQEMbW9kaWZpZWREYXRlAQhkYXRldGltZQEAAAEP"
}
I finally find the solution, hopefully, it will be useful for the community.
Basically, what needs to be done it is split the solution into four steps.
Step 1 Make the first request as in the question description and stage file to blob.
Step 2 Read blob file and get the cursor value, set it to variable
Step 3 Keep requesting data with a changed body
{"cursor" : "#{variables('cursor')}" }
Pipeline looks like this:
pipeline
Configuration of pagination looks following
pagination . It is a workaround as the server ignores this header, but we need to have something which allows sending a request in loop.
We have 2 indexes :
Conversation - Contains participants details, last message, etc
{ "_id":123,
"last_message":"Hi",
"from_phone": "+919899988888"
"to_phone":"+919899988889"
......
}
ConversationDetails - list of messages sent/received for a given participants
[{
"conv_id":123,
"message":"Hi",
"channel": "SMS"
"comm_dir":"SENT"
"created": 1592992160480
......
},
{
"conv_id":123,
"message":"Hi",
"channel": "SMS"
"comm_dir":"RECEIVED"
"created": 1592992160480
......
},
]
We need to have a field 'lastReceivedMessageSource' in every Conversation Document which is derived from Conversation Details Documents for the given conversations.
We need to migrate this data for millions of conversations. What is the fastest way to do that ?
My approach: is to fetch the values for 'n' conversations and bulk_upsert in the conversation Document.
Note: There are millions of Conversations
E.S Version: 5.6
I want to be able to return a set of counts of individual documents from a single index based on a previous set of results, and am wondering if there is a way to do it without running a separate query for each.
So, given a data set like this (simplified version of my ES documents):
{
"name": "visit",
"sessionId": "session1"
},
{
"name": "visit",
"sessionId": "session2"
},
{
"name": "visit",
"sessionId": "session3"
},
{
"name": "click",
"sessionId": "session1"
},
{
"name": "click",
"sessionId": "session3"
}
What I would like to do is be able to search for name: visit and give a count of all those. That part is easy. But I would also like to be able to now count my name: click docs that have the sessionId of the name: visit result set and return a count of how many of those name: click there were as well as the name: visit.
Is there an easy way to do this? I have looked at aggregation APIs but they all seem to not quite fit my needs. There also seems to be a parent/child relationship but it doesn't apply to my situation since both documents I want to individually get counts of are of the same type.
Expected result would be something like this:
{
"count": {
// total number of visit events since this is my start point
"visit": 3,
// the amount of click results that have sessionId
// matching my previous search's sessionId values
"click": 2
}
}
At first glance, you need to do this in two queries:
the first aggregation query to retrieve the sessionIds and
a second aggregation query filtered with those sessionIds to find the count of clicks.
I don't think it's a big deal to run those two queries, but that depends on how much data you have and how many sessionIds you want to retrieve at once.
I am trying to do application specific places search with google place api. Here is how I am adding a place:
Request:
{
"location": {
"lat": 37.760538,
"lng": -121.900879
},
"accuracy": 50,
"name": "p2p",
"types": ["other"]
}
I get success response as shown below:
Response:
{
"id" : "dfe583b1ac058750cf524f958afc5e82ade455d7",
"place_id" : "qgYvCi0wMDAwMDBhNWE4OWU4NTMzOjgwOGZlZTBhNjI3OjBjNTU1OTU4M2Q2NDI5YmM",
"reference" : "CkQxAAAAsPE72V-jhHUjj6vPy2HdC__2MhAdXanL6mlFBA4bcayRabKyMlfKFiah7U2vkoCj1P_0w9ESFSv5mfDkyufaZhIQTHBHY_jPGRHEE3EmEAGElhoUXTSylMslwHSTK5tYdstW2rOZKbw",
"scope" : "APP",
"status" : "OK"
}
When I search for this place using radar search, I get ZERO_RESULTS.
Request:
https://maps.googleapis.com/maps/api/place/radarsearch/json?key=key&radius=5000&location=37.761926,-121.891856&keyword=p2p
Response:
{
"html_attributions": [ ],
"results": [ ],
"status": "ZERO_RESULTS"
}
Is there something that I am doing the right way? Please help.
Thanks & Regards,
--Rajani
Your scope is "APP". That means you can access it (via PlaceID) from the application that created the entry only. If the location passes Google's moderation process, then it will gain scope "GOOGLE" and be accessible from the general searches.
scope — Indicates the scope of the place_id. The possible values are:
APP: The place ID is recognised by your application only. This is because your
application added the place, and the place has not yet
passed the moderation process.
GOOGLE: The place ID is available to other applications and on Google Maps.
Note: The scope field is included only in Nearby Search results and
Place Details results. You can only retrieve app-scoped places via the
Nearby Search and the Place Details requests. If the scope field is
not present in a response, it is safe to assume the scope is GOOGLE.
See: https://developers.google.com/places/documentation/search