scrapoxy - How to limit the number of instances that get created - proxy

I am using this library to setup a proxy on digital ocean and I am looking for a way to limit the number of droplets that get created in DigitalOcean so while looking at the documentation I came across this link which suggests that to limit the droplet I need to add max which I did however that does not seem to work, right now I am seeing a new droplet being created every few seconds and i want to reduce that to 2 if possible just to test things out.
This is what i have in my configuration
{
"commander": {
"password": "password"
},
"instance": {
"port": 3128,
"scaling": {
"min": 1,
"max": 2
},
"log": {
"path": "/log"
}
},
"providers": [
{
"type": "digitalocean",
"token": "the-token-goes-here",
"region": "SGP1",
"size": "s-1vcpu-2gb",
"sshKeyName": "ubuntu",
"imageName": "name-of-image",
"tags": "proxy,instance",
"max": 2
}
]
}
My understanding is by adding the max I should have been able to limit the droplets but I see no difference, is this the right place to add? What am i missing here?

Related

MINIO Is it possible to set automatic deletion from bucket of the object after the retention date?

I want to delete object after the retention date is end.
Can I do it with Bucket Lifecycle? If so, how?
And the second question, is it possible to automatically delete an object if there is a newer one available?
This will only work with a versioned bucket
Enable object lifecycle configuration on buckets to setup automatic deletion of objects after a specified number of days or a specified date.
Example:
Create a bucket lifecycle configuration which expires the objects under the prefix old/ on 2020-01-01T00:00:00.000Z date and the objects under temp/ after 7 days.
Enable bucket lifecycle configuration using mc:
{
"Rules": [{
"Expiration": {
"Date": "2020-01-01T00:00:00.000Z"
},
"ID": "OldPictures",
"Filter": {
"Prefix": "old/"
},
"Status": "Enabled"
},
{
"Expiration": {
"Days": 7
},
"ID": "TempUploads",
"Filter": {
"Prefix": "temp/"
},
"Status": "Enabled"
}
]
}
the same can be found in :https://docs.min.io/docs/minio-bucket-lifecycle-guide.html
Remove non current versions
{
"Rules": [
{
"ID": "Removing all old versions",
"Filter": {
"Prefix": "users-uploads/"
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 365
},
"Status": "Enabled"
}
]
}

Youtube Data API - /channels Endpoint not Returning Smaller Users

I am creating a networking app for musicians. I was wanting to use the Youtube Data API to let users connect their Youtube channel to their profile within my app. I got everything in place and working via making requests to URLs similar to https://www.googleapis.com/youtube/v3/channels?part=snippet,statistics&forUsername=PewDiePie&key=[YOUR_API_KEY]. This works great and returns this JSON:
{ "kind": "youtube#channelListResponse", "etag": "\"p4VTdlkQv3HQeTEaXgvLePAydmU/bj_rirVFbrVoTIOa6lCGdaXaG5M\"", "pageInfo": { "totalResults": 1, "resultsPerPage": 5 }, "items": [ { "kind": "youtube#channel", "etag": "\"p4VTdlkQv3HQeTEaXgvLePAydmU/Blp06js4r7j93y1EfKve84oXWpo\"", "id": "UC-lHJZR3Gqxm24_Vd_AJ5Yw", "snippet": { "title": "PewDiePie", "description": "I make videos.", "publishedAt": "2010-04-29T10:54:00.000Z", "thumbnails": { "default": { "url": "https://yt3.ggpht.com/a/AGF-l79FVckie4j9WT-4cEW6iu3gPd4GivQf_XNSWg=s88-c-k-c0xffffffff-no-rj-mo", "width": 88, "height": 88 }, "medium": { "url": "https://yt3.ggpht.com/a/AGF-l79FVckie4j9WT-4cEW6iu3gPd4GivQf_XNSWg=s240-c-k-c0xffffffff-no-rj-mo", "width": 240, "height": 240 }, "high": { "url": "https://yt3.ggpht.com/a/AGF-l79FVckie4j9WT-4cEW6iu3gPd4GivQf_XNSWg=s800-c-k-c0xffffffff-no-rj-mo", "width": 800, "height": 800 } }, "localized": { "title": "PewDiePie", "description": "I make videos." }, "country": "US" }, "statistics": { "viewCount": "24334379402", "commentCount": "0", "subscriberCount": "102000000", "hiddenSubscriberCount": false, "videoCount": "4054" } } ] }
Most of my app's users will be smaller musicians, likely with less than 10k youtube subscribers. Take my sister for example, this is a link to her youtube channel: https://www.youtube.com/channel/UCe4Eogv2uGaKUe4x3VNrwsg.
Whenever trying to search for her Youtube channel with the API via https://www.googleapis.com/youtube/v3/channels?part=snippet,statistics&forUsername=Audrey_Chopin&key=[YOUR_API_KEY] (and variations such as replacing Audrey_Chopin with Audrey%20Chopin or Audrey+Chopin) yield no results: { "kind": "youtube#channelListResponse", "etag": "\"p4VTdlkQv3HQeTEaXgvLePAydmU/zJL80hJ0IwMo5wddECFapC8I6Q4\"", "pageInfo": { "totalResults": 0, "resultsPerPage": 5 }, "items": [] }.
Are smaller users not supposed to be returned from this endpoint? If so, is there any way I can implement users to search for their profile without forcing the user to do the OAuth process, i.e. signing into their Youtube account?
It seems that using the /search endpoint works better for smaller channels, though there is less information available in this endpoint (I am unable to get subscriber count and video count, which was included in the "statistics" part of the /channel endpoint).
So updating
https://www.googleapis.com/youtube/v3/channels?part=snippet,statistics&forUsername=Audrey_Chopin&key=[YOUR_API_KEY]
to
https://www.googleapis.com/youtube/v3/search?part=snippet&channelType=any&maxResults=50&order=relevance&q=Audrey%20Chopin&type=channel&key=[YOUR_API_KEY]
yielded smaller channels, though without as much data as when using the /channel endpoint.
Still curious, if anybody knows, why the /channel endpoint does not return smaller channels.
Since you know the user's channel id, simply issue a query to the Channels endpoint on the URL:
https://www.googleapis.com/youtube/v3/channels?part=...&id=$CHANNEL_ID&key=$APP_KEY,
and you'll obtain all public (i.e. non-private) info attached to the referenced channel -- without needing any further authentication. Of course you can specify the part parameter as you see fit.
On the other hand, please note that querying the Search.List endpoint for snippet part is much more costly than querying the Channels.List endpoint for both snippet and statistics parts: 100 vs. 5 quota points.

AWS EC2 - starting T2 unlimited instances - bug in EC2 API?

T2 instances can now be started with an additional option to allow more CPU bursting for additional cost.
SDK: http://docs.aws.amazon.com/aws-sdk-php/v3/api/api-ec2-2016-11-15.html#runinstances
I tried it, I can switch my instances to unlimited so it should be possible.
However, I added the new configuration option to the array and nothing changed, it's still set to "standard" as before.
Here a JSON dump of the runinstances option array:
{
"UserData": "....",
"SecurityGroupIds": [
"sg-04df967f"
],
"InstanceType": "t2.micro",
"ImageId": "ami-4e3a4051",
"MaxCount": 1,
"MinCount": 1,
"SubnetId": "subnet-22ec130c",
"Tags": [
{
"Key": "task",
"Value": "test"
},
{
"Key": "Name",
"Value": "unlimitedtest"
}
],
"InstanceInitiatedShutdownBehavior": "terminate",
"CreditSpecification": {
"CpuCredits": "unlimited"
}
}
It starts the EC2 instance successfully just as before, however the CreditSpecification setting is ignored.
Amazon denies normal users to contact support, so I hope maybe someone here has a clue about it.
Hmmm... Using qualitatively the same run-instances JSON
{
"ImageId": "ami-bf4193c7",
"InstanceType": "t2.micro",
"CreditSpecification": {
"CpuCredits": "unlimited"
}
}
worked for me - the instance shows this:
T2 Unlimited Enabled
in the "description" tab after selecting this instance in the ec2 console.

SonarQube Component Tree response data

I'm having some trouble understanding some of the data in the response from the SonarQube GET api/measures/component_tree API.
Some metrics have a value attribute while others don't. I've figured out that the value displayed in the UI is the "value" unless it does not exist, then the value at the earliest period is used. The other periods are then basically deltas between measurements. Would anyone be able to provide some details around what the response values actually mean? Unfortunately, the actual API documentation that SonarQube provides doesn't give any detail around response data. Specifically, I'm wondering when a value attribute would and would not be there, what the index means since not all have the same indexes (ie. some go 1-4, others have just 3,4), and what the period data represents.
{
"metric": "new_lines_to_cover",
"periods": [
{
"index": 1,
"value": "572"
},
{
"index": 2,
"value": "572"
},
{
"index": 3,
"value": "8206"
},
{
"index": 4,
"value": "186574"
}
]
},
{
"metric": "duplicated_lines",
"value": "80819",
"periods": [
{
"index": 1,
"value": "-158"
},
{
"index": 2,
"value": "-158"
},
{
"index": 3,
"value": "-10544"
},
{
"index": 4,
"value": "-6871"
}
]
},
{
"metric": "new_line_coverage",
"periods": [
{
"index": 3,
"value": "3.9900249376558605"
},
{
"index": 4,
"value": "17.221615720524017"
}
]
},
The heuristic is very close from the truth:
if the metric starts with "new_", it means it's a metric that compute new elements on a period of time. Starting with 6.3, only the leak period is supported
otherwise, the "value" represents the raw value.
For example, to compute the number of issues:
violations computes the total number of issues
new_violations computes the number of new issues on the leak period
To know more about the leak period concept in SonarQube, please check this article.

Why Google Cloud Speech API doesn't transcript the whole audio file?

I'm trying to transcript a short interview audio file with Google Cloud Speech API (asynchronously) but it only transcribes the first half minute of the recording. I had several attempts with recordings longer than one minute and the results were the same. My question is, how can I achieve the full audio transcription for a given file?
You can find one of my use-cases here:
Upload the audio file:
POST https://speech.googleapis.com/v1beta1/speech:asyncrecognize?key={YOUR_API_KEY}
{
"config": {
"encoding": "LINEAR16",
"sampleRate": 16000,
},
"audio": {
"uri": "gs://protean-blend-146812.appspot.com/record__2017_02_02_12_02_17_greg_16000.wav",
}
}
Got the operation number in the response:
{
"name": "8977932499808116064"
}
Make a request with the operation number:
GET https://speech.googleapis.com/v1beta1/operations/8977932499808116064?key={YOUR_API_KEY}
Got the result:
{
"name": "8977932499808116064",
"metadata": {
"#type": "type.googleapis.com/google.cloud.speech.v1beta1.AsyncRecognizeMetadata",
"progressPercent": 100,
"startTime": "2017-02-02T11:21:41.346784Z",
"lastUpdateTime": "2017-02-02T11:23:03.150491Z"
},
"done": true,
"response": {
"#type": "type.googleapis.com/google.cloud.speech.v1beta1.AsyncRecognizeResponse",
"results": [
{
"alternatives": [
{
"transcript": "McGregor you have any stories about being lost that you have all the good advice well let me know in the Golden Triangle drug trafficking across the border",
"confidence": 0.8535113
}
]
},
{
"alternatives": [
{
"transcript": "we came across this Village very very poor Village People and some of the people there were really unfriendly they just started throwing rocks and my friend and we couldn't talk so we backed away went back quickly up the hill",
"confidence": 0.9027881
}
]
},
{
"alternatives": [
{
"transcript": "and we are wondering you know where to go and luckily I can see in the distance there in one tree",
"confidence": 0.8931573
}
]
}
]
}
}
The links, where I made the requests (at 'Try it!' section):
Upload: https://cloud.google.com/speech/reference/rest/v1beta1/speech/asyncrecognize
Operation response: https://cloud.google.com/speech/reference/rest/v1beta1/operations/get

Resources