Which HTTP endpoint will help me to find all the active frameworks current resource utilization?
We want this information because we want to dynamically scale Mesos cluster and our algorithm needs information regarding what resources each active framework is using.
I think to focus on the frameworks is not really what you would want to to. What you're after is probably the Mesos Slave utilization, which can be requested via calling
http://{mesos-master}:5050/master/state-summary
In the JSON answer, you'll find a slaves property which contains an array of slave objects:
{
"hostname": "192.168.0.3",
"cluster": "mesos-hw-cluster",
"slaves": [{
"id": "bd9c29d7-8530-4c5b-8c50-5d2f60dffbf6-S2",
"pid": "slave(1)#192.168.0.1:5051",
"hostname": "192.168.0.1",
"registered_time": 1456826950.99075,
"resources": {
"cpus": 12.0,
"disk": 1840852.0,
"mem": 63304.0,
"ports": "[31000-32000]"
},
"used_resources": {
"cpus": 5.75,
"disk": 0.0,
"mem": 14376.0,
"ports": "[31000-31000, 31109-31109, 31267-31267, 31699-31699, 31717-31717, 31907-31907, 31979-31980]"
},
"offered_resources": {
"cpus": 0.0,
"disk": 0.0,
"mem": 0.0
},
"reserved_resources": {},
"unreserved_resources": {
"cpus": 12.0,
"disk": 1840852.0,
"mem": 63304.0,
"ports": "[31000-32000]"
},
"attributes": {},
"active": true,
"version": "0.27.1",
"TASK_STAGING": 0,
"TASK_STARTING": 0,
"TASK_RUNNING": 7,
"TASK_FINISHED": 18,
"TASK_KILLED": 27,
"TASK_FAILED": 3,
"TASK_LOST": 0,
"TASK_ERROR": 0,
"framework_ids": ["bd9c29d7-8530-4c5b-8c50-5d2f60dffbf6-0000", "bd9c29d7-8530-4c5b-8c50-5d2f60dffbf6-0002"]
},
...
}
You could iterate over all the slave objects and calculate the overall ressource usage by summarizing the resources and then subtract the summary of the used_resources.
See
http://mesos.apache.org/documentation/latest/endpoints/master/state-summary/
http://mesos.apache.org/documentation/latest/endpoints/
Related
Java 11
SonarQube 8.9.2 LTS
For my java project the SonarQube show the next issues info:
Severity
Blocker 1.3k
Minor 1.1k
Critical 5.8k
Info 233
Major 1.3k
So I need to get this information via SonarQube WEB API.
I found only this api method:
GET http://some_url_sonar_qube/api/issues/search
And its return all issues on page = 1
And its return all issues on page = 1 with detail info
{
"total": 10049,
"p": 1,
"ps": 100,
"paging": {
"pageIndex": 1,
"pageSize": 100,
"total": 10049
},
"effortTotal": 50995,
"issues": [
{
"key": "dddd",
"rule": "css:S4670",
"severity": "CRITICAL",
...
This:
GET http://some_url_sonar_qube/api/issues/search?p=2
And its return all issues on page = 2
and so on.
Response example:
As you can see has 10049 issues. It's 100 pages.
But I need summary info. Smt like this in json format:
{
"Severity": {
"Blocker": 1300,
"Minor": 1100,
"Critical": 5800,
"Info": 233,
"Major": 1300
}
}
I'm not found api method for this
I found solution (thanks for #gawkface)
Use this method:
GET http://some_url_sonar_qube/api/issues/search?componentKeys=my_project_key&facets=severities
And here result (on section facets)
{
"total": 10049,
"p": 1,
"ps": 100,
"paging": {
"pageIndex": 1,
"pageSize": 100,
"total": 10049
},
"effortTotal": 50995,
"issues": [...],
"components": [...],
"facets": [
{
"property": "severities",
"values": [
{
"val": "CRITICAL",
"count": 5817
},
{
"val": "MAJOR",
"count": 1454
},
{
"val": "BLOCKER",
"count": 1286
},
{
"val": "MINOR",
"count": 1161
},
{
"val": "INFO",
"count": 331
}
]
}
]
}
How can you determine Hadoop cluster resources used by a completed job?
Our cluster resource manager is yarn. Access to certain yarn API endpoints is available through HTTP; for example:
curl -L http://my.hadoop.instance:8088/ws/v1/cluster/apps/application_1547448533998_502644
would return:
{
"app": {
"allocatedMB": -1,
"allocatedVCores": -1,
"amContainerLogs": "http://someNode.hadoop.instance:8042/node/containerlogs/container_e149_1547448533998_502644_01_000001/someUser",
"amHostHttpAddress": "someNode.hadoop.instance:8042",
"amNodeLabelExpression": "",
"applicationTags": "",
"applicationType": "SPARK",
"clusterId": 1547448533998,
"clusterUsagePercentage": 0.0,
"diagnostics": "",
"elapsedTime": 583889,
"finalStatus": "SUCCEEDED",
"finishedTime": 1550621490747,
"id": "application_1547448533998_502644",
"logAggregationStatus": "TIME_OUT",
"memorySeconds": 15821179,
"name": "ProjectCantor",
"numAMContainerPreempted": 0,
"numNonAMContainerPreempted": 0,
"preemptedResourceMB": 0,
"preemptedResourceVCores": 0,
"priority": 0,
"progress": 100.0,
"queue": "dsg",
"queueUsagePercentage": 0.0,
"runningContainers": -1,
"startedTime": 1550620906858,
"state": "FINISHED",
"trackingUI": "History",
"trackingUrl": "http://my.hadoop.instance:8088/proxy/application_1547448533998_502644/",
"unmanagedApplication": false,
"user": "someUser",
"vcoreSeconds": 14713
}
}
However, there are several attributes such as allocatedMB and allocatedVCores set to -1 rather than any meaningful value.
Thanks in advance.
How can I add Placement Constraints in mesosphere jobs?
I know there a GUI way to add this to Services and what I need is how can I do the same to JOBS?
This is possible but maybe not exposed in GUI. See below example from readme
{
"id": "sample-job",
"description": "A sample job that sleeps",
"run": {
"cmd": "sleep 1000",
"cpus": 0.01,
"mem": 32,
"disk": 0,
"placement": {
"constraints": [
{
"attribute": "hostname",
"operator": "LIKE",
"value": "<host-name>"
}
]
}
},
"schedules": [
{
"id": "sample-schedule",
"enabled": true,
"cron": "0 0 * * *",
"concurrencyPolicy": "ALLOW"
}
]
}
That's my first stackoverflow question.
I started with kivy and bought the book "Crating Apps in Kivy" from Dusty Phillips and all went well till he got to the openweatherapp chapter. (Which after a search other users had problems too...)
There you use the Openweather API to search for a location with a button and it should print the results in a list.
It's not the original code anymore. Others pointed out that the Link to the website changed, you have to use the API key. Of course I linked it correctly, I just typed "myAPIKey" here so it's not accesssable. The JSOn File on the bottom is what the site prints!
Also, the book formated cities in found location like that: d['name], d['sys']['country'] which I guess was also wrong, maybe the site changed here too?
My problem is that the UrlRequest from kivy doesn't seem to load anything. Or the iteration in found location is wrong, I don't know. It always says "KeyError: city" And I guess it's because it doesn't read it correctly at all. I tried a different approach with the python class requests - and it works perfectly fine! But I still want to know why this solution doesn't work.
My Questions: Where is my error with the json iteration? I have really problems with understanding because I'm also new to python.
Also: Why is request in the methode head? Where is it used?
Here the code from the book (I hope everything is typed correctly. If there is still an error with variables please point it out but I typed the code multiple times - my problem will probably be somewhere else...):
from kivy.app import App
from kivy.uix.boxlayout import BoxLayout
from kivy.properties import ObjectProperty
from kivy.network.urlrequest import UrlRequest
import json
class AddLocationForm(BoxLayout):
search_input = ObjectProperty()
search_results = ObjectProperty()
def search_location(self):
search_template = "http://api.openweathermap.org/data/2.5/forecast/daily?APPID=myAPIKey=" + "{}"
search_url = search_template.format(self.search_input.text)
request = UrlRequest(search_url, self.found_location)
def found_location(self, request, data):
data = json.loads(data.decode()) if not isinstance(data, dict) else data
cities = ["{} ({})".format(d["city"]["name"], d["city"]["country"])for d in data["list"]]
self.search_results.item_strings = cities
KV File:
AddLocationForm:
<AddLocationForm>:
orientation: "vertical"
search_input: search_box
search_results: search_results_list
BoxLayout:
height: "40dp"
size_hint_y: None
TextInput:
id: search_box
size_hint_x: 50
Button:
text: "Search"
size_hint_x: 25
on_press: root.search_location()
Button:
text: "Current Location"
size_hint_x: 25
ListView:
id: search_results_list
item_strings: []
JSON File:
{
"city": {
"id": 2761369,
"name": "Vienna",
"coord": {
"lon": 16.37208,
"lat": 48.208488
},
"country": "AT",
"population": 0
},
"cod": "200",
"message": 0.0098,
"cnt": 7,
"list": [{
"dt": 1476439200,
"temp": {
"day": 285.58,
"min": 283.71,
"max": 285.58,
"night": 283.71,
"eve": 285.43,
"morn": 285.58
},
"pressure": 985.46,
"humidity": 73,
"weather": [{
"id": 800,
"main": "Clear",
"description": "clear sky",
"icon": "01d"
}],
"speed": 7.16,
"deg": 154,
"clouds": 0
}, {
"dt": 1476525600,
"temp": {
"day": 287.33,
"min": 282.7,
"max": 291.08,
"night": 285.72,
"eve": 291.01,
"morn": 283.35
},
"pressure": 983.21,
"humidity": 71,
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}],
"speed": 4.51,
"deg": 150,
"clouds": 32,
"rain": 0.23
}, {
"dt": 1476612000,
"temp": {
"day": 286.88,
"min": 283.94,
"max": 287.15,
"night": 283.94,
"eve": 286.44,
"morn": 286.26
},
"pressure": 989.94,
"humidity": 98,
"weather": [{
"id": 501,
"main": "Rain",
"description": "moderate rain",
"icon": "10d"
}],
"speed": 4.36,
"deg": 315,
"clouds": 92,
"rain": 5.51
}, {
"dt": 1476698400,
"temp": {
"day": 287.39,
"min": 283.49,
"max": 287.39,
"night": 284.89,
"eve": 285.05,
"morn": 283.49
},
"pressure": 984.76,
"humidity": 0,
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}],
"speed": 1.64,
"deg": 132,
"clouds": 69,
"rain": 1.55
}, {
"dt": 1476784800,
"temp": {
"day": 285.91,
"min": 283.52,
"max": 285.91,
"night": 283.52,
"eve": 284.45,
"morn": 284.18
},
"pressure": 982.64,
"humidity": 0,
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}],
"speed": 3.23,
"deg": 137,
"clouds": 81,
"rain": 2.4
}, {
"dt": 1476871200,
"temp": {
"day": 283.71,
"min": 282.37,
"max": 283.71,
"night": 282.37,
"eve": 282.52,
"morn": 282.59
},
"pressure": 978.26,
"humidity": 0,
"weather": [{
"id": 502,
"main": "Rain",
"description": "heavy intensity rain",
"icon": "10d"
}],
"speed": 2.68,
"deg": 128,
"clouds": 96,
"rain": 14.37
}, {
"dt": 1476957600,
"temp": {
"day": 286.52,
"min": 282.13,
"max": 286.52,
"night": 282.13,
"eve": 282.72,
"morn": 282.19
},
"pressure": 975.05,
"humidity": 0,
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}],
"speed": 1.47,
"deg": 173,
"clouds": 13,
"rain": 2.83
}]
}
Btw, it's not a duplicate from how do i use json api in my kivy program becaue it didn't really help... The code doesn't work anymore because the API changed.
(If anyone finds this question and is looking for a working URL reader, here is my code. Don't forget to import requests)
def search_location(self):
#Get URL From Weather API + Input
search_template = "http://api.openweathermap.org/data/2.5/forecast/daily?APPID='enteryourweatherAPIKeyhere'&q=" + "{}"
search_url = search_template.format(self.search_input.text)
#Reads the Website and saves it in a string
urlresults = (requests.get(search_url)).text
#Converts Json to Python Dictionary
urlresults = json.loads(urlresults)
cities = ["{} ({})".format(urlresults["city"]["name"], urlresults["city"]["country"])]
Go to this page:
http://openweathermap.org/faq#error401
the API now requires a key. After registering on openweathermap, a key will be mailed to you. Replace the current URL with the one given in the mail.
This worked for me.
https://docs.jelastic.com/api/?class=environment.Control&member=GetStats
At the above link in the Jelastic API documentation for the GetStats method there are two parameters duration and interval.
When querying the api i can't figure out how these two parameters interact with each other.
If i query with the below i would expect 100 records at a resolution of 1 minute
/1.0/environment/control/rest/getstats?domain=[myDomiain]&session=[MySession]&duration=6000&interval=60&nodeid=[MyNode]
What i get back is 4 records for each hour so i'm unsure of how the parameters work.
Should i be using GetSumStats?
My final question would be what format are the cpu and mem stats in? MHz and Bytes?
{
"iops_used": 0,
"duration": 3600,
"cpumhz": 3,
"start": "2016-05-03 08:00:00",
"disk": 2141,
"mem": 194840,
"cpu": 12254,
"capacity": 0,
"net": {
"in_int": 703019,
"out_int": 566947,
"in_ext": 46222,
"out_ext": 367209
}
},
{
"iops_used": 0,
"duration": 3600,
"cpumhz": 3,
"start": "2016-05-03 09:00:00",
"disk": 2141,
"mem": 171992,
"cpu": 10076,
"capacity": 0,
"net": {
"in_int": 156703,
"out_int": 314023,
"in_ext": 12627,
"out_ext": 13535
}
},
{
"iops_used": 0,
"duration": 3580,
"cpumhz": 3,
"start": "2016-05-03 10:00:00",
"disk": 2141,
"mem": 172400,
"cpu": 11198,
"capacity": 0,
"net": {
"in_int": 515521,
"out_int": 551317,
"in_ext": 10329,
"out_ext": 17161
}
},
{
"iops_used": 0,
"duration": 3601,
"cpumhz": 3,
"start": "2016-05-03 11:00:00",
"disk": 2141,
"mem": 172610,
"cpu": 10032,
"capacity": 0,
"net": {
"in_int": 153394,
"out_int": 310694,
"in_ext": 10285,
"out_ext": 11210
}
}
#dlearious, for using interval equal 60 you should set duration value to 3600. This is due to the fact that Jelastic keeps detailed data hourly.
Also, you can start from minimal interval = 20.
Jelastic shows cpu in milliseconds and mem in Bytes.