How can you determine Hadoop cluster resources used by a completed job?
Our cluster resource manager is yarn. Access to certain yarn API endpoints is available through HTTP; for example:
curl -L http://my.hadoop.instance:8088/ws/v1/cluster/apps/application_1547448533998_502644
would return:
{
"app": {
"allocatedMB": -1,
"allocatedVCores": -1,
"amContainerLogs": "http://someNode.hadoop.instance:8042/node/containerlogs/container_e149_1547448533998_502644_01_000001/someUser",
"amHostHttpAddress": "someNode.hadoop.instance:8042",
"amNodeLabelExpression": "",
"applicationTags": "",
"applicationType": "SPARK",
"clusterId": 1547448533998,
"clusterUsagePercentage": 0.0,
"diagnostics": "",
"elapsedTime": 583889,
"finalStatus": "SUCCEEDED",
"finishedTime": 1550621490747,
"id": "application_1547448533998_502644",
"logAggregationStatus": "TIME_OUT",
"memorySeconds": 15821179,
"name": "ProjectCantor",
"numAMContainerPreempted": 0,
"numNonAMContainerPreempted": 0,
"preemptedResourceMB": 0,
"preemptedResourceVCores": 0,
"priority": 0,
"progress": 100.0,
"queue": "dsg",
"queueUsagePercentage": 0.0,
"runningContainers": -1,
"startedTime": 1550620906858,
"state": "FINISHED",
"trackingUI": "History",
"trackingUrl": "http://my.hadoop.instance:8088/proxy/application_1547448533998_502644/",
"unmanagedApplication": false,
"user": "someUser",
"vcoreSeconds": 14713
}
}
However, there are several attributes such as allocatedMB and allocatedVCores set to -1 rather than any meaningful value.
Thanks in advance.
Related
I have multiple requests in the same thread group and i need to pass 18 users to the 2 game id's (9 users to each gameID) and i am using counter element to create different users and gameID's.
Login Request:
42[
"get_login",
{
"uid": "EiNjkkkxgqq1ClN7gwzJEBK5XPhYKsG5",
"key": "AUEEW891WL",
"socketId":"AvKjxCGPLzgvqGxzAAb-",
"username": "",
"avatar": "",
"language": "en",
"playerMove": "",
"joinGame": "",
"replay": 0,
"gameID": 0,
"gameNo": 0,
"data": "&email=Nov${counter}#gmail.com&password=poker",
"players": 0,
"level": "",
"lastAction": "",
"game": "",
"playMoney": 1,
"role": ""
}
]
Response:
42[
"gotLogin",
{
"response": {
"playerId": 24441,
"totalWin": 0,
"points": 0,
"playerName": "672319400464",
"displayName": "DEC65",
"playerRank": "Amateur",
"playerBank": "11K",
"playerBankAmount": 11000,
"playerAvatar": "avatar1.jpg",
"playerEmail": "Nov65#gmail.com",
"playerLevel": 1,
"playerLevelStr": "Lvl. 1",
"playerLevelPrg": 0,
"playerLevelPoints": 100,
"playerGold": 0,
"playerDealerId": 1,
"playerFbId": "",
"loadVocab": "",
"dailyBonus": 0,
"token": "37333e039503825fded65f7714f04741688af294ff0a32a7c676f5c366e193de"
}
}
]
Create Game
Request:
42[
"create_game",
{
"uid": "Esdv7CHkwo1ATMfvY6NcWBTM5YB4d3nj",
"key": "AUEEW891WL",
"socketId":"AvKjxCGPLzgvqGxzAAb-",
"username": "611839990703",
"avatar": "avatar17.jpg",
"language": "en",
"playerMove": "",
"joinGame": "",
"replay": 0,
"gameID": 0,
"gameNo": 0,
"data": "&gameID=undefined&game=texas&playMoney=1&gameStyle=private-cashgame&tableName=TableOct${counter}&rakeRate=0&speed=60&sb=10&bb=20&tablelow=100&tablelimit=1000&videorequired=false&username=611839990703",
"players": 0,
"level": 347,
"lastAction": "",
"game": "",
"playMoney": 1,
"role": "1",
"token":"84d26de7404377dfa0ade2ff74676fd901852e19100068d43ac8a8d449f49a1c",
"playerId":"23910",
"displayName": "p10"
}
]
Response:
42[
"createGameRsp",
{
"response": {
"title": "Create Successful",
"result": "TableOct2 was successfully created. \n GameID: 630 \n Password: ",
"error": "",
"game": "texas",
"gameID": 630
}
}
]
Join Game:
Request:
42[
"get_joingame",
{
"uid": "Esdv7CHkwo1ATMfvY6NcWBTM5YB4d3nj",
"key": "AUEEW891WL",
"socketId":"AvKjxCGPLzgvqGxzAAb-",
"username":${userName}",
"avatar": "avatar17.jpg",
"language": "en",
"playerMove": "",
"joinGame": "",
"replay": 0,
"gameID":"${gameID}",
"gameNo": 0,
"data": "&seat=${counter2}&buyin=500&privateTable=2&clubId=0",
"players": 0,
"level": 347,
"lastAction": "ep.playNow",
"game": "texas",
"playMoney": 1,
"role": "1",
"playerId":"${playerId}",
"displayName": "p10",
"seat":${counter2}
}
]
Response:
42[
"gotJoinGame",
{
"game": "texas",
"gameID": "630",
"response": {
"error": ""
}
}
]
Right now i am able to pass one game id to only one user. But i need to pass one game id to 9 users. How is it possible?
As per JMeter documentation on variables:
Properties are not the same as variables. Variables are local to a thread; properties are common to all threads
So if you extract the gameID by one thread (virtual user) - it will be available only to this thread. If you want to use the value by other virtual user(s) - you need to convert it into a JMeter Property via __setProperty() function or suitable JSR223 Test Element
If you have 2 groups of users you can assign one gameID to users with even thread number and the other to odd users.
More information: Using JMeter Variables With Multiple Thread Groups
{ "adminTags": "", "allowedPartnerIds": "", "allowedPartnerPackages": "", "city": "", "country": "", "createdAt": 0, "deletedAt": 0, "description": "", "email": "", "fullName": "", "id": "", "indexedPartnerDataInt": 0, "indexedPartnerDataString": "", "language": "aa", "lastLoginTime": 0, "objectType": "KalturaUser", "partnerData": "", "partnerId": 0, "screenName": "", "state": "", "status": 0, "statusUpdatedAt": 0, "storageSize": 0, "tags": "", "thumbnailUrl": "", "updatedAt": 0, "userMode": 0, "zip": "", "attendanceInfo": "", "dateOfBirth": 0, "firstName": "", "gender": 0, "isAccountOwner": true, "isAdmin": true, "lastName": "", "loginEnabled": true, "password": "", "registrationInfo": "", "roleIds": "", "roleNames": "", "type": 0 }
I am pretty new to using the kaltura api. I want to dynamically adjust some of the objects via the api. For instance, the co-editor or co-partner, but I do not see in the object, which do I use or what actions in the documentations do you recommend?
If the property doesn't exist on the object (in your case the KalturaUser) then it is not updateable this object service (REST API).
You can find the documentation here.
I think you meant to update the co-editor / co-partner on the entry object - as each entry can have entitledUsersEdit / entitledUsersPublish fields which can contain list of user IDs. You can see example here.
So, after you have the user id from the object you have - you can use the baseEntry.update (from the link above) to set your user as co-editor/co-partner to the entry
When I'm trying to connect Power BI to Hadoop webhdfs, i get this error
DataSource.Error: HDFS failed to get contents from 'http://xxx.xx.x.x:50070/webhdfs/v1/myFolder/20200626150740_PERSONAL_IDS'. Status code: 503, description: 'Service Unavailable'.
When I access the URL, error like this occurs:
{
RemoteException: {
exception: "UnsupportedOperationException",
javaClassName: "java.lang.UnsupportedOperationException",
message: "op=NULL is not supported"
}
}
Anyone can help this out?
An operation (?op=...) is required when you use webhdfs REST API with http(s): scheme.
For example, when I type this in my browser:
https://<my.namenode.host>:9871/webhdfs/v1/user?op=LISTSTATUS
... I get back the list of hdfs directory /user content (similar to what I'll get with hadoop fs -ls /user plus some additional info):
// 20200709155934
// https://<my.namenode.host>:9871/webhdfs/v1/user?op=LISTSTATUS
{
"FileStatuses": {
"FileStatus": [
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 2,
"fileId": 56899,
"group": "supergroup",
"length": 0,
"modificationTime": 1592790690971,
"owner": "hdfs",
"pathSuffix": "hdfs",
"permission": "700",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
},
. . . .
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 2,
"fileId": 16390,
"group": "hive",
"length": 0,
"modificationTime": 1591715444764,
"owner": "hive",
"pathSuffix": "hive",
"permission": "1775",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
},
. . . .
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 2,
"fileId": 16454,
"group": "mazaneicha",
"length": 0,
"modificationTime": 1591035918913,
"owner": "mazaneicha",
"pathSuffix": "mazaneicha",
"permission": "720",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
},
. . . .
]
}
}
You can find the complete list of REST API operations online.
https://docs.jelastic.com/api/?class=environment.Control&member=GetStats
At the above link in the Jelastic API documentation for the GetStats method there are two parameters duration and interval.
When querying the api i can't figure out how these two parameters interact with each other.
If i query with the below i would expect 100 records at a resolution of 1 minute
/1.0/environment/control/rest/getstats?domain=[myDomiain]&session=[MySession]&duration=6000&interval=60&nodeid=[MyNode]
What i get back is 4 records for each hour so i'm unsure of how the parameters work.
Should i be using GetSumStats?
My final question would be what format are the cpu and mem stats in? MHz and Bytes?
{
"iops_used": 0,
"duration": 3600,
"cpumhz": 3,
"start": "2016-05-03 08:00:00",
"disk": 2141,
"mem": 194840,
"cpu": 12254,
"capacity": 0,
"net": {
"in_int": 703019,
"out_int": 566947,
"in_ext": 46222,
"out_ext": 367209
}
},
{
"iops_used": 0,
"duration": 3600,
"cpumhz": 3,
"start": "2016-05-03 09:00:00",
"disk": 2141,
"mem": 171992,
"cpu": 10076,
"capacity": 0,
"net": {
"in_int": 156703,
"out_int": 314023,
"in_ext": 12627,
"out_ext": 13535
}
},
{
"iops_used": 0,
"duration": 3580,
"cpumhz": 3,
"start": "2016-05-03 10:00:00",
"disk": 2141,
"mem": 172400,
"cpu": 11198,
"capacity": 0,
"net": {
"in_int": 515521,
"out_int": 551317,
"in_ext": 10329,
"out_ext": 17161
}
},
{
"iops_used": 0,
"duration": 3601,
"cpumhz": 3,
"start": "2016-05-03 11:00:00",
"disk": 2141,
"mem": 172610,
"cpu": 10032,
"capacity": 0,
"net": {
"in_int": 153394,
"out_int": 310694,
"in_ext": 10285,
"out_ext": 11210
}
}
#dlearious, for using interval equal 60 you should set duration value to 3600. This is due to the fact that Jelastic keeps detailed data hourly.
Also, you can start from minimal interval = 20.
Jelastic shows cpu in milliseconds and mem in Bytes.
Which HTTP endpoint will help me to find all the active frameworks current resource utilization?
We want this information because we want to dynamically scale Mesos cluster and our algorithm needs information regarding what resources each active framework is using.
I think to focus on the frameworks is not really what you would want to to. What you're after is probably the Mesos Slave utilization, which can be requested via calling
http://{mesos-master}:5050/master/state-summary
In the JSON answer, you'll find a slaves property which contains an array of slave objects:
{
"hostname": "192.168.0.3",
"cluster": "mesos-hw-cluster",
"slaves": [{
"id": "bd9c29d7-8530-4c5b-8c50-5d2f60dffbf6-S2",
"pid": "slave(1)#192.168.0.1:5051",
"hostname": "192.168.0.1",
"registered_time": 1456826950.99075,
"resources": {
"cpus": 12.0,
"disk": 1840852.0,
"mem": 63304.0,
"ports": "[31000-32000]"
},
"used_resources": {
"cpus": 5.75,
"disk": 0.0,
"mem": 14376.0,
"ports": "[31000-31000, 31109-31109, 31267-31267, 31699-31699, 31717-31717, 31907-31907, 31979-31980]"
},
"offered_resources": {
"cpus": 0.0,
"disk": 0.0,
"mem": 0.0
},
"reserved_resources": {},
"unreserved_resources": {
"cpus": 12.0,
"disk": 1840852.0,
"mem": 63304.0,
"ports": "[31000-32000]"
},
"attributes": {},
"active": true,
"version": "0.27.1",
"TASK_STAGING": 0,
"TASK_STARTING": 0,
"TASK_RUNNING": 7,
"TASK_FINISHED": 18,
"TASK_KILLED": 27,
"TASK_FAILED": 3,
"TASK_LOST": 0,
"TASK_ERROR": 0,
"framework_ids": ["bd9c29d7-8530-4c5b-8c50-5d2f60dffbf6-0000", "bd9c29d7-8530-4c5b-8c50-5d2f60dffbf6-0002"]
},
...
}
You could iterate over all the slave objects and calculate the overall ressource usage by summarizing the resources and then subtract the summary of the used_resources.
See
http://mesos.apache.org/documentation/latest/endpoints/master/state-summary/
http://mesos.apache.org/documentation/latest/endpoints/