I use post man and I am able to run a request perfectly fine with status 200
https://abc/api/
when I try Jmeter I keep getting 403 forbidden
GET https://abc/api/
I get
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
</body>
</html>
Sampler request
Thread Name: 0-test1-1
Sample Start: 2019-07-22 11:07:12 PDT
Load time: 485
Connect Time: 378
Latency: 485
Size in bytes: 287
Sent bytes:236
Headers size in bytes: 153
Body size in bytes: 134
Sample Count: 1
Error Count: 1
Data type ("text"|"bin"|""): text
Response code: 403
Response message: Forbidden
HTTPSampleResult fields:
ContentType: text/html
DataEncoding: null
My response headers are
HTTP/1.1 403 Forbidden
Server: aws../2.0
Date: Mon, 22 Jul 2019 18:13:06 GMT
Content-Type: text/html
Content-Length: 134
Connection: keep-alive
Note sure if that is
I also set my port to 443 because it is https
I was able to get content-type to application/json but still get 403, I do notice that my content-length is 0
I figured it out it was indeed the User-agent, I forced my Jmeter in header manager
User Agent --> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36 it worked
Maybe you need add into the header of the request
Content-Type: application/json
After setting 3 controllers to include mesos-master, mesos-slave and zookeeper. Marathon was started in last after 3 services run completed. I found INFO message at Marathon log as below
[2017-04-07 15:48:32,291] INFO Acknowledge status update for task service-dev_mysql-serv.a396a075-1ba9-11e7-ab7a-0242ac120004: TASK_FAILED (Failed to run docker container: Port mappings are only supported for bridge and user-defined networks) (mesosphere.marathon.core.task.update.impl.TaskStatusUpdateProcessorImpl:ForkJoinPool-3-worker-13)
Start for starting application on Zookeeper.
Zookeeper
$sudo docker run -it --net=host --name=zookeeper netflixoss/exhibitor:1.5.2
Mesos-master connects to Zookeeper by command line.
$sudo docker run -it --net=host \
--name jdb-network-mesos-master \
-e MESOS_PORT=5050 \
-e MESOS_ZK=zk://192.168.1.68:2181/mesos \
-e MESOS_QUORUM=1 \
-e MESOS_REGISTRY=in_memory \
-e MESOS_LOG_DIR=/var/log/mesos \
-e MESOS_WORK_DIR=/var/tmp/mesos \
-v "$(pwd)/log/mesos:/var/log/mesos" \
-v "$(pwd)/tmp/mesos:/var/tmp/mesos" \
-p 15050:5050 \
-p 12181:2181 \
docker.io/mesosphere/mesos-master:1.1.1-rc2
Mesos-Slave runing to prepare states for Marathon by specified the resource port ports:[2000-9000,20000-34000].
*MESOS-SALVE
$sudo docker run -it --net=host --privileged \
--name jdb-network-mesos-slave \
-e MESOS_PORT=5051 \
-e MESOS_MASTER=zk://192.168.1.68:2181/mesos \
-e MESOS_SWITCH_USER=0 \
-e MESOS_CONTAINERIZERS=docker,mesos \
-e MESOS_LOG_DIR=/var/log/mesos-slave \
-e MESOS_WORK_DIR=/var/tmp/mesos-slave \
-e MESOS_executor_registration_timeout=10mins \
-e MESOS_resources='ports:[2000-9000,20000-34000]' \
-e MESOS_launcher=posix \
-v "$(pwd)/log/mesos-slave:/var/log/mesos" \
-v "$(pwd)/tmp/mesos-slave:/var/tmp/mesos" \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys:/sys \
-v /cgroup:/cgroup \
-v /usr/local/bin/docker:/usr/local/bin/docker \
-v /lib64/libdevmapper.so.1.02:/lib/libdevmapper.so.1.02:ro \
-p 25051:5051 \
-p 22181:2181 \
mesosphere/mesos-slave:1.1.1-rc2
Application can connect completed between master and slave services. Zookeeper show is normal logs. Marathon run with command line below.
$sudo docker run -it \
--name jdb-network-marathon \
--rm \
--net=host \
-e MESOS_WORK_DIR=/var/lib/mesos \
-p 7070:7070 -p 32181:2181 \
--entrypoint=bash mesosphere/marathon:latest-dev
root-marathon(docker-container)$./bin/start \
--master zk://192.168.1.68:2181/mesos \
--zk zk://192.168.1.68:2181/marathon \
--http_port=7070
Json for creating container in docker is
{
"id":"/service-dev/mysql-service",
"cmd":null,
"cpus":1,
"mem":512,
"disk":512,
"instances":1,
"container":{
"docker":{
"image":"mysql",
"network":"BRIDGE",
"portMappings":[
{
"containerPort":3306,
"hostPort":31036,
"protocol":"tcp",
"name":null,
"labels":null
}
]
},
"type":"DOCKER"
}
}
Marathon-UI run on port localhost:7070/ui/ show Delayed and Wait status of application. I attach full message information creates by Marathon is
[2017-04-07 15:48:26,563] INFO Computed new deployment plan:
DeploymentPlan id=f210ab3f-c143-4aad-b051-186646b44e26,2017-04-07T15:48:26.558Z
step 1:
Restart(App(/service-dev/mysql-serv, image="mysql"List())))
(mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-1)
[2017-04-07 15:48:26,582] INFO Deploy plan with force=false:
DeploymentPlan id=f210ab3f-c143-4aad-b051-186646b44e26,2017-04-07T15:48:26.558Z
step 1:
Restart(App(/service-dev/mysql-serv, image="mysql"List())))
(mesosphere.marathon.MarathonSchedulerService:ForkJoinPool-3-worker-39)
[2017-04-07 15:48:26,582] INFO Received new deployment plan f210ab3f-c143-4aad-b051-186646b44e26, no conflicts detected (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-7)
[2017-04-07 15:48:26,586] INFO Stored new deployment plan f210ab3f-c143-4aad-b051-186646b44e26 (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-23)
[2017-04-07 15:48:26,586] INFO Launching DeploymentActor for f210ab3f-c143-4aad-b051-186646b44e26 (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-23)
[2017-04-07 15:48:26,595] INFO Updated groups/apps/pods according to plan f210ab3f-c143-4aad-b051-186646b44e26 (mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-21)
[2017-04-07 15:48:26,595] INFO Deployment acknowledged. Waiting to get processed:
DeploymentPlan id=f210ab3f-c143-4aad-b051-186646b44e26,2017-04-07T15:48:26.558Z
step 1:
Restart(App(/service-dev/mysql-serv, image="mysql"List())))
(mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-27)
[2017-04-07 15:48:26,595] INFO 192.168.1.70 - - [07/Apr/2017:15:48:26 +0000] "PUT //192.168.1.68:7070/v2/apps//service-dev/mysql-serv HTTP/1.1" 200 92 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 4 (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-459)
[2017-04-07 15:48:26,596] INFO For minimumHealthCapacity 1.0 of /service-dev/mysql-serv leave 1 instances running, maximum capacity 2, killing 0 of 0 running instances immediately. (RunSpec version 2017-04-07T15:48:26.558Z) (mesosphere.marathon.core.deployment.impl.TaskReplaceActor$:marathon-akka.actor.default-dispatcher-6)
[2017-04-07 15:48:26,601] INFO reconcile: found 0 already started instances and 0 old instances (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-7)
[2017-04-07 15:48:26,602] INFO Reconciling instances during app /service-dev/mysql-serv restart: queuing 1 new instances (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-7)
[2017-04-07 15:48:26,602] INFO Started instanceLaunchActor for /service-dev/mysql-serv version 2017-04-07T15:48:26.558Z with initial count 1 (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-7)
[2017-04-07 15:48:26,603] INFO activating matcher ActorOfferMatcher(Actorakka://marathon/user/launchQueue/1/9-service-dev_mysql-serv#1478068544). (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,603] INFO Resetting the backoff delay before restarting the runSpec (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,603] INFO Received offers WANTED notification (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,603] INFO => revive offers NOW, canceling any scheduled revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,604] INFO 2 further revives still needed. Repeating reviveOffers according to --revive_offers_repetitions 3 (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,604] INFO => Schedule next revive at 2017-04-07T15:48:31.603Z in 5000 milliseconds, adhering to --min_revive_offers_interval 5000 (ms) (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,608] INFO No tasks left to launch. Stop receiving offers for /service-dev/mysql-serv, 2017-04-07T15:48:26.558Z (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,608] INFO removing matcher ActorOfferMatcher(Actorakka://marathon/user/launchQueue/1/9-service-dev_mysql-serv#1478068544) (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:26,608] INFO Received offers NOT WANTED notification, canceling 2 revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-6)
[2017-04-07 15:48:26,608] INFO => Suppress offers NOW (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-6)
[2017-04-07 15:48:26,609] INFO Finished processing 285ef74b-aaa3-44cf-b650-d687fc6e2923-O156 from dc00cb3b8dbd. Matched 1 ops after 2 passes. ports 2000->9000,20000->31035,31037->34000; cpus 15.0; mem 70771.0; disk 4599.0 left. (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:26,609] INFO Processing LaunchEphemeral(Instance(instance [service-dev_mysql-serv.marathon-a396a075-1ba9-11e7-ab7a-0242ac120004],AgentInfo(dc00cb3b8dbd,Some(285ef74b-aaa3-44cf-b650-d687fc6e2923-S1),Vector()),InstanceState(Created,2017-04-07T15:48:26.608Z,None,None),Map(task [service-dev_mysql-serv.a396a075-1ba9-11e7-ab7a-0242ac120004] -> LaunchedEphemeral(task [service-dev_mysql-serv.a396a075-1ba9-11e7-ab7a-0242ac120004],2017-04-07T15:48:26.558Z,Status(2017-04-07T15:48:26.608Z,None,None,Created,NetworkInfo(dc00cb3b8dbd,Vector(31036),List())))),2017-04-07T15:48:26.558Z,UnreachableEnabled(300 seconds,600 seconds))) for instance [service-dev_mysql-serv.marathon-a396a075-1ba9-11e7-ab7a-0242ac120004] (mesosphere.marathon.core.launcher.impl.OfferProcessorImpl:ForkJoinPool-3-worker-9)
[2017-04-07 15:48:26,735] INFO 192.168.1.70 - - [07/Apr/2017:15:48:26 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions HTTP/1.1" 200 119 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-268)
[2017-04-07 15:48:26,881] INFO 192.168.1.70 - - [07/Apr/2017:15:48:26 +0000] "GET //192.168.1.68:7070/v2/queue HTTP/1.1" 200 32 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-226)
[2017-04-07 15:48:26,881] INFO 192.168.1.70 - - [07/Apr/2017:15:48:26 +0000] "GET //192.168.1.68:7070/v2/deployments HTTP/1.1" 200 228 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-483)
[2017-04-07 15:48:26,882] INFO 192.168.1.70 - - [07/Apr/2017:15:48:26 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv?embed=app.taskStats&embed=app.readiness HTTP/1.1" 200 898 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-477)
[2017-04-07 15:48:27,061] INFO 192.168.1.70 - - [07/Apr/2017:15:48:27 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions HTTP/1.1" 200 119 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-513)
[2017-04-07 15:48:27,062] INFO 192.168.1.70 - - [07/Apr/2017:15:48:27 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions/2017-04-07T15:48:26.558Z HTTP/1.1" 200 472 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-498)
[2017-04-07 15:48:31,758] INFO 192.168.1.70 - - [07/Apr/2017:15:48:31 +0000] "GET //192.168.1.68:7070/v2/queue HTTP/1.1" 200 32 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-226)
[2017-04-07 15:48:31,759] INFO 192.168.1.70 - - [07/Apr/2017:15:48:31 +0000] "GET //192.168.1.68:7070/v2/deployments HTTP/1.1" 200 228 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-459)
[2017-04-07 15:48:31,759] INFO 192.168.1.70 - - [07/Apr/2017:15:48:31 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv?embed=app.taskStats&embed=app.readiness HTTP/1.1" 200 898 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-268)
[2017-04-07 15:48:31,805] INFO 192.168.1.70 - - [07/Apr/2017:15:48:31 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions/2017-04-07T15:48:26.558Z HTTP/1.1" 200 472 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-477)
[2017-04-07 15:48:31,808] INFO 192.168.1.70 - - [07/Apr/2017:15:48:31 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions HTTP/1.1" 200 119 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-483)
[2017-04-07 15:48:32,285] INFO Received status update for task service-dev_mysql-serv.a396a075-1ba9-11e7-ab7a-0242ac120004: TASK_FAILED (Failed to run docker container: Port mappings are only supported for bridge and user-defined networks) (mesosphere.marathon.MarathonScheduler:Thread-407)
[2017-04-07 15:48:32,287] INFO all tasks of instance [service-dev_mysql-serv.marathon-a396a075-1ba9-11e7-ab7a-0242ac120004] are terminal, requesting to expunge (mesosphere.marathon.core.instance.update.InstanceUpdater$:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:32,289] INFO Removed app [/service-dev/mysql-serv] from tracker (mesosphere.marathon.core.task.tracker.InstanceTracker$InstancesBySpec$:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:32,290] INFO Increasing delay. Task launch delay for [/service-dev/mysql-serv] changed from [0 milliseconds] to [1 seconds]. (mesosphere.marathon.core.launchqueue.impl.RateLimiter$:marathon-akka.actor.default-dispatcher-19)
[2017-04-07 15:48:32,290] INFO receiveInstanceUpdate: instance [service-dev_mysql-serv.marathon-a396a075-1ba9-11e7-ab7a-0242ac120004] was deleted (Failed) (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-25)
[2017-04-07 15:48:32,291] INFO initiating a scale check for runSpec [/service-dev/mysql-serv] due to [instance [service-dev_mysql-serv.marathon-a396a075-1ba9-11e7-ab7a-0242ac120004]] Failed (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:32,291] INFO Acknowledge status update for task service-dev_mysql-serv.a396a075-1ba9-11e7-ab7a-0242ac120004: TASK_FAILED (Failed to run docker container: Port mappings are only supported for bridge and user-defined networks) (mesosphere.marathon.core.task.update.impl.TaskStatusUpdateProcessorImpl:ForkJoinPool-3-worker-13)
[2017-04-07 15:48:32,296] ERROR New instance instance [service-dev_mysql-serv.marathon-a396a075-1ba9-11e7-ab7a-0242ac120004] failed on agent Some(285ef74b-aaa3-44cf-b650-d687fc6e2923-S1) during app /service-dev/mysql-serv restart (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-27)
[2017-04-07 15:48:32,297] INFO Reconciling instances during app /service-dev/mysql-serv restart: queuing 1 new instances (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-27)
[2017-04-07 15:48:32,297] INFO add 1 instances to 0 instances to launch (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-27)
[2017-04-07 15:48:33,307] INFO activating matcher ActorOfferMatcher(Actorakka://marathon/user/launchQueue/1/9-service-dev_mysql-serv#1478068544). (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:33,307] INFO Received offers WANTED notification (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:33,307] INFO => revive offers NOW, canceling any scheduled revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:33,307] INFO 2 further revives still needed. Repeating reviveOffers according to --revive_offers_repetitions 3 (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:33,307] INFO => Schedule next revive at 2017-04-07T15:48:38.307Z in 5000 milliseconds, adhering to --min_revive_offers_interval 5000 (ms) (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:33,311] INFO No tasks left to launch. Stop receiving offers for /service-dev/mysql-serv, 2017-04-07T15:48:26.558Z (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-7)
[2017-04-07 15:48:33,311] INFO removing matcher ActorOfferMatcher(Actorakka://marathon/user/launchQueue/1/9-service-dev_mysql-serv#1478068544) (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-7)
[2017-04-07 15:48:33,311] INFO Received offers NOT WANTED notification, canceling 2 revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-4)
[2017-04-07 15:48:33,311] INFO => Suppress offers NOW (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-4)
[2017-04-07 15:48:33,312] INFO Finished processing 285ef74b-aaa3-44cf-b650-d687fc6e2923-O157 from dc00cb3b8dbd. Matched 1 ops after 2 passes. ports 2000->9000,20000->31035,31037->34000; cpus 15.0; mem 70771.0; disk 4599.0 left. (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-4)
[2017-04-07 15:48:33,312] INFO Processing LaunchEphemeral(Instance(instance [service-dev_mysql-serv.marathon-a7956c66-1ba9-11e7-ab7a-0242ac120004],AgentInfo(dc00cb3b8dbd,Some(285ef74b-aaa3-44cf-b650-d687fc6e2923-S1),Vector()),InstanceState(Created,2017-04-07T15:48:33.311Z,None,None),Map(task [service-dev_mysql-serv.a7956c66-1ba9-11e7-ab7a-0242ac120004] -> LaunchedEphemeral(task [service-dev_mysql-serv.a7956c66-1ba9-11e7-ab7a-0242ac120004],2017-04-07T15:48:26.558Z,Status(2017-04-07T15:48:33.311Z,None,None,Created,NetworkInfo(dc00cb3b8dbd,Vector(31036),List())))),2017-04-07T15:48:26.558Z,UnreachableEnabled(300 seconds,600 seconds))) for instance [service-dev_mysql-serv.marathon-a7956c66-1ba9-11e7-ab7a-0242ac120004] (mesosphere.marathon.core.launcher.impl.OfferProcessorImpl:ForkJoinPool-3-worker-31)
[2017-04-07 15:48:36,753] INFO 192.168.1.70 - - [07/Apr/2017:15:48:36 +0000] "GET //192.168.1.68:7070/v2/queue HTTP/1.1" 200 32 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-498)
[2017-04-07 15:48:36,753] INFO 192.168.1.70 - - [07/Apr/2017:15:48:36 +0000] "GET //192.168.1.68:7070/v2/deployments HTTP/1.1" 200 228 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-513)
[2017-04-07 15:48:36,754] INFO 192.168.1.70 - - [07/Apr/2017:15:48:36 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv?embed=app.taskStats&embed=app.readiness HTTP/1.1" 200 891 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-226)
[2017-04-07 15:48:36,810] INFO 192.168.1.70 - - [07/Apr/2017:15:48:36 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions HTTP/1.1" 200 119 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-268)
[2017-04-07 15:48:36,811] INFO 192.168.1.70 - - [07/Apr/2017:15:48:36 +0000] "GET //192.168.1.68:7070/v2/apps//service-dev/mysql-serv/versions/2017-04-07T15:48:26.558Z HTTP/1.1" 200 472 "http://192.168.1.68:7070/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" (mesosphere.chaos.http.ChaosRequestLog:qtp731256267-459)
[2017-04-07 15:48:39,015] INFO Received status update for task service-dev_mysql-serv.a7956c66-1ba9-11e7-ab7a-0242ac120004: TASK_FAILED (Failed to run docker container: Port mappings are only supported for bridge and user-defined networks) (mesosphere.marathon.MarathonScheduler:Thread-409)
[2017-04-07 15:48:39,016] INFO all tasks of instance [service-dev_mysql-serv.marathon-a7956c66-1ba9-11e7-ab7a-0242ac120004] are terminal, requesting to expunge (mesosphere.marathon.core.instance.update.InstanceUpdater$:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:39,018] INFO Removed app [/service-dev/mysql-serv] from tracker (mesosphere.marathon.core.task.tracker.InstanceTracker$InstancesBySpec$:marathon-akka.actor.default-dispatcher-4)
[2017-04-07 15:48:39,018] INFO Increasing delay. Task launch delay for [/service-dev/mysql-serv] changed from [-728 milliseconds] to [1 seconds 150 milliseconds]. (mesosphere.marathon.core.launchqueue.impl.RateLimiter$:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:39,019] INFO receiveInstanceUpdate: instance [service-dev_mysql-serv.marathon-a7956c66-1ba9-11e7-ab7a-0242ac120004] was deleted (Failed) (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:39,019] ERROR New instance instance [service-dev_mysql-serv.marathon-a7956c66-1ba9-11e7-ab7a-0242ac120004] failed on agent Some(285ef74b-aaa3-44cf-b650-d687fc6e2923-S1) during app /service-dev/mysql-serv restart (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:39,019] INFO Reconciling instances during app /service-dev/mysql-serv restart: queuing 1 new instances (mesosphere.marathon.core.deployment.impl.TaskReplaceActor:marathon-akka.actor.default-dispatcher-16)
[2017-04-07 15:48:39,019] INFO initiating a scale check for runSpec [/service-dev/mysql-serv] due to [instance [service-dev_mysql-serv.marathon-a7956c66-1ba9-11e7-ab7a-0242ac120004]] Failed (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl:marathon-akka.actor.default-dispatcher-23)
[2017-04-07 15:48:39,019] INFO add 1 instances to 0 instances to launch (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-4)
[2017-04-07 15:48:39,019] INFO Acknowledge status update for task service-dev_mysql-serv.a7956c66-1ba9-11e7-ab7a-0242ac120004: TASK_FAILED (Failed to run docker container: Port mappings are only supported for bridge and user-defined networks) (mesosphere.marathon.core.task.update.impl.TaskStatusUpdateProcessorImpl:ForkJoinPool-3-worker-1)
[2017-04-07 15:48:40,187] INFO activating matcher ActorOfferMatcher(Actorakka://marathon/user/launchQueue/1/9-service-dev_mysql-serv#1478068544). (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-10)
[2017-04-07 15:48:40,187] INFO Received offers WANTED notification (mesosphere.marathon.core.flo
Mesos-slave still run in port ranges -e MESOS_resources='ports:[2000-9000,20000-34000]'. JSON define container port 3306 and host port is 31036. It's stay on ranges of port specific number. What's error meaning in: "Port mappings are only supported for bridge and user-defined networks"?
Why mesos-slave asked to kill a task for more than 2 hours on a reboot slave?
I'm running a mesos cluster with three masters and four slaves currently on the cloud environment.
Mesos Version: 0.28.
Marathon Version: 0.15.2
I found in case, if I reboot a slave which there are docker tasks running. After rebooting, the task will be in staging state on that slave for more then 2 hours. After 2 houes later, the Marathon can launch the task on other "slave.
If check the log, I could see it stuck with "Asked to kill task" and "Ignoring kill task" for about 2 hours.
Does anyone knows why the Mesos need trying to kill the dead task for more than 2 hours?
Log after reboot:
May 11 10:12:18 euca-10-254-234-236 mesos-slave[824]: I0511 10:12:18.199795 964 slave.cpp:1891] Asked to kill task project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000
May 11 10:12:18 euca-10-254-234-236 mesos-slave[824]: W0511 10:12:18.199831 964 slave.cpp:2018] Ignoring kill task project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b because the executor 'project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 is terminating/terminated
May 11 10:12:18 euca-10-254-234-236 mesos-slave[824]: I0511 10:12:18.199872 964 slave.cpp:1891] Asked to kill task docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000
log after 2 hours later:
I0511 12:15:48.200348 963 slave.cpp:1891] Asked to kill task project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000
W0511 12:15:48.200409 963 slave.cpp:2018] Ignoring kill task project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b because the executor 'project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 is terminating/terminated
I0511 12:15:48.200429 963 slave.cpp:1891] Asked to kill task docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000
W0511 12:15:48.200438 963 slave.cpp:2018] Ignoring kill task docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b because the executor 'docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 is terminating/terminated
I0511 12:15:51.485391 964 http.cpp:190] HTTP GET for /slave(1)/state from 10.145.150.124:59955 with User-Agent='Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
I0511 12:15:51.509351 965 http.cpp:190] HTTP GET for /slave(1)/state from 10.145.150.124:59955 with User-Agent='Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
W0511 12:15:51.656379 960 slave.cpp:4979] Failed to get resource statistics for executor 'project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000: Unknown container: b2f5b385-444b-4174-9a1c-8ccd2d3184dc
W0511 12:15:51.656409 960 slave.cpp:4979] Failed to get resource statistics for executor 'docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000: Unknown container: 1ab25a1b-79fe-430b-9751-330586a1fbef
I0511 12:15:51.663321 965 http.cpp:190] HTTP GET for /slave(1)/state from 10.145.150.124:59560 with User-Agent='Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
I0511 12:15:51.671294 965 http.cpp:190] HTTP GET for /slave(1)/state from 10.145.150.124:59560 with User-Agent='Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
W0511 12:15:52.156903 962 slave.cpp:4979] Failed to get resource statistics for executor 'project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000: Unknown container: b2f5b385-444b-4174-9a1c-8ccd2d3184dc
W0511 12:15:52.156941 962 slave.cpp:4979] Failed to get resource statistics for executor 'docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000: Unknown container: 1ab25a1b-79fe-430b-9751-330586a1fbef
E0511 12:15:52.247448 962 slave.cpp:3773] Container '1ab25a1b-79fe-430b-9751-330586a1fbef' for executor 'docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 failed to start: future discarded
E0511 12:15:52.247612 962 slave.cpp:3773] Container 'b2f5b385-444b-4174-9a1c-8ccd2d3184dc' for executor 'project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 failed to start: future discarded
W0511 12:15:52.247642 962 composing.cpp:541] Container '1ab25a1b-79fe-430b-9751-330586a1fbef' is already destroyed
W0511 12:15:52.247660 962 composing.cpp:541] Container 'b2f5b385-444b-4174-9a1c-8ccd2d3184dc' is already destroyed
E0511 12:15:52.247704 962 slave.cpp:3870] Termination of executor 'docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 failed: Unknown container: 1ab25a1b-79fe-430b-9751-330586a1fbef
I0511 12:15:52.248374 962 slave.cpp:3002] Handling status update TASK_FAILED (UUID: b399e8ce-832c-4b06-a15f-3c155536b872) for task docker-registry.d1c20255-173f-11e6-b66e-d00dacb0c46b of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 from #0.0.0.0:0
E0511 12:15:52.248458 962 slave.cpp:3870] Termination of executor 'project-hub_project-hub-backend.e764cc0d-173f-11e6-b66e-d00dacb0c46b' of framework 17cd3756-1d59-4dfc-984d-3fe09f6b5730-0000 failed: Unknown container: b2f5b385-444b-4174-9a1c-8ccd2d3184dc
I am trying to make a websocket connection from Strophe.js to Ejabberd but ejabberd is not able to make a websocket connection.
Here is the detail:
ejabberd version: 16.02
Web socket headers
General
Request URL:ws://localhost:5280/websocket
Request Method:GET
Status Code:101 Switching Protocols
Response Headers
Connection:Upgrade
Sec-WebSocket-Accept:BHzI4XS3kPDZJhNt4BVAaGDvv8I=
Sec-Websocket-Protocol:xmpp
Upgrade:websocket
Request Headers
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:no-cache
Connection:Upgrade
Host:localhost:5280
Origin:http://localhost
Pragma:no-cache
Sec-WebSocket-Extensions:permessage-deflate; client_max_window_bits
Sec-WebSocket-Key:eErMAKwTSckMghBGWY3KtQ==
Sec-WebSocket-Protocol:xmpp
Sec-WebSocket-Version:13
Upgrade:websocket
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Ejabberd websocket configuration
port: 5280
module: ejabberd_http
request_handlers:
"/websocket": ejabberd_http_ws
## "/pub/archive": mod_http_fileserver
web_admin: true
http_bind: true
## register: true
captcha: false
ejabberd logs
2016-03-22 15:32:46.765 [info] <0.438.0>#ejabberd_listener:accept:333 (#Port<0.16207>) Accepted connection 127.0.0.1:57946 -> 127.0.0.1:5280
2016-03-22 15:32:46.766 [info] <0.1283.0>#ejabberd_http:init:158 started: {gen_tcp,#Port<0.16207>}
2016-03-22 15:32:46.777 [error] <0.1286.0>#ejabberd_hooks:run_fold1:368 {{badmatch,[<<"localhost">>]},[{mod_pubsub,serverhost,1,[{file,"src/mod_pubsub.erl"},{line,4012}]},{mod_pubsub,plugins,1,[{file,"src/mod_pubsub.erl"},{line,4038}]},{mod_pubsub,features,2,[{file,"src/mod_pubsub.erl"},{line,4116}]},{mod_pubsub,disco_local_features,5,[{file,"src/mod_pubsub.erl"},{line,512}]},{ejabberd_hooks,safe_apply,3,[{file,"src/ejabberd_hooks.erl"},{line,382}]},{ejabberd_hooks,run_fold1,4,[{file,"src/ejabberd_hooks.erl"},{line,365}]},{mod_caps,make_my_disco_hash,1,[{file,"src/mod_caps.erl"},{line,515}]},{mod_caps,caps_stream_features,2,[{file,"src/mod_caps.erl"},{line,188}]}]}
running hook: {disco_local_features,[{jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>},{jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>},<<>>,<<>>]}
Please help me out to understand the ejabberd log and how can I make the successful websocket connection from client to ejabberd.
It seems this is not related to Websockets, but you seem to have hit a bug that is already fix in ejabberd repository and that will be packaged as 16.03.
If you update to latest HEAD from ejabberd repository, it should solve your pubsub crash issue.
You can find it here: https://github.com/processone/ejabberd
I am trying to connect to Intersystems Cache CSP Websocket with the following line from the client:
var ws = new WebSocket("ws://" + window.location.host + "/path/ClassName.cls");
And I'm getting this error in chrome console:
WebSocket connection to 'ws://<server address>' failed: Error during WebSocket handshake: 'Connection' header is missing
The request headers:
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8,he;q=0.6
Cache-Control:no-cache
Connection:Upgrade
Cookie:CSPWSERVERID=Cache for Windows (x86-64) 2013.1.1 (Build 501_1_13062) Sun Aug 11 2013 23:20:33 EDT
Host:<host-ip>
Origin:http://<host-ip>
Pragma:no-cache
Sec-WebSocket-Extensions:permessage-deflate; client_max_window_bits
Sec-WebSocket-Key:<the key>
Sec-WebSocket-Version:13
Upgrade:websocket
User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
And the response headers:
Date:Tue, 04 Aug 2015 11:45:54 GMT
Sec-WebSocket-Accept:<some key>
Sec-WebSocket-Protocol:chat
Server:Microsoft-IIS/7.5
Transfer-Encoding:chunked
Upgrade:websocket
X-Powered-By:ASP.NET
The server is using IIS 7.5, which technically doesn't support WebSocket, but it seems that all the websocket headers are indeed in place, except one: "Connection":"Upgrade".
Is there any way to inject the "connection" header into the handshake response? Is there some client configuration that can disregard the missing header?
Thanks.
Unfortunately for WebSockets support you need IIS 8 + Windows 8 and Windows Server 2012
http://docs.intersystems.com/cache20152/csp/docbook/DocBook.UI.Page.cls?KEY=GCGI_oper_config#GCGI_websockets