I'm running Spring XD in distributed mode. But when I bring up a container the admin node sees it and only deploys some and not all streams. Here is the admin log
:11:41+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ContainerListener - Container arrived: Container{name='26c5cbfa-7f20-455c-a36c-580c98a8126--More--(76%)
...skipping 1 line
2016-10-03T19:11:41+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ContainerListener - Scheduling deployments to new container(s) in 15000 ms
2016-10-03T19:11:57+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-http-receiver': DeploymentStatus{state=incomplete}
2016-10-03T19:11:57+0000 1.3.0.RELEASE WARN DeploymentSupervisor-0 zk.ModuleRedeployer - No containers available for redeployment of http for stream cameras-http-receiver
2016-10-03T19:11:57+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-http-receiver': DeploymentStatus{state=incomplete}
2016-10-03T19:11:59+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-http-receiver-2': DeploymentStatus{state=deployed}
2016-10-03T19:12:00+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-http-receiver-3': DeploymentStatus{state=deployed}
2016-10-03T19:12:01+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-processor': DeploymentStatus{state=deployed}
2016-10-03T19:12:02+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-to-mongo': DeploymentStatus{state=incomplete}
...skipping 1 line
2016-10-03T19:12:02+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'cameras-to-mongo': DeploymentStatus{state=incomplete}
2016-10-03T19:12:03+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'mongo-sink': DeploymentStatus{state=incomplete}
2016-10-03T19:12:03+0000 1.3.0.RELEASE WARN DeploymentSupervisor-0 zk.ModuleRedeployer - No containers available for redeployment of mongodb for stream mongo-sink
2016-10-03T19:12:03+0000 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ModuleRedeployer - Deployment state for stream 'mongo-sink': DeploymentStatus{state=incomplete}
Why is it saying there's no containers available right after it recognizes the container?
Do you have any container matching criteria for the modules? It looks like one of the module instances fails to get deployed while the ModuleRedeployer is looking for some other container to deploy that module.
Related
I am trying to run Terracotta server 10.11 from https://www.terracotta.org/downloads/ to connect it with Ehcache in my Spring boot application. But the problem is that when i run Terracotta server instance using the server\bin\start-tc-server.bat it does not give any error but put the server in DIAGNOSTIC state as shown in logs below.
2022-11-24 14:22:01,660 INFO - Terracotta 5.8.5, as of 2021-12-16 at 22:21:34 UTC (Revision 3695ab2f870d94491c564e87c266555a7d1c096b from UNKNOWN)
2022-11-24 14:22:01,660 INFO - Extensions:
2022-11-24 14:22:01,679 INFO - PID is 23344
2022-11-24 14:22:01,965 INFO - Did not find configuration directory at: C:\Users\user\terracotta\config
2022-11-24 14:22:01,965 INFO - Starting node from config file: C:\Users\user\Downloads\ehcache-clustered-3.10.0-kit\ehcache-clustered-3.10.0-kit\server\conf\cluster.cfg
2022-11-24 14:22:02,266 INFO - Found only one node information in config file: C:\Users\user\Downloads\ehcache-clustered-3.10.0-kit\ehcache-clustered-3.10.0-kit\server\conf\cluster.cfg
2022-11-24 14:22:02,267 INFO - Starting unconfigured node: default-node
2022-11-24 14:22:02,271 INFO - Bootstrapped nomad system with root: C:\Users\user\terracotta\config
2022-11-24 14:22:02,277 INFO - Startup configuration of the node:
client-lease-duration=150s
client-reconnect-window=120s
cluster-name=clustered
failover-priority=availability
offheap-resources=main\:512MB
stripe.1.node.1.bind-address=0.0.0.0
stripe.1.node.1.group-bind-address=0.0.0.0
stripe.1.node.1.group-port=9430
stripe.1.node.1.hostname=localhost
stripe.1.node.1.log-dir=%H/terracotta/logs
stripe.1.node.1.name=default-node
stripe.1.node.1.port=9410
stripe.1.stripe-name=default-stripe
2022-11-24 14:22:02,280 INFO - Logging directory is not set. Logging only to the console
2022-11-24 14:22:02,293 INFO - Available Max Runtime Memory: 1820MB
2022-11-24 14:22:02,314 INFO - Creating server nodeID: NodeID[localhost:9410]
2022-11-24 14:22:02,546 INFO - Initializing LeaseServiceProvider with default lease length of 150000 ms
2022-11-24 14:22:02,548 INFO - Initializing org.terracotta.lease.service.LeaseServiceProvider#4cf92ef3
2022-11-24 14:22:02,549 INFO - Initializing org.terracotta.client.message.tracker.OOOMessageHandlerProvider#40f5b3f9
2022-11-24 14:22:02,556 INFO - Registered MBean with name: DiagnosticRequestHandler
2022-11-24 14:22:02,557 INFO - Registered Diagnostic Service: org.terracotta.nomad.server.NomadServer
2022-11-24 14:22:02,557 INFO - Registered Diagnostic Service: org.terracotta.dynamic_config.api.service.DynamicConfigService
2022-11-24 14:22:02,558 INFO - Registered Diagnostic Service: org.terracotta.dynamic_config.api.service.TopologyService
2022-11-24 14:22:02,558 INFO - Initializing org.terracotta.diagnostic.server.DiagnosticServiceProvider#1bf35727
2022-11-24 14:22:02,561 INFO - Initializing org.terracotta.diagnostic.server.extensions.DiagnosticExtensionsServiceProvider#410ee45a
2022-11-24 14:22:02,804 INFO - Initializing org.terracotta.management.service.monitoring.MonitoringServiceProvider#65c7455b
2022-11-24 14:22:02,804 INFO - Initializing org.terracotta.platform.ServerInfoProvider#240d561b
2022-11-24 14:22:02,806 INFO - Registered dynamic configuration change handler for setting client-reconnect-window: org.terracotta.dynamic_config.server.service.handler.ClientReconnectWindowConfigChangeHandler#74d20602
2022-11-24 14:22:02,809 INFO - Registered dynamic configuration change handler for setting log-dir: org.terracotta.dynamic_config.server.service.handler.NodeLogDirChangeHandler#67c6fc00
2022-11-24 14:22:02,810 INFO - Registered dynamic configuration change handler for setting failover-priority: ConfigChangeHandler#accept()
2022-11-24 14:22:02,810 INFO - Registered dynamic configuration change handler for setting public-hostname: ConfigChangeHandler#accept()
2022-11-24 14:22:02,811 INFO - Registered dynamic configuration change handler for setting public-port: ConfigChangeHandler#accept()
2022-11-24 14:22:02,811 INFO - Registered dynamic configuration change handler for setting cluster-name: ConfigChangeHandler#accept()
2022-11-24 14:22:02,812 INFO - Registered dynamic configuration change handler for setting lock-context: ConfigChangeHandler#accept()
2022-11-24 14:22:02,812 INFO - Registered dynamic configuration change handler for setting logger-overrides: org.terracotta.dynamic_config.server.service.handler.LoggerOverrideConfigChangeHandler#3ba87843
2022-11-24 14:22:02,813 INFO - Registered dynamic configuration change handler for setting tc-properties: org.terracotta.dynamic_config.server.api.SelectingConfigChangeHandler#16df9889
2022-11-24 14:22:02,815 INFO - Initializing org.terracotta.dynamic_config.server.service.DynamicConfigServiceProvider#29ca0612
2022-11-24 14:22:02,815 INFO - Registering implementation-provided service com.tc.services.PlatformServiceProvider#16b645b2
2022-11-24 14:22:02,816 INFO - Registering implementation-provided service com.tc.services.EntityMessengerProvider#3c352805
2022-11-24 14:22:02,816 INFO - Initializing com.tc.objectserver.persistence.NullPlatformStorageServiceProvider#149f57c4
2022-11-24 14:22:02,818 INFO - Registering implementation-provided service com.tc.services.LocalMonitoringProducer#5baa3715
2022-11-24 14:22:02,830 INFO - Creating 4 worker comm threads for default-node - L2_L1
2022-11-24 14:22:02,910 INFO - Registering implementation-provided service com.tc.services.CommunicatorService#7d51aa32
2022-11-24 14:22:02,920 INFO - HealthChecker Started
2022-11-24 14:22:02,952 INFO - Started the server in diagnostic mode
2022-11-24 14:22:02,967 INFO - Server started as default-node
2022-11-24 14:22:02,959 INFO - Terracotta Server instance has started diagnostic listening on all interfaces (address:/0.0.0.0 port:9410)
2022-11-24 14:22:03,177 INFO - Moved to State[ DIAGNOSTIC ]
According to the documentation it should be in ACTIVE state to be running properly. Still i tried to make connection with the server from my Spring boot application but it was also unable to reach it and gave TimeoutException.
I am using the following command to run the server instance:
./start-tc-server.bat -f C:\Users\user\Downloads\ehcache-clustered-3.10.0-kit\ehcache-clustered-3.10.0-kit\server\conf\cluster.cfg
Does anyone have any clue why its not getting to ACTIVE state ? maybe try to run it on your end and see if the server gets to ACTIVE state. Or is there anything i am missing ?
Thanks in Advance.
P.S I tried running older version of Terracotta server from the same downloads page and it easily goes to active state but i cannot use old version since it is not compaitable with Ehcache 3.x
You need to activate the server.
In the kit that you downloaded navigate to /tools/bin and you need to run config-tool.bat activate -f ../../server/conf/cluster.cfg
This will create a folder C:\Users\{user}\terracotta that will contain the configs and logs for the terracotta server, so the next time you start it will use the configs in the folder and automatically go to activate state.
If you need to change configs delete the folder, restart terracotta and activate it again using the config tool.
When I tried to connect to Nifi UI using http://localhost:8080/nifi, i am getting below error
org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:331)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:299)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:398)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2020-02-27 11:51:11,834 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2020-02-27 11:51:11,836 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector#355ee205{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2020-02-27 11:51:11,837 INFO [Thread-1] org.eclipse.jetty.server.session node 0 Stopped scavenging
Can anyone suggest what is the cause of this issue?
Nifi version- 1.9.2,installed on windows machine
Here is the nifi status logs,
12:33:16.886 [main] DEBUG org.apache.nifi.bootstrap.NotificationServiceManager - Found 0 service elements
12:33:16.896 [main] INFO org.apache.nifi.bootstrap.NotificationServiceManager - Successfully loaded the following 0 services: []
12:33:16.897 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STARTED
12:33:16.897 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STOPPED
12:33:16.898 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_DIED
12:33:16.899 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File:
12:33:16.900 [main] DEBUG org.apache.nifi.bootstrap.Command - Properties: {pid=9724}
Failed to determine if Process 9724 is running; assuming that it is not
12:33:16.902 [main] INFO org.apache.nifi.bootstrap.Command - Apache NiFi is not running
The port use by nifi is already used by another process.
you can change web server port in conf/nifi.properties
I have created a spring boot application to publish the message to the Kafka queue. For that, I am using spring cloud stream and Kafka binder as dependencies. Problem is my application is continuously trying to connect to Kafka broker if it is down for 2 minutes because of the default configuration.
I have reduced that time using the below property and set it to 1000 ms and getting the timeout exception
spring.kafka.properties.request.timeout.ms:1000.
But still, my spring application is running after the exception. I want it to fail if Kafka broker is not available to connect to. I have tried one more property for that spring.kafka.admin.fail-fast=true but still, the application is running.
I have also tried to search for some properties of spring cloud stream and Kafka binder that I can set to fail my application if Kafka broker is not available but couldn't find anything related to that.
Please, help me with this.
Please see below for the log of exception.
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopicAndPartitions(KafkaTopicProvisioner.java:351)
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopicIfNecessary(KafkaTopicProvisioner.java:325)
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopic(KafkaTopicProvisioner.java:302)
... 33 common frames omitted
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Successfully started bean 'outputBindingLifecycle'
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Starting beans in phase 2147482647
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.binding.BindableProxyFactory - Binding inputs for :interface kafka.stream.RXXXStreams
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Successfully started bean 'inputBindingLifecycle'
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Starting beans in phase 2147483547
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Successfully started bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'
2019-05-22 06:06:25 [main] DEBUG o.s.b.a.l.ConditionEvaluationReportLoggingListener -
Do you have spring-boot-web libraries as dependency? If that's the case, your application will not exit. A full log will be also very helpful.
I'm trying to crate a stream that should follow #BBCBreaking (what should have 5402612 twitter ID), but I keep getting following Http error:
2016-03-28T02:13:12+0200 1.3.1.RELEASE INFO DeploymentSupervisor-0 zk.ZKStreamDeploymentHandler - Deployment status for stream 'mystream': DeploymentStatus{state=deployed}
2016-03-28T02:13:13+0200 1.3.1.RELEASE WARN twitterSource-1-1 twitter.TwitterStreamChannelAdapter - Http error, waiting for 5 seconds before restarting
2016-03-28T02:13:19+0200 1.3.1.RELEASE WARN twitterSource-1-1 twitter.TwitterStreamChannelAdapter - Http error, waiting for 10 seconds before restarting
2016-03-28T02:13:30+0200 1.3.1.RELEASE WARN twitterSource-1-1 twitter.TwitterStreamChannelAdapter - Http error, waiting for 20 seconds before restarting
my stream command is:
stream create --name mystream --definition "twitterstream --follow='5402612' | log" --deploy
running on SpringXD: 1.3.1.RELEASE
please, any idea that why the error?
You can debug such situations by enabling DEBUG logging - log config is in the xd/config folder in .groovy files; e.g. xd-singlenode-logback.groovy.
Set the loggers for org.springframework.integration and org.springframework.xd, org.springframework.xd.dirt.server to DEBUG and add a logger for org.springframework.social.twitter also at DEBUG.
Or you can set all of org.springframework and comment out the more specific ones.
I am using teamcity server 8.0.3 (build 27540) and have set up several git projects. I am using Amazon EC2 images which are set up as cloud agents.
My setup starts up the agents and runs the tests on them when a new job is put to the queue.
I got it working fine for weeks and have not touched the server, and suddenly it stopped working.
In the cloud tabs I now get "Image error" with the message: "Image does not exist in Amazon EC2". Even though the image exists there!
If I open and save or re-create my cloud agent settings then one of the agents will be awoken by teamcity and do builds just fine, but the warning is still there and when the agent shuts down after my time delay I have to re-save the cloud agent settings to force a start on the agent.
I have tried restarting Teamcity with no avail.
This is the errors from teamcity-cloud.log
[2013-12-12 06:11:00,850] WARN [rome'{id=cp3} 1] - .clouds.amazon.image.ImageBase - Failed to fetch data of image AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31} from EC2. Image does not exist in Amazon EC2
[2013-12-12 06:11:00,958] INFO [ue; Flush Queue] - .instances.StartInstanceAction - Starting cloud instance: profile 'EC2 TC Win Chrome'{id=cp3}, AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31}, hash=EZObYYzWwxDOZ4o9svYSvGbdLqf5e7KQ, reason=Automatic start: Compatible with queued build Booking Log Client :: Win Chrome(promotionId=12008)
[2013-12-12 06:11:01,565] INFO [ue; Flush Queue] - .server.impl.CloudEventsLogger - Cloud instance start succeeded: profile 'EC2 TC Win Chrome'{id=cp3}, Amazon Instance{instanceId=i-d123cd29, imageId=i-d123cd29, amazonImageId=ami-463edb31, status: Scheduled to start}
[2013-12-12 06:11:03,135] INFO [rome'{id=cp3} 1] - r.impl.DBCloudStateManagerImpl - Image: i-d123cd29, Instance: i-d123cd29 is marked with state: running.
[2013-12-12 06:12:18,441] WARN [uled executor 4] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Running: Amazon Instance{instanceId=i-d123cd29, imageId=i-d123cd29, amazonImageId=ami-463edb31, status: Running}, profile 'EC2 TC Win Chrome'{id=cp3}
[2013-12-12 06:12:20,759] WARN [rome'{id=cp3} 1] - .clouds.amazon.image.ImageBase - Failed to fetch data of image AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31} from EC2. Image does not exist in Amazon EC2
[2013-12-12 06:13:04,668] INFO [nio-8111-exec-9] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31}, profile: profile 'EC2 TC Win Chrome'{id=cp3} was marked to CONTAIN agent
[2013-12-12 06:13:04,672] INFO [nio-8111-exec-9] - .server.impl.CloudEventsLogger - Detected cloud agent EC2-i-d123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}, profile 'EC2 TC Win Chrome'{id=cp3}, Amazon Instance{instanceId=i-d123cd29, imageId=i-d123cd29, amazonImageId=ami-463edb31, status: Running}
and from teamcity-server.log
[2013-12-12 06:13:04,648] WARN - jetbrains.buildServer.AGENT - Agent EC2-i-dd123cd29 is unauthorized with comment: Cloud instance (id=i-dd123cd29, imageId=i-dd123cd29, profileId=cp3) has gone (is not reported by cloud profile)
[2013-12-12 06:13:04,661] INFO - jetbrains.buildServer.SERVER - Agent has been registered: EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}, not running a build
[2013-12-12 06:13:04,690] INFO - tbrains.buildServer.ACTIVITIES - Agent "EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}" was authorised with comment "Virtual agent is authorized automatically."
[2013-12-12 06:13:05,022] INFO - tbrains.buildServer.ACTIVITIES - Build started; Booking Log Client :: Win Chrome {id=booking_log_client, internal id=bt18} #34 {build id=6213, promotion id=12008, branch=<default>, history=false, agent=EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}, triggered by "Nisse Nilsson" (##userId='10'). Started Thu Dec 12 06:13:04 UTC 2013}
[2013-12-12 06:13:57,586] INFO - jetbrains.buildServer.SERVER - Agent description updated for agent: EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}
[2013-12-12 06:23:54,323] INFO - jetbrains.buildServer.SERVER - Agent description updated for agent: EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}
The error appears due to:
The AMI 'ami-463edb31' is no longer available in EC2's 'EU (Ireland)' region.
source:https://aws.amazon.com/amis/amazon-public-images-basic-microsoft-windows-server-2008-64-bit