I'm using WAS ND and want to have dmgr profile with federated managed profile app.
I am creating cluster using:
AdminTask.createCluster('[-clusterConfig [-clusterName %s -preferLocal true]]' % nameOfModulesCluster)
Next, I'm configuring my WAS instance, queues, datasources, jdbc, JMS Activation Specs, factories etc.
By the time I want to create cluster member, I'm displaying:
print("QUEUES: \n" + AdminTask.listSIBJMSQueues(AdminConfig.getid('/ServerCluster:ModulesCluster/')))
print("JMS AS: \n" + AdminTask.listSIBJMSActivationSpecs(AdminConfig.getid('/ServerCluster:ModulesCluster/')))
And it returns all queues I've created earlier. But when I'm calling
AdminTask.createClusterMember('[-clusterName %(cluster)s -memberConfig [-memberNode %(node)s -memberName %(server)s -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateName default -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope cluster]]' % {'cluster': nameOfCluster, 'node': nameOfNode, 'server': nameOfServer})
AdminConfig.save()
configuration displayed earlier is... gone. Some configuration (like datasources) is still displayed in ibm/console, but queues and jms as are not. The same print is displaying nothing, but member is added to cluster.
I can't find any information using google. I've tried AdminNodeManagement.syncActiveNodes(), but it won't work since I'm using
/opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -lang jython -conntype NONE -f global.py
and AdminControl is not available.
What should I do in order to keep my configuration created before clustering? Do I have to sync it somehow?
This is the default behavior and is due to the -resourcesScope attribute in the createClusterMember command. This attribute determines how the server resources are promoted in the cluster, while adding the first cluster member.
Valid options for resourcesScope are :
Cluster: moves the resources of the first cluster member to the cluster level. The resources of the first cluster member replace the resources of the cluster. (is the default option)
Server: maintains the server resources at the new cluster member level. The cluster resources remain unchanged.
Both: copies the resources of the cluster member (server) to the cluster level. The resources of the first cluster member replace the resources of the cluster. The same resources exist at both the cluster and cluster member scopes.
Since you have set "-resourcesScope cluster" in your createClusterMember command, all configuration created at cluster scope are being deleted/replaced by the empty configurations of the new cluster member.
So, for your configurations to work, set "-resourcesScope server", such that the cluster configurations are not replaced by the cluster member configurations.
AdminTask.createClusterMember('[-clusterName %(cluster)s -memberConfig [-memberNode %(node)s -memberName %(server)s -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateName default -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope server]]' % {'cluster': nameOfCluster, 'node': nameOfNode, 'server': nameOfServer})
AdminConfig.save()
Refer "Select how the server resources are promoted in the cluster" section in https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/urun_rwlm_cluster_create2_v61.html for more details.
Related
Kafka Connect converters provide the feature of dead letter queue (DLQ) that can be configured (errors.deadletterqueue.topic.name) to store failing records. I tried configuring it on a MirrorMaker2 setup but it doesn't seem to be working as expected. My expectation is that messages that failed to replicate to target cluster are stored in the dead letter queue topic.
To test this, I simulated failures by bringing down the target cluster and expected MirrorMaker2 to create a DLQ on source cluster with failed message but didn't see the dead letter queue topic created. The Kafka documentation is not very clear on whether this configuration option works for MirrorMaker2.
Below is the configuration I used:
clusters = sourceKafkaCluster,targetKafkaCluster
sourceKafkaCluster.bootstrap.servers = xxx
targetKafkaCluster.bootstrap.servers = yyy
sourceKafkaCluster->targetKafkaCluster.enabled = true
targetKafkaCluster->sourceKafkaCluster.enabled = false
#Not sure which one of the below ones are correct.
sourceKafkaCluster->targetKafkaCluster.errors.deadletterqueue.topic.name=dlq_topic_1
sourceKafkaCluster->targetKafkaCluster.errors.deadletterqueue.topic.replication.factor=1
errors.deadletterqueue.topic.name=dlq_topic_1
errors.deadletterqueue.topic.replication.factor=1
Does the deadletterqueue configuration option work with MirrorMaker2?
This is my kafka connector properties
##
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##
# This file contains some of the configurations for the Kafka Connect distributed worker. This file is intended
# to be used with the examples, and some settings may differ from those used in a production system, especially
# the `bootstrap.servers` and those specifying replication factors.
# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
bootstrap.servers=localhost:9092
# unique name for the cluster, used in forming the Connect cluster group. Note that this must not conflict with consumer group IDs
group.id=connect-cluster
# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
# it to
key.converter.schemas.enable=false
value.converter.schemas.enable=false
# Topic to use for storing offsets. This topic should have many partitions and be replicated and compacted.
# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
# the topic before starting Kafka Connect if a specific topic configuration is needed.
# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
offset.storage.topic=__connect_offsets
offset.storage.replication.factor=1
#offset.storage.partitions=25
# Topic to use for storing connector and task configurations; note that this should be a single partition, highly replicated,
# and compacted topic. Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
# the topic before starting Kafka Connect if a specific topic configuration is needed.
# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
config.storage.topic=__connect_configs
config.storage.replication.factor=1
# Topic to use for storing statuses. This topic can have multiple partitions and should be replicated and compacted.
# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
# the topic before starting Kafka Connect if a specific topic configuration is needed.
# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
status.storage.topic=__connect_status
status.storage.replication.factor=1
#status.storage.partitions=5
# Flush much faster than normal, which is useful for testing/debugging
#offset.flush.interval.ms=10000
# These are provided to inform the user about the presence of the REST host and port configs
# Hostname & Port for the REST API to listen on. If this is set, it will bind to the interface used to listen to requests.
#rest.host.name=
#rest.port=8083
# The Hostname & Port that will be given out to other workers to connect to i.e. URLs that are routable from other servers.
#rest.advertised.host.name=
#rest.advertised.port=
# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins
# (connectors, converters, transformations). The list should consist of top level directories that include
# any combination of:
# a) directories immediately containing jars with plugins and their dependencies
# b) uber-jars with plugins and their dependencies
# c) directories immediately containing the package directory structure of classes of plugins and their dependencies
# Examples:
# plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors,
plugin.path=/usr/share/java
And this is the POST body I use for creating the Elasticsearch sink
{
"name" : "test-distributed-connector",
"config" : {
"connector.class" : "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max" : "2",
"topics.regex" : "^test[0-9A-Za-z-_]*(?<!-raw$)$",
"connection.url" : "http://elasticsearch:9200",
"connection.username": "admin",
"connection.password": "admin",
"type.name" : "_doc",
"key.ignore" : "true",
"schema.ignore" : "true",
"transforms": "TimestampRouter",
"transforms.TimestampRouter.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.TimestampRouter.topic.format": "${topic}-${timestamp}",
"transforms.TimestampRouter.timestamp.format": "YYYY.MM.dd",
"batch.size": "100",
"offset.flush.interval.ms":"60000",
"offset.flush.timeout.ms": "15000",
"read.timeout.ms": "15000",
"connection.timeout.ms": "10000",
"max.buffered.records": "1500"
}
}
The issue I met is that sometimes this sink would work and send data to Elasticsearch and showing
[2020-09-15 20:27:05,904] INFO WorkerSinkTask{id=test-distributed-connector-0} Committing offsets asynchronously using sequence number 1.......
But most of the time it would just stuck and repeat this part
[2020-09-15 20:24:29,458] INFO [Consumer clientId=consumer-4, groupId=connect-test-distributed-connector] Group coordinator kafka:9092 (id: 2147483543 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:706)
[2020-09-15 20:24:29,560] INFO [Consumer clientId=consumer-4, groupId=connect-test-distributed-connector] Discovered group coordinator kafka:9092 (id: 2147483543 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:654)
[2020-09-15 20:24:29,561] INFO [Consumer clientId=consumer-4, groupId=connect-test-distributed-connector] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:486)
One of the doubt I have is that this Elasticsearch sink would read lot's of topic with quite amount of message/data.
So that it has issue when trying to read the topic from Kafka
Because I have another Elasticsearh sink with basically identical setting with this one and that one works.
Is there any method/tweak can be done to make this Elasticseach work?
######## Update#########
Also sometimes(quite frequently) I'll see this log
[2020-09-16 09:51:18,189] WARN [Consumer clientId=consumer-6, groupId=connect-test-distributed-connector] Close timed out with 1 pending requests to coordinator, terminating client connections (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:769)
and
[2020-09-16 10:17:43,369] WARN [Consumer clientId=consumer-16, groupId=connect-test-distributed-connector-] Close timed out with 1 pending requests to coordinator, terminating client connections (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:769)
You are using a secure connection, I assume your xpack is enabled.
try changing
"connection.url" : "http://elasticsearch:9200"
to
"connection.url" : "https://elasticsearch:9200"
To all those SymmetricDS nerds over there, this one's for you all.
Right, so we have a main db, DB-01. We have 3 instances of our application running namely R1,R2,R3. Each instance has its own in-memory db namely D1,D2,D3 which it(application) is accessing respectively. We are using SymmetricDS to do one-way sync from DB-01 to D1,D2,D3. So, there is a server node, corporate C0, pointing to DB-01 and 3 client nodes, stores S1,S2,S3 pointing to D1,D2,D3 respectively.
All is working fine.
But now, we would like to introduce High Availability and there by FAILOVER into this topology i.e., at any time there will be 2 server nodes running, say Master and Slave, that would be accessing the same DB-01. If Master server goes down, clients should automatically connect to the Slave node and continue operation.
What all might be the configuration changes required to accomplish this? Are there any examples or documentations that i can reproduce to understand this concept?
We do this via clustering with 2 SymmetricDS services running on 2 app servers pointing to the High Availability (HA) connections. Then all you need is HA connections to failover like normal and Symmetric DS clustering does the rest.
Link for the user manual on clustering.
https://www.symmetricds.org/doc/3.13/html/user-guide.html#_clustering
EDIT let me get some configs for you on here service 1:
engine.name=<SDS_SERVICE_1>
db.driver=net.sourceforge.jtds.jdbc.Driver
db.url=jdbc:jtds:sqlserver://<HA_connection1>:1433/<DB>;useCursors=true;bufferMaxMemory=10240;lobBuffer=5242880
db.user=***********
db.password=***********
registration.url=http://<IP>:7004/sync/<SDS_MAIN>
sync.url=http://<IP>:7004/sync/<SDS_SERVICE_1>
group.id=<GID>
external.id=100
auto.registration=true
initial.load.create.first=true
sync.table.prefix=sym
start.initial.load.extract.job=false
cluster.lock.enabled=true
cluster.server.id=11
cluster.lock.timeout.ms=600000
cluster.lock.refresh.ms=60000
compression.level=-1
compression.strategy=0
Service 2:
engine.name=<SDS_SERVICE_2>
db.driver=net.sourceforge.jtds.jdbc.Driver
db.url=jdbc:jtds:sqlserver://<HA_connection2>:1433/<DB>;useCursors=true;bufferMaxMemory=10240;lobBuffer=5242880
db.user=***********
db.password=***********
registration.url=http://<IP>:7004/sync/<SDS_MAIN>
sync.url=http://<IP>:7004/sync/<SDS_SERVICE_2>
group.id=<GID>
external.id=100
auto.registration=true
initial.load.create.first=true
sync.table.prefix=sym
start.initial.load.extract.job=false
cluster.lock.enabled=true
cluster.server.id=12
cluster.lock.timeout.ms=600000
cluster.lock.refresh.ms=60000
compression.level=-1
compression.strategy=0
I have launched 3 Amazon EC2 instance and setup datastax cassandra as follows
1.Region - US EAST:
cassandra.yaml - configuration
a.listen_address as private IP of this instance
b.broadcast_address as public IP of this instance
c.seeds as 50.XX.XX.X1, 50.XX.XX.X2 (public-ip of node1,public-ip of node2)
cassandra-rackdc.properties - configuration
dc=DC1
rack=RAC1
dc_suffix=US_EAST_1
2.Region - US WEST:
I did same procedure as I did above.
3.Region - EU IRELAND:
The result of above configuration is
All the node working good individually. But when I do
$nodetool status on all the three node
It only listing the local node only.
I tried to achieve the following things.
1. Launch 3 cassandra node in three different region. For say, US-EAST,US-WEST,EU-IRELAND.
With Following configuration or methodology
a.Ec2MultiRegionSnitch
b.Replication staragey as SimpleStrategy
c.Replication Factor as 3
d. Read & write level as QUORUM.
I wish to attain only one thing i.e. if any two of the region is down or any two of the node down, I can survive with renaming one node.
My Questions here are
Where I did the mistake? and How to attain my requirements?
Any help or inputs are much appreciated.
Thanks.
This is what worked for me with cassandra 3.0
endpoint_snitch: Ec2MultiRegionSnitch
listen_address: <leave_blank>
broadcast_address: <public_ip_of_server>
rpc_address: 0.0.0.0
broadcast_rpc_address: <public_ip_of_server>
-seed: "one_ip_from_other_DC"
Finally, I found the resolution of my issue. I am using replication strategy as SimpleStrategy, hence I do not require to configure cassandra-rackdc.properties.
Once, I removed the file cassandra-rackdc.properties from all node, Everything working as expected.
Thanks
i have installed cassandra on two individual nodes both on Amazon.when i am trying to configure nodes to form a cluster the nodes. I am receiving the following error.
ERROR [main] 2016-05-12 11:01:26,402 CassandraDaemon.java:381 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 1 to 256.
I using these setting in cassandra.yaml file
listen_address and rpc_address to : private Ip address
seeds : Public Ip [Elastic Ip address]
num_tokens: 256
This message usually appears when num_tokens is changed after the node has been bootstrapped.
The solution is:
Stop Cassandra on all nodes
Delete the data directory (inc. datafiles, commitlog and saved_caches)
Double check that num_tokens is set to 256, initial_token is commented out and auto_bootstrap is set to true in cassandra.yaml
Start Cassandra on all nodes
This will wipe your existing cluster and cause the nodes to bootstrap from scratch again.
Cassandra doesn't support changing between vnodes and static tokens after a datacenter is bootstrapped. If you need to change from vnodes to static tokens or vice versa in an already running cluster, you'll need to create a second datacenter using the new configuration, stream your data across, and then decomission the original nodes.