I have a requirement for a small kafka setup for testing.
I'm new to kafka and I can't find a way to connect to a kafka that I setup in docker container:
I tried to use https://hub.docker.com/r/spotify/kafka/ which is a container that contains zookeeper and kafka, but I don't know what is the ip of the kafka:
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
logs:
Digest: sha256:cf8f8f760b48a07fb99df24fab8201ec8b647634751e842b67103a25a388981b
Status: Downloaded newer image for spotify/kafka:latest
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-07-12 19:45:07,018 CRIT Supervisor running as root (no user in config file)
2020-07-12 19:45:07,019 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-07-12 19:45:07,019 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-07-12 19:45:07,027 INFO RPC interface 'supervisor' initialized
2020-07-12 19:45:07,027 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-07-12 19:45:07,027 INFO supervisord started with pid 1
2020-07-12 19:45:08,031 INFO spawned: 'zookeeper' with pid 8
2020-07-12 19:45:08,034 INFO spawned: 'kafka' with pid 9
2020-07-12 19:45:09,081 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-07-12 19:45:09,082 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
I'm using a kafka-ui to connect to it. I'm trying to use:
zookeeper: localhost:2181 <--- working
kafka: localhost:9092 <--- ERROR: "The broker [localhost:9092] is reachable
but Kafka can't connect. ensure you have access to the *advertised listeners*
of the cluster and the proper authorizations."
The Spotify container is no longer maintained.
I suggest you follow the Confluent Docker quickstart guides or at least use Docker Compose rather than any all-in-one Kafka+ZooKeeper image
I will also point you at the blue help content that says "or Docker?"
but I don't know what is the ip of the kafka:
Its the same IP as Zookeeper, because you're only using a single host here
It appears you're using Linux, so it's not clear why you need Docker Machine
Related
I have setup apache atlas on EC2. After setup , I started the apache atlas using command python2 atlas_start.py and it successfully started with below log:
configured for local hbase.
hbase started.
configured for local solr.
solr started.
setting up solr collections...
starting atlas on host localhost
starting atlas on port 21000
..............................................................................................................................................................................................................................................................................................................
Apache Atlas Server started!!!
However, when I try to hit the Server URL to verify if apache atlas is up and running. It gave me connection refused error. Below is the command I used to hit the server:
curl -u username:password http://localhost:21000/api/atlas/admin/version
Also, while setup it didnt ask me for username and password SO, I am using admin:admin as username password.
you have to build the server using "mvn clean install -Pdist,", once this is done you need to navigate to the distro/target/ folder and then run the atlas_start.py from there.
Here solution
https://community.cloudera.com/t5/Support-Questions/Apache-Atlas-Started-but-with-Errors-and-Warnings/td-p/134183
The problem with this script is that starting atlas using atlas_start.py takes about 10 minutes. While its process is not completed, you face to connection refused error.
In order to see whether it is completed or not, you can check the application.log file in logs directory using tail -n 10 application.log or cat application.log. In 2.1.0 version the last 10 line of application.log is as follows:
2021-03-28 06:00:06,536 INFO - [main:] ~ Starting service org.apache.atlas.web.service.ActiveInstanceElectorService (Services:68)
2021-03-28 06:00:06,540 INFO - [NotificationHookConsumer thread-0:] ~ [atlas-hook-consumer-thread]: Starting (Logging$class:66)
2021-03-28 06:00:06,542 INFO - [main:] ~ HA is not enabled, no need to start leader election service (ActiveInstanceElectorService:103)
2021-03-28 06:00:06,542 INFO - [NotificationHookConsumer thread-0:] ~ ==> HookConsumer doWork() (NotificationHookConsumer$HookConsumer:530)
2021-03-28 06:00:06,544 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is ready, can start reading Kafka events. (NotificationHookConsumer$HookConsumer:936)
2021-03-28 06:00:06,628 WARN - [NotificationHookConsumer thread-0:] ~ [Consumer clientId=consumer-1, groupId=atlas] Error while fetching metadata with correlation id 2 : {ATLAS_HOOK=LEADER_NOT_AVAILABLE} (NetworkClient$DefaultMetadataUpdater:968)
2021-03-28 06:00:06,757 WARN - [NotificationHookConsumer thread-0:] ~ [Consumer clientId=consumer-1, groupId=atlas] Error while fetching metadata with correlation id 4 : {ATLAS_HOOK=LEADER_NOT_AVAILABLE} (NetworkClient$DefaultMetadataUpdater:968)
2021-03-28 06:00:06,932 WARN - [NotificationHookConsumer thread-0:] ~ [Consumer clientId=consumer-1, groupId=atlas] Error while fetching metadata with correlation id 6 : {ATLAS_HOOK=LEADER_NOT_AVAILABLE} (NetworkClient$DefaultMetadataUpdater:968)
2021-03-28 06:00:07,585 INFO - [main:] ~ AuditFilter initialization started (AuditFilter:64)
2021-03-28 06:00:07,585 INFO - [main:] ~ REST_API_ENABLE_DELETE_TYPE_OVERRIDE=false (AuditFilter:69)
After that you can see the login page http://localhost:21000:
or using curl command:
~ ➤ curl -u admin:admin http://localhost:21000/api/atlas/admin/version
{"Description":"Metadata Management and Data Governance Platform over Hadoop","Revision":"release","Version":"2.1.0","Name":"apache-atlas"}%
default username and password are **admin**.
There would be many reasons - do the following checks
Please check the logs which would be in the logs directory based on how you have it setup. (if you just ran it through mvn)
You need hbase(atleast) and solr/elastic/cassandra (atlas backends). Check if you are running embedded mode - where atlas will start its own backend. It is usually set as an environment variable. ie: MANAGE_LOCAL_HBASE
Make sure atlas-application.properties have the right configurations before you start it.
Hope this helps!
EDIT:
I noticed you mentioned ec2. Make sure you have configured the right security groups.
I am trying to statrt kafka and zookeeper using this docker file
version: '2'
services:
kafka:
image: landoop/fast-data-dev:cp3.3.0
hostname: kafka-host
ports:
- 29092:29092
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 2181:2181 # Zookeeper
- 9092:9092 # Kafka Broker
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
FORWARDLOGS: 0
SAMPLEDATA: 0
but I keep on receiving this error message
Cannot start service kafka: b'driver failed programming external connectivity on endpoint kafka-docker_kafka_1 (a741da2d21a00bb752fc169d579fa39bcdeef0cc88ec560d2e93b8fd287b8b5a): Error starting userland proxy: Bind for 0.0.0.0:9092 failed: port is already allocated'
even after I remove all the images and containers ...
with "sudo lsof -i tcp:9092"
I see some process are going on ... but even after I kill them, using kill -9 PIDsome other appear ...
I solved the problem; kafka and zookeeper were actually already installed locally and brew were running them in the background ... when I uninstalled them, everything was fine!
It seems that one of your containers is still using port 9092 (as sudo lsof -i tcp:9092 shows). Run docker ps to be sure that there is no container running using this port.
Moreover, stopping containers using kill -9 command is not the right way :
if you're using docker-compose (as your example shows), you can run docker-compose down to stop all services (therefore all containers defined in your file). Alternatively, if you want to shutdown a container using only docker command, you have to run docker stop <container_id_or_name>, but I do not recommend that if you're using docker-compose
when you issue a kill -9 command, you're experiencing that containers are restarting over and over because you probably have defined a restart policy on your service (https://docs.docker.com/compose/compose-file/compose-file-v2/#restart). The container started after the other one have exited (caused by kill -9) is using the same ports as the previous, so port 9092 is always used
To remove a node from 2 node cluster in AWS I ran
nodetool removenode <Host ID>
After this I was supposed to get my cluster back if I put all the cassandra.yaml and cassandra-rackdc.properties correctly.
I did it but still, I am not able to get back my cluster.
nodetool status is displaying only one node.
significant system.log on cassandra is :
INFO [main] 2017-08-14 13:03:46,409 StorageService.java:553 - Cassandra version: 3.9
INFO [main] 2017-08-14 13:03:46,409 StorageService.java:554 - Thrift API version: 20.1.0
INFO [main] 2017-08-14 13:03:46,409 StorageService.java:555 - CQL supported versions: 3.4.2 (default: 3.4.2)
INFO [main] 2017-08-14 13:03:46,445 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 198 MB and a resize interval of 60 minutes
INFO [main] 2017-08-14 13:03:46,459 MessagingService.java:570 - Starting Messaging Service on /172.15.81.249:7000 (eth0)
INFO [ScheduledTasks:1] 2017-08-14 13:03:48,424 TokenMetadata.java:448 - Updating topology for all endpoints that have changed
WARN [main] 2017-08-14 13:04:17,497 Gossiper.java:1388 - Unable to gossip with any seeds but continuing since node is in its own seed list
INFO [main] 2017-08-14 13:04:17,499 StorageService.java:687 - Loading persisted ring state
INFO [main] 2017-08-14 13:04:17,500 StorageService.java:796 - Starting up server gossip
Content of files:
cassandra.yaml : https://pastebin.com/A3BVUUUr
cassandra-rackdc.properties: https://pastebin.com/xmmvwksZ
system.log : https://pastebin.com/2KA60Sve
netstat -atun https://pastebin.com/Dsd17i0G
Both the nodes have same error log.
All required ports are open.
Any suggestion ?
It's usually a best practice to have one seed node per DC if you have just two nodes available in your datacenter. You shouldn't make every node a seed node in this case.
I noticed that node1 has - seeds: "node1,node2" and node2 has - seeds: "node2,node1" in your configuration. A node will start by default without contacting any other seeds if it can find it's IP address as first element in - seeds: ... section in the cassandra.yml configuration file. That's what you can also find in your logs:
... Unable to gossip with any seeds but continuing since node is in its own seed list ...
I suspect, that in your case node1 and node2 are starting without contacting each other, since they identify themselves as seed nodes.
Try to use just node1 for seed node in both instance's configuration and reboot your cluster.
In case of node1 being down and node2 is up, you have to change - seeds: ... section in node1 configuration to point just to node2's IP address and just boot node1.
If your nodes can't find each other because of firewall misconfiguration, it's usually a good approach to verify if a specific port is accessible from another location. E.g. you can use nc for checking if a certain port is open:
nc -vz node1 7000
References and Links
See the list of ports Cassandra is using under the following link
http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureFireWall.html
See also a detailed documentation on running multiple nodes with plenty of sample commands:
http://docs.datastax.com/en/cassandra/2.1/cassandra/initialize/initializeMultipleDS.html
This is for future reference. My problem has been solved just by opening 7000 port for same security group in AWS. Although it was open but security group was something different.
When I ran:
ec2-user#ip-Node1 ~]$ telnet Node2 7000
Trying Node2...
telnet: connect to address Node2: Connection timed out
I came to know the problem could be of the security group.
And that is how it has been solved.
About seeds I am using IP of both the nodes, like this:
-seeds: "node1,node2"
It is same on both the nodes.
I'd really appreciate some help to get cloudera manager running on AWS EC2.
Its my first install, and I'm aiming to use the AWS Free Tier to spin up a few nodes and do some training on Hadoop cluster and the cloudera distribution. I'm using the RedHat RHEL 7.2 image on AWS EC2.
I am following the instructions here... Cloudera Manager installation
I have installed cloudera manager OK, and get to the screen where it invites you to use a browser to log-in to the cloudera manager server. But that's where the problem starts. It seems the app is not listening on port 7180, so there's no hope of connecting from another machine across the network. I can't even connect locally, on the server, yet the service appears to be running OK. But its not listening on port 7180.
Q1 - How can I confirm the config is set to use port 7180.?
Q2 - are there obvious steps that I'm missing here ?
Thanks in advance,
[Edit..]
I'm beginning to wonder if the Free EC2 host is running short on memory to run cloudera manager. I saw one comment that implied that....AWS Forum post . But the process doesn't crash or report any problems in its logfile. So it must be OK, right?
[Edit.... with more diagnostic info....]
Here's a list of the diagnostics I've checked:-
SELinux is not running [for install and testing purposes],
WAN firewalls,
EC2 firewall/Security group,
Local firewall on server,
Cloudera manager log,
Is the service up and running?
Can you connect locally?
Securtity group on the EC2 instance, it contains:-
SSH and Port 7180,
Firewall/iptables/firewalld on the RedHat instance, tried:-
adding ports to iptables, then
dissabling iptables, then
adding ports to firewalld, then
dissabling the firewalld service,
$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:7180
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:7182
But I'm getting the feeling that the installation of cloudera manager is not happy, or not running correctly.
I've checked the cloudera manager log, and it ends with the following.
$ tail /var/log/cloudera-scm-server/cloudera-scm-server.log
2016-02-25 11:02:23,581 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 19264 new metrics
2016-02-25 11:02:28,920 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 0 updated metrics
2016-02-25 11:02:28,924 INFO main:com.cloudera.cmon.components.MetricSchemaManager: Cross entity aggregates processed.
And when I use tail -f, and restart the cloudera-scm-server service, the log scrolls a lot, and comes back the same state. If I search for ERROR, there are no lines with "ERR".
$ sudo service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl): [ OK ]
$ sudo systemctl status cloudera-scm-server
● cloudera-scm-server.service - LSB: Cloudera SCM Server
Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server)
Active: active (exited) since Thu 2016-02-25 12:23:03 EST; 44s ago
Docs: man:systemd-sysv-generator(8)
Process: 747 ExecStart=/etc/rc.d/init.d/cloudera-scm-server start (code=exited, status=0/SUCCESS)
So, if I try to test the service, by connecting from the local machine I get the sort of behavious that makes me thing its just not listening, and maybe not started correctly.
Try poke it with a curl from the same shell as the cloudera-scm-server service was started
$ curl localhost:7180
curl: (7) Failed connect to localhost:7180; Connection refused
$ wget localhost:7180
--2016-02-25 08:00:16-- http://localhost:7180/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:7180... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:7180... failed: Connection refused.
Try check what ports are listening on that machine, no 7180 , what's up with that???
$ netstat -nltp
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:7432 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp6 0 0 :::7432 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
Here's what to look for, and a possible solution - give it more memory...
Check the status of the cloudera-scm-server service using [depending on your flavour of linux]
$ sudo service cloudera-scm-server status
OR
$ sudo systemctl status cloudera-scm-server
Look for the status - Active: active (running)
But if you find - Active: active (exited)
you may have a problem during the startup of the cloudera-scm-server.
In which case, look at the log files for cloudera-scm-server
$sudo ls -l /var/log/cloudera-scm-server
$sudo cat /var/log/cloudera-scm-server/cloudera-scm-server.out
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000078dc58000, 265809920, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 265809920 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid831.log
[ec2-user#ip-172-31-31-166 ~]$ sudo tail -100 /var/log/cloudera-scm-server/cloudera-scm-server.out
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000078dc58000, 265809920, 0) failed; error='Cannot allocate memory' (errno=12)
Use the command top to indicate how much memory is available to your system.
Possible solution - have a look at this discussion at Cloudera forum
In this case the java heap size was too small.
As we see that heap was exhausted, assuming this is not a memory leak
or something of the sort, Cloudera Manager may need more heap to
operate. This can be configured in:
/etc/default/cloudera-scm-server You could, for instance, change "-Xmx2G" to "-Xmx3G" or "-Xmx4G" If the problem still
happens, perhaps the heap dumps will yeild some clues.
I'd suggest you tail the logs. If you are using the free tier, cloudera manager will take a while to come up... possibly up to 5 minutes or more after you start the cloudera-scm-server.
The logs should show if there are any errors, possibly issues with memory allocation since the free tier servers have limited memory available. The little snippet of log entries looks fine and typical - it will go through a long list of processes before the UI comes up on 7180.
Also while that is going on, run top or even free -g to see how much resources are being used - particularly memory.
I was having the exact same issue, cannot hit the CM login using public DNS or IP on port 7180.
Following steps will help you :
iptables stopped (service iptables stop)
SELinux disabled (got to /etc/selinux/config and disbaled the selinux)
curl/wget localhost:7180 works (check the curl status)
ufw allow 7180
service httpd status should be running.
check va/log/cloudera-scm-server log : if any error found then troubleshoot the error
cloudera-scm-server status (should be running state)
netstat -nap | grep 7180 returns (if running other service then kill it)
telnet localhost 7180 (should be connected)
Cannot connect to Cloudera Manager, not listening on port 7180
1] Check the status:
sudo service cloudera-scm-server status
*cloudera-scm-server.service - LSB: Cloudera SCM Server Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled) Active: active (exited) since UTC; 47min ago Docs: man:systemd-sysv-generator(8) rm /var/run/cloudera-scm-server.pid
NOTE : The Cloudera Manager service will not be running as it exited abnormally.
Running service cloudera-scm-server status will print following message "cloudera-scm-server dead but pid file exists".
Reason: Out of memory.
Solution : Examine the heap dump that the Cloudera Manager Server creates when it runs out of memory. The heap dump file is created in the /tmp directory, has file extension .hprof and file permission of 600. Its owner and group will be the owner and group of the Cloudera Manager server process, normally cloudera-scm:cloudera-scm.
Link : http://www.cloudera.com/documentation/manager/5-0-x/Cloudera-Manager-Diagnostics-Guide/cm5dg_troubleshooting_cluster_config.html
Check the status of `cloudera-scm-server` and follow the instructions ahead:
[root#quickstart ~]# `service cloudera-scm-server status`
By default, Cloudera's QuickStart VM manages CDH using Linux's configuration
and service management. To use Cloudera Manager instead, you must shut down
and disable the existing CDH services and then start Cloudera Manager. You can
do this by running the following command:
`sudo /home/cloudera/cloudera-manager`
[root#quickstart ~]# `sudo /home/cloudera/cloudera-manager `
`[QuickStart] Shutting down CDH services via init scripts...
JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
[QuickStart] Disabling CDH services on boot...
[QuickStart] Starting Cloudera Manager services...
[QuickStart] Deploying client configuration...
[QuickStart] Starting CM Management services...
[QuickStart] Enabling CM services on boot...
[QuickStart] Starting CDH services...`
________________________________________________________________________________
Success! You can now log into Cloudera Manager from the QuickStart VM's browser:
http://quickstart.cloudera:7180
Username: cloudera
Password: cloudera
I've installed Docker on OSX and downloaded the neo image. when I run it (using the args in the home page of the image), everything seems to work, but the last lines of the log indicate something like:
00:20:39.662 [main] INFO org.eclipse.jetty.server.Server - Started
#4761ms 2015-10-05 00:20:39.663+0000 INFO [API] Server started on:
http://022b5f3a38fc:7474/ 2015-10-05 00:20:39.663+0000 INFO [API]
Remote interface ready and available at [http://022b5f3a38fc:7474/]
which seem odd and attempting to connect my browser to either http://localhost:7474/ or the indicated http://022b5f3a38fc:7474/ results in an error
what am I missing here?
You'll want to use the IP address of the docker VM, which you can determine with this command:
docker-machine inspect default | grep IPAddress
The default IP address is 192.168.99.100
So depending on which port you exposed when running the Neo4j docker container you can access the Neo4j browser at:
http://192.168.99.100:7474
or
http://192.168.99.100:8474
Port 8474 is the the binding specified by this command:
docker run -i -t --rm --name neo4j -v $HOME/neo4j-data:/data -p 8474:7474 neo4j/neo4j
which is the example given in the documentation here