I have downloaded Flink 1.5.0 and ran the start-cluster script and the cluster seems to have started successfully:
$ ./bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host LAPTOP-HRAHBL24.
Starting taskexecutor daemon on host LAPTOP-HRAHBL24.
However, I am when I access http://localhost:8081/ the connection is being refused. I followed the question to un-comment the localhost lines in my hosts file:
https://superuser.com/questions/1008349/unable-to-connect-to-localhost-after-windows-10-upgrade
I checked the logs and I have this error in it:"/d/Programs/flink-1.5.0/bin/flink-daemon.sh: line 131: C:\Program: No such file or directory" I had this error with Flink 1.10.1, I tried searching it but couldn't find anything
Any idea what I can do to fix this?
In conf/flink-conf.yaml, set java home with this variable env.java.home
The problem was the space in "Program Files", so I set the java home in conf/flink-conf.xml instead and this error was gone.
Related
I follow the first steps to install Flink.
I can start the cluster without any problem
$ start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host DESKTOP-....
Starting taskexecutor daemon on host DESKTOP-....
But I don't get any status from
$ ps aux | grep flink
I can also not access the dashboard via localhost:8081.
There is an older post having these issues, but the solution didn't work for me, since the described conf files do no longer exist, apparently.
My JAVA_HOME is set as C:\Progra~1\Java\jdk1.8.0_311 to avoid issues with the space in Program Files.
Can you check the logs in the /logs folder? I'm suspecting that C:\Program Files\ could still cause issues because of the space there.
go to download Flink folder and try bash command
$./bin/start-cluster.sh --daemon bootstrap-server localhost:8081
and run code one more
$ ./bin/flink run examples/streaming/WordCount.jar
if you finished run above code which not issue, go to localhost:8081
This still seems to be problematic. I tried to run from Windows Subsystem for Linux (WSL).
I have the following versions: java 11.0.16 and flink 1.15.2.
sudo apt-get update
sudo apt install openjdk-11-jre-headless
export FLINK_HOME=/mnt/c/Projects/Apache/flink-1.15.2
I set the following in flink-conf.yaml
rest.port: 8081
rest.address: localhost
rest.bind-adress: 0.0.0.0
Whereby I changed the bind address for localhost to 0.0.0.0 this seems to have fixed the problem.
$FLINK_HOME/bin/start-cluster.sh
Now I can access the Flink Web Dashboard.
I am trying to restart my node by running StartNode.bat in cmd. But it is giving me the following error,
ADMU0111E: Program exiting with error: java.io.FileNotFoundException: C:\Users{userid}\workspaces\was_profiles{profilename}\config\cells\localcell\nodes\localnode\servers\nodeagent\server.xml (The system cannot find the path specified.) When I checked the above path, server.xml is present in the path C:\Users{userid}\workspaces\was_profiles{profilename}\config\cells\localcell\nodes\localnode\servers\server1\server.xml. I am not sure how to restart my node. Also, I am restarting the node, since I get ORA error invalid username/password issue while trying to test my datasource connection in IBM websphere.
Most of the posts suggest me to restart websphere server and nodeagent as well. Eventually, when I start the nodeagent, I get the above error. I tried to use syncNode command, but could not find the deployment manager host name, port details for my application.
It looks like you have single server version installed, not network deployment. In that case you dont have node and deployment manager, but just single server.
You should use commands like:
startServer.bat server1 - to start the server
stopServer.bat server1 - to stop server
I have just upgraded to Kafka 1.0 and zookeeper 3.4.10.At first, it all started fine. Stand - alone producer and consumer worked as expected. After I've ran my code for about 10 minutes, Kafka fails with this error:
[2017-11-07 16:48:01,304] INFO Stopping serving logs in dir C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.log.LogManager)
[2017-11-07 16:48:01,320] FATAL Shutdown broker because all log dirs in C:\Kafka\kafka_2.12-1.0.0\kafka-logs have failed (kafka.log.LogManager)
I have reinstalled and reconfigured Kafka 1.0 again, the same thing happened. If I try to restart, the same error occurs.
Deleting log files helps to start Kafka, but it fails again after the short run.
I have been running 0.10.2 version for a long while, and never encountered anything like this, it was very stable over the long periods of time.
I have tried to find a solution and followed instructions in the documentation.
This is not yet a production environment, it is fairly simple setup, one producer, one consumer reading from one topic.
I am not sure if this could have anything to do with zookeeper.
**Update: ** the issue has been posted at Apache JIRA board
The consensus so far seems to be that it is a Windows issue.
Ran into this issue as well, and only clearing the kafka-logs did not work. You'll also have to clear zookeeper.
Steps to resolve:
Make sure to stop zookeeper.
Take a look at your server.properties file and locate the logs directory under the following entry.
Example:
log.dirs=/tmp/kafka-logs/
Delete the log directory and its contents. Kafka will recreate the directory once it's started again.
Take a look at the zookeeper.properties file and locate the data directory under the following entry.
Example:
dataDir=/tmp/zookeeper
Delete the data directory and its contents. Zookeeper will recreate the directory once it's started again.
Start zookeeper.
<KAFKA_HOME>bin/zookeeper-server-start.sh -daemon <KAFKA_HOME>config/zookeeper.properties
Start the kakfa broker.
<KAFKA_HOME>bin/kafka-server-start.sh -daemon <KAFKA_HOME>config/server.properties
Verify the broker has started with no issues by looking at the logs/kafkaServer.out log file.
I've tried all the solutions like
Clearing Kafka Logs and Zookeeper Data (issue reoccurred after creating new topic)
Changing log.dirs path from forward slash "/" to backward slash "\" (like log.dirs=C:\kafka_2.12-2.1.1\data\kafka ) folder named C:\kafka_2.12-2.1.1\kafka_2.12-2.1.1datakafka was created and the issue did stop and the issue was resolved.
Finally I found this link, you'll get it if you google kafka log.dirs windows
Just clean the logs in C:\Kafka\kafka_2.12-1.0.0\kafka-logs and restart kafka
If at all, you are trying to execute in Windows machine, try changing path in windows way for parameter log.dirs (like log.dirs=C:\some_path\some_path_kafLogs) in server.properties in /config folder.
By default, this path will be in unix way (like /unix/path/).
This worked for me in Windows machine.
So this seems to be a windows issue.
https://issues.apache.org/jira/browse/KAFKA-6188
The JIRA is resolved, and there is an unmerged patch attached to it.
https://github.com/apache/kafka/pull/6403
so your options are:
get it running on windows and build it with the patch
run it in a unix style filesystem (linux or mac)
perhaps running it on docker in windows is worth a shot
The problem is in a concurrent working with log files of kafka. The task is a delaying of external log files changing between all Kafka threads and
Topic configuration can help:
Map<String, String> config = new HashMap<>();
config.put(CLEANUP_POLICY_CONFIG, CLEANUP_POLICY_COMPACT);
config.put(FILE_DELETE_DELAY_MS_CONFIG, "3600000");
config.put(DELETE_RETENTION_MS_CONFIG, "864000000");
config.put(RETENTION_MS_CONFIG, "86400000");
What worked for me was deleting both kafka and zookeeper log directories then configuring my log directories path in both kafka and zookeeper server.properties files (can be found in kafka/conf/server.properties) from the usual slash '/' to a backslash '\'
on windows changing to path separators '' resolved the issue, each required a double backslash ' C:\\path\\logs
Simply delete all the logs from :
C:\tmp\kafka-logs
and restart zookeeper and kafka server.
I am following these link:http://programming-tips.in/kafka-set-up-apache-kafka-on-windows/ and https://dzone.com/articles/running-apache-kafka-on-windows-os for Apache Kafka setup on a local machine.
After successfully running zookeeper for which my screen looks like:
My server.properties file looks like:
log.dirs="C:\Program Files\kafka_2.11-0.11.0.0\kafka-logs"
listeners=PLAINTEXT://:9092
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
My zookeeper.properties file looks like:
dataDir="C:\Program Files\kafka_2.11-0.11.0.0\zookeeper-data"
clientPort=2181
But, when I am trying to run the following command:
C:\Program Files\kafka_2.11-0.11.0.0>\bin\windows\kafka-server-start.bat \config\server.properties
Although, I have give all the paths correct which I have verified many times, I get this error:
The system cannot find the path specified.
What could be the possible solution?
Kafka on Windows currently doesn't support spaces in the install path. Try moving your Kafka folder so the path does not contain spaces. This issue is reported in a JIRA, and should be fixed in one of the upcoming releases.
The problem seems to be with leading '\' in your path, the following should work
C:\Program Files\kafka_2.11-0.11.0.0>bin\windows\kafka-server-start.bat config\server.properties
I have a CDH 5.1 cluster with 3 nodes. We installed it using cloudera manager automated installation.
It was running perfect until we moved the box to a different network and IP addresses changed. I tried following steps
1. Stopped service, cloudera-scm-server.
2. Stopped service, cloudera-scm-agent
3. Edit the /etc/cloudera-scm-agent/config.ini
4. change the server host to the new ip.
5. restart service, cloudera-scm-agent, cloudera-scm-server.
not working .
Then i followed
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v4-latest/Cloudera-Manager-Administration-Guide/cmag_change_hostnames.html
Not helped even after changing the ips in the PostgreSQL directly.
I found following blog :
http://www.geovanie.me/changing-ip-of-node-in-cdh-cluster/
Getting following error in the scm-agent log file
ProtocolError: <ProtocolError for 127.0.0.1/RPC2: 401 Unauthorized>
Not working ....
Can anyone please help how to change all IP addresses in a cdh 5.1 cluster safely .....
Thanks,
Amit
This is causing because of precious cloudera-scm-agent service wasn't stopped correctly, please try,
$> ps -ef | grep supervisord
$> kill -9 <processID>
then restart the agent again.
$>service cloudera-scm-agent start