I am trying to follow the v 1.2.3 tutorial for apache storm.
Trying to get the ExclamationTopology example from storm starter running.
I installed storm on my macbook using brew.
Which meant I ended up with apache storm v 1.2.3
(And that would suit me fine cos I'm trying to get to a situation where I understand stormcrawler which seems to use 1.2.3 anyway.)
So, I spent a long time trying to see if I could view the output from the example and finally think I found the output in:
/usr/local/Cellar/storm/1.2.3/libexec/logs/worders-artifacts
First question.. is that the right place to find the output?
Second Question:
the log file there is just full of messages as follows:
0.003s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:artifacts/gc.log instead. Unrecognized VM option 'PrintGCDateStamps' Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I had tried to get this whole lot running with jdk 1.8 as 1.7 was giving errors... so would that explain why this error is occurring ?
Any help or advice would be greatly appreciated...
The directory layout for the brew install of Storm looks to be a little different than a standard Storm zip install. You are probably looking in the right place, yes.
The error message you're seeing is because you're running with some JDK version greater than 9. You can either use an older JDK (e.g. 8), or upgrade Storm to 2.0.0. Finally you can also work around the issue by editing/adding the worker.childopts option in your storm.yaml, which is somewhere in your storm install directory. Something like:
worker.childopts: "-Xmx%HEAP-MEM%m -Xlog:gc*:artifacts/gc.log:time,uptime:filecount=10,filesize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
should work (this is the default in Storm 2.0.0).
Related
I had a Streamparse topology that was originally developed using Python 2. I am now trying to upgrade it to Python 3 using the 2to3 tool. I have also upgraded Streamparse to 3.15.1 (not sure which version the topology was originally developed against). When I try to run my topology using sparse run -d I get the following error:
AttributeError: 'ShellBoltSpec' object has no attribute 'componentId'
I presume it is because the ShellBoltSpec class has changed from Python 2 to Python 3 but I have no idea how to go about fixing this. Does anybody have any suggestions?
Regardless of 2to3, it could be that upgrading Streamparse to version 3.x.x broke backwards compatibility with the previous version you worked with, if that was 2.x.x or below. If so, you need to explicitly edit your code to be compatible with Steamparse 3.x.x.
My team has a monorepo where we are using bolt to manage dependencies and linking. I was trying to get the dev environment up and running on my local Windows machine but had issues install/link acorn. What I did was clone the repo, then $ bolt to install dependencies. As you can see, it showed error linking dependencies and complained about acorn.ps1 not being a symlink. I've done some research but none of the solutions I found, such as including --non-bin-links or running as admin, worked for me. I'd appreciate if you could provide some insights on this.
Windows: 10;
Yarn: 1.13.0;
Bolt: 0.22.6 - tried 0.22.4 as well but did not work;
Ok I found the solution to this issue.
acorn is not a dependency for my project, but it is one for yarn itself, as I searched for it and it only appears in yarn.lock. Meaning there's definitely something weird with my current version of Yarn, 1.13.0. So I downgraded it to 1.12.3 and according to the official documentation, installed bolt as a local package instead of a global one. It solved this issue.
I am trying to use Apache Flink for the first time for a school project, but I'm running into some trouble after installation. I have tried to follow the Quicksetup on the site, but I keep running into the same problem.
(https://ci.apache.org/projects/flink/flink-docs-stable/quickstart/setup_quickstart.html)
Console Output
The brew installation seems to work fine, this is where I ran into the first inconsistency however. Next the guide tells me to execute the following line
$ ./bin/start-cluster.sh # Start Flink
which doesn't work without installing a binary from here though
https://flink.apache.org/downloads.html
So I did that (I downloaded the Flink 1.6.2 with Hadoop 2.7 binary) and was then able to find the file and execute it. After doing this I'm supposed to go the the web frontend, and verify that a task is running. However when I go there nothing is shown.
Console Output
Web Dashbord
I started by executing the stop-cluster file, because for some reason I'm constantly getting the INFO Message
[INFO] 1 instance(s) of standalonesession are already running on Moritzs-MBP.fritz.box.
So far I have tried to to look into the flink-conf.yaml file which does set the numberOfTaskSlots to 1, so the problem does not seem to lie there. Otherwise I didn't really know where to look from there however and only tried to uninstall the and reinstall everything with no results however.
Sorry that the images are not shown, previously I had been able to fix all my code problems with existing posts and thus I can't directly post them yet. Any help would be much appreciated, thanks in advance,
Moritz
By using brew installation, the start-cluster.sh should be able to found in:
/usr/local/Cellar/apache-flink/1.6.2/libexec/bin
I'm not sure why start-cluster.sh isn't succeeding in starting up a taskmanager, but you can always (try to) start another one by hand with
$ ./bin/taskmanager.sh start
I am exploring a production-stable proxy for redis cluster called codis . It is a mentioned as a great alternative to twemproxy, especially as one of my needs is pipelining and twemproxy does not offer that.
However the documentation in English is still a WIP and the replies to github issues are in mandarin.
I am trying to install this on
Linux version 3.13.0-74-generic (buildd#lcy01-07) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #118-Ubuntu SMP
I have installed go version 1.8 and I can see the folder /usr/local/go/bin. I have added this to the PATH variable as well.
However, when on executing the command go get -u -d github.com/CodisLabs/codis, I am getting the following :
package github.com/CodisLabs/codis:
no buildable Go source files in /home/ubuntu/go/src/github.com/CodisLabs/codis
You probably want to use the updated install outlined at https://github.com/CodisLabs/codis/issues/1180#issuecomment-286660086
seems the English docs might be out of date https://github.com/CodisLabs/codis/issues/1179#issuecomment-286662505
I am trying to run hadoop in my machine (Windows x64), but I get this error when building the sources:
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.3.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionExcepti
on: 'protoc --version' did not return a version -> [Help 1]
As suggested in the documentation and explained in the BUILDING.txt, I have in my PATH vble the protoc path and from my Windows SDK I can call 'protoc --version' with no problem:
C:\Users\hernanbl\Downloads\hadoop-2.3.0-src\hadoop-common-project>protoc --version
libprotoc 2.5.0
Besides, the version is the same as used in hadoop 2.3.0 (the version I am trying to build). As defined in the documentation, I made further changes, such as declaring a Platform vble and so on.
I already googled the question and I found that I might not be the only one having this issue, but the answers don´t give me the clue to fix this...
Some ideas?? Thanks!!
I solved it by setting the environment variable pointing to the executable:
set HADOOP_PROTOC_PATH=C:\Path\To\protoc\protoc.exe
I got it.... still a solution I don´t like at all! If someone found a better one, please let me know:
Modify the class ProtocMojo, hardcoding the absolute path to the protoc.exe:
protocCommand= "C:\\development\\tools\\protoc\\protobuf-2.5.0\\src\\protoc.exe";