I'm trying to run those Flink Benchmarks:
https://github.com/dataArtisans/flink-benchmarks
I've generated the jar file using maven with that command:
mvn clean package -Pbuild-jar
Then I'm trying to run the benchmark on a Flink Cluster with that command:
./bin/flink run -c org.apache.flink.benchmark.WindowBenchmarks ~/flinkBenchmarks/target/flink-hackathon-benchmarks-0.1.jar
I've used the -c option to add to the classpath the Main of the benchmark (WindowBenchmarks) I want to run.
Finally, I get that error:
# JMH version: 1.19
# VM version: JDK 1.8.0_151, VM 25.151-b12
# VM invoker: /usr/lib/jvm/java-8-oracle/jre/bin/java
# VM options: -Dlog.file=/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/log/flink-user-client-mypc.log -Dlog4j.configuration=file:/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/conf/log4j-cli.properties -Dlogback.configurationFile=file:/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/conf/logback.xml -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.flink.benchmark.WindowBenchmarks.sessionWindow
# Run progress: 0.00% complete, ETA 00:04:00
# Fork: 1 of 3
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
<forked VM failed with exit code 1>
<stdout last='20 lines'>
</stdout>
<stderr last='20 lines'>
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
</stderr>
# Run complete. Total time: 00:00:00
Benchmark Mode Cnt Score Error Units
The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.
I don't have any previous experience with Flink and Maven so I find out what is missing. My first thought was that it's a missing dependencies error, but they look fine. Any suggestions?
Thank you in advance!
flink-benchmarks is a repository that contains sets of micro benchmarks designed to run on single machine, not on the cluster. The main functions defined in the various classes (test cases) are 'JMH' runners, not Flink programs. As such you can either execute whole benchmark suite (which takes ~1hour):
mvn -Dflink.version=1.5.0 clean install exec:exec
or if you want to execute just one benchmark, the best approach is to execute selected main function manually. For example from your IDE (don't forget about selecting flink.version, default value for the property is defined in pom.xml).
There is also a possibility to execute single benchmark from console, but I haven't tried it for very long time.
Related
I have a total of 405 tests. They are all executed fine when running on a single thread. However, when trying to run it in parallel, it seems the number of tests is not being properly allocated per thread.
So, for example, executing it using 3 threads:
mvn integration-test -Dwebdriver.remote.url=http://selenium-hub.project.svc.cluster.local:4444/wd/hub \
-Dwebdriver.remote.driver=chrome \
-Dwebdriver.driver=chrome \
-Dconfig.threads=3 \
-Dserenity.batch.size=3 \
-Dserenity.batch.number=<"from 1 to 3"> \
-Dserenity.batch.strategy=DIVIDE_BY_TEST_COUNT \
-Dserenity.take.screenshots=FOR_EACH_ACTION
After triggered maven, as according to the sample above, the tests have been allocated as follows:
Thread 1: 106
Thread 2: 96
Thread 3: 103
Total: 305
The funny thing is that those numbers vary, changing the tests count per thread on every execution.
As well, it is like it is counting 4 threads instead of 3.
I'm running those tests using Jenkins, hosted in an Openshift environment.
Found a workaround by increasing the number of threads (e.g. from 3 to 4). It looks like somehow there is a limitation regarding the number of tests executed per thread.
I will search for this config. and keep this post updated in case I find something.
i am trying to install elasticsearch with logstash for every mysql action.
i stuck when trying to install logstash .
heres my command.
[root#centos-elasticsearch tampung]# rpm -ivh logstash-6.5.2.rpm
Preparing... ################################# [100%]
Updating / installing...
1:logstash-1:6.5.2-1 ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Errno::ENOMEM: Cannot allocate memory - systemctl
spawn at org/jruby/RubyProcess.java:1566
spawn at org/jruby/RubyKernel.java:1511
popen_run at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:206
popen3 at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:102
execute at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:74
detect_systemd at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:29
detect_platform at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:24
detect at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:18
setup_defaults at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:153
execute at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:119
run at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
run at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:114
run at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132
<main> at /usr/share/logstash/lib/systeminstall/pleasewrap.rb:12
Unable to install system startup script for Logstash.
anyone know what must i do?
i tried to remove logstash,and reinstall.
but still not success.
i tried to upgrade memory..
after that i tried to reinstall again
and i get this.
Updating / installing...
1:logstash-1:6.5.2-1 ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Errno::EISDIR: Is a directory - /etc/default/logstash
write at org/jruby/RubyIO.java:3622
write at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/installer.rb:32
block in install_files at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/installer.rb:23
yield at org/jruby/RubyYielder.java:102
block in files at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/systemd.rb:24
each at org/jruby/RubyGenerator.java:99
install_files at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/installer.rb:18
run_human at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:185
execute at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:141
run at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
run at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:114
run at /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132
<main> at /usr/share/logstash/lib/systeminstall/pleasewrap.rb:12
{:timestamp=>"2018-12-07T07:44:13.341202+0000", :message=>"No platform selected. Autodetecting...", :platform=>"systemd", :version=>"219", :level=>:warn}
{:timestamp=>"2018-12-07T07:44:13.411454+0000", :message=>"Writing file", :destination=>"/etc/default/logstash"}
Unable to install system startup script for Logstash.
As I already wrote in my previous comment, the first error
Errno::ENOMEM: Cannot allocate memory
means that your system is low of phisical memory.
The second one
Errno::EISDIR: Is a directory
means that the file is a directory; and that you cannot open a directory for writing, or create or remove hard links to it.
I am using travis-ci and biicode to build my project who is depending on boost log. But boost log times are longer than 10 min so I get this message:
No output has been received in the last 10 minutes, this potentially indicates a
stalled build or something wrong with the build itself.
The build has been terminated
The build is working correctly, it's just that boost log is really long to compile with limited resources (I tried to compile it on a VM with 1 CPU and 2GB of RAM and it took almost more than 15 min)
I know this is happening because there is not enough verbose going on so I tried the following flags:
>bii cpp:build -- VERBOSE=1
In the CMakeList.txt, set BII_BOOST_VERBOSE ON as mentionnened here
Set BOOST_LOG_COMPILE_FAST_ON as explained here
Using travis_wait
Actually travis_wait seems to be the correct solution but when I put it in my .travis.yml like this
script: travis_wait bii cpp:build
It does actually doesn't output logs like usually and just time out after 20 min. I don't think the actual building is taking place
What is the correct way to handle this problem?
This is a known issue, Boost.Log takes a long time to compile.
You can use travis_wait to call bii cpp:configure, but I'm with you, I need log feedback (No pun intended). However, I have tried that too and leaded to >50min build, which means travis aborts build on free accounts :( Of course my repo does not build Boost.Log only.
Just as a note, here's part of the settings.py file from the boost-biicode repo:
#Boost.Log takes so much time to compile, leads to timeouts on Travis CI
#It was tested on Windows and linux, works 'ok' (Be careful with linking settings)
if args.ci: del packages['examples/boost-log']
I'm currently working on a solution, launching asynchronous builds while printing progress. Check this issue. It will be ready for this week :)
To speed-up your build, try to play with BII_BOOST_BUILD_J variable to set the number of threads you want for building Boost components. Here's an example:
script:
- bii cpp:configure -DBII_BOOST_BUILD_J=4
Be careful, more threads means more RAM needed to compile at a time. Be sure you don't make the travis job VM go out of memory.
I found a command-line tool called "xctest" that apparently can run the unit tests in your project. This executable lives here:
/Applications/Xcode.app/Contents/Developer/usr/bin/xctest
When I try to run this executable on my xctest bundle, I'm using:
$ ./xctest /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-abcdefghijklmnop/Build/Products/Debug/MyAppTests.xctest
However, I get the following output:
Test Suite '(null)' started at 2013-11-14 21:16:45 +0000
Test Suite '(null)' finished at 2013-11-14 21:16:45 +0000.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
There's no man page for xctest, as far as I can tell, but entering just ./xctest at the command-line yields:
Usage: xctest [--test Self | All | None | <TestCaseClassName/testMethodName>] <path of unit to be tested>
In particular, I'd like to be able to test just a particular method in a test class, which is why I'd like to use this xctest command.
I do see that there is a way to run all the tests from the command line like:
$ xcodebuild test -scheme MyApp
This runs all the unit tests and works properly (I see my unit test results, unlike when using xctest). But I'm interested in being able to run a single test method from the command-line, such as:
$ ./xctest --test MyAppTests/testExample /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-abcdefghijklmnop/Build/Products/Debug/MyAppTests.xctest
Despite what the usage message says -XCTest is the argument you need:
xctest -XCTest MyAppTests/testExample testbundle.xctest
For a direct invocation of xctest to work you may also need to set DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your built products directory. In general you need to use the same arguments and environment as Xcode does, you can see this by putting a breakpoint in one of your tests, running them through Xcode, then printing out the values of arguments and environment for [NSProcessInfo processInfo].
To avoid messing with all that note you can also modify the scheme in Xcode to run only specific tests. Under Product > Scheme > Edit Scheme select the Test action and expand the test bundle. You can use the check boxes to select the tests to run and xcodebuild's test action will then run only these tests.
I came across a strange problem about non-root users in Linux (CentOS).
I'm able to compile & run a Java Program through below commands properly :
[root#cuda1 hadoop-0.20.2]# javac EnumDevices.java
[root#cuda1 hadoop-0.20.2]# java EnumDevices
Total number of devices: 1
Name: Tesla C1060
Version: 1.3
Clock rate: 1296000 MHz
Threads per block: 512
But I need to run it through other user [B]hadoop[/B] in CentOS
[hadoop#ws37-mah-lin hadoop-0.20.2]$ javac EnumDevices.java
[hadoop#ws37-mah-lin hadoop-0.20.2]$ java EnumDevices
NVIDIA: could not open the device file /dev/nvidiactl (Permission denied).
Exception in thread "main" CUDA Driver error: 100
at jcuda.CUDA.setError(CUDA.java:1874)
at jcuda.CUDA.init(CUDA.java:62)
at jcuda.CUDA.<init>(CUDA.java:42)
at EnumDevices.main(EnumDevices.java:20)
[hadoop#ws37-mah-lin hadoop-0.20.2]$
Actually I need to run a map-reduce code but first if it runs through simple then I will go for it.
Please guide me how to solve this issue as CLASSPATH is same through all users.
Looks like you're running into a problem with device file permissions. Hadoop has nothing to do with this, neither does the Java classpath. This might be useful:
http://www.linuxquestions.org/questions/slackware-14/could-not-open-dev-nvidiactl-310026/