I am using hadoop2.7.2 on ubuntu 16.04 LTS and developed a single node setup. I am running Rumen to create job traces from log history.
I am using following command:
/usr/local/hadoop/bin/hadoopjar/usr/local/hadoop/share/hadoop/tools/lib/hadoop-rumen-2.7.2.jar org.apache.hadoop.tools.rumen.TraceBuilder
file:///usr/local/hadoop/share/hadoop/tools/sls/sample-data/job-trace.json
file:///usr/local/hadoop/share/hadoop/tools/sls/sample-data/job-topology.json
hdfs:///usr/local/hadoop/logs/userlogs/application_1486821591386_0001
But it throws an error "WARN rumen.TraceBuilder: no job found in traces" and same time it successfully create folders 'job-trace.json' and 'job-topology.json'. But job-trace.json is empty and job-topology.json has some values look like:
{
"name" : "root"
"children" : [ ]
}
I don't know why is this happening. I have already searched everywhere but not getting anything.
I solved the problem... I was giving wrong input path to Rumen. The actual path should be like this:
hdfs:///tmp/hadoop-yarn/staging/history/done/2017/03/11/000000
Related
I am new to the entire ELK Stack, and I am trying to set up Logstash. I followed all of the instructions (unzipping, setting up config file, starting Logstash). My setup is Windows 7, and my java version is 1.8.0_51.
When I run the following command (pipeline.conf is my config file):
C:\Elastic\logstash-6.2.2\bin>logstash -f pipeline.conf
I am getting the following error:
[ERROR] 2018-03-15 12:30:05.101 [main] Logstash -
java.lang.IllegalStateException:
org.jruby.exceptions.RaiseException:
(LoadError) Could not load FFI Provider:
(NotImplementedError) FFI not available:
com.kenai.jffi.Foreign.getVersion()I
See http://jira.codehaus.org/browse/JRUBY-4583
Here is what my config file:
input {
stdin {
}
}
output {
stdout {
codec => rubydebug
}
}
Any help would be appreciated. http://jira.codehaus.org/browse/JRUBY-4583 doesn't seem like a valid site. I have tried my exact process on a different machine, and Logstash works. I have been trying to look for a solution for about 2 days now. HELP PLS
Issue Resolved on the Elastic Discussion site:
https://discuss.elastic.co/t/windows-7-logstash-jruby-error/124152
I upgraded from spring-xd 1.2.1 to 1.3.0, and have both under /opt on my system. After starting xd in single node (but configured to use Zookeeper), I tried to create another stream (e.g. "time | log"), and spring-xd throws the following exception:
java.io.FileNotFoundException: /opt/spring-xd-1.2.1.RELEASE/xd/config/modules/modules.yml (Too many open files)
I changed ulimit -n 60000, but it didn't solve the problem. The strange thing is why it still points to spring-xd-1.2.1.RELEASE? I have started both xd-singlenode and xd-shell under /opt/spring-xd-1.3.1.RELEASE
EDIT: add xd-singlenode running process output just to show it's pointing to 1.3.1:
/usr/java/default/bin/java -Dspring.application.name=admin
-Dlogging.config=file:/opt/spring-xd-1.3.0.RELEASE/xd/config//
/xd-singlenode-logback.groovy -Dxd.home=/opt/spring-xd-1.3.0.RELEASE/xd
-Dspring.config.location=file:/opt/spring-xd-1.3.0.RELEASE/xd/config//
-Dxd.config.home=file:/opt
/spring-xd-1.3.0.RELEASE/xd/config//
-Dspring.config.name=servers,application
-Dxd.module.config.location=file:/opt/spring-xd-1.3.0.RELEASE/xd/config//modules/
-Dxd.module.config.name=modules -classpath
/opt/spring-xd-1.3.0.RELEASE/xd/modules/processor/scripts:/opt/spring-xd
-1.3.0.RELEASE/xd/config:/opt/spring-xd-1.3.0.RELEASE/xd/lib/activation-
...
have you updated your environment variables? specifically XD_CONFIG_LOCATION based on the error shown above.
I 'm intending to fix bugs on Elastic Search open-source project. I forked it and cloned the forked copy . Then I imported it as Maven project on Eclipse and then did Maven build . So far so good.
I opened ElasticSearchF.java file and tried to run it as a Java application.(This is as per directions written in http://www.lindstromhenrik.com/debugging-elasticsearch-in-eclipse/).
But I get an error saying path.home is not set for ElasticSearch and throws an error saying IllegalStateException.
My question is
Why is this error coming in the first place.
As I said , I want to fix bugs in ElasticSearch project.Is this the right way to set-up environment for my goal? Or should I have a client send the requests to the ElasticSearch server and then set-up debug points in Elastic Search source code. How to achieve this?
Thanks for your patience.
Update:
I did add VM argument as mentioned by one of the answerers.
Then it throws different errors and clue-less about why its throwing that.
java.io.IOException: Resource not found: "org/joda/time/tz/data/ZoneInfoMap" ClassLoader: sun.misc.Launcher$AppClassLoader#29578426
at org.joda.time.tz.ZoneInfoProvider.openResource(ZoneInfoProvider.java:210)
at org.joda.time.tz.ZoneInfoProvider.<init>(ZoneInfoProvider.java:127)
at org.joda.time.tz.ZoneInfoProvider.<init>(ZoneInfoProvider.java:86)
at org.joda.time.DateTimeZone.getDefaultProvider(DateTimeZone.java:514)
at org.joda.time.DateTimeZone.getProvider(DateTimeZone.java:413)
at org.joda.time.DateTimeZone.forID(DateTimeZone.java:216)
at org.joda.time.DateTimeZone.getDefault(DateTimeZone.java:151)
at org.joda.time.chrono.ISOChronology.getInstance(ISOChronology.java:79)
at org.joda.time.DateTimeUtils.getChronology(DateTimeUtils.java:266)
at org.joda.time.format.DateTimeFormatter.selectChronology(DateTimeFormatter.java:968)
at org.joda.time.format.DateTimeFormatter.printTo(DateTimeFormatter.java:672)
at org.joda.time.format.DateTimeFormatter.printTo(DateTimeFormatter.java:560)
at org.joda.time.format.DateTimeFormatter.print(DateTimeFormatter.java:644)
at org.elasticsearch.Build.<clinit>(Build.java:53)
at org.elasticsearch.node.Node.<init>(Node.java:138)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:157)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:177)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:278)
at org.elasticsearch.bootstrap.ElasticsearchF.main(ElasticsearchF.java:30)
[2015-06-16 18:51:36,892][INFO ][node ] [Kismet Deadly] version[2.0.0-SNAPSHOT], pid[2516], build[9b833fd/2015-06-15T03:38:40Z]
[2015-06-16 18:51:36,892][INFO ][node ] [Kismet Deadly] initializing ...
[2015-06-16 18:51:36,899][INFO ][plugins ] [Kismet Deadly] loaded [], sites []
{2.0.0-SNAPSHOT}: Initialization Failed ...
- ExceptionInInitializerError
IllegalArgumentException[An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090, completion090, XBloomFilter]]
I got help from the developer community in https://github.com/elastic/elasticsearch/issues/12737 and was able to debug it.
procedure in short would be :
1) Search for the file Elasticsearch.java/ElasticsearchF.java inside the package org.elasticsearch.bootstrap .
2) Right click -> Run Configurations...
3) In the window that pops up , Click the "Arguments" tab and under "Program arguments:" section give the value as start
and under "VM arguments:" section give the value as
-Des.path.home={path to your elasticsearch code base root folder}/core -Des.security.manager.enabled=false
4) Click "Apply" and then click "Run".
It runs now.
to check , go to localhost:9200 and you will get a message something like
{
"name" : "Raza",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.0.0-beta1",
"build_hash" : "${buildNumber}",
"build_timestamp" : "NA",
"build_snapshot" : true,
"lucene_version" : "5.2.1"
},
"tagline" : "You Know, for Search"
}
for more info on arguments
see : https://github.com/elastic/elasticsearch/commit/2b9ef26006c0e4608110164480b8127dffb9d6ad
Edit your debug/run configurations,put it on the vm arguments:
-Des.path.home=C:\github\elasticsearch\
change the C:\github\elasticsearch\ to your elasticsearch root path
the reason is some arguments in the elasticsearch.bat is missed when you debug/run it in eclipse
I am trying to import a mysql table from mysql server to elasticsearch on my MAC OSX Mavericks.
I have installed elasticsearch 1.3.1 with homebrew
Installed jdbc-river 1.3.0.4 with elasticsearch plugin --install
Installed jdk 1.7.0_67
Downloaded mysql-connector-java-5.1.28-bin.jar into
$ES_HOME/plugins/jdbc (I had to create the folders 'plugins' and
'jdbc' myself) and gave chmod 777 permission for the .jar file.
Then I ran ./bin/elasticsearch and called this command in postman in order to create a river:
PUT request.
URL: localhost:9200/_river/my_jdbc_river/_meta
Raw data:
{
"type" : "jdbc",
"jdbc" : {
"url" : "jdbc:mysql://localhost:3306/<databaseName>",
"user" : "<MysqlUserName>",
"password" : "<MysqlUserPass",
"sql" : "select * from <TableName>"
}
}
And I received the following error in the elasticsearch log in the terminal:
[2014-08-26 15:38:39,300][ERROR][org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource] while opening read connection: jdbc:mysql://localhost:3306/xcollector No suitable driver found for jdbc:mysql://localhost:3306/xcollector
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/xcollector
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.getConnectionForReading(SimpleRiverSource.java:196)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.execute(SimpleRiverSource.java:315)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:287)
at org.xbib.elasticsearch.plugin.feeder.jdbc.JDBCFeeder.fetch(JDBCFeeder.java:335)
at org.xbib.elasticsearch.plugin.feeder.jdbc.JDBCFeeder.executeTask(JDBCFeeder.java:179)
at org.xbib.elasticsearch.plugin.feeder.AbstractFeeder.newRequest(AbstractFeeder.java:362)
at org.xbib.elasticsearch.plugin.feeder.AbstractFeeder.newRequest(AbstractFeeder.java:53)
at org.xbib.pipeline.AbstractPipeline.call(AbstractPipeline.java:87)
at org.xbib.pipeline.AbstractPipeline.call(AbstractPipeline.java:14)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I have followed many posts in google to try and find the cause of the problem, does anyone have any idea what I am missing?
And if anyone knows if this procedure can be more automatic to deploy like having some kind of package manager (like npm for node.js).
Thanks in advance,
So the clue to your problem is that you had to create the folders "plugins" and "jdbc" in step 4. Both of those folders are created when you install the mysql river plugin. I can see that the mysql river plugin installed correctly in your error message - it's running but it is unable to find the jdbc driver.
Look on your drive for the correct folder - $ES_HOME should have the following folders in it:
bin
config
data
lib
logs
plugins
If it does not then $ES_HOME is set incorrectly. Copy your jdbc driver as directed into the correct folder and you should be able to resolve this problem.
What's wrong with the following logstash configuration?
input {
file {
type => "access_log"
# Wildcards work, here :)
path => [ "/root/isaac/my_logs/access_logs/gw_access_log*"]
start_position => "beginning"
}
}
output {
stdout { debug => true }
elasticsearch { embedded => true }
}
When running the above configuration, logstash is stuck on startup as follows:
[root#myvm logstash]# java -jar logstash-1.3.3-flatjar.jar agent -f logstash-complex.conf
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.3/plugin-milestones {:level=>:warn}
More importantly what are the ways to debug the issue?
I already checked that the file i am putting in the path do exist.
That isn't stuck, that's running.
you get this:
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.3/plugin-milestones {:level=>:warn}
Once logstash has started successfully
If you add -- web onto the end of your command then you should be able to see some output in Kibana web interface
If you aren't seeing messages appear in the console, first I would check that new entries are definitely being written to the file(s) that you're trying to tail. Since you're using the stdout output you should see the messages written to the console at the same time as they're going into the embedded Elasticsearch.
What I would suggest is you simplify your config by removing the elasticsearch output - this should speed up the startup time (it can take a minute or two for the embedded elasticsearch instance to start up) and focus on getting messages onto the console output first.
If you do want more verbose debug output from Logstash you can start the program with -v, -vv or -vvv for progressively more detailed debug information. E.g.:
java -jar logstash-1.3.3-flatjar.jar agent -f logstash-complex.conf -vvv
Fair warning that -vvv does produce a LOT of debug information, so start with -v and work your way up.