I have tried searching on various internet forums about the offline installation of Logstash plugins, but haven't found the solution that we can implement.
What I want to achieve is:
Install the following two logstash plugins:
logstash-filter-mutate-master
logstash-filter-aggregate-master
My Limitations Are:
At my company, there is strictly no Internet connectivity on Linux servers. Therefore, I need a solution where I could download some off-the-shelf package on my Windows machine, and then unpack it on the Linux machine, which would install the plugins I require.
I have looked at the Logstash documentation, which advises on how to install plugins offline, but that also needs a 'Staging' server which has logstash source-code and has Internet connectivity - the latter is non-existent in my organisation.
I would greatly appreciate if you could advise how to approach this. At present, when I try to install the 'mutate' plugin, here's the error I see:
$ bin/logstash-plugin install --local --no-verify
Installing...
Plugin not found, aborting
ERROR: Installation Aborted, message: Could not find gem 'ci_reporter_rspec (>= 0) java' in any of the gem sources listed in your Gemfile or installed on this machine.
Thank you.
If you want to install a logstash plugin in offline.
First way ;
You must change gemfile in logstash directory. As an example, download plugin source in somewhere of your logstash directory then modify gemfile in logstash directory with plugin name and path.
gem "logstash-input-example-plugin", :path => "./plugins/logstash-input-example-plugin"
After that just run the below command
bin/logstash-plugin install --no-verify
You can follow this link :
https://www.tutorialspoint.com/logstash/logstash_plugins.htm
Second way; using with gem file of the plugin. As an example;
bin/logstash-plugin install /path/to/logstash-output-kafka-1.0.0.gem
You can follow this link :
https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html
Related
Due to the restrictions, I was not allowed to install any packages from internet. So, This command is not useful for me inorder to install search-guard.
bin/elasticsearch-plugin install -b com.floragunn:search-guard-ssl:<version>
However, I am able to install Search Guard successfully on a different network by running the above command.
Because of this reason, I tried installing Search Guard from tar.gz or zip file by the below command as per documentation.
/usr/share/elasticsearch# bin/elasticsearch-plugin install file:///home/xxxx/xxxx/search-guard-5-5.2.0-10-sgadmin-standalone.zip
This one is failing with the below error.
-> Downloading file:///home/xxx/xxxx/search-guard-5-5.2.0-10- sgadmin-standalone.zip
[=================================================] 100%
ERROR: `elasticsearch` directory is missing in the plugin zip
I downloaded zip/tar.gz from this maven repository of search gaurd.
Is anyone also facing the same issue. If not, kindly help in solving this one.
Download this file from maven to /home/xxxx:
https://oss.sonatype.org/content/repositories/releases/com/floragunn/search-guard-5/5.2.0-11/search-guard-5-5.2.0-11.zip
Install it:
bin/elasticsearch-plugin install -b file:///home/xxxx/search-guard-5-5.2.0-11.zip
Other releases are available here: https://oss.sonatype.org/content/repositories/releases/com/floragunn/
Has anyone installed logstash-input-jmx (or any other "community" plugin for Logstash) in RHEL?
How did you manage the Ruby/gem dependencies?
After installing a recent version of Ruby, RubyGems, and Rails, the development/runtime gem dependencies have to be resolved, but attempting to install any of the gems mentioned in here, would lead to endless missing dependencies ...
Are there any packaged versions of Logstash JMX plugin around?
Thanks,
Babak.
Welcome to the house of fun. What I did was, which may be out of your scope but it might help....
$logstash_location/bin/logstash-plugin list
(as you might expect this shows what can be installed to your instance)
$logstash_location/bin/logstash-plugin install logstash-input-jmx
The plugin itself should resolve its own dependencies. I was having all sort of problems in Docker, but I had to build my Logstash tarball external to the machine. Make sure connectivity is ok too. It's a service, not local to your machine.
This post clearly describes the steps required to install a "community" logstash plugin, and can be used to install jmx plugin on RHEL. I've tried it on RHEL 7.3.
The main thing to consider is that in RHEL 7.3 the logstash can be found under /usr/share/logstash/ rather than /opt/logstash
I've been stuck on this for about a day. I downloaded and installed ElasticSearch 2.3.3 on my redhat vm. I am trying to follow the get started video on the site but not one of the installation commands from the video has worked for me, so I've tried following installation methods from the guide.
I downloaded the rpm and entered the following commands to install:
cd /usr/share/elasticsearch
bin/plugin -DproxyHost=... -DproxyPort=... install license
bin/plugin -DproxyHost=... -DproxyPort=... install marvel-agent
I then started my ElasticSearch node and went to localhost:9200/_plugin/marvel but nothing popped up.
I then thought it might be because I only downloaded the agent and not marvel, so I tried installing marvel from the cmd line, but I constantly getting a FileNotFound exception
bin/plugin -DproxyHost=... -DproxyPort=... install marvel
I then manually downloaded marvel-2.3.3.tar.gz and tried installing it manually
bin/plugin -DproxyHost=... -DproxyPort=... install --verbose file://.../marvel-2.3.3.tar.gz
but I'm told the zip is missing a plugin-descriptor.properties file
Verifying file:/home1/isuser/Downloads/marvel-2.3.3.tar.gz checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip
I then unzipped marvel-2.3.3.tar, added a .properties file, rezipped, and reinstalled. still got the same error.
I really need some help. This is very frustrating. I still need to install Kibana but if that installation is anything near as infuriating as this one I'm not looking forward to it...
EDIT:
I got past this error by moving the .properties file from within the marvel directory to the same level, but now I'm getting a new error. it keeps giving me a FileNotFoundException.
[root#den01ejm elasticsearch]# bin/plugin install -DproxyHost=www-proxy.us.oracle.com -DproxyPort=80 --verbose /home1/isuser/Desktop/up/marvel-2.3.3.tar.gz
-> Installing /home1/isuser/Desktop/up/marvel-2.3.3.tar.gz...
Trying https://download.elastic.co//home1/home1-isuser.zip ...
Failed: FileNotFoundException[https://download.elastic.co//home1/home1-isuser.zip]; nested: FileNotFoundException[https://download.elastic.co//home1/home1-isuser.zip];
Trying https://search.maven.org/remotecontent?filepath=/home1/isuser/home1-isuser.zip ...
Failed: FileNotFoundException[https://search.maven.org/remotecontent?filepath=/home1/isuser/home1-isuser.zip]; nested: FileNotFoundException[https://search.maven.org/remotecontent?filepath=/home1/isuser/home1-isuser.zip];
Trying https://oss.sonatype.org/service/local/repositories/releases/content//home1/isuser/home1-isuser.zip ...
Failed: FileNotFoundException[https://oss.sonatype.org/service/local/repositories/releases/content//home1/isuser/home1-isuser.zip]; nested: FileNotFoundException[https://oss.sonatype.org/service/local/repositories/releases/content//home1/isuser/home1-isuser.zip];
Trying https://github.com//home1/archive/isuser.zip ...
Failed: FileNotFoundException[https://github.com//home1/archive/isuser.zip]; nested: FileNotFoundException[https://github.com//home1/archive/isuser.zip];
Trying https://github.com//home1/archive/master.zip ...
Failed: FileNotFoundException[https://github.com//home1/archive/master.zip]; nested: FileNotFoundException[https://github.com//home1/archive/master.zip];
ERROR: failed to download out of all possible locations..., use --verbose to get detailed information
cd /usr/share/elasticsearch
bin/plugin -DproxyHost=... -DproxyPort=... install license
bin/plugin -DproxyHost=... -DproxyPort=... install marvel-agent
This is all that needs to be installed in Elasticsearch 2.x to make Marvel monitor Elasticsearch. The Marvel Agent monitors the node, with a dependency on the license plugin, then ships the data where ever it's configured to go (to the same node, by default).
The third plugin that you're trying to install is the Marvel UI component. Starting with ES 2.x, the UI component is a part of Kibana and not Elasticsearch.
As such, you should be installing that plugin in Kibana. As you are trying to use Marvel 2.3.3, you should be using Kibana 4.5 (so Kibana 4.5.1 as of now). From Kibana, you can install it there:
bin/kibana plugin --install elasticsearch/marvel/latest
NOTE: The syntax of the plugin commands will change in Elasticsearch 5.x and Kibana 5.x and this will be greatly simplified to literally being the same file for both Elasticsearch and Kibana, but both applications will still be needed for both sides: monitoring and the UI.
bin/elasticsearch-plugin install x-pack
bin/kibana-plugin install x-pack
I have tried the online and offline instructions for installing marvel and sense, but nothing works. I have ES 2.2.1 and Kibana 4.4.2
I tried
plugin install marvel-agent
but i get
Installing marvel-agent...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.2.0/marvel-agent-2.2.0.zip ...
ERROR: failed to download out of all possible locations..., use --verbose to get detailed information
same error for
bin/plugin install license
bin/kibana plugin --install elasticsearch/marvel/latest
so then i tried downloading the files and follow instructions on
https://www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation
i tried
plugin install file:///C:/Users/xyz/ElasticSearch/zips/marvel-agent-2.2.0.zip
but i get
-> Installing from file:/C:/Users/xyz/ElasticSearch/zips/marvel-agent-2.2.0.zip...
Trying file:/C:/Users/xyz/ElasticSearch/zips/marvel-agent-2.2.0.zip ...
Downloading ..DONE
Verifying file:/C:/Users/xyz/ElasticSearch/zips/marvel-agent-2.2.0.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
same for
plugin install file:///C:/Users/xyz/ElasticSearch/zips/license-2.2.0.zip
and finally for
kibana plugin --install marvel --url file:///C:/Users/xyz/ElasticSearch/zips/marvel-2.2.0.tar.gz
i get
Installing marvel
Attempting to transfer from file:///C:/Users/xyz/ElasticSearch/zips/marvel-2.2.0.tar.gz
Error: ENOTFOUND
Plugin installation was unsuccessful due to error "No valid url specified."
Please point out what i am doing wrong.
For me, the last installation instruction required a different format for the url.
file://c:/Temp/ElasticSearch/marvel-2.2.1.tar.gz. Basically, remove one of the forward slashes.
download the Sense tar file from the following link: https://download.elasticsearch.org/elastic/sense/sense-latest.tar.gz
open cmd form kibana\bin location
then you can install Sense using the following command:
kibana plugin -i sense -u file:PATH_TO_SENSE_TAR_FILE
I want to install Logstash for NodeJs on windows 7, but I am not able to find proper steps for the same.
Can any one please help!
There is the option of node-logstash if you want a node.js alternative to Logstash. This isn't something I'm using myself (I'm using nxlog in Windows instead) but it looks like a decent alternative to the standard JRuby Logstash if you need to forward logs from Windows.
Instructions from the readme are below:
Installation
Install NodeJS, version >= 0.10, or io.js.
Install build tools
Debian based system: apt-get install build-essential
Centos system: yum install gcc gcc-c++ make
Install zmq dev libraries: This is required to build the node zeromq module.
Debian based system: apt-get install libzmq1. Under recent releases, this package is present in default repositories. On ubuntu lucid, use this ppa. On debian squeeze, use backports.
Centos 6: yum install zeromq zeromq-devel. Before, you have to add the rpm zeromq repo : curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
Clone repository: git clone git://github.com/bpaquet/node-logstash.git && cd node-logstash
Install dependencies: npm install.
The executable is in bin/node-logstash-agent
You have scripts in dists folder to build packages. Actually, only debian is supported.
As per the comment, logstash has nothing to do with nodejs.
What you're looking to do is install Logstash on Windows, something that you can find out about by using google, there will be loads of guides out there describing how to do this.
You would then need to configure logstash to look in the right location for the log files it needs to process, and then set up filters to handle nodejs style logs (which as far as I understand aren't very well standardised). You then need to configure an output (logstash is essentially a unix pipe on steroids and needs somewhere to save the logs it has processed). Elasticsearch is the most common thing to save logs to.
Personally, in my environment, I would install logstash on a CentOS server, as it's a well established process, and ship the logs from your Windows 7 machine to the logstash server using either logstash forwarder or nxlog. That way you can have logs coming in from a number of different sources and you can still reboot your Windows machine every few days as required by Windows update without your logstash server going down.