fluent custom plugin registration throws "Unknown output plugin" - ruby

I am trying to modify an existing Fluentd plugin and use it with my customization. So I :
git cloned the plugin repo from github. git clone <url>
Edited my changes
gem build plugin.gemspec
gem install plugin.gem
Created a config mentioning the type of plugin
Ran fluentd -c <fluend_conf.conf> -vv
And I ended up getting this in fluentd log(fluent-plugin-swift is the plugin I edited and the type of the plugin is 'swift'):
2017-03-22 15:00:27 +0000 [info]: fluent/engine.rb:126:block in configure:
gem 'fluent-plugin-swift' version '0.0.1'
2017-03-22 15:00:27 +0000 [error]: fluent/supervisor.rb:369:rescue in
main_process: config error file="fluend_conf.conf" error="Unknown output plugin
'swift'. Run 'gem search -rd fluent-plugin' to find plugins"
EDIT-1: For clarifications
I have also checked the naming convention of the plugin that is to be registered.
The plugin is present in <base_dir>/lib/fluent/plugin/.
Also the name of file is out_swift.rb. And finally
inside the file, the plugin gets registered as:
Fluent::Plugin.register_output('swift', self)
Also is there any way, I can check the list of registered plugins, and the name it uses to get registered to fluend?
NOTE:
The same had previously worked for me, I had to re-do this as someone had formatted the machine I had my changes.
Please let me what is wrong in my approach?

The problem could be with the installation path, Please check the following, :
Type gem environment
You will get to see the INSTALLATION DIRECTORY like:
RubyGems Environment:
- RUBYGEMS VERSION: 2.6.8
- RUBY VERSION: 2.4.0 (2016-12-24 patchlevel 0) [x86_64-linux]
- INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/2.4.0
- USER INSTALLATION DIRECTORY: /root/.gem/ruby/2.4.0
- RUBY EXECUTABLE: /usr/local/bin/ruby
- EXECUTABLE DIRECTORY: /usr/local/bin
- SPEC CACHE DIRECTORY: /root/.gem/specs
- SYSTEM CONFIGURATION DIRECTORY: /usr/local/etc
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
Go inside the INSTALLATION DIRECTORY directory, now do
cd gem/<ur_custom_plugin>
Now check whether, your files are there or not.
If not, you got your answer.
SOLUTION:
Simply copy your files to this directory and start fluentd

As the above mentioned, this may related to which gem you used to install the plugin, for my case in CentOS7, the td-agent-gem as below works:
td-agent-gem install <pluginname>

install ruby development environment
brew install rbenv
now run fluentD from ruby environment
~/.rbenv/shims/fluentd
this refers /etc/fluent/fluent.conf
Installing the plugin from ruby environment
~/.rbenv/shims/gem install fluent-plugin-rewrite-tag-filter

Related

Why doesn't RubyMine 2018.2 detect my gems?

I'm trying to debug a Rails project within RubyMine v2018.2 (2018.2.1, 2018.2.4). When I start the editor I get a warning:
RubyMine Gem Manager: RubyMine has detected that some of the gems required for 'project' are not installed.
When I try to run RubyMine's Bundler install command I'm told
bash -c "RBENV_VERSION=2.5.0 /usr/bin/rbenv exec bundle install -V"
Activating bundler (< 2) failed:
Could not find 'bundler' (< 2) among 28 total gem(s)
Checked in 'GEM_PATH=/home/brad/.gem/ruby/2.5.0:/var/lib/gems/2.5.0:/usr/lib/x86_64-linux-gnu/rubygems-integration/2.5.0:/usr/share/rubygems-integration/2.5.0:/usr/share/rubygems-integration/all', execute `gem env` for more information
To install the version of bundler this project requires, run `gem install bundler -v '< 2'`
...and when I try to run or debug my application from the IDE I'm told "Could not find bundler (version) required by your project-path/Gemfile.lock."
Why doesn't RubyMine detect my installed Gems?
My environment
Ubuntu 18.04.01
zsh 5.4.2
rbenv 1.0.0
rbenv-binstubs # 8ecb7ca8
Rubymine 2018.2.4
I'm using rbenv to manage my Ruby versions. The system has Ruby 2.5.1 installed, but my project depends on 2.5.0. I've set 2.5.0 as the rbenv global version. Configuration for how rbenv chooses a Ruby version seems okay; I get the following results from the noninteractive commands:
> echo $RBENV_VERSION
(no response)
> cat ~/.rbenv/version
2.5.0
> rbenv global
2.5.0
After noticing rbenv wasn't working properly in some RubyMine commands I moved the eval "$(rbenv init -)" setup step from by ~/.zshrc up to ~/.zshenv and ~/.profile so it will also apply to bash and zsh login shells (per guidance here). I'm now getting matching results for interactive and login shells, but still getting unexpected behavior from the Bundle install command within RubyMine.
What is working
My application installs, builds and runs just fine from an interactive shell.
RubyMine's Ruby SDK settings correctly detect rvm and its installed versions, and all of the installed gems for that version.
What I've tried
The output from the bundle install command claims it's running
bash -c "RBENV_VERSION=2.5.0 /usr/bin/rbenv exec bundle install"
When I run this from my interactive shell or with RubyMine's "Run anything" feature (login shell) this command works fine - as does any of the following:
RBENV_VERSION=2.5.0 /usr/bin/rbenv exec bundle install
/usr/bin/rbenv exec bundle install
rbenv exec bundle install
bundle install
So I can't figure out what context RubyMine is using to run this command. It's clear from the error message that the GEM_PATH is incorrect, causing RubyMine to look in the wrong place for bundler, but I don't know where it's getting this configuration. If I bring up the "RubyGems Environment" view in RubyMine I get:
RUBYGEMS VERSION: 2.7.3
RUBY VERSION: 2.5.0 (2017-12-25 patchlevel 0) [x86_64-linux]
INSTALLATION DIRECTORY: /home/brad/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0
USER INSTALLATION DIRECTORY: /home/brad/.gem/ruby/2.5.0
RUBY EXECUTABLE: /home/brad/.rbenv/versions/2.5.0/bin/ruby
EXECUTABLE DIRECTORY: /home/brad/.rbenv/versions/2.5.0/bin
SPEC CACHE DIRECTORY: /home/brad/.gem/specs
SYSTEM CONFIGURATION DIRECTORY: /home/brad/.rbenv/versions/2.5.0/etc
RUBYGEMS PLATFORMS:
ruby
x86_64-linux
GEM PATHS:
/home/brad/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0
/home/brad/.gem/ruby/2.5.0
(...)
SHELL PATH:
/home/brad/.rbenv/versions/2.5.0/bin
/usr/lib/rbenv/libexec
/home/brad/.rbenv/plugins/ruby-build/bin
/home/brad/.rbenv/plugins/rbenv-gemset/bin
/home/brad/.rbenv/plugins/rbenv-binstubs/bin
(...rest of PATH...)
----------------------
IDE: RubyMine 2018.2.4, build #RM-182.4505.55
OS: Linux 4.15.0-38-generic[amd64]
Java: 1.8.0_152-release-1248-b8
RubyMine SDK Environment:
Sdk: rbenv: 2.5.0
Sdk Version: ver.2.5.0p0 ( revision 61468) p0
Ruby Interpreter: /home/brad/.rbenv/versions/2.5.0/bin/ruby
RVM Sdk: Rbenv sdk, gemset nullSdk Language Level: 2.5
Sdk Load Path:
~/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib
~/.rbenv/versions/2.5.0/lib/ruby/site_ruby/2.5.0
~/.rbenv/versions/2.5.0/lib/ruby/site_ruby/2.5.0/x86_64-linux
~/.rbenv/versions/2.5.0/lib/ruby/site_ruby
~/.rbenv/versions/2.5.0/lib/ruby/vendor_ruby/2.5.0
~/.rbenv/versions/2.5.0/lib/ruby/vendor_ruby/2.5.0/x86_64-linux
~/.rbenv/versions/2.5.0/lib/ruby/vendor_ruby
~/.rbenv/versions/2.5.0/lib/ruby/2.5.0
~/.rbenv/versions/2.5.0/lib/ruby/2.5.0/x86_64-linux
~/.local/share/JetBrains/Toolbox/apps/RubyMine/ch-0/182.4505.55/rubystubs25
Sdk Gem paths:
~/.gem/ruby/2.5.0/gems
~/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/bundler/gems
~/.bundle/ruby/2.5.0/gems
~/.bundle/ruby/2.5.0/bundler/gems
~/.gem/ruby/2.5.0/bundler/gems
~/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems
(full list of gems used for project)
So I think that last bit - "Sdk Gem paths" - might be the problem point, but I don't see any way to configure it.
I'm also already aware of the YouTrack issue "Rubymine 2018.2 release does not take $GEM_HOME environment variable" and don't think it's quite what's happening here. It also claims to be fixed in 2018.3 EAP1, so I installed 2018.3 EAP7 but am still seeing this problem.
I got things working! By signing out and back in. :facepalm:
To be specific, I think the key steps were
Move the $(rbenv init -) step from my .zshrc up to my .zshenv and/or to .profile
Signing out and back in.
The key insight was noticing that ~/.rbenv/shims was missing from the RubyMine Gem Environment despite showing up in both interactive and login shells (in and out of RubyMine), and realizing that while new shells were picking up my .profile changes, RubyMine itself was probably inheriting its environment variables from my session and I'd need to log out/in to to pick up the changes.
I now see ~/.rbenv/shims in my RubyMine Gem Environment, and both the Bundler Install command and my run configurations are working again.
Thank you Casper for pointing me in the right direction!
Been where you are before a bunch of times.
Under the list of scripts you can run, click "Edit Configurations". Should be in the main toolbar.
Double check that you have "Run the script in context of bundle (bundle exec)" checked on the bundler tab. This is SUPER important and is not checked by default. Make sure you check for any rake, rails or irb configurations.
Keep in mind that the run configuration is where you can override the settings for the project. It will create all kinds of problems if this is not set appropriately.
Hope this helps! Bit me a bunch of times.

Configure gem installation directory to match user installation directory

I'm familiar with the node/npm way, and trying to get set up using ruby/gem/bundler as well.
The output from gem env includes these lines:
- RUBYGEMS VERSION: 2.7.6
- RUBY VERSION: 2.5.1 (2018-03-29 patchlevel 57) [x86_64-darwin17]
- INSTALLATION DIRECTORY: /opt/local/lib/ruby2.5/gems/2.5.0
- USER INSTALLATION DIRECTORY: /Users/ssterling3/.gem/ruby/2.5.0
(I don't get why gems for ruby 2.5.1 would be installed in dir named 2.5.0, but that's not what this question is about.)
I don't want gem to ever touch anything outside of $HOME, and I want to do so in a way that I'll never have to edit even when I upgrade ruby or move the content of $HOME to a different computer where I have a different username.
(Sure, I'll have to reinstall all the gems or something, that's also not what this question is about.)
I think the right way to do it is to override the default INSTALLATION DIRECTORY and set it to always match the default USER INSTALLATION DIRECTORY.
How can I do that?
I know I can override the INSTALLATION DIRECTORY by setting $GEM_HOME, but I don't know how to get gem to output the USER INSTALLATION DIRECTORY in a way I won't have to parse before setting $GEM_HOME. I can get the INSTALLATION DIRECTORY from gem env gemdir, but gem env --user-install gemdir says Invalid option.
(Why one value has three dissimilar names and a closely related value only has the least usable analogous name are both also not what this question is about.)
I know I can configure some things in $HOME/.gemrc, but the documentation and examples I found didn't make it clear to me how to manipulate the INSTALLATION DIRECTORY. It sounded like I might be able to do it by adding the line gem: --user-install, but maybe it has to be at least three lines for install, uninstall, and update, but adding the line with gem: doesn't change the output of gem env so maybe none of those will really do what I want.
You can update environment variables. For my output:
- RUBY VERSION: 2.5.1 (2018-03-29 patchlevel 57) [x86_64-linux]
- INSTALLATION DIRECTORY: /home/ivan/.rvm/gems/ruby-2.5.1
- USER INSTALLATION DIRECTORY: /home/ivan/.gem/ruby/2.5.0
- RUBY EXECUTABLE: /home/ivan/.rvm/rubies/ruby-2.5.1/bin/ruby
- EXECUTABLE DIRECTORY: /home/ivan/.rvm/gems/ruby-2.5.1/bin
- SPEC CACHE DIRECTORY: /home/ivan/.gem/specs
- SYSTEM CONFIGURATION DIRECTORY: /home/ivan/.rvm/rubies/ruby-2.5.1/etc
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- /home/ivan/.rvm/gems/ruby-2.5.1
- /home/ivan/.rvm/gems/ruby-2.5.1#global
my file is located here:
home/ivan/.rvm/gems/ruby-2.5.1/environment
Note the environment file is into GEM PATHS first path.
Important: I should reload variables by:
source home/ivan/.rvm/gems/ruby-2.5.1/environment
or faster in most of shells:
. home/ivan/.rvm/gems/ruby-2.5.1/environment
I ended up putting this atrocity in my init scripts:
if [ ! -z "`which gem`" ]; then
GEM_HOME=`prefix=" - USER INSTALLATION DIRECTORY: "; line=\`gem env | grep "^$prefix"\`; suffix=${line#$prefix}; echo $suffix`
fi;

jekyll-multiple-languages-plugin cannot be found

I'm newbie to Ruby and Jekyll. Recently I've tried to install Jekyll Multiple Languages Plugin onto my GitLab Pages instance. I've managed to successfully add
'gem install jekyll-multiple-languages-plugin
bundle install'
into the .gitlab-ci.yml but when I try to add
gems:
jekyll-multiple-languages-plugin
into _config.yml in order to use it on the site my commit fails with the following error:
Using jekyll-watch 1.5.0
Using jekyll 3.4.3
Bundle complete! 3 Gemfile dependencies, 20 gems now installed.
Bundled gems are installed into /usr/local/bundle.
$ jekyll build -d public
Configuration file: /builds/myusername/forty-jekyll-theme/_config.yml
Dependency Error: Yikes! It looks like you don't have jekyll-multiple-languages-plugin or one of its dependencies installed. In order to use Jekyll as currently configured, you'll need to install this gem. The full error message from Ruby is: 'cannot load such file -- jekyll-multiple-languages-plugin' If you run into trouble, you can find helpful resources at https://jekyllrb.com/help/!
jekyll 3.4.3 | Error: jekyll-multiple-languages-plugin
ERROR: Job failed: exit code 1
I have used this method because any attempt to edit Gemfile ends up in commit error. I've also tried to do all presented steps except adding bundle install. In that case I get the same error, but the beginning looks like:
18 gems installed
$ gem install jekyll-multiple-languages-plugin
Successfully installed jekyll-multiple-languages-plugin-1.5.1
1 gem installed
$ jekyll build -d public
I did not manage to find the way to solve it on Stack Overflow nor other sites. For example this did not help
With the awesome support of allejo from Jekyll's IRC channel I've managed to solve the problem. Here are the steps:
I've used Gemfile. Now it looks like
source 'https://rubygems.org'
gem 'jekyll'
group :jekyll_plugins do
gem 'jekyll-multiple-languages-plugin'
end
The second modified thing was .gitlab-ci.yml (the first line - image: ruby - is also included)
image: ruby
pages:
stage: build
script:
# - gem install jekyll
- gem install bundler
# - gem install jekyll-multiple-languages-plugin
- bundle install
- bundle exec jekyll build -d public
# - jekyll build -d public
artifacts:
paths:
- public
only:
- master
It produced a bug, but it was only caused by the lack of declared language which is part of standard plugin configuration.

The 'puma' native gem requires installed build tools

I have got this error when running bundle install
Gem::InstallError: The 'puma' native gem requires installed build tools.
Please update your PATH to include build tools or download the DevKit
from 'http://rubyinstaller.org/downloads' and follow the instructions
at 'http://github.com/oneclick/rubyinstaller/wiki/Development-Kit'
I'm using RailsInstaller which already has DevKit.
I didn't have this problem before but I seem to get it after I changes the ruby version from 2.2.4 to 2.2.5. I'm using a windows computer. Any idea how to solve this issue?
I solved the problem by following these instruction:
download devkit from here.
Install the file in DevKit folder
open cmd and navigate to the folder
Copy the path of the ruby folder you installed in C drive and paste it in the DevKit\config.yml. Change ‘/’ in the path to ‘\’.
It should look like this:
# Example:
#
# ---
# - C:/ruby19trunk
# - C:/ruby192dev
#
---
- C:\Ruby22
Run ruby dk.rb install
Back to your rails folder, run bundle install and you should not have any errors

Ruby Passenger: no such file to load bundler

I installed Phusion Passenger with Nginx, configured Nginx to point to the right directory, Then I ran webapp directory and this has downloaded the gemfiles, but it can't find the gems.
When I visit the site I get the standard Passenger error page which says:
Error message:
no such file to load -- bundler
Here's the full error: http://tinypic.com/view.php?pic=vpx36r&s=7
I've do a gem install bundler so I know bundler is installed, but I think it's looking in the wrong place for the gems.
It appears as if Passenger has installed ruby-enterprise-1.8.7 and it looks like 1.8 was already installed in this box.
gem env gives me the following:
- RUBYGEMS VERSION: 1.4.2
- RUBY VERSION: 1.8.7 (2009-12-24 patchlevel 248) [x86_64-linux]
- INSTALLATION DIRECTORY: /opt/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8
- RUBY EXECUTABLE: /opt/local/ruby-enterprise-1.8.7-2010.01/bin/ruby
- EXECUTABLE DIRECTORY: /opt/local/ruby-enterprise-1.8.7-2010.01/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- /opt/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8
- /root/.gem/ruby/1.8
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://rubygems.org/
From what I've read it appears to be a path issue, but I don't know what the best course of action is to fix it.
Here's the output of
which ruby:
/opt/local/ree/bin/ruby
which bundle:
/opt/local/ree/bin/bundle
I've run bundle install in that directory and it's given me:
Your bundle is complete! It was installed into ./vendor/bundle
it looks like you have some path issues (as you already stated). try to check why you have two different paths for your ruby installation.
I mean, according to your gem env output, you have some installation on
/opt/local/ruby-enterprise-1.8.7-2010.01/
but you also have ruby and bunlder under
/opt/local/ree/
so, start from here to check if all the paths are correct and/or you haven't any double installations (well, it's just enough to check if your ENVs are coherent).
Another issue could be related to your nginx.conf, you should have something like this:
[...]
http {
passenger_root /your/path/to/passenger/gem;
passenger_ruby /path/to/ruby;
[...]
}
UPDATE (just saw additional comments with nginx.conf):
as you can see, there's a problem with paths: you have two paths for ruby and gems installs:
/opt/local/ruby-enterprise....
and
/opt/local/ree
you should remove the latter (not phisically, just review ENVs to point to the first one)
EDIT:
to change your env, you can append the following line to your /etc/bash.bashrc (or whaterver file is loaded from your shell by default):
PATH="/opt/local/ruby-enterprise-1.8.7-2010.01/bin:$PATH"
then logout and login, or for a quick test, launche this command from shell:
source /etc/bash.bashrc
now, retry to see what binary is used by default with:
which ruby
which gem
which rake
gem env
...
everything should have the /opt/local/ruby-enterprise-1.8.7-2010.01/ prefix (that's what is used from nginx and passenger configs).
UPDATE2: from the comments, it turned out that you have at least 3 ruby installs:
ruby from your package manager in /usr/lib/ruby/ (are you using ubuntu linux?)
a ruby in /opt/local/ree/
a ruby in /opt/local/ruby-enterprise-....
at this point, the best thing to do is:
uninstall all the rubys under /opt/local/ and all their gems
uninstall (purge) system provided ruby packages (you don't need it)
install and setup RVM: https://rvm.io
re-install passenger using RVM: https://rvm.io/integration/passenger/
fix nginx.conf to use passenger path and ruby

Resources