I am trying to implement a custom fluentd parser plugin. As suggested in the docs, I have created the following file called parser_unordered_multiline.rb and put it in the location /etc/td-agent/plugin/.
# #Package: Fluentd
# #Module: Unordered Multiline Parser Plugin
# #Author: Amyth Singh <mail#amythsingh.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
module Fluent
class TextParser
class UnorderedMultilineParser < Parser
Plugin.register_parser('unordered_multiline', self)
## Define configuration params
config_param :delimiter, :string, :default => " "
config_param :time_format, :string, :default => nil
## configures our parser plugin
def configure(conf)
super
if #delimiter.length != 1
raise ConfigError, "delimiter must be a single character. #{#delimiter} is not."
end
#time_parser = TimeParser.new(#time_format)
end
def parse(text)
time, key_values = text.split(" ", 2)
time = #time_parser.parse(time)
record = {}
key_values.split(#delimiter).each { |kv|
k, v = kv.split("=", 2)
record[k] = v
}
yield time, record
end
end
end
end
I am using the same code given in the example in the documentation (just with a different name) but the fluentd logs show the error:
unexpected error error="uninitialized constant Fluent::TextParser::Parser"
Following is the full stack trace
2015-12-23 14:38:23 +0530 [info]: starting fluentd-0.10.50
2015-12-23 14:38:23 +0530 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-12-23 14:38:23 +0530 [error]: unexpected error error="uninitialized constant Fluent::TextParser::Parser"
2015-12-23 14:38:23 +0530 [error]: /etc/td-agent/plugin/parser_unordered_multiline.rb:21:in `<class:TextParser>'
2015-12-23 14:38:23 +0530 [error]: /etc/td-agent/plugin/parser_unordered_multiline.rb:20:in `<module:Fluent>'
2015-12-23 14:38:23 +0530 [error]: /etc/td-agent/plugin/parser_unordered_multiline.rb:19:in `<top (required)>'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/plugin.rb:59:in `block in load_plugin_dir'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/plugin.rb:57:in `each'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/plugin.rb:57:in `load_plugin_dir'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/engine.rb:129:in `load_plugin_dir'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:369:in `block in init_engine'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:366:in `each'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:366:in `init_engine'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:129:in `block in start'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:246:in `call'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:246:in `main_process'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:221:in `block in supervise'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:220:in `fork'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:220:in `supervise'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:126:in `start'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/lib/fluent/command/fluentd.rb:160:in `<top (required)>'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.50/bin/fluentd:6:in `<top (required)>'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `load'
2015-12-23 14:38:23 +0530 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
2015-12-23 14:38:23 +0530 [error]: /usr/sbin/td-agent:7:in `load'
2015-12-23 14:38:23 +0530 [error]: /usr/sbin/td-agent:7:in `<main>'
2015-12-23 14:38:23 +0530 [info]: process finished code=256
2015-12-23 14:38:23 +0530 [warn]: process died within 1 second. exit.
And this is how my configuration file looks like.
<source>
type tail
path /var/log/mail/mail*
pos_file /var/log/td-agent/tmp/access.log
exclude_path ["/var/log/mail/*.gz", "/var/log/mail/*.zip"]
read_from_head true
time_format %M %d %H:%M:%S
port 5140
format unordered_multiline
tag mail
</source>
<match mail>
type copy
<store>
type elasticsearch
logstash_format true
flush_interval 30s
</store>
</match>
I am fairly new to ruby and fluentd, Can anybody tell me what this error is all about ?
update
According to the plugin management docs, even the path seems correct
Related
Behold:
bear#ptah:~/Pictures/Wallpapers
$ stat /home/bear/Dropbox/.#NineFoxes.org
File: /home/bear/Dropbox/.#NineFoxes.org -> polar#fennec.3781:1659418908
Size: 28 Blocks: 0 IO Block: 4096 symbolic link
Device: 10306h/66310d Inode: 57016400 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ bear) Gid: ( 1000/ bear)
Access: 2022-09-23 12:31:49.280214712 -0700
Modify: 2022-08-07 22:25:24.000000000 -0700
Change: 2022-09-23 12:31:49.532216732 -0700
Birth: 2022-09-23 12:31:49.280214712 -0700
bear#ptah:~/Pictures/Wallpapers
$ stat /home/bear/Dropbox/foo.log
File: /home/bear/Dropbox/foo.log
Size: 1471 Blocks: 8 IO Block: 4096 regular file
Device: 10306h/66310d Inode: 57016411 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ bear) Gid: ( 1000/ bear)
Access: 2022-09-23 12:31:49.280214712 -0700
Modify: 2022-09-10 23:28:04.000000000 -0700
Change: 2022-09-23 12:31:50.116221410 -0700
Birth: 2022-09-23 12:31:49.280214712 -0700
bear#ptah:~/Pictures/Wallpapers
$ irb
irb(main):001:0> File.stat '/home/bear/Dropbox/foo.log'
=> #<File::Stat dev=0x10306, ino=57016411, mode=0100644, nlink=1, uid=1000, gid=1000, rdev=0x0, size=1471, blksize=4096, blocks=8, atime=2022-09-23 12:31:49.280214712 -0700, mtime=2022-09-10 23:28:04 -0700, ctime=2022-09-23 12:31:50.11622141 -0700>
irb(main):002:0> File.stat '/home/bear/Dropbox/.#NineFoxes.org'
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
2: from (irb):2
1: from (irb):2:in `stat'
Errno::ENOENT (No such file or directory # rb_file_s_stat - /home/bear/Dropbox/.#NineFoxes.org)
irb(main):003:0>
The filename is simply correct; that is the file and Ruby fails to stat it. Linux can, everything else can. Ruby can stat a file without a hash in it, also. What is needed to render this file usable to Ruby? (No, renaming is not an option; I need to stat the file.)
Note: This is the filename that Dir.glob finds, also. This is the filename that Ruby thinks it has.
irb(main):003:0> x = Dir.glob('Dropbox/*.org', File::FNM_DOTMATCH)
=> ["Dropbox/NineFoxes.org", "Dropbox/.#NineFoxes.org", "Dropbox/Elf.org", "Dropbox/Midra.org"]
irb(main):004:0> File.stat(x[0])
=> #<File::Stat dev=0x10306, ino=57016406, mode=0100744, nlink=1, uid=1000, gid=1000, rdev=0x0, size=38102, blksize=4096, blocks=80, atime=2022-09-23 12:31:49.280214712 -0700, mtime=2022-07-29 22:00:02 -0700, ctime=2022-09-23 12:31:50.11622141 -0700>
irb(main):005:0> File.stat(x[1])
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
2: from (irb):5
1: from (irb):5:in `stat'
Errno::ENOENT (No such file or directory # rb_file_s_stat - Dropbox/.#NineFoxes.org)
irb(main):006:0> File.stat(x[2])
=> #<File::Stat dev=0x10306, ino=57016408, mode=0100644, nlink=1, uid=1000, gid=1000, rdev=0x0, size=8416, blksize=4096, blocks=24, atime=2022-09-23 12:31:49.280214712 -0700, mtime=2022-09-19 18:08:32 -0700, ctime=2022-09-23 12:31:50.120221442 -0700>
irb(main):007:0>
Additional: This is Linux, MX Linux 21, ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-gnu]. The shell is Bash.
File.stat just calls File::Stat.new and that will raise an exception if the file doesn't exist; this is the behaviour you're seeing.
Note that /home/bear/Dropbox/.#NineFoxes.org is a symlink (see the Access: 0777/lrwxrwxrwx in stat's output). Ruby's File.stat documentation doesn't say if it follows symlinks or not but File.lstat says:
Same as File::stat, but does not follow the last symbolic link. Instead, reports on the link itself.
so it is implied that File.stat will follow the symlink and stat what it points to.
So the /home/bear/Dropbox/.#NineFoxes.org symlink does exist but what the symlink references doesn't exist.
I have deployed a elasticsearch 2.2.0 Now, I'm sending logs to it using a td-agent 2.3.0-0.
The final on tag chain was..
<match extra.geoip.processed5.**>
type copy
<store>
type file
path /var/log/td-agent/sp_l5
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%dT%H%M%S%z
compress gzip
utc
</store>
<store>
type elasticsearch
host 11.0.0.174
port 9200
logstash_format true
logstash_prefix logstash_business
logstash_dateformat %Y.%m
flush_interval 5s
</store>
</match>
Now, I added a time out within the elasticsearch type block
request_timeout 45s
This is the td-agent.log with debug enabled.
2016-02-08 15:58:07 +0100 [info]: plugin/in_syslog.rb:176:listen: listening syslog socket on 0.0.0.0:5514 with udp
2016-02-08 15:59:03 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"}
2016-02-08 15:59:03 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"}
2016-02-08 16:03:33 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:03:33 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:03:35 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"}
2016-02-08 16:03:35 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"}
2016-02-08 16:08:05 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:08:05 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:08:09 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"}
2016-02-08 16:08:09 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"}
2016-02-08 16:12:40 +0100 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2016-02-08 15:59:04 +0100 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Could not push logs to Elasticsearch after 2 retries. read timeout reached" plugin_id="object:fd9738"
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:204:in `rescue in send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:194:in `send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:188:in `write'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:345:in `write_chunk'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:324:in `pop'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:321:in `try_flush'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:140:in `run'
2016-02-08 16:12:40 +0100 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2016-02-08 15:59:04 +0100 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Could not push logs to Elasticsearch after 2 retries. read timeout reached" plugin_id="object:1034980"
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:204:in `rescue in send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:194:in `send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:188:in `write'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:345:in `write_chunk'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:324:in `pop'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:321:in `try_flush'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:140:in `run'
I'm running this on AWS over ubuntu 14.04 c3.large. I have tested the access from td-agent machine creating a index, adding documents and deleting the index using curl without any problem. (To be sure, I opened all communications in my Security Groups)
Another test from td-agent machine...
root#bilbo:~# telnet 11.0.0.174 9200
Trying 11.0.0.174...
Connected to 11.0.0.174.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 320
{
"name" : "gandalf-gandalf",
"cluster_name" : "aaaa_dev",
"version" : {
"number" : "2.2.0",
"build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
"build_timestamp" : "2016-01-27T13:32:39Z",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
},
"tagline" : "You Know, for Search"
}
Connection closed by foreign host.
root#bilbo:~#
With strace I can see this
[pid 10774] connect(23, {sa_family=AF_INET, sin_port=htons(9200), sin_addr=inet_addr("11.0.0.174")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 10774] clock_gettime(CLOCK_MONOTONIC, {4876, 531526283}) = 0
[pid 10774] select(24, NULL, [23], NULL, {45, 0}) = 1 (out [23], left {44, 999925})
[pid 10774] fcntl(23, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 10774] connect(23, {sa_family=AF_INET, sin_port=htons(9200), sin_addr=inet_addr("11.0.0.174")}, 16) = 0
[pid 10774] fcntl(23, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 10774] write(23, "POST /_bulk HTTP/1.1\r\nUser-Agent: Faraday v0.9.2\r\nHost: 11.0.0.174:9200\r\nContent-Length: 4798\r\n\r\n", 97) = 97
[pid 10774] fcntl(23, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 10774] write(23, "{\"index\":{\"_index\":\"logstash_apache-2016.02.08\",\"_type\":\"fluentd\"}}\n{\"message\":\"Feb 8 16:18:47 bilbo ::apache::PRE::access: - 10.0.0.15 - control [08/Feb/2016:16:18:47 +0100] \\\"GET /status/memcached.php HTTP/1.1\\\" 200 1785 \\\"-\\\" \\\"check_http/v2.0 (monitoring-plugins 2.0)\\\" control-pre.fluzo.com:443 0\",\"n\":\"bilbo\",\"s\":\"info\",\"f\":\"local3\",\"t\":\"Feb 8 16:18:47\",\"h\":\"bilbo\",\"a\":\"apache\",\"e\":\"PRE\",\"o\":\"access\",\"ip\":\"-\",\"ip2\":\"10.0.0.15\",\"rl\":\"-\",\"ru\":\"control\",\"rt\":\"[08/Feb/2016:16:18:47 +0100]\",\"met\":\"GET\",\"pqf\":\"status/memcached.php\",\"hv\":\"HTTP/1.1\",\"st\":\"200\",\"bs\":\"1785\",\"ref\":\"-\",\"ua\":\"check_http/v2.0 (monitoring-plugins 2.0)\",\"vh\":\"control-pre.aaaa.com\",\"p\":\"443\",\"rpt\":\"0\",\"co\":null,\"ci\":null,\"la\":null,\"lo\":null,\"ar\":null,\"dm\":null,\"re\":null,\"#timestamp\":\"2016-02-08T16:19:47+01:00\"}\n{\"index\":{\"_index\":\"logstash_apache-2016.02.08\",\"_type\":\"fluentd\"}}\n{\"message\":\"Feb 8 16:18:47 bilbo ::apache::PRE::access: - 10.0.0.15 - control [08/Feb/2016:16:18:47 +0100] \\\"GET /status/core.php HTTP/1.1\\\" 200 1898 \\\"-\\\" \\\"c"..., 4798) = 4798
Seems that the connection is open.
To avoid confusion, I have installed td-agent on elasticsearch machine with the same output.
This is my elasticsearch configuration...
### MANAGED BY PUPPET ###
---
bootstrap:
mlockall: true
cluster:
name: aaaa0_dev
discovery:
zen:
minimum_master_nodes: 1
ping:
multicast:
enabled: false
unicast:
hosts:
- 11.0.0.174
gateway:
expected_nodes: 1
recover_after_nodes: 1
recover_after_time: 5m
hostname: gandalf
http:
compression: true
index:
store:
compress:
stored: true
type: niofs
network:
bind_host: 11.0.0.174
publish_host: 11.0.0.174
node:
name: gandalf-gandalf
path:
data: /var/lib/elasticsearch-gandalf
logs: /var/log/elasticsearch/gandalf
transport:
tcp:
compress: true
Any idea?
Thanks.
UPDATE
Against a real elasticsearch cluster works (3 nodes)
This is the configuration.
### MANAGED BY PUPPET ###
---
bootstrap:
mlockall: true
cluster:
name: aaaa
discovery:
zen:
minimum_master_nodes: 2
ping:
multicast:
enabled: false
unicast:
hosts:
- el0
- el1
- el2
gateway:
expected_nodes: 3
recover_after_nodes: 2
recover_after_time: 5m
hostname: kili
http:
compression: true
index:
store:
compress:
stored: true
type: niofs
network:
bind_host: 11.0.0.253
publish_host: 11.0.0.253
node:
name: kili-kili
path:
data:
- /var/lib/elasticsearch0
- /var/lib/elasticsearch1
logs: /var/log/elasticsearch/kili
transport:
tcp:
compress: true
However, kibana did create .kibana index into elasticsearch with one node. Also, y test this node with curl.
I have Ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux] and when I'm trying to use Wordmove, I'm getting following pull error:
▬▬ ✓ Pulling Database ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
local | mysqldump --host=localhost --user=USER --password=PASS wpDB > /var/www/local/wp-content/local-backup-1421774302.sql
remote | mysqldump --host=localhost --user=USER --password=PASS wpDB > /var/www/site/wp-content/dump.sql
remote | get: /var/www/site/wp-content/dump.sql /var/www/local/wp-content/dump.sql
remote | delete: /var/www/site/wp-content/dump.sql
local | adapt dump
/var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/sql_adapter.rb:44:in `gsub!': invalid byte sequence in UTF-8 (ArgumentError)
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/sql_adapter.rb:44:in `serialized_replace!'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/sql_adapter.rb:36:in `replace_field!'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/sql_adapter.rb:25:in `replace_vhost!'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/sql_adapter.rb:17:in `adapt!'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/deployer/base.rb:182:in `adapt_sql'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/deployer/ssh.rb:35:in `pull_db'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/cli.rb:47:in `block in pull'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/cli.rb:34:in `block in handle_options'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/cli.rb:32:in `each'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/cli.rb:32:in `handle_options'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/lib/wordmove/cli.rb:46:in `pull'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
from /var/lib/gems/1.9.1/gems/wordmove-1.2.0/bin/wordmove:6:in `<top (required)>'
from /usr/local/bin/wordmove:23:in `load'
from /usr/local/bin/wordmove:23:in `<main>'
Other websites run in the very same environment does not show this error. Does anyone have any idea, what can be casuing this error?
Add
sql_content.encode!('UTF-16', 'UTF-8', :invalid => :replace, :replace => '')
sql_content.encode!('UTF-8', 'UTF-16')
sql_content.force_encoding("UTF-8")
to this File,above the line 45
/root/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/wordmove-2.1.2/lib/wordmove/sql_adapter/default.rb
or this File
/var/lib/gems/2.7.0/gems/wordmove-5.2.2/lib/wordmove/sql_adapter/default.rb
Ref: https://github.com/welaika/wordmove/wiki/invalid-byte-sequence-in-UTF-8-while-pushing---pulling-db
I'm trying to execute the following command on my Heroku app:
heroku db:push postgres://username:password#localhost:5432/db_name
I successful used the same command a lot of times in the past three months but today it stopped working. It seems like it stopped working as soon as it reached 5MB in size (now it's about 8MB big). The thing is that I now started paying for the shared database so I should be allowed to have 20GB space.
This is the command output:
> heroku db:push postgres://username:password#localhost:5432/db_name --account java
Loaded Taps v0.3.23
Warning: Data in the app 'my-test-app' will be overwritten and will not be recoverable.
! WARNING: Potentially Destructive Action
! This command will affect the app: my-test-app
! To proceed, type "my-test-app" or re-run this command with --confirm my-test-app
> my-test-app
Sending schema
Schema: 100% |==========================================| Time: 00:00:46
Sending indexes
test_gift_rec: 100% |==========================================| Time: 00:00:01
test_person: 100% |==========================================| Time: 00:00:01
Sending data
14 tables, 114 records
test_bonus_aw: 100% |==========================================| Time: 00:00:00
test_money_tr: 100% |==========================================| Time: 00:00:00
test_friendsh: 100% |==========================================| Time: 00:00:00
test_friendsh: 100% |==========================================| Time: 00:00:00
test_gift_rec: 100% |==========================================| Time: 00:00:00
test_group: 100% |==========================================| Time: 00:00:00
test_country: 100% |==========================================| Time: 00:00:00
test_user_is_: 100% |==========================================| Time: 00:00:00
test_gift: 100% |==========================================| Time: 00:00:00
test_user_has: 0% | | ETA: --:--:--
Saving session to push_201204301144.dat..
!!! Caught Server Exception
HTTP CODE: 500
Taps Server Error: undefined method `symbolize_keys' for nil:NilClass
["/app/lib/taps/data_stream.rb:183:in `parse_json'", "/app/lib/taps/server.rb:89:in `block in <class:Server>'","/app/.bundle/gems/ruby/1.9.1/gems/satra-1.0/lib/sinatra/base.rb:865:in `call'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra1.0/lib/sinatra/base.rb:865:in `block in route'", "/app/.bune/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:5:in `route_eval'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `block (2 levels) in route!'", "/app/.bundle/gems/ruby/9.1/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `block in route!, "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/baseb:476:in `route!'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'", "/app/.bundle/gems/ruby/1.9.1/gems/sinat-1.0/lib/sinatra/base.rb:411:in `block in call!'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'", "/appbundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `block in invoke'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/ba.rb:566:in `catch'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra.0/lib/sinatra/base.rb:411:in `call!'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'", "/app/.bundle/gems/ruby/1.1/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `block in call'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchroze'", "/app/.bundle/gems/ruby/1.9.1/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'", "/home/heroku_rack/lib/static_assets.rb:9:in `call'", "/homheroku_rack/lib/last_access.rb:15:in `call'", "/app/.bundle/gems/ruby/1.9.1/gems/rack-1.2.1/lib/rack/urlmap.rb:47:in `block in call'", "/app/.bundleems/ruby/1.9.1/gems/rack-1.2.1/lib/rack/urlmap.rb:41:in `each'", "/app/.bundle/gems/ruby/1.9.1/gems/rack-1.2.1/lib/rack/urlmap.rb:41:in `call'", "/he/heroku_rack/lib/date_header.rb:14:in `call'", "/app/.bundle/gems/ruby/1.9.1/gems/rack-1.2.1/lib/rack/builder.rb:77:in `call'", "/app/.bundle/gems/by/1.9.1/gems/thin-1.2.7/lib/thin/connection.rb:76:in `block in pre_process'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/connection.rb::in `catch'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/connection.rb:74:in `pre_process'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2/lib/thin/connection.rb:57:in `process'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/connection.rb:42:in `receive_data'","/app/.bundle/ms/ruby/1.9.1/gems/eventmachine0.12.10/lib/eventmachine.rb:256:in`run_machine'","/app/.bundle/gems/ruby/1.9.1/gems/eventmachine0.12.10/lib/eventchine.rb:256:in `run'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/backends/base.rb:57:in `start'", "/app/.bundle/gems/ruby/1.9.1/gems/tn-1.2.7/lib/thin/server.rb:156:in `start'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/controllers/controller.rb:80:in `start'", "/app/.ndle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/runner.rb:177:in `run_command'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/lib/thin/runner.rb:143: `run!'", "/app/.bundle/gems/ruby/1.9.1/gems/thin-1.2.7/bin/thin:6:in `<top (required)>'", "/app/.bundle/gems/ruby/1.9.1/bin/thin:19:in `load'", "/a/.bundle/gems/ruby/1.9.1/bin/thin:19:in `<main>'"]
Running on Windows 7.
Please help!
You could try using the pgbackups addon if you're not changing the DB between local and production.
It is more reliable and usually faster than a db:push/db:pull.
https://devcenter.heroku.com/articles/pgbackups
deltrem#deltrem-desktop:~/ramaze web/app/blog$ ruby start.rb
I [2010-06-10 14:53:33 $1886] INFO | : activating sequel
I [2010-06-10 14:53:33 $1886] INFO | : Installing sequel
/usr/local/lib/site_ruby/1.8/rubygems/remote_fetcher.rb:124:in `initialize': Permission denied - /home/deltrem/.gem/ruby/1.8/cache/sequel-3.9.0.gem (Errno::EACCES)
from /usr/local/lib/site_ruby/1.8/rubygems/remote_fetcher.rb:124:in `open'
from /usr/local/lib/site_ruby/1.8/rubygems/remote_fetcher.rb:124:in `download'
from /usr/local/lib/site_ruby/1.8/rubygems/dependency_installer.rb:257:in `install'
from /usr/local/lib/site_ruby/1.8/rubygems/dependency_installer.rb:247:in `each'
from /usr/local/lib/site_ruby/1.8/rubygems/dependency_installer.rb:247:in `install'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:91:in `install_gem'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:102:in `temp_argv'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:89:in `install_gem'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:80:in `setup_gem'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:63:in `setup'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:62:in `each'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:62:in `setup'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:44:in `run'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:38:in `initialize'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:29:in `new'
from /usr/lib/ruby/gems/1.8/gems/ramaze-2010.04.04/lib/ramaze/setup.rb:29:in `setup'
from ./app.rb:4
from start.rb:3:in `require'
from start.rb:3
I have no idea what this is for, but the error message seems pretty clear:
Permission denied - /home/deltrem/.gem/ruby/1.8/cache/sequel-3.9.0.gem (Errno::EACCES)
You don't have permission to read or write to that path. Are you supposed to sudo this?