I have set up elasticsearch with password protected, and i am successfully able to work with elastic search by entering username=elastic and password=mypassword
but now I am trying to import mysql data into elasticsearch using logstash, when i run logstash using below command it gives error.
am i missing something?
logstash -f mysql.conf
logstash-plain.log
[2019-06-14T18:12:34,410][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-14T18:12:34,424][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-06-14T18:12:35,400][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 16, column 23 (byte 507) after output {\r\n elasticsearch {\r\n\thosts => \"http://10.42.35.14:9200/\"\r\n user => elastic\r\n password => pharma", :backtrace=>["D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-06-14T18:12:35,758][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-14T18:12:40,664][INFO ][logstash.runner ] Logstash shut down.
mysql.conf
# file: contacts-index-logstash.conf
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://52.213.22.96:3306/prbi"
jdbc_user => "myuser"
jdbc_password => "mypassword"
jdbc_driver_library => "mysql-connector-java-6.0.5.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * from tmp_j_summaryreport"
}
}
output {
elasticsearch {
hosts => "http://10.42.35.14:9200/"
user => elastic
password => myelasticpassword
index => "testing123"
}
stdout { codec => json_lines }
}
logstash.yml
# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
# pipeline:
# batch:
# size: 125
# delay: 5
#
# Or as flat keys:
#
# pipeline.batch.size: 125
# pipeline.batch.delay: 5
#
# ------------ Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
# path.data:
#
# ------------ Pipeline Settings --------------
#
# The ID of the pipeline.
#
# pipeline.id: main
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
# pipeline.batch.delay: 50
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
# path.config:
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# Periodically check if the configuration has changed and reload the pipeline
# This can also be triggered manually through the SIGHUP signal
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
#
# config.reload.interval: 3s
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# When enabled, process escaped characters such as \n and \" in strings in the
# pipeline configuration files.
#
# config.support_escapes: false
#
# ------------ Module Settings ---------------
# Define modules here. Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
# - name: MODULE_NAME
# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
# ------------ Cloud Settings ---------------
# Define Elastic Cloud settings here.
# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
# and it may have an label prefix e.g. staging:dXMtZ...
# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'
# cloud.id: <identifier>
#
# Format of cloud.auth is: <user>:<pass>
# This is optional
# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'
# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'
# cloud.auth: elastic:<password>
#
# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
# queue.type: memory
#
# If using queue.type: persisted, the directory path where the data files will be stored.
# Default is path.data/queue
#
# path.queue:
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 64mb
#
# queue.page_capacity: 64mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
# queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
# dead_letter_queue.enable: false
# If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries
# will be dropped if they would increase the size of the dead letter queue beyond this setting.
# Default is 1024mb
# dead_letter_queue.max_bytes: 1024mb
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
# path.dead_letter_queue:
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
# path.logs:
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []
#
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
#xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
#xpack.management.elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
#xpack.management.enabled: true
xpack.management.elasticsearch.hosts: "http://10.42.35.14:9200/"
#xpack.management.elasticsearch.username: logstash_system
xpack.management.elasticsearch.password: myelasticpassword
This message on the logstash log indicates that there is something wrong with your config file:
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError"
The rest of message says that the problem is in your output block
:message=>"Expected one of #, {, } at line 16, column 23 (byte 507) after output {
Double check your output configuration, it needs to be something like this:
output {
elasticsearch {
hosts => ["10.42.35.14:9200"]
user => "elastic"
password => "myelasticpassword"
index => "testing123"
}
stdout { codec => "json_lines" }
}
I'm trying to write a piece of code that tells the user if a string is formatted in a specific way. The format looks like this (it's a circle):
######\n
### ###\n
# #\n
# #\n
# #\n
# #\n
# #\n
### ###\n
######\n
\n
The newlines are included for clarity; the # character may be any character from an existing character class.
What I want to do is check if a string (from a file) contains one or more of these "circles". Multiple circles should be seperated like this:
# #\n
### ###\n
######\n
\n
######\n
### ###\n
# #\n
So this is what I've tried:
isCircle = "( #{get_i}{6}\n"
isCircle += " #{get_i}{3} #{get_i}{3}\n"
isCircle += " #{get_i} #{get_i}\n"
isCircle +=("#{get_i} #{get_i}\n"*3)
isCircle += " #{get_i} #{get_i}\n"
isCircle += " #{get_i}{3} #{get_i}{3}\n"
isCircle = " #{get_i}{6}\n\n?)*"
isCircle = Regexp.new(isCircle)
(get_i is a method that returns the aforementioned character class, correctly escaped and everything)
However, when testing this against an incorrect input string, it still tells me there's a match.
What am I doing wrong and how can I correctly perform the match?
Maybe this:
[ ]+#{6}[\n ]+#{3}[ ]+#{3}[\n ]+(?>#[ ]+#[\n\s]+)+#{3}[ ]+#{3}[\n\s]+#{6}\s*
Demo
Is this what you want to do?
Suppose you wish to search for this pattern:
pattern =
%{ ######
### ###
# #
# #
# #
# #
# #
### ###
######
}
in a text file whose contents have been read into this string:
str =
%{###
##
######
### ###
# #
# #
# #
# #
# #
### ###
######
#######
######
######
### ###
# #
# #
# #
# #
# #
### ###
######
###
}
Just use String#scan:
puts str.scan(pattern).join("\n")
######
### ###
# #
# #
# #
# #
# #
### ###
######
######
### ###
# #
# #
# #
# #
# #
### ###
######
If you instead want the line offsets of str where the pattern begins:
sarr = str.lines
parr = pattern.lines
prows = parr.size
(sarr.size-prows+1).times.select { |i| sarr[i,prows] == parr }
#==> [2, 13]
I'm running Jasmine js tests on a project. I'm using the jasmine ruby gem to run my tests. I had everything working with some sample tests earlier but when I updated the jasmine.yml file with some new tests my tests no longer run.
When I try to run from the command line with rake jasmine:ci I get a segmentation fault in PhantomJs.
My jasmine.yml file looks like this:
# src_files
#
# Return an array of filepaths relative to src_dir to include before jasmine specs.
# Default: []
#
# EXAMPLE:
#
# src_files:
# - lib/source1.js
# - lib/source2.js
# - dist/**/*.js
#
src_files:
- app/javascripts/beatView.js
# stylesheets
#
# Return an array of stylesheet filepaths relative to src_dir to include before jasmine specs.
# Default: []
#
# EXAMPLE:
#
# stylesheets:
# - css/style.css
# - stylesheets/*.css
#
stylesheets:
- assets/application.css
# helpers
#
# Return an array of filepaths relative to spec_dir to include before jasmine specs.
# Default: ["helpers/**/*.js"]
#
# EXAMPLE:
#
# helpers:
# - helpers/**/*.js
#
helpers:
- 'helpers/**/*.js'
# spec_files
#
# Return an array of filepaths relative to spec_dir to include.
# Default: ["**/*[sS]pec.js"]
#
# EXAMPLE:
#
# spec_files:
# - **/*[sS]pec.js
#
spec_files:
- 'spec/javascripts//backbone/views/beat_spec.js'
# src_dir
#
# Source directory path. Your src_files must be returned relative to this path. Will use root if left blank.
# Default: project root
#
# EXAMPLE:
#
# src_dir: public
#
src_dir:
# spec_dir
#
# Spec directory path. Your spec_files must be returned relative to this path.
# Default: spec/javascripts
#
# EXAMPLE:
#
# spec_dir: spec/javascripts
#
spec_dir: spec/javascripts
# spec_helper
#
# Ruby file that Jasmine server will require before starting.
# Returned relative to your root path
# Default spec/javascripts/support/jasmine_helper.rb
#
# EXAMPLE:
#
# spec_helper: spec/javascripts/support/jasmine_helper.rb
#
spec_helper: spec/javascripts/support/jasmine_helper.rb
# boot_dir
#
# Boot directory path. Your boot_files must be returned relative to this path.
# Default: Built in boot file
#
# EXAMPLE:
#
# boot_dir: spec/javascripts/support/boot
#
boot_dir:
# boot_files
#
# Return an array of filepaths relative to boot_dir to include in order to boot Jasmine
# Default: Built in boot file
#
# EXAMPLE
#
# boot_files:
# - '**/*.js'
#
boot_files:
# rack_options
#
# Extra options to be passed to the rack server
# by default, Port and AccessLog are passed.
#
# This is an advanced options, and left empty by default
#
# EXAMPLE
#
# rack_options:
# server: 'thin'
I don't know if the issue is in the yml file or if there is something else that I may have messed up.
I figured this one out. The issue was in my jasmine.yml file. The path for the specs was spec/javascripts/ as you can see where it specified spec_dir and then I was giving the spec file as - 'spec/javascripts//backbone/views/beat_spec.js' which caused phantom to crash because of the repeated directory names.
This code (by someone else) might have been written using an older version of Ruby because now I'm getting an error calling 'each' on a string object. The maze string below gets passed to the maze_string_to_array method. When it's run, it yields this error in `maze_string_to_array'
NoMethodError: undefined method `each' for #<String:0x00000100854ac0>
Can you explain what the problem is, and how to fix it?
def maze_string_to_array(mazestring)
#maze = []
mazestring.each do |line|
#maze.push line.chomp
end
end
Maze string
MAZE1 = %{#####################################
# # # #A # # #
# # # # # # ####### # ### # ####### #
# # # # # # # # #
# ##### # ################# # #######
# # # # # # # # #
##### ##### ### ### # ### # # # # # #
# # # # # # B# # # # # #
# # ##### ##### # # ### # # ####### #
# # # # # # # # # # # #
# ### ### # # # # ##### # # # ##### #
# # # # # # #
#####################################}
This code is unnecessarily verbose. The whole method can be written with a map and using 1.9 String#lines instead of the old 1.8.x String#each to split lines:
#maze = mazestring.lines.map(&:chomp)
Ruby 1.8 String#each used to iterate through lines. In 1.9, String#each_line does the same thing.
In Ruby 1.9 use each_line instead of each.
But it looks as if you could replace the whole method with mazestring.split(/\n/) anyway.
I am going through my home brew doctor messages and no matter what I do I can not seem to remove this message:
Warning: You have uncommitted modifications to Homebrew's core.
Unless you know what you are doing, you should run:
cd /usr/local && git reset --hard
Does anyone have any idea on how to get rid of this?
From git status /usr/local
# Library/Aliases/dwarffortress
# Library/Aliases/ultima4
# Library/Aliases/ultima7
# Library/Contributions/examples/
# Library/Formula/abfind.rb
# Library/Formula/abuse.rb
# Library/Formula/akonadi.rb
# Library/Formula/angband.rb
# Library/Formula/apc.rb
# Library/Formula/aqua-less.rb
# Library/Formula/argp-standalone.rb
# Library/Formula/attica.rb
# Library/Formula/bashreduce.rb
# Library/Formula/c10t.rb
# Library/Formula/chocolate-doom.rb
# Library/Formula/cmigemo.rb
# Library/Formula/coffee-script.rb
# Library/Formula/csstidy.rb
# Library/Formula/dosbox.rb
# Library/Formula/dotless.rb
# Library/Formula/dwarf-fortress.rb
# Library/Formula/exult.rb
# Library/Formula/flip.rb
# Library/Formula/frobtads.rb
# Library/Formula/frotz.rb
# Library/Formula/gearman-php.rb
# Library/Formula/gnu-chess.rb
# Library/Formula/gnu-go.rb
# Library/Formula/growlme.rb
# Library/Formula/imagick.rb
# Library/Formula/inform6.rb
# Library/Formula/jnethack.rb
# Library/Formula/jwhois.rb
# Library/Formula/kde-phonon.rb
# Library/Formula/kdebase-runtime.rb
# Library/Formula/kdelibs.rb
# Library/Formula/kdepimlibs.rb
# Library/Formula/libgdiplus.rb
# Library/Formula/libiconv.rb
# Library/Formula/libsgml.rb
# Library/Formula/maatkit.rb
# Library/Formula/mcrypt-php.rb
# Library/Formula/mednafen.rb
# Library/Formula/memcache-php.rb
# Library/Formula/memcached-php.rb
# Library/Formula/memcachedb.rb
# Library/Formula/midgard2-php.rb
# Library/Formula/mongo-php.rb
# Library/Formula/n2n.rb
# Library/Formula/nazghul.rb
# Library/Formula/nethack.rb
# Library/Formula/netris.rb
# Library/Formula/ninja.rb
# Library/Formula/open-tyrian.rb
# Library/Formula/orderly.rb
# Library/Formula/oxygen-icons.rb
# Library/Formula/parsley.rb
# Library/Formula/pbrt.rb
# Library/Formula/pcntl-php.rb
# Library/Formula/phpmyadmin.rb
# Library/Formula/pioneers.rb
# Library/Formula/platypus.rb
# Library/Formula/pspell-php.rb
# Library/Formula/qimageblitz.rb
# Library/Formula/robotfindskitten.rb
# Library/Formula/shared-desktop-ontologies.rb
# Library/Formula/shen.rb
# Library/Formula/slashem.rb
# Library/Formula/solr-php.rb
# Library/Formula/soprano.rb
# Library/Formula/sparse.rb
# Library/Formula/spim.rb
# Library/Formula/stone-soup.rb
# Library/Formula/strigi.rb
# Library/Formula/uggconv.rb
# Library/Formula/unnethack.rb
# Library/Formula/voldemort.rb
# Library/Formula/woof.rb
# Library/Formula/xboard.rb
# Library/Formula/xcache.rb
# Library/Formula/xdebug.rb
# Library/Formula/xmoto.rb
# Library/Formula/xu4.rb
# Library/Homebrew/test/tests
After posting on the homebrew issues log https://github.com/mxcl/homebrew/issues/11761 I was able to clear the error by running git clean -df and once that is finished brew doctor. Once complete you should see:
Your system is raring to brew.