Here is what I have specified in my yml for the logstash. I've tried multiple variations of quotes, no quotes, etc:
volumes:
- "./logstash:/etc/logstash/conf:ro"
command:
- "logstash -f /etc/logstash/conf/simplels.conf"
And simplels.conf contains this:
input {
stdin{}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout{}
}
Overall file structure is this, I'm running docker-compose up from the docker folder and getting Exit 1 on the Logstash container due to my 'command' parameter:
/docker:
docker-compose.yml
/logstash
simplels.conf
Related
I am writing a script that creates a logstash conf file and adds the configuration, and removes the existing filebeat config file and creates a new one.
I am using cat, but when I run the script, I get:
./script.sh: /etc/logstash/conf.d/apache.conf : Permission denied
./script.sh: /etc/filebeat/filebeat.yml: Permission denied
This is the script. I have tried using sudo chown -R.
Am I missing something or is there a better way to configure my file?
#!/bin/bash
sudo rm /etc/filebeat/filebeat.yml
cat > "/etc/filebeat/filebeat.yml" <<EOF
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
paths:
- /home/ubuntu/logs/.*log
setup.kibana:
output.logstash:
hosts: ["169.254.169.254:5044"]
EOF
sudo touch /etc/logstash/conf.d/apache.conf
sudo cat > "/etc/logstash/conf.d/apache.conf " <<EOF
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["169.254.169.254"]
}
}
EOF
The main problem here is because of how redirections work.
According to this answer:
All redirections (including >) are applied before executing the actual command. In other words, your shell first tries to open /etc/php5/apache2/php.ini for writing using your account, then runs a completely useless sudo cat.
You can easily solve your problem by using tee (with sudo) instead of cat. Then, your script should be like this:
#!/bin/bash
sudo rm /etc/filebeat/filebeat.yml
sudo tee "/etc/filebeat/filebeat.yml" << EOF
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
paths:
- /home/ubuntu/logs/.*log
setup.kibana:
output.logstash:
hosts: ["169.254.169.254:5044"]
EOF
sudo touch /etc/logstash/conf.d/apache.conf
sudo tee "/etc/logstash/conf.d/apache.conf" << EOF
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["169.254.169.254"]
}
}
EOF
I'm using sebp/elk's dockerised ELK. I've managed to get it running on my local machine and I'm trying to input dummy log data by SSH'ing into the docker container and running:
/opt/logstash/bin/logstash --path.data /tmp/logstash/data \
-e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'
After typing in some random text, I cannot see it indexed by elasticsearch when I visit http://localhost:9200/_search?pretty&size=1000
i am running an ELK stack in 3 docker containers through host machine ubuntu 16.04
the problem is after configuring the logstash.conf file to execute a command like "ifconfig" or "netstat -ano"i get an error. my logstash.conf file is:
input {
exec {
command => "netsat -ano"
codec => "json"
interval => 5
}
}
output{
elasticsearch { hosts => ["elasticsearch:9200"]}
}
i get this error after entering this command ( docker run -h logstash --name logstash --link elasticsearch:elasticsearch -it --rm -v "$PWD":/config-dir logstash -f /config-dir/logstash1.conf)
14:29:30.703 [[main]<exec] ERROR logstash.inputs.exec - Error while running command {:command=>"netsat -ano", :e=>#<IOError: Cannot run program "netsat" (in directory "/"): error=2, No such file or directory>, :backtrace=>["org/jruby/RubyIO.java:4380:in `popen'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-exec-3.1.2/lib/logstash/inputs/exec.rb:76:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-exec-3.1.2/lib/logstash/inputs/exec.rb:75:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-exec-3.1.2/lib/logstash/inputs/exec.rb:40:in `inner_run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-exec-3.1.2/lib/logstash/inputs/exec.rb:34:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:443:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:436:in `start_input'"]}
can anyone help please thanks in advance !
You will need to provide a full path for those commands, as the one Logstash runs with doesn't contain those directories.
input {
exec {
command => "/bin/netsat -ano"
codec => "json"
interval => 5
}
}
I am trying to read the access log s from /var/log/wtmp in elasticsearch
I can read the file when logged into the box by using last -F /var/log/wtmp
I have logstash running and sending logs to elasticsearch, here is logstash conf file.
input {
file {
path => "/var/log/wtmp"
start_position => "beginning"
}
}
output {
elasticsearch {
host => localhost
protocol => "http"
port => "9200"
}
}
what is showing in elasticsearch is
G
Once i opened the file using less , i could only see binary data.
Now logstash cant understand this data.
A logstash file like the following should work fine -
input {
pipe {
command => "/usr/bin/last -f /var/log/wtmp"
}
}
output {
elasticsearch {
host => localhost
protocol => "http"
port => "9200"
}
}
Vineeth's answer is right but the following cleaner config works as well:
input { pipe { command => "last" } }
last /var/log/wtmp and last are exactly the same.
utmp, wtmp, btmp are Unix files that keep track of user logins and logouts. They cannot be read directly because they are not regular text files. However, there is the last command which displays the information of /var/log/wtmp in plain text.
$ last --help
Usage:
last [options] [<username>...] [<tty>...]
I can read the file when logged into the box by using last -F /var/log/wtmp
I doubt that. What the -F flag does:
-F, --fulltimes print full login and logout times and dates
So, last -F /var/log/wtmp will interpret /var/log/wtmp as a username and won't print any login information.
What the -f flag does:
-f, --file <file> use a specific file instead of /var/log/wtmp
I was learning logstash. Have a very simple config file..
input {
file {
path => "D:\b.log"
start_position => beginning
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
grok {
match => { "message" => "%{LOGLEVEL:loglevel}" }
}
}
output {
stdout { codec => rubydebug }
}
The input file is just this:
INFO
I am running logstash on windows and the command is
logstash -f logstash.conf
I expect the output to be shown on the console to ensure that its working. But logstash produces no output, just the logstash config messages..
D:\Installables\logstash-2.0.0\logstash-2.0.0\bin>logstash -f logstash.conf
io/console not supported; tty will not be manipulated
Default settings used: Filter workers: 2
Logstash startup completed
I have deleted the sincedb file and tried. Is there something that i am missing?
I think this answers your question:
How to force Logstash to reparse a file?
It looks like you are missing the quotes around "beginning" and the other post recommends redirecting sincedb to dev/null. I don't know if there is a windows equivalent for that. I did use that as well, and it worked fine.
As an alternative, what I do now is to configure stdin() as input so that I don't have to worry about anything else.