Logstash-forwarder as Windows Service - elasticsearch

Question

I'm struggling to create a Windows Service for a logstash forwarder on Windows 2008 R2 Server.
My setup looks as follows:
Ubuntu Server 14.04 LTS
Elasticsearch
Logstash
Kibana
Windows Server 2008 R2:
Application logging to a certain path.
Ship logs to the ELK Stack via Logstash-forwarder
I'm currently shipping logs successfully to the ELK-Stack via Logstash forwarder compiled for Windows using the instructions here... https://github.com/elastic/logstash-forwarder. The only problem is, that I have to run the logstash forwarder in a CLI window, and I'm not able to set it up as a Windows Service.
I've tryed the following SC command, the service is created but the service will not start at all. Just returning the following error: The service did not respond to the start or control request in a timely fashion.
sc create LogstashForwarder binpath= "\"C:\_Logstash\logstash-forwarder.exe\" -config=\"C:\_Logstash\logstash-forwarder.conf\"" start= Auto displayname= "Logstash forwarder"
Unfortunately Google does not know any answer either.
Does anyone have been able to start the logstash forwarder on Windows as a Windows Service with the SC command? Some good advice will be very appreciated.

Answer 1

If your logstash configuration is correct try these steps.
Get nssm soft
Decompress the nssm zip in the bin folder of logstash
Excecute from command line nssm install logstash
Add the path to your bat on the launched config screen
Add your startup directory too.
Here you can get some more help
https://blog.basefarm.com/blog/how-to-install-logstash-on-windows-server-2012-with-kibana-in-iis/
https://github.com/verbosemode/public-notes/blob/master/logstash-windows.md
Hope this help

Answer 2

To Add to Rys' answer, Logstash-Forwarder doesn't natively read the Windows Event log. Whilst looking into how to get around this I came across this gist by Sean-M.
I modified his original script so that the Powershell script starts LSF and then pipes the event log into the stdin. I then point NSSM at the script and run that as a service. If you have your configuration file setup like this:
{
"network": {
"servers": [ "<logstash IP>:5000" ],
"timeout": 15,
"ssl ca": "C:/path/to/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"C:/inetpub/logs/LogFiles/W3SVC*/*.log"
],
"fields": { "type": "iis-w3svc" }
},
{
"paths": [
"-"
],
"fields": { "type": "windows-event" }
}
]
}
LSF will capture the JSON input and send it to Logstash. Powershell code below**:
#Requires -Version 3
param (
[string]$lognames
)
#reading security log requires elevated privileges so only read Application and System for now
[string[]]$logname = $("Application", "System" )
if ($lognames)
{
[string[]]$logname = $lognames -split ", "
}
##################################
# Functions #
##################################
function EvenSpace{
param ($word)
$tabWidth = 48
$wordTabs = $tabWidth - $word.Length
$tabNum = [Math]::Floor($($wordTabs/4)) / 2
("`t" * $tabNum)
}
## Read events, write to file
function ReadEvents {
param ([hashtable]$filter, [string]$OutFile=[String]::Empty)
## Make it look pretty if writting to stdout
try {
[object[]]$data = Get-WinEvent -FilterHashtable $filter -ErrorAction SilentlyContinue | sort RecordId
[int]$count = 0
if ((-not $data -eq $null) -or ($data.Count -gt 0)) {
$count = $data.Count
}
Write-Verbose ("Log: $($filter["LogName"])" + (EvenSpace -word $filter["LogName"]) + "Count: $count")
}
catch {
$Error[0]
Write-Verbose ""
Write-Verbose "Filter:"
$filter
return
}
if ($data.Count -gt 0) {
foreach ($event in $data) {
$json = $event | ConvertTo-Json -Compress
#$jsonbytes = #($json)
#$process.StandardInput.BaseStream.Write($jsonbytes,0,$jsonbytes.Count)
Write-Verbose $json
$process.StandardInput.WriteLine($json)
}
}
}
## Use a try/catch/finally to allow for the inputs to be closed and the process stopped
[System.Diagnostics.Process]$process = $null
$endTime = Get-Date
try
{
## Prepare to invoke the process
$processStartInfo = New-Object System.Diagnostics.ProcessStartInfo
$processStartInfo.FileName = (Get-Command .\logstash-forwarder.exe).Definition
$processStartInfo.WorkingDirectory = (Get-Location).Path
$processStartInfo.Arguments = "-config logstash-forwarder.conf"
$processStartInfo.UseShellExecute = $false
## Always redirect the input and output of the process.
## Sometimes we will capture it as binary, other times we will
## just treat it as strings.
$processStartInfo.RedirectStandardOutput = $true
$processStartInfo.RedirectStandardInput = $true
$process = [System.Diagnostics.Process]::Start($processStartInfo)
##################################
# Main Logic #
##################################
## Loop to capture events
while ($true) {
[String]::Empty | Write-Verbose
Start-Sleep -Seconds 5
$startTime = $endTime
[TimeSpan]$diff = (Get-Date) - $startTime
if ($diff.TotalHours -gt 1) {
$endTime = $startTime + (New-TimeSpan -Minutes 30)
}
else {
$endTime = Get-Date
}
Write-Verbose "Starting timespan $($startTime) -> $($endTime)"
## Supports reading multiple logs
if ($logname.Count -gt 1) {
foreach ($log in $logname) {
ReadEvents -filter #{LogName=$log; StartTime=$startTime; EndTime=$endTime} -OutFile $output
}
}
else {
ReadEvents -filter #{LogName=$logname; StartTime=$startTime; EndTime=$endTime} -OutFile $output
}
}
}
catch
{
Write-Error $error[0]|format-list -force
throw $_.Exception
}
finally
{
if($process)
{
$process.StandardInput.Close()
$process.Close()
}
}
** The script doesn't really handle LSF failing, but it serves my purposes for now.

Answer 3

Tried to add this under the Answer using nssm. You can also use the following to create the service from commandline without the UI.
Just ensure that nssm.exe is in the same directory and you run the script from there (or just edit the script).
#echo off
set BASE_DIR=C:\temp\logstash-forwarder
nssm install Logstash-Forwarder "%BASE_DIR%\logstash-forwarder.exe"
nssm set Logstash-Forwarder AppDirectory "%BASE_DIR%"
nssm set Logstash-Forwarder AppStopMethodSkip "6"
nssm set Logstash-Forwarder AppParameters "-config %BASE_DIR%\logstash-forwarder.conf"