Shipping Logs Securely to a Remote Process Group using MiNiFi - apache-nifi

I am having a little bit of challenge with NiFi…MiNiFi precisely. We use MiNiFi to ship logs from remote systems to a NiFi Instance, from there to Kafka and into Elasticsearch. We can successfully do this without https, However, recently I was tasked to do same securely using https.
Using certificates, I can connect to the NiFi UI, the challenge is that MiNiFi is unable to connect to the RPG on the remote NiFi with the error "Unable to communicate with Remote NiFi at URI https://xxxx.com:9443/nifi due to: Received fatal alert: handshake_failure" . I suspect this is because of the errors below
2018-07-23 16:27:23,083 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set
2018-07-23 16:27:23,083 INFO [main] o.apache.nifi.controller.FlowController Not enabling HTTP(S) Site-to-Site functionality because the 'nifi.remote.input.http.enabled' property is not true
I have tried to set these properties in the nifi.properties file of MiNiFi, but the file is always overwritten at each restart with default values loaded.
Please, do you have any ideas on how to resolve this?
How can I bootstrap these settings at startup in the config.yml file or any other place?

You'll need to set those in the original flow that you export from NiFi to MiNiFi. The nifi.properties of the MiNiFi instance is automatically generated from the provided config.yml file. That file is generated by using the MiNiFi Converter Toolkit to convert the exported template XML file.
For more, you can watch these videos or read the Getting Started Guide.
You'll want to look for lines like the following in the config.yml:
Security Properties:
keystore: /tmp/ssl/localhost-ks.jks
keystore type: JKS
keystore password: localtest
key password: localtest
truststore: /tmp/ssl/localhost-ts.jks
truststore type: JKS
truststore password: localtest
ssl protocol: TLS
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Remote Processing Groups:
- name: http://localhost:8080/nifi
url: http://localhost:8080/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
Input Ports:
- id: AUTOGENERATED_NIFI_PORT_ID_HERE
name: MiNiFi-input
comment: ''
max concurrent tasks: 1
use compression: false
Properties: # Deviates from spec and will later be removed when this is autonegotiated
Port: 1026
Host Name: localhost

Related

Spring cloud config server share binary file

I am using spring configuration server.
While setting up Kafka, I came across the fact that I need to somehow specify binary certificates
spring:
kafka:
ssl:
truststore:
location: /filepath/trust_cert.jks
password: 1234
keystore:
location: /filepath/keystore_cert.jks
password: 1234
Can I somehow put them on the configuration server, and in this case, what should I write to the config, where the path to the file is expected?
I really don’t want to manually upload them to each server, I would like the configuration server to give them
Of course, these urls must be protected, just like configuration server urls

Filebeat over HTTPS

I am totally newbie in elk but I'm currently deploying ELK stack via docker-compose (https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html TLS part).
Elasticsearch and Kibana work correctly in HTTPS.
However, I don't understand how to enable Filebeat over HTTPS. I would like to send my nginx logs which is located on another server (over internet, so I do not want to send logs in clear text). Everything works fine in HTTP but when I switch to HTTPS and reload Filebeat I get the following message:
Error: ... Get https://10.15.0.12:9200: x509: certificate is valid for 127.0.0.0.1, not 10.15.0.12
I know I'm doing something wrong but I don't find the answer for Filebeat over HTTPS...
Here is my Filebeat configuration :
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.15.0.12:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Authentication credentials - either API key or username/password.
username: "elastic"
password: "myelasticpassword"
Thanks in advance.
I found the error :
My self signed certificate was for 127.0.0.1 host.
I've changed the IP in the instances.yml
Then I changed my filebeat config :
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["xx.xx.xx.xx:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "elastic"
password: "mypassword"
ssl.verification_mode: none

How to run Mongodb as a service with authentication on a windows machine

remark: I am using win10.
My goal is when windows boot mongodb as a service with authentication start( you can not enter the database without authenticate) but I can not manage to do it on a windows machine ( in linux it worked)
I write here the steps I tried:
dowlnload MongoDB
change conf from default to the following
# mongod.conf
http://docs.mongodb.org/manual/reference/configuration-options/
# Where and how to store data.
storage:
dbPath: C:\MongoDB\Server\4.0\data
journal:
enabled: true
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: C:\MongoDB\Server\4.0\log\mongod.log
# network interfaces
net:
port: 27017
bindIp: 127.0.0.1
security:
authorization: enabled
setParameter:
enableLocalhostAuthBypass: false
create a Admin user in the Admin collection.
db.createUser(
{
user: "....",
pwd: "...",
roles:
[
{ role: "root", db: "admin" }
]
}
)
Made it a service:
sc.exe create MongoDB
binPath=“\”C:\MongoDB\Server\4.0\bin\mongod.exe\”
–service
config=\”C:\MongoDB\Server\4.0\bin\mongod.cfg\”” DisplayName= “MongoDB” start= “auto”
getting feedback Successful.
but when i restart the computer, mongod is not starting and if i dont specify mongod --auth i can still enter without a authentication
How can I run Mongod as service with authentication? what am i doing wrong?
When i am trying to activate the service manually I get the following error
Error photo
The issue with the security tag. I have the same issue when I wanted to start the service in Windows 10. I copy the command from Windows service properties and then run on the command prompt.
The prompt shows me the error:
Unrecognized category : security
I found the solution and it is to write the security tag with options properly.
YAML need some specific input I guess. Here it is the solution.
security:
authorization: enabled
I had the same issue.
In your mongodb.cfg, use 2 spaces (instead of TAB) to indent authorization: enabled

Setting up ELK stack

I'm completely new to ELK and trying to install the stack with some beats for our servers.
Elasticsearch, Kibana and Logstash are all installed (on server A). I followed this guide here https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html.
Filebeat template was installed as well.
I also installed filebeat on another server (server B), and was trying to test the connection
$ /usr/share/filebeat/bin/filebeat test output -c
/etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -
path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs
/var/log/filebeat
logstash: my-own-domain:5044...
connection...
parse host... OK
dns lookup... OK
addresses: 163.172.167.147
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.2
dial up... OK
talk to server... OK
Things seems to be ok, yet data from filebeat on server B doesn't seem to be sending data to logstash.
Accessing Kibana keeps redirecting me back to Create Index pattern, with the message
Couldn't find any Elasticsearch data
Any direction pointing would be really appreciated.
Can you check your filebeat.yml file and see if configuration for logs are activated :
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log

Spring Dataflow and Yarn: How to set properties properly?

How can one change the default appdeployappmaster properties ?
When I'm trying to deploy an application through Spring DataFlow YARN. I registered my app, created a stream, and click the "deploy" button. When doing so, I get the following error :
[XNIO-2 task-2] WARN o.s.c.d.s.c.StreamDeploymentController - Exception when deploying the app StreamAppDefinition [streamName=histo, name=my-app, registeredAppName=my-app, properties={spring.cloud.stream.bindings.input.destination=log, spring.cloud.stream.bindings.input.group=histo}]: java.util.concurrent.ExecutionException: org.springframework.yarn.YarnSystemException: Invalid host name: local host is: (unknown); destination host is: "null":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost; nested exception is java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "null":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost
As you can see, the deployer is unable to find the "Resource Manager" URI, Although it is well found when the Spring DataFlow Server starts.
So I only get the problem at the deployment time.
Which property should I set to fix this issue, and where would I do that ?
EDIT 1:
Following Janne Valkealahti's answer, I added the following properties in /dataflow/apps/stream/app/servers.yml, relaunched the server, and tried to re-deploy my stream.
spring:
cloud:
dataflow:
yarn:
version: 0.0.1-SNAPSHOT
deployer:
yarn:
version: 1.0.2.RELEASE
stream:
kafka:
binder:
brokers: kafka.my-domain.com:9092
zkNodes: zookeeper.my-domain.com:2181/node
# Configured for Hadoop single-node running on localhost. Replace with property values reflecting your
# actual Hadoop cluster when running in a distributed environment.
hadoop:
fsUri: hdfs://mapr.my-domain.com/referentiel/ca_category_2014/
resourceManagerHost: mapr.my-domain.com
resourceManagerPort: 8032
resourceManagerSchedulerAddress: mapr.my-domain.com:8030
session:
store-type: none
I still get the exact same message.
PS: I'm not using Ambari, I'd like to understand how it works manually first.
EDIT 2:
I solved the problem adding the -Dspring.config.location VM arg on the DataFlow Server. The given configuration is passed to the deployer, and the application is effectively deployed.
I'll write an answer for it.
You didn't tell if your installation was based on ambari or normal manual YARN install so I assume it was a latter(manual).
I think a problem is that in distribution you use the config/servers.yml has a wrong setting for resourceManagerHost as it defaults to localhost. This file is distribute only once into hdfs when streams are launched. If you have changed it after you redeploy/create stream, app in hdfs directory will not get updated. On default this file in hdfs is /dataflow/apps/stream/app/servers.yml.
This error makes sense as also dataflow yarn server controlling whole stuff also needs access to yarn resource manager to submit apps. Settings for server also comes from a same servers.yml file.
It turns out I needed to add the -Dspring.config.location JVM arg to make it work. -Dspring.config.location should point to the file containing the YARN configuration, i.e.:
spring:
cloud:
dataflow:
yarn:
version: 0.0.1-SNAPSHOT
deployer:
yarn:
version: 1.0.2.RELEASE
stream:
kafka:
binder:
brokers: kafka.my-domain.com:9092
zkNodes: zookeeper.my-domain.com:2181/node
# Configured for Hadoop single-node running on localhost. Replace with property values reflecting your
# actual Hadoop cluster when running in a distributed environment.
hadoop:
fsUri: hdfs://mapr.my-domain.com/referentiel/ca_category_2014/
resourceManagerHost: mapr.my-domain.com
resourceManagerPort: 8032
resourceManagerSchedulerAddress: mapr.my-domain.com:8030
session:
store-type: none
This configuration is then passed to the deployer app (appdeployerappmaster if I get it right).

Resources