Pass parameters from Airflow to Logstash - elasticsearch

Question

I have configured logstash to listen the logs at default airflow logs path. I want to create the index in elasticsearch as {dag_id}-{task_id}-{execution_date}-{try_number}. All these are parameters from Airflow. These are the modified values in airflow.cfg.
[core]
remote_logging = True
[elasticsearch]
host = 127.0.0.1:9200
log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
end_of_log_mark = end_of_log
write_stdout = True
json_format = True
json_fields = asctime, filename, lineno, levelname, message
These task instance details need to passed from Airflow to logstash.
dag_id,
task_id,
execution_date,
try_number
This is my logstash config file.
input {
file{
path => "/home/kmeenaravich/airflow/logs/Helloworld/*/*/*.log"
start_position => beginning
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logginapp-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
I have 2 questions. How to pass the parameters from Airflow to Logstash?
I have configured logstash to listen to the logs path. Since remote_logging is True in airfow.cfg, logs are not written to base log folder. If that is false or if I connect to Amazon S3, logs are written to base_log_folder path too. But, for me to configure logstash, logs need to be written in local folder. I use airflow version 1.10.9 . What can I do to stream my logs to Elasticsearch index.

Answer 1

To answer your first question (I assume you mean passing the logs directly to Elasticsearch), you cannot. The Airflow "Elasticsearch Logging" is not really a logging to Elasticsearch but more a configuration to enable the logging to get shipped to Elasticsearch. The naming of the attributes is (in my opinion) a little bit confusing as it suggests that you can write directly to Elasticsearch.
You can configure Airflow to read logs from Elasticsearch. See Airflow Elasticsearch documentation for more information:
Airflow can be configured to read task logs from Elasticsearch and
optionally write logs to stdout in standard or json format. These logs
can later be collected and forwarded to the Elasticsearch cluster
using tools like fluentd, logstash or others.
As you have enabled write_stdout = True, output is written to stdout. If you want the output to be written in files you have to set write_stdout = False or leave it empty. Your logstash configuration should then find the files, which answers your second question.
Cheers
Michael