Logstash multiple scheduled piplines - elasticsearch

Question

So Im want to configure multiple logstash .conf files (pipelines) with different schedule configuration for example in have file1.conf with is configuration :
file1.conf
input {
jdbc { jdbc_connection_string=>'jdbc:mysql://x.x.x.x:3306/<databasename>'
# The user we wish to execute our statement as
jdbc_user => '****'
jdbc_password => '*****'
# The path to our downloaded jdbc driver
jdbc_driver_library => 'mysql-connector-java-5.1.49.jar'
jdbc_driver_class => 'com.mysql.jdbc.Driver'
schedule => "30 * * * *"
# our query
statement => "select * from elasticsync2 "
}
filter {
grok {
match => {"date" => ["%{DATE:date_format}"]}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ['https://x.x.x.x:9200/','https://x.x.x.x:9200/']
index => '******'
user => "*********"
password => "*************"
cacert => "./certs/ca.crt"
ssl_certificate_verification => true
document_type => "data"
}
}
the second pipeline
file2.conf
input {
jdbc { jdbc_connection_string=>'jdbc:mysql://x.x.x.x:3306/<databasename>'
# The user we wish to execute our statement as
jdbc_user => '****'
jdbc_password => '*****'
# The path to our downloaded jdbc driver
jdbc_driver_library => 'mysql-connector-java-5.1.49.jar'
jdbc_driver_class => 'com.mysql.jdbc.Driver'
schedule => "20 * * * *"
# our query
statement => "select * from elasticsync "
}
filter {
grok {
match => {"date" => ["%{DATE:date_format}"]}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ['https://x.x.x.x:9200/','https://x.x.x.x:9200/']
index => '******'
user => "*********"
password => "*************"
cacert => "./certs/ca.crt"
ssl_certificate_verification => true
document_type => "data"
}
}
piplines.yml
- pipeline.id: first-pip
path.config: "/etc/logstash/conf.d/file1.conf"
queue.type: persisted
- pipeline.id: second-pip
path.config: "/etc/logstash/conf.d/file2.conf"
logstash logs
[2022-01-13T20:44:21,652][INFO ][logstash.outputs.elasticsearch][second-pip] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://x.x.x.x:9200/", "https://x.x.x.x:9200/"]}
[2022-01-13T20:44:21,656][INFO ][logstash.outputs.elasticsearch][first-pip] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://x.x.x.x:9200/", "https://x.x.x.x:9200/"]}
[2022-01-13T20:44:22,647][INFO ][logstash.outputs.elasticsearch][first-pip] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx#x.x.x.x:9200/, https://logstash_writer:xxxxxx#x.x.x.x:9200/]}}
[2022-01-13T20:44:22,647][INFO ][logstash.outputs.elasticsearch][second-pip] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx#x.x.x.x:9200/, https://logstash_writer:xxxxxx#x.x.x.x:9200/]}}
[2022-01-13T20:44:23,811][WARN ][logstash.outputs.elasticsearch][second-pip] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx#x.x.x.x:9200/"}
[2022-01-13T20:44:23,812][WARN ][logstash.outputs.elasticsearch][first-pip] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx#x.x.x.x:9200/"}
[2022-01-13T20:44:23,938][INFO ][logstash.outputs.elasticsearch][second-pip] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2022-01-13T20:44:23,946][WARN ][logstash.outputs.elasticsearch][second-pip] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-01-13T20:44:23,938][INFO ][logstash.outputs.elasticsearch][first-pip] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2022-01-13T20:44:23,951][WARN ][logstash.outputs.elasticsearch][first-pip] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-01-13T20:44:24,245][WARN ][logstash.outputs.elasticsearch][first-pip] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx#x.x.x.x:9200/"}
[2022-01-13T20:44:24,247][WARN ][logstash.outputs.elasticsearch][second-pip] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx#x.x.x.x:9200/"}
[2022-01-13T20:44:24,602][INFO ][logstash.outputs.elasticsearch][first-pip] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-01-13T20:44:24,606][INFO ][logstash.outputs.elasticsearch][second-pip] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-01-13T20:44:25,004][INFO ][logstash.javapipeline ][first-pip] Starting pipeline {:pipeline_id=>"first-pip", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/file1.conf"], :thread=>"#<Thread:0x5fda836f#/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2022-01-13T20:44:25,004][INFO ][logstash.javapipeline ][second-pip] Starting pipeline {:pipeline_id=>"second-pip", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/file2.conf"], :thread=>"#<Thread:0x84fc1d4 run>"}
[2022-01-13T20:44:27,912][INFO ][logstash.javapipeline ][second-pip] Pipeline Java execution initialization time {"seconds"=>2.9}
[2022-01-13T20:44:27,914][INFO ][logstash.javapipeline ][first-pip] Pipeline Java execution initialization time {"seconds"=>2.9}
[2022-01-13T20:44:28,025][INFO ][logstash.javapipeline ][second-pip] Pipeline started {"pipeline.id"=>"second-pip"}
[2022-01-13T20:44:28,026][INFO ][logstash.javapipeline ][first-pip] Pipeline started {"pipeline.id"=>"first-pip"}
the main problem is that when I run systemctl start logstash
it only execute the first one which is first-pip
I want to know how to synchronize schedules and pipelines to run in parallel.