How to resolve "OpenSearch Unreachable: [https://127.0.0.1:9200/][ unable to find valid certification path to requested target? - elasticsearch

Question

Hi I am trying to ingest data from logstash(oss) to Opensearch but it seems I can't connect to Opensearch from logstash.
The error log:
[avs#localhost pipeline]$ ./bin/logstash -f config/pipeline/ipv4.conf
-bash: ./bin/logstash: No such file or directory
[avs#localhost pipeline]$ cd ..
[avs#localhost config]$ cd ..
[avs#localhost logstash-7.16.2]$ ./bin/logstash -f config/pipeline/ipv4.conf
Using bundled JDK: /oss/bin/logstash-7.16.2/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /oss/data_files/logs/logstash which is now configured via log4j2.properties
[2022-01-27T11:36:13,302][INFO ][logstash.runner ] Log4j configuration path used is: /oss/bin/logstash-7.16.2/config/log4j2.properties
[2022-01-27T11:36:13,313][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-01-27T11:36:13,813][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-01-27T11:36:14,823][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-01-27T11:36:16,149][INFO ][org.reflections.Reflections] Reflections took 152 ms to scan 1 urls, producing 119 keys and 417 values
[2022-01-27T11:36:17,683][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["https://127.0.0.1:9200"]}
[2022-01-27T11:36:18,093][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://127.0.0.1:9200/]}}
[2022-01-27T11:36:18,498][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://127.0.0.1:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [https://127.0.0.1:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
The logstash pipeline file:
input {
file {
path => "/home/avs/avs_dump.csv"
start_position => "beginning"
}
}
output {
opensearch {
hosts => ["https://127.0.0.1:9200"]
auth_type => {
type => 'basic'
user => 'admin'
password => 'admin'
}
index => "cassandra"
}
file {
path => "/oss/data_files/data/logstash/zonos_ipv4.out"
}
}
and here is the opensearch.yml file:
# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: avs-subhsaree
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /oss/data_files/data/logstash
#
# Path to log files:
#
path.logs: /oss/data_files/logs/logstash
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["127.0.0.1"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
here is the files in opensearch directory that has some pem files:
.
├── esnode-key.pem
├── esnode.pem
├── jvm.options
├── jvm.options.d
├── kirk-key.pem
├── kirk.pem
├── log4j2.properties
├── opensearch.keystore
├── opensearch-observability
│   └── observability.yml
├── opensearch-reports-scheduler
│   └── reports-scheduler.yml
├── opensearch.yml
└── root-ca.pem
3 directories, 11 files
Seems logstash can't connect to opensearch because of auth type which should be SSL instead of basic but the problem is I do not know how do I obtain the required file and from where.
Anyone can point me in the right direction or a document for this would be really helpful.
Thanks