Index not found exception while sending data to Opensearch AWS using td agent - elasticsearch

Question

I have set up Opensearch in AWS. I have installed td-agent in Ubuntu 18.04. Below is my td-agent.conf file:
<source>
#type tail
path /home/rocket/PycharmProjects/EFK/log.json
pos_file /home/rocket/PycharmProjects/EFK/log.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag log
</source>
<match *log*>
#type opensearch
host search-tanz-domain-2vbjmk2d4.us-west-2.es.amazonaws.com/
port 9200
scheme https
ssl_verify false
user admin
password lah_001
index_name test
</match>
When running the td-agent I am getting below error:
2023-01-26 15:41:44 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. [404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [:9200]","index":":9200","resource.id":":9200","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [:9200]","index":":9200","resource.id":":9200","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}
So it's saying index not found which is a bit strange because as per my understanding when you send data to Opensearch or Elasticsearch then you need to create index pattern manually by using Kibana. I have never faced this error in Elasticsearch and I am only facing this issue in Opensearch while both of them looks to be same.
Edit
I have created the index using API:
I listed all the index and I can see test:
Now I again started uploading the data using td-agent but still getting the same error as above.

Answer 1

I haven't used td-agent before but based on the configuration file you provided it seems like it is trying to reach index test.
In opensearch, when you create a domain, it doesn't contain any indexes.
when you send data to opensearch or elasticsearch then you need to create index pattern manually by using kibana
I don't think this is necessarily true. You can create an index without using kibana and you can also create an index without sending any data. In fact, I think it is a better practice to create the index first and send the data later.
I think if you create the index test first it should work for you.
In Java:
OpenSearchClient client = new OpenSearchClient(
new AwsSdk2Transport(
httpClient,
host,
region,
AwsSdk2TransportOptions.builder().build()));
// create the index
String index = "test";
CreateIndexRequest createIndexRequest = new CreateIndexRequest.Builder().index(index).build();
client.indices().create(createIndexRequest);
See: Sample code for Amazon OpenSearch Service