Elastic Cloud: API endpoint - 404 Not Found on search? - elasticsearch

Question

Description
I want to query the data in our Elastic Cloud instance using the REST API with authorization via API Key.
Steps taken
I tried using the SQL API as well as the Search API. NB that I would much preferably use the SQL API.
Based on the following curl commands provided in the documentation:
curl -X POST "localhost:9200/_sql?format=txt&pretty" -H 'Content-Type: application/json' -d'
{
"query": "SELECT * FROM library WHERE release_date < \u00272000-01-01\u0027"
}
'
[source]
curl -u elastic:password https://CLUSTER_ID.REGION.PLATFORM.found.io:9243/my_index/my_type -XPOST -d '{
"title": "One", "tags": ["ruby"]
}'
{"_index":"my_index","_type":"my_type","_id":"AV3ZeXsOMOVbmlCACuwj","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
[source]
and the documentation about the REST API, I attempted the following:
import base64
import json
import requests
if __name__ == '__main__':
response1 = requests.post(
'https://{redacted-id}.northamerica-northeast1.gcp.elastic-cloud.com:9243/_sql?format=txt',
headers={
"Authorization": base64.standard_b64encode(bytes('{API_KEY_ID}:{API_KEY_KEY}', 'utf-8')),
"Content-Type": 'application/json'
},
data={
"query": """
...
"""
}
)
print('response1')
print(response1)
response2 = requests.get(
'https://{redacted-id}.northamerica-northeast1.gcp.elastic-cloud.com:9243/logs-pubsub/_search',
headers={
"Authorization": base64.standard_b64encode(bytes('{API_KEY_ID}:{API_KEY_KEY}', 'utf-8')),
"Content-Type": 'application/json'
},
data={
"query": {
# ...
}
}
)
print('response2')
print(response2)
But both queries answer with 404 - Not Found.
What did I miss? Am I missing a part of the path like /api/..., /rest/...? Or is this a misdirection from a 403 to a 404 and the issue is the API Key?
Thanks!

Answer 1

The kind folks at ElasticStack provided me with the following answer. There were two issues:
The URL above was pointing to the Kibana instance, not to the ElasticCloud instance
In order to get the appropriate URL:
Navigate to https://cloud.elastic.co/deployments/
Click on your deployment:
Click on Copy Endpoint for the Elasticsearch endpoint:
The Authorization header was malformed
The Authorization header for an ApiKey powered request is the following:
ApiKey {B64_ENCODE('API_KEY_ID:API_KEY_KEY')}
Which can be written in python as:
"ApiKey " + str(base64.standard_b64encode(bytes('API_KEY_ID:API_KEY_KEY', 'utf-8')), 'utf-8')
On a final note, the team also strongly suggested I look at their Free on-demand Elasticsearch training.
Hopefully this will be helpful to whoever passes by here!