OpenSearch docker instance only allowing HTTPS connections - elasticsearch

I'm trying to get OpenSearch configured on my local machine, and am deploying it through docker-compose using the following configuration:
opensearch:
image: opensearchproject/opensearch:1.0.0
restart: unless-stopped
ports:
- "9200:9200"
- "9300:9300"
environment:
discovery.type: single-node
The instance starts successfully, however when trying to access it through the web interface, it only accepts HTTPS connections with the default basic auth credentials (admin:admin). i.e.
https://localhost:9200 asks me to enter administrator credentials, and upon doing so, returns an expected response:
{
"name" : "a39dcf825899",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "d2ZBZDQRTyG6SvYlCmX3Iw",
"version" : {
"distribution" : "opensearch",
"number" : "1.0.0",
"build_type" : "tar",
"build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
"build_date" : "2021-07-02T23:22:21.383695Z",
"build_snapshot" : false,
"lucene_version" : "8.8.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
However when attempting to connect to the instance over HTTP, I get an empty response:
On chrome:
Using the OpenSearch Python client on a Django instance running in a separate Docker container (part of the same docker-compose.yml):
opensearchpy.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
For reference, the code I am using to connect the OpenSearch Python client to the OpenSearch instance is:
cls._os_client = OpenSearch(
[{"host": 'opensearch', "port": '9200'}],
use_ssl=False,
verify_certs=False,
ssl_assert_hostname=False,
ssl_show_warn=False
)
How can I configure OpenSearch to allow insecure HTTP connections?

You can disable security, just add DISABLE_SECURITY_PLUGIN=true to your env.

Related

find elasticsearch service endpoint

I'm on my trial to test elasticcloud. But now I got problem to create pipeline from logstash to elasticcloud. Here is my logstash.conf output
output {
stdout{codec=>rubydebug}
elasticsearch
{
hosts=>["https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243"]
index=>"testindex"
user=>elasticdeploymentcredentials
password=>elasticdeploymentcredentials
}
}
But it always returning error as:
[WARN ] 2021-03-29 12:24:50.148 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:24:55.158 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:00.163 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:25:05.170 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:10.175 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
It is possible for me to curl it with my credential as :
[root#localhost testconfig]# curl https://elasticdeploymentcredentials:elasticdeploymentcredentials#<clusterid>.asia-southeast1.gcp.elastic-cloud.com:9243
it returning
"name" : "name",
"cluster_name" : "<clusterid>",
"cluster_uuid" : "<clusteruuid>",
"version" : {
"number" : "7.12.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "build_hash",
"build_date" : "2021-03-18T06:17:15.410153305Z",
"build_snapshot" : false,
"lucene_version" : "8.8.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
am I missing something?
Instead of trying to connect to Elastic Cloud via the username/password from the deployment, try to use the Cloud_ID/Cloud_Auth combination:
output {
elasticsearch {
hosts => ["https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
cloud_id => "your cloudid from the console"
cloud_auth => "elastic:password"
}
}
The cloud_auth parameter is where you are actually going to use the username/password from the deployment. More information here:
https://www.elastic.co/guide/en/logstash/7.12/connecting-to-cloud.html

Elasticsearch GET / is returning HTML error instead of JSON response

I have recently installed Elasticsearch on RHEL and set the node name in the configuration file. Later, I started the service using the sudo systemctl start elasticsearch.service The service seems to be running as per the status command
sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-11-25 14:59:13 CET; 2h 37min ago
Docs: https://www.elastic.co
Main PID: 6565 (java)
CGroup: /system.slice/elasticsearch.service
├─6565 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...
└─6754 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Nov 25 14:59:05 hdm18 systemd[1]: Starting Elasticsearch...
Nov 25 14:59:13 hdm18 systemd[1]: Started Elasticsearch.
But the output of GET is returning an HTML page instead of JSON message
curl -X GET "localhost:9200/?pretty"
<!-- IE friendly error message walkround.
if error message from server is less than
512 bytes IE v5+ will use its own error
message instead of the one returned by
server. -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
<head>
<meta
enter code here
Any idea what am I doing wrong?
I believe you copied your command from this official link, but if you copy the curl command it will be like curl -X GET "localhost:9200/?pretty".
And above will print below correct Output.
{
"name" : "Opster",
"cluster_name" : "es_710",
"cluster_uuid" : "SZ-nvW_KSOaudmfB6e0oFg",
"version" : {
"number" : "7.10.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
"build_date" : "2020-11-09T21:30:33.964949Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
This issue is solved by unsetting the http and https proxy by using
unset http_proxy
unset https_proxy
You're using Kibana dev console syntax when it looks like you're trying to use curl. check the curl syntax when looking at tutorials.
Probably the easiest for you is to use the dev console in kibana.

How to use High Level Rest Client in Spring Data ES 3.2.0.M1

Spring Data ES 3.2.0.M1 still uses old TransportClient instead of HighLevelRestClient
Spring Data ES 3.2.0.M1 supports High Level Rest Client, see Add support for Java High Level REST Client. I've added Spring Data ES 3.2.0.M1 to the SB2 app:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
<version>3.2.0.M1</version>
</dependency>
However, still TransportClient is used. There are two indication of that: exceptions on start-up:
o.e.transport.netty4.Netty4Transport : exception caught on transport layer [NettyTcpChannel{localAddress=/127.0.0.1:61171, remoteAddress=localhost/127.0.0.1:8085}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.33.Final.jar:4.1.33.Final]
and exception stacktrace when calling ElasticsearchTemplate:
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:349)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:382)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:395)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:384)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.getSearchResponse(ElasticsearchTemplate.java:947)
Are there any config param to tell Spring Data ES to switch to new High Level Rest Client? The docs say nothing about it.
P.S. Spring Data ES 3.2.0.M1 has 6.4.3 ES client version:
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1327) ~[elasticsearch-6.4.3.jar:6.4.3]
at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[transport-netty4-client-6.4.3.jar:6.4.3]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.33.Final.jar:4.1.33.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.33.Final.jar:4.1.33.Final]
... 20 common frames omitted
back-end runs 6.4.2 version:
bash-4.4$ curl http://127.0.0.1:8085
{
"name" : "NA17WWR",
"cluster_name" : "494164851665",
"cluster_uuid" : "7t3LoK7PRp-ur6FyxSmHwQ",
"version" : {
"number" : "6.4.2",
"build_flavor" : "oss",
"build_type" : "zip",
"build_hash" : "04711c2",
"build_date" : "2018-10-16T09:16:35.059415Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
As mentioned in the issue you're referring to, the high level REST client is available in ElasticsearchRestTemplate (see PR #216) not in ElasticsearchTemplate, which they'll keep until ES 7 for backward compatibility reasons.
You can create one with the configuration below:
<bean name="elasticsearchTemplate"
class="org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate">
<constructor-arg name="client" ref="restClient"/>
</bean>
<elasticsearch:rest-client id="restClient"/>

ssh tunnel for elasticsearch

I am on a vpn which does not allow access to elasticsearch directly, so I am trying to ssh tunnel to an external box that has access.
I am tunneling with the following:
ssh -L 12345:<elastic_ip>-east-1.aws.found.io:9200
but then if I curl:
curl http://user:pass#localhost:12345
I get:
{"ok":false,"message":"Unknown cluster."}
Yet, if I try this from the box directly:
curl http://user:pass#<elastic_ip>-east-1.aws.found.io:9200
I get:
{
"status" : 200,
"name" : "instance",
"cluster_name" : “<cluster>”,
"version" : {
"number" : "1.7.2",
"build_hash" : “<build>“,
"build_timestamp" : "2015-09-14T09:49:53Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
What am I doing wrong?
Here is how you can do it using #SSH tunneling with #Putty.
Below are the steps you need to take in order to configure SSH tunneling using Putty:
Download Putty from here and install it.
Configure Putty tunneling for Elasticsearch 9300 and 9200 ports as shown in the screenshot below:
After configuring you’ll need to open the SSH connection and make sure it is connected.
You may look at the SSH event log in order to validate your tunnel. Here is a link on how to do it.
Below is an #Elasticsearch code written in #Java that shows how to connect to the remote Elasticsearch cluster using local (9090 and 9093) ports forwarded over Putty SSH client.
public class App
{
public static void main( String[] args ) throws Exception
{
Settings settings = ImmutableSettings.settingsBuilder().
put("cluster.name", "my-cluster").build();
TransportClient client = new TransportClient(settings)
.addTransportAddress(
new netSocketTransportAddress(
"localhost", 9093));
CreateIndexResponse rs = client.admin().indices().create(
new CreateIndexRequest("tunnelingindex"))
.actionGet();
System.out.println(rs.isAcknowledged());
client.close();
}
}
The code creates an index named tunnelingindex on Elasticsearch.
Hope it helps.
This is a problem of HTTP protocol. It contains also hostnames and not only IP addresses and if you issue request on the localhost, this hostname is passed to the cluster.
There are basically two solutions, both quite hacky:
Set up your elasticsearch hostname to localhost so it will recognize your query.
Set up your /etc/hosts to direct <elastic_ip>-east-1.aws.found.io to your 127.0.0.1, connect to your ssh with direct IP and then curl to the real address.

Elasticsearch with Yii 2.0: Error: Elasticsearch request failed: 7 - Failed to connect to ##.##.##.### port 9200: Connection refused

I have Elasticsearch properly configured on my server. I can do everything from the command line using cURL. I can even connect to it using cURL from a PHP script outside Yii. However, I can't seem to get it to work from within Yii 2.0.
In my config, I have:
'elasticsearch' => [
'class' => 'yii\elasticsearch\Connection',
'nodes' => [
['http_address' => 'localhost:9200'],
// configure more hosts if you have a cluster
],
],
But when I try to do a simple query in Yii, I get this error. Note how it's using my server ip address rather than 'localhost' or '172.0.0.1'. Note: I've hashed out my ip address for sercurity.
Elasticsearch Database Exception – yii\elasticsearch\Exception
Elasticsearch request failed: 7 - Failed to connect to ##.##.##.### port 9200: Connection refused
Error Info: Array
(
[requestMethod] => GET
[requestUrl] => http://##.##.##.###:9200/profiles/profile/_search
[requestBody] => {"size":100,"query":{"match_all":{}}}
[responseHeaders] => Array
(
)
[responseBody] =>
)
I was able to fix this error by updating the version of Elasticsearch to something > 1.3.0 as this is the minimum requirement for YIISOFT/YII2-ELASTICSEARCH
run curl -X GET 'http://127.0.0.1:9200' to check what version you are running.
First follow this steps to download elastic search.
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.5.2.tar.gz
mkdir es
tar -xf elasticsearch-1.5.2.tar.gz -C es
cd es
./bin/elasticsearch
Then you must be able to access to localhost:9200 and get something like this below :
{
"name" : "Sigyn",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.4.0",
"build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
"build_timestamp" : "2016-08-29T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
Then secondly,follow instruction in https://github.com/yiisoft/yii2-elasticsearch. Then you are done

Resources