We have deployed Elasticsearch(8.3) using kubernetes, ingress is defined for Elasticseach as https://elasticsearch.url.com/es, but when I am using the same to connect to Elasticsearch using Python elasticsearch package, I am getting error below:
Note: I have tried giving port number(https://elasticsearch.url.com:9200/es/)but still did not worked.
ValueError: URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')
I am using below code to connect:
from elasticsearch import Elasticsearch
client = Elasticsearch(
["https://elasticsearch.url.com/es/"],
http_auth=('username', 'password')
)
Kindly help me here how to resolve this.
the clients expect something like https://elasticsearch.url.com:9200/, as anything after the last / is considered a path/action of some sort, eg _search or an index name, for Elasticsearch to then do something with based on that context
you will likely need to remove the trailing es part of the url, then you can use https://elasticsearch.url.com:80/` (assuming ingress port 80 redirects to port 9200 for Elasticsearch)
Related
I'm trying to configure ElasticSearch data source for Grafana. I have them both running in Docker locally, both have versions 7.2.0. For Grafana I provide ES URL as http://localhost:9200, index name, time field, and ES version. All other parameters stay with the default value.
By saving my config I can see in Grafana logs next:
t=2021-02-14T14:55:58+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/<index>/_mapping remote_addr=172.17.0.1 referer="http://localhost:3000/datasources/edit/1/?utm_source=grafana_gettingstarted" error="http: proxy error: dial tcp 127.0.0.1:9200: connect: connection refused"
t=2021-02-14T14:55:58+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/<index>/_mapping status=502 remote_addr=172.17.0.1 time_ms=1 size=0 referer="http://localhost:3000/datasources/edit/1/?utm_source=grafana_gettingstarted"
I can't get why Grafana tries to get the mapping from some unknown IP. And how to configure it.
By the way, request to http://localhost:9200/<index>/_mapping returns me the correct mapping.
According to Grafana Documentation about configuration:
"URL needs to be accessible from the Grafana backend/server", so, try replacing "http://localhost:9200" to "http://elasticsearch:9200" instead. I had the same issue before, and it worked to me replacing this way :)
Plus: "elasticsearch" is the default name of Elasticsearch container (in case you are running with Docker), so that is the reason of the name.
Does anyone know how to cluster NiFi 1.5.0? I want to use dataflow.mydomain.com but... I get this error when I try to hit the loadbalancer that reads:
"The request contained an invalid host header [dataflow.mydomain.com] in the request [/nifi/]. Check for request manipulation or third-party intercept."
According to one post that I read, the problem was that the value of nifi.web.http.host had to match the value of the url.
If that's true, I don't understand how a cluster would be possible.
Thanks!
(I'm using a 3 host setup in AWS, the hosts will individually respond if I set the nifi.web.http.host to their private IP and I access it at http://[ip]/nifi/
but not if I use a loadbalancer in front of the cluster).
It is not really an issue of clustering NiFi, it is an issue of accessing it through a load balancer. A cluster does not imply a load balancer.
In the next version of NiFi there will be a new property (nifi.web.proxy.host) where you could put dataflow.mydomain.com and it would let it through.
For now I think you'd have to strip off the host header of each request at your load balancer so that it doesn't get passed on to the NiFi nodes, that it was is triggering the rejection. NiFi is inspecting the headers of the incoming request and seeing that the host header has a value that is not the host of NiFi.
I'm having a little trouble while using Logstash and http_poller as an input plugin. I would like to send my http requests through a proxy, but the only documentation I could find is:
<li> Value type is <<string,string>>
* There is no default value for this setting.
How can I define the IP of my proxy and a specific port to use?
After a few weeks of research, I found the solution:
You just have to specify the host/port within brackets;
proxy => {
host => "IP"
port => "PORT"
}
Kibana is unable to load the data from elasticsearch. I could see the below log in the elasticsearch. I am using elasticsearch version 1.4.2. Is this something related to load? Could anyone please help me?
[2015-11-05 22:39:58,505][DEBUG][action.bulk ] [Oddball] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
elastic search by default runs at http://localhost:9200
make sure you have proper URL in kibana.ymal
<pre>
# Kibana is served by a back end server. This controls which port to use.
port: 5601
# The host to bind the server to.
#host: example.com
# The Elastic search instance to use for all your queries.
elasticsearch_url: "http://localhost:9200"
</pre>
Aslo in elastic search config elasticsearch.yaml provide cluster name and http.cors.allow-origin.
<pre>
# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: elasticsearch
http.cors.allow-origin: "/.*/"
</pre>
I could solve this by setting up a new node for Elasticsearch and clearing the unassigned shards by setting the replica to 0.
According to the documentation, Elasticsearch reserves port 9300-9400 for cluster communication and port 9200-9300 for accessing the elasticsearch APIs. You get the impression that these ranges are inclusive: so port 9300 is part of the first and the second port range.
Now, my IT ops department won't like that, so hopefully I got it wrong. Anyone knows?
Elasticsearch will bind to a single port for both HTTP and the node/transport APIs.
It'll try the lowest available port first, and if it is already taken, try the next. If you run a single node on your machine, it'll only bind to 9200 and 9300.
See also: Elasticsearch Internals: Networking Introduction