I am trying to access Elastic Search via my static IP address but it's not working.
What did I try?
I created a Bitnami Elastic Search VM Instance from GCP Marketplace
I assigned a static IP to the same VM
I replaced the network.host to 0.0.0.0 inside elasticsearch.yml file
I added my static IP to network.publish_host inside elasticsearch.yml file
I added a firewall rule to allow all ports and added 0.0.0.0 as source filter
Now when trying to access Elastic Search using http://_my_static_ip:9200 I get nothing, the request fails. What am I missing here?? Any help would be appreciated. Thanks
The issue was my GCP being using an IPv6 address, I didn't know about this, this is something a developer on Fiverr told me, anyone having the same issue with Bitnami's GCP deployment needs to add the following line:
-Djava.net.preferIPv4Stack=true
to the following file:
/opt/bitnami/elasticsearch/config/jvm.options
After that restart your elasticsearc service using the following command:
sudo /opt/bitnami/ctlscript.sh restart
That should fix the issue if you have proper firewall rules set up and also added proper IPs to elasticsearch.yml config file. Read original question's What did I try? section.
I wish to run my elasticsearch remotely on gcloud VM, this is configured to run at 127.0.0.1 at a specific port 9200. How to access this from a website outside this vm? If I change the network host to 0.0.0.0 on the yml file, even 9200 port becomes inaccessible. How do I overcome this problem?
Changed network.host: [_site_ , _local_ , _global_ ]
_site_ = internal ip given by google cloud vm,
_local_ = 127.0.0.1,
_global_ = found using curl ifconfig.me,
Opened a specific port (9200) and tried to connect with global IP address.
curl to the global ip gives
>Output: Failed to connect to (_global_ ip) port 9200: connection refused.
So put network.host:0.0.0.0 and then try to allow 9200 and 9201 port and restart the elasticsearch service.If you are using ubuntu then sudo service elasticsearch restart then check by doing curl -XGET 'http://localhost:9200?pretty'.Let me know if you are still facing any issues.
Use following configurations for elasticsearch.yml
network.host: 0.0.0.0
action.auto_create_index: false
index.mapper.dynamic: false
Solved this problem by going through the logs and found out that the public ip address is re-mapped to the internal ip address, hence network.host can't be set to external ip directly. Elasticsearch yml config is as follows:
'network.host: xx.xx.xxx.xx' is set to the internal ip (given by google),
'http.cors.enabled: true',
'http.cors.allow-origin:"*", (Do not use * in production, its a security issue)
'discovery.type: single-node' in my case to make it work independently and not in a cluster
Now this sandboxed version can be accessed from outside the VM using the external IP address given by Google.
I installed hadoop cluster using bdutil (instead of click to deploy). I am not able to access job tracker page at locahost:50030/jobtracker.jsp (https://cloud.google.com/hadoop/running-a-mapreduce-job)
I am checking it locally using lynx instead of from my client browser (so localhost instead of external ip)
My setting in my config file for bdutil is
MASTER_UI_PORTS=('8088' '50070' '50030')
but after deploying the hadoop cluster when I do firewall rules list I get following
NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS
default-allow-http default 0.0.0.0/0 tcp:80,tcp:8080 http-server
default-allow-https default 0.0.0.0/0 tcp:443 https-server
default-allow-icmp default 0.0.0.0/0 icmp
default-allow-internal default 10.240.0.0/16 tcp:1-65535,udp:1-65535,icmp
default-allow-rdp default 0.0.0.0/0 tcp:3389
default-allow-ssh default 0.0.0.0/0 tcp:22
Now I dont see port 50030 in the list of rules. Why so?
so I run a command to add them (manually)
gcloud compute firewall-rules create allow-http --description "Incoming http allowed." --allow tcp:50030 --format json
Now it gets added and I can see in the output of firewall-rules list command.
But still when I do lynx locahost:50030/jobtracker.jsp I get unable to connect. Then, I run a hadoop job so that there is some output to view and then run lynx command but still see unable to connect.
Can someone tell me where I am going wrong in this complete process?
An ephemeral IP is an external IP. The difference between an ephemeral IP and a static IP is that a static IP can be reassigned to another virtual machine instance, while an ephemeral IP is released when the instance is destroyed. An ephemeral IP can be promoted to a static IP through the web UI or the gcloud command-line tool.
You can obtain the external IP of your host by querying the metadata API at http://169.254.169.254/0.1/meta-data/network. The response will be a JSON document that looks like this (pretty-printed for clarity):
{
"networkInterface" : [
{
"network" : "projects/852299914697/networks/rabbit",
"ip" : "10.129.14.59",
"accessConfiguration" : [
{
"externalIp" : "107.178.223.11",
"type" : "ONE_TO_ONE_NAT"
}
]
}
]
}
The firewall rule command seems reasonable, but you may want to choose a more descriptive name. If I saw a rule that said allow-http, I would assume it meant port 80. You may also want to restrict it to a target tag placed on your Hadoop dashboard instance; as written, your rule will allow access on that port to all instances in the current project.
How can allow only local requests for elasticsearch?
So command like:
curl -XGET 'http://localhost:9200/twitter/_settings'
can only be running on localhost and request like:
curl -XGET 'http://mydomain.com:9200/twitter/_settings'
would get rejected?
Because, from what i see, elasticsearch allows it by default.
EDIT:
According to http://www.elasticsearch.org/guide/reference/modules/network.html
you can manage bind_host parameter to allow hosts. And by default, it is set to anyLocalAddress
For elasticsearch prior to v2.0.0, if you want both http transport and internal elasticsearch transport to listen only on localhost simply add the following line to elasticsearch.yml file.
network.host: "127.0.0.1"
If you want only http transport to listen on localhost add the following line instead.
http.host: "127.0.0.1"
Starting from v2.0 elasticsearch is listening only on localhost by default. So, no additional configuration is needed.
If your final goal is to deny any requests from outside the host machine, the most reliable way would be to modify the host's iptables so that it denies any incoming requests to the service ports used by ElasticSearch (9200-9300).
If the end goal is to make sure that everyone refers to the service using an exclusive DNS, you're better off achieving this with an HTTP server that can proxy requests such as HTTPd or nginx.
I use this parameter:
http.host: "127.0.0.1"
This parameter not accept http requests for external request.
I want to setup my local development machine so that any requests for *.local are redirected to localhost. The idea is that as I develop multiple sites, I can just add vhosts to Apache called site1.local, site2.local etc, and have them all resolve to localhost, while Apache serves a different site accordingly.
I am on Windows XP.
I tried adding
127.0.0.1 *.local
to my c:\windows\system32\drivers\etc\hosts file, also tried:
127.0.0.1 .local
Neither of which seem to work.
I know I can set them up on different port numbers, but that is a pain since it is hard to remember which port is which.
I don't want to have to setup a local DNS server or anything hard, any suggestions?
Acrylic DNS Proxy (free, open source) does the job. It creates a proxy DNS server (on your own computer) with its own hosts file. The hosts file accepts wildcards.
Download from the offical website
http://mayakron.altervista.org/support/browse.php?path=Acrylic&name=Home
Configuring Acrylic DNS Proxy
To configure Acrylic DNS Proxy, install it from the above link then go to:
Start
Programs
Acrylic DNS Proxy
Config
Edit Custom Hosts File (AcrylicHosts.txt)
Add the folowing lines on the end of the file:
127.0.0.1 *.localhost
127.0.0.1 *.local
127.0.0.1 *.lc
Restart the Acrylic DNS Proxy service:
Start
Programs
Acrilic DNS Proxy
Config
Restart Acrylic Service
You will also need to adjust your DNS setting in you network interface settings:
Start
Control Panel
Network and Internet
Network Connections
Local Area Connection Properties
TCP/IPv4
Set "Use the following DNS server address":
Preferred DNS Server: 127.0.0.1
If you then combine this answer with jeremyasnyder's answer (using VirtualDocumentRoot) you can then automatically setup domains/virtual hosts by simply creating a directory.
To answer your question, you cannot use wildcards in the hosts file under Windows.
However, if you want to only change the hosts file to make new sites work.... you can configure your Apache like this and you don't have to keep editing it's config:
http://postpostmodern.com/instructional/a-smarter-mamp/
Basically a quick summary based on my setup, add the following to your apache.conf file:
LoadModule vhost_alias_module modules/mod_vhost_alias.so
NameVirtualHost *:80
<Directory "/xampp/sites">
Options Indexes FollowSymLinks Includes ExecCGI
AllowOverride All
Order allow,deny
Allow from all
</Directory>
<VirtualHost *:80>
VirtualDocumentRoot c:/xampp/sites/%-1/%-2+/
</VirtualHost>
This allows me to add an entry like:
127.0.0.1 test.dev
and then make the directory, c:\xampp\sites\dev\test and place the necessary files in there and it just works.
The other option is to use <Directory> tags in apache.conf and reference the pages from http://localhost/project/.
I don't think that it is possible.
You anyway have to modify the apache virtualroot entries every time you add a new site and location, so it's not a big work to syncronise the new name to the Windows vhost file.
Update: please check the next answer and the comments on this answer. This answer is 6 years old and not correct anymore.
To add to the great suggestions already here, XIP.IO is a fantastic wildcard DNS server that's publicly available.
myproject.127.0.0.1.xip.io -- resolves to --> 127.0.0.1
other.project.127.0.0.1.xip.io -- resolves to --> 127.0.0.1
other.machine.10.0.0.1.xip.io -- resolves to --> 10.0.0.1
(The ability to specify non-loopback addresses is fantastic for testing sites on iOS devices where you cannot access a hosts file.)
If you combine this with some of the Apache configuration mentioned in other answers, you can potentially add VirtualHosts with zero setup.
Editing the hosts file is less of a pain when you run "ipconfig /flushdns" from the windows command prompt, instead of restarting your computer.
I found a posting about Using the Windows Hosts File that also says "No wildcards are allowed."
In the past, I have just added the additional entries to the hosts file, because (as previously said), it's not that much extra work when you already are editing the apache config file.
You could talk your network administrator into setting up a domain for you (say 'evilpuppetmaster.hell') and having the wildcard there so that everything (*.evilpuppetmaster.hell') resolves to your IP
We have this working using wildcard DNS in our local DNS server: add an A record something like *.local -> 127.0.0.1
I think that your network settings will need to have the chosen domain suffix in the domain suffix search list for machines on the network, so you might want to replace .local with your company's internal domain (e.g. .int) and then add a subdomain like .localhost.int to make it clear what it's for.
So *.localhost.int would resolve to 127.0.0.1 for everybody on the network, and config file settings for all developers would "just work" if endpoints hang off that subdomain e.g. site1.localhost.int, site2.localhost.int This is pretty much the scheme we have introduced.
dnsmasq also looks nice, but I have not tried it yet:
http://ihaveabackup.net/2012/06/28/using-wildcards-in-the-hosts-file/
I have written a simple dns proxy in Python. It will read wildcard entries in /etc/hosts. See here: http://code.google.com/p/marlon-tools/source/browse/tools/dnsproxy/dnsproxy.py
I have tested in Linux & Mac OS X, but not yet in Windows.
You may try AngryHosts, which provided a way to support wildcard and regular expression. Actually, it's a hosts file enhancement and management software.
More features can be seen # http://angryhosts.com/features/
I'm using DNSChef to do that.
https://thesprawl.org/projects/dnschef/
You have to download the app, in Linux or Mac you need python to run it. Windows have their own exe.
You must create a ini file with your dns entries, for example
[A]
*.google.com=192.0.2.1
*.local=127.0.0.1
*.devServer1.com=192.0.2.3
Then you must launch the dns application with admin privileges
sudo python dnschef.py --file myfile.ini -q
or in windows
runas dnschef.exe --file myfile.ini -q
Finally you need to setup as your only DNS your local host environment (network, interface, dns or similar or in linux /etc/resolv.conf).
That's it
I made this simple tool to take the place of hosts. Regular expressions are supported.
https://github.com/stackia/DNSAgent
A sample configuration:
[
{
"Pattern": "^.*$",
"NameServer": "8.8.8.8"
},
{
"Pattern": "^(.*\\.googlevideo\\.com)|((.*\\.)?(youtube|ytimg)\\.com)$",
"Address": "203.66.168.119"
},
{
"Pattern": "^.*\\.cn$",
"NameServer": "114.114.114.114"
},
{
"Pattern": "baidu.com$",
"Address": "127.0.0.1"
}
]
#petah and Acrylic DNS Proxy is the best answer, and at the end he references the ability to do multi-site using an Apache which #jeremyasnyder describes a little further down...
... however, in our case we're testing a multi-tenant hosting system and so most domains we want to test go to the same virtualhost, while a couple others are directed elsewhere.
So in our case, you simply use regex wildcards in the ServerAlias directive, like so...
ServerAlias *.foo.local
Here is the total configuration for those trying to accomplish the goal (wildcards in dev environment ie, XAMPP -- this example assumes all sites pointing to same codebase)
hosts file (add an entry)
file: %SystemRoot%\system32\drivers\etc\hosts
127.0.0.1 example.local
httpd.conf configuration (enable vhosts)
file: \XAMPP\etc\httpd.conf
# Virtual hosts
Include etc\extra\httpd-vhosts.conf
httpd-vhosts.conf configuration
file: XAMPP\etc\extra\httpd-vhosts.conf
<VirtualHost *:80>
ServerAdmin admin#example.local
DocumentRoot "\path_to_XAMPP\htdocs"
ServerName example.local
ServerAlias *.example.local
# SetEnv APP_ENVIRONMENT development
# ErrorLog "logs\example.local-error_log"
# CustomLog "logs\example.local-access_log" common
</VirtualHost>
restart apache
create pac file:
save as whatever.pac wherever you want to and then load the file in the browser's network>proxy>auto_configuration settings (reload if you alter this)
function FindProxyForURL(url, host) {
if (shExpMatch(host, "*example.local")) {
return "PROXY example.local";
}
return "DIRECT";
}
You can use echoipdns for this (https://github.com/zapty/echoipdns).
By running echoipdns local all requests for .local subdomains are redirected to 127.0.0.1, so any domain with xyz.local etc will resolve to 127.0.0.1. You can use any other suffix also just replace local with name you want.
Echoipdns is even more powerful, when you want to use your url from other machines in network you can still use it with zero configuration.
For e.g. If your machine ip address is 192.168.1.100 you could now use a domain name xyz.192-168-1-100.local which will always resolve to 192.168.1.100. This magic is done by the echoipdns by looking at the ip address in the second part of the domain name and returning the same ip address on DNS query. You will have to run the echoipdns on the machine from which you want to access the remote system.
echoipdns also can be setup as a standalone DNS proxy, so by just point to this DNS, you can now use all the above benefits without running a special command every time, and you can even use it from mobile devices.
So essentially this simplifies the wildcard domain based DNS development for local as well as team environment.
echoipdns works on Mac, Linux and Windows.
NOTE: I am author for echoipdns.
I could not find a prohibition in writing, but by convention, the Windows hosts file closely follows the UNIX hosts file, and you cannot put wildcard hostname references into that file.
If you read the man page, it says:
DESCRIPTION
The hosts file contains information regarding the known hosts on the net-
work. For each host a single line should be present with the following
information:
Internet address
Official host name
Aliases
Although it does say,
Host names may contain any printable character other than a field delim-
iter, newline, or comment character.
that is not true from a practical level.
Basically, the code that looks at the /etc/hosts file does not support a wildcard entry.
The workaround is to create all the entries in advance, maybe use a script to put a couple hundred entries at once.
While you can't add a wildcard like that, you could add the full list of sites that you need, at least for testing, that works well enough for me, in your hosts file, you just add:
127.0.0.1 site1.local
127.0.0.1 site2.local
127.0.0.1 site3.local
...
Configuration for nginx config auto subdomain with Acrylic DNS Proxy
auto.conf file for your nginx sites folder
server {
listen 80;
server_name ~^(?<branch>.*)\.example\.com;
root /var/www/html/$branch/public;
index index.html index.htm index.php;
charset utf-8;
location / {
try_files $uri $uri/ /index.php$is_args$args;
}
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
error_log /var/log/nginx/$branch.error.log error;
sendfile off;
client_max_body_size 100m;
location ~ \.php$ {
try_files $uri /index.php =404;
fastcgi_pass php-fpm:9000;
fastcgi_index index.php;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ /\.ht {
deny all;
}
}
Add to Acrylic hosts file 127.0.0.1 example.com *.example.com and restart Acrylic service.
$branch - your subdomain name.
Set instead of root /var/www/html/$branch/public; your project path
This can be done using Pi-Hole, just edit the "/etc/hosts" and restart dns service.
nano /etc/hosts
pihole restartdns
Example:
127.0.1.1 raspberrypi
192.168.1.1 w1.dev.net
192.168.1.2 w2.dev.net
192.168.1.3 w3.dev.net
You can use a dynamic DNS client such as http://www.no-ip.com. Then, with an external DNS server CNAME *.mydomain.com to mydomain.no-ip.com.