Instead of working with google cloud, I decided to set up Kuberenetes on my own machine. I made a docker image of my hello-world web server. I set up hello-controller.yaml:
apiversion: v1
kind: ReplicationController
metadata:
name: hello
labels:
name: hello
spec:
replicas: 1
selector:
name: hello
template:
metadata:
labels:
name: hello
spec:
containers:
- name: hello
image: flaggy/hello
ports:
- containerPort: 8888
Now I want to expose the service to the world. I don't think vagrant provider has a load balancer (which seems to be the best way to do it). So I tried with NodePort service type. However, the newly created NodePort does not seem to be listened on any IP I try. Here's hello-service.yaml:
apiversion: v1
kind: Service
metadata:
name: hello
labels:
name: hello
spec:
type: NodePort
selector:
name: hello
ports:
- port: 8888
If I log into my minion I can access port 8888
$ curl 10.246.1.3:8888
Hello!
When I describe my service this is what I get:
$ kubectl.sh describe service/hello
W0628 15:20:45.049822 1245 request.go:302] field selector: v1 - events - involvedObject.name - hello: need to check if this is versioned correctly.
W0628 15:20:45.049874 1245 request.go:302] field selector: v1 - events - involvedObject.namespace - default: need to check if this is versioned correctly.
W0628 15:20:45.049882 1245 request.go:302] field selector: v1 - events - involvedObject.kind - Service: need to check if this is versioned correctly.
W0628 15:20:45.049887 1245 request.go:302] field selector: v1 - events - involvedObject.uid - 2c0005e7-1dc2-11e5-8369-0800279dd272: need to check if this is versioned correctly.
Name: hello
Labels: name=hello
Selector: name=hello
Type: NodePort
IP: 10.247.5.87
Port: <unnamed> 8888/TCP
NodePort: <unnamed> 31423/TCP
Endpoints: 10.246.1.3:8888
Session Affinity: None
No events.
I cannot find anyone listening on port 31423 which, as I gather, should be the external port for my service. I am also puzzled about IP 10.247.5.87.
I note this
$ kubectl.sh get nodes
NAME LABELS STATUS
10.245.1.3 kubernetes.io/hostname=10.245.1.3 Ready
Why is that IP different from what I see on describe for the service? I tried accessing both IPs on my host:
$ curl 10.245.1.3:31423
curl: (7) Failed to connect to 10.245.1.3 port 31423: Connection refused
$ curl 10.247.5.87:31423
curl: (7) Failed to connect to 10.247.5.87 port 31423: No route to host
$
So IP 10.245.1.3 is accessible, although port 31423 is not binded to it. I tried routing 10.247.5.87 to vboxnet1, but it didn't change anything:
$ sudo route add -net 10.247.5.87 netmask 255.255.255.255 vboxnet1
$ curl 10.247.5.87:31423
curl: (7) Failed to connect to 10.247.5.87 port 31423: No route to host
If I do sudo netstat -anp | grep 31423 on the minion nothing comes up. Strangely, nothing comes up if I do sudo netstat -anp | grep 8888 either.
There must be either some iptables magic or some interface in promiscuous mode being abused.
Would it be this difficult to get things working on bare metal as well? I haven't tried AWS provider either, but I am getting worried.
A few things.
Your single pod is 10.246.1.3:8888 - that seems to work.
Your service is 10.247.5.87:8888 - that should work as long as you are within your cluster (it's virtual - you will not see it in netstat). This is the first thing to verify.
Your node is 10.245.1.3 and your service should ALSO be on 10.245.1.3:31423 - this is the part that does not seem to be working correctly. Like service IPs, this binding is virtual - it should show up in iptables-save but not netstat. If you log into your node (minion), can you curl localhost:31423 ?
You might find this doc useful: https://github.com/thockin/kubernetes/blob/docs-debug-svcs/docs/debugging-services.md
Related
I have a spring boot application which is deployed in Kubernetes on local windows machine using minikube. I also have Elasticsearch running on my local machine (http://localhost:9200).
I want to call Elasticsearch REST endpoints from this spring boot app.
I tried solving this by creating a service without selector but not sure what am i missing.
When accessing the spring boot app using http://#minikube_ip#:#Node_Port#, i get an error "No route to host".
i tried doing minikube ssh and executing curl command, from there also i get the same error. Clearly I am missing something here.
application.yaml
elasticsearch:
hosts:
- http://my-es:80
connectTimeout: 10000
connectionRequestTimeout: 10000
socketTimeout: 10000
maxRetryTimeoutMillis: 60000
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kube-es-app
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
run: kube-es-app
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: kube-es-app
spec:
containers:
- image: elastic-search-app:latest
imagePullPolicy: Never
name: kube-es-app
ports:
- containerPort: 8080
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
---
kind: Service
apiVersion: v1
metadata:
name: my-es
spec:
ports:
- protocol: TCP
port: 80
targetPort: 9200
---
kind: Endpoints
apiVersion: v1
metadata:
name: my-es
subsets:
- addresses:
- ip: <MY_LOCAL_MACHINE_IP>
ports:
- port: 9200
Commands I executed
docker build -t elastic-search-app .
kubectl create -f deployment.yaml
kubectl expose deployment/kube-es-app --type="NodePort" --port 8080
Can anyone help please? I am stuck
If I've got the description right, the Windows machine should have vbox network adapter connected to the Host-only-network the Minikube VM is connected to.
Minikube can access the host machine directly because both are in the same network.
The Minikube is in charge of NAT-ting packages from Pods outside. What you need is to allow Elasticsearch to listen to the vbox- or all interfaces, and enable its port in the Windows firewall. Then the Elasticsearch should be available via IP address of Windows in the Host-only-network.
Apart from that, you might create a service (if you need go by name instead of IP) as discussed here:
Connect to local database from inside minikube cluster,
Minikube:Exposing mysql as a service on localhost.
I am new to Kubernetes and I am trying to create a simple front-end back-end application where front-end and back-end will have its own services. For some reason, I am not able to access back-end service by its name from front-end service.
Just because of simplicity, front-end service can be created like:
kubectl run curl --image=radial/busyboxplus:curl -i --tty
When I do a nslookup I get the following:
[ root#curl-66bdcf564-rbx2k:/ ]$ nslookup msgnc-travel
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: msgnc-travel
Address 1: 10.100.171.209 msgnc-travel.default.svc.cluster.local
Service is available by its name msgnc-travel, but when I try to do curl it:
curl msgnc-travel
it just keeps on waiting and no response is received. I have also tried
curl 10.100.171.209 and curl msgnc-travel.default.svc.cluster.local but I have the same behaviour
Any ideas why is this issue occurring?
I have successfully managed to do a "workaround" by using Ingress, but I am curious why can't I access my Spring Boot backend service directly just by providing its name?
deployment.yml looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: msgnc-travel-deployment
labels:
name: msgnc-travel-deployment
app: msgnc-travel-app
spec:
template:
metadata:
name: msgnc-travel-pod
labels:
name: msgnc-travel-pod
app: msgnc-travel-app
spec:
containers:
- name: msgnc-travel
image: bdjordjevic/msgnc-travel
ports:
- containerPort: 8080
replicas: 1
selector:
matchExpressions:
- {key: name, operator: In, values: [msgnc-travel-pod]}
- {key: app, operator: In, values: [msgnc-travel-app]}
service.yml looks like this:
apiVersion: v1
kind: Service
metadata:
name: msgnc-travel
labels:
name: msgnc-travel-service
app: msgnc-travel-app
spec:
ports:
- port: 8080
targetPort: 8080
selector:
name: msgnc-travel-pod
app: msgnc-travel-app
You are defining the service to listen at port 8080. So you are supposed to execute curl msgnc-travel:8080.
I tried running wget and this is the output I got:
wget msgnc-travel:8080
Connecting to msgnc-travel:8080 (10.98.81.45:8080)
wget: server returned error: HTTP/1.1 404
I have a minikube cluster running on Mac OSX and a simple Spring Boot REST api that connects to Redis and Mongo DB, which I have installed and running locally.
I wish not to run Redis / MongoDb in a Docker container.
I will probably run them remotely in the cloud, therefore I would probably just connect to an external IP address.
What I don't understand is what IP address I should use to connect to my localhost machine.
I start up my Minikube with VM hyperkit.
Edit:
I also tried to start using virtualbox:
minikube start --vm-driver=virtualbox
In my spring boot application, I've configured:
spring.data.mongodb.host = 10.0.2.2
spring.redis.host = 10.0.2.2
But still, I get connection errors:
This works when I run the application locally on my host machine.
For the sake of completeness, this is my yaml file:
---
apiVersion: v1
kind: Service
metadata:
name: posts-api
labels:
app: posts-api
env: dev
spec:
type: NodePort
selector:
app: posts-api
ports:
- protocol: TCP
port: 8083
name: http
---
apiVersion: v1
kind: ReplicationController
metadata:
name: posts-api
spec:
replicas: 1
template:
metadata:
labels:
app: posts-api
spec:
containers:
- name: posts-api
image: kimgysen/posts-api:latest
ports:
- containerPort: 8083
livenessProbe:
httpGet:
path: /health
port: 8083
initialDelaySeconds: 120
timeoutSeconds: 3
I'll give you the answer I gave to someone with the same problem (different tech):
Kubernetes pod unable to connect to rabbit mq instance running locally
Replace the IP and port number, and the Service and Endpoints names as appropriate.
I want to access my Grafana Kubernetes service via the kubectl proxy server, but for some reason it won't work even though I can make it work for other services. Given the below service definition, why is it not available on http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana?
grafana-service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: monitoring
name: grafana
labels:
app: grafana
spec:
type: NodePort
ports:
- name: web
port: 3000
protocol: TCP
nodePort: 30902
selector:
app: grafana
grafana-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: monitoring
name: grafana
spec:
replicas: 1
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:4.1.1
env:
- name: GF_AUTH_BASIC_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "true"
- name: GF_SECURITY_ADMIN_USER
valueFrom:
secretKeyRef:
name: grafana-credentials
key: user
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-credentials
key: password
volumeMounts:
- name: grafana-storage
mountPath: /var/grafana-storage
ports:
- name: web
containerPort: 3000
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 200Mi
cpu: 200m
- name: grafana-watcher
image: quay.io/coreos/grafana-watcher:v0.0.5
args:
- '--watch-dir=/var/grafana-dashboards'
- '--grafana-url=http://localhost:3000'
env:
- name: GRAFANA_USER
valueFrom:
secretKeyRef:
name: grafana-credentials
key: user
- name: GRAFANA_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-credentials
key: password
resources:
requests:
memory: "16Mi"
cpu: "50m"
limits:
memory: "32Mi"
cpu: "100m"
volumeMounts:
- name: grafana-dashboards
mountPath: /var/grafana-dashboards
volumes:
- name: grafana-storage
emptyDir: {}
- name: grafana-dashboards
configMap:
name: grafana-dashboards
The error I'm seeing when accessing the above URL is "no endpoints available for service "grafana"", error code 503.
With Kubernetes 1.10 the proxy URL should be slighly different, like this:
http://localhost:8080/api/v1/namespaces/default/services/SERVICE-NAME:PORT-NAME/proxy/
Ref: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#manually-constructing-apiserver-proxy-urls
As Michael says, quite possibly your labels or namespaces are mismatching. However in addition to that, keep in mind that even when you fix the endpoint, the url you're after (http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana) might not work correctly.
Depending on your root_url and/or static_root_path grafana configuration settings, when trying to login you might get grafana trying to POST to http://localhost:8001/login and get a 404.
Try using kubectl port-forward instead:
kubectl -n monitoring port-forward [grafana-pod-name] 3000
then access grafana via http://localhost:3000/
https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
The issue is that Grafana's port is named web, and as a result one needs to append :web to the kubectl proxy URL: http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana:web.
An alternative, is to instead not name the Grafana port, because then you don't have to append :web to the kubectl proxy URL for the service: http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana:web. I went with this option in the end since it's easier.
There are a few factors that might be causing this issue.
The service expects to find one or more supporting endpoints, which it discovers through matching rules on the labels. If the labels don't align, then the service won't find endpoints, and the network gateway function performed by the service will result in 503.
The port declared by the POD and the process within the container are misaligned from the --target-port expected by the service.
Either one of these might generate the error. Let's take a closer look.
First, kubectl describe the service:
$ kubectl describe svc grafana01-grafana-3000
Name: grafana01-grafana-3000
Namespace: default
Labels: app=grafana01-grafana
chart=grafana-0.3.7
component=grafana
heritage=Tiller
release=grafana01
Annotations: <none>
Selector: app=grafana01-grafana,component=grafana,release=grafana01
Type: NodePort
IP: 10.0.0.197
Port: <unset> 3000/TCP
NodePort: <unset> 30905/TCP
Endpoints: 10.1.45.69:3000
Session Affinity: None
Events: <none>
Notice that my grafana service has 1 endpoint listed (there could be multiple). The error above in your example indicates that you won't have endpoints listed here.
Endpoints: 10.1.45.69:3000
Let's take a look next at the selectors. In the example above, you can see I have 3 selector labels on my service:
Selector: app=grafana01-grafana,component=grafana,release=grafana01
I'll kubectl describe my pods next:
$ kubectl describe pod grafana
Name: grafana01-grafana-1843344063-vp30d
Namespace: default
Node: 10.10.25.220/10.10.25.220
Start Time: Fri, 14 Jul 2017 03:25:11 +0000
Labels: app=grafana01-grafana
component=grafana
pod-template-hash=1843344063
release=grafana01
...
Notice that the labels on the pod align correctly, hence my service finds pods which provide endpoints which are load balanced against by the service. Verify that this part of the chain isn't broken in your environment.
If you do find that the labels are correct, you may still have a disconnect in that the grafana process running within the container within the pod is running on a different port than you expect.
$ kubectl describe pod grafana
Name: grafana01-grafana-1843344063-vp30d
...
Containers:
grafana:
Container ID: docker://69f11b7828c01c5c3b395c008d88e8640c5606f4d865107bf4b433628cc36c76
Image: grafana/grafana:latest
Image ID: docker-pullable://grafana/grafana#sha256:11690015c430f2b08955e28c0e8ce7ce1c5883edfc521b68f3fb288e85578d26
Port: 3000/TCP
State: Running
Started: Fri, 14 Jul 2017 03:25:26 +0000
If for some reason, your port under the container listed a different value, then the service is effectively load balancing against an invalid endpoint.
For example, if it listed port 80:
Port: 80/TCP
Or was an empty value
Port:
Then even if your label selectors were correct, the service would never find a valid response from the pod and would remove the endpoint from the rotation.
I suspect your issue is the first problem above (mismatched label selectors).
If both the label selectors and ports align, then you might have a problem with the MTU setting between nodes. In some cases, if the MTU used by your networking layer (like calico) is larger than the MTU of the supporting network, then you'll never get a valid response from the endpoint. Typically, this last potential issue will manifest itself as a timeout rather than a 503 though.
Your Deployment may not have a label app: grafana, or be in another namespace. Could you also post the Deployment definition?
I'm trying to run the kubernetes guestbook app on Vagrant on my local Mac OS X machine.
I have all the nodes (master and node-1) running, by running the following:
./kubectl.sh create -f ../examples/guestbook/all-in-one/guestbook-all-in-one.yaml
As I'm running locally, I also changed the about yaml file, to use NodePort instead of LoadBalancer
Running the following:
./kubectl.sh describe service frontend returns the following:
Name: frontend
Labels: app=guestbook,tier=frontend
Selector: app=guestbook,tier=frontend
Type: NodePort
IP: 10.247.127.146
Port: <unnamed> 80/TCP
NodePort: <unnamed> 31030/TCP
Endpoints: 10.246.9.12:80,10.246.9.13:80,10.246.9.7:80
Session Affinity: None
No events.
If I try: http://10.247.127.146:31030 It doesn't conenct to the guestbook app.
Is there anything I'm doing wrong?
Thanks
nodePort is the port available on every kubernetes node.
You need to find the IP address of your kubernetes node, and then http://<nodeIP>:31030 should work.