Two of my three slave VMs are down and I can't ssh them. We have performed a hard reboot but still they are down. Any idea how to bring them back or how to debug to find the reason. Here's what jps:
3542 RunJar
9920 SecondaryNameNode
10094 ResourceManager
10244 NodeManager
8677 DataNode
31634 Jps
8536 NameNode
Here's also another detail:
ubuntu#anmol-vm1-new:~$ sudo netstat -atnp | grep 8020
tcp 0 0 10.0.1.190:8020 0.0.0.0:* LISTEN 8536/java
tcp 0 0 10.0.1.190:50957 10.0.1.190:8020 ESTABLISHED 8677/java
tcp 0 0 10.0.1.190:8020 10.0.1.190:50957 ESTABLISHED 8536/java
tcp 0 0 10.0.1.190:8020 10.0.1.193:46627 ESTABLISHED 8536/java
tcp 0 0 10.0.1.190:44300 10.0.1.190:8020 TIME_WAIT -
tcp 0 0 10.0.1.190:8020 10.0.1.190:44328 ESTABLISHED 8536/java
tcp 0 0 10.0.1.190:8020 10.0.1.193:44610 ESTABLISHED 8536/java
tcp6 0 0 10.0.1.190:44292 10.0.1.190:8020 TIME_WAIT -
tcp6 0 0 10.0.1.190:44328 10.0.1.190:8020 ESTABLISHED 10244/java
tcp6 0 0 10.0.1.190:44252 10.0.1.190:8020 TIME_WAIT -
tcp6 0 0 10.0.1.190:44247 10.0.1.190:8020 TIME_WAIT -
tcp6 0 0 10.0.1.190:44287 10.0.1.190:8020 TIME_WAIT -
When I run the following command:
hadoop fsck /
the result is:
The filesystem under path '/' is CORRUPT
Here's more details in this pastebin.
If they are down and if you cannot ssh them, that means your filesystem can be full. You have to login using VM console and clean up the file system, ssh will not work any more.
Related
I tied to implement an HTTP server in Azure Linux VM using golang. Below is the simple golang server code, listening on port 30175. And there is no firewall on that port.
package main
import (
"fmt"
"log"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}
func main() {
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":30175", nil))
}
The result of sudo netstat -tlnp is:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1605/vsftpd
tcp 0 0 127.0.0.1:3350 0.0.0.0:* LISTEN 1873/xrdp-sesman
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1697/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1379/cupsd
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 4879/8
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN 15507/9
tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN 1859/xrdp
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 2112/python
tcp6 0 0 :::22 :::* LISTEN 1697/sshd
tcp6 0 0 ::1:631 :::* LISTEN 1379/cupsd
tcp6 0 0 ::1:6010 :::* LISTEN 4879/8
tcp6 0 0 ::1:6011 :::* LISTEN 15507/9
tcp6 0 0 :::30175 :::* LISTEN 46595/HttpHandler
I can get response only in localhost, but no response from remote server:
curl localhost:30175
Hi there, I love !
curl serveripaddress:30175
not working
Your problem is that your server is listening on tcp6 stack. Try to explicitly use tcp with “0.0.0.0:6789” instead of just port “:6789”
This has nothing to do with your code. It is a typical firewall issue.
By default (on *nix platform) all the incoming traffic is blocked and all outgoing is allowed. You need to open the ports on your OS to allow incoming traffic to hit your servers. try installing ufw utility and run sudo ufw allow 30175
From the question, it seems your server is using tcp6. Ideally, it should not cause the problem because tcp6 is supposed to support both IPV4 as well as IPV6. But I would recommend you to downgrade it to tcp, if that makes sense.
You are trying to output to console. Did you try to send it as a response?
func handler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(fmt.Sprintf("Hi there, I love %s!", r.URL.Path[1:])));
}
No if you don't specify the host part of the address the server will listen on every available unicast address and every available anycast address of the system. So I would guess a problem in name resolution or routing.
This is due to Linux listen rules.
There is a reject all rule on my rules.
# listen rules
sudo iptables -L INPUT --line-numbers
sudo iptables -D INPUT 8
Should it be UDP or TCP? If it's the latter one, wouldn't it cost too much overhead on network via three-time shaking and four-time waving?
$ sudo netstat -nap | grep etcd
[sudo] password for emma:
tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN 21731/etcd
tcp 0 0 127.0.0.1:4001 0.0.0.0:* LISTEN 21731/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 21731/etcd
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 21731/etcd
Using Gstreamer latest version
gst-launch-1.0 version 1.8.1
GStreamer 1.8.1
command :
gst-launch-1.0 -m -v --gst-debug-level=2 autovideosrc ! video/x-raw,width=640,height=480 ! videoconvert ! tcpclientsink host="hidden_ip" port=1935
to stream to crtmpserver ( on ubuntu AWS ec2 ) which has following open , publicly accessible and working ports
sudo netstat -putan|grep crtmp
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 7235/crtmpserver
tcp 0 0 0.0.0.0:1935 0.0.0.0:* LISTEN 7235/crtmpserver
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 11734/crtmpserver
tcp 0 0 0.0.0.0:1935 0.0.0.0:* LISTEN 11734/crtmpserver
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 11689/crtmpserver
tcp 0 0 0.0.0.0:1935 0.0.0.0:* LISTEN 11689/crtmpserver
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 7235/crtmpserver
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 11734/crtmpserver
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 11689/crtmpserver
tcp 0 0 0.0.0.0:6665 0.0.0.0:* LISTEN 7235/crtmpserver
tcp 0 0 0.0.0.0:6665 0.0.0.0:* LISTEN 11734/crtmpserver
tcp 0 0 0.0.0.0:6665 0.0.0.0:* LISTEN 11689/crtmpserver
tcp 0 0 0.0.0.0:6666 0.0.0.0:* LISTEN 7235/crtmpserver
tcp 0 0 0.0.0.0:6666 0.0.0.0:* LISTEN 11734/crtmpserver
tcp 0 0 0.0.0.0:6666 0.0.0.0:* LISTEN 11689/crtmpserver
Error :
ERROR: from element /GstPipeline:pipeline0/GstTCPClientSink:tcpclientsink0: Error while sending data to "hidden_ip:1935".
Additional debug info:
gsttcpclientsink.c(217): gst_tcp_client_sink_render (): /GstPipeline:pipeline0/GstTCPClientSink:tcpclientsink0:
Only 236024 of 614400 bytes written: Error sending data: Broken pipe
Execution ended after 0:00:01.308461393
complete error trace
http://pastebin.com/aEFYb5q6
All help much appreciated ,
Thanks in anticipation
update :
After some workarounds on iptable and gst plugins
Im am finally getting seeking query error . Any ideas on how to solve this will be very welcomed
gst script from ubuntu 15.10
gst-launch-1.0 -m -v --gst-debug-level=2 v4l2src ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! flvmux ! rtmpsink location ='rtmp://<hidden_ip>/live'
traces from gst
flvmux gstflvmux.c:1238:gst_flv_mux_write_header:<flvmux0> downstream did not handle seeking query
............
flvmux gstflvmux.c:1246:gst_flv_mux_write_header:<flvmux0> downstream is not seekable, but streamable=false. Will ignore that and create streamable output instead
........
gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:00.213998674
UBUNTU 14.04
netstat -p outputs both "PID/Program name" in the same column. I just want "Program name" in that column. What's the easiest way to do this?
Actual Output
root#neo4j1:~# netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:ssh *:* LISTEN 1020/sshd
tcp6 0 0 [::]:ssh [::]:* LISTEN 1020/sshd
tcp6 0 0 [::]:7473 [::]:* LISTEN 31380/java
tcp6 0 0 [::]:7474 [::]:* LISTEN 31380/java
Desired Output
root#neo4j1:~# netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State Program name
tcp 0 0 *:ssh *:* LISTEN sshd
tcp6 0 0 [::]:ssh [::]:* LISTEN sshd
tcp6 0 0 [::]:7473 [::]:* LISTEN java
tcp6 0 0 [::]:7474 [::]:* LISTEN java
Try
netstat -tlp | sed 's,[0-9]\+/,,'
Using grouping with Sed:
netstat -tlp | sed 's/\(^.*\)\( [0-9]*\/\)\(.*$\)/\1\3/g'
Though RTLinuxSW's answer is much cleaner.
why it always quit when i run this ruby program ?
This is the ruby program:
require 'rubygems'
require 'eventmachine'
require 'thread'
require 'sinatra'
Thread.new do
EventMachine.run do
EM.add_periodic_timer(1) do
#do some things
end
end
end
get '/' do
'hello'
end
when i run it, i got this:
lynn#ubuntu:~/Test$ ruby em1.rb
== Sinatra/1.4.2 has taken the stage on 4567 for development with backup from Thin
>> Thin web server (v1.5.1 codename Straight Razor)
>> Maximum connections set to 1024
>> Listening on localhost:4567, CTRL+C to stop
lynn#ubuntu:~/Test$
lynn#ubuntu:~/Test$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 28 0 192.168.241.134:38223 91.189.92.11:443 CLOSE_WAIT
tcp 1 0 192.168.241.134:42978 91.189.94.25:80 CLOSE_WAIT
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
lynn#ubuntu:~/Test$
it was quit when i run this ruby program, why? here i wanna a thread to periodicaly do some things with the http request! how can i do it ?
Just run your Sinatra app under thin server and it will be fired with Eventmachine