I have to make some benchmarks for a spark-scala application.
I'm working on a cluster with 9 nodes. I access to my account with ssh -i "mykey" "user#host.com" (through Ubuntu on VM or bitvise ssh client for windows).
Now I have to test my application runtime performances but i don't know broswer the "master:8088/cluster" link from terminal or VM. I need to reach YARN web UI and HDFS. I read a lot of articles about ssh tunneling and port forwarding?
Related
Having created a Windows Azure VM and opened ports 3389 and 22 for inbound RDP and SSH connections, respectively.
I can successfully connect to the vm via RDP from a remote Windows PC.
Testing SSH connection in the Portal succeeds. However trying to connect from a remote Linux VM using SSH fails.
Given that SSH connection test within the portal succeeds, it suggests that (1) it is possible to SSH into a windows VM; and (2) there is no other config require on the server ie installing OpenSSH (or similar) / Copying over key file(s) to some location etc. However, the help steps in the Azure Portal for my Windows VM, for making remote SSH connections suggest that maybe a public key needs to exist on the server and that I need the private key on the LinuxVM I am trying to connect from.
Please could someone help me understand if ssh into windows Azure VM is possible and if so, the requirements / minimum set of steps (on the target Windows VM and the source Linux VM) I need to get to a state that I can successfully SSH.
Other posts re similar question posted have not helped me connect via ssh. I have not found a 'golden source of truth' on Microsoft docs. Maybe I missed it.
Thank you.
A Windows Server doesn’t typically come pre-built and ready to go with SSH access and it requires some setup. You can follow this to set up your Azure VM for SSH access. You can configure SSH on a Windows Azure VM for access, check out How to Set Up OpenSSH on a Windows Server. After deploying the OpenSSH, you can follow the steps about connect via SSH with client in the Azure portal on your Linux client to access that Windows VM via SSH.
I would like to connect to a remote Docker Swarm (Ubuntu) from a Windows box.
In Linux it seams that you need to update the daemon.json file.
How do you achieve this in Windows?
Thanks!
The Docker engine has two parts, the daemon service (dockerd) that's running on your Ubuntu box, we'll call it the "server". Then the docker cli is what you can run from that server (docker) or from anything like your Windows machine (docker.exe). We'll call this the "client".
The client can talk to the server over two main ways, the socket, and a TCP port. The socket is usually reserved for local connections (SSH into the server and the docker client defaults to using the socket file to talk to the local server) or SSH tunnels, which are not something that works out of the box on Windows (maybe if you try the Windows Subsystem for Linux on Windows 10).
The other connection option is TCP, which isn't enabled on the server out of the box for security reasons. It has no authentication when enabled, so you'll want to use TLS to authenticate remotely, so Docker has steps for that. It's not a 3 min solution, so many look for an easier route to solve this problem.
The easier option for enabling TLS and the TCP port on the server is to use Docker Cloud with the "Bring Your Own Swarm" feature, which manages the certificates and security for you.
I have a small cluster with docker nodes, I access it via a gateway server that I ssh into. What I would like to do, is to run e.g. Eclipse with a GUI on the cluster and access that GUI on my computer.
What I have found so far is this: http://fabiorehm.com/blog/2014/09/11/running-gui-apps-with-docker/
However, the problem I'm experiencing is that the host computer doesn't run any x-server, since it's only a node in a cluster, so I cannot mount the required directory into the container.
Is there a way to use GUI applications in a container with this setup?
i am using the following setup for hadoop's nodes web ui access :
dfs.namenode.http-address : 127.0.0.1:50070
By which i am able to access the nodes web ui link only form the local machine as :
http://127.0.0.1:50070
Is there any way by which i can make it accessible from outside as well ? say like :
http://<Machine-IP>:50070
Thanks in Advance !!
You can use hostname or ipaddress instead of localhost/127.0.0.1.
Make sure you can ping the hostname or ip from the remote machine. If you can ping it then you can able to access web ui.
To ping it
Open cmd/terminal
type the below command in remote machines
ping hostname/ip
From http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
The following table lists web interfaces that you can view on the core
and task nodes. These Hadoop interfaces are available on all clusters.
To access the following interfaces, replace slave-public-dns-name in
the URI with the public DNS name of the node. For more information
about retrieving the public DNS name of a core or task node instance,
see Connecting to Your Linux/Unix Instances Using SSH in the Amazon
EC2 User Guide for Linux Instances. In addition to retrieving the
public DNS name of the core or task node, you must also edit the
ElasticMapReduce-slave security group to allow SSH access over TCP
port 22. For more information about modifying security group rules,
see Adding Rules to a Security Group in the Amazon EC2 User Guide for
Linux Instances.
YARN ResourceManager
YARN NodeManager
Hadoop HDFS NameNode
Hadoop HDFS DataNode
Spark HistoryServer
Because there are several application-specific interfaces available on
the master node that are not available on the core and task nodes, the
instructions in this document are specific to the Amazon EMR master
node. Accessing the web interfaces on the core and task nodes can be
done in the same manner as you would access the web interfaces on the
master node.
There are several ways you can access the web interfaces on the master
node. The easiest and quickest method is to use SSH to connect to the
master node and use the text-based browser, Lynx, to view the web
sites in your SSH client. However, Lynx is a text-based browser with a
limited user interface that cannot display graphics. The following
example shows how to open the Hadoop ResourceManager interface using
Lynx (Lynx URLs are also provided when you log into the master node
using SSH).
Copy lynx http://ip-###-##-##-###.us-west-2.compute.internal:8088/
There are two remaining options for accessing web interfaces on the
master node that provide full browser functionality. Choose one of the
following:
Option 1 (recommended for more technical users): Use an SSH client to connect to the master node, configure SSH tunneling with local port
forwarding, and use an Internet browser to open web interfaces hosted
on the master node. This method allows you to configure web interface
access without using a SOCKS proxy.
to do this use the command
$ ssh -gnNT -L 9002:localhost:8088 user#example.com
where user#example.com is your username. Note the use of -g to open access to external ip addresses (beware this is a security risk)
you can check this is running using
nmap localhost
to close this ssh tunnel when done use
ps aux | grep 9002
to find the pid of your running ssh process and kill it.
Option 2 (recommended for new users): Use an SSH client to connect to the master node, configure SSH tunneling with dynamic port
forwarding, and configure your Internet browser to use an add-on such
as FoxyProxy or SwitchySharp to manage your SOCKS proxy settings. This
method allows you to automatically filter URLs based on text patterns
and to limit the proxy settings to domains that match the form of the
master node's DNS name. The browser add-on automatically handles
turning the proxy on and off when you switch between viewing websites
hosted on the master node, and those on the Internet. For more
information about how to configure FoxyProxy for Firefox and Google
Chrome, see Option 2, Part 2: Configure Proxy Settings to View
Websites Hosted on the Master Node.
This seems like insanity to me but I have been unable to find how to configure access in core-site.xml to override the web interface for the ResourceManager which by default it is available at localhost:8088/ and if Amazon think this is the way then I tend to go along with it
I recently installed the CDH distribution of Cloudera to create a 2 node cluster. From the Cloudera Manager UI, all services are running fine.
All the command line tools (hive etc ) are also working fine and I am able to read and write data to hdfs.
However the namenode (and datanode) web UI alone is not opening. Checking on netstat -a | grep LISTEN, the processes are listening on the assigned ports and there are no firewall rules which are blocking the connections ( I already disabled iptables)
I initially though that it could be a DNS issue but even the IP address is not working. The Cloudera Manager installed on the same machine on another port is opening fine.
Any pointers on how to debug this problem?
I had faced the same issue.
First it was because NAMENODE in safemode
then after because of two IP address (I have two NIC configured on CDH Cluster one internal connectivity of the servers (10.0.0.1) and other is to connect servers form Internet (192.168.0.1))
When i try to open NAMENODE GUI form any of the server connected to cluster on network 10.0.0.1 then GUI is opening and works fine but from other any other machine connected to servers by 192.168.0.1 network it fails.