I am looking for build servers, continuous integration of automation servers that can have clients (workers) that are behind NAT and that require non-root access on the clients.
This means that the worked is the one connecting to the server and asking for new jobs and not the opposite.
I am aware that it should be possible to setup VPNs or reverse ssh solutions but this is outside the scope of this question. I'm just looking for simple setups.
Here is what Jenkins documentation says about accessing slaves behind a firewall. I never tried running slaves behind a firewall myself, though.
If you decide to go with JNLP you'll need to think about how to start (or, more importantly, restart) your slaves when they go down (in my experience a slave connection may break for no apparent reason every now and then). For example, we connect via VNC to a slave machine and restart the slave via a shortcut (that runs the slave headlessly).
Related
I am trying to set up my JMeter master/slave set up on AWS EC2 instances using windows. Out of the box integration didn't work (connection refused errors) and after some investigation I got to know that RMI communication only works if the machines are on same subnet (is this true?). I found this great article and I tried to follow it as is on windows (tried running ssh port forward tunneling via cygwin) but not have any luck. So I was hoping if someone already done this with Windows and can share their experience.
Out of the box integration didn't work (connection refused errors) and
after some investigation I got to know that RMI communication only
works if the machines are on same subnet (is this true?)
It is not necessary. Even if the machines are in a different subnet and addressable via the network, RMI should work. Here make sure the Security Groups and NACLs are properly set so that both EC2 instances can communicate with each other via the network.
You can check whether you can ping(For this you need to enable ICMP in your Security Groups and also make sure other needed ports are opened for RMI) from one EC2 instance to another. If the servers can communicate with each other, then you need to troubleshoot internal firewall configuration in windows EC2 instances.
We have used https://testable.io/ AMI's from the marketplace. It takes care of all configuration and just needs to worry only about testing and the results.
https://aws.amazon.com/marketplace/pp/B073JNTVKP
All our endpoints are secured to certain IP-ranges from testable.io. If you have internal, still you need to setup those security settings.
Hope it helps.
I have a client who is remote. I need to debug some weird problem that none of my other clients are having. Before I try and set up a conference with this client, I would like to know if there is some way of remotely debugging our application.
I see that there are remote debugging tools available for Visual Studio, but from what I've read, I need to be on the same subnet. As the person is remote, this is not a possibility. Also, as I'd like to keep our connection secure, I would need to connect up some sort of encrypted tunnel (this is where I'm a little fuzzy as my networking skills are mostly theoretical).
As I understand it, an encrypted tunnel is a bridge to another (different) subnet. This is to ensure that those computers on the other side won't interfere with the local subnet computers.
So, because the client's computer is on a different subnet, I think that this is not possible. Or is it? Should there not be a way of making the client's computer show up as a virtual computer on my subnet, by forwarding packets from one subnet to another? I would think that this is theoretically possible, but I'm not exactly sure how I would go about this.
Also, at the moment, my current way that we connect to clients is through GoToMeeting, but I don't think that it supports tunneling. If not, then I may need some way of generating a tunnel, so I was also thinking of maybe using some SSH programme like PuTTY.
As I have said before, my knowledge of networking is quite theoretical, so if the tools that I am suggesting are not the correct ones, please correct me. (I'm a programmer, damm it! Not a network engineer!)
Both computers are Windows boxes. Windows 10 (client) and Windows 8.1 (development).
If you can connect to an ssh server in the remote network, you can (subject to configuration on the server) create a tunnel such that you connect to a socket on your local pic and the connection appears from the server to an endpoint on the remote network.
You'll want to investigate the -L command of OpenSSH, which combined with the PuTTY docs, should help explain what's required.
By default, the endpoint would be a port on the ssh server, but it could be a port on a different host that the remote server can connect to.
I'm not familiar with the current state of Windows SSH servers, but even if there isn't a system server to hand, you should be able to have on run 'on demand' - if you run it on a non-privileged port and by the user you want to connect in as, it shouldn't even need Admin privileges.
I'm not familiar with GoToMeeting, but the one thing with SSH tunnelling it that IT depts should be familiar with SSH. If trying that, focus on getting a working connection in, then setting up the tunnel, then connecting through it as separate steps.
Once you have an SSH connection, then it doesn't need to do something itself, and you can then investigate connecting while specifying the port forwarding, but will will need to get the basic connection working correctly first.
I spun up a Mesosphere cluster on Digital Ocean (development) and it's not allowing me to allow external (non vpn) connections to containers or apps. How can this be solved ?
To ensure that the world doesn't have access to your cluster normally, there have been iptables rules installed. By default, these allow full access inside the cluster and nothing externally.
If you're interested in running real applications, I'd recommend the following:
Put HAProxy on a single node.
Setup the haproxy-marathon-bridge script.
On the same box that you installed HAProxy on, setup iptables to allow access to the port that HAProxy is listening on.
By doing this, you'll have a single place to refer to when giving access to applications running on your Mesos cluster. No matter where the app or container is scheduled (with marathon), you'll always be able to reach it via. haproxy.
I am new to Spark and I am trying to run it on EC2. I follow the tutorial on spark webpage by using spark-ec2 to launch a Spark cluster. Then, I try to use spark-submit to submit the application to the cluster. The command looks like this:
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://ec2-54-88-9-74.compute-1.amazonaws.com:7077 --executor-memory 2G --total-executor-cores 1 ./examples/target/scala-2.10/spark-examples_2.10-1.0.0.jar 100
However, I got the following error:
ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
Please let me know how to fix it. Thanks.
You're seeing this issue because the master node of your spark-standalone cluster cant open a TCP connection back to the drive (on your machine). The default mode of spark-submit is client which runs the driver on the machine that submitted it.
A new cluster mode was added to spark-deploy that submits the job to the master where it is then run on a client, removing the need for a direct connection. Unfortunately this mode is not supported in standalone mode.
You can vote for the JIRA issue here: https://issues.apache.org/jira/browse/SPARK-2260
Tunneling your connection via SSH is possible but latency would be a big issue since the driver would be running locally on your machine.
I'm curious if you still having this issue ... But in case anyone is asking here is a brief answer. As clarified by jhappoldt, the master node of your spark-standalone cluster cant open a TCP connection back to the drive (on your local machine). Two workarounds are possible, tested and succeeded.
(1) From EC2 Management Console, create a new security group and add rules to enable TCP back and forth from your PC (public IP). (what I did was adding TCP rules inbound and outbound) ... Then add this security group to your master instance. (right click --> Networking --> Change security groups). Note: add it and don't remove the already established security groups.
This solution work well, but in your specific scenario, deploying your application from local machine to EC2 cluster, you will face further problems (resource related) so the next option is the best one
(2) Having your .jar file (or .egg) copy it to the master node using scp. You can check this link http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html for information about how to do that; and deploy your application from the master node. Note: spark is already pre-insalled so you will do nothing but write the same exact command you write on your local machine from ~/spark/bin. This shall work perfect.
Are you executing the command on your local machine, or on the created EC2 node? If you're doing it locally, make sure port 7077 is open in the security settings, as its closed to the outside by default.
I'm in need of setting up cron-based VPN connectivity from Ubuntu "Jaunty" to a Windows-based VPN over PPTP for incremental DB synchronization. Using the default Network Manager and PPTP module everything seems to work fine. However, I have 2 issues:
1) Despite checking "Use this only for resources", I still lose my local internet connection and am routed through their servers.
2) As a result of 1 I have need to automate connecting to and disconnecting from the VPN in order to perform various tasks as the machine is at the office and I am not.
Much of the information I'm finding on PPTPing from Linux to Windows involves the use of GUI Network Manager, etc. However, I need to perform these steps from the command line for the sake of automation. Can anyone point me to a quality bit of documentation for this specific case?
Best.
http://pptpclient.sourceforge.net/
I think this site has everything that you need :)