I'm trying to connect my local standalone MySQL with Cloud Fusion to create and test a data pipeline. I have deployed the driver successfully.
Also, I have configured the pipeline properties with correct values of jdbc string, user name and password but connectivity isn't getting established.
Connection String: jdbc:mysql://localhost:3306/test_database
I have also tried to test the connectivity via data wrangling option but that is also not getting succeeded.
Do I need to bring both the environments under same network by setting up some VPC and tunneling?
In your example, I see that you specified localhost in your Connection String. localhost is only advertised to other services running local to your machine, and Cloud Data Fusion (running in GCP) will not be able to reach the MySQL instance (running on your machine). Hence you're seeing the connectivity issue.
I highly recommend looking at this answer on SO that will help you setup a quick proof-of-concept.
I think that your question is more related to the way how to connect some on-premise environments to GCP networking system that gathering Google cloud instances or resources throughout VPC connection model.
Admitting the fact that GCP is actually leveraging different approaches for connection methods within a Hybrid cloud concepts, I would encourage you to learn some fundamental principles of Cloud VPN as a essential part of performing secure connection between particular VPN Peer Gateway and Cloud VPN Gateway and further creating a VPN tunnel between parties.
I guess there is even dedicated chapter in GCP documentation about Data Fusion VPC peering implementation that might be helpful in your user case.
Related
The Goal
I need to query a Redshift cluster from my golang application. The cluster is not available for public, so I want to use SSH to access the said cluster via bastion host.
Status Quo
I have an AWS Redshift cluster in a private VPC, with inbound rules to not allow any traffic from the internet, but tcp 22;
There's a bastion (which can connect to the cluster), so fowarding a port and using rsql works perfectly fine from the terminal;
I use ODBC, and the official ODBC driver from AWS;
In golang application, I use the following db implementation of the ODBC https://github.com/alexbrainman/odbc;
I can't use Devart's Redshift ODBC driver;
I can't use JDBC;
MacOS/Unix.
The Problem
The problem is pretty much trivial. When cluster is available for public and accessible from the internet, the alexbramain's library does it job. However, when the cluster is behind the wall, that's when problems kick in.
The code of the library is translated into C (system calls), I can't really debug it. While, with mysql, e.g., it's possible to register your custom dialer, it doesn't seem to be a case with ODBC.
Even when the tunnel is active, providing an ODBC DSN to the local host for some reason doesn't work. The SQLRETURN is always -1 (api/zapi_unix.go).
The Question
Did someone have such experience? How did you resolve a problem of accessing the cluster from the internet via a go app?
Thank you!
I've been trying to get our microservice setup to use the AWS RDS Proxy rather than connecting straight to the cluster for failover reasons. It connects just fine, but after it has been running for a while, there are errors that surface that end up having bad effects on any ongoing calls that happen at the time.
The error messages are:
HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#11bc3390 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
DataSource health check failed
The extra config properties I have setup in our application.properties file (besides username, password, url) are:
spring.datasource.hikari.maximum-pool-size=50
spring.datasource.hikari.minimum-idle=10
spring.datasource.hikari.max-lifetime=600000
spring.datasource.hikari.idle-timeout=180000
I have the RDS Proxy setup with pretty much the default settings for it.
Any ideas how to resolve these errors so that the connections from our microservice don't produce errors and cause any instability?
I don't have a fix. I guess you would need DEBUG logs or a debugger session to get to the core of this particular problem.
But I'm questioning the general architecture. Let's look at the docs:
Amazon RDS Proxy is a fully managed, highly available database proxy for Amazon Relational Database Service (RDS) that makes applications more scalable, more resilient to database failures, and more secure.
Your Amazon RDS Proxy instance maintains a pool of established connections to your RDS database instances, reducing the stress on database compute and memory resources that typically occurs when new connections are established.
https://aws.amazon.com/de/rds/proxy/
So the RDS Proxy is highly available and maintains a connection pool. This makes me wonder, why you also want a connection pool (Hikari) on the microservice itself.
It seems redundant. You have effectively two fail-over mechanisms in your setup.
I could imagine that this is the root of the problems you are describing.
I would suggest just using a simpler single-connection DataSource implementation to connect to the RDS Proxy.
I have one application deployed on Red Hat Linux and Oracle 12c.
As part of the application, there is an implementation of Oracle Database Change Notification. Whenever there is a change in the database, that triggers a notification back to the application server. Based on that, the application makes some decisions. Here everything is working well.
Now we are migrating application to amazon EC2 instance with our Red Hat Linux box converted into an EC2 instance and the Oracle Database migrated to cloud. I don’t know what that means, but I do have connection parameters and I'm able to connect though the JDBC driver from my application deployed in the EC2 instance.
Somehow the database change notification functionality is not working in EC2and nothing can be tracked from the application log.
Oracle Database in cloud in aws, which I consider it has to be RDS.
You need to understand the basic on how the application is deployed in aws cloud.
The EC2(with app server) must be in public subnet with security group allowing http/https traffic.
The RDS is kept in private subnet with security group attached which only allows incoming traffic from EC2 instance ( or public subnet).This is general scenario in most of the cases.
The RDS security group only allows the incoming traffic from ec2. but any traffic from your DB (Outgoing traffic) has to be explicitly allowed from the security group attached to Your RDS.
The same traffic needs to be allowed in your Ec2 instance security group.
I am unaware of your specific feature, whether it is supported in AWS RDS or not, you should take a look at AWS RDS Oracle docs
I launched one RDS instance,s3 and EC2 in AWS and its is triggered properly using lambda. Now I wish to change the change the RDS and EC2 from AWS to local machine. My lambda is triggered from s3.
How do I connect the local database through lambda in AWS?
It appears that your requirement is:
You wish to run an AWS Lambda function
Within the function, you wish to connect to a database running on your own computer (outside of AWS)
Firstly, I would not recommend this strategy. To maintain good performance, you should always have an application as close as possible to the database. This means on the same network, in the same location and not going across remote network connections or the Internet.
However, if you wish to do this, then here's some things you would need to do:
Your database will need to be accessible on the Internet, so that you can connect to it remotely. To test this, try accessing it from an Amazon EC2 instance.
The AWS Lambda function should either be configured without VPC connectivity (which means that it is connected to the Internet) or, if you have configured it for VPC connectivity, it needs to be in a Private Subnet with a NAT Gateway enabling Internet access.
(Optional) For added security, you could lock-down your database to only accept connections from a known IP address. To achieve this, you would need to use the VPC + NAT Gateway so that all traffic is coming from the Elastic IP address assigned to the NAT Gateway.
I agree with John Rotenstein that connecting your local machine to a Lambda running on AWS is probably a bad idea.
If your intention is to develop or test locally, I recommend the serverless framework, and the serverless-offline plugin. It will allow you to simulate Lambda locally, and you can pass database config values through as environment variables.
See: Running AWS Lambda and API Gateway locally: serverless-offline
Does anyone know how to use Openswan to create an IPSec tunnel to a Cisco router on EC2?
I keep reading that people can or they cannot set up the IPSec tunnels on Amazon's cloud. Is it possible or not?
If so, can someone point me to a tutorial where it was successful?
Update
AWS has just dropped the requirement to establish Border Gateway Protocol (BGP) peerings in order to use the built in VPN connectivity to an Amazon Virtual Private Cloud (VPC), see Amazon VPC - Additional VPN Features:
You can now create Hardware VPN connections to your VPC using static
routing. This means that you can establish connectivity using VPN
devices that do not support BGP such as Cisco ASA and Microsoft
Windows Server 2008 R2. You can also use Linux to establish a
Hardware VPN connection to your VPC. In fact, any IPSec VPN
implementation should work. [emphasis mine]
The outlined reason for this change specifically highlights BGP as a previous barrier to adoption of this otherwise very appealing VPN connectivity to a VPC:
First, BGP can be difficult to set up and to manage, [...]. Second, some firewalls and entry-level
routers support IPSec but not BGP. These devices are very popular in
corporate branch offices. As I mentioned above, this change
dramatically increases the number of VPN devices that can be used to
connect to a VPC. [...]
I couldn't agree more - accordingly you should now be able to facilitate Openswan (or the built in Linux IPSec stack) to connect to the respective built in VPC IPSec functionality.
Initial Answer
A related tutorial based on Racoon is available at Amazon VPC with Linux. However, before you dive into that, I highly recommend to read the referring article Connecting to Amazon VPC first, at least section Using Linux as VPN Server:
Let us say you have decided to use some linux for your VPN server. For
an IPSec newbie there is no obvious reason that this is a bad idea.
And because one soon finds a tutorial like
http://openfoo.org/blog/amazon_vpc_with_linux.html it seems possible
to perform that task. Following this tutorial you should be able to
ping the two BGP servers from your VPN server. [...] But after that point you will
start to get in trouble. Maybe you are able to connect to a server in
your VPC. But there is one thing you won’t get working stable:
Connecting from the VPC to some server in your home net
192.168.1.1/24. That is because linux has a policy-based IPSec implementation. [...]
Consequently the author concludes:
And the last and most important reason is that Openswan is not
intended to be used that way. Abusing software in a security relevant
place doesn’t seem to be a good idea. [emphasis mine]
YMMV as usual, but you have been warned ;)