Please advise on how to migrate my local mysql server to the cloud.
Currently I have a Fedora linux box and a NAS attached to it via ethernet.
I believe the best way to go about it is :
Take a mysqldump of all databases
Create an amazon RDS instance and try to load from the created mysqldump
Shift local connection to this instance
Am I on the right track ?
How should I go about doing (1). I have a username and password based access to the mysql server and it has only 1 database. I tried to follow a few links on the net but the commands did not seem to work.
Is (2) even possible ?
The end goal is to connect from local servers to DB server on AWS and be able to query seamlessly.
I've done similar migrations and I think you're on the right track.
"How should I go about doing (1)?"
Just take a mysqldump of your DB and store it in a file, e.g.:
mysqldump -h [host] -u [user] -p[password] [dbname] > dumpfilename.sql
"Is (2) even possible ?"
Absolutely. You can connect to a MySQL RDS instance just like you would connect to any other MySQL instance. The host name is refered to as "endpoint" in the AWS Management Console.
One you've created the RDS instance and setup the security group, you're ready to load the dump:
mysql -h [endpoint] -u [user] -p[password] [dbname] < dumpfilename.sql
Related
I have a postgres RDS instance which my Node.js web application running on an EC2 instance is not able to connect to. The error in my EC2 node logs is: error: password authentication failed for user "ubuntu"
I can confirm that I have the right username, password, database name, etc because it is working correctly on the development build on my machine. I copied all the .env parameters exactly into my ec2 machine for the production build. When attempting to connect to RDS on my production application web page, it fails. I have restarted my Node.js server multiple times and have rebooted the whole ec2 machine. I have confirmed that the env variables are there with printenv.
What would you recommend trying to fix this issue?
EDIT for more details: My nodejs setup should be correct because my nodejs server will call some external APIs that do not require my postgres database and those calls work properly.
EDIT2: This is strange because my username for RDS is postgres, while my username for EC2 is ubuntu. I wonder if somehow there's some clash between env variables. I checked printenv but didn't find any though
EDIT3: See comments for my workaround.
I would suggest to test the database credentials by directly connecting to RDS database using psql client on EC2 instance.
When I try to execute this command:
cd ~/.chainlink-kovan && docker run -p 6688:6688 -v ~/.chainlink-kovan:/chainlink -it --env-file=.env smartcontract/chainlink: local n
(I entered ut with my version of course)
I get this error:
The node and the database are both hosted on AWS.
This is my environment:
the issue is related to the configuration of your postgresql server.
To connect to the database you need a specially created USER with a PASSWORD, which then locks this database by starting the Chainlink node. The default postgres USER and DATABASE will not work because it is used for administrative purposes. These credentials are then added to the environmental variable where you have the correct syntax:
DATABASE_URL=postgresql://$USERNAME:$PASSWORD#$SERVER:5432/$DATABASE
You can follow those steps to create the USER with credentials:
access the postgresql server/ host via psql command line interface:
psql --host=mypostgresql.c6c8mwvfdgv0.us-west-2.rds.amazonaws.com --port=5432
Create the USER and grant all privileges:
CREATE USER youruser WITH PASSWORD 'yourpass';
GRANT ALL PRIVILEGES ON DATABASE yourdbname TO youruser;
Now you just need to change the DATABASE_URL configuration in your environmenal file (.env) and kill & restart the Chainlink node
In addition and in order to access the postgresql server hosted on AWS, you can have a look at the official documentation: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
I have a Hortonworks Hadoop cluster where the data nodes are on a separate network off of the master/head node. The only way to access the data nodes is through the master node or an edge node. From the edge node, I execute the hive command to connect into my hive database.
I cannot connect to the hive database from my desktop with DBeaver (4.3.0, 64-bit Windows) or the hive command line interface. Through DBeaver, I tried creating an SSH tunnel to my edge node and continually receive "Could not open client transport with JDBC Uri. jdbc:hive2://127.0.0.1:[port#]/[database].
Configuration for Hive/Apache Hive driver:
General Tab:
Host: dataNodeName
Port: 10000
Database/Schema: databaseName
User name: myUID
SSH Tunnel Tab (Network page):
Checked Use SSH Tunnel
Host/IP: edgeNodeServerName
Port: 22
User Name: myUID
Authentication Method: Password
Password: myPWD
Advanced
Local port: 0
Keep-Alive interval (ms): 0
When I select "Test Connection" with local port set to "0", I receive the above error message with random port numbers. If I set the local port to "10000", I receive the above error with port number "10000".
It looks like DBeaver is ignoring the generic JDBC connection settings--the host name in the created JDBC string is 127.0.0.1 instead of the data node name.
What am I missing? How do I setup DBeaver to access a Hive database located on a "hidden" network?
Is your hostname configured with the IP address mentioned in the jdbc connect syntax (127.0.0.1)?
Are you able to connect to beeline from your Unix shell?
Syntax to connect to beeline(hiveserver2):
beeline -u jdbc:hive2://<hostname>:<hive listener port>/<database> -n username> -p <password>
If you're able to connect to beeline, you should be able to connect to hive using same port number and host from DBeaver.
Hive listener port by default is configured on 10000, but there's a possibility that your admin can change the port number. Check the port number in hive-site.xml, or get it from admin.
Could you please uncheck the SSH tunnel and try?
This link has all the setup from scratch, please check if you have missed any step.
https://www.linkedin.com/pulse/query-hive-hiveserver2-from-windows-using-universal-database-nimmala
Not sure if your environment is Kerberized or not but assuming it is -
Following is what worked for me while connecting to Cloudera -
Fetch the krb5.conf or krb5.ini from your admins and place it in some directory. I normally put the file in a location where I put my keytabs.
Create jaas.conf file and place it at the same location(or the location of your choice)
jaas.conf must look like below(copy paste) -
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
doNotPrompt=true
useKeyTab=true
keyTab="C:\Users{user}\krb5cc_{user}"
useTicketCache=true
renewTGT=true
principal="{user}#DOMAIN.ORG" ;
};
Edit your dbeaver.ini file and provide the reference to both of this files(append the following lines to existing dbeaver.ini). Make sure you backup dbeaver.ini, with re installations or replacing with newer version, dbeaver.ini may get replaced, in that case you can copy the lines below from your backup dbeaver.ini file -
-Djavax.security.auth.useSubjectCredsOnly=false
-Djava.security.krb5.debug=true
-Dsun.security.krb5.debug=true
-Djava.security.krb5.conf=C:\Users{User}\Documents\Keytabs\krb5.conf
-Djava.security.auth.login.config=C:\Users{User}\Documents\Keytabs\jaas.conf
Last Step(You may need or may not)
I init my keytab before connecting. So I use Shell Commands -
Press F4 after creating the connection
Make sure in user you just put the user name for which you are initializing the keytab and nothing else. It should not be {user}#domain.org.
Use the shell commands to init the keytab
I also was having trouble configuring DBeaver to Hive, my solution was to use Cloudera's ODBC Driver. It worked a lot better then the JDBC drivers (auto-complete working, quicker, no need to run kinit), and I could automatize its creation.
The only problem is that you must be admin to install it.
When I connect to my database remotely I use ssh to connect to the remote machine, then I run sudo -u postgres psql to access PostgresSQL. The postgres user is passwordless in my OS.
I can make an SSH tunnel connect in Datagrip, but I can't seem to find a way to switch to postgres user prior to attempting to access the database.
Is there a way to do this?
First, you need to configure SSH tunnel on datasource ssh/ssl tab (host/port/username/password).
Secondly, you need to specify database credentials to your db on general tab.
Also, make sure you configured server correctly for non-local connections.
You should go to ~/.ssh/config file and set the tunnel with the user, which is used on the server, and put 'postgres' as a user name in the connection properties.
Note, it is working only in 2017.3 EAP now (release will be available this week)
My application uses the blazer gem for visualizing DB queries.
During the setup I've encountered the following error:
FATAL: no pg_hba.conf entry for host "111.22.33.44", user "blazer", database "my_db", SSL off
My application is hosted on EngineYard and uses PostgreSQL.
How can I find and modify the pg_hba.conf on EngineYard?
upd
I do have SSH access to EngineYard cloud.
Instance: General Purpose (M3) Large.
OS: EngineYard's Gentoo.
You can try the following steps. I've assumed that your DB name is my_db.
Connect to the instance via SSH (the link can be found on the EngineYard environment page)
Connect to the
database as superuser psql -U postgres -h localhost -d
my_db. If you don't have the password, check your database secrets here /data/my_db/current/config/database.yml
After connecting to DB identify location of hba file by typing SHOW
hba_file;
Quit psql by typing \q
Use previously identified path to open the hba_file file and add the missing user. E.g via vim sudo vim /db/postgresql/9.5/data/pg_hba.conf. Note the sudo command
The use should be added under # IPv4 postgres
user for 10.x with md5:
Connect to the database again
Reload the configuration via select pg_reload_conf(); command
After all steps are performed, Blazer queries should be accessible.