Running oracle queries through azure databricks - oracle

Hi I created a notebook in azure databricks to extract a table from an oracle database. I get the error DPI-1047 cannot locate a 64 bit oracle client library. I guess I have to have an oracle client library running on my cluster. I tried to research the issue and found many posts where people said I needed an init script for the oracle client. I couldnt find how to make that init script however. I also found on this website
https://medium.com/#srijansahay/connecting-sql-server-oracle-mysql-and-postgresql-from-azure-services-using-python-789e93d879b4
the following code:
%sh
mkdir -p /opt/oracle
cd /opt/oracle
wget https://download.oracle.com/otn_software/linux/instantclient/193000/instantclient-basic-linux.x64-19.3.0.0.0dbru.zip
unzip instantclient-basic-linux.x64–19.3.0.0.0dbru.zip
sudo sh -c “echo /opt/oracle/instantclient_19_3 > /etc/ld.so.conf.d/oracle-instantclient.conf”
sudo ldconfig
export LD_LIBRARY_PATH=/opt/oracle/instantclient_19_3:$LD_LIBRARY_PATH
This code is however for Linux. How do I modify it for windows? Here is what I have so far:
%sh
mkdir -p /opt/oracle
cd /opt/oracle
wget https://download.oracle.com/otn_software/nt/instantclient/19800/instantclient-basic-windows.x64-19.8.0.0.0dbru.zip
unzip instantclient-basic-windows.x64-19.8.0.0.0dbru.zip
sudo sh -c “echo /opt/oracle/instantclient_19_8 > /etc/ld.so.conf.d/oracle-instantclient.conf”
sudo ldconfig
export LD_LIBRARY_PATH=/opt/oracle/instantclient_19_8:$LD_LIBRARY_PATH
I think everything past the unzip is incorrect. Thanks for the help in advance I'm very new to databricks.

Since there are no library path and path Separation in Windows we need to use PATH for both instead. At the end of the script you place the python stuff.
Rem ic.cmd
C:
Set ORABAS = "\opt\oracle"
cd %ORABAS%
wget https://download.oracle.com/otn_software/nt/instantclient/19600/instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
unzip instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
set ORACLE_HOME=%ORABAS%\instantclient_19_3
set TNS_ADMIN=%ORACLE_HOME%
set PATH=%ORACLE_HOME%;%PATH%

Related

Trying to set GOPATH and GOROOT in AWS EC2 user data, but it is not working

I am trying to set up GOPATH GOROOT in my AWS EC2 Ubuntu 20.04 user data, but it never worked, every time I connect to the AWS EC2 and view the log in /var/log/cloud-init-output.log it always says
go: not found, but if I key in the echo part it will work.
I am trying to set up multiple EC2 with this basis, so I can't key in every instance myself.
The CloudFormation yaml user data part is below:
UserData:
Fn::Base64: |
#!/bin/bash
wget https://dl.google.com/go/go1.14.4.linux-amd64.tar.gz
tar -C /usr/local -zxvf go1.14.4.linux-amd64.tar.gz
mkdir -p ~/go/{bin,pkg,src}
echo 'export GOPATH=$HOME/go' >> ~/.bashrc
echo 'export GOROOT=/usr/local/go' >> ~/.bashrc
echo 'export PATH=$PATH:$GOPATH/bin:$GOROOT/bin' >> ~/.bashrc
echo 'export GO111MODULE=auto' >> ~/.bashrc
source ~/.bashrc
apt -y update
apt -y install mongodb wget git
systemctl start mongodb
apt -y install git gcc cmake autoconf libtool pkg-config libmnl-dev libyaml-dev
go get -u github.com/sirupsen/logrus
cd ~
git clone --recursive https://github.com/williamlin0504/free5gcWithOCF.git
cd free5gcWithOCF
make
And here is the error inside /var/log/cloud-init-output.log
Error while user data runs
Is there anyone is familiar with this, please I need some help~
In your error message, in the Makefile at line 30 there is a program bin/amf being used
This program appears to be a shell script with a problem in line 1
The nature of the problem is "go: not found"
If you have the bare word "go" in line 1 of the shell script and the path cannot find it then this is what will happen
Probably you need to alter the last line of your userdata shell script to say
PATH=/usr/local/go/bin:$PATH make
I know you have a source command earlier in the script that is supposed to set this up but it doesn't do what you think it does

WGET seems not to work with user data on AWS EC2 launch

I launch an centos AMI I created, and try to add user data as a file which looks like this:
#!/bin/bash
mkdir /home/centos/testing
cd testing
wget https://validlink
So simply, on launch, the user data creates a folder called testing and downloads this validURL which I will not put as it links to my data - however it is valid and accessible.
When I launch the instance, the folder testing is created successfully, however there is no file inside the directory.
When I ssh into the instance, and run the wget command as a sudo, the file is downloaded successfully inside the testing folder.
Why does the file not get downloaded on the ec2 launch through user data?
You have no way of knowing the current working directory when you execute the cd command. So specify full path:
cd /home/centos/testing
Try this:
#!/bin/bash
mkdir /home/centos/testing
cd /home/centos/testing
wget https://validlink
Run it using the root user.
Try this instead:
#!/bin/bash
sudo su
yum -y install wget
mkdir /home/centos/testing
cd /home/centos/testing
wget https://validlink

Bash on Windows - alias for exe files

I am using the Bash on Ubuntu on Windows, the way to run bash on Windows 10. I have the Creators update installed and the Ubuntu version is 16.04.
I was playing recently with things as npm, node.js and Docker and for docker I found it is possible to install it and run it in windows and just use the client part from bash, calling directly the docker.exe file from Windows's Program Files files folder. I just update my path variable to include the path to docker as PATH=$PATH:~/mnt/e/Program\ Files/Docker/ (put in .bashrc) and then I am able to run docker from bash calling docker.exe.
But hey this bash and I dont want to write .exe at the end of the commands (programs). I can simply add an alias alias docker="docker.exe", but then I want to use lets say docker-compose and I have to add another one. I was thinking about adding a script to .bashrc that would go over path variable and search for .exe files in every path specified in the path variable and add an alias for every occurance, but it does not seem to be a very clean solution (but I guess it would serve its purpose quite well).
Is there a simple and clean solution to achieve this?
I've faced the same problem when trying to use Docker for Windows from WSL.
Had plenty of existing shell scripts that run fine under Linux and mostly under WSL too until failing due to docker: command not found. Changing everywhere docker to docker.exe would be too cumbersome and non-portable.
Tried workaround with aliases in ~/.bashrc as here at first:
shopt -s expand_aliases
alias docker=docker.exe
alias docker-compose=docker-compose.exe
But it requires every script to be run in interactive mode and still doesn't work within backticks without script modification.
Then tried exported bash functions in ~/.bashrc:
docker() { docker.exe "$#"; }
export -f docker
docker-compose() { docker-compose.exe "$#"; }
export -f docker-compose
This works. But it's still too tedious to add every needed exe.
Finally ended up with easier symlinks approach and a modified wslshim custom helper script.
Just add once to ~/.local/bin/wslshim:
#!/bin/bash -x
cd ~/.local/bin && ln -s "`which $1.exe`" "$1" || ln -s "`which $1.ps1`" "$1" || ln -s "`which $1.cmd`" "$1" || ln -s "`which $1.bat`" "$1"
Make it executable: chmod +x ~/.local/bin/wslshim
Then adding any "alias" becomes as easy as typing two words:
$ wslshim docker
+ cd ~/.local/bin
++ which docker.exe
+ ln -s '/mnt/c/Program Files/Docker/Docker/resources/bin/docker.exe' docker
$ wslshim winrm
+ cd ~/.local/bin
++ which winrm.exe
+ ln -s '' winrm
ln: failed to create symbolic link 'winrm' -> '': No such file or directory
++ which winrm.ps1
+ ln -s '' winrm
ln: failed to create symbolic link 'winrm' -> '': No such file or directory
++ which winrm.cmd
+ ln -s /mnt/c/Windows/System32/winrm.cmd winrm
The script auto picks up an absolute path to any windows executable in $PATH and symlinks it without extension into ~/.local/bin which also resides in $PATH on WSL.
This approach can be easily extended further to auto link any exe in a given directory if needed. But linking the whole $PATH would be an overkill. )
You should be able to simply set the executable directory to your PATH. Use export to persist.
Command:
export PATH=$PATH:/path/to/directory/executable/is/located/in
In my windows 10 the solution was to install git-bash and docker for windows.
in this bash, when I press "docker" it works
for example "docker ps"
I didnt need to make an alias or change the path.
you can download git-bash from https://git-scm.com/download/win
then from Start button, search "git bash".
Hope this solution good for you

aws ec2 php exec sudo not working

We are trying to automate website deployment on aws ec2 ubuntu instance using php.
So to deploy website on aws ec2 ubuntu instance we have to follow steps
Make folder in www folder
Make conf file site-available
Execute a2ensite
restart apache
Create database
Create user and grant all access
I am trying to do all things using PHP. So I stuck at 1st steps for following code
echo shell_exec("sudo mkdir /var/www/{$_POST['environment']} 2>&1");
echo shell_exec("sudo mkdir /var/www/{$_POST['environment']}/{$_POST['site']}/public_html 2>&1");
echo shell_exec("sudo mkdir /var/www/{$_POST['environment']}/{$_POST['site']}/ssls 2>&1");
echo shell_exec("sudo mkdir /var/www/{$_POST['environment']}/{$_POST['site']}/conf 2>&1");
echo shell_exec("sudo mkdir /var/www/{$_POST['environment']}/{$_POST['site']}/sqls 2>&1");
but got following error
sudo: no tty present and no askpass program specified
I searched for some alternatives (In my mind)
Set password to sudo (Didnt get anything on google)
Connect ec2 using PHP library (Library is available but dont know how to execute commands)
Suggest anything.
That error indicates the script needs to ask the user for a password to sudo and it can't. You can allow the user running the php commands to sudo without a password. See this related answer or search for how to allow sudo without a password and you will find lots of guides.

PostgreSQL: command not found

I installed PostgreSQL via MacPorts. However, going to /opt/local/lib/postgresql84/bin I am unable to execute any of the pg commands. Does anyone know what I am doing wrong?
When you say "going to", it sounds like you're using the cd command to change to that directory. Something like:
$ cd /opt/local/lib/postgresql84/bin
$ psql
psql: command not found
Normally on Unix systems, the current directory is not part of your executable search path. So either explicitly execute psql from the current directory using ./:
$ cd /opt/local/lib/postgresql84/bin
$ ./psql
or, add the directory to your PATH:
$ export PATH=/opt/local/lib/postgresql84/bin:$PATH
$ psql
I had installed postgres (Mac,The capitan SO) with the dmg and also I got the same issue. I had to install psql through brew. Try with this :
brew install postgres
This worked for me.
Try this command:
find / -name psql
Try
whereis psql

Resources