How do I install RODBC on Mac with unixodbc and freetds? - macos

After a fairly extensive search, I noticed that a number of people are having a hard time finding a start-to-finish guide that answers this question. (At least one question notes that a solution exists, but the proposed solution does not get around the fact that, by default, RODBC attempts to compile against iODBC, which is not included with Yosemite.) I just went through this process, so I thought I would document it here in the hope that it will benefit others. I am connecting to a SQL Server database.

Using Homebrew as my OS X package manager, I can install RODBC with the following steps (assuming I have already installed R).
Install unixodbc:
$ brew install unixodbc
Install freetds (replacing /usr/local/Cellar/unixodbc/2.3.2_1 with your unixodbc directory, if necessary):
$ brew install --with-tdsver=8.0 --with-msdblib --with-unixodbc=/usr/local/Cellar/unixodbc/2.3.2_1 freetds
Configure your freetds installation (the following is a minimal configuration file):
freetds.conf
# server specific section
[global]
; tds version = 8.0
; dump file = /tmp/freetds.log
; debug flags = 0xffff
; timeout = 10
; connect timeout = 10
text size = 64512
[TESTSQL]
# insert the actual host below
host = <xxx.xx.x.xx>
port = 1433
tds version = 8.0
Test the freetds config:
$ tsql -H `<xxx.xx.x.xx>` -p 1433 -U `<username>` -P `<password>`
locale is "en_US.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> exit
Configure your unixodbc installation (the following is a minimal configuration file):
$ sudo vim /usr/local/Cellar/unixodbc/2.3.2_1/etc/odbcinst.ini
odbcinst.ini
[MSSQL]
Description = Microsoft SQL Server driver
Driver = /usr/local/Cellar/freetds/0.95.18/lib/libtdsodbc.so
(and another minimal installation file):
$ sudo vim /usr/local/Cellar/unixodbc/2.3.2_1/etc/odbc.ini
odbc.ini
[ODBC Data Sources]
TESTSQL = Test database
[TESTSQL]
Driver = MSSQL
Servername = TESTSQL
Port = 1433
Database = TMSEPRD
TDS_Version = 8.0
Test the new configuration with isql:
$ isql TESTSQL `<username>` `<password>`
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL> quit
Create a symbolic link to the files in your home directory:
$ ln -vs /usr/local/Cellar/freetds/0.95.18/etc/freetds.conf ~/.freetds.conf
$ ln -vs /usr/local/Cellar/unixodbc/2.3.2_1/etc/odbc.ini ~/.odbc.ini
$ ln -vs /usr/local/Cellar/unixodbc/2.3.2_1/etc/odbcinst.ini ~/.odbcinst.ini
Find and modify your RProfile file by appending the following line(s) of code to the file (replacing /usr/local/include with the include directory that contains your sql.h and sqlext.h files; the second line may be unnecessary if the directory does not exist):
$ vim /Library/Frameworks/R.framework/Versions/3.2/Resources/library/base/R/Rprofile
Sys.setenv(ODBC_INCLUDE="/usr/local/include")
Sys.setenv(ODBC_LIBS="/usr/local/lib")
Now download the RODBC package source (which you an download here) to your Downloads folder.
Open a new R console session and install the package (replacing RODBC_1.3-12.tar.gz with the name of your package source):
install.packages("~/Downloads/RODBC_1.3-12.tar.gz", repos=NULL, type="source")
The package should now work:
> library(RODBC)
> myconn <- odbcConnect("TESTSQL", uid="<userid>", pwd="<password>")
Thanks to Jared Folkins and Gabi Huiber for help with figuring out what directories R looks in by default for the requisite files for RODBC.

Related

unixODBC fails on mac with "[IM004] [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed"

I am trying to connect to a Microsoft SQL server from an ARM mac running macOS 11.0.1.
I have set up a conda environment with a miniconda install, and from within a Jupyter notebook I run:
import pyodbc
conn = pyodbc.connect('Driver={ODBC Driver 17 for SQL Server};Server=xxx;Database=xxx;uid=xxx;pwd=xxx;')
Error: ('IM004', "[IM004] [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed (0) (SQLDriverConnect)")
Running odbcinst -j shows no obvious issue. I think the issue is in my environment somewhere:
$ odbcinst -j
unixODBC 2.3.9
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /Users/johnmorgan/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
$ more /etc/odbcinst.ini
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/usr/local/lib/libmsodbcsql.13.dylib
UsageCount=1
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/usr/local/lib/libmsodbcsql.17.dylib
UsageCount=1
My $PATH is:
$ tr ':' '\n' <<< "$PATH"
/opt/anaconda3/envs/analysis/bin
/opt/miniconda3/bin
/opt/anaconda3/condabin
/Users/johnmorgan/anaconda3/bin
/Users/johnmorgan/anaconda3/bin
/Users/johnmorgan/anaconda2/bin
/Users/johnmorgan/anaconda/bin
//anaconda/bin
/Users/johnmorgan/anaconda/bin
/opt/local/bin
/opt/local/sbin
/Applications/anaconda/bin
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/opt/X11/bin
/Library/Apple/usr/bin
/Applications/Sublime Text.app/Contents/SharedSupport/bin
All help appreciated.
I have resolved this issue.
According to this comment the error is commonly due to a missing resource file:
https://github.com/mkleehammer/pyodbc/issues/738
The required resource file is msodbcsqlr17.rll, according to the documentation:
https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15
This the documentation indicates the resource file should be located in:
/usr/local/share/msodbcsql17/resources/en_US
(or possibly a different location, relative to the location of the driver file, see the documentation link above)
For some reason my resource file was missing.
I located a copy by inspecting the home-brew formula at:
https://github.com/microsoft/homebrew-mssql-release/blob/master/Formula/msodbcsql17%4017.1.0.1.rb
This contains the download link:
https://download.microsoft.com/download/1/9/A/19AF548A-6DD3-4B48-88DC-724E9ABCEB9A/msodbcsql-17.1.0.1.tar.gz
Within the zipped package there is a copy of msodbcsqlr17.rll
I copied this to /usr/local/share/msodbcsql17/resources/en_US
And the problem is resolved.

Git dependency errors when using Composer with private VCS repository on IBM i

More details below, but I'm running into the following error when I issue a composer install or composer update command on a project which uses a custom VCS repository for an included project. This is on an IBM i v7r3 system via SSH session. I've run out of ideas on how to troubleshoot further, so here I am. Has anyone ever seen this error or have any ideas on how to troubleshoot further?
Error:
bash-4.4$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 113 installs, 0 updates, 0 removals
- Installing sshUser/myPackage (v1.0.26): Cloning a2db0666b1
[RuntimeException]
Failed to clone gitUser:/opt/git/myPackage.git, git was not found, check that
it is installed and in your PATH env.
Could not load program git:
Dependent module /usr/local/zendphp7/lib/libz.so.1(shr_64.o) could not be loaded.
File /usr/local/zendphp7/lib/libz.so.1 is not an
archive or the file could not be read properly.
System error: Exec format error
I have a remote IBM i server which has two users. The first user "gitUser" is used for my private Git repositories (headless). I then have another user "sshUser" which is used to SSH in with and do my typical work. I have SSH keys configured to simplify connections for both users and the "gitUser" ssh key doesn't require a password, so I can use it for automated scripting.
I have a private VCS repository included in my composer.json file for a project. So long as I have the entry in the "repository" section I'm getting the above error. This occurs even if I'm not requireing a package from the repository.
Repository Entry:
"repositories": [
{
"type": "vcs",
"url": "gitUser:/opt/git/myPackage.git"
}
]
Composer on the IBM i is up to date at 1.7.2.
It's clear the issue lies with git and the inability to find a dependency in libz.so.1.
However, if I follow the chain of dependencies for the version of Git in my path, all the appropriate files exist with adequate permissions:
bash-4.4$ which git
/QOpenSys/pkgs/bin/git
bash-4.4$ dump -X64 -H /QOpenSys/pkgs/bin/git
/QOpenSys/pkgs/bin/git:
***Loader Section***
Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x000000df 0x0000433f 0x0000009d
#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000006 0x00044910 0x00000a18 0x000449ad
***Import File Strings***
INDEX PATH BASE MEMBER
0 /QOpenSys/pkgs/lib:/QOpenSys/usr/lib
1 libz.so.1 shr_64.o
2 libiconv.so.2 shr_64.o
3 libcrypto.so.1.0.0 shr_64.o
4 libpthread.a shr_xpg5_64.o
5 libc.a shr_64.o
bash-4.4$
I can confirm there are no permission failures through out all of this. So this doesn't seem to be the result of inadequate permissions. Also, I do not think the issue lies within the user profile gitUser as if I issue a composer install or composer update from another machine (with appropriate keys), everything works flawlessly. So it seems to be specific to the IBM i and the sshUser profile.
I'm starting to think Composer is using a different git executable and the dependencies of that executable do not exist. Though I assume Composer would just use my path.
Note*: I tried specifying my LIBPATH in my ./.profile with the following. But it doesn't seem to make any difference:
LIBPATH=/usr/local/zendphp7/lib
LIBPATH=$LIBPATH:/QOpenSys/pkgs/lib:/QOpenSys/usr/lib
export LIBPATH
When echoing:
bash-4.4$ echo $LIBPATH
/usr/local/zendphp7/lib:/QOpenSys/pkgs/lib:/QOpenSys/usr/lib
bash-4.4$
If I don't add the above to my ./.profile, then echo $LIBPATH prints a blank line to screen.
Solution:
I just changed the symlink in the ZENDPHP7 directory structure to the repo version. Alternatively you can manipulate your library path, but I couldn't figure out the proper way to do it on IBM i.
bash-4.4$ pwd
/usr/local/zendphp7/lib
bash-4.4$ ln -s /QOpenSys/pkgs/lib/libz.so.1 ./libz.so.1
bash-4.4$ ls -la /usr/local/zendphp7/lib/libz.so.1
lrwxrwxrwx 1 dl 0 56 Oct 11 14:21 /usr/local/zendphp7/lib/libz.so.1 -> /QOpenSys/pkgs/lib/libz.so.1
bash-4.4$
I'm guessing something is off with your path when you do the install. Is it maybe looking at the Zend PHP libraries rather than the expected ones in /usr/lib?
Notice that the errors shows a path of /usr/local/zendphp7/lib/ libz.so.1(shr_64.o) while your git dump instead shows /QOpenSys/pkgs/lib:/QOpenSys/usr/lib.

Having trouble with rsync version on mac when using it with lsyncd

I am trying to synchronize a folder on my mac with one on an ubuntu server 16.06 system that I built. The problem is with the rsync version. Here is what I did. I created a file in /var/log/lsyncd/ and called it lsyncd.conf.lua. The following are its contents:
settings {
logfile = "/var/log/lysncd/lsyncd.log", -- Sets the log file
statusFile = "/var/log/lsyncd/lsyncd-status.log" -- Sets the status log file
}
sync {
default.rsyncssh, -- uses the rsyncssh defaults Take a look here: https://github.com/axkibe/lsyncd/blob/master/default-rsyncssh.lua
delete = false, -- Doesn't delete files on the remote host eventho they're deleted at the source. This might be beneficial for some not for others
source="/Users/alixchristakaze/Desktop", -- Your source directory to watch
host="", -- The remote host (use hostname or IP)
targetdir="", -- the target dir on remote host, keep in mind this is absolute path
rsync = {
binary = "/usr/local/bin/rsync",
archive = true, -- use the archive flag in rsync
perms = true, -- Keep the permissions
owner = true, -- Keep the owner
_extra = {"-a"}, -- Sometimes permissions and owners isn't copied correctly so the _extra can be used for any flag in rsync
},
maxProcesses = 1 -- We only want to use a maximum of 1 rsync processes at same time
}
I get this error when I run this command: lsyncd /var/log/lsyncd/lsyncd.conf.lua
Warn: Using /dev/fsevents which is considered an OSX internal interface.
Warn: Functionality might break across OSX versions (This is for 10.5.X)
Warn: A hanging Lsyncd might cause Spotlight/Timemachine doing extra work.
Error: Cannot access /dev/fsevents monitor! (1:Operation not permitted)
As you can see I added this line binary = "/usr/local/bin/rsync", hoping it would tell system to use the rsync I installed through homebrew. However, I am getting the error above.
I have the latest mac version 10.12.3 and my rsync version is 3.1.2 . What am I doing wrong?

Error while connecting sparklyr to remote sparkR in Rstudio

I tried following command in my local RStudio session to connect to sparkR -
sc <- spark_connect(master = "spark://x.x.x.x:7077",
spark_home = "/home/hduser/spark-2.0.0-bin-hadoop2.7", version="2.0.0", config = list())
But, I am getting following error -
Error in start_shell(master = master, spark_home = spark_home, spark_version = version, :
SPARK_HOME directory '/home/hduser/spark-2.0.0-bin-hadoop2.7' not found
Any help?
Thanks in advance
may I ask you have you actually installed the spark into that folder?
Can you show the result of ls command in /home/ubuntu/ folder?
And sessionInfo() in R?
Let me please share with you how I am using the custom folder structure.
It is on Win, not Ubuntu but I guess it won't make much of the difference.
Using the most recent dev edition
If you would check on GitHub the RStudio guys are updating sparklyr almost every day fixing numerous reported bugs:
devtools::install_github("rstudio/sparklyr")
in my case only installation of sparklyr_0.4.12 has resolved problem with Spark 2.0 under Windows
Checking Spark availability
please check if version you're inquiring is available:
spark_available_versions()
You should see something like the line below, which indicates that the version you indend to use is actually available for your sparklyr package.
[13] 2.0.0 2.7 spark_install(version = "2.0.0", hadoop_version = "2.7")
Installation of Spark
Just to keep the order you may like to install spark in other location rather then home folder of RStudio cache.
options(spark.install.dir = "c:/spark")
Once you are sure the desire version is available it is time to install spark
spark_install(version = "2.0.0", hadoop_version = "2.7")
I'd check if it is install correctly (change it for shell ls if needed)
cd c:/spark
dir (in Win) | ls (in Ubuntu)
Now specify the location of the edition you want to use:
Sys.setenv(SPARK_HOME = 'C:/spark/spark-2.0.0-bin-hadoop2.7')
And finally enjoy the creation of connection:
sc <- spark_connect(master = "local")
I hope it helps.

Haskell, HDBC, ODBC, MySQL and Mac OS X

I'm trying to use Haskell (version 6.10.3) and HDBC to connect to a MySQL Database. I've chosen to do it with Haskell ODBC. I've used cabal to install HDBC (2.1.1) and HDBC-ODBC (2.1.0.0). I've downloaded and installed the MySQL ODBC driver (5.1.5). I used macports to install unixODBC (2.2.14_1). All of this on top of Mac OS X (10.5.8).
I've mostly been using the instructions on this page http://en.wikibooks.org/wiki/Haskell/Database. At around this point:
"# Add the mysql driver to odbcinst.ini file (under $ODBC_HOME/etc/) and your data source in $HOME/.odbc.ini."
It looks like the macports version of unixODBC installs everything under /opt/local/. I've put an odbcinst.ini into /opt/local/etc/ and I've created a .odbc.ini in my home directory which looks something like this (note that I've experimented with UID vs. USERNAME and PWD vs PASSWORD):
[ODBC Data Sources]
myodbc = MySQL ODBC 5.1 Driver
[ODBC]
Trace = 0
TraceAutoStop = 0
TraceFile =
TraceLibrary =
[myodbc]
Driver = /usr/local/lib/libmyodbc5.so
DATABASE = [hidden]
DESCRIPTION = [hidden]
SERVER = localhost
PORT = 3306
UID = [hidden]
PWD = [hidden]
PASSWORD = [hidden]
USER = [hidden]
And I've written and compiled this Haskell Program:
import Database.HDBC.ODBC
import Database.HDBC
import System
main = do
args <- getArgs
c <- connectODBC (args!!0)
tables <- getTables c
mapM_ putStrLn $ tables
When I try a DSN of "DSN=myodbc" it errors out with:
Database: SqlError
{seState = "[\"HY000\"]",
seNativeError = -1,
seErrorMsg = "connectODBC/sqlDriverConnect:
[\"1045: [unixODBC][MySQL][ODBC 5.1 Driver]Access
denied for user 'jamie'#'localhost' (using password: YES)\"]"}
However, when I try a DSN of "DSN=myodbc;UID=[hidden];PWD=[hidden]", it lists all the tables in the database.
This may be a unixODBC problem rather than a Haskell / HDBC / HDBC-ODBC problem. Running "isql myodbc" results in a "Bus Error". Running "isql -v myodbc" doesn't give any more information. Running isql [uid] [pwd] connects just fine.
iODBC, maintained and supported by my employer, has shipped as part of Mac OS X since Jaguar (10.2.x).
You'll be better off updating iODBC with all the latest patches (Apple tends to be a bit behind on these), than shifting to UnixODBC.
It's generally best to keep all your ODBC configuration in the default file locations for Mac OS X --
/Library/ODBC/odbc.ini
/Library/ODBC/odbcinst.ini
/Users/*/Library/ODBC/odbc.ini
/Users/*/Library/ODBC/odbcinst.ini
You can create symlinks from anywhere else you may want to have these files, e.g. --
ln -s ~/Library/ODBC/odbc.ini ~/.odbc.ini
Last, you may benefit from testing with a commercial ODBC driver for MySQL, such as one of my employer's offerings (two week free trial provided as part of download).

Resources