What is the correct way to install (system-wide) a D library (on a GNU system, at least) using Makefile.am?
Here is my code to install the static and shared libraries:
install-data-local:
install librdf_dlang.a librdf_dlang.so $(libdir)
The remaining question is how to install .d files for developers to use my library?
Particularly, what should be the installation directory for .d files?
If you are doing a system-wide installation of D libraries and source (interface files I presume), then the most common places are /usr/include/<project name> or /usr/local/include/<project name> as long as it does not clash with some existing C/C++ project that stores header files there. Some D programmers prefer /usr/include/d/ or /usr/local/include/d/ as well...
I for an example use /usr/di (D imports) for this purpose and my library projects have all their interface files there. I will explain why I do not like to have separate project directories there.
No matter what directory you pick, you need to update your compiler search paths.
Here is a part of my dmd.conf:
[Environment64]
DFLAGS=-I/usr/include/dmd/phobos -I/usr/include/dmd/druntime/import -I/usr/di -L-L/usr/lib64 -L--export-dynamic -fPIC
, and ldc2.conf looks like:
// default switches appended after all explicit command-line switches
post-switches = [
"-I/usr/include/d/ldc",
"-I/usr/include/d",
"-I/usr/di",
"-L-L/usr/lib64",
];
If you prefer to have a separate directory for every project, you would end up with -I<path> for each of them. - I really do not like this approach. However, it is very popular among developers so it is really up to you how to organise the D import files. I know how much developers dislike the Java approach with domain.product.packages, but this nicely fits into a single place where all D interface files are and most importantly there are no clashes because of the domain/product part...
According to Filesystem Hierarchy Standard (and e.g. this SO question)
/usr/local/include
looks a strong candidate on a "linux/unix-like system". See especially note 9:
Historically and strictly according to the standard, /usr/local is for data that must be stored on the local host (as opposed to /usr, which may be mounted across a network). Most of the time /usr/local is used for installing software/data that are not part of the standard operating system distribution (in such case, /usr would only contain software/data that are part of the standard operating system distribution). It is possible that the FHS standard may in the future be changed to reflect this de facto convention.
I have no idea about Windows.
Forgive me before I start, as I'm not a C / C++ etc programmer, a mere PHP one :) but I've been working on projects that use some others sourced from online open source repos, such as svn and git. For some of these projects, I need to install libraries and then run "./configure", "make" and then "make all" (as an example) and I do this on a "build" virtual machine to get the binaries that I need to use within my project.
The ultimate goal of some of my projects is to then take these "compiled" (if that's the correct term) binaries and place them onto a virtual machine which I would then re-distribute (according to licenses etc).
My question is this : when I build these binaries on my build machine, with all the pre-requisities that I need in order to build them in the first place ("build-essential" and "cmake" and "gcc" etc etc) - once the binaries are on my build machine (in /usr/lib for example) are they self-contained to the point that I can merely copy those /usr/lib binary files that the build created and place them in the same folder on the virtual machines that I would distribute, without the build servers having all the build components installed on them?
With all the dependencies that I would need to build the source in the place, would that finally built binary contain them all in itself, or would I have to include them on the distributed servers as well?
Would that work? Is the question a little too general and perhaps it would all depend on what I'm building?
Update from original posting after a couple of responses
I will be distributing the VMs myself, inasmuch as I will build them and then install my projects upon them. Therefore, I know the OS and environment completely. I just don't want to "bloat" them with unnecessary software that's been installed that I don't actually need because the compiled executables I will place on the distributed VMs in for example /usr/local/bin ...
That depends on how you link your program to libraries it depends on. In most cases, the default is to link dynamically, which means that you need to distribute your executable along its deps. You can check out what libraries are required to run the file using ldd command.
Theoretically, you can link everything statically, which means that library code would be compiled into executable. Thus, executable would really be self-contained, but linking statically is not always possible. This depends on actual libraries you are using and probably require playing with ./configure args when building them.
Finally, there are some liraries that always linked dynamically, such as libc. The good thing is that machine you are distributing to would surely have this library. The bad thing is that versions of these libraries may differ, and you might face ABI mismatch.
In short, if your project not huge and there is possibility to link everything statically, go this way. If not, read about AppImage and Docker.
The distribution of built libraries and headers (binary distribution) is a possible way and should work. (I do it in my projects always.)
It is not necessary that all of the libraries you built are installed into /usr/lib. To keep your target machine clean you can install it in other folder to, e.g.
/usr/local/MYLIB/lib/libmylib.so
/usr/local/MYLIB/include/mylib.h
/usr/local/MYOTHERLIB/lib/libmyotherlib.so
/usr/local/MYOTHERLIB/include/libmyotherlib.so
Advantages:
Easy installation, easy remove
All files within one subfolder, no files are missing, no mix with other libs
Disadvantage:
The loader must know the extra search path
Every source I found online says a full installation of Cygwin takes over 1 GB, but mine is only 100 MB. I was pretty sure I downloaded everything from the mirror servers, but the install took less than 5 minutes to complete instead of hours, as I'd expect if it were installing gigabytes of software.
Did Cygwin get a huge clean-up during 2012~2013, or did I do something wrong in the installation?
A full Cygwin installation can range from 23 to 112 GiB, depending on how you define "full."
Your 100 MB number tells me that you just clicked through the defaults presented by Cygwin's setup-*.exe program, selecting no optional packages, because that installs only the Base package set, which currently amounts to 0.1 GiB. Cygwin follows the modern net-connected software distribution model: it assumes you can just run setup-*.exe again and select new packages as you need them.
The Cygwin maintainers try to keep the Base category's package set as small as practical.¹ A Cygwin Base install gives you something much like an old-style Unix installation, covering little more than what POSIX specifies.
So How Do You Get a Full Installation?
The Cygwin installer does not have an obvious way to get a "full" installation, on purpose, because no one needs literally every package in the Cygwin repository.²
There is, however, a sneaky way to install everything. At the Select Packages screen...
...switch to Category view, then click the "Default" text to the right of the "All" group header. It will change to "Install," as will the corresponding text in all of the groups underneath it. This marks everything for installation.
I include this tip for completeness only. You do not want to do this! It will install gigs and gigs of stuff you will never use. Currently, there are 11242 packages in Cygwin,³ and installing every last one of them took 93 GiB of disk space for the installation tree plus 19 GiB for the download tree⁴ the last time I tried it. That gives the 112 GiB upper limit above.
All that unused software carries several costs. Even if disk space, download time, installation time, rebasing time, and local bandwidth use are of no consequence to you, consider the generous free mirror's wasted bandwidth.
I've done the experiment, so now you don't have to.
An Intelligent "Complete" Installation
I've come up with a simple set of package exclusion rules that results in a much smaller installation:
Skip all of the -debuginfo packages. Few people need these, and they take up a lot of space. Savings: about 53 GiB in the installation tree alone; more in the download tree.
It's easy to apply this rule. After selecting all packages for installation with the sneaky trick above but before you move on to the next screen, click the "Install" text next to the "Debug" category header until it switches back to "Default."
If you've already installed the debug packages, click that text until it says "Uninstall" instead.
Do not explicitly install any of the lib* packages. Let Cygwin's setup-*.exe automatically install libraries to satisfy package dependencies. Savings: about 5 GiB ⁵
To apply this rule, switch the "Libs" category to "Default" or "Uninstall" as you did with the "Debug" category. The installer will figure out which libraries you actually need in a later step.
Skip the cross-compilers and associated packages. Again, few people need these.⁶ Savings: About 4 GiB
There are two major sets of cross-development tools in Cygwin: the set for creating Cygwin executables of the other word size (i.e. 64-bit tools and libraries for 32-bit Cygwin, or vice versa) and the set for building MinGW executables of the same word size as your Cygwin installation.
To apply this rule for a 64-bit Cygwin installation, while still on the "Select Packages" screen, type cygwin32- in the package name search box at the top of that screen, then click the Default text next to each top-level category until it cycles to Default or Uninstall, as above.
Repeat that for mingw64-.
The idea is the same for 32-bit Cygwin, except that you search for and exclude packages with cygwin64- and mingw32- in their names instead.
By following this rule set, I was able to install nearly everything, taking only about 23 GiB.
Paring That Down
We can get the installation to be even smaller by excluding several other notorious disk hogs:
X11, the desktop environments, and the GUI apps together require about 11 GiB.⁷
A Cygwin Base + Devel installation comes to about 10 GiB.
A Cygwin Base + TeX category installation takes about 5 GiB. If you install only your native language's support package, it comes to about 3.7 GiB instead.
All of the -doc packages combined chew up about 5 GiB of disk space.
Someone who isn't a software developer, who doesn't use Cygwin for GUI stuff, who uses the Web for docs, and who doesn't use TeX for document creation could thus have a "full" Cygwin installation in only about 1 GiB.⁸
If you use Cygwin the way it is intended to be used, installing the base and only the extra packages you need at the moment, you probably won't even get your installation size to even those levels. I use Cygwin this way, and my installations are typically well under 1 GiB, yet they are "complete" by my lights, since they meet my current needs.
For Extra Disk-Filling Fun...
All of this testing was done with the 64-bit version of Cygwin. You can roughly double the above space requirements by installing a parallel 32-bit Cygwin installation.⁹
Doing so is not pointless. It is a viable alternative to using cross-compilers, for one thing. For another, the fundamental incompatibility of the two Cygwins means you may have need of both.
Footnotes
There are occasional threads on the Cygwin mailing list where someone argues that some very common package should be included in the Base category, such as Perl, and the result is usually that the maintainers decide not to add it to Base.
You also occasionally see the opposite: some package accidentally slips into Base, usually via an incorrect dependency. Shortly after the problem is brought to the attention of the Cygwin maintainers, the status quo ante is restored. (example)
Perhaps I am wrong.
There could be a vision-impaired Czech immigrant musician who completes US government software development contracts on the side while not busy brushing up on his technical Hindi vocabulary by translating electrical engineering reports into his adopted Mandarin.
I want to meet him. He sounds like an interesting guy.
Plus, I think I can help him with his plan to create a Tcl/Tk GUI front end for Orpie. Naturally, I will try to talk him into porting it from Ocaml to C++/Qt.
I mean, Tcl/Tk, seriously? In 2018? Let's not be ridiculous.
curl -s https://cygwin.com/packages/package_list.html | grep -c x86_64/
Cygwin's setup-*.exe doesn't delete the downloaded package files after installing them. This is useful at sites where you have multiple Cygwin installations, since you can put the download directory on a shared network drive. Each package then only has to be downloaded once at that site.
My 105 GiB upper limit assumes you will download and install to the same drive, and that you will keep the download tree in case you need to reinstall it later.
Not only does setup-*.exe not delete downloaded package files after installing them, it doesn't auto-purge old versions, so your download tree grows and grows over the years you use Cygwin. (There are scripts floating about the net to solve this problem, such as this one.)
All data storage values given in this answer are apparent disk usage numbers — du -bhs — rather than actual disk usage numbers, which would account for the file system overhead, since that varies between systems. This affects the installation tree to a much greater degree than it does the download tree since the proportion of small files is much greater in the installation tree. Expect something like +1% in the download tree and +5% in the installation tree.
You may wonder why there are libraries in the Cygwin package repository that you don't need even though you've installed "all" packages. There are several reasons:
some libraries are obsolete, but are still present on the mirrors
some libraries come in multiple alternative forms, so that people who know they need something other than the default can choose it
some libraries are there only for people writing their own programs, not to support any existing Cygwin package
Pretty much the only people who need the Cygwin cross-compilers are the people maintaining Cygwin packages, since maintainers are expected to build for both 32-bit and 64-bit Cygwin unless there is a good reason not to.
There are probably more people with a good justification for MinGW cross-development tools, but there's also the option of using MinGW and MSYS instead of Cygwin. Also, I am guessing that the number of people who do dual-stack Cygwin + MinGW development is smaller than the set of people who use one or the other exclusively, or nearly so.
It is not easy to do this exclusion, because GUI packages are scattered throughout the Cygwin packaging system, and their top-level category often contains non-GUI software, so you can't simply exclude the whole category. (e.g. Math.)
The first pass is to exclude the X11, GNOME, KDE, LXDE, MATE, Xfce, and Games categories using the same Categories view technique as above.
Then, using the search box as we did for the cross-compiler exclusion above, remove packages matching gtk, gnome, qt, and kde, optionally excluding those in the Devel, Debug or Libs packages, if you need those.
Finally, you'll have to switch to the Pending view and manually exclude a bunch of packages that weren't caught by either of those two broad exclusions: Abiword, Calligra, Celestia, Dia, Evince, Geany, gEdit, Geomview, Gimp, gLabels, GnuCash, Gnumeric, Kalzium, KMyMoney, KolourPaint, Konqueror, Krita, KStars, LyX, Marble, Pidgin, QupZilla, Scribus, Skrooge, Spectacle, Stellarium, Tellico, and Vinagre. If you don't exclude these, they'll drag back in much of what you excluded above as dependencies!
You may notice that the size of all the excluded package sets adds up to more than the 23 GiB of the "intelligent complete" installation. This is because of shared dependencies. That is to say, these sizes overlap to some extent.
If you put both setup-*.exe programs in the same directory, they will share the download tree so that all of the noarch packages are downloaded only once for both Cygwins. Between that and the fact that 32-bit and 64-bit Intel compilers generate code that differs in size, installing both Cygwins doesn't quite double the disk space requirement.
When installing packages for the first time, setup*.exe does not install every package. Only the minimal base packages from the Cygwin distribution are installed by default.
Clicking on categories and packages in the setup*.exe package installation screen will provide you with the ability to control what is installed or updated.
Clicking on the "Default" field next to the "All" category will provide you with the opportunity to install every Cygwin package.
Installing and Updating Cygwin Packages
I just did a full installation of Cygwin x64 (2018-01-03) on Windows 7 SP1 x64 Ultimate:
Cygwin folder (which by default is C:\cygwin64) has the following properties:
Size: 89.1 GiB (95,671,245,179 bytes)
Size on disk: 92.3 GB (99,148,398,592 bytes)
Contains: 1,433,494 Files, 94,363 Folders
Cygwin temporary folder where downloaded packages are stored prior to being installed (can be removed after Cygwin is installed):
Size: 18.0 GiB (19,363,639,932 bytes)
Size on disk: 18.0 GiB (19,383,767,040 bytes)
Contains: 9,711 Files, 10,354 Folders
I would like to add to this thread. This approach gives you a leaner, meaner, bare-bones, minimal Cygwin install, with just the tools/items you need. No dependency bloat, no unwanted packages, files etc.
I have been experimenting with Cygwin attempting to get a "bare-bones", minimal install. I do find that installing utilities like grep, gawk, sed and similar tools has dependencies on cygwin, base-Cygwin and sometimes unwanted tools like bash, coreutils etc.
I wanted to get only the tools and their required dlls installed and started examining the Cygwin package. I discovered that not using the setup.exe supplied by Cygwin is an alternative way to accomplish minimal Cygwin installs.
And this is how I got it done:
Download only the packages you want from any of the Cygwin mirror
sites using ftp or http. Alternatively you can use the setup.exe
supplied by Cygwin to download all the packages - download only and
no install.
Once the download is successfully completed, individual
packages like zlib, gawk, grep, libiconv are found under the
x86/release or x86_64/release directory Each package is 'tar'red and
compressed using tool 'xz' or bzip and stored in respective
directories.
To install a specific tool like sed or gawk, all that
needs to be done, is to extract the tool executable and its
dependencies (.dll)
Before you attempt the following, please ensure you have a tool like 7z.exe, xz.exe, bzip2 or other that is capable of uncompressing an .xz or bzip archive
Installing gawk example below :
Extract gawk.exe from gawk-4.1.3-1.tar.xz archive using the command 7z.exe e -so gawk-4.1.3-1.tar.xz | tar xvf -
Once that is done, you should find gawk.exe in a subfolder usually, usr/bin under the release/gawk folder
Find the dependencies for gawk, you can do this in a couple of ways.
Examine the Cygwin setup.ini file found in x86 or x86_64 folder.
Look for the string '# gawk' and in the lines after this line you should find a "requires:" line that lists the dependencies.
Mine reads like this - "requires: bash cygwin libgmp10 libintl8 libmpfr4 libreadline7"
For gawk to run, bash is not a must since we have the windows command shell. (bash is included to get a few other dlls required by gawk. However, that causes a lot more unecessary files to be installed.) The other dependencies contain files that gawk needs to run.
Extract each of the above packages using tools like 7z or xz into individual files.
After all the dependencies are extracted, copy your needed tool(s) (grep/sed/gawk) to a folder and all the dependent .dlls
You should now be able to run your tool with the minimum set of .dlls required in a bare-bones cygwin installation.
Caution : It may not be sufficient to just extract the dependencies listed in setup.ini for each tool. Sometimes, you may need to execute/run the tool to discover that there are more dlls required.
There are other means of finding out the dlls required by an .exe - you can use dumpbin from MS or dependency walker, ndepends or similar tools to find the list of dependent dlls
Consult - How do I detect the DLLs required by an application?
How do I find out which dlls an executable will load?
I also brute forced this dll info by just running the tool and installing the missing dlls listed one by one by extracting from the required packages.
When you run a tool and it errors out with a missing .dll message, search for the package that contains the dll here - https://cygwin.com/cgi-bin2/package-grep.cgi . Enter the full/partial name of the missing dll to find the name of the package containing the dll.
Eventually, I have ended up with a bare-bones cygwin install with only the tools and dlls that I need.
Example : gawk - gawk.exe and the following dlls - cygwin1.dll, cyggmp-10.dll, cygiconv-2.dll, cygintl-8.dll, cygmpfr-4.dll, cyggcc_s-seh-1.dll, cygncursesw-10.dll, cygreadline7.dll
sed - sed.exe and dlls - cygwin1.dll, cygintl-8.dll
Hope this is found useful. The Cywin installer also does dll re-basing, which I will not venture into here.
Under the devel you may need only the following:
gcc-core:GNU Compiler collection(C Open MP)
gcc-fortran:GNU Compiler collection(Fortran)
gcc-g++:GNU Compiler collection(C++)
gcc-objc:GNU Compiler collection(Objective-C)
gdb: The GNU Debugger
make: The GNU version of the "make" utility
Photo
Often one wants to LoadLibrary a system DLL like Kernel32.dll, handily LoadLibraryEx has a dwFlags parameter that allows you to specify LOAD_LIBRARY_SEARCH_SYSTEM32.
For security reasons, this can be very important as malicious attackers can subvert the default search paths that LoadLibrary looks through.
However Windows XP, which I must still support, cannot use LOAD_LIBRARY_SEARCH_SYSTEM32 as it didn't exist eons ago. This leads me to the following conclusion: I need to come up with a fully qualified path and pass that to LoadLibrary.
However, a robust implementation of this is tricky due to WoW64 and file-system redirection.
I have to worry about this possibility because the code in question lives in a library and can be called in strange environments.
My current thoughts are something like the following pseudo-code:
string Path;
BOOL Wow64Process;
if (IsWow64Process(CurrentProcess(), &Wow64Process) && Wow64Process &&
WoW64FSRedirectionIsDisabled)
Path = GetSystemWow64Directory();
else
Path = GetSystemDirectory();
Path += "\Kernel32.dll";
LoadLibrary(Path);
I have considered, and rejected using Wow64DisableWow64FsRedirection and Wow64RevertWow64FsRedirection but this is problematic as it will leave FS redirection disabled in the library's DLLMain.
Given the above, what's the best way to go about what I am trying to do?
Thanks!
In terms of loading a DLL without dependencies, you are over thinking this. If the user has disabled file system redirection in a WOW64 process then LoadLibraryEx will attempt to load the DLLs from the 64 bit system32 folder. So you can simply do exactly the same and remove all that WOW64 code.
However, the documentation says (emphasis mine):
If this value is used, %windows%\system32 is searched for the DLL and its dependencies.
So, your approach cannot replicate LOAD_LIBRARY_SEARCH_SYSTEM32 because you can only supply the full path for the DLL you load, but not for its dependencies.
The only way to hope to replicate this is to force system32 to be at the head of the DLL search path. Specifying a full path won't help you.
Finally, are you sure that LOAD_LIBRARY_SEARCH_SYSTEM32 is not available on XP64?
As an alternative to the other answers use SetDlldirectory.
The advantage ist that the current Directory isn't checked for a DLL-load.
Still open is the search via PATH... but this is in the last Position, so your DLL located in the SYSTEM32 directoryies should be loaded first using SettDllDirectory.
Also it is supported since XP SP1
On Linux/GCC I can use the -rpath flag to change an executables search path for shared libraries without tempering with environment variables.
Can this also be accomplished on Windows? As far as I know, dlls are always searched in the executable's directory and in PATH.
My scenario: I would like to put shared libraries into locations according to their properties (32/64bit/Debug/Release) without taking care of unique names. On Linux, this is easily be done via rpath, but I haven't found any way doing this on Windows yet.
Thanks for any hints!
Sadly there is no direct analogue to RPATH. There are a number of alternative possibilities, each of them most likely undesirable to you in its own special way.
Given that you need a different exe for each build flavor anyway to avoid runtime library clashes, as you might guess the easiest thing to do is to put each exe in the same folder as each set of DLLs.
As you also mentioned, the most universal method is to change the PATH variable by using a batch file to bootstrap the exe.
You could instead change the current working directory before running the program to the desired DLL folder.
You can use the function SetDllDirectory or AddDllDirectory inside your exe. This is probably the closest to an RPATH, but only works on WinXP SP1 or later.
If you're willing to alter the file name of each exe flavor, you can use the "App Paths" registry key. Each exe would need a unique filename.
The search order for DLLs in Windows is described on this page on MSDN. If you're using run-time dynamic linking, you can specify the folder when you call LoadLibrary.
"Isolated applications" is a mechanism for embedding an XML manifest that describes the DLL dependencies.