How does the same source code generate binaries for different platforms? - compilation

Many multi-platform applications seem to have common source code. How do builds generate platform specific binaries?
Is it possible to build say, a windows binary on linux or mac?

It's possible if you have an appropriate cross-compiler and libraries. For example, many programs which are available on both Linux and Windows use the MinGW toolchain on Windows, which includes a library that emulates POSIX functions using Win32 functions.

The platform a binary is compiled to run on depends on the compiler and generally, one can have the compiler compile for a target system. To that end, yes it is generally possible to compile for a system other than the one you are running on. Though you are usually better off compiling for a target system on that system.

Related

Go binary file for all platform

I have a .go file and produced the binary file using go build command from Mac. Is there a way to build a binary file which runs in windows,linux,IOS ?
I am aware we can build binary file for each of them by changing the GOOS,GOARCH params but i would like to have a single go binary file which should run in all the platforms . Please help me out of this.
Thanks in advance
No, it is not at all possible in Go or any other programming language (the executable is necessarily tailored to individual platforms and architectures).
However, to cross-compile, some tools do exist which do the cross compiling for you.
This post helps explain how to cross compile with Golang (which is pretty easy at this point).
There's also a Unix StackExchange question, https://unix.stackexchange.com/a/298283/177527, which explains why different architectures require different binaries:
The reason is because the code is compiled to machine code for a specific architecture, and machine code is very different between most processor families (ARM and x86 for instance are very different).
The binary also depends on the OS, as explained here https://softwareengineering.stackexchange.com/a/251255:
Binary Format: The executable has to conform to a certain binary format, which allows the operating system to correctly load, initialize, and start the program. Windows mainly uses the Portable Executable format, while Linux uses ELF.
System APIs: The program may be using libraries, which have to be present on the executing system. If a program uses functions from Windows APIs, it can't be run on Linux. In the Unix world, the central operating system APIs have been standardized to POSIX: a program using only the POSIX functions will be able to run on any conformant Unix system, such as Mac OS X and Solaris.
For Mac (not Windows), you can associate cross-compilation with a tool like randall77/makefat to generate a "universal binary", which will run on any architecture supported by one of the input executables.
This is currently implemented in goreleaser/goreleaser PR 2572, which means the process would be completely automated.

Cross-compile on a Linux host for various targets

I have a set of more or less portable C/C++ sources sitting on a Linux development host that I would like to be able to:
compile for 32- and 64-bit Linux targets
cross-compile for 32- and 64-bit Windows targets
cross-compile for 32- and 64-bit Mac targets
and, ideally, without any runtime dependencies on other emulation DLL's like cygwin1.dll, MinGW, etc though I could use them if there's no other choice. If I have to use them, I'd prefer statically linking their functionality to my code.
The target binary that is desired is:
a shared library (.so) for Linux and Mac targets, and
a DLL for Windows.
I have no idea how to build a cross-compiler (and the associated toolchain) from scratch. I'm hearing that pre-built cross-compiler toolchains are available for various host-and-target combinations, but I don't know where to find them, or even how to use them without running into runtime crashes/coredumps later due to pointer model subtleties (LP64, LLP64, etc), specifying wrong or inadequate compiler switches, other misconfiguration, etc.
I've so far been unable to find the relevant and complete information on the above, and whatever little I've managed to find is scattered all over the place in so many bits and pieces that I'm not even sure if all that I've read is complete or even correct (applies fully, no more no less to my case).
I'm not a compilers expert, just their regular user. Would appreciate information achieving the above compilation goals.
I would like to cross compile a library for Mac OsX on Linux and I am considering imcross. The instructions in the site are simple, but everytime you setup a crosscompiling environment you have to fix a lot of things, so I won't expect that it will be straightforward. You can check in the website that there are some limitations to this project but it is the best I came across.
Not being a priority for me now (I have other stuff to do before performing this task) I didn't setup the crossenvironment yet. I am going to do that in few days time.

Performance comparison between Windows gcc compiled & Visual Studio compiled

I'm currently compiling an open source optimization library (native C++) supplied with makefiles for use with gcc. As I am a Windows user, I'm curious on the two options I see of compiling this, using gcc with MinGW/Cygwin or manually building a Visual Studio project and compiling the source.
1) If I compile using MinGW/Cygwin + gcc, will the resulting .lib (static library) require any libraries from MinGW/Cygwin? I.e. can I distribute my compiled .lib to a Windows PC that doesn't have MinGW/Cygwin and will it still run?
2) Other than performance differences between the compilers themselves, is there an overhead associated when compiling using MinGW/Cygwin and gcc - as in does the emulation layer get compiled into the library, or does gcc build a native Windows library?
3) If speed is my primary objective of the library, which is the best method to use? I realise this is quite open ended, and I may be best running my own benchmarks, but if someone has experience here this would be great!
The whole point of Cygwin is the Linux emulation layer, and by default (ie if you don't cross-compile), binaries need cygwin1.dll to run.
This is not the case for MinGW, which creates binaries as 'native' as the ones from MSVC. However, MinGW comes with its own set of runtime libraries, in particular libstdc++-6.dll. This library can also be linked statically by using -static-libstdc++, in which case you also probably want to compile with -static-libgcc.
This does not mean that you can freely mix C++ libraries from different compilers (see this page on mingw.org). If you do not want to restrict yourself to an extern "C" interface to your library, you most likely will have to choose a single compiler and stick with it.
As to your performance concerns: Using Cygwin only causes a (minor?) penalty when actually interacting with the OS - where raw computations are concerned, only the quality of the optimizer matters.

Why does Mingw exist?

MSYS and UNIX-like build systems apart; why does Mingw exist? What I mean is, why isn't win32/64 just another target available in "vanilla GCC"? Are there technical reasons for this or are they "political"? It would seem to me that there isn't really anything special about the Windows platform that would make a GCC port to it "incompatible" with the original GCC.
Mingw isn't just the compilers - it's also the libraries and headers (and whatever other support files you might need).
GCC is just a compiler. Mingw is a full environment.
MingW is intended to allow easy porting of Posix apps to windows. It's simply a "thinner" layer than CygWin, so not everything might work where windows too different. CygWin allows provides enough insulation that even the scripting tools to build and generate code run.
Edit:
Microsoft is very fond of making there own standards, and want to put there own brand on things for which open standards already exist. i.e. OpenGL -> DirectX, Java -> .net, OpenCL is the next standard they want to spurn. This makes them a bit of a pain for non sponsored project to support them as a target. They have always turned away from anything that allows easy cross platform development, because they only want you to develop your app for there OS. It's a marketing strategy, control the brand, control the platform, and make it painful for people to support other OSes. MingW is required because GCC can generate x86 code, but even the most basic runtime support is different than Posix based OSes.
well, while you can use cross compiler to produce win32/64 binary codes from GCC, someone has to port all the C runtime libraries across though. Both Mingw and Cygwin are created for this purpose.
From MinGW's main site
Offered in easily installed binary package format, for native deployment on MS-Windows, or user-built from source, for cross-hosted use on Unix or GNU/Linux, the suite exploits Microsoft's standard system DLLs to provide the C-Runtime and Windows API.

What is the difference between Cygwin and MinGW?

I want to make my C++ project cross platform, and I'm considering using Cygwin/MinGW.
But what is the difference between them ?
Another question is whether I will be able to run the binary on a system without Cygwin/MinGW ?
As a simplification, it's like this:
Compile something in Cygwin and you are compiling it for Cygwin.
Compile something in MinGW and you are compiling it for Windows.
What is Cygwin?
Cygwin is a compatibility layer that makes it easy to port simple Unix-based applications to Windows, by emulating many of the basic interfaces that Unix-based operating systems provide, such as pipes, Unix-style file and directory access, and so on as documented by the POSIX standards. Cygwin is also bundled with a port of the GNU Compiler Collection and some other tools to the Cygwin environment.
If you have existing source code that uses POSIX interfaces, you may be able to compile it for use with Cygwin after making very few or even no changes, greatly simplifying the process of porting simple IO based Unix code for use on Windows.
Disadvantages of Cygwin
Compiling with Cygwin involves linking your program with the Cygwin run-time environment, which will typically be distributed with your program as the dynamically linked library cygwin1.dll. This library is open source and requires software using it to share a compatible open source license, even if you distribute the dll separately, because the header files and interface are included. This therefore imposes some restrictions on how you can license your code.
What is MinGW?
MinGW is a distribution of the GNU compiler tools for native Windows, including the GNU Compiler Collection, GNU Binutils and GNU Debugger. Also included are header files and libraries allowing development of native Windows applications. This therefore will act as an open source alternative to the Microsoft Visual C++ suite.
It may be possible to use MinGW to compile something that was originally intended for compiling with Microsoft Visual C++ with relatively minor modifications.
By default, code compiled in MinGW's GCC will compile to a native Windows target, including .exe and .dll files, though you could also cross-compile with the right settings, since you are basically using the GNU compiler tools suite.
Even though MingW includes some header files and interface code allowing your code to interact with the Windows API, as with the regular standard libraries this doesn't impose licensing restrictions on software you have created.
Disadvantages of MinGW
Software compiled for Windows using MinGW has to use Windows' own API for file and IO access. If you are porting a Unix/Linux application to Windows this may mean significant alteration to the code because the POSIX type API can no longer be used.
Other considerations
For any non-trivial software application, such as one that uses a graphical interface, multimedia or accesses devices on the system, you leave the boundary of what Cygwin can do for you and further work will be needed to make your code cross-platform. But, this task can be simplified by using cross-platform toolkits or frameworks that allow coding once and having your code compile successfully for any platform. If you use such a framework from the start, you can not only reduce your headaches when it comes time to port to another platform but you can use the same graphical widgets - windows, menus and controls - across all platforms if you're writing a GUI app, and have them appear native to the user.
For instance, the open source Qt framework is a popular and comprehensive cross-platform development framework, allowing the building of graphical applications that work across operating systems including windows. There are other such frameworks too. In addition to the large frameworks there are thousands of more specialized software libraries in existence which support multiple platforms allowing you to worry less about writing different code for different platforms.
When you are developing cross-platform software from the start, you would not normally have any reason to use Cygwin. When compiled on Windows, you would usually aim to make your code able to be compiled with either MingW or Microsoft Visual C/C++, or both. When compiling on Linux/*nix, you'd most often compile it with the GNU compilers and tools directly.
Cygwin is an attempt to create a complete UNIX/POSIX environment on Windows. To do this it uses various DLLs. While these DLLs are covered by GPLv3+, their license contains an exception that does not force a derived work to be covered by the GPLv3+. MinGW is a C/C++ compiler suite which allows you to create Windows executables without dependency on such DLLs - you only need the normal MSVC runtimes, which are part of any normal Microsoft Windows installation.
You can also get a small UNIX/POSIX like environment, compiled with MinGW called MSYS. It doesn't have anywhere near all the features of Cygwin, but is ideal for programmers wanting to use MinGW.
To add to the other answers, Cygwin comes with the MinGW libraries and headers and you can compile without linking to the cygwin1.dll by using -mno-cygwin flag with gcc. I greatly prefer this to using plain MinGW and MSYS.
Wikipedia does a comparison here.
From Cygwin's website:
Cygwin is a Linux-like environment for Windows. It consists of two parts: A DLL (cygwin1.dll) which acts as a Linux API emulation layer providing substantial Linux API functionality.
A collection of tools which provide Linux look and feel.
From Mingw's website:
MinGW ("Minimalistic GNU for Windows") is a collection of freely available and freely distributable Windows specific header files and import libraries combined with GNU toolsets that allow one to produce native Windows programs that do not rely on any 3rd-party C runtime DLLs
Cygwin uses a DLL, cygwin.dll, (or maybe a set of DLLs) to provide a POSIX-like runtime on Windows.
MinGW compiles to a native Win32 application.
If you build something with Cygwin, any system you install it to will also need the Cygwin DLL(s). A MinGW application does not need any special runtime.
Read these answered questions to understand the difference between Cygwin and MinGW.
Question #1: I want to create an application that I write source code once, compile it once and run it in any platforms (e.g. Windows, Linux and Mac OS X…).
Answer #1: Write your source code in
JAVA. Compile the source code once and
run it anywhere.
Question #2: I want to create an application that I write source code once but there is no problem that I compile the source code for any platforms separately (e.g. Windows, Linux and Mac OS X …).
Answer #2: Write your source code in C
or C++. Use standard header files
only. Use a suitable compiler for any
platform (e.g. Visual Studio for
Windows, GCC for Linux and XCode for
Mac). Note that you should not use any
advanced programming features to
compile your source code in all
platforms successfully. If you use
none C or C++ standard classes or
functions, your source code does not
compile in other platforms.
Question #3: In answer of question #2, it is difficult using different compiler for each platform, is there any cross platform compiler?
Answer #3: Yes, Use GCC compiler. It
is a cross platform compiler. To
compile your source code in Windows
use MinGW that provides GCC compiler
for Windows and compiles your source
code to native Windows program. Do not
use any advanced programming features
(like Windows API) to compile your
source code in all platforms
successfully. If you use Windows API
functions, your source code does not
compile in other platforms.
Question #4: C or C++ standard header files do not provide any advanced programming features like multi-threading. What can I do?
Answer #4: You should use POSIX
(Portable Operating System Interface
[for UNIX]) standard. It provides many
advanced programming features and
tools. Many operating systems fully or
partly POSIX compatible (like Mac OS
X, Solaris, BSD/OS and ...). Some
operating systems while not officially
certified as POSIX compatible, conform
in large part (like Linux, FreeBSD,
OpenSolaris and ...). Cygwin provides
a largely POSIX-compliant development
and run-time environment for Microsoft
Windows.
Thus:
To use advantage of GCC cross platform compiler in Windows, use MinGW.
To use advantage of POSIX standard advanced programming features and tools in Windows, use Cygwin.
From the point of view of porting a C program, a good way to understand this is to take an example:
#include <sys/stat.h>
#include <stdlib.h>
int main(void)
{
struct stat stbuf;
stat("c:foo.txt", &stbuf);
system("command");
printf("Hello, World\n");
return 0;
}
If we change stat to _stat, we can compile this program with Microsoft Visual C. We can also compile this program with MinGW, and with Cygwin.
Under Microsoft Visual C, the program will be linked to a MSVC redistributable run-time library: mxvcrtnn.dll, where nn is some version suffix. To ship this program we will have to include that DLL. That DLL provides _stat, system and printf. (We also have the option of statically linking the run-time.)
Under MinGW, the program will be linked to msvcrt.dll, which is an internal, undocumented, unversioned library that is part of Windows, and off-limits to application use. That library is essentially a fork of the redistributable run-time library from MS Visual C for use by Windows itself.
Under both of these, the program will have similar behaviors:
the stat function will return very limited information—no useful permissions or inode number, for instance.
the path c:file.txt is resolved according to the current working directory associated with drive c:.
system uses cmd.exe /c for running the external command.
We can also compile the program under Cygwin. Similarly to the redistributable run-time used by MS Visual C, the Cygwin program will be linked to Cygwin's run-time libraries: cygwin1.dll (Cygwin proper) and cyggcc_s-1.dll (GCC run-time support). Since Cygwin is now under the LGPL, we can package with our program, even if it isn't GPL-compatible free software, and ship the program.
Under Cygwin, the library functions will behave differently:
the stat function has rich functionality, returning meaningful values in most of the fields.
the path c:file.txt is not understood at all as containing a drive letter reference, since c: isn't followed by a slash. The colon is considered part of the name and somehow mangled into it. There is no concept of a relative path against a volume or drive in Cygwin, no "currently logged drive" concept, and no per-drive current working directory.
the system function tries to use the /bin/sh -c interpreter. Cygwin will resolve the / path according to the location of your executable, and expect a sh.exe program to be co-located with your executable.
Both Cygwin and MinGW allow you to use Win32 functions. If you want to call MessageBox or CreateProcess, you can do that. You can also easily build a program which doesn't require a console window, using gcc -mwindows, under MinGW and Cygwin.
Cygwin is not strictly POSIX. In addition to providing access to the Windows API, it also provides its own implementations of some Microsoft C functions (stuff found in msvcrt.dll or the re-distributable msvcrtnn.dll run-times). An example of this are the spawn* family of functions like spawnvp. These are a good idea to use instead of fork and exec on Cygwin since they map better to the Windows process creation model which has no concept of fork.
Thus:
Cygwin programs are no less "native" than MS Visual C programs on grounds of requiring the accompaniment of libraries. Programming language implementations on Windows are expected to provide their own run-time, even C language implementations. There is no "libc" on Windows for public use.
The fact that MinGW requires no third-party DLL is actually a disadvantage; it is depending on an undocumented, Windows-internal fork of the Visual C run-time. MinGW does this because the GPL system library exception applies to msvcrt.dll, which means that GPL-ed programs can be compiled and redistributed with MinGW.
Due to its much broader and deeper support for POSIX compared to msvcrt.dll, Cygwin is by far the superior environment for porting POSIX programs. Since it is now under the LGPL, it allows applications with all sorts of licenses, open or closed source, to be redistributed. Cygwin even contains VT100 emulation and termios, which work with the Microsoft console! A POSIX application that sets up raw mode with tcsetattr and uses VT100 codes to control the cursor will work right in the cmd.exe window. As far as the end-user is concerned, it's a native console app making Win32 calls to control the console.
However:
As a native Windows development tool, Cygwin has some quirks, like path handling that is foreign to Windows, dependence on some hard-coded paths like /bin/sh and other issues. These differences are what render Cygwin programs "non-native". If a program takes a path as an argument, or input from a dialog box, Windows users expect that path to work the same way as it does in other Windows programs. If it doesn't work that way, that's a problem.
Plug: Shortly after the LGPL announcement, I started the Cygnal (Cygwin Native Application Library) project to provide a fork of the Cygwin DLL which aims to fix these issues. Programs can be developed under Cygwin, and then deployed with the Cygnal version of cygwin1.dll without recompiling. As this library improves, it will gradually eliminate the need for MinGW.
When Cygnal solves the path handling problem, it will be possible to develop a single executable which works with Windows paths when shipped as a Windows application with Cygnal, and seamlessly works with Cygwin paths when installed in your /usr/bin under Cygwin. Under Cygwin, the executable will transparently work with a path like /cygdrive/c/Users/bob. In the native deployment where it is linking against the Cygnal version of cygwin1.dll, that path will make no sense, whereas it will understand c:foo.txt.
Other answers already hit the target. I just want to add an illustration for a quick catch.
Wikipedia Says:
MinGW forked from version 1.3.3 of Cygwin. Although both Cygwin
and MinGW can be used to port UNIX software to Windows, they
have different approaches: Cygwin aims to provide a complete POSIX layer
that provides emulations of several system calls and libraries
that exist on Linux, UNIX, and the BSD variants. The POSIX layer
runs on top of Windows, sacrificing performance where
necessary for compatibility. Accordingly, this approach requires
Windows programs written with Cygwin to run on top of a copylefted
compatibility library that must be distributed with the program, along
with the program's source code. MinGW aims to provide native
functionality and performance via direct Windows API calls. Unlike
Cygwin, MinGW does not require a compatibility layer DLL and
thus programs do not need to be distributed with source code.
Because MinGW is dependent upon Windows API calls, it cannot
provide a full POSIX API; it is unable to compile some UNIX applications that can be compiled with Cygwin. Specifically, this
applies to applications that require POSIX functionality like
fork(), mmap() or ioctl() and those that expect to be run in a
POSIX environment. Applications written using a cross-platform library that has itself been ported to MinGW, such as SDL,
wxWidgets, Qt, or GTK+, will usually compile as easily in
MinGW as they would in Cygwin.
The combination of MinGW and MSYS provides a small, self-contained
environment that can be loaded onto removable media without leaving
entries in the registry or files on the computer. Cygwin Portable
provides a similar feature. By providing more functionality, Cygwin
becomes more complicated to install and maintain.
It is also possible to cross-compile Windows applications with
MinGW-GCC under POSIX systems. This means that developers do not
need a Windows installation with MSYS to compile software that will
run on Windows without Cygwin.
Don't overlook AT&T's U/Win software, which is designed to help you compile Unix applications on windows (last version - 2012-08-06; uses Eclipse Public License, Version 1.0).
Like Cygwin they have to run against a library; in their case POSIX.DLL. The AT&T guys are terrific engineers (same group that brought you ksh and dot) and their stuff is worth checking out.
To use Cygwin in a non-free / proprietary / closed-source application, you'll need to fork out tens of thousands of dollars for a "license buyout" from Red Hat; this invalidates the standard licensing terms at a considerable cost. Google "cygwin license cost" and see first few results.
For mingw, no such cost is incurred, and the licenses (PD, BSD, MIT) are extremely permissive. At most you may be expected to supply license details with your application, such as the winpthreads license required when using mingw64-tdm.
EDIT thanks to Izzy Helianthus: The commercial license is no longer available or necessary because the API library found in the winsup subdirectory of Cygwin is now being distributed under the LGPL, as opposed to the full GPL.
Cygwin emulates entire POSIX environment, while MinGW is minimal tool set for compilation only (compiles native Win application.) So if you want to make your project cross-platform the choice between the two is obvious, MinGW.
Although you might consider using VS on Windows, GCC on Linux/Unices. Most open source projects do that (e.g. Firefox or Python).
Note that utility behaviour can genuinely vary between the two.
For example, Cygwin tar can fork - because fork() is supported in the DLL - where the mingw version cannot. This is a problem when trying to compile mysql from source.
Cygwin is designed to provide a more-or-less complete POSIX environment for Windows, including an extensive set of tools designed to provide a full-fledged Linux-like platform. In comparison, MinGW and MSYS provide a lightweight, minimalist POSIX-like layer, with only the more essential tools like gcc and bash available. Because of MinGW's more minimalist approach, it does not provide the degree of POSIX API coverage Cygwin offers, and therefore cannot build certain programs which can otherwise be compiled on Cygwin.
In terms of the code generated by the two, the Cygwin toolchain relies on dynamic linking to a large runtime library, cygwin1.dll, while the MinGW toolchain compiles code to binaries that link dynamically to the Windows native C library msvcrt.dll as well as statically to parts of glibc. Cygwin executables are therefore more compact but require a separate redistributable DLL, while MinGW binaries can be shipped standalone but tend to be larger.
The fact that Cygwin-based programs require a separate DLL to run also leads to licensing restrictions. The Cygwin runtime library is licensed under GPLv3 with a linking exception for applications with OSI-compliant licenses, so developers wishing to build a closed-source application around Cygwin must acquire a commercial license from Red Hat. On the other hand, MinGW code can be used in both open-source and closed-source applications, as the headers and libraries are permissively licensed.
MinGW (or MinGW-w64) Cygwin
-------------------- ------
Your program written Your program written
for Unix and GNU/Linux for Unix and GNU/Linux
| |
| |
V V
Heavy modifications Almost no modifications
| |
| |
V V
Compilation Compilation
Program compiled with Cygwin ---> Compatibility layer ---> Windows API
Program compiled with MinGW (or MingGW-w64) -------------> Windows API
Cygwin uses a compatibility layer, while MinGW is native. That is one of the main differences.
Cygwin is is a Unix-like environment and command-line interface for Microsoft Windows.
Mingw is a native software port of the GNU Compiler Collection (GCC) to Microsoft Windows, along with a set of freely distributable import libraries and header files for the Windows API. MinGW allows developers to create native Microsoft Windows applications.
You can run binaries generated with mingw without the cygwin environment, provided that all necessary libraries (DLLs) are present.

Resources