I would like to use pigz to compress massive tar archives.
I am using cygwin. Unfortunately, pigz is not one of the standard cygwin packages.
Anyone know how to install pigz under cygwin?
Below are the 2 techniques I tried without success:
1) The README on this webpage (or in the README file, if you download the source from here) says that you should be able to build it from source merely by
Type "make" in this directory to build the "pigz" executable.
When I do that on my machine, I get a ton of warnings starting with
pigz.c:2950:20: warning: unknown conversion type character 'j' in format [-Wformat=]
(intmax_t)g.in_tot, (intmax_t)len, tag);
and then this final error:
gcc -o pigz pigz.o yarn.o try.o deflate.o blocksplitter.o tree.o lz77.o cache.o hash.o util.o squeeze.o katajainen.o -lm -lpthread -lz
pigz.o:pigz.c:(.text+0xd4f8): undefined reference to `fsync'
collect2.exe: error: ld returned 1 exit status
make: *** [pigz] Error 1
That about exhausts my ability to build programs from source...
2) It looks like there is an old 2015 port of pigz version 2.3.3 to Cygwin Ports, the expanded cygwin package repository.
But that version out of date (the latest pigz is 2.4). Indeed, it looks like Cygwin Ports has migrated to github and searching there for pigz there finds nothing.
I am not even sure how to use Cygwin Ports! The project's homepage merely says
Follow the normal Cygwin installation instructions in order to install
any of the packages currently maintained by this project.
I assume that that means to run cygwin's setup-x86.exe, but when it asks you to "Choose A Download Site" you will need to enter some URL for Cygwin Ports.
Web searching found little information. This link says to use http://sourceware.org/cygwinports/ but setup-x86.exe soon generated an error for that URL. The instructions in this link also did not work for me.
The C99 standard specifies the j specifier for printf(). (Note that the 99 refers to 1999. It is now 2018.) You can force the pigz compilation to not assume C99 by changing __STDC_VERSION__-0 >= 199901L || __GNUC__-0 >= 3 to 0. Then it won't try to use j.
Please let me know what the values of __STDC_VERSION__, __GNUC__, and __GNUC_MINOR__ are for your compiler.
Also pigz requires POSIX compliance, which would provide the fsync() call. You can just delete the reference to fsync(), which would just result in the --synchronous and -Y options having no effect.
To follow up on comments above that I had with #varro and matzeri, I can now answer my own question: my suspicion was correct: RTools was the culprit. I found that if I temporarily removed all RTools elements from my Windows Path env var (for me: c:\Rtools\bin and c:\Rtools\mingw_32\bin), then I was able to get pigz make to work.
After doing this Path edit, I uninstalled my existing cygwin, reinstalled cygwin, installed my usual extra packages (chere, openssh, subversion, zip, unzip) and all their dependencies, installed make and all its dependencies, installed gcc-core (is the C compiler) and all its dependencies. At that point, I was able to make pigz perfectly.
There is a much easier way than compiling yourself. I had the same problem, and with a little bit of research found multiple ready-made .exe files (pigz.exe) for direct usage in Windows. I am using this one:
https://sourceforge.net/projects/pigz-for-windows/files/
The OP's main concern was: "I would like to use pigz to compress massive tar archives.", and I hope that this is a useful answer to that concern, although it does not explain how to get around the compiling problems.
Some additional notes:
The interesting thing that some folks may not be aware of is that nothing keeps us from using normal Windows binaries from within Cygwin, and vice versa. That is, even if the OP had sophisticated Cygwin / bash (or whatever) scripts which drive pigz and the whole process of compressing, he could use the ready-made pigz native Windows version linked above.
With or without Cygwin, there is no need to compile pigz yourself, unless you want the latest features or bug fixes.
Personally, I am using the native Windows pigz version from within Cygwin since a while. AFAIK, pigz has no progress bar, which is somehow inconvenient for me (from time to time I have to compress a single huge file (around 60 GB)). A convenient way to get around this is the pv utility. Since I haven't found a native Windows version of it, and since I am too lazy to compile it for Windows myself, I am using Cygwin's pv to display the progress when I let the native Windows pigz compress those huge files.
Related
I'm trying to use pp (the perl compiler) to create an application that can run independent of the perl installed library and interpreter.
It successfully creates a compiled executable although I had to use the -x -c options to get it to find dependencies successfully. It will run on my machine but when I try it on another machine I get this error so clearly there is still some dependency:
501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)
I am running it on MacOS 10.14.1 if that makes any difference. Thanks!
LWP::Protocol::https is loaded dynamically when needed, so pp has no way of knowing it's needed by default.
Solution 1
Pass -x to pp, and make sure the module is actually loaded in the run pp uses to determine the modules to include. This would probably be achieved by using LWP to make an HTTPS request during that run. --xargs=... might come in useful for this.
Solution 2
Pass -M LWP::Protocol::https to pp. You could also pass -M 'LWP::Protocol::**' to get all protocols handlers you have installed.
Solution 3
Add use LWP::Protocol::https (); to your script or an included module. Including a comment indicating why you are doing this would be appropriate.
You were building Net::SSLeay on MacOS 10.14 linking it to libssl.44.dylib which is not present on MacOS 10.12 where you try to run it.
I've found it annoying having to switch between build and test systems to find out which of the libraries are missing or incompatible and need to be packed.
I am now using the following strategy:
I use perlbrew instead of system perl.
For alien dependencies I use homebrew instead of the system libraries.
I build the packed executable using pp and run the resulting program with export DYLD_PRINT_LIBRARIES=YES being set (on the development machine)
I examine the list of loaded libraries and add all those referenced in the homebrew directory tree (/usr/local/opt/ and /usr/local/cellar/in my case) using pp -l /full/path/name -l ...
I rebuild the executable.
I still check on a target machine before deploying, but chances are very high now that it just works.
I've been trying to install LuaJIT on Windows 10 for some time following the official guide, and I actually get to install it. For example, if I execute luajit I get into the prompt. Also, luajit -v returns the version of luajit (2.0.4). And I can also execute code with luajit -e <lua code>. However, whenever I try to save bytecode with luajit -b, I get the following message:
luajit: unknown luaJIT command or jit.* modules not installed
I tried to make all sort of installations: using Cygwin, luajit-rocks, MinGW, ... However, no matter what I try, I always get the same result, and I have no clue of what to do.
Could you point me to some potential problems I might be overlooking?
I have on my system Lua 5.1 and Luarocks.
Some extra LuaJIT features are implemented as separate Lua modules (e.g. jit.bcsave for bytecode saving), and LuaJIT depends on package.path to find those modules. The suggested install location for those modules is in the default package.path, but if you override it via the LUA_PATH environment variable, you have to make sure to include that location there. One easy way to do that is to put two consecutive semicolons into LUA_PATH: Double semicolons are replaced by the compile-time default value of package.path.
You need place modules to "jit" folder near with juajit.exe. That folder include some system modules (bcsave too). package.path can dont work, becouse it hardlinked, how i understand. That folders distributed with source code.
Download lua from official sice: https://luajit.org/download.html
You can see "jit" folder inside archive:
LuaJIT-2.0.5.zip\LuaJIT-2.0.5\src\jit\
i'm tackling the problem of compiling vmime library using this guide with MinGW. As this guide states, first i need to compile libiconv library with these commands(yep i'm new to MinGW):
$ tar -xvvzf libiconv-1.13.1.tar.gz
$ cd ./libiconv-1.13.1
$ ./configure --prefix=/mingw #configures makefile, use /mingw as a prefix
$ make
$ make install
after all this commands the libiconv.dll.a appears in libiconv-1.13.1\lib.libs
directory.Also after compiling process appears the /bin directory and there is only 1 library - libcharset-1.dll.
My question is - how do i know if the library properly compiled, without errors?Should i check the output from the MSYS console? there are tons of checks, it seems pretty boring task. Thanks in advance, glad to hear any advice!
You're building a GNU Autotools package.
./configure generates the makefile(s) needed by make to build the library
on your particular system. If it thinks the library can't be built on your particular
system, it will tell you why. It might just miss some reason why you can't build
the library, because the library developer(s) have to script the tests that it runs, and might
just overlook some necessary ones. But if it misses something then make will fail.
make executes all the commands necessary to build the library on your system. If any of them fail,
then make will fail, and will tell you so unmistakably.
Likewise make install does everything necessary to install the library
under the default or specified prefix path.
Classically, unix tools (like the autootols) will inform you when something goes wrong
and not inform you that nothing went wrong.
If there a relatively simple way to make go + libxml2 + gokogiri work on windows?
I mean that I may be can install it (but at the moment I can not, stuck with Package libxml-2.0 was not found in the pkg-config search path), but then I need to provide my utilite to other people, who will never be able (or would wish ) to install lall libxml2 dependencies, modify PATH etc on windows...
It work flawless on Ubuntu...
I found this https://github.com/moovweb/gokogiri/issues/49 thats funny with installation of Gimp 2 (what?!), but I still cannot make it run with such error, I guess might be issue with PATH, but all PATH are set
$ go get github.com/moovweb/gokogiri
# github.com/moovweb/gokogiri/help
Documents\go\src\github.com\moovweb\gokogiri\help\help.go:6:25: fatal error: lib
xml/tree.h: No such file or directory
#include <libxml/tree.h>
^
compilation terminated.
# github.com/moovweb/gokogiri/xpath
Documents\go\src\github.com\moovweb\gokogiri\xpath\expression.go:4:26: fatal err
or: libxml/xpath.h: No such file or directory
#include <libxml/xpath.h>
^
compilation terminated.
You are struggling because it is hard to combine packages that were built by different people for different purposes and get your environment set up correctly. I think it is best to use MSYS2, an environment for Windows that provides a consistent set of packages for things like gcc, go, libxml2, and iconv. MSYS2 has a package manager (pacman) that helps you easily install them and keep them updated.
I don't do much programming with Go, but I am familiar with MSYS2 and it seems like I was able to get gokogiri installed using MSYS2. You should open MSYS2's "MinGW-w64 Win64 Shell" from the Start menu (mingw64_shell.bat), and try running these commands:
pacman -S mingw-w64-x86_64-{gcc,go,libxml2,iconv}
export GOROOT=/mingw64/
export GOPATH=/c/Users/David/Documents/goproj/
mkdir -p $GOPATH
go get github.com/moovweb/gokogiri
I think GOPATH should be set to the directory of your project. If you run into an error, it might be because some pacman package is required that I didn't list here.
The string mingw-w64-x86_64-{gcc,go,libxml2,iconv} gets expanded by Bash into the following list of packages:
mingw-w64-x86_64-gcc
mingw-w64-x86_64-go
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-iconv
If you are actually using 32-bit Windows, replace x86_64 with i686 in the instructions above.
If you are curious, the scripts for building those packages are here: https://github.com/Alexpux/MINGW-packages
As a disclaimer, I haven't actually compiled any go programs in MSYS2, so there could be big problems I am unaware of.
Also, one of the main developers of MSYS2 (alexpux) said this in the #msys2 IRC chat on 2015-06-21:
We not build go for a long time.
This package in very WIP state
Also see
https://github.com/Alexpux/MINGW-packages/issues/421
So you might need to fix some issues with the MSYS2 Go package and recompile it yourself to really make this work. But you have the PKGBUILD script that was used to build it, so maybe that will be less hard than what you are trying to do right now, which involves compiling/collecting every dependency of gokogiri.
MSYS2 would make your other installation of go, libxml2, and iconv obsolete. You can delete those things once you get your MSYS2 environment working.
If you are using visual studio and want to add dependency to your project then just install it using NuGet Package Manager it's easiest method.
Install command: Install-Package libxml2
The problem:
I can't seem to install perl modules correctly, JSON-2.53 in particular.
I have done the following:
Searched for a similar problem and tried its solution - did not work.
perl ".../config.h, needed by `Makefile'" not working after OSX Lion upgrade
Installed XCode command line developer utilities (c compiler, make, etc)
Read version compatibility documentation on this particular perl module: http://metacpan.org/pod/JSON
Ran the following commands to make and install the desired perl module:
$perl Makefile.PL
Welcome to JSON (v.2.53)
If you install JSON::XS v.2.27, it makes JSON faster.
************************** CAUTION **************************
This is 'JSON version 2' and there are many differences *
to version 1.xx *
Please check your applications useing old version. *
See to 'INCOMPATIBLE CHANGES TO OLD VERSION' and 'TIPS' *
Writing Makefile for JSON
(verified that the Makefile has been written)
$make
make: *** No rule to make target `/System/Library/Perl/5.12/darwin-thread-multi-2level/CORE/config.h', needed by `Makefile'. Stop.
What does that error even mean? What can I do to successfully make install this module?
Here are some additional items that may help you assist me in debugging this issue:
$which make
/Applications/Xcode.app/Contents/Developer/usr/bin/make
$which perl
/usr/bin/perl
$perl -v
This is perl 5, version 12, subversion 3 (v5.12.3) built for darwin-thread-multi-2level
I think you need to download and reinstall XCode. If I recall correctly for 10.7, after downloading Xcode from the app store it drops an installer into your Applications folder. You need to run it and try installing the command line tools again (from Xcode's prefernces pane). I know you mentioned you did this already, but a bit more background might explain why it's worth another try.
Here are the relevant lines in the Makefile from my Mac:
PERL_INC = /System/Library/Perl/5.12/darwin-thread-multi-2level/CORE
# Where is the Config information that we are using/depend on
CONFIGDEP = $(PERL_ARCHLIB)$(DFSEP)Config.pm $(PERL_INC)$(DFSEP)config.h
Later on in the Makefile CONFIGDEP is used as a dependency in a target. I believe in your case make is looking for /System/Library/Perl/5.12/darwin-thread-multi-2level/CORE/config.h and can't find it. The error you're seeing is make's obtuse way of saying file not found.
config.h contains specific information about the OS but is not needed for running scripts. It's only referenced when you want to compile a module. With stock OSX you get enough perl to execute scripts. Install XCode and you get the bits (like config.h) to do perl "development". I use quotes because you can write and run perl scripts without Xcode. But as you discovered, compiling a module requires the additional files Xcode provides. (Incidentally, RedHat does the same thing. You have to install the perl-devel package to get config.h. The perl runtime is in a separate package.)
Here are some things you can try:
Verify /System/Library/Perl/5.12/darwin-thread-multi-2level/CORE/config.h exists. If not, Xcode command line utilities were not installed properly. Try it again.
If config.h exists, check its content and make sure it looks sane. It's a C header file and consists of comments and #define statements.
If you don't have access to view config.h, you have a permission issue. Try using sudo make as a bypass. Disk Utility (found in Applications -> Utilities) might be able to permanently fix this.
You could risk changing the Makefile by removing "$(PERL_INC)$(DFSEP)config.h" from CONFIGDEP. I did this on my 10.8 Mac and it worked without issue (it passed all tests as well). However, if you don't find the root cause of your config.h issue, the next time you want to install a perl module you may find yourself right back where you started.
I had this exact same error, whilst this may not be a solution for you.... after reinstalling an updated xcode compatible with the OSX version (+rebooting after the install) I still had the error - to cut a long story short I noticed there was no config.h in /CORE/ after the error.....the solution that worked was to touch config.h and create the file first and then re-run the make. Hope this helps someone.