In Git Bash I've tried to use this command:
$ git archive -o test.tar.gz master
gzip: compressed data not written to a terminal. Use -f to force compression.
For help, type: gzip -h
The file test.tar.gz is empty, but my repository is not empty and creating a zip file works fine (contains all my source files)! Why does the tarball format fail to produce an archive?
This appears to be a compatibility problem between the way git archive wants to pipe content from tar to gzip and the way Windows handles pipes. You can generate the same error message by piping tar into gzip manually:
$ tar -c file.txt | gzip
gzip: compressed data not written to a terminal. Use -f to force compression.
For help, type: gzip -h
These two commands work for me on Windows 7, and should be functionally identical to the one you're trying:
$ git archive -o test.tar master
$ gzip test.tar
Pipe it to gzip:
git archive master | gzip > test.tar.gz
Even if you are not using Git Bash, from regular Command Prompt you can just:
git archive --format=tar.gz master > test.tar.gz
Related
I'm working on a python scrip that verify the integrity of some downloaded projects.
On my nas, I have all my compressed folder: folder1.tar.gz, folder2.tar.gz, …
On my Linux computer, the equivalent uncompressed folder : folder1, folder2, …
So, i want to compare the integrity of my files without any UnTar or download !
I think i can do it on the nas with something like (with md5sum):
sshpass -p 'pasword' ssh login#my.nas.ip tar -xvf /path/to/my/folder.tar.gz | md5sum | awk '{ print $1 }'
this give me a hash, but I don't know how to get an equivalent hash to compare with the normal folder on my computer. Maybe the way I am doing it is wrong.
I need one command for the nas, and one for the Linux computer, that output the same hash ( if the folders are the same, of course )
If you did that, tar xf would actually extract the files. md5sum would only see the file listing, and not the file content.
However, if you have GNU tar on the server and the standard utility paste, you could create checksums this way:
mksums:
#!/bin/bash
data=/path/to/data.tar.gz
sums=/path/to/data.md5
paste \
<(tar xzf "$data" --to-command=md5sum) \
<(tar tzf "$data" | grep -v '/$') \
| sed 's/-\t//' > "$sums"
Run mksums above on the machine with the tar file.
Copy the sums file it creates to the computer with the folders and run:
cd /top/level/matching/tar/contents
md5sums -c "$sums"
paste joins lines of files given as arguments
<( ...) runs a command, making its output appear in a fifo
--to-command is a GNU tar extension which allows running commands which will receive their data from stdin
grep filters out directories from the tar listing
sed removes the extraneous -\t so the checksum file can be understood by md5sum
The above assumes you don't have any very-oddly named files (for example, the names can't contain newlines)
I am using the following to export all the changed files from my master branch:
git archive --output=test_v2.zip HEAD $(git diff --name-only master)
however it gives me the following error:
error: unknown option `name-only'
usage: git archive [<options>] <tree-ish> [<path>...]
or: git archive --list
or: git archive --remote <repo> [--exec <cmd>] [<options>] <tree-ish> [<path>...]
or: git archive --remote <repo> [--exec <cmd>] --list
--format <fmt> archive format
--prefix <prefix> prepend prefix to each pathname in the archive
-o, --output <file> write the archive to this file
--worktree-attributes
read .gitattributes in working directory
-v, --verbose report archived files on stderr
-0 store only
-1 compress faster
-9 compress better
-l, --list list supported archive formats
--remote <repo> retrieve the archive from remote repository <repo>
--exec <command> path to the remote git-upload-archive command
what am I doing wrong?
it creates test_v2.zip but it is empty.
based on my research I think I should not use $ sign but I don't know what to use then.
Your example works fine in bash. But you try to run it in cmd.exe. The $() syntax is Bash syntax to start a subshell, execute the enclosed command and replace the construct with the output.
Either do the same in Git Bash (anyway the better choice) or use a cmd.exe specific construct instead if one exists.
Current script that backs up uses just tar with no compression. We need to compress the backups however we need the CPU utilization to compress these backups to happen on the backup server
Current:
tar -cvf - . | ssh user#backupserver.com "cat > ~/vps/v16/vzpbackup_${CTID}_${HNAME}_${TIMESTAMP}.tar"
New:
tar -cvf - . | ssh user#backupserver.com "cat > ~/vps/v17/vzpbackup_${CTID}_${HNAME}_${TIMESTAMP}.tar" ; ssh user#backupserver.com "cd ~/vps/v17/; tar --use-compress-program=pigz -cvf vzpbackup_${CTID}_${HNAME}_${TIMESTAMP}.tar.gz vzpbackup_${CTID}_${HNAME}_${TIMESTAMP}.tar"
Is there a better way of achieving this?
Yes. You can compress on-the-fly:
tar -cvf - . | ssh user#backupserver.com "pigz > ~/vps/v17/vzpbackup_${CTID}_${HNAME}_${TIMESTAMP}.tar.gz"
Background: since you already send a tar file via the pipe, you just need to compress the piped data on the receivers side -- and you end up with a compressed tar file.
BTW, the "New" script in your question will create a compressed tar file that contains the uncompressed tar file. This is probably not what you want.
I installed pigz via homebrew on my Macbook Air (OS X 10.10.5) to get better performance for compress/decompress.
To compress, I use tar --use-compress-program=pigz -cf test.tgz test and it's ok.
But the command to uncompress, tar --use-compress-program=pigz -xf test.tgz output error:
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Or sometimes it output:
tar: Unrecognized archive format
pigz: abort: write error on <stdout> (Broken pipe)
tar: Child process exited with status 32
tar: Error exit delayed from previous errors.
I read the manual of tar, and have no clue why it doesn't work.
I noticed that even tar --use-compress-program=gzip -xf test.tgz generate the same error. So is this a bug of OSX's tar implementation?
Note: I know pipe style pigz -d test.tgz | tar -xf works and in this case I could also just use tar -xf test.tgz which call built-in gzip. But I just want to confirm whether it is a bug.
The program works as designed: there is no provision in its command-line to pass along the options needed to use gzip for decompressing. Instead of "gzip" for decompressing, you should use the wrapped gzcat, e.g.,
tar --use-compress-program gzip -cf foo.compressed foo
tar --use-compress-program gzcat -tf foo.compressed
A quick check shows that this does not work:
tar --use-compress-program 'gzip -d' -tf foo.compress
although that could change some time (it is doable, but not done).
According to pigz's manual page, it has unpigz, which is what you can use for that program.
This is an issue between BSD tar and GNU tar.
I fixed this by installing gnu-tar from homebrew and placing that on the path.
See this similar question on superuser: https://superuser.com/questions/318809/linux-os-x-tar-incompatibility-tarballs-created-on-os-x-give-errors-when-unt
on linux I renamed gzip to gzip.sav and made a soft link to pigz so tar, dolphin, yum, and any other program that calls gzip actually calls pigz
I have a pkg file created by Install Maker for Mac.
I want to replace one file in pkg. But I must do this under Linux system, because this is a part of download process. When user starts to download file server must replace one file in pkg.
I have a solution how unpack pkg and replace a file but I dont know how pack again to pkg.
http://emresaglam.com/blog/1035
http://ilostmynotes.blogspot.com/2012/06/mac-os-x-pkg-bom-files-package.html
Packages are just .xar archives with a different extension and a specified file hierarchy. Unfortunately, part of that file hierarchy is a cpio.gz archive of the actual installables, and usually that's what you want to edit. And there's also a Bom file that includes information on the files inside that cpio archive, and a PackageInfo file that includes summary information.
If you really do just need to edit one of the info files, that's simple:
mkdir Foo
cd Foo
xar -xf ../Foo.pkg
# edit stuff
xar -cf ../Foo-new.pkg *
But if you need to edit the installable files:
mkdir Foo
cd Foo
xar -xf ../Foo.pkg
cd foo.pkg
cat Payload | gunzip -dc |cpio -i
# edit Foo.app/*
rm Payload
find ./Foo.app | cpio -o | gzip -c > Payload
mkbom Foo.app Bom # or edit Bom
# edit PackageInfo
rm -rf Foo.app
cd ..
xar -cf ../Foo-new.pkg
I believe you can get mkbom (and lsbom) for most linux distros. (If you can get ditto, that makes things even easier, but I'm not sure if that's nearly as ubiquitously available.)
Here is a bash script inspired by abarnert's answer which will unpack a package named MyPackage.pkg into a subfolder named MyPackage_pkg and then open the folder in Finder.
#!/usr/bin/env bash
filename="$*"
dirname="${filename/\./_}"
pkgutil --expand "$filename" "$dirname"
cd "$dirname"
tar xvf Payload
open .
Usage:
pkg-upack.sh MyPackage.pkg
Warning: This will not work in all cases, and will fail with certain files, e.g. the PKGs inside the OSX system installer. If you want to peek inside the pkg file and see what's inside, you can try SuspiciousPackage (free app), and if you need more options such as selectively unpacking specific files, then have a look at Pacifist (nagware).
You might want to look into my fork of pbzx here: https://github.com/NiklasRosenstein/pbzx
It allows you to stream pbzx files that are not wrapped in a XAR archive. I've experienced this with recent XCode Command-Line Tools Disk Images (eg. 10.12 XCode 8).
pbzx -n Payload | cpio -i
In addition to what #abarnert said, I today had to find out that the default cpio utility on Mountain Lion uses a different archive format per default (not sure which), even with the man page stating it would use the old cpio/odc format. So, if anyone stumbles upon the cpio read error: bad file format message while trying to install his/her manipulated packages, be sure to include the format in the re-pack step:
find ./Foo.app | cpio -o --format odc | gzip -c > Payload
#shrx I've succeeded to unpack the BSD.pkg (part of the Yosemite installer) by using "pbzx" command.
pbzx <pkg> | cpio -idmu
The "pbzx" command can be downloaded from the following link:
pbzx Stream Parser
If you are experiencing errors during PKG installation following the accepted answer, I will give you another procedure that worked for me (please note the little changes to xar, cpio and mkbom commands):
mkdir Foo
cd Foo
xar -xf ../Foo.pkg
cd foo.pkg
cat Payload | gunzip -dc | cpio -i
# edit Foo.app/*
rm Payload
find ./Foo.app | cpio -o --format odc --owner 0:80 | gzip -c > Payload
mkbom -u 0 -g 80 Foo.app Bom # or edit Bom
# edit PackageInfo
rm -rf Foo.app
cd ..
xar --compression none -cf ../Foo-new.pkg
The resulted PKG will have no compression, cpio now uses odc format and specify the owner of the file as well as mkbom.
Bash script to extract pkg: (Inspired by this answer:https://stackoverflow.com/a/23950738/16923394)
Save the following code to a file named pkg-upack.sh on the $HOME/Downloads folder
#!/usr/bin/env bash
filename="$*"
dirname="${filename/\./_}"
mkdir "$dirname"
# pkgutil --expand "$filename" "$dirname"
xar -xf "$filename" -C "$dirname"
cd "$dirname"/*.pkg
pwd
# tar xvf Payload
cat Payload | gunzip -dc |cpio -i
# cd usr/local/bin
# pwd
# ls -lt
# cp -i * $HOME/Downloads/
Uncomment the last four lines, if you are using a rudix package.
Usage:
cd $HOME/Downloads
chmod +x ./pkg-upack.sh
./pkg-upack.sh MyPackage.pkg
This was tested with the ffmpeg and mawk package from rudix.org (https://rudix.org) search for ffmpeg and mawk packages on this site.
Source : My open source projects : https://sourceforge.net/u/nathan-sr/profile/