create tar and save it to stdout - shell

I am trying to create tar from a file, which contains list of other files and saving it to stdout.
let suppose there is a file called "files-to-create" which has path of other files like /home/abc.txt /home/def.txt and I want to create tar of abc.txt,def.txt.
my script contains:
exec 100>&1
tar cf - -T files-to-sync >&100
and I am calling the script and saving it to some other file like:
/script.sh > final_tar.tar
But while creating the tar I am getting error, can somebody help me out?

You can use the following script to reach your goal, let me know if something is unclear:
Prototype 1:
$ cat scriptTar.sh
#!/bin/bash
readonly HELP="$(basename "$0") <list_of_files> <output_tar>
this script will generate a tar file composed of all files present in <list_of_files> input file
the output tar file will be saved as <output_tar>
to run the script provide the input and output filenames"
readonly INPUT_LIST_FILE=$1
readonly OUTPUT_TAR_FILE=$2
if [ -z "$INPUT_LIST_FILE" -o -z "$OUTPUT_TAR_FILE" ]
then
echo $HELP;
exit 1;
fi
tar cf - -T $INPUT_LIST_FILE > $OUTPUT_TAR_FILE
exit $?
Folder content:
$ tree .
.
├── a
│   └── abc.txt
├── b
│   └── def.txt
├── c
│   └── ghj.txt
├── files-to-sync.in
└── scriptTar.sh
3 directories, 5 files
List file content:
$ cat files-to-sync.in
./a/abc.txt
./b/def.txt
./c/ghj.txt
Execution:
$ ./scriptTar.sh files-to-sync.in output.tar
tar file content:
$ tar -tvf output.tar
-rw-rw-r-- arobert/arobert 4 2018-02-22 16:50 ./a/abc.txt
-rw-rw-r-- arobert/arobert 4 2018-02-22 16:50 ./b/def.txt
-rw-rw-r-- arobert/arobert 4 2018-02-22 16:50 ./c/ghj.txt
Or use the following script if you really want to display it on stdout:
Prototype 2 via ssh:
#!/bin/bash
readonly HELP="ERROR: $(basename "$0") <list_of_files>
this script will generate to stdout a tar file composed of all files present in <list_of_files> input file
to run the script provide the input file and redirect the output to a file"
readonly INPUT_LIST_FILE=$1
if [ -z "$INPUT_LIST_FILE" ]
then
echo $HELP;
exit 1;
fi
tar cf - -T $INPUT_LIST_FILE
Execution via ssh:
$ ssh user#localhost "cd /home/user/test_tar/; ./scriptTar.sh files-to-sync.in" > output.tar
user#localhost's password:
Content of the tar generated:
tar -tf output.tar
./a/abc.txt
./b/def.txt
./c/ghj.txt
extracting the content:
tar xvf output.tar
./a/abc.txt
./b/def.txt
./c/ghj.txt
checking the files:
more ?/*.txt
::::::::::::::
a/abc.txt
::::::::::::::
abc
::::::::::::::
b/def.txt
::::::::::::::
abc
::::::::::::::
c/ghj.txt
::::::
However if I were you, I would not only generate a tar file but add some compression (tar.gz) and transfer the file with rsync to be able to restart the download from the point where it stopped in case of transfer error.

So the proper solution is
Case1: If you are passing the list of file as an argument
you can use this:
files-to-sync=$1
tar cf - -T files-to-sync
Case2: If you want to use absolute path for the list of file
you can use this:
tar cfP - -T /path/to/the/file
use -P in case of absolute path.

Related

How to delete '$'\n' characters appended to ldd list

I am trying to copy binary dependencies to other folder, so I use ldd to see what should be copied.
However the script fails when copying. It "appears" '$'\n' characters when binary dependencies are copied.
Something is wrong, but I dont know what. Tried running command per command and cant see the fault.
What is the trouble here ?
Thanks
Script code
#!/usr/bin/env bash
set -e
chr=/home/myjail
cmds=( bash echo ls rm )
mkdir -p "$chr"/{bin,lib,lib64}
# copy commands
for app in "${cmds[#]}"; do
echo 'Added command:'"$app"
cp -v /bin/"$app" "$chr/bin"
done
# copy deps
for app in "${cmds[#]}";do
echo 'deps for:'"$app"
deps="$(ldd /bin/"$app" | egrep -o '/lib.*\.[0-9]')"
for curdep in "${deps[#]}";do
echo "$curdep"
cp -v --parents "$curdep" "$chr"
done
done
Script output
furby#debian-haptic20:~# ./depcopy.sh
Added command:bash
'/bin/bash' -> '/var/lib/haproxy/bin/bash'
Added command:echo
'/bin/echo' -> '/var/lib/haproxy/bin/echo'
Added command:ls
'/bin/ls' -> '/var/lib/haproxy/bin/ls'
Added command:mysql
'/bin/mysql' -> '/var/lib/haproxy/bin/mysql'
deps for:bash
/lib/x86_64-linux-gnu/libtinfo.so.6
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libc.so.6
/lib64/ld-linux-x86-64.so.2
cp: failed to get attributes of '/lib/x86_64-linux-gnu/libtinfo.so.6'$'\n': No such file or directory
I'm using this script for this purpose:
copydep.sh
#!/bin/bash
if [ ${#} != 2 ]
then
echo "usage $0 PATH_TO_BINARY target_folder"
exit 1
fi
path_to_binary="$1"
target_folder="$2"
# if we cannot find the the binary we have to abort
if [ ! -f "${path_to_binary}" ]
then
echo "The file '${path_to_binary}' was not found. Aborting!"
exit 1
fi
# copy the binary itself
echo "---> copy binary itself"
cp --parents -v "${path_to_binary}" "${target_folder}"
# copy the library dependencies
echo "---> copy libraries"
ldd "${path_to_binary}" | awk -F'[> ]' '{print $(NF-1)}' | while read -r lib
do
[ -f "$lib" ] && cp -v --parents "$lib" "${target_folder}"
done
Run it like this:
$ mkdir /tmp/test
$ bash copydep.sh /bin/ls /tmp/test
---> copy binary itself
/bin -> /tmp/test/bin
'/bin/ls' -> '/tmp/test/bin/ls'
---> copy libraries
/lib -> /tmp/test/lib
/lib/x86_64-linux-gnu -> /tmp/test/lib/x86_64-linux-gnu
'/lib/x86_64-linux-gnu/libselinux.so.1' -> '/tmp/test/lib/x86_64-linux-gnu/libselinux.so.1'
'/lib/x86_64-linux-gnu/libc.so.6' -> '/tmp/test/lib/x86_64-linux-gnu/libc.so.6'
'/lib/x86_64-linux-gnu/libpcre.so.3' -> '/tmp/test/lib/x86_64-linux-gnu/libpcre.so.3'
'/lib/x86_64-linux-gnu/libdl.so.2' -> '/tmp/test/lib/x86_64-linux-gnu/libdl.so.2'
/lib64 -> /tmp/test/lib64
'/lib64/ld-linux-x86-64.so.2' -> '/tmp/test/lib64/ld-linux-x86-64.so.2'
'/lib/x86_64-linux-gnu/libpthread.so.0' -> '/tmp/test/lib/x86_64-linux-gnu/libpthread.so.0'
# Result:
$ tree /tmp/test/
/tmp/test/
├── bin
│   └── ls
├── lib
│   └── x86_64-linux-gnu
│   ├── libc.so.6
│   ├── libdl.so.2
│   ├── libpcre.so.3
│   ├── libpthread.so.0
│   └── libselinux.so.1
└── lib64
└── ld-linux-x86-64.so.2
4 directories, 7 files
Your deps variable should be an array, created with mapfile and some redirection of process substitution:
mapfile -t deps < <(ldd /bin/"$app" | grep -o '/lib.*\.[0-9]')
As you have it, it's treated as a single string, which as you've seen, doesn't work.

How to tar all files.txt in directory

I have installed a bunch of packages through chimp from umVIRTLFS.
It creates a files.txt for each installed package in the directory /var/log/ulfs-package/package_name/ and I want to create a tarball containing the files listed in the files.txt.
So I want to run the command tar -zvcf package_name.tar.gz -T files.txt for each folder inside /var/log/ulfs-package.
Assuming all files.txt files are contained within a corresponding package_name directory:
ULFS=/var/log/ulfs-package
for package_name in ${ULFS}/*
do
if [ -d ${ULFS}/${package_name} ] && [ -f ${ULFS}/${package_name}/files.txt ]
then
tar -zvcf ${package_name}.tar.gz -T ${ULFS}/${package_name}/files.txt
fi
done

Bash autocompletion in user input, Makefile

I would like to setup a makefile that reads user input with Bash autocompletion, and based on that create a directory where some files would be copied into. Basically something like
$ make initializeProject
Enter destination: Dir<ectory>/Sub<directory>/ProjectDirectory
mkdir: created directory 'Directory/Subdirectory/ProjectDirectory'
'Templates/TemplateFile' -> 'Directory/Subdirectory/ProjectDirectory/TemplateFile'
where < X > stands for autocompleted parts.
As of now my makefile does not autocomplete, and it looks like this:
initProjectLaTeX:
#read -p "Enter destination: " destination; \
mkdir -pv $$destination; \
cp -iv ~/.templates/latex/* $$destination
I have tried this option, but it produces the error /bin/sh: 1: read: Illegal option -e.
How can I access Bash autocompletion here?
make uses /bin/sh as default shell and on many systems it's not the same as bash (e.g. on Debian is dash). I suspect that the -e option of read is a bash extension, that is why you get the error:
/bin/sh: 1: read: Illegal option -e
setting /bin/bash as the SHELL in the Makefile should work:
Makefile
SHELL := /bin/bash
initProjectLaTeX:
#read -e -p "Enter destination: " destination; \
mkdir -pv $$destination; \
cp -iv ~/tmp/*c $$destination
example
$ make initProjectLaTeX
Enter destination: foo/bar/baz
mkdir: created directory 'foo'
mkdir: created directory 'foo/bar'
mkdir: created directory 'foo/bar/baz'
'/home/marco/tmp/foo.c' -> 'foo/bar/baz/foo.c'
$ tree foo
foo
└── bar
└── baz
└── foo.c
2 directories, 1 file
$
completion
$ make initProjectLaTeX
Enter destination: foo
foo/ foo.c
Enter destination: foo
foo/ foo.c
Enter destination: foo
Rather than have make interact with the user, pass the desired destination to make.
Start with a modified recipe:
initProjectLaTeX:
mkdir -pn $(DESTINATION)
cp -iv ~/.templates/latex/* $(DESTINATION)
then run
make initProjectLaTeX DESTINATION=Dir<ectory>/Sub<directory>/ProjectDirectory
(bash allows completion for the value of an assignment.)

Executing unzip command from Groovy

I'm trying to run the unzip shell command from within Groovy.
I'm running the command
"unzip ~/Documents/myFile.txt.zip -d ~/Documents/".execute()
but it's not working. When I copy the exact command into my terminal it works. How can I do this from groovy?
There's no ~ as far as Groovy is concerned; use an actual path.
groovy:000> p = "ls -CF /Users/Dave".execute()
===> java.lang.UNIXProcess#2603826d
groovy:000> p.waitFor()
===> 0
groovy:000> p.in.text
===> Desktop/ Movies/ bin/
Documents/ Music/ node_modules/
Downloads/ Pictures/
Dropbox/ Public/
Library/ ScreenshotOnFail/
You can always use System.getProperty("user.home"), e.g.,
p = "ls -CF ${System.getProperty('user.home')}".execute()

How to extract nested tar.gz files easily?

I need to extract tar.gz a file. It's about 950mb. It has another 23 tar.gz files in it. Each of those 23 tar.gz files has one tar file in them. My questions is how I can easily extract all of them? Is there a commandline tool that I can use?
The structure is like the following:
foo.tar.gz
├───bar1.tar.gz
│ ├───foobar1.tar
├───bar2.tar.gz
│ ├───foobar2.tar
├───bar3.tar.gz
│ ├───foobar3.tar
├───bar4.tar.gz
│ ├───foobar4.tar
├───bar5.tar.gz
│ ├───foobar5.tar
├───bar6.tar.gz
│ ├───foobar6.tar
| ..........
| ..........
| ..........
| 23 of them
Thanks in advance.
You can use the --to-command argument to pass each extracted file to another program (on stdin). In this case, you pass it to another tar instance reading data from stdin.
tar --to-command='tar -xzvf -' -xzvf foo.tar.gz
I ended up manually extracted foo.tar.gz and using the following shell script to extract those bar*.tar.gz files.
#!/bin/bash
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:~/bin
export PATH
for next in *.tar.gz
do
echo "Untaring - $next"
tar -xzf $next -C ~/foo
done
exit 0
hope this will help someone.
Yup.
tar -xzOf foo.tar.gz bar1.tar.gz | tar -xO foobar1.tar
Should do the trick.

Resources