How to create multiple files in each directories and then compress it through tar (BASH) - bash

What I am currently struggling is to create multiple files and storing it into each directory.
I have made 100 directories like this:
mkdir ./dir.{01..100}
What I want is to create 100 text files for each directory. So that the result will show like:
click on dir.01 at home dir, which has files named: 01.txt to 100.txt
Also, I want to compress all 100 directories, each containing 100 text files into one using tar.
I am struggling with:
creating 100 text files each in 100 directories
using tar to zip all 100 directories together.
I am more interested in making creating 100 text files IN 100 directories. Also I am MUCH MORE interested in how to use tar to join all 100 directories together in specific file (fdtar) for instance.

If you are fine with empty files,
touch ./dir.{01..100}/{01..100}.txt
If you need each file to contain something, use that as the driver in a loop:
for file in ./dir.{01..100}/{01..100}.txt; do
printf "This is the file %s\n" "$file">"$file"
done
This could bump into ARG_MAX ("argument list too long") on some platforms, but it works fine on MacOS and should work fine on any reasonably standard Linux.
Splitting the loop into an inner and an outer loop could work around that problem:
for dir in ./dir.{01..100}; do
for file in {01..100}.txt; do
printf "This is file %s/%s\n" >"$dir/$file"
done
done

If I understand you need two things. First, you have 100 directories and need to create a file in each. With a for loop in bash run from the parent directory where all other directories you have created are:
for n in dir.*
do
f=`echo $n | sed s/dir\.//`
echo "This is file $n" >"$n/$f.txt"
done
Regarding tar that is even easier because tar will take multiple directories and glue them together. From the parent directory try:
tar cvf fd.tar dir.*
The c option will create the archive. v will tell tar to print all it is doing so you know what is happening. f directories.tar will create the archive with that name.
When you undo the tar operation, you will use:
tar xvf fd.tar
In this case x will extract the contents of the tar archive and will create all 100 directories for you at the directory from which you invoke it.
Note that I have used fd.tar and not fdtar as the .tar extension is the customary way to signal that the file is a tar archive.

Related

"for loop" to run rsync on relevant folders only

I have a folder contains around 650 folders (source), I generated a list of the relevant folders i want (final.txt).
i am trying to use a "for loop" to copy only the relevant sub-folders to a new location (target).
i keep getting the original content of the "source" copied to the "target".
i run:
for var in `cat final.txt` ; do rsync -ah $var source/ target/ ; done
I tried different syntax but can't seem to get what I need.
what am i doing wrong?
I expect to copy only the folders which name is in the final.txt list copied to the target (all "names" in the file are a single word, matching to some of the folder names for exactly)
ok after messing around I should have ran this (it works)
for var in `cat final.txt` ; do rsync -ah source/$var target/ ; done

Mac OS - Batch Rename All Files in Folder but Disregard All SubFolders

I have a bunch of folders that I would like to rename all the files contained within minus any subdfolders.
For example lets say I have two parent folders:
ParentFolder1 - [PF1]
ParentFolder2 - [PF2]
Each parent folder has various amounts of subfolders:
SubParentFolder1_1
SubParentFolder1_2
SubParentFolder2_1
Inside the ParentFolder and each SubParentFolder there can be files such as .mp3, .txt. etc. or more subfolders.
How would I go about renaming all and any files in this manner:
example.mp3 -> example - [PF1]
example.txt -> example - [PF2]
example.docx -> example - [PF2]
Appreciate any input!
This is a way to list files (not folders) in a range of directories and then do something with them... Specifics of renaming are up to you.
for FOLD in Parent*;
do for FILE in $(ls -p $FOLD | grep -v "/");
do echo "$FOLD/$FILE" "$FOLD/${FILE%.*}";
done; done;
Explanation:
For each folder (FOLD) in directories matching the wildcard Parent*,
list the contents, adding / to the end of directory names.
Do inverse grep on that list, leaving just the file names.
Loop through each FILE and echo out the original folder+file, followed by the folder and file with the suffix removed by patten matching.
Once you test this, you can replace echo with mv to do the actual renaming... (I've put these on separate lines to make them more readable, but would usually run this as one long command.

When compressing files (zip, tar, ect...) in SSH what determines the 'sort order' in which files are compressed?

Consider the following command run on a folder with 2TB of recursive folders and files in it.
tar -cvzf _backup.tar.gz /home/wwwsite/public_html
Consider that the folder being compressed is full of sub-folders (with hundreds of sub folders and files in them) and a naming convention that is random, but sequential, short example:
/17688EE/
/18503HH/
/19600SL/
/20659CS/
Consider that there are 10,000+ folders between each block (17000 block, then 18000 block, ect...). Naming Convention: Number 00000 + Letter A-Z, (ie: 17000AZ-17000ZA) so the folders can easily be sorted by name.
Consider that the tar command is being run in a screen with verbose output in order to check on the "progress" of that command.
screen -S compress
In theory, I had assumed I could simply look at the output of that screen, but I notice that the TARBALL does not seem to be compressing the folder in either the order they were created, nor sort them based on the name of the folder.
Therefore my question is two fold:
Other than looking at the verbose output of the TARBALL and guessing;
Is there any where to find out how long the compression process will take to complete? (such as adding a -tack command onto the TAR to show estimated time to completion, something similar to the % complete of an SCP command)
In what order does the TAR command decide to compress the folders? ( and is there a way to tell the command to "sort by" date/name during compression?)
To elaborate, after 20 min of waiting for the 17001AA-to-AZ block to compress I would figure next up would be the 17001BA-to-BZ block, but this is not the case, the verbose output shows what seem to be randomly grabbing folders without sorting by name nor date)
Simply put: What determines the sort order during compression?
If you give tar a list of directory names, the order of the entries in the tar file will match the order that readdir returns filenames from the filesystem. The fact that you are compressing the tar file has no bearing on the order.
Here is a quick example to illustrate what happens on a Linux ext4 filesystem. Other filesystems may behave differently.
First create a new directory with three files, a1, a2 and a3
$ mkdir fred
$ cd fred
$ touch a1 a2 a3
Now lets see the order that readdir returns the files. The -U option will make ls return the filenames unsorted in the order they are stored in the directory.
$ ls -U
a3 a1 a2
As you can see, on my Linux setup the files are returned in an apparently random order.
Now stick the files in a tar file. Note I'm giving tar a directory name for the input file ("." in this instance) to make sure it has to call readdir behind the scenes.
$ tar cf xxx.tar .
And finally, lets see the order that tar has stored the files.
$ tar tf xxx.tar
./
./a3
./a1
./a2
The order of the files a1, a2 and a3 matches the order that readdir returned the filenames from the filesystem. The . filename is present because it was explicitly included on the command line passed to tar.
If you want to force an order you will have to give tar a sorted list of filenames. The example below shows how to get tar to read the list of filenames from stdin, using the -T - command line option.
$ ls a* | tar cvf yyy.tar -T -
a1
a2
a3
In this toy example the list of filenames will be automatically sorted because the shell sorts the filenames that match the wildcard a*.
And just to confirm, this is what is in the tar file.
$ tar tf yyy.tar
a1
a2
a3
In your use-case a combination of the find and sort commands piped into tar should allow you to create a sorted tar file with as many entries as you like.
Something like this as a starting point.
find | sort | tar -cvzf _backup.tar.gz -T -

I want to move 100 out of 1Million files of a particular extension to a target folder in linux

I have a folder(/DIR/SRC) containing 1 Million files with different types of extensions.
I would like to pick a particular extension files from the SRC folder, may be 1000 at a time and move to a target folder which is at /DIR/TGT.
The file name has the format 1234_XXXX-XXXX_TIMESTAMP.ext
Try this one:
mv `ls -1 *.your_ext|tail -n 100` /TARGETDIR
I didn't test it, but in this way it should work.
If you want the latest 100 or alphabetically the first 100 (or, or, or,...) you can change the ls-command.
This will move all files with .your_extension in current folder to your new path
mv *.your_ext your/new/path

Bash Script to Extract split archive files using rar

Here's what I want to do:
I have thousand of split rar archives on folder name Archives.
Name of the files 0001.part1.rar 0002.part2.rar 0003.part3.rar etc.
read 0001.part1.rar
create a directory based on prefix of the file above e.g. 0001
move all files with the same prefix on the directory created above.
Extract the files within that directory
delete all rar files in that directory
Rename the extracted file based on a list of names from a text file.
Rar the renamed file with different arguments.
Move the renamed file (from step 6) to a new directory called Done.
Proceed to file 0002.part1.rar then do steps 2-8 and so forth.
Additional how would I incorporate it with cron???
This should be run once only...
After extracting the first set of rar's files change to:
file.001
file.002
file.003
etc. which I need to extract too.
Clarification on Step 6:
After extracting the second set of rar's (file.001 , file.002 etc.)
I want to rename it based on a list of names from a text file.
e.g. List of files from a text file:
0001 - GB Funds.DAT
0002 - US Bonds.DAT
0003 - SG Securities.DAT
0004 - LU Credits.DAT
Clarification on Step 7:
After renaming the file I want to move it on a new folder called "Done"
Clarification on Step 9:
Go back to the main folder with all the other archives
and continue extracting the next set of archives and
do the same steps from 1 to 8.
You can write a shell script including something like this:
# foo.sh
set -e
set -u
for i in `find -max-depth 1 -type f -name '*.rar' | sed 's/\.part*\.rar$//' | sort -u`; do
mkdir $i
mv $i.part*rar $i
cd $i
unrar x $i.part1.rar
DST=`grep $i ../rename.txt | sed 's/^[0-9]\+ - //'`
mv $i "$DST"
# and so on, rar it together again, move it done directory etc.
cd ..
done
Run it then via:
bash foo.sh
You have to clarify 6./8./9.
I don't know why do you want to run it via cron, since you only want to run it once. at is designed for running one-time jobs, or run it in a screen session.
I suggest that you do a few tests with 1-3 files from your collection and the script you end up with, before starting the whole job.

Resources