I have a folder with 28 gz files with the extension .gz and 28 files with the extension .gz.bam.
I would like to unzip all the 28 .gz files and send them to another folder. I was doing one by one as follows:
gunzip -c file1.gz > /mnt/s3/data_transfer/file1
How can I specify I want the .gz and not the .gz.bam?
This might be what you're looking for:
for f in *.gz; do gunzip -c "$f" > /mnt/s3/data_transfer/"${f%.gz}"; done
You may remove the .gz files by rm *.gz after that, if you want.
Or, alternatively
cp *.gz /mnt/s3/data_transfer/ && cd /mnt/s3/data_transfer && gunzip *.gz
Note that the latter command will gunzip all .gz files in the directory /mnt/s3/data_transfer, including the ones that exist, if any, before the cp command is executed. If you want to remove the original .gz files, replace cp with mv.
First, let's find the files:
find . -type f -name "*.gz" > foo.sh
We find the files whose name ends with gz and save them into foo.sh. Let's open it:
vim foo.sh
Now, we prepend the gunzip command to each line. Let's hit the : key, so at the bottom of the file you can enter a command. Now, let's enter this command:
%s/^/gunzip -c /
this replaces the start of each line with the text we want to prepend.
Then we append the destination path, by pressing : again and pasting the following:
%s/$/ \/some\/path/
of course, instead of \/some\/path you will need to use your own path.
Finally, add
cd /some/path
rename 's/\.gz//' *
to remove all the gz extensions from the files in /some/path (replace this with your actual path)
Note that you may need to install rename and here I assumed that the target path already existed.
Related
I have a bash script I'm trying to write
I have 2 base directories:
./tmp/serve/
./src/
I want to go through all the directories in ./tmp and copy the *.html files into the same folder path in ./src
i.e
if I have a html file in ./tmp/serve/app/components/help/ help.html -->
copy to ./src/app/components/help/ And recursively do this for all subdirectories in ./tmp/
NOTE: the folder structures should exist so just need to copy them only. If it doesn't then hopefully it could create the folder for me (not what I want) but with GIT I can track these folders to manually handle those loose html files.
I got as far as
echo $(find . -name "*.html")\n
But not sure how to actually extract the file path with pwd and do what I need to, maybe it's not a one liner and needs to be done with some vars.
something like
for i in `echo $(find /tmp/ -name "*.html")\n
do
cp -r $i /src/app/components/help/
done
going so far to create the directories would take some more time for me.
I'll try to do it on my own and see if I come up with something
but for argument sake if you do run pwd and get a response the pseudo code for that:
pwd
get response
if that directory does not exist in src create that directory
copy all the original directories contents into the new folder at /src/$newfolder
(possibly running two for loops, one to check the directory tree, and then one to go through each original directory, copying all the html files)
You process substitution to loop the output from your find command and create the destination directory(ies) and then copy the file(s):
#!/bin/bash
# accept first parameters to script as src_dir and dest values or
# simply use default values if no parameter(s) passed
src_dir=${1:-/tmp/serve}
dest=${2-src}
while read -r orig_path ; do
# To replace the first occurrence of a pattern with a given string,
# use ${parameter/pattern/string}
dest_path="${orig_path/tmp\/serve/${dest}}"
# Use dirname to remove the filename from the destination path
# and create the destination directory.
dest_dir=$(dirname "${dest_path}")
mkdir -p "${dest_dir}"
cp "${orig_path}" "${dest_path}"
done < <(find "${src_dir}" -name '*.html')
This script copy .html files from src directory to des directory (create the subdirectory if they do not exist)
Find the files, then remove the src directory name and copy them into the destination directory.
#!/bin/bash
for i in `echo $(find src/ -name "*.html")`
do
file=$(echo $i | sed 's/src\///g')
cp -r --parents $i des
done
Not sure if you must use bash constructs or not, but here is a GNU tar solution (if you use GNU tar), which IMHO is the best way to handle this situation because all the metadata for the files (permissions, etc.) are preserved:
find ./tmp/serve -name '*.html' -type f -print0 | tar --null -T - -c | tar -x -v -C ./src --strip-components=3
This finds all the .html files (-type f) in the ./tmp/serve directory and prints them nul-terminated (-print0), then sends these filenames via stdin to tar as nul-terminated literals (--null) for inclusion (-T -), creating (-c) an archive which is then sent to another tar instance which extracts (-x) the archive printing its contents along the way (optional: -v), changing directory to the destination (-C ./src) before commencing and stripping (--strip-components=3) the ./tmp/serve/ prefix from the files. (You could also cd ./tmp/serve beforehand, using find . instead, and change -C to ../../src.)
I have a folder designed in the following way:
-parentDirectory
---folder1.zip
----item1
-----item1.zip
-----item2.zip
-----item3.zip
---folder2.zip
----item1
-----item1.zip
-----item2.zip
-----item3.zip
---folder3.zip
----item1
-----item1.zip
-----item2.zip
-----item3.zip
I would like to write a bash script that will loop through and unzip the folders and then go into each subdirectory of those folders and unzip the files and name those files a certain way.
I have tried the following
cd parentDirectory
find ./ -name \*.zip -exec unzip {} \;
count=1
for fname in *
do
unzip
mv $fname $attempt{count}.cpp
count=$(($count + 1))
done
I thought the first two lines would go into the parentDirectory folder and unzip all zips in that folder and then the for loop would handle the unzipping and renaming. But instead, it unzipped everything it could and placed it in the parentDirectory. I would like to maintain the same directory structure I have.
Any help would be appreciated
excerpt from man unzip
[-d exdir]
An optional directory to which to extract files. By default, all files and subdirectories are recreated in the current directory; the -d option allows extraction in an arbitrary directory (always assuming one has permission to write to the directory).
It's doing exactly what you told it, and what would happen if you had done the same on the command line. Just tell it where to extract, since you want it to extract there.
see Ubuntu bash script: how to split path by last slash? for an example of splitting the path out of fname.
putting it all together, your command executed in the parentDirectory is
find ./ -name \*.zip -exec unzip {} \;
But you want unzip to extract to the directory where it found the file. I was going to just use backticks on dirname {} but I can't get it to work right, as it either executes on the "{}" literal before find, or never executes.
The easiest workaround was to write my own script for unzip which does it in place.
> cat unzip_in_place
unzip $1 -d `dirname $1`
> find . -name "*.zip" -exec ./unzip_in_place {} \;
You could probably alias unzip to do that automatically, but that is unwise in case you ever use other tools that expect unzip to work as documented.
I have my tar file under:
/volume1/#appstore/SynoDSApps/archiv/DE/2018_08_18__Lysto BackUp.tar.gz
With the tar command:
tar -tf "/volume1/#appstore/SynoDSApps/archiv/DE/2018_08_18__Lysto BackUp.tar.gz"
The command show me:
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/exit_codes/
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/exit_codes/code_FUNC
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/exit_codes/code_SCRI
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/login/
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/login/check_appprivilege.php
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/login/check_login.php
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/login/privilege.php
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/scripte/
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/scripte/Lysto BackUp/
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/scripte/Lysto BackUp/sys
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/webapp/scripte/Lysto BackUp/sys_func
/volume1/02_public/3rd_Party_Apps/SPK_SCRIPTS/SynoDSApps/SSH_ERROR
My Plan or better my wish is to handle it like this:
IFS=$'\n'
for PATHS in $(tar -tPf "/volume1/#appstore/SynoDSApps/archiv/DE/2018_08_18__Lysto BackUp.tar.gz")
do
SED=$(echo "$PATHS" | sed 's/.*\///')
if [[ -n "$SED" ]]
then
tar -C "${target_archiv}" -xvf "/volume1/#appstore/SynoDSApps/archiv/DE/2018_08_18__Lysto BackUp.tar.gz" "$PATHS"
#echo JA
echo "$PATHS"
fi
done
unset IFS
i only want one file of the tar and Store this to a different Directory....
but this command with the -C don´t work... it Extract all the files of the tar....
My Question is, is it possible to extract only one file of the Tar without cd to the Directory ??
Another Question: is it possible to Extract only the files of the tar without the Folders this is maybe the better way but I don´t know how...?
and no I can not tar the files without the paths of it I need them...
so this is no way for me...
I hope for help here :)
If your ultimate goal is to extract files without the full path, you can use a SED-like expression to rename the files while they are extracted, using the --xform option:
tar -C "${target_archiv}" -xvf "/volume1/#appstore/SynoDSApps/archiv/DE/2018_08_18__Lysto BackUp.tar.gz" --xform='s,^.*/,,'
The 's,^.*/,,' expression asks to substitute (s) from the beginning of the filename (^), capture everything (.*) and stop at the last slash (/) then replace it with nothing. In other words, it removes the directory structure from the filenames.
If you want to get rid of the empty folders that have been extracted, you may call this command after extracting:
find "${target_archiv}" -mindepth 1 -maxdepth 1 -type d -exec rmdir {} \;
Keep in mind it will remove all the (empty) subfolders of "${target_archiv}", even the ones that were already here before extracting the tarball. However, because rmdir will not remove directories that contain files, it will be mostly harmless to the subdirectories you had.
i have to copy all the file from source directory to destination directory , but skip all file with extension ".txt" and not to the replace the file if its already present in destination directory
example
source directory
/a/aone.js
/a/atwo.js
/b/bone.txt
/b/btwo.js
destination directory
/a/atwo.js
then it should only copy
/a/aone.js
/b/btwo.js
and skip "/a/atwo.js" because its already present in destination folder
and skip "/b/bone.txt" because its extension is ".txt"
i tried this command but this does not work
find /path/to/source/ \( ! -name "*.txt" \) -type f | cp -n /path/to/destination/ -R
cp -n /path/to/source/*(!*.txt) /path/to/destination/ -R
Assuming you can use rsync, (vaz is verbose, archive and compress - I believe the other options are self explanatory)
rsync -vaz --exclude "*.txt" /path/to/source/ /path/to/destination/
Why make it difficult. You were on the right track. A simple:
cp -an /path/to/source/*.[^t*] /path/to/destination
will copy all files from source, except those whose extension begins with a t to destination. It will do so without overwriting existing files in destination. This presumes that files do not have more than one dot. If so, then a few more lines of code will be needed.
The following will illustrate use of the above:
$ md tmp
$ md a
$ md b
$ touch a/a.{j,k,l,txt}
$ ls -1 a
a.j
a.k
a.l
a.txt
$ cp -an a/a*.[^t*] b
$ ls -1 b
a.j
a.k
a.l
using cp, you must match the proper directory depth. If you have another intervening directory, then simply add an additional wildcard. For example:
$ ls -1 dat/*/*.[^t*]
dat/a/a.j
dat/a/a.k
dat/a/a.l
dat/b/a.j
dat/b/a.k
dat/b/a.l
If your directory structure gets more complex, then go with find or rsync. Both are excellent tools and rsync can handle both local and network transfers. cp is the right tool for small jobs, but when more flexibility is needed, then grab a bigger hammer.
In past I have used following command to archive old files to one file
find . -mtime -1 | xargs tar -cvzf archive.tar
Now suppose we have 20 directories I need to make a script that goes in each directory and archives all files to different files which have same name as original file?
So suppose if I have following files in one directory named /Home/basic/
and this directory has following files:
first_file.txt
second_file.txt
third_file.txt
Now after I am done running the script I need output as follows:
first_file_05112014.tar
second_file_05112014.tar
third_file_05112014.tar
Use:
find . -type f -mtime -1 | xargs -I file tar -cvzf file.tar.gz file
I added .gz to indicate its zipped as well.
From man xargs:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names
read from standard input. Also, unquoted blanks do not terminate input
items; instead the separator is the newline character. Implies -x and -L 1.
The find command will produce a list of filepaths. -L 1 means that each whole line will serve as input to the command.
-I file will assign the filepath to file and then each occurrence of file in the tar command line will be replaced by its value, that is, the filepath.
So, for ex, if find produces a filepath ./somedir/abc.txt, the corresponding tar command will look like:
tar -czvf ./somedir/abc.txt.tar.gz ./somedir/abc.txt
which is what is desired. And this will happen for each filepath.
What about this shell script?
#!/bin/sh
mkdir /tmp/junk #Easy for me to clean up!
for p in `find . -mtime -1 -type f`
do
dir=`dirname "$p"`
file=`basename "$p"`
tar cvf /tmp/junk/${file}.tar $p
done
It uses the basename command to extract the name of the file and the dirname command to extract the name of the directory. I don't actually use the directory but I left it in there in case you might find it handy.
I put all the tar files in one place so I could delete them easily but you could easily substitute $P instead of $file if you wanted them in the same directory.