Linux bash script to copy files by list - bash

I'm new in bash and I need a help please. I have a file call list.txt containing pattern like
1210
1415
1817
What I want to do is to write a bash script which will copy all file in my current directory which name contain that pattern towards a new directory called toto.
Example of my file in the current directory :
1210_ammm.txt
1415_xdffmslk.txt
1817_lsmqlkksk.txt
201247_kksjdjdjd.txt
The goal is to copy 1210_ammm.txt, 1415_xdffmslk.txt, 1817_lsmqlkksk.txt to toto.
Transferred from an 'answer'.
My list.txt and toto directory are in my current directory. That is what I try
#!/bin/bash
while read p; do # read my list file
for i in `find -name $p -type f` # find all file match the pattern
do
cp $i toto # copy all files find into toto
done
done < partB.txt
I don't have an error but it doesn't do the job.

Here is what you need to implement :
read tokens from an input file
for each token
search the files whose name contain said token
for each file found
copy it to toto
To read tokens from the input file, you can use a read command in a while loop (and the Bash FAQ generally, and Bash FAQ 24 specifically.
To search files whose name contain a string, you can use a for loop and globbing. For example, for file in ./*test*; do echo $file; done will print the name of the files in the current directory which contain test.
To copy a file, use cp.
You can check this ideone sample for a working implementation.

Use below script:
cp "$(ls | grep -f list.txt)" toto
ls | grep -f list.txt will grep for the pattern found in list.txt in the ls output.
cp copies the matched files to toto directory.
NOTE: If list.txt and toto are not in current directory, provide absolute paths in the script.

I needed this too, I tried #Zaziln's answer, but it gave me errors. I just found a better answer. I think others will be interested too.
mapfile -t files < test1.txt
cp -- "${files[#]}" Folder/
I found it on this post --> https://unix.stackexchange.com/questions/106219/copy-files-from-a-list-to-a-folder#106231

Related

Rename files in bash based on content inside

I have a directory which has 70000 xml files in it. Each file has a tag which looks something like this, for the sake of simplicity:
<ns2:apple>, <ns2:orange>, <ns2:grapes>, <ns2:melon>. Each file has only one fruit tag, i.e. there cannot be both apple and orange in the same file.
I would like rename every file (add "1_" before the beginning of each filename) which has one of: <ns2:apple>, <ns2:orange>, <ns2:melon> inside of it.
I can find such files with egrep:
egrep -r '<ns2:apple>|<ns2:orange>|<ns2:melon>'
So how would it look as a bash script, which I can then user as a cron job?
P.S. Sorry I don't have any bash script draft, I have very little experience with it and the time is of the essence right now.
This may be done with this script:
#!/bin/sh
find /path/to/directory/with/xml -type f | while read f; do
grep -q -E '<ns2:apple>|<ns2:orange>|<ns2:melon>' "$f" && mv "$f" "1_${f}"
done
But it will rescan the directory each time it runs and append 1_ to each file containing one of your tags. This means a lot of excess IO and files with certain tags will be getting 1_ prefix each run, resulting in names like 1_1_1_1_file.xml.
Probably you should think more on design, e.g. move processed files to two directories based on whether file has certain tags or not:
#!/bin/sh
# create output dirs
mkdir -p /path/to/directory/with/xml/with_tags/ /path/to/directory/with/xml/without_tags/
find /path/to/directory/with/xml -maxdepth 1 -mindepth 1 -type f | while read f; do
if grep -q -E '<ns2:apple>|<ns2:orange>|<ns2:melon>'; then
mv "$f" /path/to/directory/with/xml/with_tags/
else
mv "$f" /path/to/directory/with/xml/without_tags/
fi
done
Run this command as a dry run, then remove --dry_run to actually rename the files:
grep -Pl '(<ns2:apple>|<ns2:orange>|<ns2:melon>)' *.xml | xargs rename --dry-run 's/^/1_/'
The command-line utility rename comes in many flavors. Most of them should work for this task. I used the rename version 1.601 by Aristotle Pagaltzis. To install rename, simply download its Perl script and place into $PATH. Or install rename using conda, like so:
conda install rename
Here, grep uses the following options:
-P : Use Perl regexes.
-l : Suppress normal output; instead print the name of each input file from which output would normally have been printed.
SEE ALSO:
grep manual

How do I extract a file name into 2 parts, making one into directory and the other one inside of it?

I'm trying to sort all mp3 files by artist and name. At the moment, they're in 1 giant file name.
E.g Artist - Song name.mp3
I want to convert this to
Artist/Song name.mp3
Here's what I've tried so far. In this case, I was using a test file named "hi\ -\ hey":
#!/bin/bash
# filters by all .mp3 extensions in the current working directory
for f in *.mp3; do
# extract artist and song name and remove spaces
artist=${f% -*}
song=${f#*- }
#make directory with the extracted artist name and move + rename the file into the directory
mkdir -p $artist
mv $f $artist/$song;
done
For some reason, it's creating a directory with the song name instead of the artist in addition to a load of errors:
mv: cannot move 'hey.mp3' to a subdirectory of itself, 'hey.mp3/hey.mp3'
mv: cannot stat 'hi': No such file or directory
mv: cannot stat '-': No such file or directory
mv: 'hey.mp3/hi' and 'hey.mp3/hi' are the same file
mv: cannot stat '-': No such file or directory
By far the simplest way of doing this is to use rename a.k.a. Perl rename.
Basically, you want to replace the sequence SPACE-DASH-SPACE with a forward slash directory separator, so the command is:
rename --dry-run -p 's| - |/|' *mp3
Sample Output
'Artist - Song name.mp3' would be renamed to 'Artist/Song name.mp3'
'Artist B - Song name 2.mp3' would be renamed to 'Artist B/Song name 2.mp3'
If that looks correct, just remove --dry-run and run it again for real. The benefits of using rename are:
it can do a dry-run to test before you run for real
it will create all necessary directories with the -p option
it will not clobber (overwrite) files without warning
you have the full power of Perl available to you and can make your renaming as sophisticated as you wish.
Note that you can install on macOS with homebrew:
brew install rename
Just in case you don't have the rename utility. A fix on your original script.
for f in *.mp3; do
# extract artist and song name and remove spaces
artist=${f%% -*}
song=${f##*- }
#make directory with the extracted artist name and move + rename the file into the directory
echo mkdir -p -- "$artist" && echo mv -- "$f" "$artist/$song"
done
Remove the echo If you're satisfied with the output.
Assuming there are many files, it's probably much faster to do this using pipes instead of a for loop. This has the additional advantage of avoiding complicated bash-specific syntax and using core unix/linux command line programs instead.
find *-*.mp3 |
sed 's,\([^-]\+\)\s*-\s*\(.*\),mkdir -p "\1"; mv "&" "\1"/"\2",' |
bash
Explanation:
This find to find all the files matching -.mp3 in the current directory.
This sed command changes each line to a command string, e.g.:
aaa - bbb.mp3
->
mkdir -p "aaa"; mv "aaa - bbb.mp3" "aaa"/"bbb.mp3"
The bash command runs each of those command strings.
you can try this.
#!/usr/local/bin/bash
for f in *.mp3
do
artist=`echo $f | awk '{print $1}' FS=-`
song=`echo $f | awk '{print $2}' FS=-`
mkdir -p $artist
mv $artist-$song $song
mv $song ./$artist
done
here I am using two variable artist and song. as your test file name is "hi\ -\ hey" so I change the awk delimiter to "-" to store variable according to it.
we don't need to use awk..by using bash parameter expansion.... it is working.
#!/usr/local/bin/bash
for f in *.mp3
do
artist=`echo ${f%-*}`
song=`echo ${f#*-}`
mkdir -p $artist
mv $artist-$song $song
mv $song ./$artist
done

Sort text files in bash

I have a number of text files in the current directory. Each file contains data about a movie.The content of each file will be according to the following format:
Movie name
Storyline
Director name
Year of release
How can I organize the files according to the name of the directors by a shell script. Movies made by same director will get moved to a folder named after the director. How can I do this by writing shell script?
The rough solution will be
for file in *; do
director=$( some code here depending on requirements clarification )
mkdir -p "$director" # quotes are crucial!
# -p suppresses errors if dir already exists.
mv -v "$file" "$director"
done
Assuming that the filename of the movie is always the first line, and the Director name is always the next to last. You could do it something similar to this:
#!/bin/bash
MOVIES_DIR=/path/to/your/movies
for FILE in ${MOVIES_DIR}/*.txt; do
# Movie name will always be the first line
MOVIE=$(sed '1q;d' "$FILE")
# Director will always be the next to last line
DIRECTOR=$(tac "$FILE" | sed '2q;d')
# Make the director folder
mkdir -p "${MOVIES_DIR}/${DIRECTOR}"
# Find inside the `MOVIES_DIR` files with the movie name with any extension
# That is NOT .txt and moves them to the proper director folder.
find "${MOVIES_DIR}" -maxdepth 1 -name "${MOVIE}.*" -not -name '*.txt' -print0 |
while IFS= read -r -d '' NAME; do
FILENAME=$(printf '%s\n' "$NAME")
mv "$FILENAME" "${MOVIES_DIR}/${DIRECTOR}"
done
done
Edit:
It was my misunderstanding that you wanted to move movies with the same name in the text file into the appropriate director folder. I now see you are intending to move the text files themselves. I downloaded all the text files from the link you provided, and it successfully sorted all of them into the appropriate folder:
#!/bin/bash
MOVIES_DIR=/path/to/your/files
for FILE in ${MOVIES_DIR}/*.txt; do
# I thought the description was on one line, but it looks like it can span
# multiple. So get the next to last line instead, since year is only one line. I also changed this in the first script.
DIRECTOR=$(tac "$FILE" | sed '2q;d')
DIRECTOR_PATH="$MOVIES_DIR/$DIRECTOR"
mkdir -p "$DIRECTOR_PATH"
# Move the current text file to the Director folder.
mv "$FILE" "$DIRECTOR_PATH"
done
The first script should sort all movies into the director folders, assuming that the text files are in the same directory as the movies.
The second script should sort all text files into the appropriate director folders.

About: extracting *.gz files and move a original file to other folder

I am almost new on shell script but don't know some commands.
I am trying to write below shell script , please give some direction.
1. Read *.gz files from specific directory
2. Extract it to other folder
3. Move a original file to another folder.
i can do it three separate shell scripts but i want it include one shell script. Then this script will be cronjob and will run every 5 minutes.
i was trying to start like below but somehow i am bit confused how to get filelist. I can do here another script but want to include in one script."
#!/bin/bash
while IFS= read file; do
gzip -c "$file" > "zipdir/$(basename "$file").gz"
done < filelist
-----------------------------------------
PS: Files are created in every 5 minutes.
There are several ways to implement what you're looking for (I would consider notify). Anyhow... this is a very simple implementation:
$ source=~/tmp/source # directory where .gz files will be created
$ target=~/tmp/target # target directory for uncompressed files
$ archive=~/tmp/archive # archive dir for .gz files
$ shopt -s nullglob # avoid retiring unexpanded paths
$ for gz in ${source}/*.gz ; do gzip -dc "$gz" > ${target}/$(basename "$gz" .gz) ; mv "$gz" ${archive}/ ; done
$ shopt -u nullglob # reset nullglob
If you know for sure "source" directory will always contain .gz files you can avoid shopt.
Another solution (not requiring shopt) is this:
find ${source} -name '*.gz' -print0 | while read -d '' -r gz; do
gzip -dc "$gz" > ${target}/$(basename "$gz" .gz)
mv "$gz" ${archive}/
done
The first line looks a little bit complicated because it manages source file names containing spaces...

Rename files in shell

I've folder and file structure like
Folder/1/fileNameOne.ext
Folder/2/fileNameTwo.ext
Folder/3/fileNameThree.ext
...
How can I rename the files such that the output becomes
Folder/1_fileNameOne.ext
Folder/2_fileNameTwo.ext
Folder/3_fileNameThree.ext
...
How can this be achieved in linux shell?
How many different ways do you want to do it?
If the names contain no spaces or newlines or other problematic characters, and the intermediate directories are always single digits, and if you have the list of the files to be renamed in a file file.list with one name per line, then one of many possible ways to do the renaming is:
sed 's%\(.*\)/\([0-9]\)/\(.*\)%mv \1/\2/\3 \1/\2_\3%' file.list | sh -x
You'd avoid running the command through the shell until you're sure it will do what you want; just look at the generated script until its right.
There is also a command called rename — unfortunately, there are several implementations, not all equally powerful. If you've got the one based on Perl (using a Perl regex to map the old name to the new name) you'd be able to use:
rename 's%/(\d)/%/${1}_%' $(< file.list)
Use a loop as follows:
while IFS= read -d $'\0' -r line
do
mv "$line" "${line%/*}_${line##*/}"
done < <(find Folder -type f -print0)
This method handle spaces, newlines and other special characters in the file names and the intermediate directories don't necessarily have to be single digits.
This may work if the name is always the same, ie "file":
for i in {1..3};
do
mv $i/file ${i}_file
done
If you have more dirs on a number range, change {1..3} for {x..y}.
I use ${i}_file instead of $i_file because it would consider $i_file a variable of name i_file, while we just want i to be the variable and file and text attached to it.
This solution from AskUbuntu worked for me.
Here is a bash script that does that:
Note: This script does not work if any of the file names contain spaces.
#! /bin/bash
# Only go through the directories in the current directory.
for dir in $(find ./ -type d)
do
# Remove the first two characters.
# Initially, $dir = "./directory_name".
# After this step, $dir = "directory_name".
dir="${dir:2}"
# Skip if $dir is empty. Only happens when $dir = "./" initially.
if [ ! $dir ]
then
continue
fi
# Go through all the files in the directory.
for file in $(ls -d $dir/*)
do
# Replace / with _
# For example, if $file = "dir/filename", then $new_file = "dir_filename"
# where $dir = dir
new_file="${file/\//_}"
# Move the file.
mv $file $new_file
done
# Remove the directory.
rm -rf $dir
done
Copy-paste the script in a file.
Make it executable using
chmod +x file_name
Move the script to the destination directory. In your case this should be inside Folder/.
Run the script using ./file_name.

Resources