How do I create a script in a command prompt (or shell script in a bash) to divide files depending on a number, say X, and put into individual folder.
Example: I have 10 files and the number X is 4 (I can set it inside the script). So, the system will create 3 folders (1st folder contain 4 files, 2nd folder contain 4 files and the last folder will contain the remaining 2 files) after running the script.
Regarding about the divide of the files. It can be either go by the date or the filename.
Example: Suppose the 10 files above is a.txt, aa.txt, b.txt, cd.txt, ef.txt, g.txt, h.txt, iii.txt, j.txt and zzz.txt. After running the script, it will create the 3 folders such that the 1st folder contain a.txt, aa.txt, b.txt, cd.txt, the 2nd folder contains ef.txt, g.txt, h.txt, iii.txt and the last folder will contain the remaining files - j.txt and zzz.txt
Based on your description, an awk one liner could achieve your goal.
check the example below, you can change "4" in xargs -n parameter with your X:
kent$ l
total 0
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 01.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 02.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 03.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 04.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 05.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 06.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 07.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 08.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 09.txt
-rw-r--r-- 1 kent kent 0 2011-09-27 11:04 10.txt
kent$ ls|xargs -n4|awk ' {i++;system("mkdir dir"i);system("mv "$0" -t dir"i)}'
kent$ tree
.
|-- dir1
| |-- 01.txt
| |-- 02.txt
| |-- 03.txt
| `-- 04.txt
|-- dir2
| |-- 05.txt
| |-- 06.txt
| |-- 07.txt
| `-- 08.txt
`-- dir3
|-- 09.txt
`-- 10.txt
#!/usr/bin/env bash
dir="${1-.}"
x="${2-4}"
let n=0
let sub=0
while IFS= read -r file ; do
if [ $(bc <<< "$n % $x") -eq 0 ] ; then
let sub+=1
mkdir -p "subdir$sub"
n=0
fi
mv "$file" "subdir$sub"
let n+=1
done < <(find "$dir" -maxdepth 1 -type f)
Works even when files have spaces and other special characters in their names.
Related
Any resources or advice would help, since I am pretty rubbish at scripting
So, I need to go to this path: /home/client/data/storage/customer/data/2020/09/15
And check to see if there are 5 or more consecutive files that contain the word "REJECTED":
ls -ltr
-rw-rw-r-- 1 root root 5059 Sep 15 00:05 customer_rlt_20200915000514737_20200915000547948_8206b49d-b585-4360-8da0-e90b8081a399.zip
-rw-rw-r-- 1 root root 5023 Sep 15 00:06 customer_rlt_20200915000547619_20200915000635576_900b44dc-1cf4-4b1b-a04f-0fd963591e5f.zip
-rw-rw-r-- 1 root root 39856 Sep 15 00:09 customer_rlt_20200915000824108_20200915000908982_b87b01b3-a5dc-4a80-b19d-14f31ff667bc.zip
-rw-rw-r-- 1 root root 39719 Sep 15 00:09 customer_rlt_20200915000901688_20200915000938206_38261b59-8ebc-4f9f-9e2d-3e32eca3fd4d.zip
-rw-rw-r-- 1 root root 12829 Sep 15 00:13 customer_rlt_20200915001229811_20200915001334327_1667be2f-f1a7-41ae-b9ca-e7103d9abbf8.zip
-rw-rw-r-- 1 root root 12706 Sep 15 00:13 customer_rlt_20200915001333922_20200915001357405_609195c9-f23a-4984-936f-1a0903a35c07.zip
Example of rejected file:
customer_rlt_20200513202515792_20200513202705506_5b8deae0-0405-413c-9a81-d1cc2171fa51REJECTED.zip
What I have so far:
!/bin/bash
YYYY=$(date +%Y);
MM=$(date +%m)
DD=$(date +%d)
#Set constants
CODE_OK=0
CODE_WARN=1
CODE_CRITICAL=2
CODE_UNKNOWN=3
#Set Default Values
FILE="/home/client/data/storage/customer/data/${YYYY}/${MM}/{DD}"
if [ ! -f $FILE ]
then
echo "NO TRANSACTIONS FOUND"
exit $CODE_CRITICAL
fi
You can do something quick in AWK:
$ cat consec.awk
/REJECTED/ {
if (match_line == NR - 1) {
consecutives++
} else {
consecutives = 1
}
if (consecutives == 5) {
print "5 REJECTED"
exit
}
match_line = NR
}
$ touch 1 2REJECTED 3REJECTED 5REJECTED 6REJECTED 7REJECTED 8
$ ls -1 | awk -f consec.awk
5 REJECTED
$ rm 3REJECTED; touch 3
$ ls -1 | awk -f consec.awk
$
This works by matching line containing REJECTED, counting consecutive lines (checked with match_line == NR - 1, which means "the last matching line was the previous line") and printing "5 REJECTED" if the number of consecutive lines is 5.
I've used ls -1 (note digit 1, not letter l) to sort by filename in this example. You could use ls -1rt (digit 1 again) to sort by file modification time, as in your original post.
unfortunately I'm quite new at bash, and I want to write a script that will start in a main directory, and check all subdirectories one by one for the presence of certain files, and if those files are present, perform an operation on them. For now, I have written a simplified version to test whether I can do the first part (checking for the files in each directory). This code runs without any errors that I can tell, but it does not echo anything to say that it has successfully found the files which I know are there.
#!/bin/bash
runlist=(1 2 3 4 5 6 7 8 9)
for f in *; do
if [[ -d {$f} ]]; then
#if f is a directory then cd into it
cd "{$f}"
for b in $runlist; do
if [[ -e "{$b}.png" ]]; then
echo "Found {$b}"
#if the file exists then say so
fi
done
cd -
fi
done
'''
Welcome to stackoverflow.
The following will do the trick (a combination of find, array, and if then else):
# list of files we are looking for
runlist=(1 2 4 8 16 32 64 128)
#find each of above anywhere below current directory
# using -maxdepth 1 because, based on on your exam you want to look one level only
# if that's not what you want then take out -maxdepth 1 from the find command
for b in ${runlist[#]}; do
echo
PATH_TO_FOUND_FILE=`find . -name $b.png`
if [ -z "$PATH_TO_FOUND_FILE" ]
then
echo "nothing found" >> /dev/null
else
# You wanted a postive confirmation, so
echo found $b.png
# Now do something with the found file. Let's say ls -l: change that to whatever
ls -l $PATH_TO_FOUND_FILE
fi
done
Here is an example run:
mamuns-mac:stack foo$ ls -lR
total 8
drwxr-xr-x 4 foo 1951595366 128 Apr 11 18:03 dir1
drwxr-xr-x 3 foo 1951595366 96 Apr 11 18:03 dir2
-rwxr--r-- 1 foo 1951595366 652 Apr 11 18:15 find_file_and_do_something.sh
./dir1:
total 0
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 1.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 8.png
./dir2:
total 0
-rw-r--r-- 1 foo 1951595366 0 Apr 11 18:03 64.png
mamuns-mac:stack foo$ ./find_file_and_do_something.sh
found 1.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 ./dir1/1.png
found 8.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 17:58 ./dir1/8.png
found 64.png
-rw-r--r-- 1 foo 1951595366 0 Apr 11 18:03 ./dir2/64.png
This question already has answers here:
How to ignore xargs commands if stdin input is empty?
(7 answers)
Closed 4 years ago.
So I have a command like: grep "\"tool\":\"SEETEST\"" * -l Which works great standalone - it prints out a list of JSON files generated for the selected tool in the current directory.
But then, if I were to pipe it to xargs ls like that:
grep "\"tool\":\"SEETEST\"" * -l | xargs ls -lSh
It prints all the files in the current directory!
How do I make it print just the matched filenames and pipe them to ls sorted by size?
If there are not matches for xargs, then it will list all files in the current directory:
#----------- current files in the directory
mortiz#florida:~/Documents/projects/bash/test$ ls -ltr
total 8
-rw-r--r-- 1 mortiz mortiz 585 Jun 18 12:13 json.example2
-rw-r--r-- 1 mortiz mortiz 574 Jun 18 12:14 json.example
#----------- using your command
mortiz#florida:~/Documents/projects/bash/test$ grep "\"title\": \"example\"" * -l
json.example
#-----------adding xargs to the previous command
mortiz#florida:~/Documents/projects/bash/test$ grep "\"title\": \"example\"" * -l | xargs ls -lSh
-rw-r--r-- 1 mortiz mortiz 574 Jun 18 12:14 json.example
#-----------adding purposely an error on "title"
mortiz#florida:~/Documents/projects/bash/test$ grep "\"titleo\": \"example\"" * -l | xargs ls -lSh
total 8.0K
-rw-r--r-- 1 mortiz mortiz 585 Jun 18 12:13 json.example2
-rw-r--r-- 1 mortiz mortiz 574 Jun 18 12:14 json.example
If you want to use xargs and grep didn't return any match, then add "-r | --no-run-if-empty" that will prevent xargs to list all the files in the current directory:
grep "\"titleo\": \"example\"" * -l | xargs -r ls -lSh
For example, I have a directory called aaa, and there are files named 11.txt,22.exe,33.mp3under aaa directory.
Now I want to make a mirror of this directory, but I only want the structure of the directory not the contents, That is to copy aaa to bbb. Files under bbb should be 11.txt.zero,22.exe.zero,33.mp3.zero. The .zero extension indicates that all the copied file should be of zero file size.
The directory generally contains subfolers.
How to achieve this using windows CMD and linux bash?
Slightly better:
Being in the directory you want to copy:
Make the directories:
find -type d -exec mkdir -p ../new_dir/{} \;
Touch the files:
find -type f -exec touch ../new_dir/{}.zero \;
Original approach
I would do it in two steps:
Copy all files with rsync -r:
rsync -r origin_dir/ target_dir/
Loop through the files in target_dir/, empty them and rename to .zero:
find target_dir/ -type f -exec sh -c 'f={}; > $f; mv $f $f.zero' \;
This of course not very optimal if the files in the original dir are quite big, because we first copy and then empty them.
Test
$ tree aa
aa
├── a
├── a1
│ └── d
├── b
├── c
└── d
1 directory, 5 files
$ ll aa/*
-rw-r--r-- 1 me me 6 Jun 12 11:16 aa/a
-rw-r--r-- 1 me me 6 Jun 12 11:16 aa/b
-rw-r--r-- 1 me me 6 Jun 12 11:16 aa/c
-rw-r--r-- 1 me me 6 Jun 12 11:16 aa/d
aa/a1:
total 0
-rw-r--r-- 1 me me 0 Jun 12 11:16 d
$ rsync -r aa/ bb/
$ tree bb
bb
├── a
├── a1
│ └── d
├── b
├── c
└── d
1 directory, 5 files
$ find bb/ -type f -exec sh -c 'f={}; > $f; mv $f $f.zero' \;
$ tree bb
bb
├── a1
│ └── d.zero
├── a.zero
├── b.zero
├── c.zero
└── d.zero
1 directory, 5 files
$ ll bb/*
-rw-r--r-- 1 me me 0 Jun 12 11:23 bb/a.zero
-rw-r--r-- 1 me me 0 Jun 12 11:23 bb/b.zero
-rw-r--r-- 1 me me 0 Jun 12 11:23 bb/c.zero
-rw-r--r-- 1 me me 0 Jun 12 11:23 bb/d.zero
bb/a1:
total 0
-rw-r--r-- 1 me me 0 Jun 12 11:23 d.zero
#echo off
setlocal enableextensions disabledelayedexpansion
set "source=c:\somewhere\aaaa"
set "target=x:\anotherplace\bbbb"
robocopy "%source%\." "%target%\." /s /e /create
for /r "%target%" %%a in (*) do ren "%%~fa" "%%~nxa.zero"
Here the robocopy command will handle the folder recursion and zero file generation. Once this is done, all the files in the target structure are renamed.
at first copy directory structure:
find aaa -type d -print | sed 's/^aaa/bbb/' | xargs mkdir -p
at second, touch the files:
find aaa -type f -print | sed 's/^aaa/bbb/' | sed 's/$/\.zero/' | xargs touch
and result:
[trak_000.COGITATOR] ➤ ls -lR aaa bbb
aaa:
total 0
drwxrwxr-x 1 trak_000 UsersGrp 0 Jun 12 12:20 q1
aaa/q1:
total 1
-rw-rw-r-- 1 trak_000 UsersGrp 194 Jun 12 12:20 test1.txt
bbb:
total 0
drwxrwxr-x 1 trak_000 UsersGrp 0 Jun 12 12:24 q1
bbb/q1:
total 0
-rw-rw-r-- 1 trak_000 UsersGrp 0 Jun 12 12:24 test1.txt.zero
Use tree command to get recursive list of source folder and then to create zero size structure use mkdir for subdirs and touch for files.
I am getting GREP invalid option error in the code below :
file=$(find . -mtime -4 |ls -lt)
for f in $file
do
po=$(echo $f|cut -d"_" -f2)
find . -mtime -4 |ls -lt|grep "$po"|while read fn
do
if [ -s $fn ]; then #checks if the file is not empty
if [ -d tmp ]; then
rm -r tmp
fi
mkdir tmp
cp -p $fn /tmp/$fn
break
fi
done
done
Basically I am trying to sort the list which I am getting from find then looping through it taking the latest non zero file for a PO.
List of file is
-rw-rw-r-- 1 loneranger loneranger 37 Jul 21 06:30 belk_po12345_20140721.log
-rw-rw-r-- 1 loneranger loneranger 24 Jul 22 06:30 belk_po12345_20140722.log
-rw-rw-r-- 1 loneranger loneranger 0 Jul 23 06:30 belk_po12345_20140723.log
-rw-rw-r-- 1 loneranger loneranger 11 Jul 24 12:00 belk_po12348_20140723.log
PO - po12345 or po12348 these are...
Basically I am trying to sort the list which I am getting from find
then looping through it taking the latest non zero file for a PO.
You might use find for all of that except the final sort:
find . -size '+1c' -type f -printf "%f %T#\n" | sort -k2
The find part search files (-type f) of more than 1 byte long (-size '+1c') and for each one print the file's base name (%f) and the modified time as seconds since Jan. 1, 1970, 00:00 GMT (%T#). After that, it is a simple sort on the second field (timestamp).
Of course, you might add all the search criterion you need on find but that's the basic idea.
And if you want to loop over the result, do as usual:
find . -size '+1c' -type f -printf "%f %T#\n" |
sort -k2 |
while read fname mts; do
# ...
done