Can sed be used with find to print Fields and Folder Name - bash

This small script :
touch ilFldsN9LS.txt
ls -l | grep "^d" > /home/userB/PLAY/LibTESTxOutputFiles/ilFldsN9LS_testTEST.txt
produces file content of this format:
drwxr-xr-x 2 userB userB 4096 Mar 23 22:40 BASH_Collection_FolderNESTY
drwxr-xr-x 2 userB userB 4096 Mar 24 17:33 BASH_Collection_Functionality
What I wish to achieve is to get output very much like the above, but using find.
Turning to the use of find, this script: (which unlike the one previous, is recursive)
find . -type d \( ! -iname ".*" \) -exec ls -ld {} \; | grep "^d" |\
| tee -a /home/UserB/PLAY/LibTESTxOutputFiles/ilFldsR9FB.txt
produces file content of this format:
drwxr-xr-x 2 UserB UserB 4096 Mar 24 17:33 ./BASH_Collection_Functionality
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 ./LibTESTxOutputFiles/AdditionalTESTresults
adding some AWK to the script, like so:
find . -type d \( ! -iname ".*" \) -exec ls -ld {} \; | grep "^d" | awk '{ sub(/\.\//, " ");print}'\
| tee -a /home/innocentxlii/PLAY/LibTESTxOutputFiles/ilFldsR9FB.txt
produces output with the ** ./ * stripped from the front of the path
and pads the gap with an extra space to ease reading:
drwxr-xr-x 2 UserB UserB 4096 Mar 24 17:33 BASH_Collection_Functionality
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 LibTESTxOutputFiles/AdditionalTESTresults
Where I have gotten stuck, is I have been trying to use sed to keep the Fields, but to
have only the last Folder in the path, listed. For example the last item above would be:
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 AdditionalTESTresults
Ideas? I tried literally dozens of sed variants but have realized something must be
wrong with this approach.

How about this: sed -r 's/(.* )(.*)\/(.*)$/\1\3/g'
Check my test runs below:
$ echo "drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir1/dir2/dir3/dir4" | sed -r 's/(.* )(.*)\/(.*)$/\1\3/g'
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir4
$ echo "drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir1/dir2/dir3" | sed -r 's/(.* )(.*)\/(.*)$/\1\3/g'
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir3
$ echo "drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir1/dir2" | sed -r 's/(.* )(.*)\/(.*)$/\1\3/g'
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir2
$ echo "drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir1" | sed -r 's/(.* )(.*)\/(.*)$/\1\3/g'
drwxr-xr-x 2 UserB UserB 4096 Mar 25 16:04 dir1
NOTE: Please note that there is a space After the 1st asterisks in the sed command

You don't need sed, you can do it all with awk
find . -type d \( ! -iname ".*" \) -exec ls -ld {} \; |
awk '{n=split($NF,a,"/"); sub($NF, " "a[n])}1'
grep "^d" is redundant since you are already finding directories with find . -type d
awk Explanation:
n=split($NF,a,"/") splits the last field (denoted by $NF) by / and assigns it to array a
n gives the length of the array
a[n] will therefore return the string following the last / (i.e. the inner most directory)
sub($NF, " "a[n]) replaces the last field (denoted by $NF) with a space for padding (as per example) + inner most directory (denoted by a[n])
awk '{...}1' the 1 outside the awk is the same as print
EDIT: RE: for cases where directory contain spaces in name
find . -type d \( ! -iname ".*" \) -exec ls -ld {} \; |
awk -F '[0-9]+:[0-9]+ ' '{n=split($NF,a,"/"); sub($NF, " "a[n])}1'
specifying the input separator with -F '[0-9]+:[0-9]+ ' (for modification time) will ensure the last field ($NF) is the file name -- regardless whether or not it contain spaces in the directory name

what about
sed 's|\./||'
that remove the first ./, so in your line
find . -type d \( ! -iname ".*" \) -exec ls -ld {} \; | sed 's|\./||'

Related

grep -l does not behave as expected on piping to xargs [duplicate]

This question already has answers here:
How to ignore xargs commands if stdin input is empty?
(7 answers)
Closed 4 years ago.
So I have a command like: grep "\"tool\":\"SEETEST\"" * -l Which works great standalone - it prints out a list of JSON files generated for the selected tool in the current directory.
But then, if I were to pipe it to xargs ls like that:
grep "\"tool\":\"SEETEST\"" * -l | xargs ls -lSh
It prints all the files in the current directory!
How do I make it print just the matched filenames and pipe them to ls sorted by size?
If there are not matches for xargs, then it will list all files in the current directory:
#----------- current files in the directory
mortiz#florida:~/Documents/projects/bash/test$ ls -ltr
total 8
-rw-r--r-- 1 mortiz mortiz 585 Jun 18 12:13 json.example2
-rw-r--r-- 1 mortiz mortiz 574 Jun 18 12:14 json.example
#----------- using your command
mortiz#florida:~/Documents/projects/bash/test$ grep "\"title\": \"example\"" * -l
json.example
#-----------adding xargs to the previous command
mortiz#florida:~/Documents/projects/bash/test$ grep "\"title\": \"example\"" * -l | xargs ls -lSh
-rw-r--r-- 1 mortiz mortiz 574 Jun 18 12:14 json.example
#-----------adding purposely an error on "title"
mortiz#florida:~/Documents/projects/bash/test$ grep "\"titleo\": \"example\"" * -l | xargs ls -lSh
total 8.0K
-rw-r--r-- 1 mortiz mortiz 585 Jun 18 12:13 json.example2
-rw-r--r-- 1 mortiz mortiz 574 Jun 18 12:14 json.example
If you want to use xargs and grep didn't return any match, then add "-r | --no-run-if-empty" that will prevent xargs to list all the files in the current directory:
grep "\"titleo\": \"example\"" * -l | xargs -r ls -lSh

Concatenate multiple directories name adding a counter in bash

I have to store the last 5 test results folders. Every time the tests are executed, a folder called "target" is created with all the results. The idea is to move that folder and add a counter (Ex.: target1, target2, ...). And store the last 5 executions.
It is important that when 5 folders exists and a new test execution creates a new folder, the oldest should be removed and all the folders renamed starting for the new one as target1 and the new last from target4 to target5
For now, I'm just storing the lastone.
rm -rf ${WORKSPACE}/target
if [ -d "${WORKSPACE}/tjba-hmi-toolkit/target" ]; then
# Control will enter here if "target" exists.
cp -r ${WORKSPACE}/tjba-hmi-toolkit/target ${WORKSPACE}/
fi
This will work with the minor glitch that first copy will have '0' suffix
$ cp -rp tmp cptest/tmp$(find cptest -type d -name 'tmp*' | wc -l)
$ ls -l cptest/
total 0
drwxr-xr-x 4 luis users 273 abr 27 14:42 tmp0
drwxr-xr-x 4 luis users 273 abr 27 14:42 tmp1
drwxr-xr-x 4 luis users 273 abr 27 14:42 tmp2
drwxr-xr-x 4 luis users 273 abr 27 14:42 tmp3
drwxr-xr-x 4 luis users 273 abr 27 14:42 tmp4`

Csh - Fetching fields via awk inside xargs

I'm struggling to understand this behavior:
Script behavior: read a file (containing dates); print a list of files in a multi-level directory tree and get their size, print the file size only, (future step: sum the overall file size).
Starting script:
cat dates | xargs -I {} sh -c "echo '{}: '; du -d 2 "/folder/" | grep {} | head"
2000-03:
1000 /folder/2000-03balbasldas
2000-04:
12300 /folder/2000-04asdwqdas
[and so on]
But when I try to filter via awk on the first field, I still get the whole line
cat dates | xargs -I {} sh -c "echo '{}: '; du -d 2 "/folder/" | grep {} | awk '{print $1}'"
2000-03:
1000 /folder/2000-03balbasldas
2000-04:
12300 /folder/2000-04asdwqdas
I've already approached it via divide-et-impera, and the following command works just fine:
du -d 2 "/folder/" | grep '2000-03' | awk '{print $1}'
1000
I'm afraid that I'm missing something very trivial, but I haven't found anything so far.
Any idea? Thanks!
Input: directory containing folders named YYYY-MM-random_data and a file containing strings:
ls -l
drwxr-xr-x 2 user staff 68 Apr 24 11:21 2000-03-blablabla
drwxr-xr-x 2 user staff 68 Apr 24 11:21 2000-04-blablabla
drwxr-xr-x 2 user staff 68 Apr 24 11:21 2000-05-blablabla
drwxr-xr-x 2 user staff 68 Apr 24 11:21 2000-06-blablabla
drwxr-xr-x 2 user staff 68 Apr 24 11:21 2000-06-blablablb
drwxr-xr-x 2 user staff 68 Apr 24 11:21 2000-06-blablablc
[...]
cat dates
2000-03
2000-04
2000-05
[...]
Expected output: sum of the disk space occupied by all the files contained in the folder whose name include the string in the file dates
2000-03: 1000
2000-04: 2123
2000-05: 1222112
[...]
======
But in particular, I'm interested in why awk is not able to fetch the column $1 I asked it to.
Ok it seems I found the answer myself after a lot of research :D
I'll post it here, hoping that it will help somebody else out.
https://unix.stackexchange.com/questions/282503/right-syntax-for-awk-usage-in-combination-with-other-command-inside-xargs-sh-c
The trick was to escape the $ sign.
cat dates | xargs -I {} sh -c "echo '{}: '; du -d 2 "/folder/" | grep {} | awk '{print \$1}'"
Using GNU Parallel it looks like this:
parallel --tag "eval du -s folder/{}* | perl -ne '"'$s+=$_ ; END {print "$s\n"}'"'" :::: dates
--tag prepends the line with the date.
{} is replaced with the date.
eval du -s folder/{}* finds all the dirs starting with the date and gives the total du from those dirs.
perl -ne '$s+=$_ ; END {print "$s\n"}' sums up the output from du
Finally there is bit of quoting trickery to get it quoted correctly.

error in script containing grep

I am getting GREP invalid option error in the code below :
file=$(find . -mtime -4 |ls -lt)
for f in $file
do
po=$(echo $f|cut -d"_" -f2)
find . -mtime -4 |ls -lt|grep "$po"|while read fn
do
if [ -s $fn ]; then #checks if the file is not empty
if [ -d tmp ]; then
rm -r tmp
fi
mkdir tmp
cp -p $fn /tmp/$fn
break
fi
done
done
Basically I am trying to sort the list which I am getting from find then looping through it taking the latest non zero file for a PO.
List of file is
-rw-rw-r-- 1 loneranger loneranger 37 Jul 21 06:30 belk_po12345_20140721.log
-rw-rw-r-- 1 loneranger loneranger 24 Jul 22 06:30 belk_po12345_20140722.log
-rw-rw-r-- 1 loneranger loneranger 0 Jul 23 06:30 belk_po12345_20140723.log
-rw-rw-r-- 1 loneranger loneranger 11 Jul 24 12:00 belk_po12348_20140723.log
PO - po12345 or po12348 these are...
Basically I am trying to sort the list which I am getting from find
then looping through it taking the latest non zero file for a PO.
You might use find for all of that except the final sort:
find . -size '+1c' -type f -printf "%f %T#\n" | sort -k2
The find part search files (-type f) of more than 1 byte long (-size '+1c') and for each one print the file's base name (%f) and the modified time as seconds since Jan. 1, 1970, 00:00 GMT (%T#). After that, it is a simple sort on the second field (timestamp).
Of course, you might add all the search criterion you need on find but that's the basic idea.
And if you want to loop over the result, do as usual:
find . -size '+1c' -type f -printf "%f %T#\n" |
sort -k2 |
while read fname mts; do
# ...
done

Folders tree with rights

In OS X and SunOS OS there is no exist the 'bash tree command'.
To plot a tree "graph" of folders i use the following instruction:
find . -type d -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
Or this to show files too.
find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
But i need another version which contains also the folders rights. Im pretty lost to add on the right side the folder rights. Anyone have any idea ??
Update:
There are any option to plot the files inside the folders and their rights too. I'm trying with this command find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g' and doing combination with the solution provided by #fedorqui but the result aren't so good.
That's the result obtained with the above command, without rights.
| | |____src
| | | |____cft2exit.c
| | | |____cft2exit_AIX
| | | |____cft2exit_SUN
| | | |____gestidt.c
| | | |____gestidt.h
| | | |____gestidt.o
| | | |____gestidt_AIX
| | | |____gestidt_SUN
| | | |____gestidt_SunOS
| | | |____makefile
| | | |____sem.a
| | | |____ut_sem.c
| | | |____ut_sem.h
| | | |____ut_sem.o
| |____data
| | |____purge.dat
| |____lost+found
You can execute ls -ld for each result of find. It will give you the permissions, other things, and then the file name. If you then pipe to awk, with awk '{print $NF, $1}' you can print both blocks of information. Finally, you pipe to your sed command. All together:
find . -type d -exec ls -ld {} \; | awk '{print $NF, $1}' | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
Test
$ find . -type d -exec ls -ld {} \; | awk '{print $NF, $1}' | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
. drwxrwxr-x
|____python drwxrwxr-x
| |____jinja2 drwxrwxr-x
| | |____bk drwxrwxr-x
| | |____infiles drwxrwxr-x
.......
In small steps:
$ find . -type d -exec ls -ld {} \;
drwxrwxr-x 7 me me 4096 Aug 15 15:35 .
drwxrwxr-x 3 me me 4096 Aug 13 14:31 ./python
drwxrwxr-x 4 me me 4096 Apr 26 15:14 ./python/jinja2
drwxrwxr-x 2 me me 4096 Apr 19 14:26 ./python/jinja2/bk
drwxrwxr-x 2 me me 4096 Apr 19 12:54 ./python/jinja2/infiles
and then
$ find . -type d -exec ls -ld {} \; | awk '{print $NF, $1}'
. drwxrwxr-x
./python drwxrwxr-x
./python/jinja2 drwxrwxr-x
./python/jinja2/bk drwxrwxr-x
./python/jinja2/infiles drwxrwxr-x
On OS X you can install tree, using homebrew:
brew install tree
or, using macports:
sudo port install tree
and then, to view directories with permissions:
$ tree -p -d
Sample output:
.
├── [drwxr-xr-x] folder1
│   └── [drwxr-xr-x] folder2
│   └── [drwxr-xr-x] folder3
│   └── [drwxr-xr-x] folder4
└── [drwxr-xr-x] folder5
├── [drwxr-xr-x] folder6
└── [drwxr-xr-x] folder7

Resources