Extract lines with duplicate last field - bash

Based on a list generated with the command
find '/patch' -name A* -exec ls -la {} \;
Get only a list with file names that appears more than once (duplicated), but I want display the full line not only the file name.
File name consider only the name of file not the full path.
Example
Based on this file:
-rw-r--r-- 1 root root 34K Jan 9 2014 /usr/share/dia/sheets/AADL.sheet
-rw-r--r-- 1 root root 952 Fev 14 07:07 /usr/share/postgresql/9.3/man/man7/ABORT.7.gz
-rw-r--r-- 1 root root 955 Jul 30 2014 /usr/share/postgresql/9.1/man/man7/ABORT.7.gz
-rw-r--r-- 1 root root 92K Abr 28 2014 /usr/share/gettext/ABOUT-NLS
-rw-r--r-- 1 root root 545 Dez 14 2013 /usr/share/dia/sheets/Automata.sheet
-rw-r--r-- 1 root root 6,7K Dez 21 2012 /usr/share/perl5/Mail/Address.pm
-rw-r--r-- 1 root root 709 Mar 3 09:03 /home/test/Address.pm
-rw-r--r-- 1 root root 709 Mar 3 11:13 /home/test/Automata.sheet
-rw-r--r-- 1 root root 520 Mar 3 11:15 /home/test/t2/Address.pm
I want get this result:
-rw-r--r-- 1 root root 952 Fev 14 07:07 /usr/share/postgresql/9.3/man/man7/ABORT.7.gz
-rw-r--r-- 1 root root 955 Jul 30 2014 /usr/share/postgresql/9.1/man/man7/ABORT.7.gz
-rw-r--r-- 1 root root 6,7K Dez 21 2012 /usr/share/perl5/Mail/Address.pm
-rw-r--r-- 1 root root 709 Mar 3 09:03 /home/test/Address.pm
-rw-r--r-- 1 root root 520 Mar 3 11:15 /home/test/t2/Address.pm
-rw-r--r-- 1 root root 545 Dez 14 2013 /usr/share/dia/sheets/Automata.sheet
-rw-r--r-- 1 root root 709 Mar 3 11:13 /home/test/Automata.sheet
Using this commands
awk -F. '{ n = split($0, a, "/"); print a[n] }' file |sort | uniq -d > filedups
I got
ABORT.7.gz
Address.pm
Automata.sheet
and after
grep -f filedups file
I get expected result.
My question:
Is there a direct way to do this in just one line using awk and/or other commands?

awk to the rescue!
starting with your initial file
$ awk '{n=split($NF,a,"/"); k=a[n]; c[k]++;
v[k]=k in v?v[k] ORS $0:$0}
END {for(k in c) if(c[k]>1) print v[k]}' file
-rw-r--r-- 1 root root 6,7K Dez 21 2012 /usr/share/perl5/Mail/Address.pm
-rw-r--r-- 1 root root 709 Mar 3 09:03 /home/test/Address.pm
-rw-r--r-- 1 root root 520 Mar 3 11:15 /home/test/t2/Address.pm
-rw-r--r-- 1 root root 545 Dez 14 2013 /usr/share/dia/sheets/Automata.sheet
-rw-r--r-- 1 root root 709 Mar 3 11:13 /home/test/Automata.sheet
-rw-r--r-- 1 root root 952 Fev 14 07:07 /usr/share/postgresql/9.3/man/man7/ABORT.7.gz
-rw-r--r-- 1 root root 955 Jul 30 2014 /usr/share/postgresql/9.1/man/man7/ABORT.7.gz

Related

Linux command du returns different size when the same content is, either an image mount to the file system, or under a directory of it

I have a qcow2 image which I have converted to raw format, and then mount to my file system:
# qemu-img convert -O raw initial-image.qcow2 raw-image.img
# kpartx -av raw-image.img
add map loop0p1 (253:1): 0 41940992 linear /dev/loop0 2048
# mount /dev/mapper/loop0p1 ~/myImage
# ls ~/myImage
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
The size of ~/myImage mountpoint is checked with du command as follows:
# du -hx --max-depth=1 ~/myImage/
0 /root/myImage/dev
0 /root/myImage/proc
0 /root/myImage/run
0 /root/myImage/sys
36M /root/myImage/etc
40K /root/myImage/root
53M /root/myImage/var
12K /root/myImage/tmp
834M /root/myImage/usr
89M /root/myImage/boot
0 /root/myImage/home
0 /root/myImage/media
0 /root/myImage/mnt
0 /root/myImage/opt
0 /root/myImage/srv
1012M /root/myImage/
At this point I copied the contents of ~/myImage and pasted them inside ~/myImage-copy directory. Then I checked the size of the newly created directory. I expected the exact same size for each directory between ~/myImage and ~/myImage-copy, but it's not:
# cp -R myImage/ ~/myImage-copy
# du -hx --max-depth=1 ~/myImage-copy/
0 /root/myImage-copy/dev
0 /root/myImage-copy/proc
0 /root/myImage-copy/run
0 /root/myImage-copy/sys
36M /root/myImage-copy/etc
40K /root/myImage-copy/root
63M /root/myImage-copy/var
12K /root/myImage-copy/tmp
862M /root/myImage-copy/usr
89M /root/myImage-copy/boot
0 /root/myImage-copy/home
0 /root/myImage-copy/media
0 /root/myImage-copy/mnt
0 /root/myImage-copy/opt
0 /root/myImage-copy/srv
1.1G /root/myImage-copy/
From the paths above with different size, I chose /usr and I followed a branch of it until I conclude to an innermost directory which has different size, in order to inspect its files. Below is shown the size difference of an innermost directory:
# du -hx --max-depth=1 ~/myImage/usr/lib64/python2.7/json/
112K /root/myImage/usr/lib64/python2.7/json/
# du -hx --max-depth=1 ~/myImage-copy/usr/lib64/python2.7/json/
164K /root/myImage-copy/usr/lib64/python2.7/json/
However, by inspecting their file content, I can't see any difference related to their size. I don't know if the second column plays a role:
# ls -lah ~/myImage/usr/lib64/python2.7/json/
total 200K
drwxr-xr-x. 2 root root 280 Feb 3 11:17 .
drwxr-xr-x. 26 root root 20K Feb 3 11:17 ..
-rw-r--r--. 1 root root 14K Aug 13 2020 decoder.py
-rw-r--r--. 2 root root 12K Aug 13 2020 decoder.pyc
-rw-r--r--. 2 root root 12K Aug 13 2020 decoder.pyo
-rw-r--r--. 1 root root 17K Aug 13 2020 encoder.py
-rw-r--r--. 2 root root 14K Aug 13 2020 encoder.pyc
-rw-r--r--. 2 root root 14K Aug 13 2020 encoder.pyo
-rw-r--r--. 1 root root 15K Aug 13 2020 __init__.py
-rw-r--r--. 2 root root 14K Aug 13 2020 __init__.pyc
-rw-r--r--. 2 root root 14K Aug 13 2020 __init__.pyo
-rw-r--r--. 1 root root 2.3K Aug 13 2020 scanner.py
-rw-r--r--. 2 root root 2.2K Aug 13 2020 scanner.pyc
-rw-r--r--. 2 root root 2.2K Aug 13 2020 scanner.pyo
-rw-r--r--. 1 root root 997 Aug 13 2020 tool.py
-rw-r--r--. 2 root root 1.3K Aug 13 2020 tool.pyc
-rw-r--r--. 2 root root 1.3K Aug 13 2020 tool.pyo
# ls -lah ~/myImage-copy/usr/lib64/python2.7/json/
total 200K
drwxr-xr-x 2 root root 280 Mar 4 17:55 .
drwxr-xr-x 26 root root 20K Mar 4 17:55 ..
-rw-r--r-- 1 root root 14K Mar 4 17:55 decoder.py
-rw-r--r-- 1 root root 12K Mar 4 17:55 decoder.pyc
-rw-r--r-- 1 root root 12K Mar 4 17:55 decoder.pyo
-rw-r--r-- 1 root root 17K Mar 4 17:55 encoder.py
-rw-r--r-- 1 root root 14K Mar 4 17:55 encoder.pyc
-rw-r--r-- 1 root root 14K Mar 4 17:55 encoder.pyo
-rw-r--r-- 1 root root 15K Mar 4 17:55 __init__.py
-rw-r--r-- 1 root root 14K Mar 4 17:55 __init__.pyc
-rw-r--r-- 1 root root 14K Mar 4 17:55 __init__.pyo
-rw-r--r-- 1 root root 2.3K Mar 4 17:55 scanner.py
-rw-r--r-- 1 root root 2.2K Mar 4 17:55 scanner.pyc
-rw-r--r-- 1 root root 2.2K Mar 4 17:55 scanner.pyo
-rw-r--r-- 1 root root 997 Mar 4 17:55 tool.py
-rw-r--r-- 1 root root 1.3K Mar 4 17:55 tool.pyc
-rw-r--r-- 1 root root 1.3K Mar 4 17:55 tool.pyo
In addition, I checked the md5 hash of the files above as a view of their content. It seems there is no content differences between the same files:
# md5sum ~/myImage/usr/lib64/python2.7/json/*
598c681c82c582ca3f17950b4d5413c1 /root/myImage/usr/lib64/python2.7/json/decoder.py
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage/usr/lib64/python2.7/json/decoder.pyc
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage/usr/lib64/python2.7/json/decoder.pyo
007a9954ca6641b29564a0f0cb55096b /root/myImage/usr/lib64/python2.7/json/encoder.py
366d44efbfce1e5b4a3397b15f202765 /root/myImage/usr/lib64/python2.7/json/encoder.pyc
366d44efbfce1e5b4a3397b15f202765 /root/myImage/usr/lib64/python2.7/json/encoder.pyo
dd5db0c7fb7c531be4e14feebbdb52e8 /root/myImage/usr/lib64/python2.7/json/__init__.py
791c8dca2a86ce50ead04e918a17b508 /root/myImage/usr/lib64/python2.7/json/__init__.pyc
791c8dca2a86ce50ead04e918a17b508 /root/myImage/usr/lib64/python2.7/json/__init__.pyo
8d6660f10863f99ffdd9a95eeddd7b64 /root/myImage/usr/lib64/python2.7/json/scanner.py
59865f486b742aea959693c04cfce057 /root/myImage/usr/lib64/python2.7/json/scanner.pyc
59865f486b742aea959693c04cfce057 /root/myImage/usr/lib64/python2.7/json/scanner.pyo
ad879e2ba247d3d4a5cc71fe22db91d0 /root/myImage/usr/lib64/python2.7/json/tool.py
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage/usr/lib64/python2.7/json/tool.pyc
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage/usr/lib64/python2.7/json/tool.pyo
# md5sum ~/myImage-copy/usr/lib64/python2.7/json/*
598c681c82c582ca3f17950b4d5413c1 /root/myImage-copy/usr/lib64/python2.7/json/decoder.py
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage-copy/usr/lib64/python2.7/json/decoder.pyc
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage-copy/usr/lib64/python2.7/json/decoder.pyo
007a9954ca6641b29564a0f0cb55096b /root/myImage-copy/usr/lib64/python2.7/json/encoder.py
366d44efbfce1e5b4a3397b15f202765 /root/myImage-copy/usr/lib64/python2.7/json/encoder.pyc
366d44efbfce1e5b4a3397b15f202765 /root/myImage-copy/usr/lib64/python2.7/json/encoder.pyo
dd5db0c7fb7c531be4e14feebbdb52e8 /root/myImage-copy/usr/lib64/python2.7/json/__init__.py
791c8dca2a86ce50ead04e918a17b508 /root/myImage-copy/usr/lib64/python2.7/json/__init__.pyc
791c8dca2a86ce50ead04e918a17b508 /root/myImage-copy/usr/lib64/python2.7/json/__init__.pyo
8d6660f10863f99ffdd9a95eeddd7b64 /root/myImage-copy/usr/lib64/python2.7/json/scanner.py
59865f486b742aea959693c04cfce057 /root/myImage-copy/usr/lib64/python2.7/json/scanner.pyc
59865f486b742aea959693c04cfce057 /root/myImage-copy/usr/lib64/python2.7/json/scanner.pyo
ad879e2ba247d3d4a5cc71fe22db91d0 /root/myImage-copy/usr/lib64/python2.7/json/tool.py
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage-copy/usr/lib64/python2.7/json/tool.pyc
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage-copy/usr/lib64/python2.7/json/tool.pyo
So, what is the reason behind the size difference of the same content in du output?
My operating system is CentOS Linux release 7.9.2009.

Clean ten days and older logs using Shell

[root#amp logs]# ls -l
total 0
-rw-r--r-- 1 root root 0 Nov 23 17:51 lb-quarzcenter.log
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-01
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-02
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-03
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-04
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-05
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-06
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-07
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-08
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-10
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-11
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-12
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-13
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-14
-rw-r--r-- 1 root root 0 Nov 23 17:51 lb-quarzcenter.log.2019-02-15
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-09
How do I match a string with year-month-date and delete 10 days ago file according to lb-quarzcenter.log.*?
now=$(date +%s) # express current time as seconds since 1970-01-01
(( ten_days_ago = now - 60*60*24*10 )) # subtract 864000 seconds (10 days) from that
date_minus_ten=$(date +%F --date="#$ten_days_ago") # express that number as a YYYY-MM-DD
for filename in lb-quartzcenter.log.* ; do # loop over all matching files
filedate="${filename/lb-quartzcenter.log./}" # remove the filename part before the timestamp
if [[ $filedate < $date_minus_ten ]] ; then # if filename remainder is lexicographically lower...
rm -f "$filename" # ... remove the file
fi
done

Make nl to enumerate from a specified line

I intent to enumerate the files and dirs
[root#iz2ze9wve43n2nyuvmsfx5z /]# ls -lh |nl
1 total 60K
2 lrwxrwxrwx 1 root root 7 Jun 15 22:08 bin -> usr/bin
3 dr-xr-xr-x. 5 root root 4.0K Jun 15 22:10 boot
4 drwxr-xr-x 19 root root 2.9K Sep 17 11:35 dev
5 drwxr-xr-x. 85 root root 4.0K Sep 18 22:44 etc
6 drwxr-xr-x. 2 root root 4.0K Sep 17 08:25 home
7 lrwxrwxrwx 1 root root 7 Jun 15 22:08 lib -> usr/lib
8 lrwxrwxrwx 1 root root 9 Jun 15 22:08 lib64 -> usr/lib64
9 drwx------. 2 root root 16K Oct 15 2017 lost+found
10 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 media
11 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 mnt
12 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 opt
13 dr-xr-xr-x 86 root root 0 Sep 17 11:35 proc
14 drwxrwxrwx. 21 root root 4.0K Oct 22 08:11 root
15 drwxr-xr-x 24 root root 660 Oct 24 19:10 run
16 lrwxrwxrwx 1 root root 8 Jun 15 22:08 sbin -> usr/sbin
17 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 srv
18 dr-xr-xr-x 13 root root 0 Oct 21 19:50 sys
19 drwxrwxrwt. 11 root root 4.0K Oct 24 05:19 tmp
20 drwxr-xr-x. 13 root root 4.0K Jun 15 22:08 usr
21 drwxr-xr-x. 19 root root 4.0K Jun 15 22:08 var
It starts with 1 which I desire is 0.
How could make nl to enumerate from a specified line.
You just need to use the -v option, and specify to start from 0; your instruction becoming:
ls -lh |nl -v 0
Lets have a look at nl --help:
$ nl --help
Usage: nl [OPTION]... [FILE]...
Write each FILE to standard output, with line numbers added.
With no FILE, or when FILE is -, read standard input.
Mandatory arguments to long options are mandatory for short options too.
...
-v, --starting-line-number=NUMBER first line number for each section
Therefore:
$ nl -v0 <<EOF
> a
> b
> EOF
0 a
1 b
Note that it does not really matter whether you want to enumerate directories or anything else, which is why you probably should strip this part from your question to make it more specific.

Listing filenames and timestamps in subdirectories recursively in UNIX

The command that I've been using is "ls -lR". The results usually look like this:
.:
total 4
lrwxrwxrwx 1 root root 9 Oct 11 03:35 dos -> /root/dos
drwxr-xr-x 2 root root 80 Oct 11 03:35 folder1
drwxr-xr-x 2 root root 100 Oct 11 03:35 folder2
-rw-r--r-- 1 root root 242 Oct 11 03:35 hello.c
./folder1:
total 0
-rw-r--r-- 1 root root 0 Oct 11 03:25 file1001
-rw-r--r-- 1 root root 0 Oct 11 03:35 file1002
./folder2:
total 0
-rw-r--r-- 1 root root 0 Oct 11 03:39 file2001
-rw-r--r-- 1 root root 0 Oct 11 03:45 file2002
How do I optimize the command so that it would only display the following?
./folder1:
Oct 11 03:25 | file1001
Oct 11 03:35 | file1002
./folder2:
Oct 11 03:39 | file2001
Oct 11 03:45 | file2002
Here's something that might work:
~/mydir ls -lR | grep -vi "total" | egrep -o "^\.\/.*|^\..*|([A-Z])\w+\s+[0-9]+\s+[0-9]+\s+.*"
.:
Sep 4 2015 es_jira.py
Sep 4 2015 es_slurp_jira.py
Aug 21 2015 __init__.py
./plugins:
Sep 4 2015 __init__.py
Sep 4 2015 lrt_report.py
Sep 11 2015 mr_fingerprint.py
Mar 6 2016 mr_tunable.py
Dec 1 2015 plugin.py
Dec 1 2015 test
Dec 1 2015 utils.py
./plugins/test:
Sep 4 2015 _test_ca_space_plugin.py
Sep 4 2015 _test_lrt_report_plugin.py
Sep 4 2015 _test_mr_failover_plugin.py
Sep 4 2015 _test_mr_fingerprint_plugin.py
Dec 1 2015 _test_mr_tunable_plugin.py
Sep 4 2015 _test_spacedays_plugin.py
If you want to start adding tabs for nested lines and stuff, you're looking for a script and variable work, which is doable in a one-liner, but gets more complicated than a quick and dirty grep.

How to set world permissions to be the same as group permissions?

How would you go about changing permissions for a file or in a directory recursively in such a way that group permissions would be copied over to world permissions, with no other changes? For example, to go from this directory listing:
drwxr-x--- 2 septi septi 4096 Jun 29 01:14 example.d
-rw-r----- 1 septi septi 0 Jun 29 01:14 example.r
-rwxr-x--- 1 septi septi 0 Jun 29 01:14 example.x
...to:
drwxr-xr-x 2 septi septi 4096 Jun 29 01:14 example.d
-rw-r--r-- 1 septi septi 0 Jun 29 01:14 example.r
-rwxr-xr-x 1 septi septi 0 Jun 29 01:14 example.x
From the chmod(1) man page (relevant parts extracted):
-R Change the modes of the file hierarchies rooted in the files
instead of just the files themselves.
And:
The symbolic mode is described by the following grammar:
who ::= a | u | g | o
op ::= + | - | =
perm ::= r | s | t | w | x | X | u | g | o
The who symbols "u", "g", and "o" specify the user, group, and
other parts of the mode bits, respectively. The who symbol a is
equivalent to ugo.
The perm symbols represent the portions of the mode bits as follows:
g The group permission bits in the original mode of the file.
So for you:
chmod -R o=g *
Example:
$ ls -l
total 0
drwxr-x--- 2 carl staff 68 Jun 28 10:25 example.d
-rw-r----- 1 carl staff 0 Jun 28 10:25 example.r
-rwxr-x--- 1 carl staff 0 Jun 28 10:25 example.x
$ chmod -R o=g *
$ ls -l
total 0
drwxr-xr-x 2 carl staff 68 Jun 28 10:25 example.d
-rw-r--r-- 1 carl staff 0 Jun 28 10:25 example.r
-rwxr-xr-x 1 carl staff 0 Jun 28 10:25 example.x

Resources