shell script to find first and last occourance of a string - shell

Question

I have prepared a shell script to do the following in a 50 node hadoop cluster:
list all the log files related to my application in each server
print the last modified timestamp, hostname, filename
sort the log files from 50 nodes as per the modified time stamp
current output format is:
2016-07-11-01:06 server1 MY_APPLICATION-worker-6701.log.6.gz
2016-07-12-05:23 server1 MY_APPLICATION-worker-6701.log.7.gz
2016-07-13-08:38 server2 MY_APPLICATION-worker-6701.log
2016-07-13-10:38 server3 MY_APPLICATION-worker-6701.log.out
2016-07-13-10:38 server2 MY_APPLICATION-worker-6701.log.err
2016-07-13-10:38 server5 MY_APPLICATION-worker-6701.log
2016-07-15-10:22 server4 MY_APPLICATION-worker-6703.log.out
2016-07-15-10:22 server3 MY_APPLICATION-worker-6703.log.err
2016-07-15-10:22 server2 MY_APPLICATION-worker-6703.log
.
totallogs=""
for server in $(cat all-hadoop-cluster-servers.txt); do
logs1="$(ssh user_id#$server 'ls /var/log/hadoop/storm/ -ltr --time-style="+%Y-%m-%d-%H:%M" | grep MY_APPLICATION | awk -v host=$HOSTNAME "{print \$6, host, \$7}"' )"
if [ -z "${logs1}" ]; then
continue
else
logs1+="\n"
totallogs+=$logs1
fi
done
for el in "${totallogs[#]}"
do
printf "$el"
done | sort
How to find the first occurrence of the "unique-ID" and last occurrence of the "unique-ID" in each log file along with the above output.
Expected output format is:
time_stamp hostname filename first-unique-ID last-unique-ID
2016-07-11-01:06 server1 MY_APPLICATION-worker-6701.log.6.gz 1467005065878 1467105065877
2016-07-12-05:23 server1 MY_APPLICATION-worker-6701.log.7.gz 1467105065878 1467205065860
2016-07-13-08:38 server2 MY_APPLICATION-worker-6701.log 1467205065861 1467305065852
2016-07-13-10:38 server3 MY_APPLICATION-worker-6701.log.out
2016-07-13-10:38 server2 MY_APPLICATION-worker-6701.log.err
2016-07-13-10:38 server5 MY_APPLICATION-worker-6701.log 1467305065853 1467405065844
2016-07-15-10:22 server4 MY_APPLICATION-worker-6703.log.out
2016-07-15-10:22 server3 MY_APPLICATION-worker-6703.log.err
2016-07-15-10:22 server2 MY_APPLICATION-worker-6703.log 1467405065845 1467505065853
Sample log file:
DEBUG | 2008-09-06 10:51:44,848 | unique-ID >>>>>> 1467205065861
DEBUG | 2008-09-06 10:51:44,817 | DefaultBeanDefinitionDocumentReader.java | 86 | Loading bean definitions
DEBUG | 2008-09-06 10:51:44,848 | AbstractBeanDefinitionReader.java | 185 | Loaded 5 bean definitions from location pattern [samContext.xml]
INFO | 2008-09-06 10:51:44,848 | XmlBeanDefinitionReader.java | 323 | Loading XML bean definitions from class path resource [tmfContext.xml]
DEBUG | 2008-09-06 10:51:44,848 | DefaultDocumentLoader.java | 72 | Using JAXP provider [com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl]
DEBUG | 2008-09-06 10:51:44,848 | BeansDtdResolver.java | 72 | Found beans DTD [http://www.springframework.org/dtd/spring-beans.dtd] in classpath: spring-beans.dtd
DEBUG | 2008-09-06 10:51:44,848 | unique-ID >>>>>> 1467205065862
DEBUG | 2008-09-06 10:51:44,864 | DefaultBeanDefinitionDocumentReader.java | 86 | Loading bean definitions
DEBUG | 2008-09-06 10:51:45,458 | AbstractAutowireCapableBeanFactory.java | 411 | Finished creating instance of bean 'MS-SQL'
DEBUG | 2008-09-06 10:51:45,458 | DefaultSingletonBeanRegistry.java | 213 | Creating shared instance of singleton bean 'MySQL'
DEBUG | 2008-09-06 10:51:45,458 | AbstractAutowireCapableBeanFactory.java | 383 | Creating instance of bean 'MySQL'
DEBUG | 2008-09-06 10:51:45,458 | AbstractAutowireCapableBeanFactory.java | 459 | Eagerly caching bean 'MySQL' to allow for resolving potential circular references
DEBUG | 2008-09-06 10:51:45,458 | AbstractAutowireCapableBeanFactory.java | 411 | Finished creating instance of bean 'MySQL'
DEBUG | 2008-09-06 10:51:45,458 | DefaultSingletonBeanRegistry.java | 213 | Creating shared instance of singleton bean 'Oracle'
DEBUG | 2008-09-06 10:51:45,458 | AbstractAutowireCapableBeanFactory.java | 383 | Creating instance of bean 'Oracle'
DEBUG | 2008-09-06 10:51:45,458 | AbstractAutowireCapableBeanFactory.java | 459 | Eagerly caching bean 'Oracle' to allow for resolving potential circular references
DEBUG | 2008-09-06 10:51:45,473 | AbstractAutowireCapableBeanFactory.java | 411 | Finished creating instance of bean 'Oracle'
DEBUG | 2008-09-06 10:51:45,473 | DefaultSingletonBeanRegistry.java | 213 | Creating shared instance of singleton bean 'PostgreSQL'
DEBUG | 2008-09-06 10:51:45,473 | AbstractAutowireCapableBeanFactory.java | 383 | Creating instance of bean 'PostgreSQL'
DEBUG | 2008-09-06 10:51:45,473 | AbstractAutowireCapableBeanFactory.java | 459 | Eagerly caching bean 'PostgreSQL' to allow for resolving potential circular references
DEBUG | 2008-09-06 10:51:45,473 | AbstractAutowireCapableBeanFactory.java | 411 | Finished creating instance of bean 'PostgreSQL'
INFO | 2008-09-06 10:51:45,473 | SQLErrorCodesFactory.java | 128 | SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
DEBUG | 2008-09-06 10:52:44,817 | DefaultBeanDefinitionDocumentReader.java | 86 | Loading bean definitions
DEBUG | 2008-09-06 10:52:44,848 | unique-ID >>>>>> 1467205065864

Answer 1

grep 'uniqueID' sample_log_file | sed -n '1p;$p'

Answer 2

Since you're already using awk, you could change your awk program
"{print \$6, host, \$7}"
to
"{ first=last=\"\"; path=\"/var/log/hadoop/storm/\"\$7; while (getline var <path) if (split(var, arr, \">>>>>>\") > 1) { if (!first) first=arr[2]; last=arr[2] } print \$6, host, \$7, \"\t\", first, last }"
in order to let it do the work.