Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
I want to number each line that gets outputted when i list a directory, so that instead of typing out the full name of the object, i can identify it with a number in the list. In Bash.
Ex. os-list is a directory I use to store numerous of objects that are ever changing.
os1.xxx.iso is the object name.
From
ls os-list
os1-xxx.iso
os2-xxx.iso
What is the path?: os1-xxx.iso
To
ls os-list
[1]os1-xxx.iso
[2]os2-xxx.iso
What is the path? 1
What is the term that im looking for this kind of operation in bash?
The command select can be used:
files=$(ls os-list)
select choice in ${files[#]}; do
break
done
echo "${choice}"
You can modify this to your needs, just look for more examples with select.
I would change the prompt (PS3="What is the path: ") and replace the break in the select loop (check for a valid response).
I want to number each line that gets outputted when i list a directory
For your exact desired output format:
ls | nl | sed 's/^[ \t]*//' | sed -r 's/^[0-9]*()/[\0]/' | sed 's/\t//'
or
ls | cat -n | sed 's/^[ \t]*//' | sed -r 's/^[0-9]*()/[\0]/' | sed 's/\t//'
If you only want to have number as reference, and not in your exact desired format then simply:
ls | cat -n
or
ls | nl
would suffice, sed pipe is added to enclose given number in square brackets and remove starting/trailing spaces to conform to your desired output. Admitedly, this could be done with awk as well, pipe is not optimized, just given as reference point.
Edit:
with awk like so:
ls | cat -n | awk '{print "[" NR "]"$2}'
Selecting filename based on index (example with index 12 given):
ls | cat -n | awk '{print "[" NR "]"$2}' | grep "^\[12\]" | sed 's/^\[12\]//'
Note of caution: this approach supposes that between listing and selecting no file is added (if file is added in between and your sort order is messed up 12th file in listing and 12th file in select might ended not being the same file).
Related
This question already has answers here:
Remove empty lines in a text file via grep
(11 answers)
Closed 4 years ago.
Exists way to remove empty lines with cat myfile | grep -w #something ?
I looking for simple way for remove empty lines from my output like in the way the presented above.
This really belongs on the codegolfing stackexchange because it's not related to how anyone would ever write a script. However, you can do it like this:
cat myfile | grep -w '.*..*'
It's equivalent to the more canonical grep ., but adds explicit .*s on either side so that it will always match the complete line, thereby satisfying the word boundary conditions imposed by -w
You can pipe your output to awk to easily remove empty lines
cat myfile | grep -w #something | awk NF
EDIT: so... you just want cat myfile | awk NF?
if you have to use grep, you can do grep myfile -v '^[[:blank:]]*$'
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
Is grep capable of providing the line number on which the specified word appears?
Also, is possible to use grep to search for a word starting from some certain line downward?
Use grep -n to get the line number of a match.
I don't think there's a way to get grep to start on a certain line number. For that, use sed. For example, to start at line 10 and print the line number and line for matching lines, use:
sed -n '10,$ { /regex/ { =; p; } }' file
To get only the line numbers, you could use
grep -n 'regex' | sed 's/^\([0-9]\+\):.*$/\1/'
Or you could simply use sed:
sed -n '/regex/=' file
Combining the two sed commands, you get:
sed -n '10,$ { /regex/= }' file
You can call tail +[line number] [file] and pipe it to grep -n which shows the line number:
tail +[line number] [file] | grep -n /regex/
The only problem with this method is the line numbers reported by grep -n will be [line number] - 1 less than the actual line number in [file].
Or You can use
grep -n . file1 |tail -LineNumberToStartWith|grep regEx
This will take care of numbering the lines in the file
grep -n . file1
This will print the last-LineNumberToStartWith
tail -LineNumberToStartWith
And finally it will grep your desired lines(which will include line number as in orignal file)
grep regEX
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
Let say i have file like below.
ABC_DEF_G-1_P-249_8.CSV
I want to cut to be like this below.
ABC_DEF_G-1_P-249_
I use this awk command to do that like below.
ls -lrt | grep -i .CSV | tail -1 | awk -F ' ' '{print $8}' | cut -c 1-18
Question is, if the number 1, is growing, how to make the substring is dynamic
example like below...
ABC_DEF_G-1_P-249_
....
ABC_DEF_G-10_P-249_
ABC_DEF_G-11_P-249_
...
ABC_DEF_G-1000_P-249_
To display the file names of all .CSV without everything after the last underscore, you can do this:
for fname in *.CSV; do echo "${fname%_*}_"; done
This removes the last underscore and evertyhing that follows it (${fname%_*}), and then appends an underscore again. You can assign that, for example, to another variable.
For an example file list of
ABC_DEF_G-1_P-249_9.CSV
ABC_DEF_G-10_P-249_8.CSV
ABC_DEF_G-1000_P-249_4.CSV
ABC_DEF_G-11_P-249_7.CSV
ABC_DEF_G-11_P-249_7.txt
this results in
$ for fname in *.CSV; do echo "${fname%_*}_"; done
ABC_DEF_G-1_P-249_
ABC_DEF_G-10_P-249_
ABC_DEF_G-1000_P-249_
ABC_DEF_G-11_P-249_
You can do this with just ls and grep
ls -1rt | grep -oP ".*(?=_\d{1,}\.CSV)"
If you are concerned about the output of ls -1, as mentioned in the comments you can use find as well
find -type f -printf "%f\n" | grep -oP ".*(?=_\d{1,}\.CSV)"
Outputs:
ABC_DEF_G-1_P-249
ABC_DEF_G-1000_P-249_
This assumes you want everything except the _number.CSV, if it needs to be case insensitive then you can the -i flag to the grep. The \d{1,} allows for the number between _ and .CSV to grow from one to many digits. Also doing it this way you don't have to worry about if the number 1 in your example increases:
ABC_DEF_G-1_P-249
You should not be parsing ls. Perhaps you are looking for something like this:
base=$(printf "%s\n" * | grep -i .CSV | tail -1 | awk -F ' ' '{print $8}' | cut -c 1-18)
However, that's a useless use of grep you want to get rid of right there -- Awk does everything grep does, and everything tail does, too, and actually, everything cut does as well. The grep can also be avoided by using a better wildcard, though:
base=$(printf "%s\n" *.[Cc][Ss][Vv] | awk 'END { print substr($8, 1, 18) }')
In the shell itself, you can do much the same thing with no external processes at all. Proposing a suitable workaround would perhaps require a better understanding of what you are trying to accomplish, though.
I have two files containing list of files. I need to check what files are missing in the list of second file. Problem is that I do not have to match full name, but only need to match last 19 Characters of the file names.
E.g
MyFile12343220150510230000.xlsx
and
MyFile99999620150510230000.xlsx
are same files.
This is a unique problem and I don't know how to start. Kindly help.
awk based solution:
$ awk '
{start=length($0) - 18;}
NR==FNR{a[substr($0, start)]++; next;} #save last 19 characters for every line in file2
{if(!a[substr($0, start)]) print $0;} #If that is not present in file1, print that line.
' file2.list file.list
First you can use comm to match the exact file names and obtain a list of files not matchig. Then you can use agrep. I've never used it, but you might find it useful.
Or, as last option, you can do a brute force and for every line in the first file search into the second:
#!/bin/bash
# Iterate through the first file
while read LINE; do
# Find the section of the filename that has to match in the other file
CHECK_SECTION="$(echo "$LINE" | sed -nre 's/^.*([0-9]{14})\.(.*)$/\1.\2/p')"
# Create a regex to match the filenames in the second file
SEARCH_REGEX="^.*$CHECK_SECTION$"
# Search...
egrep "$SEARCH_REGEX" inputFile_2.txt
done < inputFile_1.txt
Here I assumed the filenames end with 14 digits that must match in the other file and a file extension that can be different from file to file but that has to match too:
MyFile12343220150510230000.xlsx
| variable | 14digits |.ext
So, if the first file is FILE1 and the second file is FILE2 then if the intention is only to identify the files in FILE2 that don't exist in FILE1, the following should do:
tmp1=$(mktemp)
tmp2=$(mktemp)
cat $FILE1 | rev | cut -c -19 | sort | uniq > ${tmp1}
cat $FILE2 | rev | cut -c -19 | sort | uniq > ${tmp2}
diff ${tmp1} ${tmp2} | rev
rm ${tmp1} ${tmp2}
In a nutshell, this reverses the characters on each line, and extracts the part you're interested in, saving to a temporary file, for each list of files. The reversal of characters is done since you haven't said whether or not the length of filenames is guaranteed to be constant---the only thing we can rely on here is that the last 19 characters are of a fixed format (in this case, although the format is easily inferred, it isn't really relevant). The sort is important in order for the diff to show you what's not in the second file that is in the first.
If you're certain that there will only ever be files missing from FILE2 and not the other way around (that is, files in FILE2 that don't exist in FILE1), then you can clean things up by removing the cruft introduced by diff, so the last line becomes:
diff ${tmp1} ${tmp2} | rev | grep -i xlsx | sed 's/[[:space:]]\+.*//'
The grep limits the output to those lines with xlsx filenames, and the sed removes everything on a line from the first space encountered onwards.
Of course, technically this only tells you what time-stamped-grouped groups of files exist in FILE1 but not FILE2--as I understand it, this is what you're looking for (my understanding of your problem description is that MyFile12343220150510230000.xlsx and MyFile99999620150510230000.xlsx would have identical content). If the file names are always the same length (as you subsequently affirmed), then there's no need for the rev's and the cut commands can just be amended to refer to fixed character positions.
In any case, to get the final list of files, you'll have to use the "cleaned up" output to filter the content of FILE1; so, modifying the script above so that it includes the "cleanup" command, we can filter the files that you need using a grep--the whole script then becomes:
tmp1=$(mktemp)
tmp2=$(mktemp)
missing=$(mktemp)
cat $FILE1 | rev | cut -c -19 | sort | uniq > ${tmp1}
cat $FILE2 | rev | cut -c -19 | sort | uniq > ${tmp2}
diff ${tmp1} ${tmp2} | rev | grep -i xlsx | sed 's/[[:space:]]\+.*//' > ${missing}
grep -E "("`echo $(<${missing}) | sed 's/[[:space:]]/|/g'`")" ${tmp1}
rm ${tmp1} ${tmp2} ${missing}
The extended grep command (-E) just builds up an "or" regular expression for each timestamp-plus-extension and applies it to the first file. Of course, this is all assuming that there will never be timestamp-groups that exist in FILE2 and not in FILE1--if this is the case, then the "diff output processing" bit needs to be a little more clever.
Or you could use your standard coreutil tools:
for i in $(cat file1 file2 | sort | uniq -u); do
grep -q "$i" f1.txt && \
echo "f2 missing '$i'" || \
echo "f1 missing '$i'"
done
It will identify which non-common entries are missing from which file. You can also manipulate the non-common filenames in any way you like, e.g. parameter expansion/substring extraction, substring removal, or character indexes.
I am new to shell scripting so i need some help need how to go about with this problem.
I have a directory which contains files in the following format. The files are in a diretory called /incoming/external/data
AA_20100806.dat
AA_20100807.dat
AA_20100808.dat
AA_20100809.dat
AA_20100810.dat
AA_20100811.dat
AA_20100812.dat
As you can see the filename of the file includes a timestamp. i.e. [RANGE]_[YYYYMMDD].dat
What i need to do is find out which of these files has the newest date using the timestamp on the filename not the system timestamp and store the filename in a variable and move it to another directory and move the rest to a different directory.
For those who just want an answer, here it is:
ls | sort -n -t _ -k 2 | tail -1
Here's the thought process that led me here.
I'm going to assume the [RANGE] portion could be anything.
Start with what we know.
Working Directory: /incoming/external/data
Format of the Files: [RANGE]_[YYYYMMDD].dat
We need to find the most recent [YYYYMMDD] file in the directory, and we need to store that filename.
Available tools (I'm only listing the relevant tools for this problem ... identifying them becomes easier with practice):
ls
sed
awk (or nawk)
sort
tail
I guess we don't need sed, since we can work with the entire output of ls command. Using ls, awk, sort, and tail we can get the correct file like so (bear in mind that you'll have to check the syntax against what your OS will accept):
NEWESTFILE=`ls | awk -F_ '{print $1 $2}' | sort -n -k 2,2 | tail -1`
Then it's just a matter of putting the underscore back in, which shouldn't be too hard.
EDIT: I had a little time, so I got around to fixing the command, at least for use in Solaris.
Here's the convoluted first pass (this assumes that ALL files in the directory are in the same format: [RANGE]_[yyyymmdd].dat). I'm betting there are better ways to do this, but this works with my own test data (in fact, I found a better way just now; see below):
ls | awk -F_ '{print $1 " " $2}' | sort -n -k 2 | tail -1 | sed 's/ /_/'
... while writing this out, I discovered that you can just do this:
ls | sort -n -t _ -k 2 | tail -1
I'll break it down into parts.
ls
Simple enough ... gets the directory listing, just filenames. Now I can pipe that into the next command.
awk -F_ '{print $1 " " $2}'
This is the AWK command. it allows you to take an input line and modify it in a specific way. Here, all I'm doing is specifying that awk should break the input wherever there is an underscord (_). I do this with the -F option. This gives me two halves of each filename. I then tell awk to output the first half ($1), followed by a space (" ")
, followed by the second half ($2). Note that the space was the part that was missing from my initial suggestion. Also, this is unnecessary, since you can specify a separator in the sort command below.
Now the output is split into [RANGE] [yyyymmdd].dat on each line. Now we can sort this:
sort -n -k 2
This takes the input and sorts it based on the 2nd field. The sort command uses whitespace as a separator by default. While writing this update, I found the documentation for sort, which allows you to specify the separator, so AWK and SED are unnecessary. Take the ls and pipe it through the following sort:
sort -n -t _ -k 2
This achieves the same result. Now you only want the last file, so:
tail -1
If you used awk to separate the file (which is just adding extra complexity, so don't do it sheepish), you can replace the space with an underscore again with sed:
sed 's/ /_/'
Some good info here, but I'm sure most people aren't going to read down to the bottom like this.
This should work:
newest=$(ls | sort -t _ -k 2,2 | tail -n 1)
others=($(ls | sort -t _ -k 2,2 | head -n -1))
mv "$newest" newdir
mv "${others[#]}" otherdir
It won't work if there are spaces in the filenames although you could modify the IFS variable to affect that.
Try:
$ ls -lr
Hope it helps.
Use:
ls -r -1 AA_*.dat | head -n 1
(assuming there are no other files matching AA_*.dat)
ls -1 AA* |sort -r|tail -1
Due to the naming convention of the files, alphabetical order is the same as date order. I'm pretty sure that in bash '*' expands out alphabetically (but can not find any evidence in the manual page), ls certainly does, so the file with the newest date, would be the last one alphabetically.
Therefore, in bash
mv $(ls | tail -1) first-directory
mv * second-directory
Should do the trick.
If you want to be more specific about the choice of file, then replace * with something else - for example AA_*.dat
My solution to this is similar to others, but a little simpler.
ls -tr | tail -1
What is actually does is to rely on ls to sort the output, then uses tail to get the last listed file name.
This solution will not work if the filename you require has a leading dot (e.g. .profile).
This solution does work if the file name contains a space.