Cleanup a path list in bash, removing all children paths (subdirectories) - bash

Question

I have a file containing a list of paths like the following:
/some/path
/some/path/file
/some/path/subpath/file
/some/otherpath
/some/otherpath/file
As of today, I iterate of that list, check if the path exists, and if so, I delete the path / file.
This works, but isn't very optimized since once a directory is deleted, I can safely assume that all of it's children are deleted too.
I also happen to use that list in rsync as exclude list, which is quite cpu extensive because the list can be quite big.
I'd like to clean that list before using it, meaning that if /some/path exists, all children paths ie /some/path/* can be safely removed from the list.
The result list of the example above should look like
/some/path
/some/otherpath
The list is already sorted, meaning there won't be a case like
/some/path/file
/some/path
What's the fastest way to do so in GNU bash ?
Thanks.
[EDIT]
Source list is generated as follows:
rsync creates a list of files in paths, using grep and sed to 'clean' rsync output
# rsync operation explanation
# (command || :) = Return code 0 regardless of command return code
# (grep -E \"^-|^d|^l\" || :) = Be sure line begins with '-' or 'd' or 'l' (rsync semantics for file, directory or symlink)
# (sed -r 's/^.{10} +[0-9,]+ [0-9/]{10} [0-9:]{8} //' || :) = Remove everything before timestamps
# (awk 'BEGIN { FS=\" -> \" } ; { print \$1 }' || :) = Only show output before ' -> ' in order to remove symlink destinations
# (grep -v \"^\.$\" || :) = Removes line containing current directory sign '.'
rsync --list-only -rlptgoDE8 /path1 | (grep -E \"^-|^d|^l\" || :) |
(sed -r 's/^.{10} +[0-9,]+ [0-9/]{10} [0-9:]{8} //' || :) |
(awk 'BEGIN { FS=\" -> \" } ; { print \$1 }' || :) |
(grep -v \"^\.$\" || :) | sort > /tmp/path1_list
rsync --list-only -rlptgoDE8 /path2 | (grep -E \"^-|^d|^l\" || :) |
(sed -r 's/^.{10} +[0-9,]+ [0-9/]{10} [0-9:]{8} //' || :) |
(awk 'BEGIN { FS=\" -> \" } ; { print \$1 }' || :) |
(grep -v \"^\.$\" || :) | sort > /tmp/path2_list
comm -23 /tmp/path1_list /tmp/path2_list > final_list
Purpose of final_list is to have a list of files which are present in /path1 but not in /path2
[/EDIT]
[EDIT2]
I use rsync to create the file lists because I need to honor rsync exclusion patterns, which I can't with other utilities, hence the whole rsync decoding used for list generation.
The whole project is about statefull file synchronization hosted at https://github.com/deajan/osync
[/EDIT2]
[EDIT3]
michael's answer based on awk works great, except for specific corner cases like:
/some/path
/some/path-whatever
/some/path/file
/some/path/subpath/file
/some/otherpath
/some/otherpath/file
Overall, I could "dedupe" some of my lists from 48k lines to 50. Not a perfect solution, but does the job so far.
[/EDIT3]

Answer 1

Considering that a sorted file of paths would have redundant paths listed after matching substrings, e.g.,
/one
/one/two # dupe, matches /one
/one/two/three # dupe, matches /one
/two/three
/two/three/four # dupe, matches /two/three
Then if you go thru the file, and if the current line contains the substring above it (or, specifically, the shortest substring above it), then just skip those lines:
LC_COLLATE=C sort -u file.txt | awk '
BEGIN { prev="^dummy/" }
$0 ~ prev { print "# skip: " $0; next }
$0 !~ prev { print $0; prev="^"$0"/" }'
This prefixes lines to skip with # so you can see what's omitted, feel free to remove that once you verify this might work.
Notes:
I'm ignoring the difference between files and directories, because checking would be messy & a lot slower.
Also, /path/to/file should not cause /path/to/file2 to be skipped, even though it's a substring as coded, hence I use a regex like ^string/ to assume every entry is a directory to avoid that problem.
Technically, having both pattern matches is probably redundant, unless you need to tweak them.
Maybe make sure there are no blank lines or other oddities in the input, e.g., sort input.txt | grep '^/' | awk ...
I added LC_COLLATE=C so that /path/one/ would sort ahead of /path/one2, otherwise the original assumption above (sorted substrings indicates subdirs) doesn't hold. (It might not hold in other cases, as well, but then at least there still will be fewer duplicates, but still some duplicates.) I just noticed this problem right as I was posting, and perhaps you'll discover other corner cases, so please do test :-)

Answer 2

You need two while loops. The outer while loop reads the base paths. And the inner while loops reads the lines to drop.
#! /bin/bash
exec 0< full-list
exec 1> reduced-list
read -r line1
while :; do
echo "$line1"
while :; do
if read -r line2; then
case $line2 in
"$line1"/*)
continue
;;
*)
line1=$line2
break
;;
esac
else
exit 0
fi
done
done
The above code assumes that there is at least one line in the full list. If this can not be guaranteed, you have to add an additional if statement after the first read.