How to fast sum values in directed graph in shell? - bash

Question

I have a directed graph with like 2000 nodes stored in a file. Each line represents an edge from the node stored in the first column to the node stored in the second column, it is even easy to visualize the data for example in dot(1). Columns are separated by tabs, rows separated by newlines and nodes are named with any of the a-zA-Z0-9_ characters. Tree can have multiple roots, it may have cycles, which should be ignored. I don't care about cycles, they are redundant, but they can happen in the input. Below I am presenting an example of the graph, with tr to substitute spaces for tabs and here-document, to easy reproduce the input file:
tr ' ' '\t' <<EOF >connections.txt
str1 str2
str2 str3
str3 str4
str100 str2
str100 str101
str101 str102
EOF
I have also a list of some node in the graph, called heads. These will be the starting nodes, ie. heads:
tr ' ' '\t' <<EOF >heads.txt
str1
str100
EOF
And I have also a list of associated "cost" with each node. Example with some random data:
tr ' ' '\t' <<EOF >cost.txt
str1 1
str2 5
str3 10
str4 548
str100 57
str101 39
str102 23
EOF
I want to sum the "cost" of each node while traversing the tree from nodes stored in head.txt and print the cost with some traversing information for each leaf.
I want to:
for each node in heads.txt
sum the cost from costs.txt of the node into some variable
find that node in connections.txt
find what does this node connect to
and repeat the algorithm for each of the nodes the node connects to
when the node is connected with nothing, print the sum of costs
Ideally the script would look like:
$ script.sh heads.txt connections.txt cost.txt
str1->str2->str3->str4 1+5+10+548 564
str100->str2->str3->str4 57+5+10+548 620
str100->str101->str102 57+39+23 119
And I even have written this, and it works:
#!/bin/bash
set -euo pipefail
headsf=$1
connectionsf=$2
costf=$3
get_cost() {
grep "^$1"$'\t' "$costf" | cut -f2 || echo 0
}
get_conn() {
grep "^$1"$'\t' "$connectionsf" | cut -f2
}
check_conns() {
grep -q "^$1"$'\t' "$connectionsf"
}
f_output() {
printf "%s\t%s\n" "$1" "$2"
}
f() {
local func cost
func="$1"
cost=$(get_cost "$func")
if ! check_conns "$func"; then
f_output "${2:+$2->}$func" "${3:+$3+}$cost"
return
fi
get_conn "$func" |
while IFS=$'\t' read -r calls; do
if [ "$func" = "$calls" ]; then
echo "$func is recursive" >&2
continue
fi
if <<<"$2" grep -q -w "$calls"; then
printf "$2 calls recursive $calls\n" >&2
continue
fi
f "$calls" "${2:+$2->}$func" "${3:+$3+}$cost"
done
}
while IFS= read -r head; do
f "$head" "" ""
done < "$headsf" |
while IFS=$'\t' read -r func calc; do
tmp=$(<<<$calc bc)
printf "%s\t%s\t%s\n" "$func" "$calc" "$tmp"
done |
column -t -s $'\t'
However it is impossibly slow on bigger inputs. Even with sample files here (only 6 lines) the script takes 200ms on my machine. How can I speed it up? Can the inputs be sorted, joined somehow to speed it up (grep doesn't care if the input is sorted)? Can this be done faster in awk or other unix tools?
I would like to limit myself to bash shell and standard *unix tools, coreutils, moreutils, datamash and such. I tried doing it in awk, but failed, I have no idea how to do find something recursively in the input in awk. It this feels to me "doable" in a shell script really fast.

Answer 1

Since no one has posted an answer yet, here is an awk solution as a starting point:
#!/usr/bin/awk -f
BEGIN {
FS=OFS="\t"
}
FILENAME=="connections.txt" {
edges[$1,++count[$1]]=$2
next
}
FILENAME=="cost.txt" {
costs[$1]=$2
next
}
FILENAME=="heads.txt" {
f($1)
}
function f(node,
path,cost,sum,prev,sep1,sep2,i) {
if(node in prev)
# cycle detected
return
path=path sep1 node
cost=cost sep2 costs[node]
sum+=costs[node]
if(!count[node]) {
print path,cost,sum
}
else {
prev[node]
for(i=1;i<=count[node];++i)
f(edges[node,i],path,cost,sum,prev,"->","+")
delete prev[node]
}
}
Make it read connections.txt and cost.txt before heads.txt.
Its output (padded):
$ awk -f tst.awk connections.txt cost.txt heads.txt
str1->str2->str3->str4 1+5+10+548 564
str100->str2->str3->str4 57+5+10+548 620
str100->str101->str102 57+39+23 119

Answer 2

You say you want only standard tools, but you also mention using dot on your data, so I'm assuming you have the other graphviz utilities available... in particular, gvpr, which is like awk for graphs:
#!/usr/bin/env bash
graph=$(mktemp)
join -t$'\t' -j1 -o 0,1.2,2.2 -a2 \
<(sort -k1,1 connections.txt) \
<(sort -k1,1 cost.txt) |
awk -F$'\t' 'BEGIN { print "digraph g {" }
{ printf "%s [cost = %d ]\n", $1, $3
if ($2 != "") printf "%s -> %s\n", $1, $2 }
END { print "}" }' > "$graph"
while read root; do
gvpr -a "$root" '
BEGIN {
int depth;
int seen[string];
string path[int];
int costs[int];
}
BEG_G {
$tvtype = TV_prepostfwd;
$tvroot = node($, ARGV[0]);
}
N {
if ($.name in seen) {
depth--;
} else {
seen[$.name] = 1;
path[depth] = $.name;
costs[depth] = $.cost;
depth++;
if (!fstout($) && path[0] == ARGV[0]) {
int i, c = 0;
for (i = 0; i < depth - 1; i++) {
printf("%s->", path[i]);
}
printf("%s\t", $.name);
for (i = 0; i < depth - 1; i++) {
c += costs[i];
printf("%d+", costs[i]);
}
c += $.cost;
printf("%d\t%d\n", $.cost, c);
}
}
}' "$graph"
done < heads.txt
rm -f "$graph"
Running this after creating your data files:
$ ./paths.sh
str1->str2->str3->str4 1+5+10+548 564
str100->str2->str3->str4 57+5+10+548 620
str100->str101->str102 57+39+23 119
Or, since it's so ubiquitous it might as well be standard, a sqlite-based solution. This one doesn't even require bash/zsh/ksh93, unlike the above.
$ sqlite3 -batch -noheader -list <<EOF
.separator "\t"
CREATE TABLE heads(node TEXT);
.import heads.txt heads
CREATE TABLE costs(node TEXT PRIMARY KEY, cost INTEGER) WITHOUT ROWID;
.import cost.txt costs
CREATE TABLE connections(from_node TEXT, to_node TEXT
, PRIMARY KEY(from_node, to_node)) WITHOUT ROWID;
.import connections.txt connections
WITH RECURSIVE paths(tail, path, costs, cost) AS
(SELECT h.node, h.node, c.cost, c.cost
FROM heads AS h
JOIN costs AS c ON h.node = c.node
UNION ALL
SELECT conn.to_node, p.path || '->' || conn.to_node
, p.costs || '+' || c.cost, p.cost + c.cost
FROM paths AS p
JOIN connections AS conn ON conn.from_node = p.tail
JOIN costs AS c ON c.node = conn.to_node
)
SELECT path, costs, cost FROM paths AS p
WHERE tail NOT IN (SELECT from_node FROM connections)
ORDER BY path;
EOF
str1->str2->str3->str4 1+5+10+548 564
str100->str101->str102 57+39+23 119
str100->str2->str3->str4 57+5+10+548 620