replace terms with associated abbreviations from other file, in case of matching - macos

Question

I have two files:
1. Pattern file = pattern.txt
2. File containing different terms = terms.txt
pattern.txt contain two columns, separated by ;
In the first column I have several terms and in the second column abbreviations,
associated to the first column, same line.
terms.txt contain single words and terms defined by single words but also
by a combination of words.
pattern.txt
Berlin;Brln
Barcelona;Barcln
Checkpoint Charly;ChckpntChrl
Friedrichstrasse;Fridrchstr
Hall of Barcelona;HllOfBarcln
Paris;Prs
Yesterday;Ystrdy
terms.txt
Berlin
The Berlinale ended yesterday
Checkpoint Charly is still in Friedrichstrasse
There will be a fiesta in the Hall of Barcelona
Paris is a very nice city
The target is to replace terms with standardised abbreviations and to find out which terms
have no abbreviation.
As result I would like to have two files.
The first file is a new terms file, with terms replaced by abbreviations where it could be replaced.
The second file containing a list with all terms that doesn't have an abbreviation.
The output is case insensitive, I don't make difference between "The" and "the".
new_terms.txt
Brln
The Berlinale ended Ystrdy
ChckpntChrl is still in Fridrchstr
There will be a fiesta in the HllOfBarcln
Prs is a very nice city
terms_without_abbreviations.txt
a
be
Berlinale
city
ended
fiesta
in
is
nice
of
still
The
There
very
will
I will appreciate your help and thanks in advance for your time and hints!

Answer 1

This is mostly what you need:
BEGIN { FS=";"; }
FNR==NR { dict[tolower($1)] = $2; next }
{
line = "";
count = split($0, words, / +/);
for (i = 1; i <= count; i++) {
key = tolower(words[i]);
if (key in dict) {
words[i] = dict[key];
} else {
result[key] = words[i];
}
line = line " " words[i];
}
print substr(line, 2);
}
END {
count = asorti(result, sorted);
for (i = 1; i <= count; i++) {
print result[sorted[i]];
}
}

Answer 2

Ok, so I had a bit of a crack, but will explain issues:
If you have multiple changes in pattern.txt that can pertain to a single line, the first change will make its change and the second will not (eg. Barcelona;Barcln and Hall of Barcelona;HllOfBarcln, obviously if Barcln has already been done when you get to the longer version it will not longer exist and so no change made)
Similar to above, there is no abbreviation for the word 'Hall' so again if we assume above is true and only the first change was made, your new file for changes will include hall as not having an abbreviation
#!/usr/bin/awk -f
BEGIN{
FS = ";"
IGNORECASE = 1
}
FNR == NR{
abbr[tolower($1)] = $2
next
}
FNR == 1{ FS = " " }
{
for(i = 1; i <= NF; i++){
item = tolower($i)
if(!(item in abbr) && !(item in twa)){
twa[item]
print item > "terms_without_abbreviations.txt"
}
}
for(i in abbr)
gsub("\\<"i"\\>", abbr[i])
print > "new_terms.txt"
}
There are probably other gotchas to look for but it is a vague direction. Not sure how you would get around my points above??