How to replace nth field in a csv file with mapped value from another file? - bash

Question

I have a csv file in the following format:
23:56:00,5,1,7,99,100,101
23:56:30,5,1,7,98,199,191
23:57:00,6,1,6,99,99,98
23:57:30,5,2,6,97,99,199
...
And a map file in the following format:
1:10
2:12
3:30
4:aa
5:16
6:11
7:bb
What I'm trying to accomplish is to replace the fields in columns 2,3 and 4 in the first csv files with the values they map to in the map file.
For example in the above case the final output I want is this:
23:56:00,16,10,bb,99,100,101
23:56:30,16,10,bb,98,199,191
23:57:00,11,10,11,99,99,98
23:57:30,16,12,11,97,99,199
What would be the best way to do this? I was trying to figure out a way using awk/sed but I'm not sure how to access multiple files inside awk, and if that is even the best way to do this. There will be a lot of repetitions since its a large file so I don't think checking for a mapping each time is the right way to do this.
Is there a way to store the map in to a hash table inside the shell script, and then replace using the hash mapping?

Answer 1

Try with:
awk '
BEGIN { FS = OFS = "," }
FNR == NR {
split($0, f, /:/)
map[f[1]] = f[2]
next
}
{
for (i=2; i<=4; i++) {
if ($i in map) { $i = map[$i] }
}
}
{ print }
' mapfile csvfile
It reads the map file first and saves data in an associative array that is compared with fields 2, 3 and 4 from the csv file. The result yields:
23:56:00,16,10,bb,99,100,101
23:56:30,16,10,bb,98,199,191
23:57:00,11,10,11,99,99,98
23:57:30,16,12,11,97,99,199

Answer 2

One pure Bash possibility (with Bash version≥4):
Slurp the map file in an associative array and process your csv file:
#!/bin/bash
declare -A map=()
while IFS=: read -r k v; do
[[ -z "$k$v" ]] && continue # ignore empty lines
map[$k]=$v
done < mapfile.txt
IFS=,
while read -r -a ary; do
[[ -z "${ary[#]}" ]] && continue # ignore empty lines
ary[1]=${map[${ary[1]}]}
ary[2]=${map[${ary[2]}]}
ary[3]=${map[${ary[3]}]}
echo "${ary[*]}"
done < csvfile.txt
If the keys in your map file are non-negative integers, you don't need associative arrays, and just replace the line declare -A map=() with map=().
It might not be the most efficient since Bash is not the fastest to process data, but it works well!
Btw, there are no error checkings whatsoever, so be sure you apply this script to well-formated files.
On your example, this yields:
23:56:00,16,10,bb,99,100,101
23:56:30,16,10,bb,98,199,191
23:57:00,11,10,11,99,99,98
23:57:30,16,12,11,97,99,199

Answer 3

Perl solution. Hashes exist in the recent versions of bash, but I prefer a real programming language when working with them.
#!/usr/bin/perl
use warnings;
use strict;
open my $MAP, '<', '1.map' or die $!;
my %map;
while (<$MAP>) {
chomp;
my ($key, $value) = split /:/;
$map{$key} = $value;
}
open my $CSV, '<', '1.csv' or die $!;
while (<$CSV>) {
my #fields = split /,/;
s/(.*)/$map{$1}/ for #fields[1, 2, 3];
print join ',' => #fields;
}

Answer 4

Another awk
awk -F",|:" 'FNR==NR {a[$1]=$2;next} {print $1":"$2":"$3,a[$4],a[$5],a[$6],$7,$8,$9}' OFS=, map csv
23:56:00,16,10,bb,99,100,101
23:56:30,16,10,bb,98,199,191
23:57:00,11,10,11,99,99,98
23:57:30,16,12,11,97,99,199