multi string search and replace - bash

Question

I have 2 text files
File1 has more than 400K lines. Each line is similar to this sample:
hstor,table,"8bit string",ABCD,0000,0,4,19000101,today
File2 has a list of new 8bit strings to replace the current ones in file1 while preserving the rest in file1.
So file1 goes from
hstor,table,"OLD 8bit string",ABCD,0000,0,4,19000101,today
to
hstor,table,"NEW 8bit string",ABCD,0000,0,4,19000101,today
I can't sed 400K times
How can I script this so that all the OLD 8bit strings in file1 are replaced with the NEW 8bit strings listed in file2?

Answer 1

This might work for you (GNU sed):
sed 's#.*#s/[^,]*/&/3#' file2 | cat -n | sed -f - file1
This converts file2 into a sed script file and then runs it on file1.
The first sed script takes each line in file2 and changes it to substitution command which replaces the third field in the target with the contents of the current line of file2.
This is piped into a cat command which inserts line numbers which will be used by the sed script to address each substitution command.
The final sed command uses the /dev/stdin to read in a sed script and runs it against the input file file1.

Answer 2

In case you need to do this multiple times and performance is important, I wrote a program in C to do this. It's a modified version of this code. I know you did not use any C-tag, but I got the impression that your main concern was to just get the job done.
NOTE:
I take no responsibility for it. It is quite a bit of a quick hack, and I do assume some stuff. One assumption is that the string you want to replace does not contain any commas. Another is that no line is longer than 100 bytes. A third assumption is that the input files are named file and rep respectively. If you want to try it out, make sure to inspect the data afterwards. It writes to stdout so you just redirect the output to a new file. It does the job in around two seconds.
Here is the code:
#include <stdio.h>
#include <stdlib.h>
#include <memory.h>
int main()
{
/* declare a file pointer */
FILE *infile;
FILE *replace;
char *buffer;
char *rep_buffer;
long numbytes;
long rep_numbytes;
/* open an existing file for reading */
infile = fopen("file", "r");
replace = fopen("rep", "r");
/* quit if the file does not exist */
if(infile == NULL)
return 1;
if(replace == NULL)
return 1;
/* Get the number of bytes */
fseek(infile, 0L, SEEK_END);
numbytes = ftell(infile);
fseek(replace, 0L, SEEK_END);
rep_numbytes = ftell(replace);
/* reset the file position indicator to
the beginning of the file */
fseek(infile, 0L, SEEK_SET);
fseek(replace, 0L, SEEK_SET);
/* grab sufficient memory for the
buffer to hold the text */
buffer = (char*)calloc(numbytes, sizeof(char));
rep_buffer = (char*)calloc(rep_numbytes, sizeof(char));
/* memory error */
if(buffer == NULL)
return 1;
if(rep_buffer == NULL)
return 1;
/* copy all the text into the buffer */
fread(buffer, sizeof(char), numbytes, infile);
fclose(infile);
fread(rep_buffer, sizeof(char), rep_numbytes, replace);
fclose(replace);
char line[100]={0};
char *i=buffer;
char *r=rep_buffer;
while(i<&buffer[numbytes-1]) {
int n;
/* Copy from infile until second comma */
for(n=0; i[n]!=','; n++);
n++;
for(; i[n]!=','; n++);
n++;
memcpy(line, i, n);
/* Copy a line from replacement */
int m;
for(m=0; r[m]!='\n'; m++);
memcpy(&line[n], r, m);
/* Skip corresponding text from infile */
int k;
for(k=n; i[k]!=','; k++);
/* Copy the rest of the line */
int l;
for(l=k; i[l]!='\n'; l++);
memcpy(&line[n+m], &i[k], l-k);
/* Next line */
i+=l;
r+=m+1;
/* Print to stdout */
printf("%s", line);
}
/* free the memory we used for the buffer */
free(buffer);
free(rep_buffer);
}