storage in awk when used within shell script - shell

Question

I am writing a shell script program in which I am internally calling an awk script. Here is my script below.
for FILE in `eval echo{0..$fileIterator}`
{
if(FILE == $fileIterator)
{
printindicator =1;
}
grep RECORD FILEARRAY[FILE]| awk 'for(i=1;i<=NF;i++) {if($i ~ XXXX) {XARRAY[$i]++}} END {if(printIndicator==1){for(element in XARRAY){print element >> FILE B}}'
I hope I am clear with my code . Please let me know if you need any other details.
ISSUE
My motivation in this program is to traverse through all the files an get the lines that has "XXXX" in all the files and store the lines in an array. That is what I am doing here. Finally I need to store the contents of the array variable into a file. I can store the contents at each and every step like the below
{if($i ~ XXXX) {XARRAY[$i]++; print XARRAY[$i] >> FILE B}}
But the reason behind not going to this approach is each time I need to do an I/O operation and for this the time taken is much and that is why I am converting that into inmemory everytime and then at last dumping the in memory array(XARRAY) into the file.
The problem I am facing here is that. The shell script calls the awk everytime, the data's are getting stored in the array(XARRAY) but for the next iteration, the previous content of XARRAY is getting deleted and it puts the new content as this assumes this as a new array. Hence at last when I print the contents, it prints only the lately updated XARRAY and not all the data that is expected from this.
SUGGESTIONS EXPECTED
1) How to make the awk script realize that the XARRAY is an old one and not the new one when it is being called everytime in each iteration.
2) One of the alternative is to do an I/O everytime. But I am not interested in this. Is there any other alternative other than this. Thank you.

Answer 1

This post involves combining shell script and awk script to solve a problem. This is very often a useful approach, as it can leverage the strengths of each, and potentially keep the code from getting ugly in either!
You can indeed "preserve state" with awk, with a simple trick: leveraging a coprocess from the shell script (bash, ksh, etc. support coprocess).
Such a shell script launches one instance of awk as a coprocess. This awk instance runs your awk code, which continuously processes its lines of input, and accumulates stateful information as desired.
The shell script continues on, gathering up data as needed, and passes data to the awk coprocess whenever ready. This can run in a loop, potentially blocking or sleeping, potentially acting as a long-running background daemon. Highly versatile!
In your awk script, you need a strategy for triggering the output of the stateful data it has been accumulating. The simplest, is to have an END{} action which triggers when awk stdin closes. If you need output data sooner than that, at each line of input the awk code has a chance to output its data.
I have successfully used this approach many times.

Answer 2

Ouch, can't tell if it is meant to be real or pseudocode!
You can't make awk preserve state. You would either have to save it to a temporary file or store it in a shell variable, the contents of which you'd pass to later invocations. But this is all too much hassle for what I understand you want to achieve.
I suggest you omit the loop, which will allow you to call awk only once with just some reordering. I assume FILE A is the FILE in the loop and FILE B is something external. The reordering would end up something very roughly like:
grep RECORD ${FILEARRAY[#]:0:$fileIterator} | awk 'for(i=1;i<=NF;i++) {if($i ~ XXXX) {XARRAY[$i]++}} END {for(element in XARRAY){print element >> FILEB}'
I move the filename expansion to the grep call and removed the whole printIndicator check.
It could all be done even more efficiently (the obvious one being removal of grep), but you provided too little detail to make early optimisation sensible.
EDIT: fixed the loop iteration with the info from the update. Here's a loopy solution, which is immune to new whitespace issues and too long command lines:
for FILE in $(seq 0 $fileIterator); do
grep RECORD "${FILEARRAY[$FILE]}"
done |
awk 'for(i=1;i<=NF;i++) {if($i ~ XXXX) {XARRAY[$i]++}} END {for(element in XARRAY){print element >> FILEB}'
It still runs awk only once, constantly feeding it data from the loop.
If you want to load the results into an array UGUGU, do the following as well (requires bash 4):
mapfile UGUGU < FILEB

Answer 3

results=$(for loop | awk{for(element in XARRAY)print element})..
I declared result as an array so for every "element" that is being printed it should store in results[1], results[2].
But instead of this, it is performing the below ...
Lets assume
element = "I am fine"(First iteration of for loop),
element = "How are you" (Second iteration of for loop).
My expected result in accordance to this is,
results[1]= "I am fine" and results[2] = "How are you" ,
but the output I am getting is results[1]= "I" results[2]= "am". I dont know why it is delimiting by space .. Any suggestions regarding this