passing bash variable for awk column specifier - bash

Question

There are loads of threads about passing a shell variable to awk, and I've figured that out easily enough, but the variable I want to pass is the column specifier variable ($1,$2 etc)
Given that the shell uses these variables as default command line argument variables as well, this is getting confusing.
In this script I'm just sorting and joining 2 files together, but in order to begin generalising the script a little, I want to be able to specify on the command line, the field in the key file that awk should be taking as its sort-specifier.
What am I doing wrong here? (I'm only just getting to grips with awk and the oneliner was adapted slightly from here.
keyfile="$1"
filetosort="$2"
field="$3"
awk -v a="$field"
paste "$keyfile" <(awk 'NR==FNR{o[FNR]=a; next} {t[$1]=$0} END{for(x=1; x<=FNR; x++){y=o[x]; print t[y]}}' $keyfile $filetosort)
EDIT Added example in/output
Keyfile: (10 random lines from file)
PVClumt18 PAK_2199 PAK_01997
PVClopt2 PAK_2091 PAK_01895
PVCcif7 PAK_1975 PAK_01793
PVClopT12 PAU_02101 PAU_02063
PVCpnf20 PAK_3524 PAK_03184
PVClopt3 PAK_2090 PAK_01894
PVClopT11 PAU_02102 PAU_02064
PVCunit2_11 plu1698 PLT_01726
PVClumT9 afp10 PAU_02198
PVCunit2_17 plu1692 PLT_01720
File to sort:
PAU_02064 1pqx 1pqx_A 37.4 13 0.00035 31.4 >1pqx_A Conserved hypothetical protein; ZR18,structure, autostructure,spins,autoassign, northeast structural genomics consortium; NMR {Staphylococcus aureus subsp} SCOP: d.267.1.1 PDB: 2ffm_A 2m6q_A 2m8w_A
PAK_01997 5ftj 5ftj_A 99.9 1.6e-26 4.2e-31 229.2 >5ftj_A Transitional endoplasmic reticulum ATPase; hydrolase, single-particle, AAA ATPase; HET: ADP OJA; 2.30A {Homo sapiens} PDB: 3cf1_A* 3cf3_A* 3cf2_A* 5ftk_A* 5ftl_A* 5ftm_A* 5ftn_A* 1r7r_A* 5c19_A 5c1b_A* 5c18_A* 3cf0_A*
PAK_01894 3j9q 3j9q_A 99.9 1.8e-29 4.6e-34 215.9 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
PAK_03184 1xju 1xju_A 99.4 4.1e-17 1.1e-21 98.8 >1xju_A Lysozyme; secreted inactive conformation, hydrolase; 1.07A {Enterobacteria phage P1} SCOP: d.2.1.3
PAK_01793 5a3a 5a3a_A 50.8 6 0.00016 31.4 >5a3a_A SIR2 family protein; transferase, P-ribosyltransferase, metalloprotein, NAD-depen lipoylation, regulatory enzyme, rossmann fold; 1.54A {Streptococcus pyogenes} PDB: 5a3b_A* 5a3c_A*
PLT_01720 3ggm 3ggm_A 54.2 4.9 0.00013 26.2 >3ggm_A Uncharacterized protein BT9727_2919; bacillus cereus group., structural genomics, PSI-2, protein structure initiative; 2.00A {Bacillus thuringiensis serovarkonkukian}
PLT_01726 3h2t 3h2t_A 96.8 8e-06 2.1e-10 82.6 >3h2t_A Baseplate structural protein GP6; viral protein, virion; 3.20A {Enterobacteria phage T4} PDB: 3h3w_A 3h3y_A
PAK_01895 3j9q 3j9q_A 100.0 2.5e-35 6.4e-40 248.6 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
PAU_02198 4jiv 4jiv_D 69.6 1.6 4.2e-05 27.5 >4jiv_D VCA0105, putative uncharacterized protein; PAAR-repeat motif, membrane piercing, type VI secretion SYST vibrio cholerae VGRG2; HET: PLM STE ELA; 1.90A {Vibrio cholerae o1 biovar eltor}
PAU_02063 4yap 4yap_A 31.1 20 0.00052 29.1 >4yap_A Glutathione S-transferase homolog; GSH-lyase GSH-dependent; 1.11A {Sphingobium SP} PDB: 4g10_A 4yav_A*
Thus I need to sort and match the rows based on column 3 in the keyfile, and column 1 in the file to sort.
And the resulting file: (The duplication of columns 3 & 4 was something I was planning to sort out after)
PVClumt18 PAK_2199 PAK_01997 PAK_01997 5ftj 5ftj_A 99.9 1.6e-26 4.2e-31 229.2 >5ftj_A Transitional endoplasmic reticulum ATPase; hydrolase, single-particle, AAA ATPase; HET: ADP OJA; 2.30A {Homo sapiens} PDB: 3cf1_A* 3cf3_A* 3cf2_A* 5ftk_A* 5ftl_A* 5ftm_A* 5ftn_A* 1r7r_A* 5c19_A 5c1b_A* 5c18_A* 3cf0_A*
PVClopt2 PAK_2091 PAK_01895 PAK_01895 3j9q 3j9q_A 100.0 2.5e-35 6.4e-40 248.6 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
PVCcif7 PAK_1975 PAK_01793 PAK_01793 5a3a 5a3a_A 50.8 6 0.00016 31.4 >5a3a_A SIR2 family protein; transferase, P-ribosyltransferase, metalloprotein, NAD-depen lipoylation, regulatory enzyme, rossmann fold; 1.54A {Streptococcus pyogenes} PDB: 5a3b_A* 5a3c_A*
PVClopT12 PAU_02101 PAU_02063 PAU_02063 4yap 4yap_A 31.1 20 0.00052 29.1 >4yap_A Glutathione S-transferase homolog; GSH-lyase GSH-dependent; 1.11A {Sphingobium SP} PDB: 4g10_A 4yav_A*
PVCpnf20 PAK_3524 PAK_03184 PAK_03184 1xju 1xju_A 99.4 4.1e-17 1.1e-21 98.8 >1xju_A Lysozyme; secreted inactive conformation, hydrolase; 1.07A {Enterobacteria phage P1} SCOP: d.2.1.3
PVClopt3 PAK_2090 PAK_01894 PAK_01894 3j9q 3j9q_A 99.9 1.8e-29 4.6e-34 215.9 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
PVClopT11 PAU_02102 PAU_02064 PAU_02064 1pqx 1pqx_A 37.4 13 0.00035 31.4 >1pqx_A Conserved hypothetical protein; ZR18,structure, autostructure,spins,autoassign, northeast structural genomics consortium; NMR {Staphylococcus aureus subsp} SCOP: d.267.1.1 PDB: 2ffm_A 2m6q_A 2m8w_A
PVCunit2_11 plu1698 PLT_01726 PLT_01726 3h2t 3h2t_A 96.8 8e-06 2.1e-10 82.6 >3h2t_A Baseplate structural protein GP6; viral protein, virion; 3.20A {Enterobacteria phage T4} PDB: 3h3w_A 3h3y_A
PVClumT9 afp10 PAU_02198 PAU_02198 4jiv 4jiv_D 69.6 1.6 4.2e-05 27.5 >4jiv_D VCA0105, putative uncharacterized protein; PAAR-repeat motif, membrane piercing, type VI secretion SYST vibrio cholerae VGRG2; HET: PLM STE ELA; 1.90A {Vibrio cholerae o1 biovar eltor}
PVCunit2_17 plu1692 PLT_01720 PLT_01720 3ggm 3ggm_A 54.2 4.9 0.00013 26.2 >3ggm_A Uncharacterized protein BT9727_2919; bacillus cereus group., structural genomics, PSI-2, protein structure initiative; 2.00A {Bacillus thuringiensis serovarkonkukian}

Answer 1

When you pass awk -v a="$field", the specification of the awk variable a is only good for that single awk command. You can't expect a to be available in a completely different invocation of awk.
Thus, you need to put it in-place directly:
$ bashvar="2"
$ echo 'foo bar baz' | awk -v awkvar="$bashvar" '{print $awkvar}'
bar
Or in your case:
field=1
awk -v a="$field" '
NR==FNR {
o[FNR]=$a;
next;
}
{ t[$1] = $0 }
END {
for(x=1; x<=FNR; x++) {
y=o[x]
printf("%s\t%s\n", y, t[y])
}
}' "$keyfile" "$filetosort"
Points of note:
Our printf here is emitting both the key and the value, so there's no need to use paste to put the keyfile values back in.
$a is used to treat the awk variable a (assigned from shell variable field) as a variable name itself, and to perform an indirect reference -- thus, looking up the relevant column number.
Always, always quote your shell variables on expansion. Otherwise, you have no way of knowing how many argument to awk will be generated by the expansion of $keyfile -- it could be 0 (if there are no characters in the string not found in IFS); it could be 1, but it could also be a completely unbounded number (input file.txt would become two arguments, input and file.txt; * input * .txt would have each * replaced with a list of files).

Answer 2

I'm going to add this as an answer because it does resolve the question as I posed it, notwithstanding Charles' excellent advice about the (myriad) areas I was going wrong.
Altering the above code with Charles' point about the separate awk commands, I can now invoke the following (sorry yes, it's still using paste)...
#!/bin/bash
keyfile="$1"
filetosort="$2"
indexfield="$3"
paste "$keyfile" <(awk -v field="$indexfield" 'NR==FNR{o[FNR]=$field; next} {t[$1]=$0} END{for(x=1; x<=FNR; x++){y=o[x]; print t[y]}}' "$keyfile" "$filetosort")
I was missing the $ before the variable I was calling in the awk command, which is (part) of why my original code wasn't working, as well as not including the awk variable declaration in a single awk call.
Thus, bash sortandmatch.sh keyfile filetosort 3 produces the output I want:
PVCunit2_5 plu1704 PLT_01732 PLT_01732 4etv 4etv_A 39.0 12 0.00032 27.6 >4etv_A Ryanodine receptor 2; phosphorylation, cardiac, metal transport; 1.65A {Mus musculus}
PVCunit2_4 plu1705 PLT_01733 PLT_01733 3j9q 3j9q_A 99.9 7.2e-30 1.9e-34 219.0 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
XVC_pnf15 XBW1_RS06910 XBW1_RS06910 XBW1_RS06910 1fi0 1fi0_A 69.2 1.7 4.4e-05 22.8 >1fi0_A VPR protein, R ORF protein; helix, viral protein; NMR {Synthetic} SCOP: j.11.1.1
PVCcif7 PAU_01999 PAU_01967 PAU_01967 5a3a 5a3a_A 47.5 7.3 0.00019 30.9 >5a3a_A SIR2 family protein; transferase, P-ribosyltransferase, metalloprotein, NAD-depen lipoylation, regulatory enzyme, rossmann fold; 1.54A {Streptococcus pyogenes} PDB: 5a3b_A* 5a3c_A*
PVClumT15 PAU_02233 PAU_02192 PAU_02192 1tdp 1tdp_A 22.1 37.0 0.00096 27.2 >1tdp_A Carnobacteriocin B2 immunity protein; four-helix bundle, antimicrobial protein; NMR {Carnobacterium maltaromaticum} SCOP: a.29.8.1
XVC_pnf3 XBW1_RS06850 XBW1_RS06850 XBW1_RS06850 3eaa 3eaa_A 87.7 0.13 3.4e-06 35.7 >3eaa_A EVPC; T6SS, unknown function; 2.79A {Edwardsiella tarda}
PVCunit1_4 afp4 PAU_02778 PAU_02778 3j9q 3j9q_A 99.9 3.6e-29 9.5e-34 214.6 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
PVCunit2_3 plu1706 PLT_01734 PLT_01734 3j9q 3j9q_A 100.0 1.6e-34 4.3e-39 253.7 >3j9q_A Sheath; pyocin, bacteriocin, sheath, structural protein; 3.50A {Pseudomonas aeruginosa}
PVClumt17 PAK_2200 PAK_01998 PAK_01998 3k8p 3k8p_C 34.7 16.0 0.00041 34.1 >3k8p_C DSL1, KLLA0C02695P; intracellular trafficking, DSL1 complex, multisubunit tethering complex, snare proteins; 2.60A {Kluyveromyces lactis}
PVClopT12 PAU_02101 PAU_02063 PAU_02063 4yap 4yap_A 31.1 20 0.00052 29.1 >4yap_A Glutathione S-transferase homolog; GSH-lyase GSH-dependent; 1.11A {Sphingobium SP} PDB: 4g10_A 4yav_A*