Speed up data comparison in powershell acquired via Import-CSV - performance

Question

Simple question, but tough problem.
I have 2 CSV files exported from Excel, one with 65k rows and one with about 50k. I need to merge the data from those 2 files based on that condition :
where File1.Username -eq File2.Username
Note that the datatype for the username property in both files is this :
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
And obviously looping through 65k x 50k objects properties to compare takes..well, 1 day and 23 hours as I estimated when I measured a script run on only 10 rows.
I am considering several solutions at this point, like splitting the CSV files and running different portions in different powershell sessions simultaneously while giving real time priority to powershell.exe but that's cumbersome, and I haven't tested that option so I can't report on the real gain of performance.
I wondered if I should change the datatype rather, and use for instance .ToString.GetHashCode() but I tried that option too and oddly enough the execution time was quicker when comparing string VS string than hash sum integer VS hash sum integer.
So long story short, I am looking for a superfast way to compare 65k x 50k string variables.
Any help would be greatly appreciated :)
Thanks!
Elaborating example:
Ok here's a metaphorical example. Suppose you have a database containing the names and equipment of astronauts (SPACE), and another one containing the names and equipment of
marine explorers(OCEAN).
So in the SPACE dataset you'll have for instance:
First Name,Last name, Username, space gear,environment.
And then the first row of data would be like :
Neil,Armstrong,Stretch,spacesuit,moon
In the OCEAN Dataset you'd have :
First Name,Last name, Username, birthdate, diving gear,environment
with the following data:
Jacques,Cousteau,Jyc,1910-06-11,diving suit,ocean
Now suppose that at some point Neil Armstrong had himself registered to a diving course and was added the the OCEAN dataset.
In the OCEAN Dataset you'd now have :
First Name,Last name, Username, birthdate, diving gear,environment
with the following data:
Jacques,Cousteau,Jyc,1910-06-11,diving suit,ocean
Neil,Armstrong,Stretch,1930-08-05,diving suit,ocean
The person who handed me the data over gave me a third dataset which was a "mix" of the other 2 :
In the MIXED Dataset you'd now have :
Dataset,First Name,Last name, Username, birthdate, diving gear, space gear,environment
with the following data:
ocean,Jacques,Cousteau,Jyc,1910-06-11,diving suit,,ocean
space,Neil,Armstrong,Stretch,1930-08-05,,space suit,moon
ocean,Neil,Armstrong,Stretch,1930-08-05,diving suit,,ocean
So my task is to make the dataset MIXED looking like this:
First Name,Last name, Username, birthdate, diving gear, space gear,environment
Jacques,Cousteau,Jyc,1910-06-11,diving suit,,ocean
Neil,Armstrong,Stretch,1930-08-05,diving suit,space suit,(moon,ocean)
And to top it all off, there's a couple of profoundly stupid scenarios that can happen:
1) A same guy could be in either SPACE Dataset or OCEAN Dataset more than once, but with different usernames.
2) Two completely different users could share the same username in the SPACE Dataset, but NOT in the OCEAN Dataset.User names there are unique. Yes, you read that correctly, both Cousteau and Armstrong could potentially have the same username.
I've indeed already looked at the possibility of having the data cleaned up a little bit before getting my teeth stuck in that task, but that's not possible.
I have to take the context as it is, can't change anything.
So the first thing I did was to segregate the number of records for the username field, Group-Object -Property Username, and my work was focused on the cases were a given user was, like Neil Armstrong, in both datasets.
When there is only 1 record, like Cousteau, it's straight forward, I leave it as it is. When there's one record in each dataset I need to merge data, and when there is more than 2 records for one username then it's fair to say that it is a complete mess, although I don't mind leaving those as they are just now (especially because thousands of records have a [string]::IsNullOrEmpty($Username) = $true so they count as a number greater than 2 records..)
I hope it makes more sense?
At the moment I want to focus on the cases where a given username is showing up once in both SPACE and OCEAN datasets, I know its not complicated but the algorithm I am using makes the whole process super slow :
0 - Create an empty array
1 - Get rows from SPACE dataset
2 - Get rows from OCEAN dataset
3 - Create a hashtable containing the properties of both datasets where properties aren't empty
4 - Create a psobject to encapsulate the hashtable
5 - Add that object to the array
And that is taking ages because I am talking about 65k records in SPACE and about 50k records in OCEAN.
SO I was wondering if there's a better way of doing this?
Thanks !