Way to check for duplicates before writing into a file? - go

Question

So I wrote a small script that takes text files as an input, reads every line and tries to validate it as an email. If it passes, it writes the line into a new ('clean') file, if it doesn't pass, it strips it of spaces and tries to validate it again. Now, if it passes this time, it writes the line into a new file and if it fails, it ignores the line.
Thing is, such as it is, my script may write duplicate emails into the output files. How should I go around that and check for duplicates present in the output file before writing?
Here's the relevant code:
// create reading and writing buffers
scanner := bufio.NewScanner(r)
writer := bufio.NewWriter(w)
for scanner.Scan() {
email := scanner.Text()
// validate each email
if !correctEmail.MatchString(email) {
// if validation didn't pass, strip and lowercase the email and store it
email = strings.Replace(email, " ", "", -1)
// validate the email again after cleaning
if !correctEmail.MatchString(email) {
// if validation didn't pass, ignore this email
continue
} else {
// if validation passed, write clean email into file
_, err = writer.WriteString(email + "\r\n")
if err != nil {
return err
}
}
} else {
// if validation passed, write the email into file
_, err = writer.WriteString(email + "\r\n")
if err != nil {
return err
}
}
}
err = writer.Flush()
if err != nil {
return err
}

Answer 1

Create a type that implements writer then create a custom WriteString
Inside WriteString open the file where you store your emails, iterate over each email and save the new emails.

Answer 2

You may use a Go built-in map as a set like this:
package main
import (
"fmt"
)
var emailSet map[string]bool = make(map[string]bool)
func emailExists(email string) bool {
_, ok := emailSet[email]
return ok
}
func addEmail(email string) {
emailSet[email] = true
}
func main() {
emails := []string{
"duplicated#golang.org",
"abc#golang.org",
"stackoverflow#golang.org",
"duplicated#golang.org", // <- Duplicated!
}
for _, email := range emails {
if !emailExists(email) {
fmt.Println(email)
addEmail(email)
}
}
}
Here is the output:
duplicated#golang.org
abc#golang.org
stackoverflow#golang.org
You may try the same code at The Go Playground.