Why does `utf8.Valid()` consider some valid UTF-8 code as invalid? - go

Question

$ cat main.go
package main
import (
"fmt"
"unicode/utf8"
"io/ioutil"
"os"
"log"
)
func main() {
str, err := ioutil.ReadFile(os.Args[1])
if err != nil { log.Fatal(err) }
fmt.Println(utf8.Valid([]byte(str)))
}
$ go run main.go <(echo $'\xed\xa0\xb5')
false
utf8.Valid() says '\xed\xa0\xb5' is invalid.
\xed: 11101101
\xa0: 10100000
\xb5: 10110101
But there are the three bytes in binary format. 111xxxxx means it is a three bytes characters. According to the table on the wiki page, the following two bytes should valid.
https://en.wikipedia.org/wiki/UTF-8
Does anybody know why utf8.Valid() shows the three byte character as invalid UTF-8 character?

Answer 1

As per the codepage layout table on Wikipedia, while the hex sequence \xed\xa0\xb5 follows the encoding rules for the codepoint at 0xd835, it is considered invalid as it matches the codepoint reserved for surrogate halves in UTF-16. When looking at the codepoint table for Unicode, you'll see that the codepoints from 0xD800 to 0xDFFF are all unassigned and marked as "explicitly invalid in UTF-8".
Unicode and ISO/IEC 10646 do not assign actual characters to any of the code points in the D800–DFFF range — these code points only have meaning when used
in surrogate pairs. Hence an individual code point from a surrogate pair does not represent a character, is invalid unless used in a surrogate pair, and is
unconditionally invalid in UTF-32 and UTF-8 (if strict conformance to the standard is applied).