WS Federation Validate Throws Missing Configuration Exception with provided MetadataAddress - .net-6.0

Question

im struggling with the Configuration of a WsFederation Authentication, and i dont know whats missing here, because what it asks for is clearly provided.
Maybe somebody has an idea what could be the problem.
But Lets dive in some code, first we start with the AddWsFederation Configuration:
builder.AddWsFederation($"{scheme}:fed", "WS Federation Connect", options =>
{
options.ClaimsIssuer = "https://internal.identity.provider";
options.Wtrealm = "https://localhost:5001/";
options.Wreply = "https://localhost:5001/";
options.MetadataAddress = "https://internal.identity.provider/v2/FederationMetadata/2007-06/FederationMetadata.xml";
options.RequireHttpsMetadata = true;
options.CallbackPath = "/signin-wsf-fe";
options.Validate();
});
As you can see i provide a MetadataAddress, but the options.Validate() code thows still the following exception:
Provide MetadataAddress, Configuration, or ConfigurationManager to WsFederationOptions
Im not very expirenced with this type of authentication, so im pretty much at a loss.
In case someone wants to see the structure of the FederationMetadata.xml
<EntityDescriptor ID="<<some id>>" entityID="http://internal.identity.provider/" xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
<SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />
<Reference URI="#_3fa19ded-fa24-4d70-9f80-6255e9f860e4">
<Transforms>
<Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
<Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />
<DigestValue>some-value</DigestValue>
</Reference>
</SignedInfo>
<SignatureValue>some signature value</SignatureValue>
<KeyInfo>
<X509Data>
<X509Certificate>some certificate </X509Certificate>
</X509Data>
</KeyInfo>
</Signature>
<RoleDescriptor xsi:type="fed:SecurityTokenServiceType"
protocolSupportEnumeration="http://docs.oasis-open.org/wsfed/federation/200706"
ServiceDescription="my Identity"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:fed="http://docs.oasis-open.org/wsfed/federation/200706">
<KeyDescriptor use="signing">
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<X509Data>
<X509Certificate>some value</X509Certificate>
</X509Data>
</KeyInfo>
</KeyDescriptor>
<fed:TokenTypesOffered><fed:TokenType Uri="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1" />
<fed:TokenType Uri="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0" />
</fed:TokenTypesOffered>
<fed:ClaimTypesOffered>
<auth:ClaimType Uri="http://schemas.microsoft.com/ws/2008/06/identity/claims/role" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
<auth:ClaimType Uri="http://internal.identity.provider/claims/contactid" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
<auth:ClaimType Uri="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
<auth:ClaimType Uri="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
</fed:ClaimTypesOffered>
<fed:SecurityTokenServiceEndpoint>
<wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing">
<wsa:Address>https://internal.identity.provider/v2/issue/wstrust/mixed/username</wsa:Address>
</wsa:EndpointReference>
</fed:SecurityTokenServiceEndpoint>
<fed:SecurityTokenServiceEndpoint>
<wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing">
<wsa:Address>https://internal.identity.provider/v2/issue/wstrust/mixed/certificate</wsa:Address>
</wsa:EndpointReference>
</fed:SecurityTokenServiceEndpoint>
<fed:PassiveRequestorEndpoint>
<wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing">
<wsa:Address>https://internal.identity.provider/v2/issue/hrd</wsa:Address>
</wsa:EndpointReference>
</fed:PassiveRequestorEndpoint>
</RoleDescriptor>
<RoleDescriptor xsi:type="fed:ApplicationServiceType"
protocolSupportEnumeration="http://docs.oasis-open.org/wsfed/federation/200706"
ServiceDescription="my Identity"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:fed="http://docs.oasis-open.org/wsfed/federation/200706">
<KeyDescriptor use="signing">
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<X509Data>
<X509Certificate>some value</X509Certificate>
</X509Data>
</KeyInfo>
</KeyDescriptor>
<fed:TokenTypesOffered>
<fed:TokenType Uri="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1" />
<fed:TokenType Uri="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0" />
</fed:TokenTypesOffered>
<fed:ClaimTypesOffered>
<auth:ClaimType Uri="http://schemas.microsoft.com/ws/2008/06/identity/claims/role" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
<auth:ClaimType Uri="http://internal.identity.provider/claims/contactid" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
<auth:ClaimType Uri="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
<auth:ClaimType Uri="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn" xmlns:auth="http://docs.oasis-open.org/wsfed/authorization/200706" />
</fed:ClaimTypesOffered>
<fed:PassiveRequestorEndpoint>
<wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing">
<wsa:Address>https://internal.identity.provider/v2/issue/hrd</wsa:Address>
</wsa:EndpointReference>
</fed:PassiveRequestorEndpoint>
</RoleDescriptor>
</EntityDescriptor>
In case someone has an idea what i am missing here, feel free to leave a comment, or even en answer would be much apricated.
Best Regards and Thanks in Advance