Apt-get: Only Basic auth using server hostname with https not working - apt

Question

I have a local web server which is hosting all my debian packages from another machine I am trying to do apt-get update/upgrade to fetch the Package index list and upgrade the machine using https but only basic authorization as my web server is configured to do only basic auth and I do not want to change that to certificate based auth.
apt-get update with https basic auth works fine (i.e the client is able to skip the cert based authentication) when I use IP address of the web server but as soon as I try to use hostname of the web server then it doesn't work I keep getting the error "gnutls_handshake() failed: A TLS warning alert has been received."
Config for IP scenario which works with basic auth without certs
APT Config under apt.conf.d with IP:
Debug::Acquire::https "true";
Acquire::https::10.2.20.1 {
Verify-Host "false";
Verify-Peer "false";
};
source.list.d with IP:
deb [arch=amd64] https://username:password#10.2.20.1:443/foo bar test
Debugs when it works
0% [Working]* About to connect() to 10.2.20.1 port 443 (#0)
* Trying 10.2.20.1... * connected
* found 164 certificates in /etc/ssl/certs/ca-certificates.crt
* server certificate verification SKIPPED
* Server auth using Basic with user 'username'
> GET /foo/dists/bar/Release.gpg HTTP/1.1
Authorization: Basic
Config for hostname scenario doesn't work with basic auth without certs
APT Config under apt.conf.d with IP:
Debug::Acquire::https "true";
Acquire::https::my-foo-test.com {
Verify-Host "false";
Verify-Peer "false";
};
source.list.d with IP:
deb [arch=amd64] https://username:password#my-foo-test.com:443/foo bar test
Debug with TLS warning when hostname is used
root#my:~# apt-get update
0% [Working]* About to connect() to my-foo-test.com port 443 (#0)
* Trying 10.2.20.1... * connected
* found 164 certificates in /etc/ssl/certs/ca-certificates.crt
* gnutls_handshake() failed: A TLS warning alert has been received.
* Closing connection #0
Ign https://my-foo-test.com repo Release.gpg
I have resolved IP to hostname locally on my machine where I am running apt-get update using /etc/hosts file
Entry from /etc/hosts file
10.2.20.1 my-foo-test.com
Event tried below way but didn't work, tried putting this into apt.conf.d/ that didn't work either
apt-get update -o Debug::Acquire::https=true -o Acquire::https::Verify-Host=false -o Acquire::https::Verify-Peer=false -o Dir::Etc::SourceList="/etc/apt/sources.list.d/mysource.list" update
Thanks for the help!