ASP.NET WebAPI StatusCode: 401, ReasonPhrase: 'Unauthorized' - asp.net-web-api

Question

The scenario I am talking is local debugging scenario as I have not deployed to azure yet.
I have a MVC Web App authenticating against AAD that calls WebAPI
I have the right permission set in the AAD tenant, the reason I say it set correctly because I am able to get the AccessToken for the WebAPI, but then when I try to call the WebAPI it gives me the following error
{StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Pragma: no-cache
X-SourceFiles: =?UTF-8?B?xxxxxxxxxx
Cache-Control: no-cache
Date: Sun, 27 Mar 2016 02:06:31 GMT
Server: Microsoft-IIS/10.0
WWW-Authenticate: Bearer
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Content-Length: 61
Content-Type: application/json; charset=utf-8
Expires: -1
}}
Code
string userObjectID = ClaimsPrincipal.Current.FindFirst("http://schemas.microsoft.com/identity/claims/objectidentifier").Value;
NaiveSessionCache nsc = new NaiveSessionCache(userObjectID);
AuthenticationContext authContext = new AuthenticationContext(Startup.Authority, nsc);
ClientCredential credential = new ClientCredential(clientId, appKey);
result = authContext.AcquireTokenSilent(todoListClientId, credential, new UserIdentifier(userObjectID, UserIdentifierType.UniqueId));
Uri webApiUri = new Uri(string.Format("{0}{1}", todoListBaseAddress, "/api/TodoList"));
string accessTknString = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(result.AccessToken));
HttpClient client = new HttpClient();
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, webApiUri);
request.Headers.Add(System.Net.HttpRequestHeader.Authorization.ToString(), string.Format("Bearer {0}", accessTknString));
HttpResponseMessage response = await client.SendAsync(request);