How to configure StandardOauth2AccessTokenProvider properly? - apache-nifi

Question

We are currently trying to access the Google Spreadsheet REST API through InvokeHTTP.
However, we are struggling to get the StandardOauth2AccessTokenProvider configured properly.
We upgraded our Nifi instance to version 1.16.3 in order to provide the scope, however, we are not sure which data needs to be put into the Client ID and Client Secret.
We tried the following options:
Create Oauth2 credentials in the Google Cloud console and provide these
Combine the above approach with a google username and password (in User Password mode)
Create a service account and provide its credentials
Whenever we try to send any request to the Google Spreadsheets API, we always get the same stack trace:
2022-09-29 11:47:18,193 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=86d7424e-0183-1000-cd72-d2071e1add8b] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.io.UncheckedIOException: OAuth2 access token request failed
java.io.UncheckedIOException: OAuth2 access token request failed
at org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider.getAccessDetails(StandardOauth2AccessTokenProvider.java:330)
at org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider.acquireAccessDetails(StandardOauth2AccessTokenProvider.java:289)
at org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider.getAccessDetails(StandardOauth2AccessTokenProvider.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
at com.sun.proxy.$Proxy108.getAccessDetails(Unknown Source)
at org.apache.nifi.processors.standard.InvokeHTTP.initOauth2AccessTokenProvider(InvokeHTTP.java:864)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1662)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (StringReader); line: 1, column: 2]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2005)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4761)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4667)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
at org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider.getAccessDetails(StandardOauth2AccessTokenProvider.java:324)
... 26 common frames omitted
Is this due to a possible Consent Screen? Any pointers on what we are doing wrong?