We are Using Apache Nifi to Publish events to PubsubLite using (PublishGCPubSubLite 1.17.0) Nifi processor.
GCP SA json files contains below field:
{
"type": "service_account",
"project_id": "",
"private_key_id": "",
"private_key": "",
"client_email": "",
"client_id": "",
"auth_uri": "",
"token_uri": "",
"auth_provider_x509_cert_url": "",
"client_x509_cert_url": ""
}
We tried using type:"service_account" and it is getting failed to communicate. and we are getting the below error,
PublishGCPubSubLite[id=7f9baf80-0184-1000-ffff-ffff8ec321b3] Failed to create Google Cloud PubsubLite Publisher: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
From Nifi server we tested the connectivity and credential, it is working:
./google-cloud-sdk/bin/gcloud auth login --cred-file gcp.json
Authenticated with service account credentials for: [#.iam.gserviceaccount.com].
Your current project is [None]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
Please suggest how to make connection to PubsubLite on Apache Nifi
Related
I am working with Google Cloud in Go and following this article by John Hanley:
https://www.jhanley.com/google-cloud-improving-security-with-impersonation/
and mashed it with this SO answer:
How to authenticate Google APIs (Google Drive API) from Google Compute Engine and locally without downloading Service Account credentials?
The credentials are successfully saved to, "application_default_credentials.json":
Notice: "type": "impersonated_service_account"
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[sa#example-2021.iam.gserviceaccount.com]:generateAccessToken",
"source_credentials": {
"client_id": "...apps.googleusercontent.com",
"client_secret": "...",
"refresh_token": "...",
"type": "authorized_user"
},
"type": "impersonated_service_account"
}
My code which produces an unknown credential type: "impersonated_service_account" error:
package main
import (
...
"cloud.google.com/go/storage"
"golang.org/x/oauth2"
"google.golang.org/api/docs/v1"
"google.golang.org/api/drive/v3"
"google.golang.org/api/impersonate"
"google.golang.org/api/option"
...
)
var Config.GoogleServiceAccount string = "sa#example-2021.iam.gserviceaccount.com"
func main(){
_ = getTokenAsImpersonator()
}
// From: https://pkg.go.dev/google.golang.org/api/impersonate#example-CredentialsTokenSource-ServiceAccount
func getTokenAsImpersonator() oauth2.TokenSource {
ctx := context.Background()
// Base credentials sourced from ADC or provided client options.
ts, err := impersonate.CredentialsTokenSource(ctx, impersonate.CredentialsConfig{
TargetPrincipal: Config.GoogleServiceAccount,
Scopes: []string{"https://www.googleapis.com/auth/cloud-platform"},
// Delegates: []string{"bar#project-id.iam.gserviceaccount.com"},
})
if err != nil {
log.Fatal(err)
}
return ts
}
The 'unknown credential type: "impersonated_service_account"' error:
google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: unknown credential type: "impersonated_service_account"
Have I done something wrong or is this a bug?
UPDATE
Answering John's questions from the comments:
1.
a) What is the value of the environment variable GOOGLE_APPLICATION_CREDENTIALS?
GOOGLE_APPLICATION_CREDENTIALS=/Users/x/.config/gcloud/application_default_credentials.json
b) What command did you use to generate application_default_credentials.json?
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/accounts.reauth,openid --impersonate-service-account=[sa#example-2021.iam.gserviceaccount.com]
Response:
Credentials saved to file: [/Users/x/.config/gcloud/application_default_credentials.json]
c)Which OS and version?
MacOS 10.13.6
d)gcloud --version?
Google Cloud SDK 343.0.0
app-engine-go
app-engine-python 1.9.91
bq 2.0.69
cloud-datastore-emulator 2.1.0
core 2021.05.27
gsutil 4.62
If you can create a minimum example ...
I have updated the example code above.
At some point I had used the CLI to impersonate an account:
gcloud config set auth/impersonate_service_account <service account>
Then later on when trying to use the application default credentials command it wraps your credentials with the service account credentials.
gcloud auth application-default login
What you end up with is a file that looks like this:
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/example#example-project.iam.gserviceaccount.com:generateAccessToken",
"source_credentials": {
"client_id": "123abc.apps.googleusercontent.com",
"client_secret": "XXXXXXXXX",
"refresh_token": "XXXXXXXXX",
"type": "authorized_user"
},
"type": "impersonated_service_account"
}
This appears to cause a lot of problems with third party services such as terraform.
What is strange is that Terraform is just making API calls to Google using Google SDKs, so really its something to do with Google.
You need to remove the impersonation:
gcloud config unset auth/impersonate_service_account
And then run the application default credential command again:
gcloud auth application-default login
Now if you check your file it should look like this:
{
"client_id": "XXXXXXXXX",
"client_secret": "XXXXXXXXX",
"quota_project_id": "example-project",
"refresh_token": "XXXXXXXXXX",
"type": "authorized_user"
}
I was hitting the same issue when I was trying to impersonate an account so I could run Terraform commands as a service account instead of my personal account but it doesn't like that.
EDIT: Rereading you question it sounds like you're in the same boat as me. We want to use service accounts without physically downloading the keys. This is even mentioned by Google as best practice. But doing so is causing issues with their own SDKs.
I had the same issue running GCP Terraform provider tests. You can specify the Service Account Terraform have to impersonate setting the env variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT (documentation).
Configuration steps:
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=SERVICE_ACCOUNT#PROJECT_ID.iam.gserviceaccount.com
gcloud auth application-default login
Below is my request body. When I hit the request using postman it throws forbidden with 403 error code but when I remove the "paymentOkUrl": "https://ei-prelive.dasitha.com:8289/general/api-gateway/v1/upad/pb/termUrl" from request body it executes successfully. I found it blocks any thing starts from "https://". I am using AWS ECS (with EC2) to host my springboot app.
I am new to AWS can anyone tell me what should I change in the aws configurations to solve this problem.
Request:
{
"paymentProvider": "ELAVON_REALEX",
"systemId": "30",
"orderId": "26518203_3afb594b-f25e-40de-bca2-650a1a234b53",
"amount": "14100",
"currencyCode": "GBP",
"cardDetails": {
"cardType": "MASTERCARD",
"cardNumber": "5269275301730688",
"expiryDate": "0424",
"cardHolderName": "Perera H T P M"
},
"paymentOkUrl": "https://ei-prelive.dasitha.com:8289/general/api-gateway/v1/upad/pb/termUrl"
}
Response:
{403 Forbidden}
I am following the below to set up a config server in PCF and access the application in PCF.
https://github.com/pcf-guides/gs-configuration-server
I have added the config server from the PCF marketplace and it is bind with the app.
The config server is pointed to the following Git repository:
https://github.com/pcf-guides/configuration-server-config-repo.git
The VCAP_SERVICES is like below :
"p-config-server": [
{
"binding_name": null,
"credentials": {
"access_token_uri": "https://p-spring-cloud-
services.uaa.run.pivotal.io/oauth/token",
"client_id": "p-config-server-d837bb39-4cf1-47ce-994b-03257852a7f6",
"client_secret": "XXsWzX6IhKME",
"uri": "https://config-9f0e115f-dbb0-42e8-981a-e70e2cd62570.cfapps.io"
},
"instance_name": "sconfigserver",
"label": "p-config-server",
"name": "sconfigserver",
"plan": "trial",
"provider": null,
"syslog_drain_url": null,
"tags": [
"configuration",
"spring-cloud"
Whenever I am trying to access the application authentication error is coming :
Full authentication is required to access this resource.
I dont have any security configuration in the application.
In your application.properties file, you should add the following
management.security.enabled=false
This will disable security altogether.
I tried this in my main application class and worked fine for me
#SpringBootApplication(exclude = {SecurityAutoConfiguration.class , ManagementWebSecurityAutoConfiguration.class})
I'm following the OData tutorial at https://blogs.sap.com/2017/05/21/step-4-with-sap-s4hana-cloud-sdk-calling-an-odata-service/comment-page-1/ and I'm getting an error when I try to retrieve business partners using DefaultBusinessPartnerService.
The relevant piece of code is:
DefaultBusinessPartnerService businessPartnerService = new DefaultBusinessPartnerService();
System.err.println("criated default business partner");
List<BusinessPartner> partners = businessPartnerService
.getAllBusinessPartner()
.select(BusinessPartner.BUSINESS_PARTNER,
BusinessPartner.LAST_NAME,
BusinessPartner.FIRST_NAME)
//.filter(BusinessPartner.BUSINESS_PARTNER_CATEGORY.eq(CATEGORY_VENDOR))
.orderBy(BusinessPartner.LAST_NAME, Order.ASC)
.execute(new ErpEndpoint(new ErpConfigContext()));
response.setContentType("application/json");
response.getWriter().write(new Gson().toJson(partners));
My ErpQueryEndpoint configuration is as follows:
#Mon May 14 15:27:09 BRT 2018
URL=https\://host\:port
Name=ErpQueryEndpoint
TrustAll=TRUE
Type=HTTP
Password=Password
Authentication=BasicAuthentication
User=Username
Where host, port, Username and Password have been replaced by the correct values.
When I query http://localhost:8080/s4integration-application/businesspartners I get the following error:
The endpoint responded with HTTP error code 403.
No service found for namespace , name API_BUSINESS_PARTNER, version 0001
Full error message:
{
"error": {
"code": "/IWFND/MED/170",
"message": {
"lang": "en",
"value": "No service found for namespace , name API_BUSINESS_PARTNER, version 0001"
},
"innererror": {
"application": {
"component_id": "",
"service_namespace": "/SAP/",
"service_id": "API_BUSINESS_PARTNER",
"service_version": "0001"
},
"transactionid": "C83CB3D2A1420000E005AF97B0836AD5",
"timestamp": "20180514182746.3576100",
"Error_Resolution": {
"SAP_Transaction": "Run transaction /IWFND/ERROR_LOG on SAP Gateway hub system (System Alias ) and search for entries with the timestamp above for more details",
"SAP_Note": "See SAP Note 1797736 for error analysis (https://service.sap.com/sap/support/notes/1797736)"
},
"errordetails": []
}
}
}
It seems that the endpoint is not configured on the SAP system (it's an S/4 Hana system). I'm not sure if I have to add something else to the URL besides the host and port or if there is some other configuration that has to be done on the SAP system.
Instructions for activating OData APIs from the SAP S/4HANA backend can be found here: help.sap.com/viewer/cdc25c83b63e482586b31b8acd49cf2f/1610%20003/… Just ignore the notion of the Fiori app.
I am converting a PHP application to access Google calendar to Go. I used this step by step to get started.
All went smoothly, but when I run quickstart.go, I get the following error:
Unable to parse client secret file to config: oauth2/google: missing
redirect URL in the client_credentials.json exit status 1
Content of the client_secret.json is:
{
"installed":{
"client_id":"***********content.com",
"project_id":"*******",
"auth_uri":"https://accounts.google.com/o/oauth2/auth",
"token_uri":"https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs"
}
}
That client_secret.json file is located at the root of my GOPATH as instructed in the step by step
I already have aOAuth 2.0 client ID for my PHP app that works just fine in PHP. I just want to use that one in the a new Go application to access multiple user calendars, but when I download the json file attached to that ID, I am getting the error above. Maybe the quickstart.go is not meant for that usage.
Any hints?
When you create OAuth credentials at https://console.developers.google.com/apis/credentials the dialog initially prompts you to "Configure your OAuth client" and you can choose between "Web application", "Desktop app", etc.
The client.json obtained for the generated OAuth credentials may not contain a "Return URL", depending on the type chosen initially.
For example, for "Web application" the client.json does not have a redirect URL:
{
"web": {
"client_id": "x.apps.googleusercontent.com",
"project_id": "x",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_secret": "x"
}
}
While for a "Desktop app" it has:
{
"installed": {
"client_id": "x.apps.googleusercontent.com",
"project_id": "x",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_secret": "x",
"redirect_uris": [
"urn:ietf:wg:oauth:2.0:oob",
"http://localhost"
]
}
}
The Go oauth.google module always requires a return URI: https://github.com/golang/oauth2/blob/0f29369cfe4552d0e4bcddc57cc75f4d7e672a33/google/google.go#L61