I want to have a user which has a read-only access to a given index. I have read the Elasticsearch documentation and learnt that this can be achieved using the xpack API provided by Elasticsearch as a security feature. Now I am using "IBM Cloud Databases for Elasticsearch" and it comes with Elasticsearch 6.8.4, I am successfully able to communicate with it via Python and REST APIs as well and can manage to create index document etc but I am not able to use any of the xpack methods at all, not even the basic ones like get_role or get_user, it gives an error that I have attached herewith. I have also tried a same Elasticsearch version locally deployed on my machine and I am successfully able to use all the xpack methods. Below are the examples of how I am trying to use get_user method using requests and elasticsearch python.
Here is the requests method used and the response:
# Get User via requests
endpoint = "https://9fb4-f729-4d0c-86b1-da.eb46187.databases.appdomain.cloud:31248/_xpack/security/user"
header = {'Authorization': access_token,
'Content-Type': 'application/json',
'Accept': 'application/json'}
requests.get(url=endpoint,
auth=(cred_elastic["username"],cred_elastic['password']),
verify='./cert.pem',
headers=header).json()
Response:
{'error': {'root_cause': [{'type': 'security_exception',
'reason': 'Unexpected exception indices:data/read/get'}],
'type': 'security_exception',
'reason': 'Unexpected exception indices:data/read/get'},
'status': 500}
Here is python elasticsearch same method and response:
# Creating Elasticsearch Object
context = create_default_context(cadata=cred_elastic['tls_certificate'])
es = Elasticsearch(cred_elastic['endpoint'],
ssl_context=context,
http_auth=(cred_elastic['username'],
cred_elastic['password']))
es.security.get_user()
Response:
TransportError: TransportError(405, 'Incorrect HTTP method for uri [/_security/user] and method [GET], allowed: [POST]', 'Incorrect HTTP method for uri [/_security/user] and method [GET], allowed: [POST]')
Additionally, in the second method, the error is different but if instead I use put_user, it throws the exact same 500 error the former method throws.
I am using the default user and service credentials that IBM Cloud creates for authentication.
Update: This is the link to the service that I am using (Contains Documentation link as well):
https://cloud.ibm.com/catalog/services/databases-for-elasticsearch
That's because IBM Cloud Databases for Elasticsearch doesn't use xpack. So, if you're attempting to use it, it won't work. Currently, they only have one type of user.
Related
I'm attempting to collect details about all processors on two installed instances of Nifi, versions 1.19.0 and 1.20.0. I want to use Nifi to do this so I can perform some minor ETLs and save the resultant dataset in a database. Both instances are configured as standalone authentication with SSL.
However, when I configure the InvokeHTTP processor how I think it should be to aquire a token, I'm getting a 400 status code and a response.body of "The username and password must be specified." on both instances.
Here's the super basic flow:
Nifi Flow
Here's what the error looks like:
Attributes of failed response
Here's the current config of the InvokeHTTP processor on the 1.20.0 instance which we can focus on from here out since they are both responding similarly.
Postman - InvokeHTTP Config
When I run the request with Postman, I get the expected token in the response body. And I'm able to make subsequent requests (the above config was a response from https://nifi06:8443/nifi-api/processors/39870615-0186-1000-e7d7-c59fa621ca7d in Postman).
I've tried the following:
Adding dynamic attributes for username/password as in the above configuration.
I've added them as JSON to the preceding Generate FlowFile processor's custom text.
And I've also tried using the Request Username/Request Password properties of the InvokeHTTP processor.
All return with the same response.body of "The username and password must be specified."
This seems it should be really simple and I'm sure I've been staring at it too long to see what I'm missing. Can anyone tell me where I need to specify the username/password?
Thanks.
I've got an API gateway v2 which is using a proxy integration to lambda.
I'm trying to pass a custom header value via a GET call to my API.
...
fetch(new Request(refreshUrl, {
method: 'GET',
headers: new Headers({
'refreshToken': user?.tokens.refreshToken
})
}))
...
I can't for the life of me figure out how to get APIGW to pass that header value to my lambda. When I log the event received in my lambda, my "refreshToken" header is nowhere to be seen.
I've tried to add parameter mapping such as the following to my lambda integration in apigw with no success:
$request.header.refreshtoken
And the documentation for such scenarios with AWS only seems to show for APIGW v1.
Solved:
The issue wasn't related to APIGW. The request parameter mapping is correct.
The problem was that I was proxying the api via cloud front and I needed to whitelist the header in my distribution.
I have a self-hosted Grafana instance and a Elastic Cloud hosted Elasticsearch instance with Kibana. I'm hoping to get the two connected so I can query Elasticsearch directly from Grafana. I've gone through the official docs and haven't found them helpful here.
Here's the form I see when adding a datasource in Grafana:
Here's my Elastic Cloud deployment applications:
The URL in the Grafana form is coming from the Elasticsearch "Copy Endpoint" link.
The user defined here is a user I set up in the Elasticsearch "Stack Management" section with the kibana_system user role.
I've tried several of the options defined and cannot get a successful response from my ES instance.
Depending on the settings I've tried, it will either be a 400 error, 403 error, 503 error, or the Laravel Elasticsearch error: Types cannot be provided in get mapping requests, unless include_type_name is set to true..
With my current settings, I'm hitting the Laravel error above. I would assume this means my credentials are correct, but something else is happening in the verification request.
The verification request is going to https://[grafanahost]/api/datasources/proxy/17/logs*/_mapping
{
"error":{
"root_cause":[
{
"type":"illegal_argument_exception",
"reason":"Types cannot be provided in get mapping requests, unless include_type_name is set to true."
}
],
"type":"illegal_argument_exception",
"reason":"Types cannot be provided in get mapping requests, unless include_type_name is set to true."
},
"status":400
}
Thank you!
I have a google cloud api gateway deployed to send requests to a cloud run service.
The cloud run service hosts a laravel docker container image and to authenticate with my authenticated pages, I need to send an Authorization header (Authorization: Bearer my-user-token-here).
When I send the request directly to the cloud run service, I am able to get the response I need with the Authorization header set. But, when I send the request through the api gateway, I always get an unauthenticated message showing the header is missing in the api request to the cloud run. I am not sure of this though.
I can't find any useful documentation on google cloud api gateway to suggest whether cloud run drops the header.
I am also not sure whether the error is from the openapi.yaml. So far I realized I cannot use the v3 of the openapi documentation but rather v2 as api gateway does not support v2. In the v2 of the openapi docs, the securityDefinitions don't support Authorization header Bearer token but instead supports Authorization header basic.
My Openapi yaml
# openapi2-run.yaml
swagger: "2.0"
info:
title: my-api
description: my custom api
version: 1.0.0
schemes:
- https
produces:
- application/json
consumes:
- application/json
x-google-backend:
address: https://some-cloud-run-url
basePath: /api
host: my-api.nw.gateway.dev
x-google-endpoints:
- name: "my-api.nw.gateway.dev"
allowCors: True
paths:
/user:
get:
summary: Requested user details.
operationId: UserDetails
responses:
"200":
description: Return Requested User Details.
schema:
type: string
"default":
description: Unexpected error
The surprising fact is that if I send the request either locally or directly to the cloud run, it works and I get no authentication error, but when I use the api-gateway, then I get the error. So I am guessing it has to do with the header going missing when the request reaches the cloud run, probably because the yaml definition I have here does not have an authorization header.
We have an API gateway instance which sends requests to cloud functions.
If any incoming requests have an Authorization header, the gateway maps the header details into an X-Forwarded-Authorization header in the request to the cloud function.
I assume it's the same for requests to Cloud Run. I don't have any experience with Laravel to know if it has options to look in the forwarded header, though.
Actually you can ignore it by setting the disable_auth in x-google-backend.
The document is not in google gateway, but in google endpoint as follow.
https://cloud.google.com/endpoints/docs/openapi/openapi-extensions
By the document it said:
When configuring your target backend, you may not want to use IAP or IAM to authenticate requests from ESPv2 if either of these conditions apply:
The backend should allow unauthenticated invocations.
The backend requires the original Authorization header from the API client and cannot use X-Forwarded-Authorization (described in the jwt_audience section).
So in your particular case, you just need to modify a single block like this:
x-google-backend:address:
https://some-cloud-run-url
disable_auth: True
And it will work like a charm.
Beware that once you decide to do the authorization yourself, you cannot set the securityDefinitions in the gateway config. The gcp gateway will throw 401 if you do this.
We are using our own logging solution because stackdriver is su...bpar. I want to pull the metrics on how many unacknowledged messages there are in the pubsub. Started to read the docs on that and they are all over the place.
Found this page:
https://cloud.google.com/monitoring/api/metrics
Despite being under the api it does not describe any api calls, but does contain the description of the metric I want to extract.
Now I am thinking I need to use the monitoring api to extract what I need somehow:
https://cloud.google.com/monitoring/api/ref_v3/rest/
So I use the api explorer to try a couple of methods:
https://developers.google.com/apis-explorer/#search/monitoring/monitoring/v3/monitoring.projects.groups.list
I query and gives me an available url:
GET https://monitoring.googleapis.com/v3/projects/myprojectname/groups?key={YOUR_API_KEY}
I go to my project's console (api & credentials page) and generate an api key without restrictions and paste it in trying to curl.
curl https://monitoring.googleapis.com/v3/projects/myproject/groups?key=myrandomkeylkjlkj
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
Why is this happening? How can I get the metrics? I went to the url provided but it explains oauth token creation and has nothing regarding the api keys. I just need to curl things to make sure I am going the right way.
Why does this have to be so hard? Killed several hours of my life trying to get this.
curl -H "Authorization: Bearer $(gcloud config config-helper --format='value(credential.access_token)')" https://monitoring.googleapis.com/v3/projects/myproject/groups