I wanted to add a read only user to my cluster, my app prefixes all its indexes with myapp_.
Following https://www.elastic.co/blog/user-impersonation-with-x-pack-integrating-third-party-auth-with-kibana (what a strange title for the only actually usable blog post on this...) I have first added a role with
curl -XPOST '$ELASTIC_URL:9200/_xpack/security/role/name_of_readonly_role' \
-H 'Content-Type: application/json' \
-d'{"indices":[{"names":"myapp_*","privileges":["read"]}]}'
and then added it to a user:
curl -XPOST $ELASTIC_URL:9200/_xpack/security/user/name_of_user \
-H 'Content-Type: application/json' \
-d'{"roles":["name_of_readonly_role"],"password":"some_password"}'
but when opening $ELASTIC_URL:9200 I got
action [cluster:monitor/main] is unauthorized for user
what's next?
There's a complete dearth of examples for this as far as I can see, to fix this problem the role command needs to be re-run with -d'{"cluster":["monitor"], "indices":[{"names":"myapp_*","privileges":["read"]}]}' (same curl command works for creating or updating roles). This seems to leak the name of all indexes but not much else aside from their names and I was fine with that. And even that seems to be not enough for some apps like the ElasticSearch Head brower extension, I needed to add the index level monitor privilege as well: -d'{"cluster":["monitor"], "indices":[{"names":"myapp_*","privileges":["read", "monitor"]}]}'. Role changes are automatically applied to users.
I still have no idea what the "/main" relates to in the error message but this works.
Related
I want to set up a dev environment of Hasura on my local machine, that replicates my existing production (same tables, same schema, same data).
What are the required steps to achieve this task?
I've found this process to work well.
Create a clean empty local postgresql database and Hasura instance. To update an existing local database, drop it and recreate it.
Dump the schema and data from your existing Hasura server (as per the answer by #protob, but with clean_output set so that manual changes to the output do not have to be made. See pgdump for details.
curl --location --request POST 'https://example.com/v1alpha1/pg_dump' \
--header 'Content-Type: application/json' \
--header 'X-Hasura-Role: admin' \
--header 'Content-Type: text/plain' \
--header 'x-hasura-admin-secret: {SECRET}' \
--data-raw '{ "opts": ["-O", "-x","--inserts", "--schema", "public"], "clean_output": true}' > hasura-db.sql
Import the schema and data locally:
psql -h localhost -U postgres < hasura-db.sql
The local database has all the migrations because we copied the latest schema, so just mark them as applied:
# A simple `hasura migrate apply --skip-execution` may work too!
for x in $(hasura migrate status | grep "Not Present" | awk '{ print $1 }'); do
hasura migrate apply --version $x --skip-execution
done
# and confirm the updated status
hasura migrate status
Now finally apply the Hasura metadata using the hasura CLI:
hasura metadata apply
Enjoy your new instance!
Backup the database.
Run Hasura with the database.
Make sure Hasura metadata is synced.
Hasura has a special endpoint for executing pg_dump on the Postgres instance.
Here is a sample CURL request:
curl --location --request POST 'https://your-remote-hasura.com/v1alpha1/pg_dump' \
--header 'Content-Type: application/json' \
--header 'X-Hasura-Role: admin' \
--header 'Content-Type: text/plain' \
--data-raw '{
"opts": ["-O", "-x","--inserts", "--schema", "public"]
}'
It outputs the schema and data in psql format.
You can use a tool such as Postman for convenience to import, test and run the CURL query.
Please follow the pg_dump documentation to adjust needed opts.
i.e. the above query uses "--inserts" opt, which produces "INSERT INTO" statements in the output.
The output can be copied, pasted and imported directly to Hasura Panel SQL Tab ("COPY FROM stdin" statements result in errors when inserted in the panel).
http://localhost:8080/console/data/sql
Before import, comment out or delete the line CREATE SCHEMA public; from query, because it already exists.
You also have to select tables and relations to be tracked, during or after executing the query.
If the amout of data is bigger, it might be better to use CLI for import.
I am migrating my code from Elasticsearch-5.6 to Elasticsearch-7.0. What is the allowed mapping type that I should use?
As per the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/removal-of-types.html#_schedule_for_removal_of_mapping_types
For Elasticsearch 7.x
"... indexing a document no longer requires a document type. ... _doc is a permanent part of the path, and represents the endpoint name rather than the document type."
This seems pretty clear, but I was able to execute both of the following successfully:
curl -XPUT "http://localhost:9200/twitter/doc/1" -H 'Content-Type: application/json' -d'{"x":"val"}'
curl -XPUT "http://localhost:9200/twitter/_doc/1" -H 'Content-Type: application/json' -d'{"x":"val"}'
As per the documentation, inserting at /index_name/doc should not function, as /_doc is part of the endpoint-path.
Am I missing something from the documentation?
(My migration strategy would depend on this, as Elasticsearch-5 does not allow type names with preceding '_' (e.g. '_doc') and I would want to change my code to write to 'doc' if that works with Elasticsearch-7)
When I try to store anything in elasticsearch, An error says that:
TransportError(403, u'cluster_block_exception', u'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];')
I already inserted about 200 millions documents in my index. But I don't have an idea why this error is happening.
I've tried:
curl -u elastic:changeme -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster.blocks.read_only":false}}'
As mentioned here:
ElasticSearch entered "read only" mode, node cannot be altered
And the results is:
{"acknowledged":true,"persistent":{"cluster":{"blocks":{"read_only":"false"}}},"transient":{}}
But nothing changed. what should I do?
Try GET yourindex/_settings, this will show yourindex settings. If read_only_allow_delete is true, then try:
PUT /<yourindex>/_settings
{
"index.blocks.read_only_allow_delete": null
}
I got my issue fixed.
plz refer to es config guide for more detail.
The curl command for this is
curl -X PUT "localhost:9200/twitter/_settings?pretty" -H 'Content-Type: application/json' -d '
{
"index.blocks.read_only_allow_delete": null
}'
Last month I facing the same problem, you can try this code on your Kibana Dev Tools
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
I hope it helps
I had faced the same issue when my disk space was full,
please see the steps that I did
1- Increase the disk space
2- Update the index read-only mode, see the following curl request
curl -XPUT -H "Content-Type: application/json"
http://localhost:9200/_all/_settings -d
'{"index.blocks.read_only_allow_delete": null}'
This happens because of the default watermark disk usage of Elastic Search. Usually it is 95% of disk size.
This happens when Elasticsearch thinks the disk is running low on space so it puts itself into read-only mode.
By default Elasticsearch's decision is based on the percentage of disk space that's free, so on big disks this can happen even if you have many gigabytes of free space.
The flood stage watermark is 95% by default, so on a 1TB drive you need at least 50GB of free space or Elasticsearch will put itself into read-only mode.
For docs about the flood stage watermark see https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html.
Quoted from part of this answer
One solution is to disable it enitrely (I found it useful in my local and CI setup). To do it run the 2 commands:
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
Tagging into this later on as I just encountered the problem myself - I accomplished the following steps.
1) Deleted older indexes to free up space immediately - this brought me to around 23% free.
2) Update the index read-only mode.
I still had the same issue. I checked the Dev Console to see what might be locked still and none were. Restarted the cluster and had the same issue.
Finally under index management I selected the indexes with ILM lifecycle issues and picked to reapply ILM step. Had to do that a couple of times to clear them all out but it did.
The problem may be a disk space problem, i had this problem despite i cleaned many space my disk, so, finally i delete the data folder and it worked: sudo rm -rf /usr/share/elasticsearch/data/
This solved the issue;
PUT _settings { "index": { "blocks": { "read_only_allow_delete": "false" }
}
I have configuration parametr current_build_date (User Defined Parameter) I want just to set this parameter to current date by API TeamCity.
On docs I have seen this:
http://teamcity:8111/httpAuth/app/rest/buildTypes/<buildTypeLocator>/parameters/<parameter_name>
I know my Build configuration ID, but I can't understand how by this to make buildTypeLocator.
I assume result will be something like this:
curl -u Login:Password \
-X PUT \
-d 'valueOfMyParam' \
-H 'Content-Type: text/plain' \
http://teamcity:8111/httpAuth/app/rest/buildTypes/<buildTypeLocator>/parameters/current_build_date
I will realy appreciate if somebody who knows TeamCity API will help me with this problem.
I made attempt just to pass instead of buildTypeLocator my Build configuration ID and I got ERROR:
[17:08:25][Step 3/3] Error has occurred during request processing (Not Found).
[17:08:25][Step 3/3] Error: jetbrains.buildServer.server.rest.errors.NotFoundException: No project found by name or internal/external id 'BuildConfigurationID'.
If there are any problems or ambiguities with my question please add comment, i'll try to fix it.
If you browse the REST API endpoints in a browser you'll be able to see the format of the build locator.
Visit http://teamcity:8111/httpAuth/app/rest/buildTypes/ and you'll see the entries have a href attribute that contains the buildLocator (generally a property:value combination)
You'll then be able to navigate using that url / communicate via the API
Hope this helps
I solved problem: build type locator was id:Build configuration ID
current_build_date=`date +%%Y-%%m-%%d:%%H:%%M:%%S`
echo $current_build_date;
curl -u Login:Password \
-X PUT \
-d $current_build_date \
-H 'Content-Type: text/plain' \
https://teamcity.billing.ru/httpAuth/app/rest/buildTypes/id:Build
configuration ID/parameters/current_build_date
Having a many-to-many relationship between users and groups. I would like to know how to update this relationship with SDR. This is what I've tried so far after reading the docs.
curl -X POST -H 'Content-Type: text/uri-list' -d 'http://localhost:8080/rest/users/5' http://localhost:8080/rest/groups/1/users
Expected result: Add user 5 to group 1.
Actual result: 405 Method Not Allowed.
curl -X PUT -H 'Content-Type: text/uri-list' -d 'http://localhost:8080/rest/users/5' http://localhost:8080/rest/groups/1/users
Expected result: Replace all members of group 1 with user 5.
Actual result: Works as expected.
curl -X PUT -H 'Content-Type: text/uri-list' -d #members.txt http://localhost:8080/rest/groups/1/users
Where the file members.txt has:
http://localhost:8080/rest/users/5
http://localhost:8080/rest/users/6
http://localhost:8080/rest/users/7
Expected result: Replace all members of group 1 with the users 5, 6 and 7.
Actual result: Only last user (in this case 7) gets added.
Could someone provide an example on how to ADD a single URI to an association?. Also if possible, how to add or replace an association with multiple URIs?
After re-reading the documentation, it does indeed say POST should add to the collection.
My experience has been to use PATCH to add to the collection.
To further the answer: You should be able to use PUT CONTENT-TYPE: text/uri-list with a content body having multiple URIs. Each URI is separated by a line break "\n"
Try this:
curl -v -X POST -H "Content-Type: text/uri-list" -d "http://localhost:8080/rest/users/5" http://localhost:8080/rest/groups/1/users