What is the correct way to use the REST API to add a relationship between entities in apache atlas? Looking at the docs for the REST API, I find it difficult to tell what some of the fields mean, which are required or not (and what happens if they are not entered), or what default values should be (since the examples use what appear to be placeholder values (eg. what is provenanceType or the propagateTags field appears to expect some kind of enumeration value, but never specifies valid options)).
Could someone provide any example of what this would look like in real / valid values? Eg. if had 2 entities E1 and E2 already added to Atlas and wanted to establish a relationship between the two, would want to do something like...
curl -X POST --header 'Content-Type: application/json;charset=UTF-8' --header 'Accept: application/json' -d '{<simplified json>}' 'https://atlas-server-hostname:21000/v2/relationship'
Trying
[hph_etl#HW03 ~]$ curl -vv -u admin:admin -X POST --header 'Content-Type: application/json;charset=UTF-8' --header 'Accept: application/json' -d '{ \
"createTime": 1565135406, \
"createdBy": "hph_etl", \
"end1": { \
"guid": "2ddcda5b-2489-4636-a9ab-12b199c02422", \
"typeName": "hdfs_path" \
}, \
"end2": { \
"guid": "a33f45de-13d0-4a30-9df7-b0e02eb0dfd5", \
"typeName": "hdfs_path" \
}, \
"guid": "2ddcda5b-2489-4636-a9ab-12b199c02422", \
"propagateTags": "TWO_TO_ONE", \
"status": "ACTIVE", \
"typeName": "hdfs_path" \
}' 'http://HW03.co.local:21000/api/atlas/v2/relationship'
* About to connect() to HW03.co.local port 21000 (#0)
* Trying 172.18.4.48...
* Connected to HW03.co.local (172.18.4.48) port 21000 (#0)
* Server auth using Basic with user 'admin'
> POST /v2/relationship HTTP/1.1
> Authorization: Basic xxxxxx
> User-Agent: curl/7.29.0
> Host: HW03.co.local:21000
> Content-Type: application/json;charset=UTF-8
> Accept: application/json
> Content-Length: 442
>
* upload completely sent off: 442 out of 442 bytes
< HTTP/1.1 404 Not Found
< Date: Wed, 07 Aug 2019 01:07:44 GMT
< Set-Cookie: ATLASSESSIONID=xxxxxx;Path=/;HttpOnly
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Type: text/html;charset=utf-8
< Content-Length: 2265
< Server: Jetty(9.3.14.v20161028)
<
<!doctype html>
<!--
....
*
* http://www.apache.org/licenses/LICENSE-2.0
....
-->
<!--[if gt IE 8]>
<script type="text/javascript">
function Redirect() {
window.location.assign("login.jsp");
}
Redirect();
</script>
<![endif]-->
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if gt IE 7]>
<script src="js/external_lib/es5-shim.min.js"></script>
<script src="js/external_lib/respond.min.js"></script>
<![endif]-->
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Atlas</title>
....
</head>
<body>
<div id="wrapper">
<!-- Page Content Begin -->
<div id="page-content-wrapper">
<div class="page-title clearfix">
<h3>Looking for something?</h3>
<p>We're sorry. The web address you're looking for is not a functioning page in Apache Atlas. Please try navigating from Apache Atlas Home</p>
</div>
</div>
<!-- Page Content End -->
</div>
</body>
</html>
* Connection #0 to host HW03.co.local left intact
does not work and not sure what to do about the error message. Even the simple example of
curl -vv -u admin:admin -X POST --header 'Content-Type: application/json;charset=UTF-8' --header 'Accept: application/json' -d '{ \
"end1": { \
"guid": "2ddcda5b-2489-4636-a9ab-12b199c02422" \
}, \
"end2": { \
"guid": "a33f45de-13d0-4a30-9df7-b0e02eb0dfd5" \
}, \
"typeName": "AtlasRelationshipDef" \
}' 'http://HW03.ucera.local:21000/api/atlas/v2/relationship'
throws a similarly uninformative error
* upload completely sent off: 211 out of 211 bytes
< HTTP/1.1 500 Internal Server Error
< Date: Mon, 12 Aug 2019 19:57:44 GMT
< Set-Cookie: ATLASSESSIONID=xxxxxx;Path=/;HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Server: Jetty(9.3.14.v20161028)
<
* Connection #0 to host HW03.co.local left intact
There was an error processing your request. It has been logged (ID 6d64bc3a1a910e46)
Checking the logs in /var/logs/atlas/application.log on the atlas host server, I can see yet another uninformative error message...
[hph_etl#HW03 atlas]$ cat application.log | grep -C 2 6d64bc3a1a910e46
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2019-08-12 09:57:44,880 ERROR - [pool-2-thread-10 - 8a5535b1-6544-4f9b-b3ad-8bec5e8d6fcd:] ~ Error handling a request: 6d64bc3a1a910e46 (ExceptionMapperUtil:32)
javax.ws.rs.WebApplicationException
at com.sun.jersey.server.impl.uri.rules.TerminatingRule.accept(TerminatingRule.java:66)
Note that for getting the guids required for the relationship linking, even Hortonworks seems to provide only a poor solution.
What could be going wrong here? Are there any better docs than those linked to for understanding the API?
Atlas relationship between existing entities can be created either using entity GUIDs or uniqueAttributes in end1 and end2 which can be qualifiedName or any other unique attribute .
Please do note that top level typeName is the relationship def typeName while typeName inside end1 and end2 is entity typeName.
In case of relationship between hive_table and hive_db the relationship def typeName is: hive_table_db
So, if you want to create a relationship between hive_table and hive_db, the request would be:
POST: /api/atlas/v2/relationship
{
"typeName": "hive_table_db",
"end1": {
"typeName": "hive_table",
"uniqueAttributes": {
"qualifiedName": "db.table#cluster"
}
},
"end2": {
"typeName": "hive_db",
"uniqueAttributes": {
"qualifiedName": "db#cluster"
}
}
}
For predefined Atlas types you can find the relationship typeName from its definition inside relationshipAttributeDefs field
GET: /api/atlas/v2/types/typedef/name/hive_db
Which gives the following response:
{
"category": "ENTITY",
"guid": "78c44290-2ed8-461b-953d-3965d9bb44ca",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1548175553859,
"updateTime": 1548175822249,
"version": 2,
"name": "hive_db",
"description": "hive_db",
"typeVersion": "1.2",
"serviceType": "hive",
"attributeDefs": [
{
"name": "clusterName",
"typeName": "string",
"isOptional": false,
"cardinality": "SINGLE",
"valuesMinCount": 1,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": true,
"includeInNotification": true,
"searchWeight": -1
},
{
"name": "location",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1
},
{
"name": "parameters",
"typeName": "map<string,string>",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1
},
{
"name": "ownerType",
"typeName": "hive_principal_type",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1
}
],
"superTypes": [
"Asset"
],
"subTypes": [],
"relationshipAttributeDefs": [
{
"name": "tables",
"typeName": "array<hive_table>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": -1,
"valuesMaxCount": -1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1,
"constraints": [
{
"type": "ownedRef"
}
],
"relationshipTypeName": "hive_table_db",
"isLegacyAttribute": false
},
{
"name": "ddlQueries",
"typeName": "array<hive_db_ddl>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": -1,
"valuesMaxCount": -1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1,
"constraints": [
{
"type": "ownedRef"
}
],
"relationshipTypeName": "hive_db_ddl_queries",
"isLegacyAttribute": false
},
{
"name": "meanings",
"typeName": "array<AtlasGlossaryTerm>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": -1,
"valuesMaxCount": -1,
"isUnique": false,
"isIndexable": false,
"includeInNotification": false,
"searchWeight": -1,
"relationshipTypeName": "AtlasGlossarySemanticAssignment",
"isLegacyAttribute": false
}
]
}
Further, you need to make sure the typeName in end1 and end2 is as per the relationship def, which you can check in type definition:
GET: /api/atlas/v2/types/typedef/name/hive_table_db
{
"category": "RELATIONSHIP",
"guid": "79257a2c-407c-4c0b-b3ae-04b1b3a8d649",
"createdBy": "root",
"updatedBy": "root",
"createTime": 1548175553894,
"updateTime": 1548175553894,
"version": 1,
"name": "hive_table_db",
"description": "hive_table_db",
"typeVersion": "1.0",
"serviceType": "hive",
"attributeDefs": [],
"relationshipCategory": "COMPOSITION",
"propagateTags": "NONE",
"endDef1": {
"type": "hive_table",
"name": "db",
"isContainer": false,
"cardinality": "SINGLE",
"isLegacyAttribute": true
},
"endDef2": {
"type": "hive_db",
"name": "tables",
"isContainer": true,
"cardinality": "SET",
"isLegacyAttribute": false
}
}
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -u admin:admin 'http://localhost:21000/api/atlas/v2/relationship' -d '
{
"typeName": "???",
"end1": {"guid" : ""},
"end2": {"guid" : ""}
}
'
1. How to fullfil <???>
go to http://localhist:21000/api/atlas/v2/types/typedef/name/<YOUR ENTITY>
to find relationshipAttributeDefs for this ENTITY
You need "relationshipTypeName" where "rdbms_foreign_key_column_references"="???"
2. than exec
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -u admin:admin 'http://localhost:21000/api/atlas/v2/relationship' -d '
{
"typeName": "rdbms_foreign_key_column_references",
"end1": {"guid" : ""},
"end2": {"guid" : ""}
}
'
you will have error
"errorMessage":"Relationship end is invalid. Expected rdbms_foreign_key but is NULL"
where you will know the needs ENTITY
Related
I am trying to link to an article that has an alias associated with it. I have not been able to find any documentation on how this is done.
Here is what I have done so far:
Step 1: Created an Article by posting it to the /articles API:
curl -X 'POST' \
'https://mysubdomain.vanilladevelopment.com/api/v2/articles' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'x-transient-key: mykey' \
-d '{
"body": "demo",
"draftID": 0,
"format": "text",
"knowledgeCategoryID": 512,
"name": "Article Title"
}'
I received positive confirmation that an article was created:
{
"articleID": 987,
"articleRevisionID": 1348,
"knowledgeCategoryID": 512,
"breadcrumbs": [
{
"name": "Learners",
"url": "https://mysubdomain.vanilladevelopment.com/english/kb/documentation-learners"
}
],
"knowledgeBaseID": 4,
"name": "Article Title",
"body": "demo",
"outline": [],
"excerpt": "demo",
"seoDescription": null,
"seoName": null,
"slug": "987-article-title",
"sort": 14,
"score": 0,
"views": 0,
"url": "https://mysubdomain.vanilladevelopment.com/english/kb/articles/987-article-title",
"insertUserID": 9,
"dateInserted": "2022-08-19T13:34:33+00:00",
"updateUserID": 9,
"dateUpdated": "2022-08-19T13:34:33+00:00",
"status": "published",
"featured": false,
"dateFeatured": null,
"locale": "en",
"translationStatus": "up-to-date",
"foreignID": null
}
Step 2: Created an alias to the article using the articles/{id}/aliases API
curl -X 'PUT' \
'https://mysubdomain.vanilladevelopment.com/api/v2/articles/987/aliases' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'x-transient-key: mykey' \
-d '{
"aliases": [
"andy987"
]
}'
I received positive confirmation that the alias was created:
{
"articleID": 987,
"articleRevisionID": 1348,
"knowledgeCategoryID": 512,
"breadcrumbs": [
{
"name": "Learners",
"url": "https://mysubdomain.vanilladevelopment.com/english/kb/documentation-learners"
}
],
"knowledgeBaseID": 4,
"name": "Article Title",
"body": "demo",
"outline": [],
"excerpt": "demo",
"seoDescription": null,
"seoName": null,
"slug": "987-article-title",
"sort": 14,
"score": 0,
"views": 0,
"url": "https://mysubdomain.vanilladevelopment.com/english/kb/articles/987-article-title",
"insertUserID": 9,
"dateInserted": "2022-08-19T13:34:33+00:00",
"updateUserID": 9,
"dateUpdated": "2022-08-19T13:34:33+00:00",
"aliases": [
"andy987"
],
"status": "published",
"featured": false,
"dateFeatured": null,
"locale": "en",
"translationStatus": "up-to-date",
"foreignID": null
}
My question is how do I refer to the alias (i.e. andy987) from within another article ? I have tried:
https://mysubdoamin.vanilladevelopment.com/english/kb/articles/andy987
https://mysubdoamin.vanilladevelopment.com/english/kb/andy987
https://mysubdoamin.vanilladevelopment.com/andy987
and none of these work.
<<<<<<<<<<
I've been working on creating a script utilizes curl and variables. Doing some searches I found how to place a variable in the data portion of the curl but now I'm getting errors from the curl command.
(Some parts of the code removed to keep passwords out, everything is working except for entering the data into the --data portion of curl)
curlData=$(cat <<EOF
{
"rebootClient": false,
"createPseudoClientRequest": {
"registerClient": true,
"clientInfo": {
"clientType": 0,
}
},
"packages": [
{
"packageId": 702,
"packageName": "File System",
"packageId": 51,
"packageName": "MediaAgent"
}
],
},
"entities": [
{
"clientId": 0,
}
]
}
EOF
)
shopt -u nocasematch #Sets options back to being case sensitive
echo "$csName"
curl -vv --location --request POST "http://$csName:81/SearchSvc/CVWebService.svc/InstallClient" --header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header "Authtoken: $token" \
-d "$curlData"
Doing a bash -x everything looks correct
"rebootClient": false,
"createPseudoClientRequest": {
"registerClient": true,
"clientInfo": {
"clientType": 0,
}
},
"packages": [
{
"packageId": 702,
"packageName": "File System",
"packageId": 51,
"packageName": "MediaAgent"
}
],
"clientAuthForJob": {
},
"entities": [
{
"clientId": 0,
"clientName": "srybrcost",
}
]
}'
But every time I get Request body is empty or format is invalid
I am using Packetbeat to monitor the requests/responses into/out of Elasticsearch client nodes using the http protocol watcher on port 9200. I am sending the output of Packetbeat through Logstash, and then from there out to a different instance of Elasticsearch. We have compression support enabled in the Elasticsearch that is being monitored, so I occasionally see requests with "Accept-Encoding: gzip, deflate" headers returning responses that are gzipped. Unfortunately, I have not been able to decode any of these gzip responses using any tools I have at my disposal (including the web-based converters, the gzip command line tool, and using Zlib::GzipReader in a Logstash ruby filter script). They all report that it is not a gzip format.
Does anyone know why I can't seem to decode the gzip content?
I have provided a sample of the filter I'm using in Logstash to try to do this on the fly as the event passes through Logstash (and it always reports that http.response.body is not in gzip format).
filter {
if [type] == "http" {
if [http][response][headers][content-encoding] == "gzip" {
ruby {
init => "
require 'zlib'
require 'stringio'
"
code => "
body = event.get('[http][response][body]').to_s
sio = StringIO.new(body)
gz = Zlib::GzipReader.new(sio)
result = gz.read.to_s
event.set('[http][response][body]', result)
"
}
}
}
}
I'm also providing a sample of the logged event here which includes the gzip content in case you would like to try to decompress it yourself:
{
"_index": "packetbeat-6.2.3-2018.05.19",
"_type": "doc",
"_id": "oH0bemMB2mAXfg5euIiP",
"_score": 1,
"_source": {
"server": "",
"client_server": "",
"bytes_in": 160,
"bytes_out": 361,
"#timestamp": "2018-05-19T20:33:46.470Z",
"client_port": 55863,
"path": "/",
"type": "http",
"client_proc": "",
"query": "GET /",
"port": 9200,
"host": "gke-main-production-elastic-clients-5728bab3-t1z8",
"#version": "1",
"responsetime": 0,
"fields": {
"nodePool": "production-elastic-clients"
},
"response": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-encoding: gzip\r\ncontent-length: 250\r\n\r\n\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
"proc": "",
"request": "GET / HTTP/1.1\r\nUser-Agent: vscode-restclient\r\nhost: es-http-dev.elastic-prod.svc.cluster.local:9200\r\naccept-encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n",
"beat": {
"name": "gke-main-production-elastic-clients-5728bab3-t1z8",
"version": "6.2.3",
"hostname": "gke-main-production-elastic-clients-5728bab3-t1z8"
},
"status": "OK",
"method": "GET",
"client_ip": "10.24.20.6",
"http": {
"response": {
"phrase": "OK",
"headers": {
"content-encoding": "gzip",
"content-length": 250,
"content-type": "application/json; charset=UTF-8"
},
"body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
"code": 200
},
"request": {
"params": "",
"headers": {
"connection": "keep-alive",
"user-agent": "vscode-restclient",
"content-length": 0,
"host": "es-http-dev.elastic-prod.svc.cluster.local:9200",
"accept-encoding": "gzip, deflate"
}
}
},
"tags": [
"beats",
"beats_input_raw_event"
],
"ip": "10.24.41.5"
},
"fields": {
"#timestamp": [
"2018-05-19T20:33:46.470Z"
]
}
}
And this is the response for that message that I receive at the client after it has been decompressed successfully by the client:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 250
{
"name": "es-client-7688c8d9b9-qp9l7",
"cluster_name": "esprod",
"cluster_uuid": "8iRwLMMSR72F76ZEONYcUg",
"version": {
"number": "5.6.3",
"build_hash": "1a2f265",
"build_date": "2017-10-06T20:33:39.012Z",
"build_snapshot": false,
"lucene_version": "6.6.1"
},
"tagline": "You Know, for Search"
}
I had a different situation and was able to resolve my issue. Posting it here, see if it helps your case.
I was using postman tool to test my REST API services locally. My Packetbeat used following config.
type: http
ports: [80, 8080, 8000, 5000, 8002]
send_all_headers: true
include_body_for: ["application/json", "x-www-form-urlencoded"]
send_request: true
send_response: true
I was getting following output in body.
I was able to get http.response.body in clear text when i added following to my postman request.
Accept-Encoding: application/json
I'm working in a Jenkinsfile and trying to parse the following return output. I can get the uuid but I can't get name. Looking for some guidance.
Jenkinsfile
node('ansible'){
stage('Get VM List'){
def content = sh (returnStdout: true, script: "curl -X GET --header 'Content-Type: application/json' --header 'Accept: application/octet-stream' 'http://someurlapi'").trim()
def vmList = readJSON text: content;
//Works
echo vmList[0].uuid
}}
Return Output
[
{
"num": XX,
"ip": "XX.XX.XX.XX",
"type": "KVM",
"name": "machinename",
"state": "Running",
"ram": 4096,
"ram-display": "4 GiB",
"zpool": {
"name": "zpool",
"compression": "lz4",
"mountpoint": "\/mnt",
"mounted": true
},
"uuid": "d7622bd3-ed3d-5000-ae01-89ab294933r1",
"autostart": false,
"cpu": 2
}]
I figured it out I changed it to
echo vmList[0]["name"]
I am setting up new payment method in e-shop.
The requirements are:
URL: POST https://payparts2.privatbank.ua/ipp/v2/payment/create
server-server
Headers
Accept: application/json;
Accept-Encoding: UTF-8;
Content-Type: application/json; charset=UTF-8;
Body
{
"storeId": "",
"orderId": "",
"amount": 300.00,
"currency": "980",
"partsCount": 6,
"merchantType": "PP",
"products": [
{
"name": "TV",
"count": 2,
"price": 100.00
},
{
"name": "microwave",
"count": 1,
"price": 200.00
}
],
"responseUrl": "http://shop.com/response",
"redirectUrl": "http://shop.com/redirect",
"signature": ""
}
Here's my code:
http://jsfiddle.net/olga_smelo/hasb5kkr/9/
I've got an error:
FAILContent type 'application/x-www-form-urlencoded;charset=UTF-8' not supporteduk_UA
, but I set "Content type" as "application/json" in ajax
Help me please!