Elasticsearch/Filebeat _meta/fields.yml not loading into kibana - elasticsearch

I'm assigned to a filebeat plugin of our product. I wrote a IngestPipeline and de default mapping is in various ways unfortunately so i wanted to change this with the _meta/fields.yml.
I configured the yml file accordingly to:
https://www.elastic.co/guide/en/beats/devguide/7.8/filebeat-modules-devguide.html#_metafields_yml_2
https://www.elastic.co/guide/en/beats/devguide/current/event-fields-yml.html
https://github.com/elastic/kibana/issues/82273
My yml file looks like this:
- key: pac
title: "pac"
description: Description missing
pac Module
fields:
- name: pac.log
type: group
description: Description missing
fields:
- name: deo
type: group
description: Description missing
fields:
- name: duration
type: long
description: Duration of the deo
- name: category
type: keyword
description: Category of the deo
- name: owner
type: keyword
description: User of the deo
- name: version
type: float
description: Version of the deo
- name: name
type: keyword
description: Name of the deo
- name: reference
type: double
description: Referencenumber of the deo
- name: state
type: keyword
description: State of the deo
- name: status
type: keyword
description: Status of the deo
- name: trigger
type: group
description: Description missing
fields:
- name: category
type: keyword
description: Category of the deo-trigger
- name: name
type: text
description: Name of the deo-trigger
- name: path
type: text
description: Full-Text of the trigger properties
- name: provider
type: keyword
description: Supplier of the deo-trigger
- name: wiring
type: group
description: Description missing
fields:
- name: async
type: boolean
description: If deos wired asynchronously
- name: deoId
type: keyword
description: Identification number of the deo
- name: execute
type: boolean
description: If deo is executed or not
- name: owner
type: keyword
description: User of the deo
- name: shared
type: boolean
description: Deo was shared
- name: stopOnError
type: boolean
description: If deo stopped on Error
- name: do
type: group
description: Description missing
fields:
- name: name
type: keyword
description: Name of the do task
- name: state
type: group
description: Description missing
fields:
- name: from
type: keyword
description: State from which the do task was used
- name: to
type: keyword
description: State to which the do task was switched
- name: esa
type: group
description: Description missing
fields:
- name: connection
type: keyword
description: Connection status of the ESA
- name: name
type: keyword
description: Name of the ESA
- name: state
type: group
description: Description missing
fields:
- name: from
type: keyword
description: State from which the ESA Connection was established
- name: to
type: keyword
description: State to which the ESA Connection was established
- name: monitor
type: group
description: Description missing
fields:
- name: heap
type: group
description: Description missing
fields:
- name: bytes
type: long
description: Used bytes of the heap
- name: pct
type: float
description: Percentage of the maximum available bytes
- name: heapgc
type: group
description: Description missing
fields:
- name: bytes
type: long
description: Used bytes of the heapgc
- name: pct
type: float
description: Percentage of the maximum available bytes
- name: service
type: group
description: Description missing
fields:
- name: class
type: keyword
description: Class of the service
- name: duration
type: long
description: How long the service call took
- name: name
type: keyword
description: Name of the service call
- name: operation
type: keyword
description: Operation type of the service call
- name: success
type: boolean
description: Was the service call successfull or not
- name: system
type: group
description: Description missing
fields:
- name: category
type: keyword
description: Category of the System
- name: priority
type: keyword
description: Priority of the System
- name: monitor
type: group
description: Description missing
fields:
- name: cpu
type: group
description: Description missing
fields:
- name: pct
type: float
description: Percentage of the cpu usage
- name: wiring
type: text
description: Description missing
- name: meta
type: text
description: Description missing
- name: tags
type: keyword
description: Description missing
- name: timestamp
type: date
description: Description missing
- name: level
type: keyword
description: Description missing
- name: logger
type: keyword
description: Description missing
I would like to have the field types as configured in the fields.yml file. But at first no mapping is available and if i start to send logs the mapping for my fields is created automatically with unfortunate types.

The yaml file works as it is in the description above. My Problem was that my collegue from whom i inherited the project didn't use the filebeat build script. So the uploaded fields.yml file wasn't extended by my fields. I reconfigured the fields.yaml path inside the filebeat.yml configuration file and now it works.

Related

how to convert yaml file data to a string in bash

How to convert YAML file data into string output, so that we can use it as a body parameter in the below curl request with the correct escape pattern.
curl request
curl -s -H "Accept: application/json" -H "Content-Type: application/json" --location --request POST "${HOST}/api/v1/projects" --header "Authorization: Bearer "$token"" -d '{"name":"'${PROJECT_NAME}'","openAPISpec":"none","isFileLoad": "true","openText": '${openText}',"source": "API"}'
---
swagger: '2.0'
info:
description: 'This is a sample server Petstore server. You can find out more about
Swagger at [http://swagger.io](http://swagger.io) or on [irc.freenode.net, #swagger](http://swagger.io/irc/). For
this sample, you can use the api key `special-key` to test the authorization filters.'
version: 1.0.6
title: Swagger Petstore
termsOfService: http://swagger.io/terms/
contact:
email: apiteam#swagger.io
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
host: petstore.swagger.io
basePath: "/v2"
tags:
- name: pet
description: Everything about your Pets
externalDocs:
description: Find out more
url: http://swagger.io
- name: store
description: Access to Petstore orders
- name: user
description: Operations about user
externalDocs:
description: Find out more about our store
url: http://swagger.io
schemes:
- https
- http
paths:
"/pet/{petId}/uploadImage":
post:
tags:
- pet
summary: uploads an image
description: ''
operationId: uploadFile
consumes:
- multipart/form-data
produces:
- application/json
parameters:
- name: petId
in: path
description: ID of pet to update
required: true
type: integer
format: int64
- name: additionalMetadata
in: formData
description: Additional data to pass to server
required: false
type: string
- name: file
in: formData
description: file to upload
required: false
type: file
responses:
'200':
description: successful operation
schema:
"$ref": "#/definitions/ApiResponse"
security:
- petstore_auth:
- write:pets
- read:pets
"/pet":
post:
tags:
- pet
summary: Add a new pet to the store
description: ''
operationId: addPet
consumes:
- application/json
- application/xml
produces:
- application/json
- application/xml
parameters:
- in: body
name: body
description: Pet object that needs to be added to the store
required: true
schema:
"$ref": "#/definitions/Pet"
responses:
'405':
description: Invalid input
security:
- petstore_auth:
- write:pets
- read:pets
put:
tags:
- pet
summary: Update an existing pet
description: ''
operationId: updatePet
consumes:
- application/json
- application/xml
produces:
- application/json
- application/xml
parameters:
- in: body
name: body
description: Pet object that needs to be added to the store
required: true
schema:
"$ref": "#/definitions/Pet"
responses:
'400':
description: Invalid ID supplied
'404':
description: Pet not found
'405':
description: Validation exception
security:
- petstore_auth:
- write:pets
- read:pets
"/pet/findByStatus":
get:
tags:
- pet
summary: Finds Pets by status
description: Multiple status values can be provided with comma separated strings
operationId: findPetsByStatus
produces:
- application/json
- application/xml
parameters:
- name: status
in: query
description: Status values that need to be considered for filter
required: true
type: array
items:
type: string
enum:
- available
- pending
- sold
default: available
collectionFormat: multi
responses:
'200':
description: successful operation
schema:
type: array
items:
"$ref": "#/definitions/Pet"
'400':
description: Invalid status value
security:
- petstore_auth:
- write:pets
- read:pets
"/pet/findByTags":
get:
tags:
- pet
summary: Finds Pets by tags
description: Multiple tags can be provided with comma separated strings. Use
tag1, tag2, tag3 for testing.
operationId: findPetsByTags
produces:
- application/json
- application/xml
parameters:
- name: tags
in: query
description: Tags to filter by
required: true
type: array
items:
type: string
collectionFormat: multi
responses:
'200':
description: successful operation
schema:
type: array
items:
"$ref": "#/definitions/Pet"
'400':
description: Invalid tag value
security:
- petstore_auth:
- write:pets
- read:pets
deprecated: true
"/pet/{petId}":
get:
tags:
- pet
summary: Find pet by ID
description: Returns a single pet
operationId: getPetById
produces:
- application/json
- application/xml
parameters:
- name: petId
in: path
description: ID of pet to return
required: true
type: integer
format: int64
responses:
'200':
description: successful operation
schema:
"$ref": "#/definitions/Pet"
'400':
description: Invalid ID supplied
'404':
description: Pet not found
security:
- api_key: []
post:
tags:
- pet
summary: Updates a pet in the store with form data
description: ''
operationId: updatePetWithForm
consumes:
- application/x-www-form-urlencoded
produces:
- application/json
- application/xml
parameters:
- name: petId
in: path
description: ID of pet that needs to be updated
required: true
type: integer
format: int64
- name: name
in: formData
description: Updated name of the pet
required: false
type: string
- name: status
in: formData
description: Updated status of the pet
required: false
type: string
responses:
'405':
description: Invalid input
security:
- petstore_auth:
- write:pets
- read:pets
delete:
tags:
- pet
summary: Deletes a pet
description: ''
operationId: deletePet
produces:
- application/json
- application/xml
parameters:
- name: api_key
in: header
required: false
type: string
- name: petId
in: path
description: Pet id to delete
required: true
type: integer
format: int64
responses:
'400':
description: Invalid ID supplied
'404':
description: Pet not found
security:
- petstore_auth:
- write:pets
- read:pets
"/store/order":
post:
tags:
- store
summary: Place an order for a pet
description: ''
operationId: placeOrder
consumes:
- application/json
produces:
- application/json
- application/xml
parameters:
- in: body
name: body
description: order placed for purchasing the pet
required: true
schema:
"$ref": "#/definitions/Order"
responses:
'200':
description: successful operation
schema:
"$ref": "#/definitions/Order"
'400':
description: Invalid Order
"/store/order/{orderId}":
get:
tags:
- store
summary: Find purchase order by ID
description: For valid response try integer IDs with value >= 1 and <= 10. Other
values will generated exceptions
operationId: getOrderById
produces:
- application/json
- application/xml
parameters:
- name: orderId
in: path
description: ID of pet that needs to be fetched
required: true
type: integer
maximum: 10
minimum: 1
format: int64
responses:
'200':
description: successful operation
schema:
"$ref": "#/definitions/Order"
'400':
description: Invalid ID supplied
'404':
description: Order not found
delete:
tags:
- store
summary: Delete purchase order by ID
description: For valid response try integer IDs with positive integer value.
Negative or non-integer values will generate API errors
operationId: deleteOrder
produces:
- application/json
- application/xml
parameters:
- name: orderId
in: path
description: ID of the order that needs to be deleted
required: true
type: integer
minimum: 1
format: int64
responses:
'400':
description: Invalid ID supplied
'404':
description: Order not found
"/store/inventory":
get:
tags:
- store
summary: Returns pet inventories by status
description: Returns a map of status codes to quantities
operationId: getInventory
produces:
- application/json
parameters: []
responses:
'200':
description: successful operation
schema:
type: object
additionalProperties:
type: integer
format: int32
security:
- api_key: []
"/user/createWithArray":
post:
tags:
- user
summary: Creates list of users with given input array
description: ''
operationId: createUsersWithArrayInput
consumes:
- application/json
produces:
- application/json
- application/xml
parameters:
- in: body
name: body
description: List of user object
required: true
schema:
type: array
items:
"$ref": "#/definitions/User"
responses:
default:
description: successful operation
"/user/createWithList":
post:
tags:
- user
summary: Creates list of users with given input array
description: ''
operationId: createUsersWithListInput
consumes:
- application/json
produces:
- application/json
- application/xml
parameters:
- in: body
name: body
description: List of user object
required: true
schema:
type: array
items:
"$ref": "#/definitions/User"
responses:
default:
description: successful operation
"/user/{username}":
get:
tags:
- user
summary: Get user by user name
description: ''
operationId: getUserByName
produces:
- application/json
- application/xml
parameters:
- name: username
in: path
description: 'The name that needs to be fetched. Use user1 for testing. '
required: true
type: string
responses:
'200':
description: successful operation
schema:
"$ref": "#/definitions/User"
'400':
description: Invalid username supplied
'404':
description: User not found
put:
tags:
- user
summary: Updated user
description: This can only be done by the logged in user.
operationId: updateUser
consumes:
- application/json
produces:
- application/json
- application/xml
parameters:
- name: username
in: path
description: name that need to be updated
required: true
type: string
- in: body
name: body
description: Updated user object
required: true
schema:
"$ref": "#/definitions/User"
responses:
'400':
description: Invalid user supplied
'404':
description: User not found
delete:
tags:
- user
summary: Delete user
description: This can only be done by the logged in user.
operationId: deleteUser
produces:
- application/json
- application/xml
parameters:
- name: username
in: path
description: The name that needs to be deleted
required: true
type: string
responses:
'400':
description: Invalid username supplied
'404':
description: User not found
"/user/login":
get:
tags:
- user
summary: Logs user into the system
description: ''
operationId: loginUser
produces:
- application/json
- application/xml
parameters:
- name: username
in: query
description: The user name for login
required: true
type: string
- name: password
in: query
description: The password for login in clear text
required: true
type: string
responses:
'200':
description: successful operation
headers:
X-Expires-After:
type: string
format: date-time
description: date in UTC when token expires
X-Rate-Limit:
type: integer
format: int32
description: calls per hour allowed by the user
schema:
type: string
'400':
description: Invalid username/password supplied
"/user/logout":
get:
tags:
- user
summary: Logs out current logged in user session
description: ''
operationId: logoutUser
produces:
- application/json
- application/xml
parameters: []
responses:
default:
description: successful operation
"/user":
post:
tags:
- user
summary: Create user
description: This can only be done by the logged in user.
operationId: createUser
consumes:
- application/json
produces:
- application/json
- application/xml
parameters:
- in: body
name: body
description: Created user object
required: true
schema:
"$ref": "#/definitions/User"
responses:
default:
description: successful operation
securityDefinitions:
api_key:
type: apiKey
name: api_key
in: header
petstore_auth:
type: oauth2
authorizationUrl: https://petstore.swagger.io/oauth/authorize
flow: implicit
scopes:
read:pets: read your pets
write:pets: modify pets in your account
definitions:
ApiResponse:
type: object
properties:
code:
type: integer
format: int32
type:
type: string
message:
type: string
Category:
type: object
properties:
id:
type: integer
format: int64
name:
type: string
xml:
name: Category
Pet:
type: object
required:
- name
- photoUrls
properties:
id:
type: integer
format: int64
category:
"$ref": "#/definitions/Category"
name:
type: string
example: doggie
photoUrls:
type: array
xml:
wrapped: true
items:
type: string
xml:
name: photoUrl
tags:
type: array
xml:
wrapped: true
items:
xml:
name: tag
"$ref": "#/definitions/Tag"
status:
type: string
description: pet status in the store
enum:
- available
- pending
- sold
xml:
name: Pet
Tag:
type: object
properties:
id:
type: integer
format: int64
name:
type: string
xml:
name: Tag
Order:
type: object
properties:
id:
type: integer
format: int64
petId:
type: integer
format: int64
quantity:
type: integer
format: int32
shipDate:
type: string
format: date-time
status:
type: string
description: Order Status
enum:
- placed
- approved
- delivered
complete:
type: boolean
xml:
name: Order
User:
type: object
properties:
id:
type: integer
format: int64
username:
type: string
firstName:
type: string
lastName:
type: string
email:
type: string
password:
type: string
phone:
type: string
userStatus:
type: integer
format: int32
description: User Status
xml:
name: User
externalDocs:
description: Find out more about Swagger
url: http://swagger.io
Below is the successful string output I got from the request payload when I use 'Inspect' from the google browser.
So when we convert the YAML file data to string from the Linux bash and use it in the curl request mentioned above for '${openText}' field we are expected to have the string like the one below the string output.
I tried to yq tool but don't know to use it or whether it will give the exact result.
Expected/Close-To-Expected is string output.
"---\nswagger: '2.0'\ninfo:\n description: 'This is a sample server Petstore server. You can find out more about\n Swagger at [http://swagger.io](http://swagger.io) or on [irc.freenode.net, #swagger](http://swagger.io/irc/). For\n this sample, you can use the api key `special-key` to test the authorization filters.'\n version: 1.0.6\n title: Swagger Petstore\n termsOfService: http://swagger.io/terms/\n contact:\n email: apiteam#swagger.io\n license:\n name: Apache 2.0\n url: http://www.apache.org/licenses/LICENSE-2.0.html\nhost: petstore.swagger.io\nbasePath: \"/v2\"\ntags:\n- name: pet\n description: Everything about your Pets\n externalDocs:\n description: Find out more\n url: http://swagger.io\n- name: store\n description: Access to Petstore orders\n- name: user\n description: Operations about user\n externalDocs:\n description: Find out more about our store\n url: http://swagger.io\nschemes:\n- https\n- http\npaths:\n \"/pet/{petId}/uploadImage\":\n post:\n tags:\n - pet\n summary: uploads an image\n description: ''\n operationId: uploadFile\n consumes:\n - multipart/form-data\n produces:\n - application/json\n parameters:\n - name: petId\n in: path\n description: ID of pet to update\n required: true\n type: integer\n format: int64\n - name: additionalMetadata\n in: formData\n description: Additional data to pass to server\n required: false\n type: string\n - name: file\n in: formData\n description: file to upload\n required: false\n type: file\n responses:\n '200':\n description: successful operation\n schema:\n \"$ref\": \"#/definitions/ApiResponse\"\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n \"/pet\":\n post:\n tags:\n - pet\n summary: Add a new pet to the store\n description: ''\n operationId: addPet\n consumes:\n - application/json\n - application/xml\n produces:\n - application/json\n - application/xml\n parameters:\n - in: body\n name: body\n description: Pet object that needs to be added to the store\n required: true\n schema:\n \"$ref\": \"#/definitions/Pet\"\n responses:\n '405':\n description: Invalid input\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n put:\n tags:\n - pet\n summary: Update an existing pet\n description: ''\n operationId: updatePet\n consumes:\n - application/json\n - application/xml\n produces:\n - application/json\n - application/xml\n parameters:\n - in: body\n name: body\n description: Pet object that needs to be added to the store\n required: true\n schema:\n \"$ref\": \"#/definitions/Pet\"\n responses:\n '400':\n description: Invalid ID supplied\n '404':\n description: Pet not found\n '405':\n description: Validation exception\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n \"/pet/findByStatus\":\n get:\n tags:\n - pet\n summary: Finds Pets by status\n description: Multiple status values can be provided with comma separated strings\n operationId: findPetsByStatus\n produces:\n - application/json\n - application/xml\n parameters:\n - name: status\n in: query\n description: Status values that need to be considered for filter\n required: true\n type: array\n items:\n type: string\n enum:\n - available\n - pending\n - sold\n default: available\n collectionFormat: multi\n responses:\n '200':\n description: successful operation\n schema:\n type: array\n items:\n \"$ref\": \"#/definitions/Pet\"\n '400':\n description: Invalid status value\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n \"/pet/findByTags\":\n get:\n tags:\n - pet\n summary: Finds Pets by tags\n description: Multiple tags can be provided with comma separated strings. Use\n tag1, tag2, tag3 for testing.\n operationId: findPetsByTags\n produces:\n - application/json\n - application/xml\n parameters:\n - name: tags\n in: query\n description: Tags to filter by\n required: true\n type: array\n items:\n type: string\n collectionFormat: multi\n responses:\n '200':\n description: successful operation\n schema:\n type: array\n items:\n \"$ref\": \"#/definitions/Pet\"\n '400':\n description: Invalid tag value\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n deprecated: true\n \"/pet/{petId}\":\n get:\n tags:\n - pet\n summary: Find pet by ID\n description: Returns a single pet\n operationId: getPetById\n produces:\n - application/json\n - application/xml\n parameters:\n - name: petId\n in: path\n description: ID of pet to return\n required: true\n type: integer\n format: int64\n responses:\n '200':\n description: successful operation\n schema:\n \"$ref\": \"#/definitions/Pet\"\n '400':\n description: Invalid ID supplied\n '404':\n description: Pet not found\n security:\n - api_key: []\n post:\n tags:\n - pet\n summary: Updates a pet in the store with form data\n description: ''\n operationId: updatePetWithForm\n consumes:\n - application/x-www-form-urlencoded\n produces:\n - application/json\n - application/xml\n parameters:\n - name: petId\n in: path\n description: ID of pet that needs to be updated\n required: true\n type: integer\n format: int64\n - name: name\n in: formData\n description: Updated name of the pet\n required: false\n type: string\n - name: status\n in: formData\n description: Updated status of the pet\n required: false\n type: string\n responses:\n '405':\n description: Invalid input\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n delete:\n tags:\n - pet\n summary: Deletes a pet\n description: ''\n operationId: deletePet\n produces:\n - application/json\n - application/xml\n parameters:\n - name: api_key\n in: header\n required: false\n type: string\n - name: petId\n in: path\n description: Pet id to delete\n required: true\n type: integer\n format: int64\n responses:\n '400':\n description: Invalid ID supplied\n '404':\n description: Pet not found\n security:\n - petstore_auth:\n - write:pets\n - read:pets\n \"/store/order\":\n post:\n tags:\n - store\n summary: Place an order for a pet\n description: ''\n operationId: placeOrder\n consumes:\n - application/json\n produces:\n - application/json\n - application/xml\n parameters:\n - in: body\n name: body\n description: order placed for purchasing the pet\n required: true\n schema:\n \"$ref\": \"#/definitions/Order\"\n responses:\n '200':\n description: successful operation\n schema:\n \"$ref\": \"#/definitions/Order\"\n '400':\n description: Invalid Order\n \"/store/order/{orderId}\":\n get:\n tags:\n - store\n summary: Find purchase order by ID\n description: For valid response try integer IDs with value >= 1 and <= 10. Other\n values will generated exceptions\n operationId: getOrderById\n produces:\n - application/json\n - application/xml\n parameters:\n - name: orderId\n in: path\n description: ID of pet that needs to be fetched\n required: true\n type: integer\n maximum: 10\n minimum: 1\n format: int64\n responses:\n '200':\n description: successful operation\n schema:\n \"$ref\": \"#/definitions/Order\"\n '400':\n description: Invalid ID supplied\n '404':\n description: Order not found\n delete:\n tags:\n - store\n summary: Delete purchase order by ID\n description: For valid response try integer IDs with positive integer value.\n Negative or non-integer values will generate API errors\n operationId: deleteOrder\n produces:\n - application/json\n - application/xml\n parameters:\n - name: orderId\n in: path\n description: ID of the order that needs to be deleted\n required: true\n type: integer\n minimum: 1\n format: int64\n responses:\n '400':\n description: Invalid ID supplied\n '404':\n description: Order not found\n \"/store/inventory\":\n get:\n tags:\n - store\n summary: Returns pet inventories by status\n description: Returns a map of status codes to quantities\n operationId: getInventory\n produces:\n - application/json\n parameters: []\n responses:\n '200':\n description: successful operation\n schema:\n type: object\n additionalProperties:\n type: integer\n format: int32\n security:\n - api_key: []\n \"/user/createWithArray\":\n post:\n tags:\n - user\n summary: Creates list of users with given input array\n description: ''\n operationId: createUsersWithArrayInput\n consumes:\n - application/json\n produces:\n - application/json\n - application/xml\n parameters:\n - in: body\n found\n put:\n tags:\n - user\n summary: Updated user\n \n password:\n type: string\n phone:\n type: string\n userStatus:\n type: integer\n format: int32\n description: User Status\n xml:\n name: User\nexternalDocs:\n description: Find out more about Swagger\n url: http://swagger.io\n\n"
You are setting JSON as the curl input, but you have YAML. So:
yq -p yaml -o json file.yaml | # convert YAML to JSON
curl -s -H "Accept: application/json" \
-H "Content-Type: application/json" \
--location \
--request POST "${HOST}/api/v1/projects" \
--header "Authorization: Bearer $token" \
-d "#/dev/stdin" https://<URL>
If you need to modify the YAML file, you can process it before with the yq command
The curl request header specifies it accepts 'application/json' but you're not passing in JSON.
To convert a yaml to a JSON string, instead of using sed do:
openText=$(yq -oj petstore.yaml)
Disclaimer: I wrote yq
I tried the below code.
openText=$(sed -z 's/\n/\\n/g' petstore.yaml | tr -d ' ')
curl -s -H "Accept: application/json" -H "Content-Type: application/json" --location --request POST "${HOST}/api/v1/projects" --header "Authorization: Bearer "$token"" -d '{"name":"'${PROJECT_NAME}'","openAPISpec":"none","isFileLoad": "true","openText": '${openText}',"source": "API"}'
String output seems to be ok with ( sed -z 's/\n/\n/g' petstore.yaml | tr -d ' ')
---\nswagger:'2.0'\ninfo:\ndescription:'ThisisasampleserverPetstoreserver.Youcanfindoutmoreabout\nSwaggerat[http://swagger.io](http://swagger.io)oron[irc.freenode.net,#swagger](http://swagger.io/irc/).For\nthissample,youcanusetheapikey`special-key`totesttheauthorizationfilters.'\nversion:1.0.6\ntitle:SwaggerPetstore\ntermsOfService:http://swagger.io/terms/\ncontact:\nemail:apiteam#swagger.io\nlicense:\nname:Apache2.0\nurl:http://www.apache.org/licenses/LICENSE-2.0.html\nhost:petstore.swagger.io\nbasePath:"/v2"\ntags:\n-name:pet\ndescription:EverythingaboutyourPets\nexternalDocs:\ndescription:Findoutmore\nurl:http://swagger.io\n-name:store\ndescription:AccesstoPetstoreorders\n-name:user\ndescription:Operationsaboutuser\nexternalDocs:\ndescription:Findoutmoreaboutourstore\nurl:http://swagger.io\nschemes:\n-https\n-http\npaths:\n"/pet/{petId}/uploadImage":\npost:\ntags:\n-pet\nsummary:uploadsanimage\ndescription:''\noperationId:uploadFile\nconsumes:\n-multipart/form-data\nproduces:\n-application/json\nparameters:\n-name:petId\nin:path\ndescription:IDofpettoupdate\nrequired:true\ntype:integer\nformat:int64\n-name:additionalMetadata\nin:formData\ndescription:Additionaldatatopasstoserver\nrequired:false\ntype:string\n-name:file\nin:formData\ndescription:filetoupload\nrequired:false\ntype:file\nresponses:\n'200':\ndescription:successfuloperation\nschema:\n"$ref":"#/definitions/ApiResponse"\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\n"/pet":\npost:\ntags:\n-pet\nsummary:Addanewpettothestore\ndescription:''\noperationId:addPet\nconsumes:\n-application/json\n-application/xml\nproduces:\n-application/json\n-application/xml\nparameters:\n-in:body\nname:body\ndescription:Petobjectthatneedstobeaddedtothestore\nrequired:true\nschema:\n"$ref":"#/definitions/Pet"\nresponses:\n'405':\ndescription:Invalidinput\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\nput:\ntags:\n-pet\nsummary:Updateanexistingpet\ndescription:''\noperationId:updatePet\nconsumes:\n-application/json\n-application/xml\nproduces:\n-application/json\n-application/xml\nparameters:\n-in:body\nname:body\ndescription:Petobjectthatneedstobeaddedtothestore\nrequired:true\nschema:\n"$ref":"#/definitions/Pet"\nresponses:\n'400':\ndescription:InvalidIDsupplied\n'404':\ndescription:Petnotfound\n'405':\ndescription:Validationexception\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\n"/pet/findByStatus":\nget:\ntags:\n-pet\nsummary:FindsPetsbystatus\ndescription:Multiplestatusvaluescanbeprovidedwithcommaseparatedstrings\noperationId:findPetsByStatus\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:status\nin:query\ndescription:Statusvaluesthatneedtobeconsideredforfilter\nrequired:true\ntype:array\nitems:\ntype:string\nenum:\n-available\n-pending\n-sold\ndefault:available\ncollectionFormat:multi\nresponses:\n'200':\ndescription:successfuloperation\nschema:\ntype:array\nitems:\n"$ref":"#/definitions/Pet"\n'400':\ndescription:Invalidstatusvalue\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\n"/pet/findByTags":\nget:\ntags:\n-pet\nsummary:FindsPetsbytags\ndescription:Multipletagscanbeprovidedwithcommaseparatedstrings.Use\ntag1,tag2,tag3fortesting.\noperationId:findPetsByTags\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:tags\nin:query\ndescription:Tagstofilterby\nrequired:true\ntype:array\nitems:\ntype:string\ncollectionFormat:multi\nresponses:\n'200':\ndescription:successfuloperation\nschema:\ntype:array\nitems:\n"$ref":"#/definitions/Pet"\n'400':\ndescription:Invalidtagvalue\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\ndeprecated:true\n"/pet/{petId}":\nget:\ntags:\n-pet\nsummary:FindpetbyID\ndescription:Returnsasinglepet\noperationId:getPetById\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:petId\nin:path\ndescription:IDofpettoreturn\nrequired:true\ntype:integer\nformat:int64\nresponses:\n'200':\ndescription:successfuloperation\nschema:\n"$ref":"#/definitions/Pet"\n'400':\ndescription:InvalidIDsupplied\n'404':\ndescription:Petnotfound\nsecurity:\n-api_key:[]\npost:\ntags:\n-pet\nsummary:Updatesapetinthestorewithformdata\ndescription:''\noperationId:updatePetWithForm\nconsumes:\n-application/x-www-form-urlencoded\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:petId\nin:path\ndescription:IDofpetthatneedstobeupdated\nrequired:true\ntype:integer\nformat:int64\n-name:name\nin:formData\ndescription:Updatednameofthepet\nrequired:false\ntype:string\n-name:status\nin:formData\ndescription:Updatedstatusofthepet\nrequired:false\ntype:string\nresponses:\n'405':\ndescription:Invalidinput\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\ndelete:\ntags:\n-pet\nsummary:Deletesapet\ndescription:''\noperationId:deletePet\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:api_key\nin:header\nrequired:false\ntype:string\n-name:petId\nin:path\ndescription:Petidtodelete\nrequired:true\ntype:integer\nformat:int64\nresponses:\n'400':\ndescription:InvalidIDsupplied\n'404':\ndescription:Petnotfound\nsecurity:\n-petstore_auth:\n-write:pets\n-read:pets\n"/store/order":\npost:\ntags:\n-store\nsummary:Placeanorderforapet\ndescription:''\noperationId:placeOrder\nconsumes:\n-application/json\nproduces:\n-application/json\n-application/xml\nparameters:\n-in:body\nname:body\ndescription:orderplacedforpurchasingthepet\nrequired:true\nschema:\n"$ref":"#/definitions/Order"\nresponses:\n'200':\ndescription:successfuloperation\nschema:\n"$ref":"#/definitions/Order"\n'400':\ndescription:InvalidOrder\n"/store/order/{orderId}":\nget:\ntags:\n-store\nsummary:FindpurchaseorderbyID\ndescription:ForvalidresponsetryintegerIDswithvalue>=1and<=10.Other\nvalueswillgeneratedexceptions\noperationId:getOrderById\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:orderId\nin:path\ndescription:IDofpetthatneedstobefetched\nrequired:true\ntype:integer\nmaximum:10\nminimum:1\nformat:int64\nresponses:\n'200':\ndescription:successfuloperation\nschema:\n"$ref":"#/definitions/Order"\n'400':\ndescription:InvalidIDsupplied\n'404':\ndescription:Ordernotfound\ndelete:\ntags:\n-store\nsummary:DeletepurchaseorderbyID\ndescription:ForvalidresponsetryintegerIDswithpositiveintegervalue.\nNegativeornon-integervalueswillgenerateAPIerrors\noperationId:deleteOrder\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:orderId\nin:path\ndescription:IDoftheorderthatneedstobedeleted\nrequired:true\ntype:integer\nminimum:1\nformat:int64\nresponses:\n'400':\ndescription:InvalidIDsupplied\n'404':\ndescription:Ordernotfound\n"/store/inventory":\nget:\ntags:\n-store\nsummary:Returnspetinventoriesbystatus\ndescription:Returnsamapofstatuscodestoquantities\noperationId:getInventory\nproduces:\n-application/json\nparameters:[]\nresponses:\n'200':\ndescription:successfuloperation\nschema:\ntype:object\nadditionalProperties:\ntype:integer\nformat:int32\nsecurity:\n-api_key:[]\n"/user/createWithArray":\npost:\ntags:\n-user\nsummary:Createslistofuserswithgiveninputarray\ndescription:''\noperationId:createUsersWithArrayInput\nconsumes:\n-application/json\nproduces:\n-application/json\n-application/xml\nparameters:\n-in:body\nname:body\ndescription:Listofuserobject\nrequired:true\nschema:\ntype:array\nitems:\n"$ref":"#/definitions/User"\nresponses:\ndefault:\ndescription:successfuloperation\n"/user/createWithList":\npost:\ntags:\n-user\nsummary:Createslistofuserswithgiveninputarray\ndescription:''\noperationId:createUsersWithListInput\nconsumes:\n-application/json\nproduces:\n-application/json\n-application/xml\nparameters:\n-in:body\nname:body\ndescription:Listofuserobject\nrequired:true\nschema:\ntype:array\nitems:\n"$ref":"#/definitions/User"\nresponses:\ndefault:\ndescription:successfuloperation\n"/user/{username}":\nget:\ntags:\n-user\nsummary:Getuserbyusername\ndescription:''\noperationId:getUserByName\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:username\nin:path\ndescription:'Thenamethatneedstobefetched.Useuser1fortesting.'\nrequired:true\ntype:string\nresponses:\n'200':\ndescription:successfuloperation\nschema:\n"$ref":"#/definitions/User"\n'400':\ndescription:Invalidusernamesupplied\n'404':\ndescription:Usernotfound\nput:\ntags:\n-user\nsummary:Updateduser\ndescription:Thiscanonlybedonebytheloggedinuser.\noperationId:updateUser\nconsumes:\n-application/json\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:username\nin:path\ndescription:namethatneedtobeupdated\nrequired:true\ntype:string\n-in:body\nname:body\ndescription:Updateduserobject\nrequired:true\nschema:\n"$ref":"#/definitions/User"\nresponses:\n'400':\ndescription:Invalidusersupplied\n'404':\ndescription:Usernotfound\ndelete:\ntags:\n-user\nsummary:Deleteuser\ndescription:Thiscanonlybedonebytheloggedinuser.\noperationId:deleteUser\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:username\nin:path\ndescription:Thenamethatneedstobedeleted\nrequired:true\ntype:string\nresponses:\n'400':\ndescription:Invalidusernamesupplied\n'404':\ndescription:Usernotfound\n"/user/login":\nget:\ntags:\n-user\nsummary:Logsuserintothesystem\ndescription:''\noperationId:loginUser\nproduces:\n-application/json\n-application/xml\nparameters:\n-name:username\nin:query\ndescription:Theusernameforlogin\nrequired:true\ntype:string\n-name:password\nin:query\ndescription:Thepasswordforloginincleartext\nrequired:true\ntype:string\nresponses:\n'200':\ndescription:successfuloperation\nheaders:\nX-Expires-After:\ntype:string\nformat:date-time\ndescription:dateinUTCwhentokenexpires\nX-Rate-Limit:\ntype:integer\nformat:int32\ndescription:callsperhourallowedbytheuser\nschema:\ntype:string\n'400':\ndescription:Invalidusername/passwordsupplied\n"/user/logout":\nget:\ntags:\n-user\nsummary:Logsoutcurrentloggedinusersession\ndescription:''\noperationId:logoutUser\nproduces:\n-application/json\n-application/xml\nparameters:[]\nresponses:\ndefault:\ndescription:successfuloperation\n"/user":\npost:\ntags:\n-user\nsummary:Createuser\ndescription:Thiscanonlybedonebytheloggedinuser.\noperationId:createUser\nconsumes:\n-application/json\nproduces:\n-application/json\n-application/xml\nparameters:\n-in:body\nname:body\ndescription:Createduserobject\nrequired:true\nschema:\n"$ref":"#/definitions/User"\nresponses:\ndefault:\ndescription:successfuloperation\nsecurityDefinitions:\napi_key:\ntype:apiKey\nname:api_key\nin:header\npetstore_auth:\ntype:oauth2\nauthorizationUrl:https://petstore.swagger.io/oauth/authorize\nflow:implicit\nscopes:\nread:pets:readyourpets\nwrite:pets:modifypetsinyouraccount\ndefinitions:\nApiResponse:\ntype:object\nproperties:\ncode:\ntype:integer\nformat:int32\ntype:\ntype:string\nmessage:\ntype:string\nCategory:\ntype:object\nproperties:\nid:\ntype:integer\nformat:int64\nname:\ntype:string\nxml:\nname:Category\nPet:\ntype:object\nrequired:\n-name\n-photoUrls\nproperties:\nid:\ntype:integer\nformat:int64\ncategory:\n"$ref":"#/definitions/Category"\nname:\ntype:string\nexample:doggie\nphotoUrls:\ntype:array\nxml:\nwrapped:true\nitems:\ntype:string\nxml:\nname:photoUrl\ntags:\ntype:array\nxml:\nwrapped:true\nitems:\nxml:\nname:tag\n"$ref":"#/definitions/Tag"\nstatus:\ntype:string\ndescription:petstatusinthestore\nenum:\n-available\n-pending\n-sold\nxml:\nname:Pet\nTag:\ntype:object\nproperties:\nid:\ntype:integer\nformat:int64\nname:\ntype:string\nxml:\nname:Tag\nOrder:\ntype:object\nproperties:\nid:\ntype:integer\nformat:int64\npetId:\ntype:integer\nformat:int64\nquantity:\ntype:integer\nformat:int32\nshipDate:\ntype:string\nformat:date-time\nstatus:\ntype:string\ndescription:OrderStatus\nenum:\n-placed\n-approved\n-delivered\ncomplete:\ntype:boolean\nxml:\nname:Order\nUser:\ntype:object\nproperties:\nid:\ntype:integer\nformat:int64\nusername:\ntype:string\nfirstName:\ntype:string\nlastName:\ntype:string\nemail:\ntype:string\npassword:\ntype:string\nphone:\ntype:string\nuserStatus:\ntype:integer\nformat:int32\ndescription:UserStatus\nxml:\nname:User\nexternalDocs:\ndescription:FindoutmoreaboutSwagger\nurl:http://swagger.io\n\n
But java app throws below error.
{"timestamp":"2022-12-08T20:47:59.420+0000","status":400,"error":"Bad Request","message":"JSON parse error: Unexpected character ('-' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value; nested exception is com.fasterxml.jackson.core.JsonParseException: Unexpected character ('-' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: (PushbackInputStream); line: 1, column: 111]","path":"/api/v1/projects"}
The conversion is working. But needs to be put it in the right format/structure to pass it through the java app.
Will work on it and post here upon success.

Yaml - Swagger, Duplicate mapping key

I've created a simple API, however I am having problems documenting it with Swagger as I get error: Duplicate mapping key on line 200. It is the line which says: "INVALID TOKEN" on route to update a switch: api/switches/update/{id}
I've tried creating also definition specific for update, but it was the same as Create so I think its okay to use Create definition for update as well. Not sure, if this could be a part of the problem.
swagger: "2.0"
info:
version: "1.0.0"
title: Mechanical Switches API
host: localhost:4000
basePath: /
schemes:
- http
- https
consumes:
- application/json
produces:
- application/json
- application/xml
- text/xml
- text/html
paths:
/api/welcome:
get:
description: Sharing the love with welcome message
summary: Welcome message
operationId: getWelcomeMessage
tags:
- user
responses:
'200':
description: "OK"
/api/user/register:
post:
description: Register a new user
summary: Create new user
operationId: postNewUser
tags:
- user
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
parameters:
- in: body
name: User registration
description: The user to create
schema:
$ref: "#/definitions/userCreate"
/api/user/login:
post:
description: Login as user
summary: Login user
operationId: loginExistingUser
tags:
- user
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
parameters:
- in: body
name: User login
description: The user to login
schema:
$ref: "#/definitions/userLogin"
/api/switches/create:
post:
description: Create a new switch
summary: New Switch
operationId: postNewSwitch
tags:
- switches
responses:
'200':
description: "OK"
'400':
description: "ACCESS DENIED"
'401':
description: "INVALID TOKEN"
parameters:
- in: body
name: Create switch
description: Create a new switch
schema:
$ref: "#/definitions/switchCreate"
/api/switches:
get:
description: Get all the switches
summary: All Switches
operationId: getAllSwitches
tags:
- switches
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/{id}:
get:
parameters:
- name: id
in: path
required: true
type: integer
format: int64
description: Get a specific switch
summary: Specific switch
operationId: getSingleSwitch
tags:
- switches
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/type/linear:
get:
description: Get all linear switches
summary: Linear switches
operationId: getLinearSwitches
tags:
- switches
- type linear
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/type/tactile:
get:
description: Get all tactile switches
summary: Tactile switches
operationId: getTactileSwitches
tags:
- switches
- type tactile
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/type/clicky:
get:
description: Get all clicky switches
summary: Clicky switches
operationId: getClickySwitches
tags:
- switches
- type clicky
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/mount/5pin:
get:
description: Get all switches with 5-pin mount
summary: All 5-pin
operationId: getAll5pin
tags:
- switches
- 5pin
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/mount/3pin:
get:
description: Get all switches with 3-pin mount
summary: All 3-pin
operationId: getAll3pin
tags:
- switches
- 3pin
responses:
'200':
description: "OK"
'400':
description: "NOT OK"
/api/switches/update/{id}:
put:
parameters:
- name: id
in: path
required: true
type: integer
format: int64
description: Update a specific switch
summary: Update switch
operationId: updateSwitch
tags:
- switches
- update
responses:
'200':
description: "OK"
'400':
description: "ACCESS DENIED"
'401':
description: "INVALID TOKEN"
parameters:
- in: body
name: Update existing switch
description: Update a switch
schema:
$ref: "#/definitions/switchCreate"
/api/switches/delete/{id}:
delete:
parameters:
- name: id
in: path
required: true
type: integer
format: int64
description: Delete a specific switch
summary: Delete switch
operationId: deleteSwitch
tags:
- switches
- delete
responses:
'200':
description: "OK"
'400':
description: "ACCESS DENIED"
'401':
description: "INVALID TOKEN"
definitions:
userCreate:
properties:
userName:
type: string
email:
type: string
password:
type: string
required:
- userName
- email
- password
userLogin:
properties:
email:
type: string
password:
type: string
required:
- email
- password
switchCreate:
properties:
model:
type: string
brand:
type: string
switchCollection:
type: string
switchType:
type: string
actuationForce:
type: string
preTravel:
type: string
totalTravel:
type: string
stemStructure:
type: string
mount:
type: string
lifespan:
type: string
colors:
type: string
manufacturer:
type: string
required:
- model
- brand
- switchCollection
- switchType
- actuationForce
- preTravel
- totalTravel
- stemStructure
- mount
- lifespan
- colors
- manufacturer
The "duplicate mapping key" error occurs because the /api/switches/update/{id} definition contains two parameters sections:
/api/switches/update/{id}:
put:
parameters: # <------
- name: id
in: path
required: true
type: integer
format: int64
description: Update a specific switch
...
responses:
...
parameters: # <------
- in: body
name: Update existing switch
description: Update a switch
schema:
$ref: "#/definitions/switchCreate"
You need to merge the parameters into a single list:
/api/switches/update/{id}:
put:
parameters:
- name: id
in: path
required: true
type: integer
format: int64
- in: body
name: Update existing switch
description: Update a switch
schema:
$ref: "#/definitions/switchCreate"

How to build multimodule gradle project with each module having its own Open API spec and the parent has all spec merge together on build

I am designing one application, the application should merge OpenAPI-3 specification files into a single file.
Consider the following open api v3 schema files color.yaml and book.yaml
color.yaml
openapi: 3.0.1
info:
title: OpenAPI definition
version: v0
servers:
- url: http://localhost:8080
description: Generated server url
paths:
/api/color/{name}:
get:
tags:
- color-controller
operationId: getColor
parameters:
- name: name
in: path
required: true
schema:
type: string
responses:
"200":
description: OK
content:
'*/*':
schema:
$ref: '#/components/schemas/Color'
components:
schemas:
Color:
type: object
properties:
name:
type: string
red:
type: integer
format: int32
green:
type: integer
format: int32
blue:
type: integer
format: int32
book.yaml
openapi: 3.0.1
info:
title: OpenAPI definition
version: v0
servers:
- url: http://localhost:8080
description: Generated server url
paths:
/api/book/{name}:
get:
tags:
- book-controller
operationId: getBook
parameters:
- name: name
in: path
required: true
schema:
type: string
responses:
"200":
description: OK
content:
'*/*':
schema:
$ref: '#/components/schemas/Book'
components:
schemas:
Book:
type: object
properties:
name:
type: string
iban:
type: string
</code>
The parent module should merge these files into a single master yaml spec file
**merged.yaml**
openapi: 3.0.1
info:
title: My title
version: 1.0.0-SNAPSHOT
servers:
- url: http://localhost:8080
description: Generated server url
paths:
/api/book/{name}:
get:
tags:
- book-controller
operationId: getBook
parameters:
- name: name
in: path
required: true
style: simple
explode: false
schema:
type: string
responses:
"200":
description: OK
content:
'*/*':
schema:
$ref: '#/components/schemas/Book'
/api/color/{name}:
get:
tags:
- color-controller
operationId: getColor
parameters:
- name: name
in: path
required: true
style: simple
explode: false
schema:
type: string
responses:
"200":
description: OK
content:
'*/*':
schema:
$ref: '#/components/schemas/Color'
components:
schemas:
Book:
type: object
properties:
name:
type: string
iban:
type: string
Color:
type: object
properties:
name:
type: string
red:
type: integer
format: int32
green:
type: integer
format: int32
blue:
type: integer
format: int32
Also when I do changes to specific sub-module spec file it should reflect in parent spec file and should show in swagger-ui for testing.
Regarding the merging part of your question:
You can use APIMatic's API spec merge feature to first merge your specs and then transform the merged output into OpenAPI's format. Here are the steps:
Structure your directory as follows:
dir\
spec1\
color.yaml
spec2\
book.yaml
APIMATIC-META.json
A minimalistic APIMATIC-META.json can look like this to enable merging:
{
"MergeConfiguration": {
"MergedApiName": "My title",
"MergeApis": true,
"MergeSettings": {
"SkipCodeGenValidation": true
}
}
}
ZIP the directory, upload it and transform it via their website to OpenAPI v3 to get your merged output. Here is a link that provides step by step guide on uploading and performing a transformation: https://docs.apimatic.io/manage-apis/api-merging/#transforming-the-zipped-file
If you are looking to automate the process, APIMatic has an API too: https://www.apimatic.io/docs/api#/http/api-endpoints/transformation/transform-via-file

How to pass EMR cluster ID in the CloudWatch Alarm

I am trying to create a SNS alarm for my EMR cluster so when EMR cluster is failed i should get notified .
But my issue is i am not able to pass Cluster ID as JobFlowId in the CloudWatch Alarm .
I am create all resources using CloudFomartion Templet .
When i use REF to refer Cluster Id i get below error .
Error
Template validation error: Template format error: Unresolved resource dependencies [FinancialLineItemEmrCluster] in the Resources block of the template
Here is my templet .
I am having issue specially in at JobFlowId while creating the resources EMRAlarm
AWSTemplateFormatVersion: '2010-09-09'
Description: TRF SDI Optmization Full File Creation
Parameters:
AppName:
Default: trfsdioptmization
Description: trfsdioptmization.
Type: String
Environment:
Type: String
Default: nonprod
FinancialIdentifier:
Type: String
Default: 123456789
ApplicationAssetInsightId:
Type: String
Default: 12345678
EnvironmentType:
Type: String
AllowedValues:
- "prod"
- "PRE-PRODUCTION"
- "QUALITY ASSURANCE"
- "nonprod"
Default: nonprod
ResourceOwner:
Type: String
Default: sudarshan.kumar#abcd.com
EnvironmentPhase:
Type: String
Default: nonprod
RegionAbbreviation:
Default: us-east-1
Description: Region Abbreviation e.g. us-east-1 for us-east
Type: String
Resources:
TRFSDIFullfileGeneration:
Type: "AWS::DataPipeline::Pipeline"
#DeletionPolicy: Retain
Properties:
Name: !Sub "${ApplicationAssetInsightId}-tr-fr-${EnvironmentPhase}-${RegionAbbreviation}-${AppName}-DataPipeline"
Description: "Pipeline to create full file for TRFSDI full file Optmization"
Activate: false
PipelineObjects:
-
Id: "FinancialLineItemActivity"
Name: "FinancialLineItemActivity"
Fields:
-
Key: "type"
StringValue: "EmrActivity"
-
Key: "runsOn"
RefValue: "FinancialLineItemEmrCluster"
-
Key: "step"
StringValue: "command-runner.jar,spark-submit,--master,yarn-cluster,--deploy-mode,cluster,--class,start.EntryFileCreation,s3://205147-trf-fr-nonprdo-us-east-1-trfsdioptmization/AJAR/SparkJob-0.1-jar-with-dependencies.jar,FinancialLineItem"
Id: "Default"
Name: "Default"
Fields:
-
Key: "type"
StringValue: "Default"
-
Key: "scheduleType"
StringValue: "ONDEMAND"
-
Key: "failureAndRerunMode"
StringValue: "CASCADE"
-
Key: "role"
StringValue: "DataPipelineDefaultRole"
-
Key: "resourceRole"
StringValue: "DataPipelineDefaultResourceRole"
-
Key: "pipelineLogUri"
StringValue: "s3://205147-tr-fr-nonprod-us-east-1-trfsdioptmization/EMRLOGS"
-
Id: "FinancialLineItemEmrCluster"
Name: "FinancialLineItemEmrCluster"
Fields:
-
Key: "terminateAfter"
StringValue: "30 Minutes"
-
Key: "releaseLabel"
StringValue: "emr-5.9.0"
-
Key: "masterInstanceType"
StringValue: "m3.xlarge"
-
Key: "coreInstanceType"
StringValue: "m3.2xlarge"
-
Key: "coreInstanceCount"
StringValue: "2"
-
Key: "type"
StringValue: "EmrCluster"
-
Key: "applications"
StringValue: "spark"
-
Key: "subnetId"
StringValue: "subnet-86febcab"
-
Key: "onSuccess"
RefValue: "FinancialLineItem_Success"
-
Key: "onFail"
RefValue: "FinancialLineItem_Fail"
EMRAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: "Raise alarm if apps running on EMR cluster is killed"
Namespace: AWS/ElasticMapReduce
MetricName: AppsKilled
Dimensions:
- Name: 205147-TRFSDIOPTmization
JobFlowId: !Ref FinancialLineItemEmrCluster
Statistic: Average
Period: 15
EvaluationPeriods: '1'
ComparisonOperator: GreaterThanOrEqualToThreshold
Threshold: 1
AlarmActions:
- "AWSTRF_HEALTH"
There is no resource of the type AWS::EMR::Cluster in your template.
You are referencing something called FinancialLineItemEmrCluster in your CloudWatch Alarm. From the context, I am assuming you are trying to reference a EMR job. However, since you have no parameter or resource in your template named FinancialLineItemEmrCluster, you can not access is.
EMR cluster IDs generally look like this: j-1ABCD123AB1A. You have several options:
If this cluster is in another template, you could create a CloudFormation export in that stack, and use !ImportValue in your template to import it.
Alternatively, you could use a Parameter in your template and pass the ClusterId that way. Add this as a parameter:
Example:
FinancialLineItemEmrCluster:
Description: 'Your EMR cluster id. Example: j-1ABCD123AB1A'
Type: String
A third alternative is to just hardcode it into your template.
In any case, you can not refer directly to a resource in another stack.
If you have not createt an EMR::Cluster at all, and have no Cluster Id, you need to create one first. You could add one to your Template using the AWS::EMR::Cluster resource:
FinancialLineItemEmrCluster:
Type: AWS::EMR::Cluster
Properties:
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: "m3.xlarge"
Market: "ON_DEMAND"
Name: "Master"
CoreInstanceGroup:
InstanceCount: 2
InstanceType: "m3.xlarge"
Market: "ON_DEMAND"
Name: "Core"
TerminationProtected: true
Name: "TestCluster"
JobFlowRole: "EMR_EC2_DefaultRole"
ServiceRole: "EMR_DefaultRole"
ReleaseLabel: "emr-4.2.0"

Swagger Editor 3.0 How to code component schemas for java maps

I have an response schema that contains java URI, Set and Map.
I can't find any example how to code this yaml part for swagger.
How to write the URI, Set and Map part in yaml components: schemas: see below.
My response java code looks like below.
public class ShowStaticInfo {
private String Id;
private long time;
private URI poster;
private Set<Tag> tags;
private HashMap<String,Tag> objectCreated;
And the written yaml code like this.
As stated above this yaml needs to be changed.
openapi: "3.0.1"
info:
title: Mobile backend
version: 1.0.0
license:
name: Apache 2.0
servers:
- url: https://development.cybercom.com/services/6
description: Development server
- url: https://staging.cybercom.com/services
description: Staging server
- url: https://production.cybercom.com/services
description: Production server
paths:
/buildinfo:
get:
description: Returns the build information (Version and Time stamp).
operationId: getBuildInfo
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/BuildInfo'
parameters:
- in: header
name: LBPATH
schema:
type: string
/countries/{countryId}/cities/{cityId}/showinfo/static:
get:
description: Returns a list of static show information for a city.
operationId: getShowStaticInfo
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/ShowStaticInfo'
parameters:
- in: path
name: countryId
required: true
schema:
type: string
- in: path
name: cityId
required: true
schema:
type: string
- in: header
name: Accept-Language
schema:
type: string
- in: header
name: LBPATH
schema:
type: string
components:
schemas:
BuildInfo:
properties:
version:
type: string
timestamp:
type: string
status:
type: string
ShowStaticInfo:
properties:
poster:
$ref: '#/components/schemas/URI'
time:
type: integer
format: int64
showId:
type: string
tags:
$ref: '#/components/schemas/Set'
showObjectCreated:
$ref: '#/components/schemas/HashMap'

Resources