switch from WebSocket to RawSocket Transport - autobahn

I am new to crossbar/autobahn framework and to network programming in general, recently at work we would like to experiment to move from WebSocket transport to RawSocket Transport with using MessagePackSerializer and do performance comparison. I am requesting to the framework developers to please guide me to convert following config to use raw socket. will using auth settings from ws section should work ?
{
"version": 2,
"workers": [
{
"type": "router",
"options": {
"pythonpath": ["../components"]
},
"realms": [
{
"name": "realm1",
"roles": [
{
"name": "system",
"authorizer": "com.XXXX.auth.authorize"
},
{
"name": "user",
"authorizer": "com.XXXX.auth.authorize"
},
{
"name": "service",
"authorizer": "com.XXXX.auth.authorize"
}
]
}
],
"transports": [
{
"type": "rawsocket",
"endpoint": {
"type": "unix",
"path": "/tmp/crossbar-socket"
}
},
{
"type": "web",
"endpoint": {
"type": "tcp",
"port": 8080
},
"paths": {
"status": {
"type": "resource",
"classname": "health_web_indicator.run",
"extra": {
"DVMT_VERSION": "2.0.1",
"HEALTHY_INDICATOR": ":) DVMT UP",
"UNHEALTHY_INDICATOR": ":( DVMT DOWN",
"MSSQL_DRIVER": "FreeTDS",
"MSSQL_SERVER": "XXXX.dv.local",
"MSSQL_PORT": "1515",
"MSSQL_DATABASE": "XXXX",
"MSSQL_USER": "XXXX",
"MSSQL_PASSWORD": "XXXX",
"SUPERVISOR_PROCESSES": ["XXXX", "XXXX", "XXXX", "XXXX", "XXXX"],
"SUPERVISOR_SERVER_SOCKET": "/crossbar/supervisord.sock"
}
},
"ws": {
"type": "websocket",
"auth": {
"anonymous": {
"type": "dynamic",
"authenticator": "com.XXXX.auth.authenticate"
},
"ticket": {
"type": "dynamic",
"authenticator": "com.XXXX.auth.authenticate"
}
},
"debug": true,
"options": {
"auto_ping_interval": 0,
"auto_ping_timeout": 60000,
"auto_ping_size": 4,
"compression": {
"deflate": {
"request_no_context_takeover": false,
"request_max_window_bits": 11,
"no_context_takeover": false,
"max_window_bits": 11,
"memory_level": 4
}
}
}
}
}
}
],
"components": [
{
"id": "auth",
"type": "class",
"classname": "auth.AuthComponent",
"realm": "realm1",
"role": "trusted",
"extra": {
"prefix": "com.XXXX.auth",
"system_role": "system",
"anonymous_role": "anonymous",
"mssql": {
"driver": "FreeTDS",
"server": "XXXX.dv.local",
"port": "1515",
"database": "XXXX",
"uid": "XXXX",
"pwd": "XXXX",
"autocommit": true
},
"heartbeat_rate": 30.0,
"ldap_api_url": "http://XXXX:80/rssapi/dvmsapi/admin/GetUserLdapGroups/{username}",
"enforce_cert_match": false,
"cert_sub_pattern": "[\\s_-]"
}
}
]
}
]
}
also will it be easy enough for you to please give me an example for using autobahnpython as a client for rawsocket transport. https://crossbar.io/docs/RawSocket-Transport/#example---connecting-transport

Related

Creating/Manipulating Elasticsearch Data View from filebeat.yml

I created some kibana dashboards and they are referencing to a kibana data view. This data view is currently default generated by Filebeat when i'm uploading the filebeat setup, etc. Therefore the data view gets a random Id and my dashboard gets a mising reference since those id's are changing. Now i want to configure somewhere the default created data view since my index- and template-pattern have fixed names and i could create a fixed data view. Then my Dashboards wouldn't go into "missing references". I searched the following sites:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html
https://github.com/elastic/kibana/issues
I created the Dashboards within the kibana UI and exported them for easier integration within my filebeat project.
My filebeat.yml file
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
setup.template.name: pac-filebeat-%{[agent.version]}
setup.template.pattern: pac-filebeat-%{[agent.version]}
setup.template.fields: ${path.config}/fields.yml
setup.dashboards.enabled: false
setup.dashboards.directory: ${path.config}\kibana\custom
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "localhost:5601"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "http"
index: pac-filebeat-%{[agent.version]}
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
This is one example Dashboard.
{
"attributes": {
"description": "",
"hits": 0,
"kibanaSavedObjectMeta": {
"searchSourceJSON": {
"query": {
"query": "",
"language": "kuery"
},
"filter": []
}
},
"optionsJSON": {
"useMargins": true,
"syncColors": false,
"syncTooltips": false,
"hidePanelTitles": false
},
"panelsJSON": [
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 0,
"y": 0,
"w": 9,
"h": 10,
"i": "70779a3d-7880-416d-9736-8a374af8ae41"
},
"panelIndex": "70779a3d-7880-416d-9736-8a374af8ae41",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsLegacyMetric",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-2a27a515-43d0-4aae-8764-467983de9faf"
},
{
"type": "index-pattern",
"name": "99c2853a-48e3-4042-badf-9c0829cec119",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"
}
],
"state": {
"visualization": {
"layerId": "2a27a515-43d0-4aae-8764-467983de9faf",
"accessor": "10b01daf-1c3d-46d1-a014-b381ac70019c",
"layerType": "data"
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [
{
"meta": {
"index": "99c2853a-48e3-4042-badf-9c0829cec119",
"alias": "pac.log.level : ERROR",
"type": "custom",
"key": "query",
"value": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"should": [
{
"match": {
"pac.log.level": "ERROR"
}
}
],
"minimum_should_match": 1
}
}
],
"should": [],
"must_not": []
}
},
"disabled": false,
"negate": false
},
"query": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"should": [
{
"match": {
"pac.log.level": "ERROR"
}
}
],
"minimum_should_match": 1
}
}
],
"should": [],
"must_not": []
}
},
"$state": {
"store": "appState"
}
}
],
"datasourceStates": {
"indexpattern": {
"layers": {
"2a27a515-43d0-4aae-8764-467983de9faf": {
"columns": {
"10b01daf-1c3d-46d1-a014-b381ac70019c": {
"label": "Errors",
"customLabel": true,
"dataType": "number",
"operationType": "count",
"isBucketed": false,
"scale": "ratio",
"sourceField": "pac.log.level",
"filter": {
"query": "pac.log.level : ERROR",
"language": "kuery"
},
"params": {
"emptyAsNull": true
}
}
},
"columnOrder": [
"10b01daf-1c3d-46d1-a014-b381ac70019c"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
},
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 9,
"y": 0,
"w": 12,
"h": 10,
"i": "2bbc2f60-dfb2-4545-a725-9f9bec323e4b"
},
"panelIndex": "2bbc2f60-dfb2-4545-a725-9f9bec323e4b",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsXY",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-a0f69cd9-6726-4551-9a91-cb330cf0b16d"
}
],
"state": {
"visualization": {
"legend": {
"isVisible": true,
"position": "right"
},
"valueLabels": "hide",
"fittingFunction": "None",
"xTitle": "",
"yTitle": "",
"axisTitlesVisibilitySettings": {
"x": false,
"yLeft": false,
"yRight": true
},
"tickLabelsVisibilitySettings": {
"x": true,
"yLeft": false,
"yRight": true
},
"labelsOrientation": {
"x": 0,
"yLeft": 0,
"yRight": 0
},
"gridlinesVisibilitySettings": {
"x": true,
"yLeft": false,
"yRight": true
},
"preferredSeriesType": "bar_horizontal_percentage_stacked",
"layers": [
{
"layerId": "a0f69cd9-6726-4551-9a91-cb330cf0b16d",
"seriesType": "bar_horizontal_percentage_stacked",
"accessors": [
"cbf5323b-0d81-4411-afa3-91087f062874"
],
"yConfig": [],
"layerType": "data",
"splitAccessor": "a98ba577-d539-4ab3-b798-2bddcb2efd3f",
"palette": {
"type": "palette",
"name": "status"
}
}
]
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [],
"datasourceStates": {
"indexpattern": {
"layers": {
"a0f69cd9-6726-4551-9a91-cb330cf0b16d": {
"columns": {
"a98ba577-d539-4ab3-b798-2bddcb2efd3f": {
"label": "ESA states",
"dataType": "string",
"operationType": "terms",
"scale": "ordinal",
"sourceField": "pac.log.esa.connection",
"isBucketed": true,
"params": {
"size": 3,
"orderBy": {
"type": "column",
"columnId": "cbf5323b-0d81-4411-afa3-91087f062874"
},
"orderDirection": "desc",
"otherBucket": true,
"missingBucket": false,
"parentFormat": {
"id": "terms"
}
},
"customLabel": true
},
"cbf5323b-0d81-4411-afa3-91087f062874": {
"label": "Connected",
"dataType": "number",
"operationType": "unique_count",
"scale": "ratio",
"sourceField": "pac.log.esa.connection",
"isBucketed": false,
"filter": {
"query": "pac.log.esa.connection : Connected",
"language": "kuery"
},
"params": {
"emptyAsNull": true
},
"customLabel": true
}
},
"columnOrder": [
"a98ba577-d539-4ab3-b798-2bddcb2efd3f",
"cbf5323b-0d81-4411-afa3-91087f062874"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
},
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 21,
"y": 0,
"w": 7,
"h": 10,
"i": "ef851b6b-222a-4b70-a7c3-ec6a85bdf4d1"
},
"panelIndex": "ef851b6b-222a-4b70-a7c3-ec6a85bdf4d1",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsLegacyMetric",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-d8719702-b241-4931-8bd3-d706f1403c25"
}
],
"state": {
"visualization": {
"layerId": "d8719702-b241-4931-8bd3-d706f1403c25",
"accessor": "312ded46-d0e9-4dcf-9779-b0ca10249a90",
"layerType": "data"
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [],
"datasourceStates": {
"indexpattern": {
"layers": {
"d8719702-b241-4931-8bd3-d706f1403c25": {
"columns": {
"312ded46-d0e9-4dcf-9779-b0ca10249a90": {
"label": "Failed DEO Executions",
"customLabel": true,
"dataType": "number",
"operationType": "count",
"isBucketed": false,
"scale": "ratio",
"sourceField": "pac.log.deo.status",
"filter": {
"query": "pac.log.deo.status : failed",
"language": "kuery"
},
"params": {
"emptyAsNull": true
}
}
},
"columnOrder": [
"312ded46-d0e9-4dcf-9779-b0ca10249a90"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
},
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 0,
"y": 10,
"w": 14,
"h": 11,
"i": "7bcd2a3e-54c1-4b09-a0b6-d41c339c2ae6"
},
"panelIndex": "7bcd2a3e-54c1-4b09-a0b6-d41c339c2ae6",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsXY",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-cd91feb0-0927-4465-b0fd-b5bac50af734"
}
],
"state": {
"visualization": {
"title": "Empty XY chart",
"legend": {
"isVisible": true,
"position": "right",
"showSingleSeries": true
},
"valueLabels": "hide",
"preferredSeriesType": "line",
"layers": [
{
"layerId": "cd91feb0-0927-4465-b0fd-b5bac50af734",
"accessors": [
"8ccc17fc-529e-4460-9925-149e77e37dd1"
],
"position": "top",
"seriesType": "line",
"showGridlines": false,
"layerType": "data",
"xAccessor": "43aa69d1-f806-4f87-a2b8-57718e85400a"
}
],
"yTitle": "",
"axisTitlesVisibilitySettings": {
"x": true,
"yLeft": false,
"yRight": true
}
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [],
"datasourceStates": {
"indexpattern": {
"layers": {
"cd91feb0-0927-4465-b0fd-b5bac50af734": {
"columns": {
"43aa69d1-f806-4f87-a2b8-57718e85400a": {
"label": "#timestamp",
"dataType": "date",
"operationType": "date_histogram",
"sourceField": "#timestamp",
"isBucketed": true,
"scale": "interval",
"params": {
"interval": "auto",
"includeEmptyRows": true,
"dropPartials": false
}
},
"8ccc17fc-529e-4460-9925-149e77e37dd1": {
"label": " System cpu usage",
"dataType": "number",
"operationType": "median",
"sourceField": "pac.log.system.monitor.cpu.pct",
"isBucketed": false,
"scale": "ratio",
"params": {
"emptyAsNull": true,
"format": {
"id": "percent",
"params": {
"decimals": 2
}
}
},
"customLabel": true
}
},
"columnOrder": [
"43aa69d1-f806-4f87-a2b8-57718e85400a",
"8ccc17fc-529e-4460-9925-149e77e37dd1"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
}
],
"timeRestore": false,
"title": "ESA overview",
"version": 1
},
"coreMigrationVersion": "8.5.3",
"id": "d29c4ac0-ab68-11ed-bfcf-939f8111002e",
"migrationVersion": {
"dashboard": "8.5.0"
},
"references": [
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "70779a3d-7880-416d-9736-8a374af8ae41:indexpattern-datasource-layer-2a27a515-43d0-4aae-8764-467983de9faf",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "70779a3d-7880-416d-9736-8a374af8ae41:99c2853a-48e3-4042-badf-9c0829cec119",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "2bbc2f60-dfb2-4545-a725-9f9bec323e4b:indexpattern-datasource-layer-a0f69cd9-6726-4551-9a91-cb330cf0b16d",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "ef851b6b-222a-4b70-a7c3-ec6a85bdf4d1:indexpattern-datasource-layer-d8719702-b241-4931-8bd3-d706f1403c25",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "7bcd2a3e-54c1-4b09-a0b6-d41c339c2ae6:indexpattern-datasource-layer-cd91feb0-0927-4465-b0fd-b5bac50af734",
"type": "index-pattern"
},
{
"id": "c6a11f70-ab68-11ed-bfcf-939f8111002e",
"name": "tag-c6a11f70-ab68-11ed-bfcf-939f8111002e",
"type": "tag"
}
],
"type": "dashboard",
"updated_at": "2023-02-13T07:14:45.551Z",
"version": "WzE1NTI1NiwzMl0="
}
This would be the used tag inside the above dashboard
{
"attributes": {
"color": "#52bc22",
"description": "",
"name": "custom"
},
"coreMigrationVersion": "8.5.3",
"id": "c6a11f70-ab68-11ed-bfcf-939f8111002e",
"migrationVersion": {
"tag": "8.0.0"
},
"references": [],
"type": "tag",
"updated_at": "2023-02-13T07:39:07.543Z",
"version": "WzE1NTM0NywzMl0="
}
The error message:
error dashboard asset: returned 200 to import file: 1 error: error: missing_references, asset ID=d29c4ac0-ab68-11ed-bfcf-939f8111002e; asset type=dashboard; references=[{Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d}]. Response: {"successCount":1,"success":false,"warnings":[],"successResults":[{"type":"tag","id":"c6a11f70-ab68-11ed-bfcf-939f8111002e","meta":{"title":"custom","icon":"tag"},"overwrite":true}],"errors":[{"id":"d29c4ac0-ab68-11ed-bfcf-939f8111002e","type":"dashboard","meta":{"title":"ESA overview","icon":"dashboardApp"},"error":{"type":"missing_references","references":[{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"}]},"overwrite":true}]}
kind regardes
You can configure Filebeat to use your own dashboard using the following settings:
setup.dashboards.url
setup.dashboards.file
setup.dashboards.index
setup.dashboards.string_replacements (<-- probably the one you're looking for)

Logstash Indexing error “Index -1 out of bounds for length 0”

I want to use ELK stack to analyze some Kubernetes audit logs. They're sent to the Logstash webhook as JSON. Here's my config file :
input{
http {
port => 8888
codec => "json"
type => "json"
}
}
filter{
json {
source => "message"
}
}
output{
file {
path => "/home/ubuntu/logstash-kubernetes/audit.log"
}
elasticsearch{
hosts => "localhost:9200"
index => "kubernetes"
}
}
The output file is populated with good looking entries, I'm happy with it. But when it comes to exporting to Elasticsearch, I'm getting the following error :
elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"kubernetes", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x71740f4b>], :response=>{"index"=>{"_index"=>"kubernetes", "_type"=>"_doc", "_id"=>"x6QKJHYBkIKyNOo2Q-z8", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"array_index_out_of_bounds_exception", "reason"=>"Index -1 out of bounds for length 0"}}}}}
It only happens for some entries. Let me show you an generated entry that was processed without error :
{
"_index": "kubernetes",
"_type": "_doc",
"_id": "xKQKJHYBkIKyNOo2POwJ",
"_version": 1,
"_score": 0,
"_source": {
"headers": {
"content_type": "application/json",
"request_method": "POST",
"accept_encoding": "gzip",
"request_path": "/",
"http_host": "****:8888",
"http_accept": "application/json, */*",
"content_length": "2813",
"http_version": "HTTP/1.1",
"http_user_agent": "Go-http-client/1.1"
},
"#version": "1",
"host": "****",
"apiVersion": "audit.k8s.io/v1",
"items": [
{
"requestObject": {
"type": "Normal",
"firstTimestamp": "2020-12-02T15:20:19Z",
"involvedObject": {
"namespace": "kube-system",
"name": "kube-scheduler",
"apiVersion": "v1",
"resourceVersion": "2573",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d",
"kind": "Endpoints"
},
"message": "****1_938d6a87-bccb-4ea0-bd6b-31fc27e48b7a became leader",
"eventTime": null,
"source": {
"component": "default-scheduler"
},
"reason": "LeaderElection",
"count": 1,
"reportingInstance": "",
"apiVersion": "v1",
"reportingComponent": "",
"lastTimestamp": "2020-12-02T15:20:19Z",
"metadata": {
"name": "kube-scheduler.164cef6d1f3ed55e",
"namespace": "kube-system",
"creationTimestamp": null
},
"kind": "Event"
},
"requestURI": "/api/v1/namespaces/kube-system/events",
"responseStatus": {
"code": 201,
"metadata": {}
},
"user": {
"groups": [
"system:authenticated"
],
"username": "system:kube-scheduler"
},
"annotations": {
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\"",
"authorization.k8s.io/decision": "allow"
},
"stageTimestamp": "2020-12-02T15:20:19.090835Z",
"userAgent": "kube-scheduler/v1.19.4 (linux/amd64) kubernetes/d360454/scheduler",
"objectRef": {
"namespace": "kube-system",
"name": "kube-scheduler.164cef6d1f3ed55e",
"apiVersion": "v1",
"resource": "events"
},
"responseObject": {
"type": "Normal",
"firstTimestamp": "2020-12-02T15:20:19Z",
"involvedObject": {
"namespace": "kube-system",
"name": "kube-scheduler",
"apiVersion": "v1",
"resourceVersion": "2573",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d",
"kind": "Endpoints"
},
"message": "****1_938d6a87-bccb-4ea0-bd6b-31fc27e48b7a became leader",
"eventTime": null,
"source": {
"component": "default-scheduler"
},
"reason": "LeaderElection",
"count": 1,
"reportingInstance": "",
"apiVersion": "v1",
"reportingComponent": "",
"lastTimestamp": "2020-12-02T15:20:19Z",
"metadata": {
"name": "kube-scheduler.164cef6d1f3ed55e",
"namespace": "kube-system",
"creationTimestamp": "2020-12-02T15:20:19Z",
"managedFields": [
{
"operation": "Update",
"fieldsType": "FieldsV1",
"time": "2020-12-02T15:20:19Z",
"manager": "kube-scheduler",
"apiVersion": "v1",
"fieldsV1": {
"f:source": {
"f:component": {}
},
"f:lastTimestamp": {},
"f:reason": {},
"f:firstTimestamp": {},
"f:type": {},
"f:involvedObject": {
"f:kind": {},
"f:namespace": {},
"f:name": {},
"f:apiVersion": {},
"f:uid": {},
"f:resourceVersion": {}
},
"f:message": {},
"f:count": {}
}
}
],
"resourceVersion": "2576",
"selfLink": "/api/v1/namespaces/kube-system/events/kube-scheduler.164cef6d1f3ed55e",
"uid": "573e4082-7c38-473c-9116-a114be97a47d"
},
"kind": "Event"
},
"level": "RequestResponse",
"requestReceivedTimestamp": "2020-12-02T15:20:19.021481Z",
"auditID": "cf291b3f-f5fa-4e43-a177-adf9cbfc0197",
"sourceIPs": [
"****"
],
"stage": "ResponseComplete",
"verb": "create"
}
],
"#timestamp": "2020-12-02T15:20:19.094Z",
"type": "json",
"metadata": {},
"kind": "EventList"
}
}
And here's the JSON for that error happens :
{
"headers": {
"content_type": "application/json",
"request_method": "POST",
"accept_encoding": "gzip",
"request_path": "/",
"http_host": "****:8888",
"http_accept": "application/json, */*",
"content_length": "2668",
"http_version": "HTTP/1.1",
"http_user_agent": "Go-http-client/1.1"
},
"#version": "1",
"host": "****",
"apiVersion": "audit.k8s.io/v1",
"items": [
{
"requestObject": {
"metadata": {
"name": "kube-scheduler",
"namespace": "kube-system",
"creationTimestamp": "2020-12-02T14:36:33Z",
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"****38d6a87-bccb-4ea0-bd6b-31fc27e48b7a\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-02T15:18:52Z\",\"renewTime\":\"2020-12-02T15:18:52Z\",\"leaderTransitions\":67}"
},
"managedFields": [
{
"operation": "Update",
"fieldsType": "FieldsV1",
"time": "2020-12-02T15:18:23Z",
"manager": "kube-scheduler",
"apiVersion": "v1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:control-plane.alpha.kubernetes.io/leader": {}
}
}
}
}
],
"resourceVersion": "2531",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/kube-scheduler",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d"
},
"apiVersion": "v1",
"kind": "Endpoints"
},
"requestURI": "/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s",
"responseStatus": {
"code": 200,
"metadata": {}
},
"user": {
"groups": [
"system:authenticated"
],
"username": "system:kube-scheduler"
},
"annotations": {
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\"",
"authorization.k8s.io/decision": "allow"
},
"stageTimestamp": "2020-12-02T15:18:52.067347Z",
"userAgent": "kube-scheduler/v1.19.4 (linux/amd64) kubernetes/d360454/leader-election",
"objectRef": {
"namespace": "kube-system",
"name": "kube-scheduler",
"apiVersion": "v1",
"resourceVersion": "2531",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d",
"resource": "endpoints"
},
"responseObject": {
"metadata": {
"name": "kube-scheduler",
"namespace": "kube-system",
"creationTimestamp": "2020-12-02T14:36:33Z",
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"****1_938d6a87-bccb-4ea0-bd6b-31fc27e48b7a\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-02T15:18:52Z\",\"renewTime\":\"2020-12-02T15:18:52Z\",\"leaderTransitions\":67}"
},
"managedFields": [
{
"operation": "Update",
"fieldsType": "FieldsV1",
"time": "2020-12-02T15:18:52Z",
"manager": "kube-scheduler",
"apiVersion": "v1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:control-plane.alpha.kubernetes.io/leader": {}
}
}
}
}
],
"resourceVersion": "2542",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/kube-scheduler",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d"
},
"apiVersion": "v1",
"kind": "Endpoints"
},
"level": "RequestResponse",
"requestReceivedTimestamp": "2020-12-02T15:18:52.062449Z",
"auditID": "a45860c7-e0c9-4724-be2e-705a1b2955f4",
"sourceIPs": [
"****"
],
"stage": "ResponseComplete",
"verb": "update"
}
],
"#timestamp": "2020-12-02T15:20:22.001Z",
"type": "json",
"metadata": {},
"kind": "EventList"
}
(I removed some personnal data from the JSONs)
Both JSONs appears as valid. I noticed that the JSONs with responseStatus.code = 201 were correctly processed, and those with code = 200 were throwing this error.
I'm getting a bit mad so if anyone can help with that, I'd be very pleased !
The error is thrown on this line because subfields has a length of 0 and hence it is trying to access index -1, which is not allowed.
The method that splits the field names into pathsis called splitAndValidatePath and what it does is to split field names when it encounters . (i.e. a dot).
In your second document, you have two fields called "." (i.e. it is just a dot) and they are not really valid.
".": {},
If you remove them both, your document will be indexed correctly.
Another way of solving this is to disable indexing for the parent f:annotations object field and that will do the trick... provided, of course, you don't need to search for content within f:annotations.
I face same problem with fluentd.
As per #val suggestion. Fixed by below.
PUT /[index_name]/_mapping
{
"properties" : {
"metadata" : {
"properties" : {
"managedFields" : {
"properties" : {
"fieldsV1" : {
"properties" : {
"f:metadata" : {
"type" : "object",
"enabled": false
}
}
}
}
}
}
}
}
}

Is it possible to assign public static IPs to individual VMs in Azure VMSS?

There is a document describing how to allocate a public IP per VM in VMSS: https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-networking#public-ipv4-per-virtual-machine
But it is not clear how to assign public static IP per VM. Is it possible?
Unfortunately, Azure does not provide control of the Public IP allocation method per instance in VMSS. You can see the all supported Properties of
VirtualMachineScaleSetPublicIPAddressConfigurationProperties object in the latest ARM API version.
However, after my validation, when you restart the instance or VMSS scale-in or scale-out, the public IP address of existing instances is not changed. The public IP address of instances will be updated unless you stop the instance of VMSS.
Update
Currently, you can manage it with IpPublicPrefix. Note that IpPublicPrefix requires a standard SKU load balancer and public IP address. Here is a working sample. You can check the public IP address of the instances in VMSS with the REST API.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vmSku": {
"type": "string",
"defaultValue": "Standard_A1_v2",
"metadata": {
"description": "Size of VMs in the VM Scale Set."
}
},
"windowsOSVersion": {
"type": "string",
"defaultValue": "2019-Datacenter",
"allowedValues": [
"2008-R2-SP1",
"2012-Datacenter",
"2012-R2-Datacenter",
"2016-Datacenter",
"2019-Datacenter"
],
"metadata": {
"description": "The Windows version for the VM. This will pick a fully patched image of this given Windows version. Allowed values: 2008-R2-SP1, 2012-Datacenter, 2012-R2-Datacenter & 2016-Datacenter, 2019-Datacenter."
}
},
"vmssName": {
"type": "string",
"minLength": 3,
"maxLength": 61,
"metadata": {
"description": "String used as a base for naming resources. Must be 3-61 characters in length and globally unique across Azure. A hash is prepended to this string for some resources, and resource-specific information is appended."
}
},
"instanceCount": {
"type": "int",
"defaultValue": 3,
"minValue": 1,
"maxValue": 100,
"metadata": {
"description": "Number of VM instances (100 or less)."
}
},
"singlePlacementGroup": {
"type": "bool",
"defaultValue": true,
"metadata": {
"description": "When true this limits the scale set to a single placement group, of max size 100 virtual machines. NOTE: If singlePlacementGroup is true, it may be modified to false. However, if singlePlacementGroup is false, it may not be modified to true."
}
},
"adminUsername": {
"type": "string",
"defaultValue": "vmssadmin",
"metadata": {
"description": "Admin username on all VMs."
}
},
"adminPassword": {
"type": "securestring",
"metadata": {
"description": "Admin password on all VMs."
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
},
"platformFaultDomainCount": {
"type": "int",
"defaultValue": 1,
"metadata": {
"description": "Fault Domain count for each placement group."
}
},
"publicIPPrefixes_pubprefix_name": {
"defaultValue": "vmsspublicprefix",
"type": "string"
}
},
"variables": {
"namingInfix": "[toLower(substring(concat(parameters('vmssName'), uniqueString(resourceGroup().id)), 0, 9))]",
"longNamingInfix": "[toLower(parameters('vmssName'))]",
"addressPrefix": "10.0.0.0/16",
"subnetPrefix": "10.0.0.0/24",
"virtualNetworkName": "[concat(variables('namingInfix'), 'vnet')]",
"publicIPAddressName": "[concat(variables('namingInfix'), 'pip')]",
"subnetName": "[concat(variables('namingInfix'), 'subnet')]",
"loadBalancerName": "[concat(variables('namingInfix'), 'lb')]",
"publicIPAddressID": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]",
"lbProbeID": "[resourceId('Microsoft.Network/loadBalancers/probes',variables('loadBalancerName'), 'tcpProbe')]",
"natPoolName": "[concat(variables('namingInfix'), 'natpool')]",
"bePoolName": "[concat(variables('namingInfix'), 'bepool')]",
"lbPoolID": "[resourceId('Microsoft.Network/loadBalancers/backendAddressPools',variables('loadBalancerName'),variables('bePoolName'))]",
"natStartPort": 50000,
"natEndPort": 50119,
"natBackendPort": 3389,
"nicName": "[concat(variables('namingInfix'), 'nic')]",
"ipConfigName": "[concat(variables('namingInfix'), 'ipconfig')]",
"frontEndIPConfigID": "[resourceId('Microsoft.Network/loadBalancers/frontendIPConfigurations',variables('loadBalancerName'),'loadBalancerFrontEnd')]",
"osType": {
"publisher": "MicrosoftWindowsServer",
"offer": "WindowsServer",
"sku": "[parameters('windowsOSVersion')]",
"version": "latest"
},
"imageReference": "[variables('osType')]"
},
"resources": [
{
"type": "Microsoft.Network/loadBalancers",
"apiVersion": "2020-06-01",
"name": "[variables('loadBalancerName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/publicIPAddresses', variables('publicIPAddressName'))]"
],
"sku": {
"name": "Standard"
},
"properties": {
"frontendIPConfigurations": [
{
"name": "LoadBalancerFrontEnd",
"properties": {
"publicIPAddress": {
"id": "[variables('publicIPAddressID')]",
"name": "Standard"
}
}
}
],
"backendAddressPools": [
{
"name": "[variables('bePoolName')]"
}
],
"inboundNatPools": [
{
"name": "[variables('natPoolName')]",
"properties": {
"frontendIPConfiguration": {
"id": "[variables('frontEndIPConfigID')]"
},
"protocol": "Tcp",
"frontendPortRangeStart": "[variables('natStartPort')]",
"frontendPortRangeEnd": "[variables('natEndPort')]",
"backendPort": "[variables('natBackendPort')]"
}
}
],
"loadBalancingRules": [
{
"name": "LBRule",
"properties": {
"frontendIPConfiguration": {
"id": "[variables('frontEndIPConfigID')]"
},
"backendAddressPool": {
"id": "[variables('lbPoolID')]"
},
"protocol": "Tcp",
"frontendPort": 80,
"backendPort": 80,
"enableFloatingIP": false,
"idleTimeoutInMinutes": 5,
"probe": {
"id": "[variables('lbProbeID')]"
}
}
}
],
"probes": [
{
"name": "tcpProbe",
"properties": {
"protocol": "Tcp",
"port": 80,
"intervalInSeconds": 5,
"numberOfProbes": 2
}
}
]
}
},
{
"type": "Microsoft.Network/publicIPPrefixes",
"apiVersion": "2020-11-01",
"name": "[parameters('publicIPPrefixes_pubprefix_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard",
"tier": "Regional"
},
"properties": {
"prefixLength": 28,
"publicIPAddressVersion": "IPv4",
"ipTags": []
}
},
{
"type": "Microsoft.Compute/virtualMachineScaleSets",
"apiVersion": "2020-06-01",
"name": "[variables('namingInfix')]",
"location": "[parameters('location')]",
"sku": {
"name": "[parameters('vmSku')]",
"tier": "Standard",
"capacity": "[parameters('instanceCount')]"
},
"dependsOn": [
"[resourceId('Microsoft.Network/loadBalancers', variables('loadBalancerName'))]",
"[resourceId('Microsoft.Network/virtualNetworks', variables('virtualNetworkName'))]",
"[resourceId('Microsoft.Network/publicIPPrefixes',parameters('publicIPPrefixes_pubprefix_name'))]"
],
"properties": {
"overprovision": true,
"upgradePolicy": {
"mode": "Automatic"
},
"singlePlacementGroup": "[parameters('singlePlacementGroup')]",
"platformFaultDomainCount": "[parameters('platformFaultDomainCount')]",
"virtualMachineProfile": {
"storageProfile": {
"osDisk": {
"caching": "ReadWrite",
"createOption": "FromImage"
},
"imageReference": "[variables('imageReference')]"
},
"osProfile": {
"computerNamePrefix": "[variables('namingInfix')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPassword')]"
},
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "[variables('nicName')]",
"properties": {
"primary": true,
"ipConfigurations": [
{
"name": "[variables('ipConfigName')]",
"properties": {
"subnet": {
"id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('virtualNetworkName'), variables('subnetName'))]"
},
"loadBalancerBackendAddressPools": [
{
"id": "[variables('lbPoolID')]"
}
],
"loadBalancerInboundNatPools": [
{
"id": "[resourceId('Microsoft.Network/loadBalancers/inboundNatPools', variables('loadBalancerName'), variables('natPoolName'))]"
}
],
"publicipaddressconfiguration": {
"name": "pub1",
"properties": {
"idleTimeoutInMinutes": 15,
"publicIPAddressVersion": "IPv4",
"publicIPPrefix":{
"id": "[resourceId('Microsoft.Network/publicIPPrefixes',parameters('publicIPPrefixes_pubprefix_name'))]"
}
}
}
}
}
]
}
}
]
}
}
}
},
{
"type": "Microsoft.Network/publicIPAddresses",
"apiVersion": "2020-06-01",
"name": "[variables('publicIPAddressName')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard"
},
"properties": {
"publicIPAllocationMethod": "Static",
"dnsSettings": {
"domainNameLabel": "[variables('longNamingInfix')]"
}
}
},
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2020-06-01",
"name": "[variables('virtualNetworkName')]",
"location": "[parameters('location')]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('addressPrefix')]"
]
},
"subnets": [
{
"name": "[variables('subnetName')]",
"properties": {
"addressPrefix": "[variables('subnetPrefix')]"
}
}
]
}
},
{
"type": "Microsoft.Insights/autoscaleSettings",
"apiVersion": "2015-04-01",
"name": "autoscalehost",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachineScaleSets/', variables('namingInfix'))]"
],
"properties": {
"name": "autoscalehost",
"targetResourceUri": "[resourceId('Microsoft.Compute/virtualMachineScaleSets', variables('namingInfix'))]",
"enabled": true,
"profiles": [
{
"name": "Profile1",
"capacity": {
"minimum": "1",
"maximum": "10",
"default": "1"
},
"rules": [
{
"metricTrigger": {
"metricName": "Percentage CPU",
"metricResourceUri": "[resourceId('Microsoft.Compute/virtualMachineScaleSets', variables('namingInfix'))]",
"timeGrain": "PT1M",
"statistic": "Average",
"timeWindow": "PT5M",
"timeAggregation": "Average",
"operator": "GreaterThan",
"threshold": 50
},
"scaleAction": {
"direction": "Increase",
"type": "ChangeCount",
"value": "1",
"cooldown": "PT5M"
}
},
{
"metricTrigger": {
"metricName": "Percentage CPU",
"metricResourceUri": "[resourceId('Microsoft.Compute/virtualMachineScaleSets', variables('namingInfix'))]",
"timeGrain": "PT1M",
"statistic": "Average",
"timeWindow": "PT5M",
"timeAggregation": "Average",
"operator": "LessThan",
"threshold": 30
},
"scaleAction": {
"direction": "Decrease",
"type": "ChangeCount",
"value": "1",
"cooldown": "PT5M"
}
}
]
}
]
}
}
]
}

ECS Service stuck on creation

Trying to deploy a CloudFormation template gets stuck with 'AWS::ECS::Service' stuck on 'CREATE_IN_PROGRESS' with status 'Resource creation Initiated'.
Can anyone please tell me if there's a configuration problem with my CloudFormation template?
How would i troubleshoot this?
CloudFormation Template
Thank you!
"FargateTasksCluster": {
"Type": "AWS::ECS::Cluster"
},
"FargateTasksLogGroup": {
"Type": "AWS::Logs::LogGroup"
},
"MytaskTask": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"ContainerDefinitions": [{
"Name": "my-task",
"Image": "mountainpass/infrastructure:helloworld",
"Environment": [],
"LogConfiguration": {
"LogDriver": "awslogs",
"Options": {
"awslogs-region": {
"Fn::Sub": "${AWS::Region}"
},
"awslogs-group": {
"Fn::Sub": "${FargateTasksLogGroup}"
},
"awslogs-stream-prefix": "fargate"
}
}
}],
"Family": "my-task",
"NetworkMode": "awsvpc",
"ExecutionRoleArn": {
"Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole"
},
"TaskRoleArn": {
"Fn::Sub": "${IamRoleLambdaExecution}"
},
"RequiresCompatibilities": ["FARGATE"],
"Memory": "0.5GB",
"Cpu": 256
}
},
"MytaskService": {
"Type": "AWS::ECS::Service",
"Properties": {
"Cluster": {
"Fn::Sub": "${FargateTasksCluster}"
},
"LaunchType": "FARGATE",
"ServiceName": "my-task",
"DesiredCount": 1,
"TaskDefinition": {
"Fn::Sub": "${MytaskTask}"
},
"NetworkConfiguration": {
"AwsvpcConfiguration": {
"AssignPublicIp": "DISABLED",
"SecurityGroups": [],
"Subnets": ["subnet-2cc37974", "subnet-3c959d5b", "subnet-49d7d200"]
}
}
}
}

SAM template for API Gateway has errors within Visual Studio (as part of Serverless Application solution)

I am trying to create a SAM template (serverless.template) in visual studio to publish my API Gateway. I have a couple of errors being produced within the template validation (in Visual Studio) that I'm unable to resolve. When published, the stack deploys, but there is no usage plan(s) or api key(s) created (or rather, they are created but not viewable in the console, they come up as 'invalid reference' if you try to view them via Cloudformation > Resources, and they don't show up at all in the API Gateway console).
The errors are:
ServerlessRestApiDeployment26aad1646f is an unknown reference
"ServerlessRestApiProdStage": {
"Type": "AWS::ApiGateway::Stage",
"Properties": {
"DeploymentId": {
"Ref": "ServerlessRestApiDeployment26aad1646f"
},
"RestApiId": {
"Ref": "ServerlessRestApi"
},
"StageName": "Prod"
}
},
And
ServerlessRestApiProdStage is an invalid type for this reference
"APIGatewayHeartInHandKey": {
"Type": "AWS::ApiGateway::ApiKey",
"DependsOn": [
"ServerlessRestApi",
"ServerlessRestApiProdStage"
],
"Properties": {
"Name": "HeartInHandApiKey",
"Description": "Api Key for Heart In Hand",
"Enabled": true,
"GenerateDistinctId": true,
"StageKeys": [
{
"RestApiId": {
"Ref": "ServerlessRestApi"
},
"StageName": {
"Ref": "ServerlessRestApiProdStage"
}
}
]
}
},
The full SAM template shown below.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Transform": "AWS::Serverless-2016-10-31",
"Description": "API Gateway to access InSite data-store",
"Resources": {
"Get": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::Get",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/",
"Method": "GET"
}
}
}
}
},
"GetTableBasic": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::GetTableBasic",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/tables/{tableid}/{columnid}",
"Method": "GET"
}
}
}
}
},
"GetColumnList": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::GetColumnList",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/list/columns/{tableid}",
"Method": "GET"
}
}
}
}
},
"GetTableList": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::GetTableList",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/list/tables",
"Method": "GET"
}
}
}
}
},
"PostClickCollectNotification": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::PostClickCollectNotification",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/datagw/general/webhook/ccnotify",
"Method": "POST"
}
}
}
}
},
"PostClickCollectStockUpdate": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::PostClickCollectStockUpdate",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/datagw/general/post/sohupdate",
"Method": "POST"
}
}
}
}
},
"GetTableResponse": {
"Type": "AWS::Serverless::Function",
"Properties": {
"VpcConfig": {
"SecurityGroupIds": [
"sg-111a1476"
],
"SubnetIds": [
"subnet-3029a769",
"subnet-5ec0b928"
]
},
"Handler": "AWSServerlessInSiteDataGw::AWSServerlessInSiteDataGw.Functions::GetTableResponse",
"Runtime": "dotnetcore2.0",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole",
"AWSLambdaVPCAccessExecutionRole",
"AmazonSSMFullAccess"
],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "datagw/general/table/get/{tableid}",
"Method": "GET"
}
}
}
}
},
"ServerlessRestApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Description":"This is a placeholder for the description of this web api",
"ApiKeySourceType":"HEADER",
"Body": {
"info": {
"version": "1.0",
"title": {
"Ref": "AWS::StackName"
}
},
"paths": {
"/list/tables": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetTableList.Arn}/invocations"
}
},
"responses": {}
}
},
"/list/columns/{tableid}": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetColumnList.Arn}/invocations"
}
},
"responses": {}
}
},
"datagw/general/table/get/{tableid}": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetTableResponse.Arn}/invocations"
}
},
"responses": {}
}
},
"/": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Get.Arn}/invocations"
}
},
"responses": {}
}
},
"/tables/{tableid}/{columnid}": {
"get": {
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetTableBasic.Arn}/invocations"
}
},
"responses": {}
}
}
},
"swagger": "2.0"
}
}
},
"ServerlessRestApiProdStage": {
"Type": "AWS::ApiGateway::Stage",
"Properties": {
"DeploymentId": {
"Ref": "ServerlessRestApiDeployment26aad1646f"
},
"RestApiId": {
"Ref": "ServerlessRestApi"
},
"StageName": "Prod"
}
},
"CustomLambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "lambdaAccessApiKeys",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"apigateway:GET"
],
"Resource": {
"Fn::Sub": [
"arn:aws:apigateway:ap-southeast-2::/apikeys/${__keyId__}",
{
"__keyId__": {
"Ref": "APIGatewayHeartInHandKey"
}
}
]
}
}
]
}
}
]
}
},
"GetApiKeyValueLambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"ZipFile": {
"Fn::Join": [
"\n",
[
"import json",
"import boto3",
"client = boto3.client('apigateway')",
"def lambda_handler(event, context):",
" response = client.get_api_key(",
" apiKey= event['apiKeyId'],",
" includeValue = True",
")",
" return {",
" 'statusCode': 200,",
" 'body': response['value']}"
]
]
}
},
"Handler": "index.lambda_handler",
"Runtime": "python3.6",
"Timeout": 30,
"Role": {
"Fn::GetAtt": [
"CustomLambdaExecutionRole",
"Arn"
]
}
}
},
"APIGatewayUsagePlanInternal": {
"Type": "AWS::ApiGateway::UsagePlan",
"Properties": {
"ApiStages": [
{
"ApiId": {
"Ref": "ServerlessRestApi"
},
"Stage": {
"Ref": "ServerlessRestApiProdStage"
}
}
],
"Description": "Internal Apps Usage Plan",
"UsagePlanName": "Insite-datagw-InternalAppPlan"
}
},
"APIGatewayUsagePlanExternal": {
"Type": "AWS::ApiGateway::UsagePlan",
"Properties": {
"ApiStages": [
{
"ApiId": {
"Ref": "ServerlessRestApi"
},
"Stage": {
"Ref": "ServerlessRestApiProdStage"
}
}
],
"Description": "External Apps Usage Plan",
"UsagePlanName": "InSite-datagw-ExternalAppPlan"
}
},
"APIGatewayHeartInHandKey": {
"Type": "AWS::ApiGateway::ApiKey",
"DependsOn": [
"ServerlessRestApi",
"ServerlessRestApiProdStage"
],
"Properties": {
"Name": "HeartInHandApiKey",
"Description": "Api Key for Heart In Hand",
"Enabled": true,
"GenerateDistinctId": true,
"StageKeys": [
{
"RestApiId": {
"Ref": "ServerlessRestApi"
},
"StageName": {
"Ref": "ServerlessRestApiProdStage"
}
}
]
}
},
"LinkHeartInHandKey": {
"Type": "AWS::ApiGateway::UsagePlanKey",
"Properties": {
"KeyId": {
"Ref": "APIGatewayHeartInHandKey"
},
"KeyType": "API_KEY",
"UsagePlanId": {
"Ref": "APIGatewayUsagePlanInternal"
}
}
},
"APIGatewayPricelineSiteKey": {
"Type": "AWS::ApiGateway::ApiKey",
"DependsOn": [
"ServerlessRestApi",
"ServerlessRestApiProdStage"
],
"Properties": {
"Name": "PricelineSiteApiKey",
"Description": "Api Key for Priceline Website",
"Enabled": true,
"GenerateDistinctId": true,
"StageKeys": [
{
"RestApiId": {
"Ref": "ServerlessRestApi"
},
"StageName": {
"Ref": "ServerlessRestApiProdStage"
}
}
]
}
},
"LinkPricelineSiteKey": {
"Type": "AWS::ApiGateway::UsagePlanKey",
"Properties": {
"KeyId": {
"Ref": "APIGatewayPricelineSiteKey"
},
"KeyType": "API_KEY",
"UsagePlanId": {
"Ref": "APIGatewayUsagePlanInternal"
}
}
}
},
"Outputs": {
"ApiURL": {
"Description": "API endpoint URL for Prod environment",
"Value": {
"Fn::Sub": "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
}
}
}
}
it looks like you're trying to use features of the AWS::Serverless::Api resource from SAM without defining an AWS::Serverless::Api in your template.
In order to fix the issues you've brought up, it looks like you need to:
Remove the ServerlessRestApiProdStage resource
Sam will generate this resource (and the deployment resource) for you if you use an AWS::Serverless::Api resource.
Convert your AWS::ApiGateway::RestApi resource into an AWS::Serverless::Api resource:
Remove ApiKeySourceType property and add "x-amazon-apigateway-api-key-source" : "HEADER", to the swagger,
Change the Type from AWS::ApiGateway::RestApi to AWS::Serverless::Api
"ServerlessRestApi": {
"Type": "AWS::Serverless::RestApi",
"Properties": {
"Description":"This is a placeholder for the description of this web api",
"Body": {
"info": {
"version": "1.0",
"title": {
"Ref": "AWS::StackName"
}
},
"x-amazon-apigateway-api-key-source" : "HEADER",
"paths": {
...

Resources