Creating/Manipulating Elasticsearch Data View from filebeat.yml - elasticsearch

Question

I created some kibana dashboards and they are referencing to a kibana data view. This data view is currently default generated by Filebeat when i'm uploading the filebeat setup, etc. Therefore the data view gets a random Id and my dashboard gets a mising reference since those id's are changing. Now i want to configure somewhere the default created data view since my index- and template-pattern have fixed names and i could create a fixed data view. Then my Dashboards wouldn't go into "missing references". I searched the following sites:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html
https://github.com/elastic/kibana/issues
I created the Dashboards within the kibana UI and exported them for easier integration within my filebeat project.
My filebeat.yml file
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
setup.template.name: pac-filebeat-%{[agent.version]}
setup.template.pattern: pac-filebeat-%{[agent.version]}
setup.template.fields: ${path.config}/fields.yml
setup.dashboards.enabled: false
setup.dashboards.directory: ${path.config}\kibana\custom
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "localhost:5601"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "http"
index: pac-filebeat-%{[agent.version]}
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
This is one example Dashboard.
{
"attributes": {
"description": "",
"hits": 0,
"kibanaSavedObjectMeta": {
"searchSourceJSON": {
"query": {
"query": "",
"language": "kuery"
},
"filter": []
}
},
"optionsJSON": {
"useMargins": true,
"syncColors": false,
"syncTooltips": false,
"hidePanelTitles": false
},
"panelsJSON": [
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 0,
"y": 0,
"w": 9,
"h": 10,
"i": "70779a3d-7880-416d-9736-8a374af8ae41"
},
"panelIndex": "70779a3d-7880-416d-9736-8a374af8ae41",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsLegacyMetric",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-2a27a515-43d0-4aae-8764-467983de9faf"
},
{
"type": "index-pattern",
"name": "99c2853a-48e3-4042-badf-9c0829cec119",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"
}
],
"state": {
"visualization": {
"layerId": "2a27a515-43d0-4aae-8764-467983de9faf",
"accessor": "10b01daf-1c3d-46d1-a014-b381ac70019c",
"layerType": "data"
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [
{
"meta": {
"index": "99c2853a-48e3-4042-badf-9c0829cec119",
"alias": "pac.log.level : ERROR",
"type": "custom",
"key": "query",
"value": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"should": [
{
"match": {
"pac.log.level": "ERROR"
}
}
],
"minimum_should_match": 1
}
}
],
"should": [],
"must_not": []
}
},
"disabled": false,
"negate": false
},
"query": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"should": [
{
"match": {
"pac.log.level": "ERROR"
}
}
],
"minimum_should_match": 1
}
}
],
"should": [],
"must_not": []
}
},
"$state": {
"store": "appState"
}
}
],
"datasourceStates": {
"indexpattern": {
"layers": {
"2a27a515-43d0-4aae-8764-467983de9faf": {
"columns": {
"10b01daf-1c3d-46d1-a014-b381ac70019c": {
"label": "Errors",
"customLabel": true,
"dataType": "number",
"operationType": "count",
"isBucketed": false,
"scale": "ratio",
"sourceField": "pac.log.level",
"filter": {
"query": "pac.log.level : ERROR",
"language": "kuery"
},
"params": {
"emptyAsNull": true
}
}
},
"columnOrder": [
"10b01daf-1c3d-46d1-a014-b381ac70019c"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
},
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 9,
"y": 0,
"w": 12,
"h": 10,
"i": "2bbc2f60-dfb2-4545-a725-9f9bec323e4b"
},
"panelIndex": "2bbc2f60-dfb2-4545-a725-9f9bec323e4b",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsXY",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-a0f69cd9-6726-4551-9a91-cb330cf0b16d"
}
],
"state": {
"visualization": {
"legend": {
"isVisible": true,
"position": "right"
},
"valueLabels": "hide",
"fittingFunction": "None",
"xTitle": "",
"yTitle": "",
"axisTitlesVisibilitySettings": {
"x": false,
"yLeft": false,
"yRight": true
},
"tickLabelsVisibilitySettings": {
"x": true,
"yLeft": false,
"yRight": true
},
"labelsOrientation": {
"x": 0,
"yLeft": 0,
"yRight": 0
},
"gridlinesVisibilitySettings": {
"x": true,
"yLeft": false,
"yRight": true
},
"preferredSeriesType": "bar_horizontal_percentage_stacked",
"layers": [
{
"layerId": "a0f69cd9-6726-4551-9a91-cb330cf0b16d",
"seriesType": "bar_horizontal_percentage_stacked",
"accessors": [
"cbf5323b-0d81-4411-afa3-91087f062874"
],
"yConfig": [],
"layerType": "data",
"splitAccessor": "a98ba577-d539-4ab3-b798-2bddcb2efd3f",
"palette": {
"type": "palette",
"name": "status"
}
}
]
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [],
"datasourceStates": {
"indexpattern": {
"layers": {
"a0f69cd9-6726-4551-9a91-cb330cf0b16d": {
"columns": {
"a98ba577-d539-4ab3-b798-2bddcb2efd3f": {
"label": "ESA states",
"dataType": "string",
"operationType": "terms",
"scale": "ordinal",
"sourceField": "pac.log.esa.connection",
"isBucketed": true,
"params": {
"size": 3,
"orderBy": {
"type": "column",
"columnId": "cbf5323b-0d81-4411-afa3-91087f062874"
},
"orderDirection": "desc",
"otherBucket": true,
"missingBucket": false,
"parentFormat": {
"id": "terms"
}
},
"customLabel": true
},
"cbf5323b-0d81-4411-afa3-91087f062874": {
"label": "Connected",
"dataType": "number",
"operationType": "unique_count",
"scale": "ratio",
"sourceField": "pac.log.esa.connection",
"isBucketed": false,
"filter": {
"query": "pac.log.esa.connection : Connected",
"language": "kuery"
},
"params": {
"emptyAsNull": true
},
"customLabel": true
}
},
"columnOrder": [
"a98ba577-d539-4ab3-b798-2bddcb2efd3f",
"cbf5323b-0d81-4411-afa3-91087f062874"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
},
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 21,
"y": 0,
"w": 7,
"h": 10,
"i": "ef851b6b-222a-4b70-a7c3-ec6a85bdf4d1"
},
"panelIndex": "ef851b6b-222a-4b70-a7c3-ec6a85bdf4d1",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsLegacyMetric",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-d8719702-b241-4931-8bd3-d706f1403c25"
}
],
"state": {
"visualization": {
"layerId": "d8719702-b241-4931-8bd3-d706f1403c25",
"accessor": "312ded46-d0e9-4dcf-9779-b0ca10249a90",
"layerType": "data"
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [],
"datasourceStates": {
"indexpattern": {
"layers": {
"d8719702-b241-4931-8bd3-d706f1403c25": {
"columns": {
"312ded46-d0e9-4dcf-9779-b0ca10249a90": {
"label": "Failed DEO Executions",
"customLabel": true,
"dataType": "number",
"operationType": "count",
"isBucketed": false,
"scale": "ratio",
"sourceField": "pac.log.deo.status",
"filter": {
"query": "pac.log.deo.status : failed",
"language": "kuery"
},
"params": {
"emptyAsNull": true
}
}
},
"columnOrder": [
"312ded46-d0e9-4dcf-9779-b0ca10249a90"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
},
{
"version": "8.5.3",
"type": "lens",
"gridData": {
"x": 0,
"y": 10,
"w": 14,
"h": 11,
"i": "7bcd2a3e-54c1-4b09-a0b6-d41c339c2ae6"
},
"panelIndex": "7bcd2a3e-54c1-4b09-a0b6-d41c339c2ae6",
"embeddableConfig": {
"attributes": {
"title": "",
"visualizationType": "lnsXY",
"type": "lens",
"references": [
{
"type": "index-pattern",
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "indexpattern-datasource-layer-cd91feb0-0927-4465-b0fd-b5bac50af734"
}
],
"state": {
"visualization": {
"title": "Empty XY chart",
"legend": {
"isVisible": true,
"position": "right",
"showSingleSeries": true
},
"valueLabels": "hide",
"preferredSeriesType": "line",
"layers": [
{
"layerId": "cd91feb0-0927-4465-b0fd-b5bac50af734",
"accessors": [
"8ccc17fc-529e-4460-9925-149e77e37dd1"
],
"position": "top",
"seriesType": "line",
"showGridlines": false,
"layerType": "data",
"xAccessor": "43aa69d1-f806-4f87-a2b8-57718e85400a"
}
],
"yTitle": "",
"axisTitlesVisibilitySettings": {
"x": true,
"yLeft": false,
"yRight": true
}
},
"query": {
"query": "",
"language": "kuery"
},
"filters": [],
"datasourceStates": {
"indexpattern": {
"layers": {
"cd91feb0-0927-4465-b0fd-b5bac50af734": {
"columns": {
"43aa69d1-f806-4f87-a2b8-57718e85400a": {
"label": "#timestamp",
"dataType": "date",
"operationType": "date_histogram",
"sourceField": "#timestamp",
"isBucketed": true,
"scale": "interval",
"params": {
"interval": "auto",
"includeEmptyRows": true,
"dropPartials": false
}
},
"8ccc17fc-529e-4460-9925-149e77e37dd1": {
"label": " System cpu usage",
"dataType": "number",
"operationType": "median",
"sourceField": "pac.log.system.monitor.cpu.pct",
"isBucketed": false,
"scale": "ratio",
"params": {
"emptyAsNull": true,
"format": {
"id": "percent",
"params": {
"decimals": 2
}
}
},
"customLabel": true
}
},
"columnOrder": [
"43aa69d1-f806-4f87-a2b8-57718e85400a",
"8ccc17fc-529e-4460-9925-149e77e37dd1"
],
"incompleteColumns": {}
}
}
}
},
"internalReferences": [],
"adHocDataViews": {}
}
},
"enhancements": {}
}
}
],
"timeRestore": false,
"title": "ESA overview",
"version": 1
},
"coreMigrationVersion": "8.5.3",
"id": "d29c4ac0-ab68-11ed-bfcf-939f8111002e",
"migrationVersion": {
"dashboard": "8.5.0"
},
"references": [
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "70779a3d-7880-416d-9736-8a374af8ae41:indexpattern-datasource-layer-2a27a515-43d0-4aae-8764-467983de9faf",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "70779a3d-7880-416d-9736-8a374af8ae41:99c2853a-48e3-4042-badf-9c0829cec119",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "2bbc2f60-dfb2-4545-a725-9f9bec323e4b:indexpattern-datasource-layer-a0f69cd9-6726-4551-9a91-cb330cf0b16d",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "ef851b6b-222a-4b70-a7c3-ec6a85bdf4d1:indexpattern-datasource-layer-d8719702-b241-4931-8bd3-d706f1403c25",
"type": "index-pattern"
},
{
"id": "454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d",
"name": "7bcd2a3e-54c1-4b09-a0b6-d41c339c2ae6:indexpattern-datasource-layer-cd91feb0-0927-4465-b0fd-b5bac50af734",
"type": "index-pattern"
},
{
"id": "c6a11f70-ab68-11ed-bfcf-939f8111002e",
"name": "tag-c6a11f70-ab68-11ed-bfcf-939f8111002e",
"type": "tag"
}
],
"type": "dashboard",
"updated_at": "2023-02-13T07:14:45.551Z",
"version": "WzE1NTI1NiwzMl0="
}
This would be the used tag inside the above dashboard
{
"attributes": {
"color": "#52bc22",
"description": "",
"name": "custom"
},
"coreMigrationVersion": "8.5.3",
"id": "c6a11f70-ab68-11ed-bfcf-939f8111002e",
"migrationVersion": {
"tag": "8.0.0"
},
"references": [],
"type": "tag",
"updated_at": "2023-02-13T07:39:07.543Z",
"version": "WzE1NTM0NywzMl0="
}
The error message:
error dashboard asset: returned 200 to import file: 1 error: error: missing_references, asset ID=d29c4ac0-ab68-11ed-bfcf-939f8111002e; asset type=dashboard; references=[{Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d} {Type:index-pattern ID:454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d}]. Response: {"successCount":1,"success":false,"warnings":[],"successResults":[{"type":"tag","id":"c6a11f70-ab68-11ed-bfcf-939f8111002e","meta":{"title":"custom","icon":"tag"},"overwrite":true}],"errors":[{"id":"d29c4ac0-ab68-11ed-bfcf-939f8111002e","type":"dashboard","meta":{"title":"ESA overview","icon":"dashboardApp"},"error":{"type":"missing_references","references":[{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"},{"type":"index-pattern","id":"454f0a8e-d5b8-4c8d-9dbe-a1a61782d24d"}]},"overwrite":true}]}
kind regardes

Answer 1

You can configure Filebeat to use your own dashboard using the following settings:
setup.dashboards.url
setup.dashboards.file
setup.dashboards.index
setup.dashboards.string_replacements (<-- probably the one you're looking for)