NiFi - Non-Heap Used getting larger - apache-nifi

NiFi Cluster
2 Node(CPU: 4/8, Memory: 32GB; NiFi JVM: 1GB) NiFi Cluster.
This NiFi is for ETL use, main processor is ExecuteSQL. Total Processor reach 2K.
Here is the problem: Non-head usage took more then 3GB memory which jvm just set to 1GB.
jvm setting
java.arg.2=-Xms1G
java.arg.3=-Xmx1G
// other args set to default
usage
{
"systemDiagnostics": {
"aggregateSnapshot": {
"totalNonHeap": "7.56 GB",
"totalNonHeapBytes": 8118382592,
"usedNonHeap": "7.38 GB",
"usedNonHeapBytes": 7927382112,
"freeNonHeap": "182.15 MB",
"freeNonHeapBytes": 191000480,
"maxNonHeap": "-2 bytes",
"maxNonHeapBytes": -2,
"nonHeapUtilization": "-1.232114368E9%",
"totalHeap": "2 GB",
"totalHeapBytes": 2147483648,
"usedHeap": "1.66 GB",
"usedHeapBytes": 1784031344,
"freeHeap": "346.62 MB",
"freeHeapBytes": 363452304,
"maxHeap": "2 GB",
"maxHeapBytes": 2147483648,
"heapUtilization": "83.0%",
"availableProcessors": 16,
"processorLoadAverage": 7.529999999999999,
"totalThreads": 768,
"daemonThreads": 617,
"uptime": "218:04:38.693",
"flowFileRepositoryStorageUsage": {
"freeSpace": "727.24 GB",
"totalSpace": "984.06 GB",
"usedSpace": "256.82 GB",
"freeSpaceBytes": 780865024000,
"totalSpaceBytes": 1056621576192,
"usedSpaceBytes": 275756552192,
"utilization": "26.0%"
},
"contentRepositoryStorageUsage": [
{
"identifier": "default",
"freeSpace": "727.24 GB",
"totalSpace": "984.06 GB",
"usedSpace": "256.82 GB",
"freeSpaceBytes": 780865024000,
"totalSpaceBytes": 1056621576192,
"usedSpaceBytes": 275756552192,
"utilization": "26.0%"
}
],
"provenanceRepositoryStorageUsage": [
{
"identifier": "default",
"freeSpace": "727.24 GB",
"totalSpace": "984.06 GB",
"usedSpace": "256.82 GB",
"freeSpaceBytes": 780865024000,
"totalSpaceBytes": 1056621576192,
"usedSpaceBytes": 275756552192,
"utilization": "26.0%"
}
],
"garbageCollection": [
{
"name": "G1 Young Generation",
"collectionCount": 442716,
"collectionTime": "12:12:02.124",
"collectionMillis": 43922124
},
{
"name": "G1 Old Generation",
"collectionCount": 8,
"collectionTime": "00:00:20.826",
"collectionMillis": 20826
}
],
"statsLastRefreshed": "11:24:32 CST",
"versionInfo": {
"niFiVersion": "1.9.2",
"javaVendor": "Oracle Corporation",
"javaVersion": "1.8.0_161",
"osName": "Linux",
"osVersion": "4.4.0-93-generic",
"osArchitecture": "amd64",
"buildTag": "nifi-1.9.2-RC2",
"buildRevision": "ff01ff6",
"buildBranch": "NIFI-6169-RC2",
"buildTimestamp": "04/03/2019 15:25:53 CST"
}
}
}
}
How could I low the non-head usage?
Is it about the almost 2k processors? or other reasons?
update
add jmap -histo:live [pid] > histo-live.txt file
histo-live.txt

Related

Jinja2 for loop doesn't follow original dictionary order

So I'm using a .j2 template file to generate a report regarding primary and secondary disk devices in a system.
I have to print the primary disk first and I'm covering only sd* and vd* device naming scheme.
Problem is the for loop always do a reverse lookup on the dictionary.
Here's the a ansible_devices variable on my system.
ansible1 | SUCCESS => {
"ansible_facts": {
"ansible_devices": {
"dm-0": {
"holders": [],
"host": "",
"links": {
"ids": [
"dm-name-rhel_rhel8-root",
"dm-uuid-LVM-DOufRu3rljRtRzbfEbU3jOt0XPsF4dTm8muvelhzXFaWdS0ivguPLcwB1EQaTdOj"
],
"labels": [],
"masters": [],
"uuids": [
"26028eb5-091e-41f5-993c-8639ded7c1c5"
]
},
"model": null,
"partitions": {},
"removable": "0",
"rotational": "1",
"sas_address": null,
"sas_device_handle": null,
"scheduler_mode": "",
"sectors": "146800640",
"sectorsize": "512",
"serial": "VB011d5a18",
"size": "70.00 GB",
"support_discard": "0",
"vendor": null,
"virtual": 1
},
"dm-1": {
"holders": [],
"host": "",
"links": {
"ids": [
"dm-name-rhel_rhel8-swap",
"dm-uuid-LVM-DOufRu3rljRtRzbfEbU3jOt0XPsF4dTmv7C9OKBjTw1x1xQ4VeF7G2iGYeCHi6Y2"
],
"labels": [],
"masters": [],
"uuids": [
"15cb7bae-eec8-421e-a324-5c8087b31b17"
]
},
"model": null,
"partitions": {},
"removable": "0",
"rotational": "1",
"sas_address": null,
"sas_device_handle": null,
"scheduler_mode": "",
"sectors": "4317184",
"sectorsize": "512",
"serial": "VB011d5a18",
"size": "2.06 GB",
"support_discard": "0",
"vendor": null,
"virtual": 1
},
"sda": {
"holders": [],
"host": "IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)",
"links": {
"ids": [
"ata-VBOX_HARDDISK_VB011d5a18-fc7b8d75",
"scsi-0ATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75",
"scsi-1ATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75",
"scsi-SATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75"
],
"labels": [],
"masters": [],
"uuids": []
},
"model": "VBOX HARDDISK",
"partitions": {
"sda1": {
"holders": [],
"links": {
"ids": [
"ata-VBOX_HARDDISK_VB011d5a18-fc7b8d75-part1",
"scsi-0ATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75-part1",
"scsi-1ATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75-part1",
"scsi-SATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75-part1"
],
"labels": [],
"masters": [],
"uuids": [
"7431a3cf-0ed2-4813-9df7-083f8ee4f6ae"
]
},
"sectors": "2097152",
"sectorsize": 512,
"size": "1.00 GB",
"start": "2048",
"uuid": "7431a3cf-0ed2-4813-9df7-083f8ee4f6ae"
},
"sda2": {
"holders": [
"rhel_rhel8-swap",
"rhel_rhel8-root"
],
"links": {
"ids": [
"ata-VBOX_HARDDISK_VB011d5a18-fc7b8d75-part2",
"lvm-pv-uuid-hBGpnr-FYc6-95QG-dsoT-qD3k-Imd0-0yoWzd",
"scsi-0ATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75-part2",
"scsi-1ATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75-part2",
"scsi-SATA_VBOX_HARDDISK_VB011d5a18-fc7b8d75-part2"
],
"labels": [],
"masters": [
"dm-0",
"dm-1"
],
"uuids": []
},
"sectors": "266336256",
"sectorsize": 512,
"size": "127.00 GB",
"start": "2099200",
"uuid": null
}
},
"removable": "0",
"rotational": "1",
"sas_address": null,
"sas_device_handle": null,
"scheduler_mode": "mq-deadline",
"sectors": "268435456",
"sectorsize": "512",
"serial": "VB011d5a18",
"size": "128.00 GB",
"support_discard": "0",
"vendor": "ATA",
"virtual": 1
},
"sdb": {
"holders": [],
"host": "IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)",
"links": {
"ids": [
"ata-VBOX_HARDDISK_VB6207845b-7aa2bea6",
"scsi-0ATA_VBOX_HARDDISK_VB6207845b-7aa2bea6",
"scsi-1ATA_VBOX_HARDDISK_VB6207845b-7aa2bea6",
"scsi-SATA_VBOX_HARDDISK_VB6207845b-7aa2bea6"
],
"labels": [],
"masters": [],
"uuids": []
},
"model": "VBOX HARDDISK",
"partitions": {},
"removable": "0",
"rotational": "1",
"sas_address": null,
"sas_device_handle": null,
"scheduler_mode": "mq-deadline",
"sectors": "10485760",
"sectorsize": "512",
"serial": "VB6207845b",
"size": "5.00 GB",
"support_discard": "0",
"vendor": "ATA",
"virtual": 1
}
},
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false
}
As it can be seen, the order is as;
dm-0
dm-1
sda
sdb
My template file is below;
{% for dev in ansible_devices %}
{% if dev == 'sda' %}
primary device {{ dev }} found on {{ ansible_hostname }}
{% endif %}
{% if dev == 'sdb' %}
second device {{ dev }} found on {{ ansible_hostname }}
{% endif %}
{% if dev == 'vda' %}
primary device {{ dev }} found on {{ ansible_hostname }}
{% endif %}
{% if dev == 'vdb' %}
second device {{ dev }} found on {{ ansible_hostname }}
{% endif %}
{% endfor %}
{% if ('sdb' not in ansible_devices) and ('vdb' not in ansible_devices) %}
no second device found on {{ ansible_hostname }}
{% endif %}
If I execute this within a playbook, the following content is generated;
--- before
+++ after: /home/ansible/.ansible/tmp/ansible-local 12580ell0d0oq/tmp1qs9mu2y/devices.j2
## -0,0 +1,2 ##
+primary device sda found on ansible3
+no second device found on ansible3
changed: [ansible3]
--- before
+++ after: /home/ansible/.ansible/tmp/ansible-local-12580ell0d0oq /tmpej185qb4/devices.j2
## -0,0 +1,2 ##
+second device sdb found on ansible2
+primary device sda found on ansible2
changed: [ansible2]
--- before
+++ after: /home/ansible/.ansible/tmp/ansible-local-12580ell0d0oq/tmpkbcmuhqa/devices.j2
## -0,0 +1,2 ##
+primary device sda found on ansible4
+no second device found on ansible4
changed: [ansible4]
--- before
+++ after: /home/ansible/.ansible/tmp/ansible-local-12580ell0d0oq/tmpvkm3uid6/devices.j2
## -0,0 +1,2 ##
+second device sdb found on ansible1
+primary device sda found on ansible1
changed: [ansible1]
Somehow, the iteration is going in reverse order, I've seen this in all the alternative tests for the given scenario.
Am I missing something here?

Possible bug NOT_AVAILABLE_IN_COUNTRY

I am trying to enable policy via Android Management API.
{
"name": "enterprises/LC03hx99sc/policies/default",
"version": "2",
"applications": [
{
"packageName": "app.caredirect.caredirecthub2",
"installType": "KIOSK"
},
{
"packageName": "com.android.settings",
"installType": "FORCE_INSTALLED",
"defaultPermissionPolicy": "GRANT"
},
{
"packageName": "com.amazon.dee.app",
"installType": "FORCE_INSTALLED",
"defaultPermissionPolicy": "GRANT"
}
],
"systemUpdate": {
"type": "WINDOWED",
"startMinutes": 10,
"endMinutes": 1439
},
"debuggingFeaturesAllowed": true
}
The store listing says for app.caredirect.caredirecthub2 that it is available in EVERY country.
Despite this I get the following error:
{
"name": "enterprises/LC03hx99sc/devices/33ddb8137f60b585",
"managementMode": "DEVICE_OWNER",
"state": "ACTIVE",
"appliedState": "ACTIVE",
"nonComplianceDetails": [{
"settingName": "applications",
"nonComplianceReason": "APP_NOT_INSTALLED",
"packageName": "app.caredirect.caredirecthub2",
"installationFailureReason": "NOT_AVAILABLE_IN_COUNTRY"
}, {
"settingName": "persistentPreferredActivities",
"nonComplianceReason": "APP_NOT_INSTALLED",
"packageName": "app.caredirect.caredirecthub2"
}],
"enrollmentTime": "2020-09-23T15:02:52.290Z",
"lastStatusReportTime": "2020-09-23T15:02:57.915Z",
"lastPolicySyncTime": "2020-09-23T15:02:53.754Z",
"appliedPolicyVersion": "1",
"apiLevel": 28,
"softwareInfo": {
"androidVersion": "9",
"androidDevicePolicyVersionCode": 1317100,
"androidDevicePolicyVersionName": "13.17.10.v32",
"androidBuildNumber": "TB-X605F_S210208_200807_ROW",
"deviceKernelVersion": "3.18.120-perf-g1d02637-dirty",
"bootloaderVersion": "unknown",
"androidBuildTime": "2020-08-07T10:08:52Z",
"securityPatchLevel": "2020-08-01",
"primaryLanguageCode": "en-US",
"deviceBuildSignature": "1c41fedecacbe89b81c242ecf74929f7d57322b64f8179c1c78a390a3494f14a",
"systemUpdateInfo": {
"updateStatus": "UP_TO_DATE"
}
},
"hardwareInfo": {
"brand": "Lenovo",
"hardware": "qcom",
"manufacturer": "LENOVO",
"serialNumber": "HA0Y8JQX",
"model": "Lenovo TB-X605F"
},
"appliedPolicyName": "enterprises/LC03hx99sc/policies/default",
"memoryInfo": {
"totalRam": "1909563392",
"totalInternalStorage": "3121049600"
},
"userName": "enterprises/LC03hx99sc/users/106404730049843005283",
"enrollmentTokenName": "enterprises/LC03hx99sc/enrollmentTokens/1019LI5YbxD4MNgIOfRKRxxKtK4ZLzRBZTGlL0Tw5Ns",
"previousDeviceNames": ["enterprises/LC03hx99sc/devices/3832aa78af19d796", "enterprises/LC03hx99sc/devices/326e9eaec5f835db", "enterprises/LC03hx99sc/devices/3f22065004da99b8", "enterprises/LC03hx99sc/devices/3027d6b05179f8ac", "enterprises/LC03hx99sc/devices/3011e18ade8e09f3", "enterprises/LC03hx99sc/devices/36ea1951ef4e088f", "enterprises/LC03hx99sc/devices/30fb71bc476a4d1d", "enterprises/LC03hx99sc/devices/3695657a941cba4a"],
"securityPosture": {
"devicePosture": "SECURE"
},
"ownership": "COMPANY_OWNED"
}
What is causing NOT_AVAILABLE_IN_COUNTRY ?
This is preventing the whole kiosk setup from working.
thanks

DC/OS marathon Virtual network not working

I installed DC/OS with 3 masters and 3 agents and face a problem with virtual networking. Here is my Marathon app spec:
{
"id": "/nginx",
"cmd": null,
"cpus": 1,
"mem": 128,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "nginx",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"servicePort": 10002,
"protocol": "tcp",
"name": "main1",
"labels": {
"VIP_0": "9.0.0.0:34562"
}
}
],
"privileged": false,
"parameters": [],
"forcePullImage": false
}
},
"portDefinitions": [
{
"port": 10002,
"protocol": "tcp",
"labels": {}
}
]
}
I see the following in the DC/OS virtual network section:
VIRTUAL NETWORK NAME | SUBNET | AGENT PREFIX LENGTH
dcos 9.0.0.0/8 24
The containers stays in waiting for a long time. If I remove the port mapping section it runs successfully.
Basically I need to know how to work with this new virtual network, and fix the service discovery and load balancing without using any extra stuff.
Took me some time to figure it out as well...
You need to:
Remove all ports assignment in the task definition
Describe the name of the network to attach to (default network created is named "dcos")
{
"id": "yourtask",
"container": {
"type": "DOCKER",
"docker": {
"image": "your/image",
"network": "USER"
}
},
"acceptedResourceRoles" : [
"slave_public"
],
"ipAddress": {
"networkName": "dcos"
},
"instances": 2,
"cpus": 0.2,
"mem": 128
}

When I try to send a request to Paysafe, I keep getting the following response form the API

Here is my code that I am using to make my request.
{
"total_amount": "1000",
"currency_code": "USD",
"merchant_ref_num": "12345678",
"customer_notification_email": "test#test.com",
"profile": {
"firstName": "Test",
"lastName": "Sample",
"merchantCustomerId": "12345678"
},
"billingDetails": {
"city": "Montreal",
"country": "CA",
"street": "123 Main Apt 200",
"zip": "H1H1H1",
"state": "QC",
"phone": "555-555-5555"
},
"extendedOptions": [
{
"key": "authType",
"value": "auth"
},
{
"key": "orderTimeout",
"value": 2592000
},
{
"key": "suppressCustomerEmail",
"value": true
},
{
"key": "silentPost",
"value": false
}
],
"callback": [
{
"format": "get",
"rel": "on_success",
"uri": "https://secure4137.hostgator.com/~cmedei/optimal/callbacks/callback.php",
"returnKeys": [
"id",
"profile.paymentToken",
"profile.id",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.authType",
"transaction.status",
"transaction.currencyCode",
"transaction.merchantRefNum",
"transaction.card.brand",
"transaction.card.country",
"transaction.card.expiry",
"transaction.card.lastDigits",
"transaction.card.threeDEnrolment",
"transaction.card.threeDResult",
"transaction.card.type",
"transaction.paymentType",
"transaction.prepaidcard.lastDigits"
],
"retries": 3,
"synchronous": true
},
{
"format": "get",
"rel": "on_decline",
"uri": "https://secure4137.hostgator.com/~cmedei/optimal/callbacks/callback.php",
"returnKeys": [
"id",
"profile.paymentToken",
"profile.id",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.authType",
"transaction.status",
"transaction.currencyCode",
"transaction.merchantRefNum",
"transaction.card.brand",
"transaction.card.country",
"transaction.card.expiry",
"transaction.card.lastDigits",
"transaction.card.threeDEnrolment",
"transaction.card.threeDResult",
"transaction.card.type",
"transaction.paymentType",
"transaction.prepaidcard.lastDigits"
],
"retries": 3,
"synchronous": true
}
],
"redirect": [
{
"rel": "on_success",
"uri": "http://localhost:8080/hosted_api/main/success/",
"returnKeys": [
"id",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.status",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.riskReasonCode",
"transaction.card.lastDigits",
"transaction.card.brand",
"transaction.card.type"
]
},
{
"rel": "on_decline",
"uri": "http://localhost:8080/hosted_api/main/failure/",
"returnKeys": [
"id",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.status",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.riskReasonCode",
"transaction.card.lastDigits",
"transaction.card.brand",
"transaction.card.type"
]
},
{
"rel": "on_error",
"uri": "http://localhost:8080/hosted_api/main/error/",
"returnKeys": [
"id",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.status",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.riskReasonCode",
"transaction.card.lastDigits",
"transaction.card.brand",
"transaction.card.type"
]
},
{
"rel": "on_timeout",
"uri": "http://localhost:8080/hosted_api/main/timeout/",
"returnKeys": [
"id",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.status",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.riskReasonCode",
"transaction.card.lastDigits",
"transaction.card.brand",
"transaction.card.type"
]
},
{
"rel": "on_hold",
"uri": "http://localhost:8080/hosted_api/main/onhold/",
"returnKeys": [
"id",
"transaction.confirmationNumber",
"transaction.amount",
"transaction.status",
"transaction.errorCode",
"transaction.errorMessage",
"transaction.riskReasonCode",
"transaction.card.lastDigits",
"transaction.card.brand",
"transaction.card.type"
]
}
],
"shoppingCart": [
{
"amount": "1000",
"quantity": "1",
"description": "Fast Order"
}
]
}
Here is what I am getting back as the response
{
"error": {
"code": 401,
"message": "Not authorised"
}
}
I receive the same error even if I remove the JSON code. So I am assuming that my credentials are incorrect.
<?php
header("Content-type:application/json");
header("Authorization:application/Basic Og==");
?>
The information that you provided seems to be off. The Authorization in your PHP header seems to be pretty short. There could be a few reasons.
You should be using the Base64 encoding for your key. This is what is needed in order to pass in the information in the header.
base64_encode($APIKey)
Once you do this the information will be passed in correctly. It should look like this.
<?php
header("Content-type:application/json");
header("Authorization:application/Basic cTRlajZHRW5YWXJkUk9pS3JySEo6UEFBMGViNmU0M2Q2MmFkNTk5OTg");
?>
There is also one more thing.
I am not sure what endpoint that you are using but make sure that you are pointing to the correct place. The reason being is that when you are sending to the wrong place, it cannot validate the key and the system will just not know what you are sending in.
Hope that this helps!

Where can I get a list of Ansible pre-defined variables?

I see that Ansible provide some pre-defined variables that we can use in playbooks and template files. For example, the host IP address is ansible_eth0.ipv4.address. Googleing and searching the docs I couldn't find a list of all available variables.
Would someone list them for me?
From the FAQ:
How do I see a list of all of the ansible_ variables?
Ansible by default gathers “facts” about the machines under management, and these facts can be accessed in playbooks and in templates. To see a list of all of the facts that are available about a machine, you can run the setup module as an ad hoc action:
ansible -m setup hostname
This will print out a dictionary of all of the facts that are available for that particular host. You might want to pipe the output to a pager.This does NOT include inventory variables or internal ‘magic’ variables. See the next question if you need more than just ‘facts’.
Here is the output for my vagrant virtual machine called scdev:
scdev | success >> {
"ansible_facts": {
"ansible_all_ipv4_addresses": [
"10.0.2.15",
"192.168.10.10"
],
"ansible_all_ipv6_addresses": [
"fe80::a00:27ff:fe12:9698",
"fe80::a00:27ff:fe74:1330"
],
"ansible_architecture": "i386",
"ansible_bios_date": "12/01/2006",
"ansible_bios_version": "VirtualBox",
"ansible_cmdline": {
"BOOT_IMAGE": "/vmlinuz-3.2.0-23-generic-pae",
"quiet": true,
"ro": true,
"root": "/dev/mapper/precise32-root"
},
"ansible_date_time": {
"date": "2013-09-17",
"day": "17",
"epoch": "1379378304",
"hour": "00",
"iso8601": "2013-09-17T00:38:24Z",
"iso8601_micro": "2013-09-17T00:38:24.425092Z",
"minute": "38",
"month": "09",
"second": "24",
"time": "00:38:24",
"tz": "UTC",
"year": "2013"
},
"ansible_default_ipv4": {
"address": "10.0.2.15",
"alias": "eth0",
"gateway": "10.0.2.2",
"interface": "eth0",
"macaddress": "08:00:27:12:96:98",
"mtu": 1500,
"netmask": "255.255.255.0",
"network": "10.0.2.0",
"type": "ether"
},
"ansible_default_ipv6": {},
"ansible_devices": {
"sda": {
"holders": [],
"host": "SATA controller: Intel Corporation 82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode] (rev 02)",
"model": "VBOX HARDDISK",
"partitions": {
"sda1": {
"sectors": "497664",
"sectorsize": 512,
"size": "243.00 MB",
"start": "2048"
},
"sda2": {
"sectors": "2",
"sectorsize": 512,
"size": "1.00 KB",
"start": "501758"
},
},
"removable": "0",
"rotational": "1",
"scheduler_mode": "cfq",
"sectors": "167772160",
"sectorsize": "512",
"size": "80.00 GB",
"support_discard": "0",
"vendor": "ATA"
},
"sr0": {
"holders": [],
"host": "IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)",
"model": "CD-ROM",
"partitions": {},
"removable": "1",
"rotational": "1",
"scheduler_mode": "cfq",
"sectors": "2097151",
"sectorsize": "512",
"size": "1024.00 MB",
"support_discard": "0",
"vendor": "VBOX"
},
"sr1": {
"holders": [],
"host": "IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)",
"model": "CD-ROM",
"partitions": {},
"removable": "1",
"rotational": "1",
"scheduler_mode": "cfq",
"sectors": "2097151",
"sectorsize": "512",
"size": "1024.00 MB",
"support_discard": "0",
"vendor": "VBOX"
}
},
"ansible_distribution": "Ubuntu",
"ansible_distribution_release": "precise",
"ansible_distribution_version": "12.04",
"ansible_domain": "",
"ansible_eth0": {
"active": true,
"device": "eth0",
"ipv4": {
"address": "10.0.2.15",
"netmask": "255.255.255.0",
"network": "10.0.2.0"
},
"ipv6": [
{
"address": "fe80::a00:27ff:fe12:9698",
"prefix": "64",
"scope": "link"
}
],
"macaddress": "08:00:27:12:96:98",
"module": "e1000",
"mtu": 1500,
"type": "ether"
},
"ansible_eth1": {
"active": true,
"device": "eth1",
"ipv4": {
"address": "192.168.10.10",
"netmask": "255.255.255.0",
"network": "192.168.10.0"
},
"ipv6": [
{
"address": "fe80::a00:27ff:fe74:1330",
"prefix": "64",
"scope": "link"
}
],
"macaddress": "08:00:27:74:13:30",
"module": "e1000",
"mtu": 1500,
"type": "ether"
},
"ansible_form_factor": "Other",
"ansible_fqdn": "scdev",
"ansible_hostname": "scdev",
"ansible_interfaces": [
"lo",
"eth1",
"eth0"
],
"ansible_kernel": "3.2.0-23-generic-pae",
"ansible_lo": {
"active": true,
"device": "lo",
"ipv4": {
"address": "127.0.0.1",
"netmask": "255.0.0.0",
"network": "127.0.0.0"
},
"ipv6": [
{
"address": "::1",
"prefix": "128",
"scope": "host"
}
],
"mtu": 16436,
"type": "loopback"
},
"ansible_lsb": {
"codename": "precise",
"description": "Ubuntu 12.04 LTS",
"id": "Ubuntu",
"major_release": "12",
"release": "12.04"
},
"ansible_machine": "i686",
"ansible_memfree_mb": 23,
"ansible_memtotal_mb": 369,
"ansible_mounts": [
{
"device": "/dev/mapper/precise32-root",
"fstype": "ext4",
"mount": "/",
"options": "rw,errors=remount-ro",
"size_available": 77685088256,
"size_total": 84696281088
},
{
"device": "/dev/sda1",
"fstype": "ext2",
"mount": "/boot",
"options": "rw",
"size_available": 201044992,
"size_total": 238787584
},
{
"device": "/vagrant",
"fstype": "vboxsf",
"mount": "/vagrant",
"options": "uid=1000,gid=1000,rw",
"size_available": 42013151232,
"size_total": 484145360896
}
],
"ansible_os_family": "Debian",
"ansible_pkg_mgr": "apt",
"ansible_processor": [
"Pentium(R) Dual-Core CPU E5300 # 2.60GHz"
],
"ansible_processor_cores": "NA",
"ansible_processor_count": 1,
"ansible_product_name": "VirtualBox",
"ansible_product_serial": "NA",
"ansible_product_uuid": "NA",
"ansible_product_version": "1.2",
"ansible_python_version": "2.7.3",
"ansible_selinux": false,
"ansible_swapfree_mb": 766,
"ansible_swaptotal_mb": 767,
"ansible_system": "Linux",
"ansible_system_vendor": "innotek GmbH",
"ansible_user_id": "neves",
"ansible_userspace_architecture": "i386",
"ansible_userspace_bits": "32",
"ansible_virtualization_role": "guest",
"ansible_virtualization_type": "virtualbox"
},
"changed": false
}
The current documentation now has a complete chapter about Discovering variables: facts and magic variables.
ansible -m setup hostname
Only gets the facts gathered by the setup module.
Gilles Cornu posted a template trick to list all variables for a specific host.
Template (later called dump_variables):
HOSTVARS (ANSIBLE GATHERED, group_vars, host_vars) :
{{ hostvars[inventory_hostname] | to_yaml }}
PLAYBOOK VARS:
{{ vars | to_yaml }}
Playbook to use it:
- hosts: all
tasks:
- template:
src: templates/dump_variables
dest: /tmp/ansible_variables
- fetch:
src: /tmp/ansible_variables
dest: "{{inventory_hostname}}_ansible_variables"
After that you have a dump of all variables on every host, and a copy of each text dump file on your local workstation in your tmp folder. If you don't want local copies, you can remove the fetch statement.
This includes gathered facts, host variables and group variables.
Therefore you see ansible default variables like group_names, inventory_hostname, ansible_ssh_host and so on.
There are 3 sources of variables in Ansible:
Variables gathered from facts. You can get them by running command: ansible -m setup hostname
Built-in (pre-defined) Ansible variables (AKA 'magic' variables). They are documented in Ansible documentation: http://docs.ansible.com/playbooks_variables.html#magic-variables-and-how-to-access-information-about-other-hosts
Here is the list extracted from Ansible 1.9 documentation:
group_names
groups
inventory_hostname
ansible_hostname
inventory_hostname_short
play_hosts
delegate_to
inventory_dir
inventory_file
Variables passed to ansible via command line. But obviously you know what they are
I use this simple playbook:
---
# vars.yml
#
# Shows the value of all variables/facts.
#
# Example:
#
# ansible-playbook vars.yml -e 'hosts=localhost'
#
- hosts: localhost
tasks:
- fail: "You must specify a value for `hosts` variable - e.g.: ansible-playbook vars.yml -e 'hosts=localhost'"
when: hosts is not defined
- hosts: "{{ hosts }}"
tasks:
- debug: var=vars
- debug: var=hostvars[inventory_hostname]
There is lot of variables defined as Facts -- http://docs.ansible.com/ansible/playbooks_variables.html#information-discovered-from-systems-facts
"ansible_all_ipv4_addresses": [
"REDACTED IP ADDRESS"
],
"ansible_all_ipv6_addresses": [
"REDACTED IPV6 ADDRESS"
],
"ansible_architecture": "x86_64",
"ansible_bios_date": "09/20/2012",
"ansible_bios_version": "6.00",
"ansible_cmdline": {
"BOOT_IMAGE": "/boot/vmlinuz-3.5.0-23-generic",
"quiet": true,
"ro": true,
"root": "UUID=4195bff4-e157-4e41-8701-e93f0aec9e22",
"splash": true
},
"ansible_date_time": {
"date": "2013-10-02",
"day": "02",
"epoch": "1380756810",
"hour": "19",
"iso8601": "2013-10-02T23:33:30Z",
"iso8601_micro": "2013-10-02T23:33:30.036070Z",
"minute": "33",
"month": "10",
"second": "30",
"time": "19:33:30",
"tz": "EDT",
"year": "2013"
},
"ansible_default_ipv4": {
"address": "REDACTED",
"alias": "eth0",
"gateway": "REDACTED",
"interface": "eth0",
"macaddress": "REDACTED",
"mtu": 1500,
"netmask": "255.255.255.0",
"network": "REDACTED",
"type": "ether"
},
"ansible_default_ipv6": {},
"ansible_devices": {
"fd0": {
"holders": [],
"host": "",
"model": null,
"partitions": {},
"removable": "1",
"rotational": "1",
"scheduler_mode": "deadline",
"sectors": "0",
"sectorsize": "512",
"size": "0.00 Bytes",
"support_discard": "0",
"vendor": null
},
"sda": {
"holders": [],
"host": "SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)",
"model": "VMware Virtual S",
"partitions": {
"sda1": {
"sectors": "39843840",
"sectorsize": 512,
"size": "19.00 GB",
"start": "2048"
},
"sda2": {
"sectors": "2",
"sectorsize": 512,
"size": "1.00 KB",
"start": "39847934"
},
"sda5": {
"sectors": "2093056",
"sectorsize": 512,
"size": "1022.00 MB",
"start": "39847936"
}
},
"removable": "0",
"rotational": "1",
"scheduler_mode": "deadline",
"sectors": "41943040",
"sectorsize": "512",
"size": "20.00 GB",
"support_discard": "0",
"vendor": "VMware,"
},
"sr0": {
"holders": [],
"host": "IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)",
"model": "VMware IDE CDR10",
"partitions": {},
"removable": "1",
"rotational": "1",
"scheduler_mode": "deadline",
"sectors": "2097151",
"sectorsize": "512",
"size": "1024.00 MB",
"support_discard": "0",
"vendor": "NECVMWar"
}
},
"ansible_distribution": "Ubuntu",
"ansible_distribution_release": "precise",
"ansible_distribution_version": "12.04",
"ansible_domain": "",
"ansible_env": {
"COLORTERM": "gnome-terminal",
"DISPLAY": ":0",
"HOME": "/home/mdehaan",
"LANG": "C",
"LESSCLOSE": "/usr/bin/lesspipe %s %s",
"LESSOPEN": "| /usr/bin/lesspipe %s",
"LOGNAME": "root",
"LS_COLORS": "rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:",
"MAIL": "/var/mail/root",
"OLDPWD": "/root/ansible/docsite",
"PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"PWD": "/root/ansible",
"SHELL": "/bin/bash",
"SHLVL": "1",
"SUDO_COMMAND": "/bin/bash",
"SUDO_GID": "1000",
"SUDO_UID": "1000",
"SUDO_USER": "mdehaan",
"TERM": "xterm",
"USER": "root",
"USERNAME": "root",
"XAUTHORITY": "/home/mdehaan/.Xauthority",
"_": "/usr/local/bin/ansible"
},
"ansible_eth0": {
"active": true,
"device": "eth0",
"ipv4": {
"address": "REDACTED",
"netmask": "255.255.255.0",
"network": "REDACTED"
},
"ipv6": [
{
"address": "REDACTED",
"prefix": "64",
"scope": "link"
}
],
"macaddress": "REDACTED",
"module": "e1000",
"mtu": 1500,
"type": "ether"
},
"ansible_form_factor": "Other",
"ansible_fqdn": "ubuntu2.example.com",
"ansible_hostname": "ubuntu2",
"ansible_interfaces": [
"lo",
"eth0"
],
"ansible_kernel": "3.5.0-23-generic",
"ansible_lo": {
"active": true,
"device": "lo",
"ipv4": {
"address": "127.0.0.1",
"netmask": "255.0.0.0",
"network": "127.0.0.0"
},
"ipv6": [
{
"address": "::1",
"prefix": "128",
"scope": "host"
}
],
"mtu": 16436,
"type": "loopback"
},
"ansible_lsb": {
"codename": "precise",
"description": "Ubuntu 12.04.2 LTS",
"id": "Ubuntu",
"major_release": "12",
"release": "12.04"
},
"ansible_machine": "x86_64",
"ansible_memfree_mb": 74,
"ansible_memtotal_mb": 991,
"ansible_mounts": [
{
"device": "/dev/sda1",
"fstype": "ext4",
"mount": "/",
"options": "rw,errors=remount-ro",
"size_available": 15032406016,
"size_total": 20079898624
}
],
"ansible_nodename": "ubuntu2.example.com",
"ansible_os_family": "Debian",
"ansible_pkg_mgr": "apt",
"ansible_processor": [
"Intel(R) Core(TM) i7 CPU 860 # 2.80GHz"
],
"ansible_processor_cores": 1,
"ansible_processor_count": 1,
"ansible_processor_threads_per_core": 1,
"ansible_processor_vcpus": 1,
"ansible_product_name": "VMware Virtual Platform",
"ansible_product_serial": "REDACTED",
"ansible_product_uuid": "REDACTED",
"ansible_product_version": "None",
"ansible_python_version": "2.7.3",
"ansible_selinux": false,
"ansible_ssh_host_key_dsa_public": "REDACTED KEY VALUE"
"ansible_ssh_host_key_ecdsa_public": "REDACTED KEY VALUE"
"ansible_ssh_host_key_rsa_public": "REDACTED KEY VALUE"
"ansible_swapfree_mb": 665,
"ansible_swaptotal_mb": 1021,
"ansible_system": "Linux",
"ansible_system_vendor": "VMware, Inc.",
"ansible_user_id": "root",
"ansible_userspace_architecture": "x86_64",
"ansible_userspace_bits": "64",
"ansible_virtualization_role": "guest",
"ansible_virtualization_type": "VMware"
The debug module can be used to analyze variables. Be careful running the following command. In our setup it generates 444709 lines with 16MB:
ansible -m debug -a 'var=hostvars' localhost
I am not sure but it might be necessary to enable facts caching.
If you need just one host use the host name as a key for the hostvars hash:
ansible -m debug -a 'var=hostvars.localhost' localhost
This command will display also group and host variables.
Note the official docs on connection configuration variables or "behavioral" variables - which aren't listed in host vars, appears to be List of Behavioral Inventory Parameters in the Inventory documentation.
P.S. The sudo option is undocumented there (yes its sudo not ansible_sudo as you'd expect ...) and probably a couple more aren't, but thats best doc I've found on em.
Some variables are not available on every host, e.g. ansible_domain and domain. If the situation needs to be debugged, I login to the server and issue:
user#server:~$ ansible -m setup localhost | grep domain
[WARNING]: provided hosts list is empty, only localhost is available
"ansible_domain": "prd.example.com",
I know this question has been answered already, but I feel like there are a whole other set of pre-defined variables not covered by the ansible_* facts. This documentation page covers the directives (variables that modify Ansible's behavior), which I was looking for when I came across this page.
This includes some common and some specific use-case directives:
become: Controls privilege escalation (sudo)
delegate_to: run task on another host (like running on localhost)
serial: allows you to run a play across a specific number/percentage of hosts before moving onto next set
https://github.com/f500/ansible-dumpall
FYI: this github project shows you how to list 90% of variables across all hosts. I find it more globally useful than single host commands. The README includes instructions for building a simple inventory report. It's even more valuable to run this at the end of a playbook to see all the Facts. To also debug Task behaviour use register:
The result is missing a few items:
- included YAML file variables
- extra-vars
- a number of the Ansible internal vars described here: Ansible Behavioural Params

Resources