Let me explain the problem and context. This is a server for a database solution. The database was created with docker, and added a volume to the server. Then all docker installation path was moved to the volume added to the server (for security and backup mantain). Then for monitoring, i added a metricbeat agent to capture data, like disk and other stuff, but for this context occurs the problem.
Im searching for a specific mount (the is a volume mount), and when in terminal type df -aTh | grep "/dev" for show filesystem, its show this image:
Then in metricbeat.yaml i have this configuration for system module:
- module: system
period: 30s
metricsets: ["filesystem"]
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|etc|host|hostfs)($|/)'
Notice in last line i omitted "dev" , because i want to obtain the mount volume "/dev/sda" high lighted in the screen shot. But when i discoverd in kibana, that device is not showed, and i dont now why, must be showed.
Thanks for reading and help :) . All of this if for monitoring, and show data in grafana. But i can't find the filesystem "/dev/sda/", for the disk dashboard...
From the documentation about the setting filesystem.ignore_types:
A list of filesystem types to ignore. Metrics will not be collected from filesystems matching these types. This setting also affects the fsstats metricset. If this option is not set, metricbeat ignores all types for virtual devices in systems where this information is available (e.g. all types marked as nodev in /proc/filesystems in Linux systems). This can be set to an empty list ([]) to make filebeat report all filesystems, regardless of type.
If you check the file /proc/filesystems you can see which files are marked as "nodev". Is it possible that ext4 is marked as nodev?
Can you try to set the setting filesystem.ignore_types: [] to see if the file is now considered?
Related
I use the ansible filesystem module to format the data disks of a newly provisioned database cluster.
- name: Format data disk
community.general.filesystem:
fstype: ext4
dev: /dev/sdc
...
But what I want is a way to automatically ckeck before formatting to make sure it doesn't run if the disk is already formatted, although I noticed that the module seems to do that checking on every run but I'm still not sure of its behavior.
Any thoughts?
Yes, the module already checks for you. According to the documentation:
If state=present, the filesystem is created if it doesn’t already exist, that is the default behaviour if state is omitted.
So, the module will not reformat the device if it's already formatted with the provided fstype. To force a reformat, you can change the fstype to something else, or set force=true.
I have windows build agents for Jenkins running in EC2, and I would like to make use of the ephemeral disks coming from the "d" type instances (C5ad.4xl for example gives 2 x 300GB NVMe) to take advantage of the high IO available on those disks.
Since they are build agents the ephemeral nature of the drives is fine. I need something that will detect, provision and mount those disks as a drive in Windows basically regardless of size and number. I can do this easily (LVM or software RAID etc.) in Linux but although there is a guide from 2014 here for achieving this, it doesn't seem to work on Windows Server 2019 and the latest instances.
That same post makes reference to new commandlets added from Server 2012 R2 but those do not support converting the disks to dynamic (which is a key step needed to stripe them done by diskpart in the original post's code), so they cannot be used to directly do what is required.
Are there any other options to make this work dynamically, ideally with powershell (or similar) that can be passed to the Jenkins agent at boot time as part of its config?
Windows now has Storage Pools and these can be used to do what is needed here. This code successfully detected multiple disks, added them to the pool striped, used the max size available and mounted the new volume on the drive letter "E":
# get a list of the disks that can be pooled
$PhysicalDisks = (Get-PhysicalDisk -CanPool $true)
# only take action if there actually are disks
if ($PhysicalDisks) {
# create storage pool using the discovered disks, called ephemeral in the standard subsystem
New-StoragePool –FriendlyName ephemeral -StorageSubSystemFriendlyName "Windows Storage*" –PhysicalDisks $PhysicalDisks
# Create a virtual disk, striped (simple resiliency in its terms), use all space
New-VirtualDisk -StoragePoolFriendlyName "ephemeral" -FriendlyName "stripedephemeral" -ResiliencySettingName Simple -UseMaximumSize
# initialise the disk
Get-VirtualDisk -FriendlyName 'stripedephemeral'|Initialize-Disk -PartitionStyle GPT -PassThru
# create a partition, use all available size (this will pop up if you do it interactively to format the drive, not a problem when running as userdata via Jenkins config)
New-Partition -DiskNumber 3 -DriveLetter 'E' -UseMaximumSize
# format as NTFS to make it useable
Format-Volume -DriveLetter E -FileSystem NTFS -Confirm:$false
# this creates a folder on the drive to use as the workspace by the agent
New-Item -ItemType Directory -Force -Path E:\jenkins\workspace
}
There are some assumptions here about the number of disks, and that will vary based on the instance type, but generally it will take any ephemeral disks it finds, if there is more than one it will stripe across them, and then uses the entire disk size available once the volume has been created/formatted. This can all be wrapped in <powershell></powershell> and added to the user data section of the Jenkins agent config so that it is run at boot.
As of a few weeks ago we added filebeat, metricbeat and apm to our dotnet core application ran on our kubernetes cluster.
It works all nice and recently we discovered filebeat and metricbeat are able to write a different index upon several rules.
We wanted to do the same for APM, however searching the documentation we can't find any option to set the name of the index to write to.
Is this even possible, and if yes how is it configured?
I also tried finding the current name apm-* within the codebase but couldn't find any matches upon configuring it.
The problem which we'd like to fix is that every space in kibana gets to see the apm metrics of every application. Certain applications shouldn't be within this space so therefore i thought a new apm-application-* index would do the trick...
Edit
Since it shouldn't be configured on the agent but instead in the cloud service console. I'm having troubles to 'user-override' the settings to my likings.
The rules i want to have:
When an application does not live inside the kubernetes namespace default OR kube-system write to an index called apm-7.8.0-application-type-2020-07
All other applications in other namespaces should remain in the default indices
I see you can add output.elasticsearch.indices to make this happen: Array of index selector rules supporting conditionals and formatted string.
I tried this by copying the same i had for metricbeat and updated it to use the apm syntax and came to the following 'user-override'
output.elasticsearch.indices:
- index: 'apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}'
when:
not:
or:
- equals:
kubernetes.namespace: default
- equals:
kubernetes.namespace: kube-system
but when i use this setup it tells me:
Your changes cannot be applied
'output.elasticsearch.indices.when': is not allowed
Set output.elasticsearch.indices.0.index to apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}
Set output.elasticsearch.indices.0.when.not.or.0.equals.kubernetes.namespace to default
Set output.elasticsearch.indices.0.when.not.or.1.equals.kubernetes.namespace to kube-system
Then i updated the example but came to the same conclusion as it was not valid either..
In your ES Cloud console, you need to Edit the cluster configuration, scroll to the APM section and then click "User override settings". In there you can override the target index by adding the following property:
output.elasticsearch.index: "apm-application-%{[observer.version]}-{type}-%{+yyyy.MM.dd}"
Note that if you change this setting, you also need to modify the corresponding index template to match the new index name.
I need to gather a list of all mounted "mount points" that the local file system has access to.
This includes:
Any ordinarily mounted volume under /Volumes.
Any NFS volume that's currently mounted under /net.
Any local or remote file system mounted with the "mount" command or auto-mounted somehow.
But I need to avoid accessing any file systems that can be auto-mounted but are currently not mounted. I.e, I do not want to cause any auto-amounting.
My current method is as follows:
Call FSGetVolumeInfo() in a loop to gather all known volumes. This will give me all local drives under /Volumes as well as /net, /home, and NFS mounts under /net.
Call FSGetVolumeParms() to get each volume's "device ID" (this turns out to be the mount path for network volumes).
If the ID is a POSIX path (i.e. it's starting with "/"), I use readdir() on its path's parent to check whether the parent dir contains actually the mount point item (e.g. if ID is /net/MyNetShare, then I readdir /net). If it's not available, I assume this is a auto-mount point with a yet-unmounted volume and therefore exclude it from my list of mounted volumes.
Lastly, if the volume appears mounted, I check if it contains any items. If it does, I add it to my list.
Step 3 is necessary to see whether the path is actually mounted. If I'd instead call lstat() on the full path, it would attempt to automount the file system, which I need to avoid.
Now, even though the above works most of the time, there are still some issues:
The mix of calls to the BSD and Carbon APIs, along with special casing the "device ID" value, is rather unclean.
The FSGetVolumeInfo() call gives me mount points such as "/net" and "/home" even though these do not seem to be actual mount points - the mount points would rather appear inside these. For example, if I'd mount a NFS share at "/net/MyNFSVolume", I'd gather both a "/net" point and a "/net/MyNFSVolume", but the "/net" point is no actual volume.
Worst of all, sometimes the above process still causes active attempts to contact the off-line server, leading to long timeouts.
So, who can show me a better way to find all the actually mounted volumes?
By using the BSD level function getattrlist(), asking for the ATTR_DIR_MOUNTSTATUS attribute, one can test the DIR_MNTSTATUS_TRIGGER flag.
This flag seems to be only set when an automounted share point is currently unreachable. The status of this flag appears to be directly related to the mount status maintained by the automountd daemon that manages re-mounting such mount points: As long as automountd reports that a mount point isn't available, due to the server not responding, the "trigger" flag is set.
Note, however, that this status is not immediately set once a network share becomes inaccessible. Consider this scenario:
The file /etc/auto_master has this line added at the end:
/- auto_mymounts
The file /etc/auto_mymounts has the following content:
/mymounts/MYSERVER1 -nfs,soft,bg,intr,net myserver1:/
This means that there will be a auto-mounting directory at /mymounts/MYSERVER1, giving access to the root of myserver1's exported NFS share.
Let's assume the server is initially reachable. Then we can browse the directory at /mymounts/MYSERVER1, and the DIR_MNTSTATUS_TRIGGER flag will be cleared.
Next, let's make the server become unreachable by simply killing the network connection (such as removing the ethernet cable to turning off Wi-Fi). At this point, when trying to access /mymounts/MYSERVER1 again, we'll get delays and timeouts, and we might even get seemingly valid results such as non-empty directory listings despite the unavailable server. The DIR_MNTSTATUS_TRIGGER flag will remain cleared at this point.
Now put the computer to sleep and wake it up again. At this point, automountd tries to reconnect all auto-mounted volumes again. It will notice that the server is offline and put the mount point into "trigger" state. Now the DIR_MNTSTATUS_TRIGGER flag will be set as desired.
So, while this trigger flag is not the perfect indicator to tell when the remote server is unreachable, it's good enough to tell when the server has become offline for a longer time, as it's usually happening when moving the client computer between different networks, such as between work and home, with the computer being put to sleep in between, thus causing the automountd daemon to detect the reachability of the NFS server.
I attached an EBS volume to my EC2 instance, turned it into a EXT3 file system, and successfully mounted it. However, I was thrown off a little bit initially mainly due to what the AWS console said my EBS device ID was.
According to the AWS console:
i-xxxxxxx :/dev/sdf (attached)
I took this to mean that my attached EBS device id was /dev/sdf. So when I attempted to turn the device into a file system using this device id, I received the following error message.
ubuntu#ip-xx-xx-xx-xx:~$ mkfs -t ext3 /dev/sdf
mke2fs 1.42 (29-Nov-2011)
Could not stat /dev/sdf --- No such file or directory
The device apparently does not exist; did you specify it correctly?
Then after researching a little bit, I found this article and then subsequently found out by running cat /proc/partitions that my true device id was /dev/xvdf not /dev/sdf.
My question is why is the AWS console saying it's /dev/sdf when it's actually /dev/xvdf? I think there has to be some kind of logical explanation for this.
AWS provides the following message/warning when attaching a volume via the AWS Management Console:
Note: Newer linux kernels may rename your devices to /dev/xvdf through
/dev/xvdp internally, even when the device name entered here (and
shown in the details) is /dev/sdf through /dev/sdp.
I don't have any upstream source for this information handy, but Jay Rum's answer to the (not longer relevant) temporary problem EBS Disks starting as device /dev/xvde, but mapped as /dev/sda attributes this functionality to the xen-blkfront driver:
The "xen-blkfront" driver, which allows the virtual machines (i.e. the
EC2 instances) to access the underlying block devices, traditionally
mapped sda, sdb... to xvda, xvdb..., [...]
Finally, cyberx86' answer to How do I access the attached volume in Amazon EC2 provides a detailed and illustrated explanation of this device naming mismatch and how to deal with it, i.e. identify the currently available devices etc.
Note: This question had already been answered in Aug 24 '12, but that answer with 6 upvotes got deleted in May 01 '13 by the Community Moderator (i.e. a resp. automation process) for non transparent reasons (apparently because the User was removed) - anyway, I've added back a slight variation of the original content from my point of view.