Azure DevOps build pipeline self-hosted agent "No space left on device" - bash

Question

I am running a build pipeline on Azure that runs on a private build server (Red Hat Enterprise Linux) running Self Hosted Agent. This build pipeline only has 1 Job and 2 Tasks where the 1st task is it basically SSH's into a Repo server we have (different server that just holds big files) generates an ISO image on that Repo server, then uses curl to put that ISO back on the the build server where the Azure Pipeline agent is running in the stereotypical $(Build.ArtifactStagingDirectory) Azure uses for Artifacts.
This 1st task succeeds, and the ISO is generated and copied over to the build server, but the "Publish Artifact" stage keeps failing. It's trying to publish to the path $(Build.ArtifactStagingDirectory) but produces an error message, with more logs:
No space left on device
I already went in a cleared all the directories and files that exceeded > 1GB in this working directory `/home/azure/vsts/_work
I'm not an expert with Linux. When I run df -h and view the filesystem, there are a bunch in the list. Is there a way to know what partition I'm actually using for this Azure pipeline agent that using the /home/azure/vsts/_work directory?
My df -h list looks like:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 19G 19G 28K 100% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 8.0K 3.9G 1% /dev/shm
tmpfs 3.9G 138M 3.8G 4% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdc 100G 152M 100G 1% /glusterfs
/dev/sda1 488M 119M 334M 27% /boot
/dev/mapper/vg_root-lv_var 997M 106M 891M 11% /var
/dev/mapper/vg_docker-lv_docker 50G 3.1G 44G 7% /var/lib/docker/overlay2
/dev/mapper/vg_root-lv_log 997M 46M 952M 5% /var/log
/dev/mapper/vg_root-lv_crash 997M 33M 965M 4% /var/crash
/dev/mapper/vg_root-lv_root_logins 29M 1.8M 27M 6% /var/log/root_logins
/dev/mapper/vg_root-lv_core 125M 6.6M 119M 6% /var/core
/dev/mapper/vg_root-lv_repo 997M 83M 915M 9% /var/cache/yum
/dev/mapper/vg_root-lv_home 997M 33M 965M 4% /export/home
/dev/mapper/vg_root-lv_logins 93M 5.0M 88M 6% /var/log/logins
/dev/mapper/vg_root-lv_audit 725M 71M 655M 10% /var/log/audit
tmpfs 799M 0 799M 0% /run/user/0
walkie1-ap2.nextgen.com:/hdd-volume0 200G 2.3G 198G 2% /gluster-hdd
If anyone could provide some insight I'd greatly appreciate it.
End of error log:
[2020-05-06 05:49:09Z ERR JobRunner] Caught exception from job steps StepsRunner: System.IO.IOException: No space left on device
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode)
at Microsoft.VisualStudio.Services.Agent.PagingLogger.NewPage()
at Microsoft.VisualStudio.Services.Agent.PagingLogger.Write(String message)
at Microsoft.VisualStudio.Services.Agent.Worker.ExecutionContext.Write(String tag, String message)
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunAsync(IExecutionContext jobContext, IList`1 steps)
at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken)```

Answer 1

I can't reproduce same issue on my side, but I think you can check this article for a trouble-shooting.
As I know this task itself will take extra space when being executed. You can try a bash command to make copy of content under path $(Build.ArtifactStagingDirectory) to double the size of those content, if this action throws same No space left on device error?
And in build pipeline there's one clean option to clean the caches before executing the job, enable it to check if it helps:
If it's yaml pipeline, try something like:
workspace:
clean: outputs | resources | all # what to clean up before the job runs
and
steps:
- checkout: self | none | repository name # self represents the repo where the initial Pipelines YAML file was found
clean: boolean # if true, run `execute git clean -ffdx && git reset --hard HEAD` before fetching
See Yaml schema.

Answer 2

So I was a n00b here, and the solution was simply to clean out space from the main directory we used to store our large ISO files:
/dev/mapper/vg_root-lv_root 19G 19G 28K 100% /
This is a custom VM we used to run builds on Azure and I wasn't accustomed to the error message. But yes, if anyone says this message and is using a custom build agent, its definitely a space issue.