In R/factoextra, when I run fviz_nbclust() it's show cannot allocate vector of size.... Is there any other package or way can solver it ? Thanks!
library(factoextra)
fviz_nbclust(kmeans_data_scaled,kmeans, method = "wss")
Error: cannot allocate vector of size 165.4 Gb
Related
I'm trying to use the VK_EXT_external_memory_host extension https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/VK_EXT_external_memory_host.html. I'm not sure what the difference is between vk::ExternalMemoryHandleTypeFlagBits::eHostAllocationEXT and eHostMappedForeignMemoryEXT but I've been failing to get either to work. (I'm using VulkanHpp).
void* data_ptr = getTorchDataPtr();
uint32_t MEMORY_TYPE_INDEX;
auto EXTERNAL_MEMORY_TYPE = vk::ExternalMemoryHandleTypeFlagBits::eHostAllocationEXT;
// or vk::ExternalMemoryHandleTypeFlagBits::eHostMappedForeignMemoryEXT;
vk::MemoryAllocateInfo memoryAllocateInfo(SIZE_BYTES, MEMORY_TYPE_INDEX);
vk::ImportMemoryHostPointerInfoEXT importMemoryHostPointerInfoEXT(
MEMORY_FLAG,
data_ptr);
memoryAllocateInfo.pNext = &importMemoryHostPointerInfoEXT;
vk::raii::DeviceMemory deviceMemory( device, memoryAllocateInfo );
I'm getting Result::eErrorOutOfDeviceMemory when the constructor of DeviceMemory calls vkAllocateMemory if EXTERNAL_MEMORY_TYPE = eHostAllocationEXT and zeros in the memory if EXTERNAL_MEMORY_TYPE = eHostMappedForeignMemoryEXT (I've checked the py/libtorch tensor I'm importing is non-zero, and that my code successfully copies and readbacks a different buffer).
All values of MEMORY_TYPE_INDEX produce the same behaviour (except when MEMORY_TYPE_INDEX overflows).
The set bits of the bitmask returned by getMemoryHostPointerPropertiesEXT is suppose to give the valid values for MEMORY_TYPE_INDEX.
auto pointerProperties = device.getMemoryHostPointerPropertiesEXT(
EXTERNAL_MEMORY_TYPE,
data_ptr);
std::cout << "memoryTypeBits " << std::bitset<32>(pointerProperties.memoryTypeBits) << std::endl;
}
But if EXTERNAL_MEMORY_TYPE = eHostMappedForeignMemoryEXT then vkGetMemoryHostPointerPropertiesEXT returns Result::eErrorInitializationFailed, and if EXTERNAL_MEMORY_TYPE = eHostAllocationEXT, then the 8th and 9th bits are set. But this is the same regardless of whether data_ptr is a cuda pointer 0x7ffecf400000 or a cpu pointer 0x2be7c80 so I'm feeling something has gone wrong.
I'm also unable to get the extension VK_KHR_external_memory_capabilities which is required by VK_KHR_external_memory which is a requirement of the extension we are using VK_EXT_external_memory_host. I'm using vulkan version 1.2.162.0.
The eErrorOutOfDeviceMemory is strange as we are not supposed to be allocating any memory, I'd be glad if someone could speculate about this.
I believe that host memory is cpu memory, thus:
vk::ExternalMemoryHandleTypeFlagBits::eHostAllocationEXT wont work because the pointer is to device memory (gpu).
vk::ExternalMemoryHandleTypeFlagBits::eHostMappedForeignMemoryEXT wont work because the memory is not mapped by the host (cpu).
Is there anyway to import local device memory into vulkan? does it have to be host mapped?
Probably not https://stackoverflow.com/a/54801938/11998382.
I think the best option, for me, is to map some vulkan memory, and copy the pytorch cpu tensor across. The same data would be uploaded to the gpu twice but this doesn't really matter I suppose.
my projec used the buffer both in GPU and CPU.
code are similar like this:
xxx = HalideBuffer_n(width, heght, 1); // buffer in CPU
xxx.device_malloc(device); // buffer in GPU
PS: using HalideBuffer_n = Halide::Runtime::Buffer<uint8_t, 3>
i can use
xxx.device_free();
to free the buffer in GPU.
but which API can i choose to free the buffer in CPU？
i have used such as free or deallocate, but it seems dont work.
A Halide::Buffer is a shared pointer to the underlying allocation. xxx.deallocate() drops the reference to it. If this is the only copy of that HalideBuffer_n object then it will free the underlying memory too. If it's not freeing it means that another copy of that HalideBuffer_n object exists somewhere.
I tried to mount a parition with block size larger than 4kb.
the operation failed with the following error:
Function not implemented
(I tried enabling Huge Pages, got the same error).
After some research I found that this is probably due to the blocksize being larger than the os page size.
As I understand the page size is determined at compile time,
so I'd like to try compile a kernel with different page size.
You can use these kernel configurations from make menuconfig and change the page size.
CONFIG_<ARCH>_4K_PAGES
CONFIG_<ARCH>_16K_PAGES
CONFIG_<ARCH>_64K_PAGES
I just tried adding debugging information in kmalloc returned pointer but after my changes my kernel is not coming up. i have changes as follows :-
Suppose some module request a memory of x size , then in kmalloc i increased the size with (x+ debug_information).
At the beginning of memory allocated pointer I have stored following information
calling function address(4 Byte),size(4byte), and header(byte) and at end memory size one footer was added and address returned to calling module is (ptr +12).
Now while free memory using kfree , i am decreasing pointer address by 12 bytes (ptr -12).
Now after doing these changes my kernel is not coming up.
Even i done simple code changes like increase requested memory of size (requested_size +4), return pointer to module of ptr +4,
while freeing memory decrease this pointer with 4, now this also not working .
What could be possible issue, why we can not add debug information before and after kmalloc returned pointer ?
I am trying to allocate memory
NSImage*originalLocationImage;
NSURL *fileURL = [NSURL fileURLWithPath:originalLocation]; //originalLocation is file path in my disk
originalLocationImage = [[NSImage alloc]initByReferencingURL:fileURL];
NSBitmapImageRep *sourceRep = [[NSBitmapImageRep alloc]initWithData:[SourceImage TIFFRepresentation]] //this line fails to allocate memory
and it fails for large Images/
and throws error
malloc: *** mmap(size=268435456) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
for small images it works properly.
any help??
Using libjpeg might be an option, it seems to be able to do an optimized sub-scaling up to 16: http://jpegclub.org/djpeg/.
libjpeg is plain C, so you can call it from Objective-C.
I'd question why you want to load all of the image into memory in the first place - surely dealing with it like a tile set would be a better option?
You are getting this message because, whilst you might have this much memory free, you don't have this much contiguous memory free
If all you want is a preview, you could read it line by line, skipping every n lines and for each line, skip every n pixels. You're left with a (way) smaller image that you can pass through more sophisticated scaling algorithms if necessary, or you can just save it as your thumbnail image or whatever.
If you only want to create a preview of the image, CGImageSource has APIs specifcally for that. They are very first and don't require reading the entire image into memory.