Out-of-core: swap vs plamsa object store

Hi there, it is exciting to see modin have out-of-core support. While reading through the implementations, I found that this is implemented using Ray’s plasma store feature.

I’m curious about the difference between swap memory and plasma store: it seems to me that when the main memory is not sufficient for holding the data, the OS will also move some memory pages from the main memory to the disk. So, why using plasma proactively move objects to disk is better than leveraging the swap mechanism of the OS?

Hi @Weiyuan_Wu, thanks for posting!

Effectively, the Ray plasma store is using the OS buffer cache paging mechanism to exceed memory. In the very near future, Ray is going to have an API for out of core to push objects to S3 or perhaps some other store. We are excited about this feature coming soon from Ray.