One of the few things Apple got badly wrong in the original Macintosh forty years ago was memory. It came with just 128 KB of RAM, barely sufficient for demonstration purposes, and by the autumn/fall of 1984 had been replaced by the 'Fat Mac' with 512 KB costing an extra $700. Apple proved quick to demonstrate that memory for its new Mac products was never going to come cheap.
Until the release of System 7 in May 1991, Macs couldn't use virtual memory when running Mac OS, and memory management was primitive to say the least. Even in System 8.6 eight years later, virtual memory remained optional. Some apps required it, while others couldn't run when it was enabled. Most users stuck with only enabling it when their software needed it, and made do with the limitations of physical memory of 384 MB or even less. The maximum my Blue & White Power Mac G3/350 could accommodate was just 1 GB. As apps were far more conservative in their memory requirements, this worked better than you might expect.
My Power Mac G3 worked well with Mac OS taking its lion's share of just over 50 MB, my mail client with less than 7 MB, and the whole of Microsoft Word in under 20 MB. But apps could and did run out of memory, when they would simply quit with an error alert.
Memory leaks still plagued Mac OS 8, and many users had to resort to utilities like R Fronabarger's freeware Memory Mapper to track free memory and try to understand what was going on.
One memory problem never fixed in Mac OS 8.x occurred in many apps including Web browsers, Microsoft Office 98, and others. Using these led to a progressive reduction in the amount of contiguous free memory, until eventually the whole Mac crashed. This appeared worst in Macs with most physical memory, and although some patches were produced by third-parties, none was a complete solution. The only workaround was to keep an eye on memory, and restart the Mac before crashing became likely.
In those days, you had to set the amount of memory to be allocated to each app in the Finder's Get Info dialog. Getting this right was usually a matter of trial and error.
Mac OS X was completely different, with virtual memory a permanent feature, and greatly improved management by the kernel. But memory leaks continued, and we learned the pain brought by those in Mach zone memory, memory blocks allocated for use by the kernel and its extensions. That happened as recently as macOS Catalina 10.15.6, resulting in kernel panics. Those were preceded by the kernel complaining of zone_map_exhaustion
in the log:
03:21:09.447981+0100 kernel zone_map_exhaustion: Zone map size 12240662528, capacity 12884901888 [jetsam limit 95%]
03:21:09.448533+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6544393440
03:21:09.449437+0100 kernel kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.
Memory leaks, fortunately not affecting Mach zones this time, also troubled macOS 12.0.1.
With the advent of Apple silicon Macs came the greatest change in memory management and use since the release of Mac OS X twenty years earlier: instead of having separate physical memory for devices like GPUs, M-series chips use Unified Memory, one pool for use by CPU cores, GPU, and much else apart from the Secure Enclave.
Although it's now over 23 years since Mac OS X was first fully released, take a look in the App Store at the products claiming to 'clean up' or somehow manage memory, as if we were still in the days of Classic Mac OS. If anyone can explain to me how those apps aren't making fraudulent claims, I'd be very grateful, as I stopped trying to manage my Mac's memory when I switched from Mac OS 9 to Mac OS X, and haven't looked back.