Then today I realized I was relying on garbage collection far too much and realizing it wasn't collecting things as quickly/efficiently as I had liked, but most of that was my fault.
The way I was generating the world was creating far too many objects that weren't being used after the world was generated. I was seeing memory use easily climb towards 1.5GB on world creation. When you loaded a saved world the memory would be about 512MB, which is closer to what I wanted, but generation would climb far too high. I swapped my unnecessary cloned objects to references to the base objects and world generation sticks at the same ~512MB the game uses to run.
I only noticed this because I realized that I had put my save on a separate thread, so I thought, "Hey, seeing as how it is on it's own thread, and saving takes a while, why not make a copy of the world, and let the player keep playing while it's saving". While good in theory, making a copy of the world was doubling the memory use, and would freak out lower memory systems, especially on the larger world size (it would push something like 3GB of memory, not to mention this is a 32-bit app so it would freak on any system after 2GB of usage).
With the smaller map size, which is still quite large, and getting rid of unnecessary objects I've kept the memory usage down to appropriate levels. Now I can go back to focusing on gameplay elements.