Ys IV and V got bigger, then they scaled it down again for Ys IV.
You mean Ys VI was scaled down, I assume. That's probably because it was their first original Ys game on 3D capable hardware. It's sort of like how TES II: Daggerfall is way bigger than TES III: Morrowind. They didn't want another sprite-based game, so they had to downsize the world.
"Remember kid, there's heroes and there's legends. Heroes get remembered, but legends never die. Follow your heart kid, and you'll never go wrong."