I left the game for a bit and when I came back I was in a different place!

When you die, you respawn by default at Vieda Port. You can change this option
in the options tab.








But I was in a town!

Not all towns are completely safe :)








I found a key item, but the Crystal Sphere icon hasn't changed.

Some areas have more than one key item. Also, you may not have the required
skill levels to get that item so don't worry if you left the game on overnight
and the icon is still there.








Is it possible for me to ruin my build by making the wrong decisions?

Not really, no. It is possible to level up unoptimally in some cases and have
less stat points compared to other people, but the level cap is so far beyond
the end of the game content that it doesn't matter.