Henry Jenkins states in his essay 'Game Design as Narrative Architecture' that games are experience spaces. Game spaces are more important to him than the narrative itself. He thinks that creating the stage for the narrative is more important than the story that will be told. Many games tell their story through exploration of the game space which makes the environment itself indispensable.
According to Jenkins games need a well designed, thoughtful world environment to deliver the story.
Personally, I really like his statement, cause a well designed game world will help underline the story and how we experience it. Really many games, as already mentioned, are driven by exploring the world and by that discovering the storyline. It can make us take detours so we get more out of the narrative or take short cuts to discover faster, because we can't wait for the next thing to happen.
As a live rpg player I know I can imagine these game worlds in my head. They will only ever be as good as my imaginenation goes, and maybe never be as good as the author imagined. So having a graphical world I can explore will help me take in the story as it was imagined.
On another note a graphical world will never turn a horrible story into a grand one. It does only so much to help deliver it. If the story sucks, it sucks. So designing a world to help your story is great, depending on it to make the story isn't too great.
As everything in life, there has to be a balance between both.