I think a lot of these answers really touch on the truth. Here’s my theory, and I’m not in any way a scientist or a student of this phenomenon, but here’s my two cents nonetheless. Your brain uses dreams to sort things out…your brain is basically an organic supercomputer and what it’s really doing when it dreams is to calculate things…kind of like when you shut down your computer and it installs the updates. If you think of your brain like a computer, think of it like the computer update or processing something to give you a result.
In the case of your computer, when you shut it down, when you come back to it, what you want to see is the computer updated with the latest functionality. Or if you are having it calculate something, when it’s done, what you are interested in, what has meaning to you is the result. Now, your computer (as well as your brain while dreaming) made millions of calculations, there is a great deal of information that your computer had internally that you never get to see, nor do you really have any interest in seeing…it has no value to you how the computer got from point a to point b, you just need to operate at point b from now on.
Your brain is the same thing…these dreams are its internal processors going through millions of permutations, working things out. When you wake up with a fresh perspective on something, what you care about is that fresh perspective, you don’t really care how you got from where you were yesterday to where you were today, well, at least not in any way that couldn’t be described as “trivial”...the actual permutations have no meaning and no use in your conscious mind, and therefore they remain relegated to the processors, i.e. your unconscious mind.
But because the brain is part of you, you can’t help but retain some of that awareness of the unconscious mind’s workings on waking up. It’s kind of like the restarting process for your computer…you see the title screen, some of that is still there, you are still coming out of a sleep state and therefore you still have the ability to bring some of your unconscious mind forth. If you are passive about it, it recedes and probably all that data is like RAM, it’s thrown out the next time your mind has to make some calculations in the unconscious. But, if you take ACTION, by accessing that data, and writing it down, then it’s kind of like copying the files from your unconscious mind, where the cache will be cleared, and your conscious mind where everything sticks around. It’s only the dreams that have a real impact that you spend a lot of your conscious mind thinking about that tend to stick with you for the long haul…writing them down accomplishes that data transfer.
I could be wrong….