What's the best way to migrate my files to my new computer?

I have a mac and could use migration assistant, but I’m a bit reticent, because I’d have to migrate the whole user, and isn’t one of the benefits of getting a new machine the fact that you get rid of all the crap that builds up on your old one? Why would I want to move that all over? Then again, I don’t want to leave anything critical behind if I do the files manually using an external drive. What will I be forgetting to move if I do the manual route?