After a car accident that was settled on the spot, could the other driver take any kind of legal action against me?

I was in bumper to bumper traffic yesterday, lost control of my footing and rear-ended another car. The guy and I both pull over to check for damages. He expressed he did not want to get our insurances involved in this. So we both agree not to exchange insurance information and there was no police report and in returned we agree to an amount of money that would take care of the minor damage to his bumper. After I paid the fee on the spot we both went on our ways. The question is, could he turn around, change his mind and seek some kind of legal action against me?