What is the origin of the slur "nuts"?

It’s closely related to the term ‘nuthouse’ = mental hospital

From the OED:

The earliest known use of the noun nuthouse is in the 1900s.

OED’s earliest evidence for nuthouse is from 1906, in the writing of Chauncey M’Govern.

As for Nuts:

Where does the adjective nuts come from?
Earliest known use – late 1700s

The earliest known use of the adjective nuts is in the late 1700s.

OED’s earliest evidence for nuts is from 1785, in a dictionary by Francis Grose, antiquary.

^^^ but that doesn’t explain the logic behind the comparison.

I found this:
“Nut is slang for head. And nut case means head case; i.e, mind/brain injury/illness”
If you look at a cracked walnut you can see it resembles a brain cross section.

There are two main theories:

1. It is derived from a shortening of the phrase “off one’s nut,” which itself comes from the association between heads and nuts that @anniereborn mentioned. To be off one’s nut is to be acting foolish or irrational, which are also symptoms that people tend to associate—rightly or wrongly—with mental illness.

2. It is derived from the phrase “to be nuts about.” To be nuts about something means to be extremely enthusiastic about it and always have it in your head. But excess enthusiasm can come off as extreme and obsessive, which is why the other version of the phrase is “to be crazy about.” And since the only difference between the two phrases is the substitution of “crazy” for “nuts,” the two came to be seen as interchangeable.

Unfortunately, there’s no definitive proof for either theory.

Good to see you @CunningLinguist!