How near does a black hole has to be to affect our planet?

Well it entirely depends upon its mass.
In case of the black hole that was just imaged:
https://xkcd.com/2135/

There is a black hole in the center of the Milky Way galaxy. It is holding the Milky Way galaxy together and as search is affecting the Earth.

The word “affect” is too nebulous. What “affect” are you looking for? That black hole in the middle determines our galaxy’s center of mass? The Milky Way rotates around that mass and we’re in for the ride. How’s that for an affect? I hate questions like this that make my head race. That huge mass can grab a double star system, suck one star up and hurl the other outward at extragalactic velocities. So a black hole can send things our way capable of disrupting the oort cloud or kuiper belt effectively raining junk down into the inner solar system. A black hole transiting our neighborhood of stars is going to effect the sun first, and we might well bear many nasty consequences from that. But the vagaries possible with this sort of question kind of drive me crazy.

^^ Are you nuts? You just answered the question. Relax!