Here is an interesting passage from Etymology online, The American Civil War
If you peel back this slave-holder vs. anti-slavery fight over Western territories, you again meet a clash of economic interests. Why did the North fight so hard to prevent slaveholders being allowed to carry their institution into Missouri or Kansas or other western territories? I’d answer that question by asking another one: Why would a textile mill worker in Lancashire, England, undertake the expense and hazard of a voyage to America, only to go to work on the same mule, for the same wage, in an American mill? Because cheap land in the western territories offered him the prospect of making just enough to quit the factory and set himself up as a farmer.
Hamilton, in his 1791 “Report on Manufactures,” anticipated this: “Many, whom Manufacturing views would induce to emigrate, would afterwards yield to the temptations, which the particular situation of this Country holds out to Agricultural pursuits.”
That’s why New England mill owners resisted the expansion of slave labor and cotton plantations into the Louisiana Purchase territories and the land acquired from Mexico. Immigrants didn’t flock to Alabama and Mississippi, because the plantation system that had been created there didn’t provide the lure of cheap land for family farms. If Kansas and Nebraska had turned into Alabama and Mississippi, that would have cut off an essential inducement to immigrants, who gave the Northern factories cheap labor.
In reality, the lure was usually just a tease. A fraction of the immigrants to Northern mill towns eventually made it west and set up as farmers. The rest cycled from one row of tenaments to another, from Lowell to Montreal to Pittsburgh to Albany, dragging families and debt with them.
The wrangling over slavery in the territories, like the tariff, was part of the bigger picture of one region trying to break out of the original partnership compact and impose its will, its might, and its values on the whole of America. This seems a natural development to us, now, but only because it has been so effectively done. What comes to pass always seems foreordained.
And prejudice was not exclusively a Southern attribute. It is also worth remembering that De Tocqueville, French political thinker and historian best known for his works Democracy in America, noted that “race prejudice seems stronger in those states that have abolished slavery than in those where it still exists, and nowhere is it more intolerant than in those states where slavery was never known.”