Tyler Cowen argued in the New York Times that fear of international trade is basically irrational in-group loyalty, which implies a fear of outsiders, and thus, trade. He argues these fears are unfounded, because the effects of trade with outsiders is no different than the effects of productivity increases at home. Dani Rodrik responded with the argument that people really fear competing with those who play 'unfairly', giving the example that if he lost his job to a psychologist who was allowed to use torture to elicit preferences, this he feels, would be unfair, even if this evil academic produced valuable work. More concretely, Americans don't like competing with countries where they have fewer regulations, less worker safety, environmental protection, etc.
I think you can dismiss Rodrick's argument by noting that Americans are just as wary of imports from Canada, Japan and Europe, as they are from imports from Mexico, China, and Africa. Cowen alludes to this in his rejoiner.
I think it's mainly the Buchanan targeted benefits-diffuse costs story. The issue is not so much Americans are against free trade, but at the margin, every import has a strong group of supporters for quotas or tariffs, and a broad group against such quotas and tariffs. The logic implies that the people who give money, lobby, and vote on this issue are always against tariffs. General arguments on the principle of free trade are to abstract for most voters.