Can I set payment restrictions on a product level?

Yes, sure. You can use a SKU product attribute in the conditions. Product attributes can be found under the “Product Subselection” condition.

Note that you need to set quantity to “equals or greater than 1” and SKU to “is one of”: