Can I completely remove the Add to cart button for all user groups for specific products?

Yes, on the corresponding Product Information tab you will be able to choose whether Hide Price should be enabled, and if yes, for which customer groups it should work.