Can I completely remove the Add to cart button for all user groups for specific products?

Yes, on the corresponding Product Information tab you will be able to choose whether Hide Price should be enabled, and if yes, for which customer groups it should work.

Magento Hide Price by Amasty

See more details on Hide Price page
Rate the answer?