To achieve this you need to create a custom attribute for bestsellers and enable it for using in Sorting options: catalog > attributes > manage attributes
Product with values set in the custom attributes will be shown on the top of the list, while the remaining products will be sorted based on actual data. Please see more details on how to create custom attributes for bestsellers and most viewed options in our User Guide on Product Sorting.
Yes, this can be configured in the Category Display Settings. Please navigate to Catalog > Manage Categories, select necessary Category from the category tree, open the Display Settings tab, uncheck the ‘Use Config Settings’ checkbox and choose the sorting option you’d like to set up as default for this category in the ‘Default Product Listing Sort By’ field:
Note that configurable and bundle products are composite and their quantity is 0. That is why sorting by quantity is not supported for them. The option ‘Use qty to determine if the product is `Out of Stock`’ doesn’t work for configurable and bundle product types as well for the same reason.
In case if sorting doesn't work there might be a conflict with other extensions. We have worked out a special tool for the case. Go to System>Configuration>Amasty Extensions>Extensions&Notifications>Troubleshooter, then find the Improved Sorting extension and click the 'Fix' button. Providing that the issue hasn't been solved, send a request ticket to our support team.