Yes, or you could hide categories based on product count. That will require changes to your "Categories" widgets.
To get the number of products in a specific category, you can use the PHP variable:
To get the total number of products in a category and its subcategories, you can use the PHP variable:
Then perform some conditional logic in your views to display or not display the categories.