I'm guessing that you were checking and unchecking merchants on the Store Merchants page... This would be incorrect.
To specify merchants on a category basis, here's what you should do:
1. Go here: The Factory > Your Store > Store menu > Store Merchants
2. Check all merchants and click save
3. Go to the category which you want to only search from a few merchants
4. Click the add products to this category link
5. In the search form, add a new filter and choose merchants from the drop down list. Then only check the merchants you want to search in your search form.
That should do it.