The answer is yes, it does matter, each and every plugin you use has to be processed before an actual page is generated, so all the plugin files are compiled in order to get the intended output.

I’ve been working with a plugin that was mentioned on the wp-hackers list last week, called debug queries, to optimize the performance on some of my sites, including this one.

…

“Heavy” plugins
It turns out, that of the 16 queries my site now still does for a single post page, 7 are for the related posts plugin I’m using, YARPP, and if you’re wondering: yes that’s making me reconsider the plugin. I’m going to try a couple [of] other plugins and see how good they do in both relevance and (database) performance.