Objectively quantifying the classification accuracy of Intrusion Detection Systems (IDSs) is of fundamental importance. Evaluation metrics have been proposed to measure the effectiveness of traditional IDSs, but none of those metrics seems suitable to evaluate the distributed collaborative IDSs that are generally employed in Wireless Sensor Networks (WSNs). This is because in WSNs each IDS output (i.e., alarm or absence of alarm) results from a consensus decision among several nodes, as opposed to an individual decision of a single node. In this paper, we present the trust probability (Pt) metric, which is defined as the probability that an IDS draws the right conclusion in its collaborative decision-making process. This metric is computed based on the properties of the individual nodes that contribute to the IDS global conclusions. We provide numerical examples as well as a detailed analysis of Pt. Moreover, we show how Pt can be used to find the best operating point of a given IDS and to compare different IDSs. Finally, since Pt is a measure of how much each IDS global output should be trusted, we discuss how this metric can be used in real-time to rank IDS alerts.