Every day, security engineers cope with a flow of cyber security incidents. While most incidents trigger routine reactions, others require orders of magnitude more effort to investigate and resolve. How security operation teams in organizations should tune their response to tame extreme events remains unclear. Analyzing the statistical properties of sixty thousand security events collected over six years at a large organization, we find that the distribution of costs induced by security incidents is in general highly skewed, following a power law tail distribution. However, this distribution of incident severity becomes less skewed over time, suggesting that the organization under scrutiny has managed to reduce the impact of large events. We illustrate this result with a case study focused on the empirical effects of full disk encryption on the severity of incidents involving lost or stolen devices.