Photon-counting energy resolving detectors are subject to intense research interest, and there is a need for a general framework for performance assessment of these detectors. The commonly used linear-systems theory framework, which measures detector performance in terms of noise-equivalent quanta (NEQ) and detective quantum efficiency (DQE) is widely used for characterizing conventional X-ray detectors but does not take energy-resolving capabilities into account. We extend this framework to encompass energy-resolving photon-counting detectors and elucidate how the imperfect energy response of real-world detectors affects imaging performance. We generalize NEQ and DQE to matrix-valued quantities as functions of spatial frequency, and show how these can be calculated from simple Monte Carlo simulations. To demonstrate how the new metrics can be interpreted, we compute them for simplified models of fluorescence and Compton scatter in a photon-counting detector and for a Monte Carlo model of a CdTe detector with 0.5 x 0.5 mm^2 pixels. Our results show that the ideal-linear-observer performance for any detection or material quantification task can be calculated from the proposed metrics. Off-diagonal elements in these matrices are shown to be related to imperfect energy resolution. The Monte Carlo model of the CdTe detector predicts a zero-frequency dose efficiency relative to an ideal detector of 0.86 and 0.65 for detecting water and bone, respectively. When the task instead is to quantify these materials, the corresponding values are 0.34 for water and 0.26 for bone. We have shown that the matrix-valued NEQ and DQE metrics contain sufficient information for calculating the dose efficiency for both detection or quantification tasks, the task having any spatial and energy dependence. This framework will be beneficial for the development of photon-counting X-ray detectors.