Cybersecurity organizations invest heavily in developing their people. Analysts attend technical courses, incident responders earn certifications, and security teams participate in awareness programs. These investments are essential, but they leave an important question unanswered.
An analyst may understand how ransomware operates without recognizing the early signs of lateral movement. An incident response team may know its procedures but struggle to coordinate containment decisions. An executive may approve an incident response plan without having practiced the business decisions required during a major disruption.
The distinction is not between training and practical experience. Both matter. The distinction is between acquiring knowledge and demonstrating operational capability.
A mature cybersecurity readiness program must address both.
The limitation of training-based metrics
Most organizations can report how many employees completed cybersecurity training, which certifications they hold, and how many learning hours were delivered.
These metrics describe educational activity. They do not necessarily describe operational performance.
Consider a Security Operations Center (SOC) that has completed training on credential-based attacks. The analysts may understand Kerberos authentication, recognize suspicious PowerShell commands, and know how attackers exploit privileged accounts.
But during an actual incident, several additional capabilities become critical:
- Can the team distinguish malicious authentication activity from legitimate administrative behavior?
- Can analysts correlate endpoint, identity, and network telemetry?
- Can they determine the scope of a compromise before recommending containment?
- Can they escalate the incident with sufficient evidence for a timely decision?
- Can they preserve investigative evidence while coordinating recovery?
These are observable behaviors, not simply knowledge requirements.
The NIST NICE Workforce Framework (SP 800-181 Rev. 1) provides a useful foundation for this distinction. Rather than defining cybersecurity work primarily through course titles or certifications, NICE describes work using tasks, knowledge, and skills. Organizations can use this approach to define what a role must actually accomplish and then assess whether its personnel can perform those tasks.
Readiness is a system capability, not an individual qualification
Cyber incidents rarely respect organizational boundaries. A ransomware investigation may involve SOC analysts, incident responders, identity administrators, infrastructure teams, legal advisers, business continuity personnel, and executive leadership.
Each participant may be competent individually, yet the overall response may still fail.
For example, an analyst identifies suspicious activity and recommends disabling a compromised privileged account. The identity team hesitates because the account supports a critical business application. Meanwhile, the incident commander is waiting for confirmation of the affected systems.
No single participant necessarily lacks technical knowledge. The weakness lies in coordination, decision authority, and the ability to act under uncertainty. This is why readiness must be assessed at multiple levels:
Individual capability
Can each person perform the technical and procedural tasks required by their role?
Team capability
Can participants share evidence, establish priorities, and coordinate actions efficiently?
Organizational capability
Can technical findings be translated into business decisions, with clear accountability and appropriate authority?
NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities, explicitly distinguishes training, exercises, and tests. Its guidance supports using exercises to evaluate plans and identify procedural or capability gaps rather than relying on instruction alone.
The publication dates to 2006, but the underlying distinction remains useful: knowing a plan and successfully executing it are different outcomes.
Turn learning objectives into observable performance
An effective readiness program begins by defining what successful performance looks like.
“Understand how to respond to credential compromise.”
“Given authentication logs, endpoint telemetry, and a simulated compromised account, identify the suspicious activity, establish the likely scope, document supporting evidence, and recommend a justified containment action.”
The second objective can be evaluated. A practical assessment design should connect four elements:
| Element | Example |
|---|---|
| Role | SOC analyst |
| Required capability | Investigate suspected credential compromise |
| Observable evidence | Investigation timeline, correlated events, escalation decision |
| Evaluation criteria | Accuracy, completeness, timeliness, and quality of reasoning |
The assessment should evaluate the decisions made, not merely whether participants reach a predefined answer.
For example, rapidly disabling an account is not automatically the correct decision. In some circumstances, the action may disrupt critical services or interfere with evidence collection. A stronger evaluation considers whether the analyst recognized those consequences, communicated the trade-offs, and followed the appropriate authorization process.
A practical example: from classroom knowledge to incident response
Consider the following fictional exercise, designed to illustrate a performance-based assessment.
An organization operates a hybrid environment with on-premises Active Directory, cloud identities, and centralized security monitoring. At the beginning of the exercise, the SOC receives an alert indicating unusual authentication activity involving a privileged account. The scenario unfolds in stages.
Detection and initial investigation
Participants receive authentication records, endpoint alerts, and selected network events. They must determine whether the activity is suspicious, identify the affected identities, and establish an initial incident timeline.
The evaluator observes whether analysts validate their hypotheses against available evidence rather than relying solely on alert severity.
Scope and escalation
Additional evidence suggests that the account may have been used to access another system. Participants must investigate possible lateral movement, communicate their findings, and decide whether to escalate the incident.
The evaluation considers the completeness of the evidence, the quality of the escalation, and whether important uncertainties were communicated.
Containment decision
The compromised account is discovered to have dependencies involving a critical business service. The team must recommend a containment strategy that considers both security risk and operational impact.
This tests coordination between technical teams and business decision-makers.
Recovery and review
The exercise concludes with a recovery discussion and a structured review of decisions, evidence, and missed opportunities.
The objective is not simply to declare the incident resolved. It is to determine which capabilities were demonstrated, which were missing, and what must improve.
A scenario of this kind can reveal weaknesses that would be difficult to identify through a conventional knowledge assessment. For example, participants may demonstrate excellent technical investigation skills but fail to communicate the business implications of their findings.
That is a readiness gap worth addressing.
Measure evidence, not participation
Exercise participation is a useful administrative metric, but it should not be confused with operational readiness. Organizations should establish evaluation criteria before an exercise begins. Possible measures include:
| Measure | What it helps evaluate |
|---|---|
| Time to validate suspicious activity | Initial investigation efficiency |
| Evidence completeness | Whether conclusions are supported |
| Time to appropriate escalation | Communication and decision flow |
| Containment decision quality | Technical judgment and risk awareness |
| Procedure adherence | Execution of established response processes |
| Cross-team coordination | Ability to operate across organizational boundaries |
| Improvement on retest | Whether identified weaknesses were addressed |
These measures require interpretation. A faster investigation is not necessarily better if critical evidence is overlooked. A slower containment decision may be justified when business dependencies are significant.
Organizations should therefore combine quantitative measures with structured qualitative evaluation. They should also avoid comparing results across exercises of different difficulty without accounting for scenario complexity, available telemetry, and participant responsibilities.
Close the loop: assessment, improvement and retesting
One of the most common weaknesses in exercise programs is treating the exercise itself as the final deliverable. A team completes a simulation, receives feedback, and returns to normal operations. The findings are documented, but the organization never verifies whether the weaknesses were corrected.
This approach measures activity without establishing improvement. A more effective process follows a continuous cycle:
- Assess
- Identify gaps
- Improve
- Retest
- Update readiness evidence
For example, suppose an exercise reveals that analysts can identify compromised credentials but cannot reliably determine which systems were accessed. The immediate response should not automatically be another general cybersecurity course. The organization should investigate the underlying cause.
Is the problem insufficient analyst knowledge? Missing telemetry? Poorly documented investigation procedures? Inadequate access to relevant systems?
Each cause requires a different intervention. A knowledge gap may justify targeted training. A telemetry gap may require engineering changes. A coordination problem may call for a revised escalation process and a joint exercise.
The CISA Tabletop Exercise Package guidance recommends producing an After-Action Report and Improvement Plan that identifies corrective actions, assigns responsibility, and establishes implementation timelines. This creates a connection between exercise findings and organizational change.
Retesting then provides evidence of whether the intervention worked.
A practical starting point for security leaders
Organizations do not need to begin with a large cyber range or a complex, multi-day simulation. A focused readiness assessment can begin with one critical role, one relevant threat scenario, and a small number of measurable objectives. A practical starting approach is:
Select a critical capability
For example, investigating privileged account compromise or coordinating ransomware containment.
Define the required performance
Identify the tasks, decisions, and outputs that demonstrate successful execution.
Establish a baseline
Use a focused tabletop, technical lab, or controlled simulation to observe current performance.
Document specific gaps
Separate individual skill gaps from process, technology, and coordination weaknesses.
Assign improvements and retest
Verify that corrective actions produce observable changes.
As the program matures, organizations can introduce more complex scenarios, cross-functional participation, and longitudinal measurement.
The important shift is methodological: move from measuring how much training was delivered to measuring what the organization can demonstrate.
Readiness must be demonstrated
Cybersecurity training builds essential knowledge. Certifications can validate important technical competencies. Neither should be dismissed.
But neither, on its own, establishes that an organization can investigate, coordinate, decide, and recover effectively during a real incident.
Operational readiness emerges when knowledge is combined with realistic practice, clear responsibilities, observable performance, and continuous improvement. For security leaders, the central question should evolve:
“How many people have we trained?”
“What can our people and teams demonstrably do, under realistic conditions, and what evidence supports that conclusion?”
References and further reading
- NIST — Workforce Framework for Cybersecurity (NICE Framework), SP 800-181 Rev. 1 ↗
A framework for describing cybersecurity work through tasks, knowledge, and skills. Useful for defining role-based capabilities and performance expectations.
- NIST — Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities, SP 800-84 ↗
Guidance on designing and evaluating training, exercises, and tests to assess organizational capabilities.
- CISA — Cybersecurity Tabletop Exercise Package (CTEP) ↗
Practical exercise planning guidance, including facilitator materials, evaluation templates, and after-action improvement planning.
- NIST — Cybersecurity Framework 2.0 ↗
A broader cybersecurity risk management framework that incorporates continuous improvement and lessons learned.
Editorial note: This article presents an original CySource perspective informed by publicly available cybersecurity frameworks and guidance. The exercise scenario is illustrative and does not describe an actual client engagement, measured research outcome, or independently verified case study.