2.2 - Troubleshoot monitoring and alerting (silent rules, INSUFFICIENT_DATA, missing findings)
AWS Certified Security Specialty objective 2.2 covers the failure modes of a monitoring stack that looks configured but stays silent. IAM and other global service events are emitted only in us-east-1, so a rule built elsewhere to catch policy changes never fires. An alarm sitting permanently in INSUFFICIENT_DATA usually means the metric filter has published nothing yet, because CloudWatch metric filters only create a data point when the pattern matches; treating missing data as not breaching fixes the noise. A rule that matches events but never invokes its Lambda target is missing the resource-based policy granting events.amazonaws.com permission to invoke, and an alarm whose history shows the SNS action ran while nobody got mail points to an unconfirmed subscription, or to an SNS topic encrypted with a customer managed KMS key whose policy does not let CloudWatch generate a data key. Memory is not an EC2 default metric, so the CloudWatch agent must be installed. EventBridge patterns match exact values, not prefixes, and rules on a custom bus never see events delivered to the default bus. Findings stop when a member account is not invited, or Inspector coverage misses instances without the SSM agent.
Rule on IAM changes never fires = global service events only land in us-east-1. Alarm stuck in INSUFFICIENT_DATA = the metric filter has published no data point yet. Rule matches but Lambda never runs = missing resource policy for events.amazonaws.com.
Practice questions
1. An alarm visibly moves to the ALARM state in the console, yet its SNS topic is never notified and the alarm history shows no action entry. What should be inspected?
- The retention period of the log group behind the metric
- Whether alarm actions are enabled on the alarm (correct answer)
- Whether the metric is published with a resolution of one second, which prevents CloudWatch from attaching any notification action to the alarms built on top of it
- The number of subscribers on the destination topic
An alarm whose ActionsEnabled flag is false still evaluates and still changes state, but it never runs its actions, and the history records the state change without any action line. Retention concerns how long log lines survive, one second resolution is a valid alarm source, and even a topic with zero subscribers would still show that the action ran.
2. An EventBridge rule invokes a Lambda function that is sometimes throttled, and a few security events are lost during those windows. Which configuration prevents the loss?
- A dead letter queue on the rule target, with a retry policy (correct answer)
- A longer timeout on the Lambda function
- A second identical rule on the same bus, so that any event missed by the first delivery attempt is picked up again by the duplicate rule a few seconds later
- An SNS topic placed between the rule and the function
EventBridge retries a failed delivery and then drops the event unless a dead letter queue is configured on the target, so the DLQ is what turns a lost event into a recoverable one. A longer timeout does nothing about throttling, a duplicate rule would receive the same event and be throttled the same way, and inserting SNS moves the problem without giving any durable landing place.
3. An EventBridge rule created in eu-west-1 to catch IAM policy changes never fires, although the changes clearly happen. What is the most likely cause?
- EventBridge rules must always be created in the account's home region, which is defined once at account creation and cannot be changed afterwards
- The rule needs a resource based policy allowing the iam.amazonaws.com service principal to invoke it
- CloudTrail must be configured as an explicit event source on the custom bus before it will publish anything
- IAM is a global service whose events are delivered in us-east-1, not in eu-west-1 (correct answer)
Global services such as IAM, STS, CloudFront and Route 53 record their events in us-east-1, so an EventBridge rule watching them has to live there and can then forward to another region if needed. There is no such thing as an immutable account home region for rules, EventBridge targets do not require a policy for iam.amazonaws.com, and CloudTrail already publishes management events to the default bus without being registered as a source.
4. An alarm built on a metric filter for failed console logins sits permanently in INSUFFICIENT_DATA instead of OK. What is the usual explanation?
- The alarm threshold is set higher than the maximum value the metric filter is technically able to publish for a single evaluation period
- The SNS topic attached to the alarm action has not been confirmed by its subscriber
- CloudWatch alarms cannot be built on metrics produced by metric filters
- No matching log line has occurred, so the filter publishes no data point at all (correct answer)
A metric filter emits a data point only when a line matches, so a quiet period leaves the metric empty and the alarm in INSUFFICIENT_DATA. The fix is to set a default value of zero on the filter or to treat missing data as notBreaching. Threshold height only decides OK versus ALARM, an unconfirmed subscription would affect notification delivery and not alarm state, and metric filter metrics are perfectly valid alarm sources.
5. An alarm reaches the ALARM state and its history confirms the SNS action ran, yet the on call engineer receives no email. What should be checked first?
- Whether the CloudWatch metric has enough resolution
- Whether the email subscription has been confirmed (correct answer)
- Whether the alarm period is shorter than the metric publication interval, which would cause CloudWatch to discard the notification before it reaches the topic
- Whether the alarm is attached to a dashboard
An SNS email subscription stays in PendingConfirmation until the recipient clicks the confirmation link, and SNS silently skips pending endpoints. Metric resolution and dashboard attachment have no bearing on delivery, and a period shorter than the publication interval produces missing data points rather than a discarded notification.
6. A team creates a CloudWatch alarm on memory utilisation of an EC2 instance, but the metric list is empty. Why?
- Detailed monitoring is disabled, and until it is turned on the hypervisor withholds every operating system level metric from the CloudWatch namespace
- Memory metrics only exist for instances launched from an AWS provided AMI
- The instance is not registered with AWS Systems Manager
- Memory is a guest level metric that requires the CloudWatch agent (correct answer)
The hypervisor cannot see inside the guest, so memory, swap and disk usage are only published if the CloudWatch agent runs in the instance and has a role allowing PutMetricData. Detailed monitoring merely raises EC2 metric frequency from five minutes to one, the AMI origin is irrelevant, and Systems Manager registration eases agent deployment without being the reason the metric is absent.