Non-Production API Endpoints for the Amazon Cloudwatch Fails to Log to CloudTrail Resulting in Silent Permission Enumeration

How it was found

Summary: Typically, when an adversary gains access to stolen AWS IAM credentials they will frequently test those credentials to see what access they have. They do this by performing API calls and seeing which succeed and which fail. There are even automated tools to make this process easier. For defenders and security professionals, this behavior serves as a golden opportunity for detection as it likely involves generating a large number of failed API call attempts. If an adversary could enumerate permissions without logging to CloudTrail, they could perform this activity invisibly.

There are many categories of CloudTrail bypass. The specific variant we will be focused on in this report has been referred to as "non-production endpoint permission enumeration CloudTrail bypass". If you would like to learn more about it, you can find more details here.

We have found 4 non-production endpoints for the cloudwatch service which could be used with standard IAM credentials, and do not log the tested cloudwatch operation to CloudTrail. While it is good that these results do not prove access to customer partition data, the endpoints can still be used for permission enumeration without logging to CloudTrail.

AWS has previously stated that this type of vulnerability should be reported. Specifically, "For isolated non-production endpoints that do not log to CloudTrail but are otherwise callable with normal credentials and exhibit normal IAM permission behavior, AWS considers the CloudTrail logging bypass of such endpoints also to be a security issue. If you find an API or APIs on an endpoint with these characteristics, please contact the AWS Security Team at ████████".

Description:

Automated validation tested the candidate endpoint with two principals:

• admin: privileged role used to observe authorized candidate endpoint behavior.
• noperm: no-permission role used to confirm the candidate endpoint exhibits useful IAM authorization behavior.

For each endpoint in this report, the candidate endpoint call did not generate a matching CloudTrail event for either principal, and the candidate endpoint responses differed between the privileged and no-permission principals.

Steps To Reproduce:

Using a sufficiently privileged IAM user or role, perform the following AWS CLI operation against one of the affected candidate endpoints:

aws cloudwatch list-dashboards --region us-east-1 --cli-connect-timeout 10 --cli-read-timeout 10 --endpoint-url ████████ --no-verify-ssl

Wait 10-15 minutes and confirm that the request does not generate a matching CloudTrail event. Next, perform the same candidate endpoint call with a principal that does not have permission for the target action:

aws cloudwatch list-dashboards --region us-east-1 --cli-connect-timeout 10 --cli-read-timeout 10 --endpoint-url ████████ --no-verify-ssl

The second request also does not generate a matching CloudTrail event. An adversary can perform these calls and, depending on the response of the API, determine if an identity they have compromised does or does not have permission to perform the operation.

Additionally, depending on if a principal does or does not have permission to perform the action, the response from the affected endpoint will be different. Example evidence from the validation run is below.

Example: list-dashboards against ████████

CloudTrail lookup for the admin candidate endpoint call:

• region us-east-1: matched_any=False, matched_marker=False, events=0, sources=none

CloudTrail lookup for the noperm candidate endpoint call:

• region us-east-1: matched_any=False, matched_marker=False, events=0, sources=none

admin candidate response:

{
    "DashboardEntries": []
}

noperm candidate response:

An error occurred (AccessDenied) when calling the ListDashboards operation: User: arn:aws:sts::████████:assumed-role/noperm/vm-noperm-20260601T181312Z is not authorized to perform: cloudwatch:ListDashboards on resource: arn:aws:cloudwatch::████████:dashboard/* because no identity-based policy allows the cloudwatch:ListDashboards action

Supporting Material/References:

• Indicate the Amazon service or product that this vulnerability occurs on:

cloudwatch

• What type of Amazon AWS account(s) is needed to verify or reproduce this vulnerability?:

Standard commercial partition account

• Estimated CVSS score and vector string:

4.3 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N/CR:H/IR:H/AR:H

• Estimated CWEs (comma separated):

CWE-778: Insufficient Logging

• Have you already publicly disclosed any information on this issue? If so, when and where?:

This specific example? No. This general technique? Yes, a lot actually. I have blogged about it here and spoke about it at Black Hat USA 2023 (hxxps://www.youtube.com/watch?v=YP2XNAbB_Nw) and fwd:cloudsec 2023 (hxxps://www.youtube.com/watch?v=61C_lEQ5qNM) [I have broken these links because HackerOne appeared to complain about them?]

Additionally, we have reported similar findings for different services:

• aiops
• bedrock-agent
• neptune-graph
• transcribe
• storagegateway
• sso-admin
• ssm-quicksetup
• securityhub
• route53
• redshift-data
• neptune-graph
• lakeformation and m2
• health
• glue
• globalaccelerator
• forecast
• events
• elasticache
• datazone part 2
• docdb-elastic
• devicefarm
• datazone
• cloudwatch
• bedrock
• bedrock-agent
• ssm

The following operation exhibited this behavior:

• list-dashboards: 4 endpoint(s)

The following is a list of endpoints we found that exhibited this behavior:

Please note: We follow an industry standard 90-day vulnerability disclosure policy. You can read more about our policy here.

Impact

Summary:

An adversary can enumerate permissions of compromised credentials for the cloudwatch service without logging to CloudTrail.