upvote
I'm officially off of AWS so don't have any consoles to check against, but back on a laptop.

Based on docs and some of the concerns about this happening to someone else, I would probably start with the following:

1. Check who/what created those EC2s[0] using the console to query: eventSource:ec2.amazonaws.com eventName:RunInstances

2. Based on the userIdentity field, query the following actions.

3. Check if someone manually logged into Console (identity dependent) [1]: eventSource:signin.amazonaws.com userIdentity.type:[Root/IAMUser/AssumedRole/FederatedUser/AWSLambda] eventName:ConsoleLogin

4. Check if someone authenticated against Security Token Service (STS) [2]: eventSource:sts.amazonaws.com eventName:GetSessionToken

5. Check if someone used a valid STS Session to AssumeRole: eventSource:sts.amazonaws.com eventName:AssumeRole userIdentity.arn (or other identifier)

6. Check for any new IAM Roles/Accounts made for persistence: eventSource:iam.amazonaws.com (eventName:CreateUser OR eventName:DeleteUser)

7. Check if any already vulnerable IAM Roles/Accounts modified to be more permissive [3]: eventSource:iam.amazonaws.com (eventName:CreateRole OR eventName:DeleteRole OR eventName:AttachRolePolicy OR eventName:DetachRolePolicy)

8. Check for any access keys made [4][5]: eventSource:iam.amazonaws.com (eventName:CreateAccessKey OR eventName:DeleteAccessKey)

9. Check if any production / persistent EC2s have had their IAMInstanceProfile changed, to allow for a backdoor using EC2 permissions from a webshell/backdoor they could have placed on your public facing infra. [6]

etc. etc.

But if you have had a compromise based on initial investigations, probably worth while getting professional support to do a thorough audit of your environment.

[0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c...

[1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c...

[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-...

[3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/s...

[4] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...

[5] https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-...

[6] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R...

reply
this is helpful. i will look for the logs.

Also some more observations below:

1) some 20 organisations were created within our Root all with email id with same domain (co.jp) 2) attacker had created multiple fargate templates 3) they created resources in 16-17 AWS regions 4) they requested to raise SES,WS Fargate Resource Rate Quota Change was requested, sage maker Notebook maintenance - we have no need of using these instances (recd an email from aws for all of this) 5) in some of the emails i started seeing a new name added (random name @outlook.com)

reply
It does sound like you've been compromised by an outfit that has got automation to run these types of activities across compromised accounts. A Reddit post[0] from 3 years ago seems to indicate similar activities.

Do what you can to triage and see what's happened. But I would strongly recommend getting a professional outfit in ASAP to remediate (if you have insurance notify them of the incident as well - as often they'll be able to offer services to support in remediating), as well as, notify AWS that an incident has occurred.

[0] https://www.reddit.com/r/aws/comments/119admy/300k_bill_afte...

reply
RunInstances
reply