Asked a clanker to correlate the test data (link in parent) with: https://raw.githubusercontent.com/aws/aws-emulator-evaluatio...

And output something meaningful for some of the more popular services:

  Service            Tests  All  Happy  Reject  Ctrl  Data  Solo  Integ  Wire#  Wire%
  -----------------  -----  ---  -----  ------  ----  ----  ----  -----  -----  -----
  Data Firehose        288  77%    85%     91%   92%   72%   89%    59%     24    62%
  SES                  132  72%    65%     79%   79%   42%   79%     0%     21    52%
  EventBridge          889  71%    74%     64%   72%   68%   83%    67%     75    84%
  KMS                  432  70%    82%     69%   81%   68%   97%    36%     69    84%
  Step Functions       907  70%    79%     53%   69%   67%   73%    65%     68    76%
  DynamoDB            1437  69%    70%     61%   62%   65%   82%    53%    243    62%
  SNS                  594  69%    72%     55%   62%   68%   75%    65%     39    69%
  SQS                 1480  68%    70%     67%   61%   68%   94%    62%     29    45%
  Secrets Manager      414  68%    78%     65%   66%   72%   79%    57%     26    77%
  Cognito UserPools    781  65%    67%     66%   82%   36%   75%    33%    193    64%
  S3                  2488  62%    62%     57%   61%   57%   83%    51%    296    68%
  CloudFront           627  62%    62%     63%   70%   46%   71%    41%    190    22%
  API Gateway         1009  61%    64%     59%   77%   41%   75%    43%    200    73%
  SES v2               210  61%    63%     59%   75%   48%   77%    40%     44    30%
  Route 53             130  61%    63%     59%   62%   53%   69%     0%     49    67%
  CloudWatch          1186  60%    66%     51%   66%   53%   77%    36%    152    55%
  ECR                  392  60%    55%     75%   67%   49%   87%    27%     74    65%
  Kinesis              380  59%    69%     44%   63%   50%   62%    54%     25    44%
  IAM                  991  58%    63%     50%   60%   34%   78%    41%    228    57%
  Batch                274  58%    62%     59%   78%   20%   65%    17%     19    47%
  CloudWatch Logs     1016  58%    61%     54%   71%   47%   78%    41%    185    59%
  OpenSearch           342  57%    62%     52%   70%   29%   59%    50%     98    48%
  API Gateway v2       292  57%    64%     52%   74%   38%   73%    33%    107    27%
  EKS                  349  57%    60%     52%   65%   10%   59%    38%     55    62%
  Glue                 646  54%    51%     55%   66%   32%   69%    40%    155    43%
  Lambda              2808  53%    53%     42%   58%   48%   48%    54%    171    42%
  ECS                  808  51%    45%     52%   62%   23%   62%    34%    118    44%
  Redshift             241  49%    57%     47%   53%   52%   47%    57%     50    46%
  STS                  218  49%    58%     46%   56%   30%   57%    34%     22    50%
  ELBv2                596  45%    43%     47%   55%   19%   58%    30%    111    63%
  RDS                  659  43%    54%     45%   54%   35%   51%    31%     99    28%
  ElastiCache          186  42%    43%     49%   54%    0%   47%    17%     27    41%
  EC2                 2043  40%    35%     36%   39%    5%   44%    32%    429    33%
  CloudFormation      2019  36%    33%     31%   30%   40%   78%    24%   2161     3%
  AppSync              419  34%    35%     35%   54%    1%   44%     0%     91    13%
  SSM                  566  28%    28%     27%   27%   25%   26%    30%    175    11%
  Athena               420  22%    20%     15%    7%   28%   16%    37%     37     0%
  EC2 Auto Scaling     440  21%    19%     23%   19%   18%   21%    20%     89    16%

Pass rates from the release's scores.json (commit 7753983), joined with the eval suite's manifest.jsonl. Tests = behavioral checks touching the service. All = overall pass rate. Happy/Reject = valid requests vs. requests AWS should refuse. Ctrl/Data = control plane vs. data plane. Solo/Integ = single-service vs. multi-service checks. Wire# / Wire% = count and pass rate of recorded request/response replay tests (a test passes only if every call matches).

Edit: Also worth a look: https://ministack.org/blog/aws-emulator

I believe AWS already autogenerates their API clients (e.g. boto3) based on some authoritative internal service api spec so I imagine that spec could be used upstream of this tool as well.

    MySchema:
      type: object
      properties:
        enum_member:   { enum: [foo, bar] }
        enum_member_2: { enum: [quz, qux] }
      if:   { properties: { enum_member: { const: bar } } }
      then: { required: [enum_member_2] }

Yeah whatever they did internally to get service teams to actually adopt AWS Config ~5 years ago needs to happen here or it’s going to get stuck with marquee services and that’s it. Especially given this is a much much larger ask of each service.

There are many AWS services. Simply offering an alternative to a few of the most used could be a good business. It doesn't have to cover all AWS in all release versions, just enough to get and hold some clients