trek10inc / awsets Goto Github PK

View Code? Open in Web Editor NEW

197.0 197.0 25.0 508 KB

A utility for crawling an AWS account and exporting all its resources for further analysis.

License: MIT License

Go 99.98% Makefile 0.02%

awsets's People

Contributors

Stargazers

Watchers

awsets's Issues

Installation fails via Homebrew

I'm unable to install via Homebrew. Further, visiting the Homebrew website, I cannot find any reference to the Trek10 tap.

❯ brew tap trek10inc/tap

Error: Invalid formula: /opt/homebrew/Library/Taps/trek10inc/homebrew-tap/Formula/awsets.rb
formulae require at least a URL
Error: Cannot tap trek10inc/tap: invalid syntax in tap!

cli-option "--profile"

Hi,

I have several profiles in my .aws/credentials and don't have the [default] one.
So, I can't choose which profile to use (for example like in aws-cli) and have an error:

2020/09/17 14:37:23 unable to load regions: failed to query regions: EC2RoleRequestError: no EC2 instance role found
caused by: exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: host is down

It's because after checking default profile in .aws/credentials and envs - it tries to use InstanceRole.
I think it will be very convenient (for example):

awsets list --include iam/user --profile dev

Do you think the --profile arg will be useful?

Distribution via brew once we have a 1.0.0

Hello,

Would be great to be able to do brew install awssets. Once this reaches 1.0.0 or when you have time, would you mind updating the distribution mechanism to brew for Mac binaries?

ref: https://medium.com/@ben.lafferty/deploying-go-cli-applications-316e9cca16a4

After installing awsets, it does not work as a command.

After installing GO and then awsets, it does not work as a command.
GO111MODULE="on" go get github.com/trek10inc/awsets/cmd/awsets
Is there another step before we can run it as a command?

❯ GO111MODULE="on" go get github.com/trek10inc/awsets/cmd/awsets
go: github.com/trek10inc/awsets/cmd/awsets upgrade => v0.0.0-20200903202143-ec27b76811e6

❯ awsets --help
zsh: command not found: awsets

Add relations to Log Groups

Currently, Log Groups are being queried, but they are not tied to anything. Where possible, add a relationship to the appropriate Lambda, EC2, ECS, EKS, etc resource.

Feature request: Add DynamoDB Backups & relations to DDB tables

New flag to pass aws credentials or a way to run awsets without aws cli and creds configured?

Hi,

Awesome work on this project!

I've read the documentation and I didn't find a way to run awsets list -o all.json without having aws cli configured (most probably I'm missing something).

How we can run awsets list -o all.json without aws cli configured?

I would gladly help with work on this project, but I'm still a beginner in Go..

Update: Or role-arn?

no --include but len(include) in Listers function is still equal to 1

When debugging !25 I found that even when an --include is not in the command it is still creating this list with 1 element of an empty string.

Improve ARN parsing

Currently, ARN parsing is very rudimentary. It needs updated to handle the more complicated formats, including ones with versions.

CloudWatch QueryDefinitions

Hey there,

Any change of supporting CloudWatch QueryDefinitions ?

awsets is awesome :D

Thanks!

DynamoDB backups of deleted tables excluded

First, thanks for adding DDB backups to 0.6.1.

Commit 0925415... / issue #14 added DDB backups as sub-resources to ddb/table. DynamoDB backups can exist of now-deleted tables that still need to be cleaned up.

To gather those, I think there needs to be a separate resource type ddb/backup added instead of backups being a sub-resource listed on existing tables. Another option would be to make a fake ddb/table resource with a status of DELETED and put those backups as sub-resources of.

Bug: Listers returning data from wrong services.

Bug:

The following Listers returned data from rds and are "Type"ing as their own.

docdb/cluster
docdb/instance
neptune/dbcluster
neptune/dbclustersnapshot
neptune/dbinstance

Details

I do not have any docdb clusters.
Listing for docdb/cluster returns data.
These records indicate that the resources are of "Type": "docdb/cluster".
The rest of that record contains values from rds clusters. [Account, Region, Id, Name, Attributes, Tags, Relations]
Same thing happens with above Listers. i.e. They are returning rds/* resource information and are "Type"ing as their own.
This may be happening with other listers. This is just what I ran into.

command ran:

awsets list --profile "my-profile" --show-progress --regions "us-east-1" --include "docdb" --output "docdb-output.json"
awsets list --profile "my-profile" --show-progress --regions "us-east-1" --include "neptune" --output "neptune-output.json"

Version

awsets - version: 1.0.4	commit: ba77e77d0c6b00fbdd59712a0b4d08c49f2d9b61	date: 2021-12-10T17:21:29Z

Fix S3 bucket location

Update code after this bug is fixed in the AWS Go SDK so that S3 buckets include region

Handle Unmapped resources

The following is a list of resource type & count from a recent scan. Support for these resources should be prioritized, or they need to be explicitly excluded in the mappings.

AWS::CodeStar::SeedRepository,008
AWS::Route53Resolver::ResolverEndpoint,004
AWS::KinesisAnalytics::ApplicationOutput,006
AWS::DirectoryService::SimpleAD,001
AWS::ApiGateway::Account,023
AWS::EC2::TransitGatewayRoute,005
AWS::EC2::VPNGatewayRoutePropagation,020
AWS::Inspector::AssessmentTarget,004
AWS::S3::AccessPoint,011
AWS::IAM::UserToGroupAddition,014
AWS::CDK::Metadata,006
AWS::CodeStar::SyncResources,029
AWS::Route53Resolver::ResolverQueryLoggingConfig,001
AWS::RDS::DBSecurityGroup,003
AWS::ServiceCatalog::LaunchRoleConstraint,146
AWS::DLM::LifecyclePolicy,024
AWS::Inspector::ResourceGroup,001
AWS::Chatbot::SlackChannelConfiguration,005
AWS::EC2::ClientVpnEndpoint,003
AWS::Route53Resolver::ResolverRule,010
AWS::SecretsManager::SecretTargetAttachment,006
AWS::KinesisAnalytics::Application,006
AWS::RDS::EventSubscription,004
AWS::CodeStarNotifications::NotificationRule,006
AWS::Route53Resolver::ResolverRuleAssociation,010
AWS::OpsWorksCM::Server,001
AWS::RDS::OptionGroup,013
AWS::ServiceDiscovery::Service,006
AWS::DirectoryService::MicrosoftAD,002
AWS::ServiceCatalog::LaunchNotificationConstraint,140
AWS::ServiceCatalog::PortfolioProductAssociation,146
AWS::Route53::RecordSetGroup,148
AWS::ElasticLoadBalancingV2::ListenerCertificate,009
AWS::IAM::ServiceLinkedRole,035
AWS::SecurityHub::Hub,001
AWS::Inspector::AssessmentTemplate,004
AWS::ServiceDiscovery::PrivateDnsNamespace,003
AWS::Route53Resolver::ResolverQueryLoggingConfigAssociation,001
AWS::StepFunctions::Activity,003

[FEAT-REQ] profile/account support

the code currently supports AWS_PROFILE env var, do you think you could like that back to the name of the cache file, or smuggle account number into there somewhere?

I would love to have a cache of a few of the clients i work on, at the moment I have to do a little bit of files/folder manipulation to add this 'external' metadata.

Open to your thoughts.

Terry

Lots of exceed maximum number of attempts and fail.

Steps:
Running ./awsets list -o all.json with a custom AWS_PROFILE set.

Stack:

1: failed job us-west-1 - AWSGreengrassGroup
 with error: failed to list greengrass groups: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 04eafedd-fdaf-4442-a1c1-113415c22ffc
3: failed job us-west-1 - AWSGreengrassCoreDefinition
 with error: failed to list greengrass core definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: c0bf6a76-77e8-4fbe-a890-61c2801710c1
2: failed job us-west-1 - AWSGreengrassFunctionDefinition
 with error: failed to list greengrass function definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 688f3272-5b39-41a8-aa6c-c238081bf0e4
6: failed job ap-northeast-3 - AWSEc2TransitGateway
 with error: InvalidAction: The action DescribeTransitGateways is not valid for this web service.
	status code: 400, request id: 01331f76-be70-4a48-a5cc-d4f892b48bc4
2: failed job ap-northeast-3 - AWSCloudFormationStackSet
 with error: ValidationError: AWS CloudFormation StackSets is not supported in this region
	status code: 400, request id: b4041a21-b14f-4a9a-b775-be75c2df9962
7: failed job us-west-1 - AWSGreengrassResourceDefinition
 with error: failed to list greengrass resource definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 275297d2-96f5-4efa-8aa5-ba5748a35e61
0: failed job ap-northeast-3 - AWSApiGatewayApiKey
 with error: AccessDeniedException:
	status code: 403, request id: d89513e7-db9c-4da9-8796-e2534586e0e5
0: failed job us-west-1 - AWSGreengrassDeviceDefinition
 with error: failed to list greengrass device definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 614d40e2-a902-4691-9d10-277f674df0a9
5: failed job us-west-1 - AWSGreengrassSubscriptionDefinition
 with error: failed to list greengrass subscription definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 636d7bd6-51af-4369-8c75-bdbfff9d5960
9: failed job us-west-1 - AWSGreengrassConnectorDefinition
 with error: failed to list greengrass connector definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: d34de1f4-69ba-4ec6-b020-039b2972e769
9: failed job us-west-1 - AWSGreengrassLoggerDefinition
 with error: failed to list greengrass logger definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: f4cdb082-0f2c-4294-9a82-c0aa185e1018
6: failed job ap-northeast-3 - AWSApiGatewayVpcLink
 with error: AccessDeniedException:
	status code: 403, request id: e6bc79c1-2879-4787-b836-f786bb65716e
6: failed job ap-northeast-3 - AWSApiGatewayV2DomainName
 with error: failed to list apigatewayv2 domain names: AccessDeniedException:
	status code: 403, request id: 5c487e09-35bc-4b6d-aed7-01da76ee3128
7: failed job ap-northeast-3 - AWSApiGatewayDomainName
 with error: AccessDeniedException:
	status code: 403, request id: b54f89b7-1409-41ac-b9fb-773b083d6784
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3278f33]

goroutine 41 [running]:
github.com/trek10inc/awsets/arn.ParseP(...)
	/home/runner/work/awsets/awsets/arn/main.go:32
github.com/trek10inc/awsets/lister.AWSEcsCluster.List(0xc00041baa0, 0x9, 0x42d8840, 0xc00023e9c0, 0x42d2ce0, 0xc0004395e0, 0x42d2ae0, 0xc0000f2cc0, 0xc0013be450, 0x2, ...)
	/home/runner/work/awsets/awsets/lister/ecs_cluster.go:80 +0x973
github.com/trek10inc/awsets.List.func1(0xc00003cb40, 0xc000456180, 0xc00043a1c0, 0xc00045e4a0, 0x7, 0xc00053e060)
	/home/runner/work/awsets/awsets/main.go:206 +0x9ba
created by github.com/trek10inc/awsets.List
	/home/runner/work/awsets/awsets/main.go:171 +0x1e7

awssets accepts include with --

Hello,

Discovered this by accident, trying to list only EC2 resources.

./awsets list -o all.json include ec2 scans all the resources, instead of throwing an error.
./awsets list -o all.json --include ec2 works correctly

ECS Service raise error

This command:

Only empty results are returned

awsets list --regions eu-central-1 -o all.json --include ec2 yields an empty file:

❯ cat all.json
[]%
``

Verbose output:

❯ awsets list --regions eu-central-1 -o all.json --include ec2 -v
regions: [eu-central-1]
resource types: [ec2/eip ec2/flowlog ec2/image ec2/instance ec2/internetgateway ec2/keypair ec2/launchtemplate ec2/natgateway ec2/networkacl ec2/networkinterface ec2/routetable ec2/securitygroup ec2/snapshot ec2/subnet ec2/transitgateway ec2/volume ec2/vpc ec2/vpcpeering ec2/vpngateway]
querying 0 combinations
9: finished worker
1: finished worker
2: finished worker
0: finished worker
7: finished worker
3: finished worker
4: finished worker
6: finished worker
5: finished worker
8: finished worker


I double checked and `aws ec2 describe-instances` works as expected. What could be the issue?

Add additional relationships

Review greengrass device relationships
CF Distro & S3 bucket