Giter VIP home page Giter VIP logo

awsets's Issues

Only empty results are returned

awsets list --regions eu-central-1 -o all.json --include ec2 yields an empty file:

❯ cat all.json
[]%
``

Verbose output:

❯ awsets list --regions eu-central-1 -o all.json --include ec2 -v
regions: [eu-central-1]
resource types: [ec2/eip ec2/flowlog ec2/image ec2/instance ec2/internetgateway ec2/keypair ec2/launchtemplate ec2/natgateway ec2/networkacl ec2/networkinterface ec2/routetable ec2/securitygroup ec2/snapshot ec2/subnet ec2/transitgateway ec2/volume ec2/vpc ec2/vpcpeering ec2/vpngateway]
querying 0 combinations
9: finished worker
1: finished worker
2: finished worker
0: finished worker
7: finished worker
3: finished worker
4: finished worker
6: finished worker
5: finished worker
8: finished worker


I double checked and `aws ec2 describe-instances` works as expected. What could be the issue?

Installation fails via Homebrew

I'm unable to install via Homebrew. Further, visiting the Homebrew website, I cannot find any reference to the Trek10 tap.

❯ brew tap trek10inc/tap

Error: Invalid formula: /opt/homebrew/Library/Taps/trek10inc/homebrew-tap/Formula/awsets.rb
formulae require at least a URL
Error: Cannot tap trek10inc/tap: invalid syntax in tap!

Bug: Listers returning data from wrong services.

Bug:

The following Listers returned data from rds and are "Type"ing as their own.

  1. docdb/cluster
  2. docdb/instance
  3. neptune/dbcluster
  4. neptune/dbclustersnapshot
  5. neptune/dbinstance

Details

  • I do not have any docdb clusters.
  • Listing for docdb/cluster returns data.
  • These records indicate that the resources are of "Type": "docdb/cluster".
  • The rest of that record contains values from rds clusters. [Account, Region, Id, Name, Attributes, Tags, Relations]
  • Same thing happens with above Listers. i.e. They are returning rds/* resource information and are "Type"ing as their own.
  • This may be happening with other listers. This is just what I ran into.

command ran:

awsets list --profile "my-profile" --show-progress --regions "us-east-1" --include "docdb" --output "docdb-output.json"
awsets list --profile "my-profile" --show-progress --regions "us-east-1" --include "neptune" --output "neptune-output.json"

Version

awsets - version: 1.0.4	commit: ba77e77d0c6b00fbdd59712a0b4d08c49f2d9b61	date: 2021-12-10T17:21:29Z

awssets accepts include with --

Hello,

Discovered this by accident, trying to list only EC2 resources.

  1. ./awsets list -o all.json include ec2 scans all the resources, instead of throwing an error.
  2. ./awsets list -o all.json --include ec2 works correctly

Lots of exceed maximum number of attempts and fail.

Steps:
Running ./awsets list -o all.json with a custom AWS_PROFILE set.

Stack:

1: failed job us-west-1 - AWSGreengrassGroup
 with error: failed to list greengrass groups: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 04eafedd-fdaf-4442-a1c1-113415c22ffc
3: failed job us-west-1 - AWSGreengrassCoreDefinition
 with error: failed to list greengrass core definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: c0bf6a76-77e8-4fbe-a890-61c2801710c1
2: failed job us-west-1 - AWSGreengrassFunctionDefinition
 with error: failed to list greengrass function definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 688f3272-5b39-41a8-aa6c-c238081bf0e4
6: failed job ap-northeast-3 - AWSEc2TransitGateway
 with error: InvalidAction: The action DescribeTransitGateways is not valid for this web service.
	status code: 400, request id: 01331f76-be70-4a48-a5cc-d4f892b48bc4
2: failed job ap-northeast-3 - AWSCloudFormationStackSet
 with error: ValidationError: AWS CloudFormation StackSets is not supported in this region
	status code: 400, request id: b4041a21-b14f-4a9a-b775-be75c2df9962
7: failed job us-west-1 - AWSGreengrassResourceDefinition
 with error: failed to list greengrass resource definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 275297d2-96f5-4efa-8aa5-ba5748a35e61
0: failed job ap-northeast-3 - AWSApiGatewayApiKey
 with error: AccessDeniedException:
	status code: 403, request id: d89513e7-db9c-4da9-8796-e2534586e0e5
0: failed job us-west-1 - AWSGreengrassDeviceDefinition
 with error: failed to list greengrass device definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 614d40e2-a902-4691-9d10-277f674df0a9
5: failed job us-west-1 - AWSGreengrassSubscriptionDefinition
 with error: failed to list greengrass subscription definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: 636d7bd6-51af-4369-8c75-bdbfff9d5960
9: failed job us-west-1 - AWSGreengrassConnectorDefinition
 with error: failed to list greengrass connector definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: d34de1f4-69ba-4ec6-b020-039b2972e769
9: failed job us-west-1 - AWSGreengrassLoggerDefinition
 with error: failed to list greengrass logger definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
	status code: 429, request id: f4cdb082-0f2c-4294-9a82-c0aa185e1018
6: failed job ap-northeast-3 - AWSApiGatewayVpcLink
 with error: AccessDeniedException:
	status code: 403, request id: e6bc79c1-2879-4787-b836-f786bb65716e
6: failed job ap-northeast-3 - AWSApiGatewayV2DomainName
 with error: failed to list apigatewayv2 domain names: AccessDeniedException:
	status code: 403, request id: 5c487e09-35bc-4b6d-aed7-01da76ee3128
7: failed job ap-northeast-3 - AWSApiGatewayDomainName
 with error: AccessDeniedException:
	status code: 403, request id: b54f89b7-1409-41ac-b9fb-773b083d6784
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3278f33]

goroutine 41 [running]:
github.com/trek10inc/awsets/arn.ParseP(...)
	/home/runner/work/awsets/awsets/arn/main.go:32
github.com/trek10inc/awsets/lister.AWSEcsCluster.List(0xc00041baa0, 0x9, 0x42d8840, 0xc00023e9c0, 0x42d2ce0, 0xc0004395e0, 0x42d2ae0, 0xc0000f2cc0, 0xc0013be450, 0x2, ...)
	/home/runner/work/awsets/awsets/lister/ecs_cluster.go:80 +0x973
github.com/trek10inc/awsets.List.func1(0xc00003cb40, 0xc000456180, 0xc00043a1c0, 0xc00045e4a0, 0x7, 0xc00053e060)
	/home/runner/work/awsets/awsets/main.go:206 +0x9ba
created by github.com/trek10inc/awsets.List
	/home/runner/work/awsets/awsets/main.go:171 +0x1e7

Improve ARN parsing

Currently, ARN parsing is very rudimentary. It needs updated to handle the more complicated formats, including ones with versions.

cli-option "--profile"

Hi,

I have several profiles in my .aws/credentials and don't have the [default] one.
So, I can't choose which profile to use (for example like in aws-cli) and have an error:

2020/09/17 14:37:23 unable to load regions: failed to query regions: EC2RoleRequestError: no EC2 instance role found
caused by: exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: host is down

It's because after checking default profile in .aws/credentials and envs - it tries to use InstanceRole.
I think it will be very convenient (for example):

awsets list --include iam/user --profile dev

Do you think the --profile arg will be useful?

DynamoDB backups of deleted tables excluded

First, thanks for adding DDB backups to 0.6.1.

Commit 0925415... / issue #14 added DDB backups as sub-resources to ddb/table. DynamoDB backups can exist of now-deleted tables that still need to be cleaned up.

To gather those, I think there needs to be a separate resource type ddb/backup added instead of backups being a sub-resource listed on existing tables. Another option would be to make a fake ddb/table resource with a status of DELETED and put those backups as sub-resources of.

After installing awsets, it does not work as a command.

After installing GO and then awsets, it does not work as a command.
GO111MODULE="on" go get github.com/trek10inc/awsets/cmd/awsets
Is there another step before we can run it as a command?

❯ GO111MODULE="on" go get github.com/trek10inc/awsets/cmd/awsets
go: github.com/trek10inc/awsets/cmd/awsets upgrade => v0.0.0-20200903202143-ec27b76811e6
❯ awsets --help
zsh: command not found: awsets

Add relations to Log Groups

Currently, Log Groups are being queried, but they are not tied to anything. Where possible, add a relationship to the appropriate Lambda, EC2, ECS, EKS, etc resource.

Handle Unmapped resources

The following is a list of resource type & count from a recent scan. Support for these resources should be prioritized, or they need to be explicitly excluded in the mappings.

AWS::CodeStar::SeedRepository,008
AWS::Route53Resolver::ResolverEndpoint,004
AWS::KinesisAnalytics::ApplicationOutput,006
AWS::DirectoryService::SimpleAD,001
AWS::ApiGateway::Account,023
AWS::EC2::TransitGatewayRoute,005
AWS::EC2::VPNGatewayRoutePropagation,020
AWS::Inspector::AssessmentTarget,004
AWS::S3::AccessPoint,011
AWS::IAM::UserToGroupAddition,014
AWS::CDK::Metadata,006
AWS::CodeStar::SyncResources,029
AWS::Route53Resolver::ResolverQueryLoggingConfig,001
AWS::RDS::DBSecurityGroup,003
AWS::ServiceCatalog::LaunchRoleConstraint,146
AWS::DLM::LifecyclePolicy,024
AWS::Inspector::ResourceGroup,001
AWS::Chatbot::SlackChannelConfiguration,005
AWS::EC2::ClientVpnEndpoint,003
AWS::Route53Resolver::ResolverRule,010
AWS::SecretsManager::SecretTargetAttachment,006
AWS::KinesisAnalytics::Application,006
AWS::RDS::EventSubscription,004
AWS::CodeStarNotifications::NotificationRule,006
AWS::Route53Resolver::ResolverRuleAssociation,010
AWS::OpsWorksCM::Server,001
AWS::RDS::OptionGroup,013
AWS::ServiceDiscovery::Service,006
AWS::DirectoryService::MicrosoftAD,002
AWS::ServiceCatalog::LaunchNotificationConstraint,140
AWS::ServiceCatalog::PortfolioProductAssociation,146
AWS::Route53::RecordSetGroup,148
AWS::ElasticLoadBalancingV2::ListenerCertificate,009
AWS::IAM::ServiceLinkedRole,035
AWS::SecurityHub::Hub,001
AWS::Inspector::AssessmentTemplate,004
AWS::ServiceDiscovery::PrivateDnsNamespace,003
AWS::Route53Resolver::ResolverQueryLoggingConfigAssociation,001
AWS::StepFunctions::Activity,003

[FEAT-REQ] profile/account support

Hi

the code currently supports AWS_PROFILE env var, do you think you could like that back to the name of the cache file, or smuggle account number into there somewhere?

I would love to have a cache of a few of the clients i work on, at the moment I have to do a little bit of files/folder manipulation to add this 'external' metadata.

Open to your thoughts.

Terry

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.