Scout2 doesn't appear to work with ResourceGroupsTaggingAPI about scoutsuite HOT 12 CLOSED

Thanks for testing it out ! I'll give it a try with your code and see what's missing so that I can update the wiki if needed.

On the other hand, I'm wondering what type of data you're looking for using this service? I'm not familiar with it but wonder what kind of security-related data you'd get out of it?

from scoutsuite.

In our case, we're trying to enforce meaningful tagging of resources, then use those tags to correlate said resources to applications and organizations internally. Ideally, all services would gather tags in the initial Scout2 run, but the boto3 commands to get tags from each service varies. Which would mean extending every service to make the additional API call to get tags

Instead, I want to use ResourceGroupsTaggingAPI to return everything tagged on the account, with an ARN as the identifier. Within Scout2, my thought was that there could be a rule that associates the tags with the resources after the fact. That way, I could write a rule to check whether resources are tagged (assuming they can be), but also for tracking and accountability of what group or application spun up the offending resource.

from scoutsuite.

Thanks for the explanation ! I had a chance to give it a look and there are a few things:

1. The main reason why you don't see anything parsed/stored is that, in the metadata file, the response element should be "ResourceTagMappingList". If you double check the boto3 doc at http://boto3.readthedocs.io/en/latest/reference/services/resourcegroupstaggingapi.html#ResourceGroupsTaggingAPI.Client.get_resources, the "response" attribute from the metadata map has to correspond to the name of the list attribute returned by the AWS API. Sometimes if you do "list_users" you get "users", and other times when you do "get_resources", you get "ResourceTagMappingList"... since there isn't much consistency across services, this field has to be looked up from the AWS API refs or boto3 doc.

2. If you just made this change, you would get a bunch of exceptions due to "ResourceARNS" not being declared in the "resource_id_map" (AWSScout2/configs/init.py). This is because you haven't created a custom parsing function as documented in https://github.com/nccgroup/Scout2/wiki/HowTo:-fetch-configuration-for-a-new-service-or-resource#step-4-create-resource-specific-parsing-functions. I just updated the wiki to have Step 3bis for testing.

git diff AWSScout2//configs/__init__.py
diff --git a/AWSScout2/configs/__init__.py b/AWSScout2/configs/__init__.py
index 0af8a16..d4a2877 100644
--- a/AWSScout2/configs/__init__.py
+++ b/AWSScout2/configs/__init__.py
@@ -1,5 +1,6 @@
 
 resource_id_map = {
+    'resource_arns': 'ResourceARN',
     'network_interfaces': 'NetworkInterfaceId',
     'peering_connections': 'VpcPeeringConnectionId',
     'subnet_groups': 'DBSubnetGroupName'

If you do the above change, and also change

+   "resources": {
+                "ResourceARN": {

into

+   "resources": {
+                "resource_arns": {

in the metadata you should have code that actually fetches the data now. I also just updated the wiki page to mention that, for consistency, I like to use snake_case and plural for the resource names in the Scout2 data structure.

Final point, looking at http://boto3.readthedocs.io/en/latest/reference/services/resourcegroupstaggingapi.html#ResourceGroupsTaggingAPI.Client.get_resources, I see that this service's pagination token is called PaginationToken as opposed to Marker or NextToken for every other service I've worked with so far... this will require a change in opinel as well to make sure the handle_truncated_response method iterates through all resources. I expect the change to look like that but haven't had a chance to test it yet:

diff --git a/opinel/utils/aws.py b/opinel/utils/aws.py
index a1e9f5b..6872d12 100644
--- a/opinel/utils/aws.py
+++ b/opinel/utils/aws.py
@@ -125,7 +125,7 @@ def handle_truncated_response(callback, params, entities):
             for entity in entities:
                 if entity in response:
                     results[entity] = results[entity] + response[entity]
-            for marker_name in ['NextToken', 'Marker']:
+            for marker_name in ['NextToken', 'Marker', 'PaginationToken']:
                 if marker_name in response and response[marker_name]:
                     params[marker_name] = response[marker_name]
                     marker_found = True

from scoutsuite.

You know, I still can't get anything out of it. From what I can see, the parse_resource_arn function doesn't even run. The class gets called, and I can do a print that comes through, but nothing inside my function actually seems to run.

Were you able to get actual results from this API in testing, or is this theoretical that it should work? Nothing I've tried gets me any data through Scout2. Using boto3 directly, it does work.

from scoutsuite.

Yeah I was able to get data - I'll push something to a branch

from scoutsuite.

Done - Pushed to branch issue228

from scoutsuite.

Ah, I see it works on pub cloud. It does not work for GovCloud, for some reason. All other services pull successfully in GovCloud. Boto3 itself can also successfully pull this API in GovCloud. Any ideas on why Scout2 wouldn't, considering the other services (like EC2) work regardless of region?

from scoutsuite.

Is there any way to get a fully debug-enabled build of opinel and scout2? Seems like this is some kind of error/empty return value on GovCloud, but there's no meaningful output to tell me what it's doing or why it's not getting results.

from scoutsuite.

Have you tried running with --debug ? If that's not enough, you can use virtualenv, clone the opinel and scout2 repos, and modify them as you wish to get the debug information you need.

I don't have access to a govcloud account which prevents me from testing in this partition. My first guess would be that resource groups tagging APIs aren't available in gov cloud. It's not listed on the AWS website (https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services); I suggest you open a ticket w/ the AWS support to check with them?

from scoutsuite.

@afchill - Did you get an answer from AWS? Is it because the service is not available in govcloud ?

from scoutsuite.

This could be extremely valuable if it works correctly.

from scoutsuite.

@afchill any updates on this?

from scoutsuite.