ashim888 / awis Goto Github PK

A python script to query Amazon's Alexa Web Information Service (AWIS).

License: GNU General Public License v3.0

Python 100.00%

awis's Introduction

awis

A python script that generates a custom url and query string used to query Amazon's Alexa Web Information Service (AWIS). You may want to check the getting started section on the FAQ.

UrlInfo request

from myawis import *

obj = CallAwis(Access_Key_ID, Secret_Access_Key)
urlinfo = obj.urlinfo("www.example.com")

More info in the docs on UrlInfo

Output sample

import pprint

pprint.pprint(flatten_urlinfo(urlinfo))

{'CategoryData.0.AbsolutePath': 'Top/Computers/Software/Configuration_Management/Tools/Git',
 'CategoryData.0.Title': 'Tools/Git',
 'CategoryData.1.AbsolutePath': 'Top/Computers/Open_Source/Project_Hosting',
 'CategoryData.1.Title': 'Open Source/Project Hosting',
 'ContactInfo.CompanyStockTicker': None,
 'ContactInfo.DataUrl.#text': 'github.com',
 'ContactInfo.DataUrl.@type': 'canonical',
 'ContactInfo.Email': None,
 'ContactInfo.OwnerName': None,
 'ContactInfo.PhoneNumbers.PhoneNumber': None,
 'ContactInfo.PhysicalAddress': None,
 'ContentData.AdultContent': None,
 'ContentData.DataUrl.#text': 'github.com',
 'ContentData.DataUrl.@type': 'canonical',
 'ContentData.Language': None,
 'ContentData.LinksInCount': '81310',
 'ContentData.OwnedDomains': None,
 'ContentData.SiteData.Description': 'GitHub is the best place to share code '
                                     'with friends, co-workers, classmates, '
                                     'and complete strangers. Over four '
                                     'million people use GitHub to build '
                                     'amazing things together.',
 'ContentData.SiteData.Title': 'GitHub',
 'ContentData.Speed.MedianLoadTime': '1675',
 'ContentData.Speed.Percentile': '53',
 'ContributingSubdomain.codeload.github.com.Months.1.PageViews.PerUser': '1.26',
 'ContributingSubdomain.codeload.github.com.Months.1.PageViews.Percentage': '0.58%',
 'ContributingSubdomain.codeload.github.com.Months.1.Reach.Percentage': '2.39%',
[...]
 'ContributingSubdomain.status.github.com.Months.1.PageViews.PerUser': '1.1',
 'ContributingSubdomain.status.github.com.Months.1.PageViews.Percentage': '0.07%',
 'ContributingSubdomain.status.github.com.Months.1.Reach.Percentage': '0.34%',
 'Country.AU.Contribution.PageViews': '0.8%',
 'Country.AU.Contribution.Users': '1.0%',
 'Country.AU.Rank': '74',
[...]
 'Country.US.Contribution.PageViews': '21.0%',
 'Country.US.Contribution.Users': '21.3%',
 'Country.US.Rank': '45',
 'OutputTimestamp': '2017-08-14T18:48:45.026723',
 'Related.DataUrl.#text': 'github.com',
 'Related.DataUrl.@type': 'canonical',
 'RelatedLink.0.DataUrl.#text': 'zenofshen.com/posts/ajax-sortable-lists-tutorial',
 'RelatedLink.0.DataUrl.@type': 'canonical',
 'RelatedLink.0.NavigableUrl': 'http://zenofshen.com/posts/ajax-sortable-lists-tutorial',
 'RelatedLink.0.Title': 'zen of shen - script.aculo.us Ajax Sortable Lists '
                        'Tutorial',
[...]
 'RelatedLink.9.DataUrl.#text': 'www.ubuntu.com/',
 'RelatedLink.9.DataUrl.@type': 'canonical',
 'RelatedLink.9.NavigableUrl': 'http://www.ubuntu.com/',
 'RelatedLink.9.Title': 'Ubuntu Linux',
 'TrafficData.DataUrl.#text': 'github.com',
 'TrafficData.DataUrl.@type': 'canonical',
 'TrafficData.Rank': '58',
 'UsageStatistic.Days.1.PageViews.PerMillion.Delta': '-44.22%',
 'UsageStatistic.Days.1.PageViews.PerMillion.Value': '944',
 'UsageStatistic.Days.1.PageViews.PerUser.Delta': '-14.06%',
 'UsageStatistic.Days.1.PageViews.PerUser.Value': '4.28',
 'UsageStatistic.Days.1.PageViews.Rank.Delta': '40',
 'UsageStatistic.Days.1.PageViews.Rank.Value': '92',
 'UsageStatistic.Days.1.Rank.Delta': '+26',
 'UsageStatistic.Days.1.Rank.Value': '84',
 'UsageStatistic.Days.1.Reach.PerMillion.Delta': '-34.99%',
 'UsageStatistic.Days.1.Reach.PerMillion.Value': '8,380',
 'UsageStatistic.Days.1.Reach.Rank.Delta': '+19',
 'UsageStatistic.Days.1.Reach.Rank.Value': '83',
[...]
 'UsageStatistic.Months.3.PageViews.PerMillion.Delta': '+5.46%',
 'UsageStatistic.Months.3.PageViews.PerMillion.Value': '1,573',
 'UsageStatistic.Months.3.PageViews.PerUser.Delta': '-3.69%',
 'UsageStatistic.Months.3.PageViews.PerUser.Value': '5.220',
 'UsageStatistic.Months.3.PageViews.Rank.Delta': '-1',
 'UsageStatistic.Months.3.PageViews.Rank.Value': '49',
 'UsageStatistic.Months.3.Rank.Delta': '-3',
 'UsageStatistic.Months.3.Rank.Value': '58',
 'UsageStatistic.Months.3.Reach.PerMillion.Delta': '+9.67%',
 'UsageStatistic.Months.3.Reach.PerMillion.Value': '12,830',
 'UsageStatistic.Months.3.Reach.Rank.Delta': '-1',
 'UsageStatistic.Months.3.Reach.Rank.Value': '67'}

TrafficHistory request

from myawis import *

obj = CallAwis(Access_Key_ID, Secret_Access_Key)
obj.traffichistory("www.example.com",RANGE, START)
# RANGE is optional. Defaults to 31
# START is optional. Defaults to 20070801

More info in the docs on TrafficHistory

awis's People

Contributors

Stargazers

Watchers

Forkers

jkgeyti michaelhuffman isaacbernat nishith83 jmiller656 damonboynton liehendi11 wusenyan mikkokotila smeilz starenka matricali bradwaite freshlybuilt dearyhud vincent-lefrere

awis's Issues

Request has expired

When I make a request, I get a "request has expired" error. Any suggestions?

update pip package with the latest version

Hey guys, I think the latest version on pypi is 0.2.3, can you update it to 0.2.4 there? Thanks.

authorization problem

Hello, I installed form pip and am doing:

from myawis import *

access = 'x'
secret = 'x'

obj = CallAwis(access, secret)
urlinfo = obj.urlinfo("www.example.com")

and I get:

{'OutputTimestamp': '20180520T152730Z',
 'Request.Errors.Error.ErrorCode': '"User: '
                                   'arn:aws:iam::192208424764:user/alexa is '
                                   'not authorized to perform: awis:GET on '
                                   'resource: '
                                   'arn:aws:awis:us-west-1:192208424764:/api"'}

But this credential is ok in the AWS side. Any ideas?

"KeyError: 'Response'" on given example in home page

I tried the basic example in the home page but I get a KeyError with flatten_urlinfo :

from myawis import *
obj = CallAwis(Access_Key_ID, Secret_Access_Key)
urlinfo = obj.urlinfo("www.example.com")
b = flatten_urlinfo(urlinfo)

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "...eb-virt\lib\site-packages\myawis\__init__.py", line 194, in flatten_urlinfo
    flatten(info["aws:UrlInfoResponse"]["Response"]["UrlInfoResult"]["Alexa"])
KeyError: 'Response'

flatten_urlinfo function is missing

Following the instructions provided in the README.md, I get the error on flatten_urlinfo():

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-9304a8f0e37b> in <module>()
----> 1 pprint.pprint(flatten_urlinfo(urlinfo))

NameError: name 'flatten_urlinfo' is not defined

TrafficHistory issue with adding in parameters

Evening,

I'm trying to use the traffic history response group but I'm unable to add in the range and start parameters successfully. When I just run the query it returns successfully with the default values, I've tried a few combinations but I can't seem to get it to come together.

I'm also struggling to parse the response, I am receiving a successful response but I follow the example of URLInfo at the top of the README doc I encounter some issues.

KeyError with 'Response' from UrlInfo

Following the example code on the README.md and using the git master, I'm getting the below error message.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-9304a8f0e37b> in <module>()
----> 1 pprint.pprint(flatten_urlinfo(urlinfo))

/Users/mikko/dev/norway/lib/python2.7/site-packages/myawis/__init__.pyc in flatten_urlinfo(urlinfo, shorter_keys)
    161     _result = {}
    162     info = xmltodict.parse(str(urlinfo))
--> 163     flatten(info["aws:UrlInfoResponse"]["Response"]["UrlInfoResult"]["Alexa"])
    164     _result["OutputTimestamp"] = create_timestamp()
    165     return _result

KeyError: 'Response'

NotImplementedError: .OwnedDomain contains a list we don't know how to flatten.

For some domains like yahoo.com and amazon.com I'm getting the following error:

Traceback (most recent call last):
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/myawis/__init__.py", line 134, in flatten
    items = value.items()
AttributeError: 'list' object has no attribute 'items'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/telegram/ext/dispatcher.py", line 270, in process_update
    handler.handle_update(update, self)
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/telegram/ext/messagehandler.py", line 169, in handle_update
    return self.callback(dispatcher.bot, update, **optional_args)
  File "rankcheckbot.py", line 101, in instantrank
    data = flatten_urlinfo(urlinfo)
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/myawis/__init__.py", line 163, in flatten_urlinfo
    flatten(info["aws:UrlInfoResponse"]["Response"]["UrlInfoResult"]["Alexa"])
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/myawis/__init__.py", line 159, in flatten
    flatten(v, ".".join([prefix, k]))
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/myawis/__init__.py", line 159, in flatten
    flatten(v, ".".join([prefix, k]))
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/myawis/__init__.py", line 159, in flatten
    flatten(v, ".".join([prefix, k]))
  File "/Users/mostafalameei/Sites/rankcheckbot/lib/python3.6/site-packages/myawis/__init__.py", line 156, in flatten
    raise NotImplementedError(msg)
NotImplementedError: .OwnedDomain contains a list we don't know how to flatten.

NameError: name Access_Key_ID is not defined

I'm getting this error when running my python script:
NameError: name 'Access_Key_ID' is not defined

P.S: I have removed my real access key from the above code.

Categories return Nothing

I want to get the categories but it returns Nothing. Do you know the reasons?

obj.urlinfo("www.instagram.com", response_group="Categories")

<?xml version="1.0" encoding="utf-8"?>
<aws:UrlInfoResponse xmlns:aws="http://awis.amazonaws.com/doc/2005-10-05">
<aws:Response xmlns:aws="http://awis.amazonaws.com/doc/2005-07-11">
<aws:OperationRequest>
<aws:RequestId>f2a24d64-4970-4f62-8bbe-1644ad6d7546</aws:RequestId>
</aws:OperationRequest>
<aws:UrlInfoResult>
<aws:Alexa>
<aws:Request>
<aws:Arguments>
<aws:Argument>
<aws:Name>url</aws:Name>
<aws:Value>www.instagram.com</aws:Value>
</aws:Argument>
<aws:Argument>
<aws:Name>responsegroup</aws:Name>
<aws:Value>Categories</aws:Value>
</aws:Argument>
</aws:Arguments>
</aws:Request>
<aws:Related>
<aws:DataUrl type="canonical">instagram.com/</aws:DataUrl>
</aws:Related>
</aws:Alexa>
</aws:UrlInfoResult>
<aws:ResponseStatus xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/">
<aws:StatusCode>Success</aws:StatusCode>
</aws:ResponseStatus>
</aws:Response>
</aws:UrlInfoResponse>

Doesn't work as V2 signing was deprecated

I loved this module, but it no longer works as only V4 signing works now.

Is it possible for someone to update this to V4 signing? Thanks so much!

Error in response

obj=CallAwis('www.dealsea.com','Rank', 'my id', 'my key')

obj.urlinfo()

Returns:

\nRequestExpiredRequest has expired. Timestamp date is 2017-08-31T00:41:56.381165e6ee50f7-5ca7-a1f4-2866-8a34076f48e6

How do I get the actual data?

Thanks a lot for creating this package! I might be missing something obvious as I was not able to get the actual data from Alexa.

I did get a positive response (just no actual data) using this:

obj=CallAwis('cnn.com','History',access_key_id,secret_access_key)

and then this:

obj.traffichistory()

In which case I get the response:

<?xml version="1.0" encoding="unicode-escape"?>\n<aws:TrafficHistoryResponse xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/"><Response><OperationRequest><RequestId>01c60df3-0086-ea87-5331-38811f0f7cca</RequestId></OperationRequest><TrafficHistoryResult><Alexa>\n<TrafficHistory>\n<Range>31</Range>\n<Site>www.cnn.com</Site>\n<Start>2007-08-01</Start>\n<HistoricalData/>\n</TrafficHistory>\n</Alexa></TrafficHistoryResult><aws:ResponseStatus><aws:StatusCode>Success</aws:StatusCode></aws:ResponseStatus></Response></aws:TrafficHistoryResponse>

Maybe it would help to include some expected outputs in the README to avoid doubt.

missing deps in the pip package

bs4 and requests are missing. Giving that the module only runs when these are present, maybe better to include it to setup.py. If you like I can make a PR with this change.

cat_browse should call 'CategoryBrowse' not 'CategoryListings'

These are 2 distinct endpoints - cat_browse should call 'CategoryBrowse' not 'CategoryListings'
parameter domain is unused (and doesn't make any sense for either call)

Sites Linking In

Is it possible to get the sites linking in from this wrapper?