meumobi / ams.connect Goto Github PK

I've setted start=2017-01-13 and expect results from 13/01 until yesterday but it returns results from 12/01 to 13/01

when requesting rubicon for a large period it raises error due to timeout (see logs below), could we increae this timeout

2017-01-26T15:13:44.116605+00:00 app[web.1]: 10.141.27.204 - - [26/Jan/2017:15:13:14 +0000] "GET /rubicon?start=2016-12-21&end=2016-12-31 HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
2017-01-26T15:13:44.106375+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/rubicon?start=2016-12-21&end=2016-12-31" host=limitless-sea-32755.herokuapp.com request_id=bfb9fb9d-a87e-4639-b348-2e4cf87e2c3a fwd="186.205.3.207" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0

Precisa agora colocar logs, pode ser monolog mesmo, que é recomendado com o framework, as principais exception que precisamos loggar hoje são:
API response
errors
response time and size
Connection timeout
Mapping error, campo nao existe

93fb5d7#commitcomment-21759041

The Adserving report data will be received via attachments in mail box to be defined;

• The service will call the Lib EmailReader to get the attachment with the data for a specific day;
• Then this will append this new data to a file accessed by the AdServingTable Class;
  • For now we'll maintain this file in a project folder, but when it grow in size, we can migrate to a database like mongodb or firebase;
• To prevent duplicated data to be appended, it will keep a PID Lock to check the last date of report data extracted;

for all providers cpm should be equal to (#(revenue/impr prises) * 1000)

Metrics and Dimensions

dimensions = [DATE, AD_UNIT_CODE]
metrics = [AD_REQUESTS, MATCHED_AD_REQUESTS, EARNINGS, AD_REQUESTS_RPM]

• date = date
• impr. reçues = AD_REQUESTS
• impr. prises = MATCHED_AD_REQUESTS
• revenu = EARNINGS*0.98
• key = AD_UNIT_CODE
• inventaire = "AdNetwork Fill"
• cpm = AD_REQUESTS_RPM

GET https://www.googleapis.com/adsense/v1.4/accounts/pub-3651508233665711/reports?startDate=2017-01-29&endDate=2017-01-30&dimension=AD_UNIT_CODE&dimension=DATE&metric=EARNINGS&metric=MATCHED_AD_REQUESTS&metric=AD_REQUESTS_RPM&metric=AD_REQUESTS&key={YOUR_API_KEY}

tested with https://limitless-sea-32755.herokuapp.com/rubicon?start=2017-01-12&end=2017-01-20

This feature should facilitate the debug

Sublime Skinz : Fillrate manquant (le calcul comme pour les autres partenaires. Pour Sublime, impressions reçues = impressions envoyées)

For rubicon if size_id = {2, 15}, then check from AdServing if 'impr envoyé's is present, if not then impr. envoyées (ie. 2) = impr. reçues (57) - impr. prises (57) from same zone_id and size_id resp. 57, 10
The biggest format redirect to smallest

we should sync AMS.Connect with daylight saving time to use correct time offset requesting rubicon (los Angeles location)

see #9 for more details

In order to compute reports of providers for last month, we need to save on AMS.connect the Adserving of last month. To achieve it I suggest to allow processing a monthly reports sent by email. The admin could sent the latest report (no matter if it's not the latest month) to amsconnect+latest@... and if we call /adserving?email=amsconnect+latest@... it will process it.

request often reach timeout, we should investigate if it's possible to do it Async or improve Service to not load AdServing and Correlation Table for each row

when we request /adtech for a star date, ams connect returns results from date + 1. For example http://limitless-sea-32755.herokuapp.com/adserving?start=2017-03-26 returns results of 27/03/2017

Documentation: http://www.adtech-demo.com/API (also reachable on https://adtech.fluidtopics.net/search#!search;query=api)

API Access:

• USERNAME: platform.api.1502.1
• PASSWORD: U(912!kpw%
• SERVER_URL: https://ws-api.adtech.de
• LOGIN_SERVER_URL: https://iq.adtech.de

Adapting the configuration for SSO

1. Ensure that ADTECH has enabled your ONE BY AOL: AdServer UI login to be SSO
   compatible
2. Replace {USERNAME} with your SSO user name
3. Replace {PASSWORD} with your SSO password
4. Replace {SERVER_URL} with your location’s server URL.
5. Replace {LOGIN_SERVER_URL} with your location’s login server URL.
6. You are now ready to run the SSO code.

Standardize the responses for services and errors

The Lumen Framework already has classes for managing responses, we can use them to keep the same standard for all the responses (data responses, error responses and general messages)
Ref: https://lumen.laravel.com/docs/5.4/responses

permitir selecionar um período com params Dates (Y-m-d) "start" e "end" (correspondem aos labels usados no sitebuilder para items.type==promotions), recomendo usar os mesmos para ser consistente. No caso do sublime, que não permite selecionar período, usar o valor "start" como filtro e ignorar "end".

In order to get full 24 hours of report, the timezone in request must be set to California timezone

To request a report in the account's local time zone, all you need to do is set the useTimezoneReporting parameter to true (it defaults to false).

Source: https://developers.google.com/adsense/management/timezone

Normalize the enviroment to run the keep the compatibility when running local and in heroku platform.

Ref: https://devcenter.heroku.com/articles/container-registry-and-runtime

Add following field
fillRate = "impressions reçues" / "impressions prises" * 100 + '%'

To streamline investigations it should display:

• request url and headers
• response body and headers

AdServing is a static file updated by cron on AMS.Connect and using following format:

Impressions envoyées,	Date,	Key
5177,	1/23/2017,	394466-67
5687,	1/23/2017,	394466-43
26115,	1/23/2017,	395228-10

Be careful the date use format MMDDYYYY

Both fields added on AMS.Format are described below:

impressions envoyées = $line["Imps. sans défaut"]
discrepencies = (1 -$line["Imps. sans défaut"]/$array["impressions reçues"] )*100 + '%'

if I call the provider report and cancel request before server send response, then the temp file is not removed from /public

Report all adserving keys missing 'lumen.WARNING: AdServing Key Not Found {"key":503487,"date":"2017-03-24"} []'

I've tested to request sublime after 9pm from Rio and AMS.Connect returns following msg:
cURL Error :date format should be YYYY-mm-dd AND before today

When we request monthly datas (from 01-02 to 28-02 for example) of rubicon, a lot of rows are missing. For instance 392430-57 is missing for 02-02

seems first request after long time idle. See logs below. The request was https://limitless-sea-32755.herokuapp.com/adsense, but I've observed it also for rubicon:

in AMSPresenter.php line 26
at Application->Laravel\Lumen\Concerns\{closure}(2, 'fopen(csvOutput1485961973.csv): failed to open stream: No such file or directory', '/app/app/Services/AMSPresenter.php', 26, array('file' => 'csvOutput1485961973.csv'))
at fopen('csvOutput1485961973.csv', 'r') in AMSPresenter.php line 26
at AMSPresenter->echoCsv('csvOutput1485961973.csv') in AdsensePresenter.php line 61
at AdsensePresenter->present(array(array('AD_UNIT_CODE' => '1022077801', 'DATE' => '2017-01-31', 'AD_REQUESTS' => '117', 

this data will be provided by "correspondance" table

• date: provided by AMS.Connect if unique day
• impr. reçues: "NA"
• impr. prises: impr
• revenue: rev
• key: zone
• inventaire: "AMS Market Place"
• ecpm: (revenue/impr prises) * 1000
• annonceur: "sublime"

Create the presenter method for the Adtech Provider

Above it follow an example file with the Adtech Data:
343944_1502_1_1489805144101_csv.txt

Create a library to

• connect and read emails with imap credentials
  • Ref: https://garrettstjohn.com/article/extracting-attachments-from-emails-with-php/
• get emails by filter (recipient or title and date)
  • Ref: http://php.net/manual/pt_BR/function.imap-search.php
• function to get the attachments from email and return content
  • Ref: https://garrettstjohn.com/article/extracting-attachments-from-emails-with-php/

Check why this error occurs on adtech provider for the 'last day':

http://limitless-sea-32755.herokuapp.com/adtech

Changes to the filesystem on one dyno are not propagated to other dynos and are not persisted across deploys and dyno restarts. A better and more scalable approach is to use a shared resource such as a database or queue.

Source: https://devcenter.heroku.com/articles/how-heroku-works#dyno-manager

• criteo: por dia Y-m-d, com start e end dates
• rubicon: por datetime Y-m-dTHⓂ️s, com start e end datetimes. Então o dia corresponde ao intervalo Y-m-d 00:00:00 - Y-m-d 23:59:59
• sublime: por dia, sem permitir selecionar períodos, somente um dia de cada vez

use meumobi favicon. Download the asset and put it on project

In order to compute premium reports for larger period then one day, I suggest to allow processing monthly reports sent by email. The admin could sent the latest report (no matter if it's not the latest month) to amsconnect+latest@... and if we call /adtech?email=amsconnect+latest@... it will process it.

Before API only returns datas from a given day we should loop from start to end dates and merge responses.
Each loop should update the response to add "date" file, not provided by API