proversity-org / edx-analytics-pipeline Goto Github PK

Test the AnswerDistributionWorkflow (performance) task(s)

Hive metastore credentials/permissions are not right. Fix them.

Connect to ElasticSearch for module engagement tasks to work as per Gabe's advice

Created date not being pulled through by ImportEnrollmentsIntoMysql task. Needed for dashboard to load.

Test CourseActivityWeeklyTask

Update master for production

Create LearnerCourseEngagement task including activity/engagement per unit of course.

Test CourseActivityDailyTask

Setup and deploy pipeline on a server for harambee,
setup insights and data api on prod server.

Progress:
Pipeline server created and can execute the hadoop pi job without error, ImportsEnrollments job venv has been created but there is an error coming up from luigi staging missing parameter about defaul_log_level or something.

Insights installed and database migrated: requires SSO setup.
data api installation failed due to database rights of the insights user on the reports database. re-try tonight.

@renevatium shhht but fyi. sofar we looking mkay :)

https://proversity-team.slack.com/archives/C24145UFM/p1497331744499891

Test working tasks with s3 input. Need to update edx-deployment/analytics/override.cfg

Sync all imported hdfs data to an s3 bucket for backup.

opaque_key_util.py:33 - Unable to parse course_id

https://proversity-team.slack.com/files/andrea/F5W4K1SHE/pasted_image_at_2017_06_20_10_01_pm.png

Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

Aggregate vertical module acceptance data

CourseActivityDailyTask doesn't work properly for Insights.

Test ModuleEngagementWorkflowTask

Luigi (on EMR) not creating password stores, debug this.

Upgrade luigi from 1.0.17 to 1.0.22. v1.3.0 works but has config issues.

Test InsertToMysqlCourseEnrollByCountryWorkflow task

Some of the tasks on harambee analytics pipeline server has been failing.

find and fix issues with.
"ImportEnrollmentsIntoMysql, InsertToMysqlAllVideoTask and ModuleEngagementWorkflowTask"

InsertToMysqlAllVideoTask and ModuleEngagementWorkflowTask are failing with the could not override table with empty result set error, looks like one of the database tables are not being transported due to this, either find and add a setting or just force it somehow. donno really.

ImportEnrollmentsIntoMysql fails due to the enrollment event task not being run for the certain period. i.e. the event task is not being run before the insert task

Increase Hadoop/Yarn/Hive memory allocation using deployment configuration settings.