see #457

Please see this PR: #201

so cms/djangoapps/search_api/views.py ALLOWED_ORIGIN needs to accept domains using Django's sites framework: https://docs.djangoproject.com/en/2.0/ref/contrib/sites/ (yeah, this refs to Django 2.0, that's ok)

@bryanlandia is the trinity feature branch:
trinityedu/feature/customregistrationfields

ready to be merged into trinity/master and trinity/release?

Right now I'm setting the production branch to this feature branch, but I'd like to get that sorted out before the launch.

Contribute both #873 and appsembler/edx-theme-codebase#147 to edX.

1. Add tests for the Juniper production s3 cache: #907

2. Address Anders comment: #907 (review)

I'm in favor of getting this out quickly to unblock Juniper.

Longer term, we probably want to consider some things:

• instead of adding our stuff directly to openedx.core.storage.ProductionS3Storage, we probably ought to make our own storage class that just subclasses it to override url() and add the explicit caching, then use AppsemblerProductionS3Storage in our config.
• having honeycomb tracing on every call to url() is useful at the moment for debugging/verifying this fix, but is probably more verbose than we're going to want to dial it down later.
• instead of commenting out the setting of the key prefix here and putting a random value in the config in appsembler/configuration#361 maybe we could instead set EDX_PLATFORM_REVISION to a git hash or cloud build revision id during the deploy (with something like an ansible --extra-vars=EDX_PLATFORM_REVISION=$REVISION_ID in the deploy script. That would achieve the same effect of making sure that the cache is cleared on each deploy.

Contribute the work of RED-1223 in #683.

Contribute the fixes below:

A) #884 has a fix for the incorrect get_current_site logic. The fix is contributable.

The bug is resolved by replacing:

Site.objects.get_current() (Django) --> get_current_site() (edX's)

Because the Django get_current returns the default site (with settings.SITE_ID) always.

The edX function is more site aware and returns the current site by using the request.site value which is the desired behaviour and would prevent the HTML emails from showing example.com in the email body as described in RED-1886.

B) #925 fix activation email server error in celery

The error happened because of the incomplete example.com fix in #884

This commit fixes the issue by getting back the mandatory emulate_http_request but now with a better method to know the site using our appsembler.sites.utils helpers.

This should work on celery tasks while printing the correct site domain instead of example.com as it used to do before RED-1886 was fixed.

We're enabling tests a bit by bit starting from basic pep8 (#403) up to the user_api django app (#581).

In this process we're skipping tens of tests. Some of them are important and related to SAML for example.

Those tests are:

• Sometimes marked as @unittest.skip,
• Other times marked as @unittest.expectedFailure,
• Otherwise they could be simply commented out with an inline comment starting usually with # Appsembler: failing test for reason such and such ...

The idea behind skipping those tests is to enable an easy way to run as much tests as reasonably possible within our engineering team capacity constraints. The biggest win from the enabled tests is detecting some regression tests when we make them. Indeed, it is theoretically impossible to detect all regression tests.

• What's the plan for those tests? There's no plan as of writing this issue.
• How to find an exhaustive list of all skipped tests? The only reasonable way is to run all the tests using $ pytest -v on both appsembler/tahoe/develop and on open-release/hawthorn.2 and find the missing tests from the vanilla Hawthorn. It's a cumbersome but reliable method.
• I want to change this, and I want to improve our testing. What can I do? Great! Head out to:
  • Testing Efforts at Appsembler
  • Decision: Require tests for each bug fix or new feature?

#559 seems like a good fix to contribute upstream.

Problem with xblock-utils:ProxyTransNode.merge_translation method

The merge_translation method isn't cleaning up after the merge and leaves stray translations in the following way:

• Calling `i18n.helpers.translate('fr_CA', 'Thank you.') returns the untranslated 'Thank you.' ✔️
• Calling `i18n.helpers.xblock_translate('fr_CA', 'Thank you.') returns 'Merci !' ✔️
• Calling i18n.helpers.translate('fr_CA', 'Thank you.') again returns the the translated 'Merci !' ❌ which means that the merge_translation` didn't clean up after itself

Workaround: Call translate then xblock_translate in the tahoe_translate command.

Long term fix: Work with upstream to fix the cleanup process around translation.activate(...)

This was discovered while working on #948

It's not clear to note which site the CourseEnrollmentAllowed came from, and therefore it's likely that collision might happen between sites.

I'm adding it here until moving it to Jira or somewhere else.

The third_party_auth migration, 0007_samlconfiguration_slo_redirect_url fails when upgrading Ficus to Ginkgo because the field already exists.

One workaround is to fake this migration,

./manage.py lms migrate third_party_auth 0007_samlconfiguration_slo_redirect_url --fake

Then run the migrations as usual

I might be missing something, but ...

https://github.com/appsembler/edx-platform/blob/appsembler/eucalyptus/master/openedx/core/djangoapps/theming/finders.py#L50

looks wrong to me. Shouldn't it append /customer_specific? My customer specific JS files are not being copied to staticfiles; instead, the base edx-platform ones are.

See https://github.com/appsembler/edx-theme-customers/blob/eucalyptus/trinity/lms/static/js/student_account/views/account_settings_fields.js

This is output from the collectstatic handler

(Pdb) path
u'js/student_account/views/account_settings_fields.js'
(Pdb) l
 94  	            handler = self.copy_file
 95  	
 96  	        found_files = OrderedDict()
 97  	        for finder in get_finders():
 98  	            import pdb;pdb.set_trace()
 99  ->	            for path, storage in finder.list(self.ignore_patterns):
100  	                # Prefix the relative path if the source storage contains it
101  	                if getattr(storage, 'prefix', None):
102  	                    prefixed_path = os.path.join(storage.prefix, path)
103  	                else:
104  	                    prefixed_path = path
(Pdb) n
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py(101)collect()
-> if getattr(storage, 'prefix', None):
(Pdb) 
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py(102)collect()
-> prefixed_path = os.path.join(storage.prefix, path)
(Pdb) 
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py(106)collect()
-> if prefixed_path not in found_files:
(Pdb) 
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py(107)collect()
-> found_files[prefixed_path] = (storage, path)
(Pdb) 
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py(108)collect()
-> handler(path, prefixed_path, storage)
(Pdb) s
--Call--
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py(298)copy_file()
-> def copy_file(self, path, prefixed_path, source_storage):
(Pdb) a
self = <django.contrib.staticfiles.management.commands.collectstatic.Command object at 0x7f3f580eea90>
path = js/student_account/views/account_settings_view.js
prefixed_path = eucalyptus-theme-codebase/js/student_account/views/account_settings_view.js
source_storage = <openedx.core.djangoapps.theming.storage.ThemeStorage object at 0x7f3f4b34c950>
(Pdb) 

I'm trying to collect static assets from the Trinity Euc. customer specific theme, and the prefixed path is missing customer_specific.

@natea @bezidejni @bryanlandia @annatpt @emrob (and anybody else I forgot to include) -
Does anybody know if we're still using our Intercom integration with Open edX?
https://github.com/appsembler/edx-platform/tree/appsembler/master/common/djangoapps/appsembler

There was a bit of refactoring on the edx-platform side of things that broke our Intercom functionality in Dogwood. I'm wondering how important that functionality is and whether or not we still use it. I know for a while our testdrives would use the chat widget to start a conversation with potential customers/leads. Is that still the case?

We can remove this line:

and the associated appsembler.txt file from our fork of the edx-platform repo if we pull the contents from that file and add something to our server-vars.yml files like:

EDXAPP_EXTRA_REQUIREMENTS:
  - name: 'git+https://github.com/org/project@branch#egg=egg_name'

We've ported deleted command from Hawthorn:

• #990
• #1067

We should revert both PRs before moving to Koa.

• Contributes #1091

Background

https://appsembler.atlassian.net/browse/BLACK-2163
By the end of 2021 YouTube Deprecated their Timedtext API. This caused problem for our customers to download automatically the transcripts for their YouTube video for their videos in Open edX. Open edX team is aware of this error https://openedx.atlassian.net/browse/TNL-9460, but there was no movement on solving this issue from open edx end.
We tried to solve this by replacing the API with a new YouTube API, but the new API requires OAuth authentication and only works with videos in the same channel and even with corresponding changes in the codebase the Transcripts didn't work.

Our solution

I developed a new API that responds with SRT version of transcripts for a given YouTube video ID for example https://us-central1-penn-state.cloudfunctions.net/youtube-transcript?video_id=AcZZlbWRyUM You can try this with different videos just replace what comes after video_id= with your YouTube video ID.
After receiving the Transcripts from our API we make a call to transcripts/upload to upload the transcript for a given unit component location, and we store the transcripts. To make it visible in Studio and LMS.
The functionality should work as the following video

Filmage.2022-02-28_204527.mp4

#191 introduced a change to make the ENABLE_COURSE_DISCOVERY configurable from the lms.env.json file on devstack. However the solution used another feature.

The suggestion is to refactor it:

FEATURES['ENABLE_COURSE_DISCOVERY'] = ENV_TOKENS['FEATURES'].get('ENABLE_COURSE_DISCOVERY', FEATURES['ENABLE_COURSE_DISCOVERY'])

Contribute the #751 (comment) fix to edX upstream:

RED-1615. The same error is reproducible in open-release/juniper.master which is weird.

TODO

• Figure out why DEBUG = False in tests. This unusual.

  • Apparently it's not a new thing in Django: https://stackoverflow.com/questions/7447134/how-do-you-set-debug-to-true-when-running-a-django-test

That's a great fix. Would you like to push this to upstream @grozdanowski?

Originally posted by @OmarIthawi in #498 (comment)

Test Drive user reported not being able to download CSV grade reports from
http://lms.edxdemo.appsembler.com/courses/course-v1:TheSoftwareGuild+SG101+2016_Summer/instructor#view-data_download

and I can confirm. Links are like:

• file:///tmp/edx-s3/grades/course-v1%253ATheSoftwareGuild%252BSG101%252B2016_Summer/TheSoftwareGuild_SG101_2016_Summer_problem_grade_report_2016-09-06-1816.csv
• file:///tmp/edx-s3/grades/course-v1%253ATheSoftwareGuild%252BSG101%252B2016_Summer/TheSoftwareGuild_SG101_2016_Summer_grade_report_2016-09-06-1815.csv

On prod AMC in django shell, I did:

from django.conf import settings
settings.GRADES_DOWNLOAD
{u'STORAGE_TYPE': u'S3', u'ROOT_PATH': u'grades', u'BUCKET': u'edx-amcalpha-grade-bucket'}

Not sure why the method on the S3ReportStore for returning links is returning file:/// links but it is. Configuration looks correct.

https://github.com/appsembler/edx-platform/blob/appsembler/amc/testdrive/lms/djangoapps/instructor_task/models.py#L316

USE_S3_FOR_CUSTOMER_THEMES is somewhat a very bad method of choosing a storage class.

In devstack it is mixed with static files, while it should be a media file.

The proper solution for doing such operation is to use a configurable storage class:

# production
CUSTOMER_THEMES_STORAGE_CLASS = 'storages.backends.s3boto.S3BotoStorage'
CUSTOMER_THEMES_STORAGE_PARAMS = {
  'location': 'customer_themes',
}

# dev
CUSTOMER_THEMES_STORAGE_CLASS = 'django.core.files.storage.FileSystemStorage'
CUSTOMER_THEMES_STORAGE_PARAMS = {}

This way it's fully configurable and the code won't care where it's running.

Also, this needs a cash busting mechanism. So perhaps at save, the css file should be suffixed with some GUID:

appsembler-the-red.staging-tahoe-hawthorn.appsembler.com.e044f8e7-e4bd-44b2-9c46-6af817b1ebbd.css

Deleting all appsembler-the-red.staging-tahoe-hawthorn.appsembler.com.* files before that wouldn't be a bad idea.

In #257 we added few tests for certificates. test_visible_only_to_course_staff is now broken after the Hawthorn merge.

$ pytest lms/djangoapps/instructor/tests/test_certificates.py::CertificatesInstructorDashTest::test_visible_only_to_course_staff

Someday we should fix it 😅

The i18n command in #948 is a community-wide problem.

The migration, https://github.com/appsembler/edx-platform/blob/appsembler/ginkgo/develop/common/djangoapps/third_party_auth/migrations/0007_samlconfiguration_slo_redirect_url.py, adds the 'slo_redirect_url ' field that already exists in the data imported from the Ficus instance.

Workaround is to

./manage.py lms migrate third_party_auth 0007_samlconfiguration_slo_redirect_url --fake --settings=<aws|devstack_appsembler>

Original PR:

Allow importing arbitrary urlpatterns from Python modules

• write tests
• reach out to edX and see if they're interested in this functionality
• generalize implementation

Both the Site Configuration Django app and the edx-organizations repository have substantial hacks to enable Tahoe multi-tenancy.

This is a technical debt that we should address by moving those hacks into a new app, let's call it tahoe-sites, or even to the existing appsembler/sites Django application.

We've had a lot of migration issues and RuntimeError: Model class organizations.models.Organization doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS. exceptions to keep it for the next release.

• Refactoring the edx-organizations package doc
• RED-46 - We need to review and refactor our Organization, Sites and Users mapping

Contribute: #983 (RED-2361)

Problem description: Certificates have a Django signal, when a certificate object is created and has a passing status (downloadable or generating) but the course or the grading change, and it ends up in a non passing grade for the learner, the certificate change automatically to notpassing.
That's fine except when there is an Exception certificate created from the instructor dashboard, then it should never change to notpassing.

Contribute both #953 and #956

This exception below is raised because there's an attempt to make the ModuleI18nService fallback to itself due to the use of the magical ModuleI18nService.__getattr__ which returns django.utils.translation._fallback when asked for ModuleI18nService._fallback. RED-2263.

This creates a infinite recursion that needs to break.

File "xblockutils/templatetags/i18n.py", line 41, in merge_translation
    translation.merge(i18n_service)
...
File "python3.5/gettext.py", line 169, in add_fallback
    self._fallback.add_fallback(fallback)
RecursionError: maximum recursion depth exceeded"

Tests

I wasn't able to reproduce the issue via scripts or tests. The issue has extended beyond the regular bug timebox of 4 hours.

Contribute #964 upstream.

Fixes the exception below by using exc_info=True to print a full stack trace.

AttributeError /auth/complete/{backend}/
'KeyError' object has no attribute 'message'

Contribute the PRs below:

1. #959

   RED-2176: This fixes a bug with Course Access Groups which relies on the USER_ACCOUNT_ACTIVATED signal to be sent after activating the account.

2. #1017

   Fixes RED-2176.