doctoctoc / doctoctocbot Goto Github PK

Bots should follow verified users who are members or potential members.

top_statusid_lst(hourdelta) & help_statusid_lst(hourdelta) do not take into account human moderation & they exclude questions spread in a thread if the hashtag is not present in the root status.

Needs rewriting.

File "/code/src/bot/onstatus.py", line 29, in get_status_communities
for c in Community.objects.filter(active=True).remove(community):
AttributeError: 'QuerySet' object has no attribute 'remove'

2021-02-10 11:04:13,055 - celery.app.trace - ERROR - Task moderation.tasks.poll_moderation_dm[927b139b-93d4-4495-a12a-13c918bb6e57] raised unexpected: AttributeError("'QuerySet' object has no attribute 'remove'")
Traceback (most recent call last):
File "/code/src/bot/onstatus.py", line 63, in triage_status
tweetdj = Tweetdj.objects.get(statusid=status_id)
File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 406, in get
raise self.model.DoesNotExist(
conversation.models.Tweetdj.DoesNotExist: Tweetdj matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 650, in protected_call
return self.run(*args, **kwargs)
File "/code/src/moderation/tasks.py", line 301, in poll_moderation_dm
process_cat_mi(cat_mi, moderated_su_mi, moderator_su_mi, mod_mi)
File "/code/src/moderation/tasks.py", line 355, in process_cat_mi
triage_status(status_id, community)
File "/code/src/bot/onstatus.py", line 68, in triage_status
status = get_status_communities(status_id, community)
File "/code/src/bot/onstatus.py", line 29, in get_status_communities
for c in Community.objects.filter(active=True).remove(community):
AttributeError: 'QuerySet' object has no attribute 'remove'

For instance: https://twitter.com/bebetologue/status/1079382611238240256

hashtag0 is True, but the hashtag was present only in the quoted tree, which is not what we wanted

landing.templatetags.landing_tags.hashtag_lst

hashtag list is printed with 'and', we also need 'or'

upgrade opengraph tags on moderator list page

moderation.templatetags.auth_extras.is_moderator is based on old moderator category but moderators are now defined by table moderation.Moderator

status containing both tracked hashtags (doctocotc & docstoctoc) show up as having only one or the other.
hashtag retained seems to be the last one occurring in the tweet:

"doctoctoc ... docstoctoc" -> docstoctoc

"docstoctoc ... doctoctoc" -> doctoctoc

text shows shortened url like https://t.co/sMEITEXASU
it should show the real hyperlink

• Currently some modules of the project contain hard coded constant variables that should be shared across the different modules and that are, for some, already present in the configuration file. For example the 'addtolist' module contain a hard coded 'bot_id' variable.
• Currently several modules implement their own way to load the configuration file, instead they should all use the 'getConfig' method of the 'cfg' module. For example in the 'addtolist' module on line 14 to 16.
• It seems there was a move to use json based configuration file, maybe that should be reconsidered (see footnotes 1 & 2) ?

1. https://softwareengineering.stackexchange.com/questions/351126/how-bad-of-an-idea-is-it-to-use-python-files-as-configuration-files
2. https://revelry.co/json-configuration-file-format

moderation: attempt to send DM systematically after follow or follow request to invite user to self verify and give other info

use django-money?

AuthAlreadyAssociated at /oauth/complete/twitter/

Environment:

Request Method: GET
Request URL: http://localhost:XXXX/oauth/complete/twitter/?redirect_state=xxx&oauth_token=xxx&oauth_verifier=xxx

Django Version: 2.0.13
Python Version: 3.7.2
Installed Applications:
['users',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.sites',
'django.contrib.staticfiles',
'rest_framework',
'doctocnet',
'landing',
'social_django',
'moderation',
'conversation',
'bot',
'dm',
'charts',
'display',
'timeline',
'crispy_forms',
'paypal.standard.ipn',
'crowdfunding',
'registration',
'django_countries',
'customer',
'debug_toolbar',
'status',
'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'social_django.middleware.SocialAuthExceptionMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware']

Traceback:

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/core/handlers/exception.py" in inner
35. response = get_response(request)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
128. response = self.process_exception_by_middleware(e, request)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
126. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
54. return view_func(*args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_django/utils.py" in wrapper
49. return func(request, backend, *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_django/views.py" in complete
33. *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/actions.py" in do_complete
43. user = backend.complete(user=user, *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/backends/base.py" in complete
40. return self.auth_complete(*args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/utils.py" in wrapper
259. return func(*args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/backends/oauth.py" in auth_complete
184. return self.do_auth(access_token, *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/utils.py" in wrapper
259. return func(*args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/backends/oauth.py" in do_auth
195. return self.strategy.authenticate(*args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_django/strategy.py" in authenticate
107. return authenticate(*args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/contrib/auth/init.py" in authenticate
70. user = _authenticate_with_backend(backend, backend_path, request, credentials)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/django/contrib/auth/init.py" in _authenticate_with_backend
116. return backend.authenticate(*args, **credentials)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/backends/base.py" in authenticate
80. return self.pipeline(pipeline, *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/backends/base.py" in pipeline
83. out = self.run_pipeline(pipeline, pipeline_index, *args, **kwargs)

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/backends/base.py" in run_pipeline
113. result = func(*args, **out) or {}

File "/home/elkcloner/.virtualenvs/doctoctocbot/lib/python3.7/site-packages/social_core/pipeline/social_auth.py" in social_user
23. raise AuthAlreadyAssociated(backend, msg)

Exception Type: AuthAlreadyAssociated at /oauth/complete/twitter/
Exception Value: This account is already in use.

Test for protected (API get credentials or db) before IncomingFriendship to save API?

SyntaxError: Missing parentheses in call to 'print'

http://doctoctoc.net/financement/progress/
http://doctoctoc.net/financement/fund/
Should display default html if the number of projects is 0 in database.

CC @Z4rsi0

tweet.favourite()

add bool field option in Community model

Check if follower before sending category DM

I run the bot on Debian Stretch. I got a warning following the introduction of the last pieces of code that depend on sqlalchemy:

/home/xxx/.local/lib/python3.5/site-packages/psycopg2/init.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.
""")

Question:

Should I use

• Debian package python3-psycopg2
• pip3 install psycopg2
• pip3 install psycopg2-binary

I guess this psycopg2 dependency comes from the use of JSONB type in model/model.py:
from sqlalchemy.dialects.postgresql import JSONB

Any idea @natolh?

donors with repeat donations appear multiple times
donor should appear only once in the category corresponding to the sum of his donations

See https://github.com/DocTocToc/doctoctocbot/wiki/Analyse-thread

Traverse the thread towards the root tweet (1st tweet of the thread), retweet root tweet only if question mark present in thread and all tweets up to the root are authored by the same user.

We can't get social links data from protected accounts at once.
We need to wait until they accept our follow request before starting the moderation process.

& is displayed as "&" on scroll

If you delete the category of a user who was already moderated through a DM, the category reappears after the next call to poll_moderation_dm()

In Order to comply with GDPR, allow user to see and ask for a review of his professional category in the user views.

Only the original status should be retweeted, not answers to another status or quoted-retweets containing the hashtag.

function users.utils.follower_level is used to determine the API access on the home page
how do we deal with rate limit?
now in the event of a rate limit, the gunicorn thread is paused and times out... :(
we need to used cached value of update_social_ids and update asynchronously in a celery task

On Twitter sign in, update Django username if Twitter username has changed.

https://www.codementor.io/siddhism/using-bower-to-manage-static-files-with-django-h0kd5bq29

moderation.moderate.addtoqueue
check if queue for user already exists

users are added to database some time before we create a moderation
their username can change
then the moderation DM contains the old @, which cannot be clicked
moderation is not possible

fix: update profile systematically before each moderation request

Look for question mark in the whole thread, not only in the first tweet.

getting 404 error when downloading profile pic

if a user didn't upload a profile pic, her PP is the default one
default_profile_400x400.png

it doesn't show up in displayed tweets

web app: "follow thread" option to be notified of answers

https://blog.ganssle.io/articles/2019/11/utcnow.html

in Docker image pip is pinned to RUN pip install pip==21.0.1
because of an old bug with packges compatibility

we should try to use the lastest version of pip

This thread containing a question mark and the hashtag, authored by a MD in the list, was not RTed. Why?
https://twitter.com/ElliotReid_MD/status/1112822277442297862

Implementing a user database to store bot's contributors will allow :

• Exceed the 5.000 members limit of Twitter lists (Twitter lists are used to keep track of contributors).
• Allow to store more information about each user locally (currently the retweet script access only the contributors Twitter ID via a text file).

We could try to enable Eventlet for celery workers to improve performance without spending more on the VPS. See http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html