oduwsdl / raintale Goto Github PK

A model for handling presets can be found in the existing tellstory command.

The challenge for the GUI will be displaying these presets when a user selects them. We also do not want a user to be able to tell a story with Twitter or Facebook as those will be in their own scripts.

If the user provides the Twitter template for the "Create Story with Template" script, the script will fail with no useful error message currently. We have to make sure a meaningful error message is provided in such a scenario.

We can update the storyteller code to account for this.

https://github.com/oduwsdl/raintale/blob/master/raintale/storytellers/storyteller.py

if template_contents[0:34] == '{# RAINTALE MULTIPART TEMPLATE #}\n':
 display error message

As identified by @ato, an organization may only wish to accept Raintale templates from trusted individuals.

This will require two actions:

1. when completing the documentation for the Raintale GUI, we will have to provide at least a link to instructions for creating a superuser and adding users via the Django administrator interface
2. we will need to disable the Registration link on the main page if a user requests it during or after installation

Addressing #1 will not be accomplished until we complete the Raintale GUI documentation.

Addressing #2 requires that an installer disable the /accounts/register/ endpoint with the following steps.

After reviewing Wooey's source code and testing locally, I've determined that the installer can disable that by setting WOOEY_REGISTER_URL to None. inside settings/user_settings.py.

Finally, to ensure that only authenticated users are allowed to execute Raintale scripts, the installer will set WOOEY_ALLOW_ANONYMOUS from settings/user_settings.py to False.

So, to summarize, install-gui.sh should:

• accept a flag indicating that the user wishes to install the system without registration
• if that flag is set, it will set WOOEY_REGISTER_URL to None inside settings/user_settings.py
• if that flag is set, it will set WOOEY_ALLOW_ANONYMOUS to False inside settings/user_settings.py

Raintale already supports the link and text story element types in JSON input. An image story element type would allow users to handle images in a special way with their templates. It should not be handled that differently from text. This would just give users an additional key with which to control the look of output in templates.

A potential additional key is video. Because the user may need to other key types, this may require more thought than just adding if statements.

Following the postgres instructions after installing from RPM on a fresh CentOS install running set-raintale-database.sh fails with this error:

changing to /opt/raintale/raintale-gui/../raintale_with_wooey
Traceback (most recent call last):
  File "/opt/raintale/raintale-virtualenv/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 25, in <module>
    import psycopg2 as Database
ModuleNotFoundError: No module named 'psycopg2'

Installing the system python38-psychopg2 package did not help. Installing it with pip inside the raintale-virtualenv resolves the error:

sudo dnf group install -y 'Development Tools'
sudo dnf install -y python38-devel postgresql-devel
sudo /opt/raintale/raintale-virtualenv/bin/pip install psycopg2

Recommend either adding this step to the section about configuring raintale for postgres or shipping psycopg2 in the package.

As reported by @ato, the help text for the story file argument in Raintale scripts is not as descriptive or user-friendly as it could be.

A suggestion for a replacement: "An input file containing URLs of archived web pages (e.g., mementos, captures, URI-Ms) for use in the story)."

set-raintale-database.sh creates a file /etc/raintale.conf containing the database details supplied by the command-line and password prompt. It then adds a section to user_settions.py that references these as environment variables. However it never actually sets these as environment variables when calling manage.py migrate, so the placeholder values like "raintale_password" are used instead of the values entered by the user. The migrate command therefore fails with:

psycopg2.OperationalError: FATAL:  password authentication failed for user "raintale"

I worked around this by adding this to set-raintale-database.sh before the migrate command:

source "${raintale_conf}"
export DATABASE_NAME DATABASE_PORT DATABASE_HOST DATABASE_USER DATABASE_PASSWORD

Sometimes MementoEmbed has a transient error when processing memento content. Right now Raintale crashes and instructs the user to just rerun the program if something goes wrong. It should, instead, retry n times before giving up, like the OTMT does.

The date and version in CITATION.cff must match the date and version used elsewhere in the software.

Raintale was originally built without caching support because it only issued calls to MementoEmbed. Storytellers like the video storyteller require making additional HTTP(S) requests to web archives. Caching may also help with testing.

The requests-cache library is used elsewhere in the DSA toolkit for this purpose.

This is different from including an imagereel in a tweet. This would be a single tweet that contains a video story created from all mementos submitted as a list of URI-Ms.

As suggested by @ato, it would be cool if the user could, instead of submitting a file containing a list of URI-Ms, they could instead paste a list of URI-Ms into a textarea as input.

The Raintale GUI production environment needs more robust infrastructure than provided by our Docker installation.

By default Wooey/Django uses SQLite as a database. SQLite allows for concurrent reads, but locks and queues concurrent writes. This is great for a single user, but not so good for a system that will satisfy multiple users. Django supports changing the backend database to something else.

Based on the Wooey documentation, it will use the default database for queuing, but RabbitMQ is "a more robust system" than the default database because RabbitMQ is specially designed for this queuing.

We should at least point to documentation that helps users set this up. We could also update our installer to alter the correct Django settings files as needed by administrators.

After enabling postgres support according to the documentation jobs get stuck with status "Waiting" and raintale-celery service logs the following error:

Nov 16 07:52:32 localhost.localdomain celery[71877]: psycopg2.OperationalError: FATAL: password authentication failed for user "raintale"

Fixed by adding this line to [Service] section in /etc/systemd/system/raintale-celery.service:

EnvironmentFile=/etc/raintale.conf

and then restarting with:

sudo systemctl daemon-reload
sudo systemctl restart raintale-celery

We can finish the Create Video Story script by adapting what we've learned from the existing Create Story From Template script. Much of the structure is the same.

The storyteller class we need to use is VideoStoryTeller as found in raintale/storytellers/video.py.

Video stories do not support a template, so the code does not need to support one as input.

A user can view their Raintale story if they know which file link to click on, but this is not friendly.

A better way would be to have a "View Story" button as @himarshaj and I have discussed. It would have to be aware of how to handle different story types, for example:

• for Create Story From Preset, Create Story From Template, and Create Video Story this button will link to the file that was generated as part of the storytelling process
• for Tell Story With Facebook and Tell Story With Twitter this button will link to the associated social media post

We need to figure out how Django's template engine can tell the difference. Maybe there is a control flow statement we can use that will change behavior based on the job type. A second challenge is how to read the URL from the output in the case of social media stories so we can alter the behavior of this button.

The template controlling this behavior is in raintale-gui/templates/jobs/job_view.html.

Refer to oduwsdl/hypercane#51.

The code already exists for doing this in Raintale. It just needs to be exposed so Wooey can render its GUI properly.

We need to update the Tell Story With Twitter script to apply the TwitterStoryTeller class to generating these stories.

I know that some of this work has been accomplished already because we already have test stories on Twitter. It likely just needs to be checked in.

Raintale templates are based on Jinja2 templates. With the addition of preferences, Raintale templates are no longer Jinja2 compatible, but Jinja2 filters would still be useful to users.

The differences:

• Preferences instruct Raintale how to generate the value for a variable. Example: choosing the 3rd best sentence from the document.
• Filters instruct Jinja2 how to alter the value of a variable after creation. Example: converting the text to uppercase.

Raintale should still support Jinja2 filters.

Preferences are handled like so:
{{ variable|prefer key=value,key2=value2 }}

Jinja2 filters could be added to the mix like so:
{{ variable|prefer key=value,key2=value2|filter1|filter2(arg1, arg2)|filter3 }}

...which I think is ~8:31. The clock in the raintale logo looks like the familiar 10:10. If there is meaning to that, disregard this suggestion.

The code already exists for doing this in Raintale. It just needs to be exposed so Wooey can render its GUI properly.

We need to update the Tell Story With Facebook script to apply the FacebookStoryTeller class to generate these stories.

Once #28 is complete, its code could be used as a model for how to do this with FacebookStoryTeller.

Use the same CSS solution as the one from oduwsdl/hypercane#56.

The Wooey documentation states that users start it by running both Django and Celery. The commands they provide for starting Wooey are:

# celery -A your_project_name worker -c 1 --beat -l info
# python manage.py runserver

We've been trying to determine why jobs in Wooey appear to be blocking and thus only execute sequentially. The problem is with the -c 1 argument, which tells Celery to only start 1 process to process the queue. If we remove -c 1 then it will start a number of processes equal to the number of CPUs present in the system, allowing for Wooey (and Raintale) to execute more than one process at a time.

We need to remove -c 1 from the GUI startup script in start-gui.sh:

For reference, I have already done this with Hypercane's GUI startup.

Wooey does not currently support Django 3.2.5 and the Raintale GUI installer downgrades Django to 3.1.8 so that our system will work. The author of Wooey assures me that a fix will be included in the next Wooey version. Once that version is available, we need to remove the lines downgrading Django from our installer.

Due to issues downloading //fonts.googleapis.com/css?family=Pacifico, Firefox does not render Raintale's WUI for quite a long time.

In Hypercane, removing line 14 from hypercane-gui/templates/base.html addressed the issue, as mentioned in oduwsdl/hypercane#61.

The Raintale WUI needs to be documented once completed. The documentation should include screenshots and descriptions of how to use Raintale as well as how to install the system.

They are confusing to us and will likely be so for users. Until we can articulate how to use them, we should remove them.

We just need someone to remove the lines from raintale-gui/templates/jobs/job_view.html:

This is a request for a new template to be included with Raintale.

Storyteller: Twitter

Template code is below:

{# RAINTALE MULTIPART TEMPLATE #}
{# RAINTALE TITLE PART #}
{{ title }}

{% if generated_by is not none %}Story By: {{ generated_by }}{% endif %}

{% if collection_url is not none %}{{ collection_url }}{% endif %}
{# RAINTALE ELEMENT PART #}

{{ element.surrogate.title }}

{{ element.surrogate.memento_datetime }}

{{ element.surrogate.urim }}

{# RAINTALE ELEMENT MEDIA #}
{{ element.surrogate.imagereel }}

set-raintale-database.sh hangs after printing:

verifying that database is empty
discovered 0 tables in the database
database is empty, continuing
writing database information to /opt/raintale/raintale-gui/../hypercane_with_wooey/hypercane_with_wooey/settings/user_settings.py

It appears to be stuck with this grep command reading from stdin:

This script should take into account the lessons learned from #19, #20, and #21.

Ideally, we would create an RPM install for RedHat-based systems and a DEB install for Debian-based systems, but that may be a bit too much to test for the duration of the IIPC Grant. A tarball containing the necessary files and an installation script is likely enough. To make it administrator-friendly, we could apply Makeself as well. This way the user can download a single file, execute it, and it will extract our content, execute our script, and start up the Raintale GUI.

Wooey has a confusing setup with respect to Downloads. We will be implementing a "View Story" button in #26 which will make the existing Download button redundant. We should remove the Download button so that it does not confuse users.

The code to do so is here:

writing database information to /opt/raintale/raintale-gui/../hypercane_with_wooey/hypercane_with_wooey/settings/user_settings.py
/opt/raintale/raintale-gui/set-raintale-database.sh: line 118: /opt/raintale/raintale-gui/../hypercane_with_wooey/hypercane_with_wooey/settings/user_settings.py: No such file or directory

I assume that should be raintale_with_wooey not hypercane_with_wooey. There are some other references to hypercane in the script that probably don't belong:

Our existing start-gui.sh and stop-gui.sh scripts work well for Docker deployment, but, as noted by @ato, we need something more robust for general server installation.

@ato has provided a Gist here that provides this capability:
https://gist.github.com/ato/69fc6952ce57d329a402a6f9f128f6e1

We need to include these as separate files in the raintale/raintale-gui/ folder to be included in a future installer. Several people, including Olga from the IIPC, have asked that we attempt to make services like the Raintale GUI easier to install, and @ato's systemd work goes a long way to making this happen.

We will have to independently test this on another systemd-capable machine as well.

Per feedback from @ato, if we allow users to specify an output file, they could abuse the server and overwrite existing system files. We really do not need to provide this option. To avoid this security issue, we should do two things for all Raintale scripts that have output files:

1. remove the option of specifying an output filename from all GUI scripts - this addresses the security concern
2. make the default output filename output.dat or story-file.dat with an appropriate extension - I'm leaning toward story-file.dat or story.dat or something like that -- just make sure it is descriptive and matches the work done in #26

This should not affect stories that use social media because there is no output filename.