As a GH action:

on:
  workflow_dispatch:
    inputs:
      force_generate_all:
        description: 'Force generate all posts'
        required: false
        type: boolean
        default: false

      - name: Run script
        run: python generate_posts.py --force_generate_all ${{ github.event.inputs.force_generate_all }}

Bayesian beagle - Vlogger: Make Your Dream A Vlog

Vlogger AI system creates complex vlogs from text using a Large Language Model and video diffusion model. State-of-the-art results.

https://bayesian-beagle.netlify.app/posts/vlogger_make_your_dream_a_vlog/2024-01-17-vlogger_make_your_dream_a_vlog

Bayesian beagle - Prompt Weight Experiments for LLM Instruction Fine-Tuning

Study examines impact of prompt token classification loss weighting on LLaMA models fine-tuned on instruction tasks. Results vary based on dataset length.

https://bayesian-beagle.netlify.app/posts/prompt_weight_experiments_for_llm_instruction_fine_tuning/2024-01-24-prompt_weight_experiments_for_llm_instruction_fine_tuning

Issue Description

Currently, our static website built with Quarto is generating pages using a .jsonl file, where each record corresponds to an Arxiv ID. We aim to replace this with a SQLite database to enhance scalability and maintainability.

Steps to Implement

1. Create a SQLite Database

• Design Database Schema: Reflect the structure of the .jsonl file in the database schema.
• Initialize Database: Create a SQLite database with tables based on the schema.

2. Data Migration

• Migration Script: Develop a Python script to migrate data from the .jsonl file to the SQLite database.
• Data Integrity: Ensure all data is accurately transferred.

3. Update Python Scripts for Data Retrieval

• Modify Existing Scripts: Adapt scripts that currently read the .jsonl file to instead interact with the SQLite database.
• Implement Database Queries: Use Python libraries like sqlite3 to handle database operations.
• Fetch Records by ID: Ensure the updated scripts can retrieve specific records using Arxiv IDs.

4. Testing

• Test Data Retrieval: Thoroughly test the updated scripts for accurate database interactions.
• Validate Page Generation: Compare newly generated pages with existing ones for consistency.

5. Documentation

• Update Documentation: Reflect the transition from .jsonl to SQLite in the project documentation.
• Setup Instructions: Include guidelines for database setup and script execution.

6. Deployment

• Merge and Deploy: After testing, merge the changes into the main branch and update the live website.

Additional Considerations

• Performance: Optimize query performance for large datasets.
• Backup and Recovery: Implement a strategy for database backup.
• CI/CD Pipeline: Update CI/CD processes to include the new database setup.

Request for Contributions

Contributions are welcome! If you're interested in helping with this transition, please comment on this issue or submit a pull request.

#15

Currently use approx word counts by whitespace before sending to API; will truncate.

Also skips if OpenAI comes back with error.

Need a better solution to check with OpenAI's tokenizer.

Perhaps create a third file that is known false positives

Possible idea: summarization quality

Here are the main functions of the code:

1. convert_to_folder_name: This function takes a string as input and converts it to a folder name format by replacing spaces, slashes, question marks, colons, commas, and hyphens with underscores. It uses the translate method of strings with a translation table to perform the replacement.

2. create_qmd_file: This function takes an example dictionary and an output folder path as input. It extracts relevant information from the example dictionary, such as the title, publish date, and image URL. It also creates a folder name based on the title using convert_to_folder_name. It then creates a QMD file path by combining the output folder path, folder name, and current date. It checks if the file already exists and returns if it does. Otherwise, it uses Jinja2 templating to render a QMD file content with the example data. It creates the output sub-folder if it doesn't exist and writes the rendered content to the file.

3. generate_qmd: This is the main command of the script. It takes an input JSONL file path and an output folder path as input. It opens the JSONL file and iterates over each line. It loads the line as a JSON object and calls the create_qmd_file function to generate a QMD file for each example.

The code uses the typer package for command-line interface (CLI) parsing. It defines a CLI command generate_qmd that accepts the input JSONL file and output folder as arguments.

Suggestions for improving the code for programming best practices:

1. Use type hints: Add type hints to function and variable declarations to improve code readability and maintainability.

2. Use docstrings: Add docstrings to functions to provide a description of their purpose and usage.

3. Handle exceptions: Add exception handling to catch and handle any potential errors that may occur during file operations, such as opening, reading, or writing files.

4. Separate concerns: Consider breaking down the create_qmd_file function into smaller, more focused functions to improve code organization and readability.

5. Use pathlib functions: Instead of manually constructing file paths using string concatenation, use the pathlib module's functions to manipulate file paths. For example, use Path(output_folder) / folder_name / file_name to create the file path.

6. Use f-strings: Instead of concatenating strings using the + operator, use f-strings to improve string formatting and readability. For example, use f"File saved: {file_path}" instead of "File saved: " + file_path.

7. Use a logger: Instead of printing messages directly to the console, consider using a logging library like logging to log messages with different levels of severity.

8. Error handling for existing directories: Add error handling to check if the output folder already exist s and whether it is a valid directory before writing files to it.

9. Unit tests: Add unit tests to verify the correctness of the functions and handle edge cases.

Set up local runner https://github.com/wesslen/bayesian-beagle/settings/actions/runners

For Mixtral, found a `tldr' with code:

"subtitle": "Fine-tuning embeddings improves item retrieval in conversational recommendation agents.\n```python\n\n```"

Prevented quarto from rendering. Need to keep an eye out.

• Create Python script to summarize monthly posts by category
• Create new GH Action to run every month