Giter VIP home page Giter VIP logo

riven's Introduction

GitHub Repo stars Issues License Contributors Discord

Plex torrent streaming through Real Debrid and 3rd party services like Overseerr, Mdblist, etc.

Services currently supported:

Service Supported
Real Debrid
Plex
Overseerr
Mdblist
Trakt
Jackett
Plex Watchlist RSS
Torrentio
Orionoid
Jackett
Listrr

| and more to come!

Check out out Project Board to stay informed!

Please add feature requests and issues over on our Issue Tracker or join our Discord to chat with us!

We are constantly adding features and improvements as we go along and squashing bugs as they arise.


Table of Contents


ElfHosted

ElfHosted is a geeky open-source PaaS which provides all the "plumbing" (hosting, security, updates, etc) for your self-hosted apps.

Important

ElfHosted ❤️ Riven 100%

Riven is an "Elf-icial" app in the ElfHosted app catalogue - A whopping 😻😻 100% 😻😻 of your subscription goes directly to Riven developers, who can usually be found in the #elf-riven channel in the ElfHosted Discord Server.

Curious how it works? Here's an explainer video!

Tip

ElfHosted "Infinite Streaming" bundles

Riven is pre-packaged with Plex, Zurg, and symlinks, and ready-to-go, in these convenient bundles:

Self Hosted

Docker Compose

Copy over the contents of docker-compose.yml to your docker-compose.yml file.

Note

You can check out the docker-compose-full.yml file to get an idea of how things tie together.

Then run docker compose up -d to start the container in the background. You can then access the web interface at http://localhost:3000 or whatever port and origin you set in the docker-compose.yml file.

Tip

On first run, Riven creates a settings.json file in the data directory. You can edit the settings from frontend, or manually edit the file and restart the container or use .env or docker-compose environment variables to set the settings (see .env.example for reference).

What is ORIGIN ?

ORIGIN is the URL of the frontend on which you will access it from anywhere. If you are hosting Riven on a vps with IP address 123.45.67.890 then you will need to set the ORIGIN to http://123.45.67.890:3000 (no trailing slash). Similarly, if using a domain name, you will need to set the ORIGIN to http://riven.example.com:3000 (no trailing slash). If you change the port in the docker-compose.yml file, you will need to change it in the ORIGIN as well.

Running outside of Docker

To run outside of docker you will need to have node (v18.13+) and python (3.10+) installed. Then clone the repository

git clone https://github.com/rivenmedia/riven.git && cd riven

and open two terminals in the root of the project and run the following commands in each.

First terminal:

cd frontend
npm install
npm run build
ORIGIN=http://localhost:3000 BACKEND_URL=http://127.0.0.1 node build

Read above for more info on ORIGIN.

Second terminal:

pip install poetry
poetry install --without dev
poetry run python backend/main.py

Symlinking settings

rclone_path should point to your rclone mount that has your torrents on your host.

library_path should point to the location of the mount in plex container

    "symlink": {
        "rclone_path": "/mnt/zurg",
        "library_path": "/mnt/library"
    }

Plex libraries that are currently required to have sections:

Type Categories
Movies movies, anime_movies
Shows shows, anime_shows

Note

Currently, these Plex library requirements are mandatory. However, we plan to make them customizable in the future to support additional libraries as per user preferences.


Development

Welcome to the development section! Here, you'll find all the necessary steps to set up your development environment and start contributing to the project.

Prerequisites

Ensure you have the following installed on your system:

  • Node.js (v18.13+)
  • Python (3.10+)
  • Poetry (for Python dependency management)
  • Docker (optional, for containerized development)

Initial Setup

  1. Clone the Repository:

    git clone https://github.com/rivenmedia/riven.git && cd riven
  2. Install Backend Dependencies:

    pip install poetry
    poetry install
  3. Install Frontend Dependencies:

    cd frontend
    npm install
    cd ..

Using make for Development

We provide a Makefile to simplify common development tasks. Here are some useful commands:

  • Initialize the Project:

    make
  • Start the Development Environment: This command stops any previous containers, removes old images, and rebuilds the image using cached layers. Any changes in the code will trigger a rebuild.

    make start
  • Restart the Container:

    make restart
  • View Logs:

    make logs

Development without make

If you prefer not to use make and Docker, you can manually set up the development environment with the following steps:

  1. Start the Backend:

    poetry run python backend/main.py
  2. Start the Frontend:

    cd frontend
    npm run dev

Additional Tips

  • Environment Variables: Ensure you set the ORIGIN environment variable to the URL where the frontend will be accessible. For example:

    export ORIGIN=http://localhost:3000
  • Code Formatting: We use Black for Python and Prettier for JavaScript. Make sure to format your code before submitting any changes.

  • Running Tests:

    poetry run pytest

By following these guidelines, you'll be able to set up your development environment smoothly and start contributing to the project. Happy coding!


Contributing

We welcome contributions from the community! To ensure a smooth collaboration, please follow these guidelines:

Submitting Changes

  1. Open an Issue: For major changes, start by opening an issue to discuss your proposed modifications. This helps us understand your intentions and provide feedback early in the process.
  2. Pull Requests: Once your changes are ready, submit a pull request. Ensure your code adheres to our coding standards and passes all tests. Commits should follow conventional-commits specification.

Code Formatting

  • Backend: We use Black for code formatting. Run black on your code before submitting.
  • Frontend: We use Prettier for code formatting. Run prettier on your code before submitting.
  • Line Endings: Use CRLF line endings unless the file is a shell script or another format that requires LF line endings.

Dependency Management

We use Poetry for managing dependencies. Poetry simplifies dependency management by automatically handling package versions and resolving conflicts, ensuring consistency across all environments.

Setting Up Your Environment

  1. Install Poetry: If you haven't already, install Poetry using pip install poetry.
  2. Install Dependencies: After cloning the repository, navigate to the project's root directory and run poetry install. This command installs all necessary dependencies as defined in the pyproject.toml file and creates an isolated virtual environment.
  3. Activate Virtual Environment: You can activate the virtual environment using poetry shell or run commands directly using poetry run <command>.

Adding or Updating Dependencies

  • Add a Dependency: Use poetry add <package-name> to add a new dependency.
  • Update a Dependency: Use poetry update <package-name> to update an existing dependency.

Running Tests and Linters

Before submitting a pull request, ensure your changes are compatible with the project's dependencies and coding standards. Use the following commands to run tests and linters:

  • Run Tests: poetry run pytest
  • Run Linters: poetry run ruff check backend and poetry run isort --check-only backend

By following these guidelines, you help us maintain a high-quality codebase and streamline the review process. Thank you for contributing!



License

This project is licensed under the GNU GPLv3 License - see the LICENSE file for details

riven's People

Contributors

aamorin avatar ayushsehrawat avatar centerionware avatar davidemarcoli avatar dependabot[bot] avatar dreulavelle avatar drmxrcy avatar funkypenguin avatar gaisberg avatar hjongedijk avatar ipromknight avatar justheretobreakthings avatar kingpin avatar lostb1t avatar omnunum avatar zombieapps avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

riven's Issues

Episode/Shows not being created properly sometimes?

Exception has occurred: AttributeError
'Program' object has no attribute 'core_manager'
  File "/home/spoked/docker/iceberg/backend/main.py", line 29, in run_in_thread
    yield
  File "/home/spoked/docker/iceberg/backend/main.py", line 55, in <module>
    app.program.start()
  File "/home/spoked/docker/iceberg/backend/program/__init__.py", line 36, in start
    self.core_manager = ServiceManager(self.media_items, True, Content, Plex, Scraping, Debrid, Symlinker)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/utils/service_manager.py", line 10, in __init__
    self.initialize_services(services)
  File "/home/spoked/docker/iceberg/backend/utils/service_manager.py", line 28, in initialize_services
    new_service = service(self.media_items)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/program/content/__init__.py", line 21, in __init__
    self._get_content()
  File "/home/spoked/docker/iceberg/backend/program/content/__init__.py", line 35, in _get_content
    service.run()
  File "/home/spoked/docker/iceberg/backend/program/content/overseerr.py", line 55, in run
    container = self.updater.create_items(new_items)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/program/updaters/trakt.py", line 38, in create_items
    item = future.result()
           ^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/program/updaters/trakt.py", line 61, in _create_item
    item = create_item_from_imdb_id(imdb_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/program/updaters/trakt.py", line 148, in create_item_from_imdb_id
    return _map_item_from_data(data, media_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/program/updaters/trakt.py", line 112, in _map_item_from_data
    item["number"] = getattr(data, "number")
                     ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'types.SimpleNamespace' object has no attribute 'number'

During handling of the above exception, another exception occurred:

  File "/home/spoked/docker/iceberg/backend/program/__init__.py", line 61, in stop
    for service in self.core_manager.services:
                   ^^^^^^^^^^^^^^^^^
  File "/home/spoked/docker/iceberg/backend/main.py", line 31, in run_in_thread
    app.program.stop()
  File "/home/spoked/docker/iceberg/backend/main.py", line 53, in <module>
    with server.run_in_thread():
AttributeError: 'Program' object has no attribute 'core_manager

line 112 of trakt is,

        case "episode":
            item["number"] = getattr(data, "number")
            return_item = Episode(item)

item imdb_id is tt1475582 which shows as a show from trakt reponse but ends up as an episode type

[Bug] Failed building wheel of lxml on python12 & python11

pip install -r requirements.txt fails on Ubuntu on python11 & python12. Works fine on python10.

image

On further investigation, it is related to the version of lxml, removing it fixed the issue but raised another issue

[2023-12-08 14:48:15 | DEBUG] <settings.load> - Settings loaded from settings.json
Traceback (most recent call last):
  File "/home/mini/projects/iceburg/src/main.py", line 10, in <module>
    from utils.ui_helpers import CustomJSONEncoder
  File "/home/mini/projects/iceburg/src/utils/ui_helpers.py", line 3, in <module>
    from flask.json import JSONEncoder
ImportError: cannot import name 'JSONEncoder' from 'flask.json' (/home/mini/.pyenv/versions/3.12.0/lib/python3.12/site-packages/flask/json/__init__.py)

Seems like newer version of flask.json doesn't have JSONEncoder.

PTN fails to get CAM if multiple quality in pattern

  • Aquaman.and.the.Lost.Kingdom.2023.1080p.HDRip.CAM.AUDIO.Snoopy.mkv

PTN picks up HDRip as the quality instead of seeing CAM. We need to add a secondary check to check for cam..

This would probably be better to just do a custom regex check instead of using PTN to check for Cam or TS

[Feature] Enhancement Ideas

I’m just going to throw various ideas out there:

  • The obvious stuff like Trakt/MDBlist support, Overseerr support
  • Content tracking - the software should not only check for content availability in Plex (ie if something becomes unavailable and Zurg can’t fix it, leading to the item being removed, Iceburg should re-search for the item) but also track availability per version
    • Meaning if the 4k version that was requested goes down, Iceburg searches for the 4k version even if a 1080p version remains available
  • Search logic - only search versions that are available (ie only search a 4k version if that content was released in 4k)
  • Ignore flow - it would be great to periodically unignore when items are considered available to avoid early searches ignoring items
    • Arrs flow - possible idea that if an item cannot be found but is considered available that some form of Usenet workflow is supported (could simply be if S03E05 is not found, the ignored item is marked as requested in Sonarr)
  • Notifications - on release snatch, on item added etc
  • Continuing series flag - flag items as either continuing or not to ensure that a series that resumes airing continues to pull new content

App crashing due to batch_size out of range/zero

In trakt.py, The batch size being 0 results in multiple unhandled exceptions.

Traceback (most recent call last):
  File "/home/raju/PycharmProjects/iceberg/backend/main.py", line 26, in run_in_thread
    yield
  File "/home/raju/PycharmProjects/iceberg/backend/main.py", line 52, in <module>
    app.program.start()
  File "/home/raju/PycharmProjects/iceberg/backend/program/__init__.py", line 33, in start
    self.core_manager = ServiceManager(self.media_items, True, Content, Plex, Scraping, Debrid, Symlinker)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/raju/PycharmProjects/iceberg/backend/utils/service_manager.py", line 10, in __init__
    self.initialize_services(services)
  File "/home/raju/PycharmProjects/iceberg/backend/utils/service_manager.py", line 28, in initialize_services
    new_service = service(self.media_items)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/raju/PycharmProjects/iceberg/backend/program/content/__init__.py", line 20, in __init__
    self._get_content()
  File "/home/raju/PycharmProjects/iceberg/backend/program/content/__init__.py", line 34, in _get_content
    service.run()
  File "/home/raju/PycharmProjects/iceberg/backend/program/content/overseerr.py", line 52, in run
    container = self.updater.create_items(new_items)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/raju/PycharmProjects/iceberg/backend/program/updaters/trakt.py", line 33, in create_items
    imdb_id_batches = [imdb_ids[i:i + batch_size] for i in range(0, len(imdb_ids), batch_size)]
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: range() arg 3 must not be zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/raju/PycharmProjects/iceberg/backend/main.py", line 50, in <module>
    with server.run_in_thread():
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/raju/PycharmProjects/iceberg/backend/main.py", line 28, in run_in_thread
    app.program.stop()
  File "/home/raju/PycharmProjects/iceberg/backend/program/__init__.py", line 58, in stop
    for service in self.core_manager.services:
                   ^^^^^^^^^^^^^^^^^
AttributeError: 'Program' object has no attribute 'core_manager'

[Feature Request] Ignore List

Would be nice to have the ability to have an ignore list where items get added after x attempts or can be manually added. Could be controlled from the front end. Maybe ability to add items by dynamic search/imdb id.

Torrentio hitting rate limit

Once we hit torrentio rate limit we will keep requesting forever, this needs to addressed.

[2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:07 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle [2023-12-17 19:28:08 | ERROR] <torrentio.run> - Rate limit exceeded!, trying again next cycle

Torrentio 500: Handler Error

Error: 500 b'{"err":"handler error"}'
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 687, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
ValueError: unsupported format character 'C' (0x43) at index 46
Call stack:
  File "/usr/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/iceberg/backend/program/scrapers/__init__.py", line 28, in run
    service.run()
  File "/iceberg/backend/program/scrapers/torrentio.py", line 38, in run
    scraped_amount += self._scrape_show(item)
  File "/iceberg/backend/program/scrapers/torrentio.py", line 59, in _scrape_show
    scraped_amount += self._scrape_items(episodes)
  File "/iceberg/backend/program/scrapers/torrentio.py", line 65, in _scrape_items
    data = self.api_scrape(item)
  File "/iceberg/backend/program/scrapers/torrentio.py", line 130, in api_scrape
    response = get(f"{url}.json", retry_if_failed=False)
  File "/iceberg/backend/utils/request.py", line 106, in get
    return _make_request(
  File "/iceberg/backend/utils/request.py", line 90, in _make_request
    return ResponseObject(response, response_type)
  File "/iceberg/backend/utils/request.py", line 30, in __init__
    self.data = self.handle_response(response)
  File "/iceberg/backend/utils/request.py", line 37, in handle_response
    logger.error(response.url, response.reason)
Message: 'https://torrentio.strem.fun/sort=qualitysize%7Cqualityfilter=480p,scr,cam,unknown/stream/show/tt4728568:3:9.json'
Arguments: ('Internal Server Error',)

[Bug] Error message: Unable to fetch settings data. API is down.

Used the docker compose i thought we need to add token from UI

[2023-12-30 09:17:18 | DEBUG] <settings.get> - Get (realdebrid) returned: {'api_key': 'REDACTED'}
[2023-12-30 09:17:18 | ERROR] <realdebrid.__init__> - Realdebrid settings incorrect or not premium, retrying in 2...

Docker Fully Supported

  • Issues with PUID/PGID are still showing up. This needs to be handled better, with less logging as well. I have a fix for this, just need to test locally a little more.
  • Add Dockerfile's for local development, with a Makefile. Which has start, stop, restart, build.. and anything else that can be useful for local development. These need to be handling frontend and backend separately.
  • Update Docker Compose in readme of repo to reflect best settings for users in production.

Will add more as I come across things for Docker

[Feature] Hide url in logs

Hey,

The api keys are already automatically hidden from logs, could we also do the same for the url? That will make is easier to provide logs when reporting issues.

activ

Feature Request: Check for Missing Episodes in Plex

Feature Request: Check for Missing Episodes in Plex

Description:

I propose enhancing Iceburg to check for missing episodes within Plex, providing users with an efficient way to manage their library.

Details:

  • Functionality: Iceburg could scan a user's Plex library to identify missing episodes.
  • Action on Missing Episodes: Options to automatically add missing episodes or log them in a file.
  • User Configuration: Allow customization such as directories to search or handling preferences.
  • Notification: Notify users about missing episodes and actions taken.

Benefits:

  • Enhanced User Experience: Simplifies managing missing episodes within Plex.
  • Efficiency: Saves time for users maintaining a complete library.
  • Flexibility: Customizable options suit different user preferences.

Request:

I request the Iceburg team to consider implementing this feature, benefiting Plex users and enhancing Iceburg's utility.

Thank you for your consideration.

Best regards,

KAKA

Feature Request: Integration of "Your Friends’ Watchlist" RSS Feed from Plex

Feature Request: Integration of "Your Friends’ Watchlist" RSS Feed from Plex

Description:

I propose adding a feature to Iceburg that allows integration with "Your Friends’ Watchlist" RSS feed from Plex.

Details:

  • Functionality: Enable Iceburg to access and interpret "Your Friends’ Watchlist" RSS feed from Plex.
  • Use Case: Download Friend's watchlist .

Request:

I kindly request the Iceburg development team to consider implementing the integration of "Your Friends’ Watchlist" RSS feed from Plex.

Thank you for considering this feature addition.

Best regards,

KAKA

[BUG] Plex Watchlist Issue: 'NoneType' object has no attribute 'get'

Starting Container with 1000:1000 permissions...
Initialization complete. Executing main process...
[2023-12-18 14:29:39 | DEBUG] <settings.load> - Settings loaded from /iceberg/data/settings.json
[2023-12-18 14:29:40 | DEBUG] <settings.get> - Get (mdblist) returned: {'lists': ['29073'], 'api_key': 'REDACTED', 'update_interval': 80}
[2023-12-18 14:29:41 | DEBUG] <settings.get> - Get (overseerr) returned: {'url': 'http://overseerr:5055', 'api_key': 'REDACTED'}
[2023-12-18 14:29:41 | DEBUG] <settings.get> - Get (plex) returned: {'user': 'REDACTED', 'token': 'REDACTED', 'url': 'http://192.168.1.154:32400', 'watchlist': 'https://rss.plex.tv/9b11874b-f7f5-476d-83ce-REDACTED'}
[2023-12-18 14:29:42 | INFO] <mdblist.run> - Added 50 items
[2023-12-18 14:29:42 | INFO] <overseerr.run> - Added 7 items
[2023-12-18 14:29:43 | DEBUG] <plex_watchlist._get_items_from_plex_watchlist> - Found 50 items
Traceback (most recent call last):
  File "/iceberg/backend/main.py", line 22, in <module>
    app.program = Program()
                  ^^^^^^^^^
  File "/iceberg/backend/program/__init__.py", line 64, in __init__
    Content(self.media_items),  # Content must be first
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/iceberg/backend/program/content/__init__.py", line 28, in __init__
    service.run()
  File "/iceberg/backend/program/content/plex_watchlist.py", line 41, in run
    container = self.updater.create_items(new_items)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/iceberg/backend/program/updaters/trakt.py", line 36, in create_items
    get_items.append(item)
  File "/iceberg/backend/program/media.py", line 289, in append
    self.sort("requested_at", True)
  File "/iceberg/backend/program/media.py", line 279, in sort
    self.items.sort(key=lambda item: item.get(by), reverse=reverse)
  File "/iceberg/backend/program/media.py", line 279, in <lambda>
    self.items.sort(key=lambda item: item.get(by), reverse=reverse)
                                     ^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

[Feature Request] User Authentication

Would be good to have a single user auth. I plan on adding this to my FQDN.

I can manage it through Cloudflare ACL's or Authentik or something.. but would be cool for a simple login form

[BUG] Frontend port 8080

Running in Docker.

WEB UI home page message:

Something went wrong
Error code: 500

Error message: Unable to fetch user data. API is down.

stdout home page accessed:

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: connect ECONNREFUSED 127.0.0.1:8080
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '127.0.0.1',
    port: 8080
  }
}

Clicking on Status or Settings from home page:

Container exits. stdout throws:

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: connect ECONNREFUSED 127.0.0.1:8080
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '127.0.0.1',
    port: 8080
  }
}
TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: connect ECONNREFUSED 127.0.0.1:8080
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '127.0.0.1',
    port: 8080
  }
}
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^
[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "[object Object]".] {
  code: 'ERR_UNHANDLED_REJECTION'
}
Node.js v20.10.0

Active Ports:

d4005c968e4a:/iceberg# netstat -tulpn | grep LISTEN
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      1/node
tcp        0      0 127.0.0.11:46839        0.0.0.0:*               LISTEN      -

Looks like port 8080 may not be getting activated.

[BUG] tt5321682 causes a crash

Wit the help of Gb I added extra logging and found that tt5321682 crashes iceberg consitently.

[2023-12-29 12:04:51 | DEBUG] <trakt._map_item_from_data> - tt5321682
Iceberg  | Traceback (most recent call last):
Iceberg  |   File "/iceberg/backend/main.py", line 22, in <module>
Iceberg  |     app.program = Program()
Iceberg  |                   ^^^^^^^^^
Iceberg  |   File "/iceberg/backend/program/__init__.py", line 59, in __init__
Iceberg  |     Content(self.media_items),  # Content must be first
Iceberg  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
Iceberg  |   File "/iceberg/backend/program/content/__init__.py", line 28, in __init__
Iceberg  |     service.run()
Iceberg  |   File "/iceberg/backend/program/content/overseerr.py", line 40, in run
Iceberg  |     container = self.updater.create_items(new_items)
Iceberg  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Iceberg  |   File "/iceberg/backend/program/updaters/trakt.py", line 27, in create_items
Iceberg  |     item = self._create_item(imdb_id)
Iceberg  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Iceberg  |   File "/iceberg/backend/program/updaters/trakt.py", line 43, in _create_item
Iceberg  |     item = create_item_from_imdb_id(imdb_id)
Iceberg  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Iceberg  |   File "/iceberg/backend/program/updaters/trakt.py", line 131, in create_item_from_imdb_id
Iceberg  |     return _map_item_from_data(data, media_type)
Iceberg  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Iceberg  |   File "/iceberg/backend/program/updaters/trakt.py", line 93, in _map_item_from_data
Iceberg  |     item["number"] = getattr(data, "number")
Iceberg  |                      ^^^^^^^^^^^^^^^^^^^^^^^
Iceberg  | AttributeError: 'types.SimpleNamespace' object has no attribute 'number'

plex_update looping - causes Plex to crash some times

Iceberg will continuously try to update and match Plex items.. this causes Plex to loop scanning the libraries... eventually causing Plex to stall out or crash. Need to be more strict on when we update/match.. refreshing more specifically.

I imagine this will probably get resolved around the same time symlinking gets introduced.

[Feature] Rclone Integration

  • Add Rclone into Iceberg container for Iceberg to manage everything from scraping and getting content, to symlinking it without user intervention/configuration.

[Feature] Multi-Episode Scraping Support

Hard to explain this issue but I'll my best.

Iceberg selects the hash below for Reacher S02E01 which is a multi episode pack. However it only selects the first episode in real debrid. When it then tries to select the same torrent for E02 & E03 it doesn't since it thinks the torrent is already downloaded.

image

[2023-12-22 08:25:19 | DEBUG] <torrentio._scrape_items> - Found 39 streams for Reacher S1
[2023-12-22 08:25:20 | DEBUG] <torrentio._scrape_items> - Found 25 streams for Reacher S2E1
[2023-12-22 08:25:21 | DEBUG] <torrentio._scrape_items> - Found 25 streams for Reacher S2E2
[2023-12-22 08:25:21 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Reacher season 1
[2023-12-22 08:25:22 | DEBUG] <torrentio._scrape_items> - Found 25 streams for Reacher S2E3
[2023-12-22 08:25:23 | DEBUG] <torrentio._scrape_items> - Found 19 streams for Reacher S2E4
[2023-12-22 08:25:23 | INFO] <torrentio.run> - Scraped 5 streams
[2023-12-22 08:25:24 | DEBUG] <realdebrid._download_item> - Downloaded Reacher S1
[2023-12-22 08:25:24 | INFO] <realdebrid.download> - Downloaded 1 cached releases
[2023-12-22 08:25:26 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Reacher season 2 episode 1
[2023-12-22 08:25:29 | DEBUG] <realdebrid._download_item> - Downloaded Reacher S2E1
[2023-12-22 08:25:30 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Reacher season 2 episode 2
[2023-12-22 08:25:31 | DEBUG] <realdebrid._is_downloaded> - Torrent already downloaded
[2023-12-22 08:25:32 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Reacher season 2 episode 3
[2023-12-22 08:25:33 | DEBUG] <realdebrid._is_downloaded> - Torrent already downloaded

Switch to creating items from TheTVDB and TMDb (just like Arrs)

Instead of using Trakt we could have items created from these databases instead to ensure accuracy.

In theory, ALL content services should have items created from that using the appropriate ID. We can always convert ID's if the content service doesn't provide the correct ID.

Possibly use pydantic to handle mapping data we use.. this probably gonna be a large rewrite if we go in that direction 🤔

    item = {
        "title": getattr(data, "title", None),              # 'Game of Thrones'
        "year": getattr(data, "year", None),                # 2011
        "status": getattr(data, "status", None),            # 'ended', 'released', 'returning series'
        "aired_at": formatted_aired_at,                     # datetime.datetime(2011, 4, 17, 0, 0)
        "is_anime": is_anime,                               # True"
        "imdb_id": getattr(data.ids, "imdb", None),         # 'tt0496424'
        "tvdb_id": getattr(data.ids, "tvdb", None),         # 79488
        "tmdb_id": getattr(data.ids, "tmdb", None),         # 1399
        "genres": getattr(data, "genres", None),            # ['Action', 'Adventure', 'Drama', 'Fantasy']
        "network": getattr(data, "network", None),          # 'HBO'
        "country": getattr(data, "country", None),          # 'US'
        "language": getattr(data, "language", None),        # 'en'
        "requested_at": datetime.now(),                     # datetime.datetime(2021, 4, 17, 0, 0)
    }

Just food for thought.. as the Arrs use these 2 databases and have proven to be pretty reliable...

Remove Special chars in parser?

Без лица.BDRemux..mkv
Игра смерти (Deaths) Сезон 1
Капитан.Марвел.2.2023_1080p_[scarfilm].mkv
Монарх - Наследие монстров (Monarch - Legacy of Monsters) Сезон 1
Наполеон.2023.WEBRip.[1080p].NNMClub.mkv
Настоящая любовь.1993.Director's Cut.UHD.Blu-Ray.Remux.2160p.mkv
Падение Олимпа (Olympus Has Fallen) 2013, США, боевик, триллер, BDRemux 1080p GORESEWAGE.mkv

Maybe an allow_foreign setting with a default value of false

Exception in plex.py

I am getting the following exception.

INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
Exception in thread Plex:
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/iceberg/backend/program/plex.py", line 69, in run
    self._update_sections()
  File "/iceberg/backend/program/plex.py", line 161, in _update_sections
    if log_string:
       ^^^^^^^^^^
UnboundLocalError: cannot access local variable 'log_string' where it is not associated with a value

Removing the IF block does seem to fix the exception

if log_string:
    logger.debug("Updated section %s for %s", section.title, log_string)

Iceberg goes into a loop sometimes

Not entirely sure what causes the loop, but here is a log snippet showing the loop. It will keep going like that forever.
I have already redone the config and deleted the pal files, but it comes back anyway.

Iceberg  | [2024-01-15 09:06:41 | DEBUG] <realdebrid._determine_best_stream> - No cached release found for Doctor Who S07
Iceberg  | [2024-01-15 09:06:43 | DEBUG] <orionoid._scrape_item> - Found 3 streams for Doctor Who S07
Iceberg  | [2024-01-15 09:06:43 | DEBUG] <realdebrid._determine_best_stream> - No cached release found for Doctor Who S08
Iceberg  | [2024-01-15 09:06:44 | DEBUG] <orionoid._scrape_item> - Found 2 streams for Doctor Who S08
Iceberg  | [2024-01-15 09:06:44 | DEBUG] <realdebrid._determine_best_stream> - No cached release found for Doctor Who S09
Iceberg  | [2024-01-15 09:06:45 | DEBUG] <orionoid._scrape_item> - Found 7 streams for Doctor Who S09
Iceberg  | [2024-01-15 09:06:46 | DEBUG] <orionoid._scrape_item> - Could not find streams for Doctor Who S10
Iceberg  | [2024-01-15 09:06:46 | DEBUG] <realdebrid._determine_best_stream> - No cached release found for Doctor Who S11
Iceberg  | [2024-01-15 09:06:47 | DEBUG] <orionoid._scrape_item> - Found 5 streams for Doctor Who S11
Iceberg  | [2024-01-15 09:06:48 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:48 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:49 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:49 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:50 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:50 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:50 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:51 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:51 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:52 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:52 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:52 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:53 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:53 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:54 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:54 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:55 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:55 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:56 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:56 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded
Iceberg  | [2024-01-15 09:06:56 | DEBUG] <realdebrid._determine_best_stream> - Found cached release for Doctor Who S12
Iceberg  | [2024-01-15 09:06:56 | DEBUG] <realdebrid._is_downloaded> - Torrent for Doctor Who S12 already downloaded

Orionoid not finding anything a lot

Example:

Iceberg  | [2024-01-19 03:03:01 | DEBUG] <torrentio._scrape_item> - Found 11 streams for Reacher S02E08
Iceberg  | [2024-01-19 03:03:02 | DEBUG] <orionoid._scrape_item> - Could not find streams for Reacher S02E08
Iceberg  | [2024-01-19 03:11:45 | DEBUG] <realdebrid._download_item> - Downloaded Reacher S02E08
Iceberg  | [2024-01-19 03:20:19 | DEBUG] <symlink._symlink> - Created symlink for Reacher S02E08
Iceberg  | [2024-01-19 03:28:46 | DEBUG] <plex.update_item_section> - Updated section TV Shows for Reacher S02E08
Iceberg  | [2024-01-19 00:55:27 | DEBUG] <torrentio._scrape_item> - Found 9 streams for Boy Swallows Universe S01
Iceberg  | [2024-01-19 00:55:27 | DEBUG] <orionoid._scrape_item> - Could not find streams for Boy Swallows Universe S01
Iceberg  | [2024-01-18 20:17:56 | DEBUG] <torrentio._scrape_item> - Found 32 streams for Mad Max Beyond Thunderdome
Iceberg  | [2024-01-18 20:17:57 | DEBUG] <orionoid._scrape_item> - Could not find streams for Mad Max Beyond Thunderdome
Iceberg  | [2024-01-18 20:25:04 | DEBUG] <realdebrid._download_item> - Downloaded Mad Max Beyond Thunderdome
Iceberg  | [2024-01-18 20:32:09 | DEBUG] <symlink._symlink> - Created symlink for Mad Max Beyond Thunderdome

10% CPU Intensive report

Need to look into what kind of toll it takes on the host machine and if any of it can be mitigated. Will require some testing to make sure we're not bogging down the system, or taking up too many resources.

Wrong library is updated

Looks like it is trying to use the first movie or tv library it finds rather than the one that has the right path

Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Fear the Invisible Man
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Secrets in the Marriage
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for He Went That Way
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Attack of the 50 Foot Camgirl
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Queen: Rock Montreal & Live Aid
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Girl in the Video
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for True Justice: Family Ties
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Destroy All Neighbors
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Roadkill
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for Lion of Judah Legacy
Iceberg  | [2024-01-18 05:41:39 | DEBUG] <symlink._symlink> - Created symlink for The Painter
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Fear the Invisible Man
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Queen: Rock Montreal & Live Aid
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for He Went That Way
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Roadkill
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Secrets in the Marriage
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for The Painter
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for True Justice: Family Ties
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Girl in the Video
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Destroy All Neighbors
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Attack of the 50 Foot Camgirl
Iceberg  | [2024-01-18 05:41:42 | DEBUG] <plex.update_item_section> - Updated section Concerts for Lion of Judah Legacy

image
image
image

Fix `_is_released` for determining when item is released

<jackett._scrape_item> - Could not find streams for Wonka

This movie isn't even out yet. Overseerr shows only theatrical release, but no digital or physical.

Googling this movie..
Theatrical Release: December 15, 2023
Digital Release: January 30, 2024
Physical Release: March 12, 2024

imdb id is tt6166392

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.