VoxNovel is an innovative program that leverages the capabilities of booknlp to analyze literature, attribute quotations to specific characters, and generate a tailored audiobook where each character has a distinct voice through coqui tts. This not only provides an immersive audiobook experience but also brings each character to life with a unique voice, making the listening experience much more engaging.
All Coqui TTS models-(Tacotron, Tacotron2, Glow-TTS, Speedy-Speech, Align-TTS, FastPitch, FastSpeech, FastSpeech2, SC-GlowTTS, Capacitron, OverFlow, Neural HMM TTS, Delightful TTS, โTTS, VITS, ๐ธ YourTTS, ๐ข Tortoise, ๐ถ Bark), and STYLETTS2.
๐๐๏ธ Accents you can give each character with the default cloning model (XTTS)
- They also allow them to speak these languages, but the quotation attribution won't correctly identify for anything thats not English. English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Turkish (tr), Russian (ru), Dutch (nl), Czech (cs), Arabic (ar), Chinese (zh-cn), Japanese (ja), Hungarian (hu), Korean (ko)(as well as a folder of individual mp4 chatper files with ebook image embedded in them if you want that)
High Quality XTTS V2 Demos
Guardians_of_Ga.Hoole_10._Lasky._Kathryn_-_Coming_Of_Hoole_chapter_4.mp4
๐๐ More Demo Audio files :)
High Quality Tortoise Demos
272463996-c4f8dfdf-c5bd-4771-ab1a-6131c22a67b2.mp4
Super fast audio Balacoon Demos
271878548-53b694a9-5ddd-4174-82bc-07aff22d2330.mp4
271876316-530e8781-c77c-4424-89cd-a02df363b0bf.mp4
**Super High Quality testing with fine tuned models **
AUDIO_5811.mp4
You can fine tune your own Xtts models with around 6+ minutes of audio for free with this colab https://colab.research.google.com/drive/1GiI4_X724M8q2W-zZ-jXo7cWTV7RfaH-
Explore and run the interactive version of the Headless VoxNovel project directly on Google Colab! Get started here.
GUI Part 1 (BookNLP Processor) Info/Features
-"Process File" button: Click and it'll ask you to select a ebook file.GUI Part 2 (Coqui TTS GUI) Info/Features
- Select TTS Model Dropdown: This selects the TTS model that will be used for voice cloning.
- Include fast Voice Models Checkbox: (Fast generate at cost of audio quality) Click this to be able to see every other model and singular voices supported by Coqui TTS.
- It will update the "Select TTS Model" Dropdown for voice cloning models to also include (List of values to be added).
- It will update the Dropdown for voices to select for each character to also include (List of values to be added).
- Make all audio generate with Narrator voice Checkbox: This will make every character's audio be generated with the voice you have selected for the Narrator when you click the "Generate audio" button.
- Clone new voice Button: Click this to add a new voice you can clone (make sure you have a reference audio file on hand).
- Add Fine-tuned Xtts model to voice actor Button: If you have a folder containing all the parameters of a fine-tuned Xtts model of a specific voice, then you can click this to make that voice actor clone with that fine-tuned Xtts model, to provide much better voice cloning results.
- Character voices Dropdowns: These are the dropdowns for selecting the Voice Actor (and the Accent of each character if using XTTS).
- (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
- When you select a voice It will play the audio sample of that voice, if it's a fast voice model voice and a refrence audio does not exist, then it will generate one to play.
- (2): The Accents available to select from for this character. (Optional, Default is English).
- (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
- Chapter Delimiter Field: Will change the default chapter delimiter (The string that's used to identify chapters).
- Silence Duration in milliseconds (ms) Field: This will change the amount of milliseconds in between each combined chunk of audio.
- Select TTS Language Dropdown: This will let you select the default Accent used for every character which has not had the Accent manually selected for.
- Loading bar: Will give an approximate amount of time left. (Estimate, you probably won't see accurate predictions until it's been running for 5 min).
- Annotated book preview Block: This will show the entirety of the book with each character's lines color-coded.
- You can click on a line while the audiobook is being generated to hear what that generated line sounds like. But only if the line has already had audio generated for it; if not, it'll play nothing.
- Load Book Button: Clicking this will reload the color-coded annotated book view, it will just randomize the selected colors for each character's lines.
- Generate Audio Button: Will start generating the full audiobook.
- Select random voices Button (Will only be visible if the "include fast Voice Models" checkbox is checked): Will Select an auto-gender-inferred fast model voice for every character except for the narrator's voice.
GUI Part 3 (Book Viewer) Info/Features
-It's hard to explain its more of a playground if you mess around with it then you should get how it works. But it can be used to fine tune the audiobook -Close out of the window when your done with it.Run in this order:
๐ค Headless VoxNovel Google Colab
Explore and run the interactive version of the Headless VoxNovel project directly on Google Colab! Get started here.
๐ณ Docker (Sound not working in gui yet)
๐ณ Headless Docker
Docker headless m1 ๐Mac
-
cd ~
-
git clone https://github.com/DrewThomasson/VoxNovel.git
-
sudo docker run -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:headless_m1_latest
Headless Docker ๐ง Linux/Intel ๐Mac
-
cd ~
-
git clone https://github.com/DrewThomasson/VoxNovel.git
-
sudo docker run -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless
-
cd ~
-
git clone https://github.com/DrewThomasson/VoxNovel.git
-
sudo docker run --gpus all -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless
Headless Docker ๐ฅ๏ธ Windows
Follow these steps to set up the VoxNovel project on a Windows system using PowerShell:
-
Navigate to your user profile directory:
cd $env:USERPROFILE
-
Clone the VoxNovel repository from GitHub:
git clone https://github.com/DrewThomasson/VoxNovel.git
To run the VoxNovel application in a Docker container on your CPU:
docker run -v "${env:USERPROFILE}/VoxNovel/:/VoxNovel/" -it athomasson2/voxnovel:latest_headless
If you have an NVIDIA GPU and want to accelerate processing, use the following command:
docker run --gpus all -v "${env:USERPROFILE}/VoxNovel/:/VoxNovel/" -it athomasson2/voxnovel:latest_headless
๐ณ GUI Docker (Sound not working in gui yet)
๐ง Linux Docker
1. `cd ~`git clone https://github.com/DrewThomasson/VoxNovel.git
sudo docker run --gpus all -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /dev/snd:/dev/snd --device /dev/snd -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest
๐ Mac Docker
This guide provides instructions on how to run a Docker container with a graphical user interface on macOS using XQuartz for X11 forwarding and volume mounting.
- Download and install XQuartz from XQuartz website.
- Open XQuartz.
- Go to
XQuartz
->Preferences
. - In the
Security
tab, enable Allow connections from network clients. - Restart XQuartz to apply these settings.
Open a terminal and run the following command to allow connections from your local machine to XQuartz:
xhost + $(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}')
Run the following command to start your Docker container. This command configures the GUI to display on your host and mounts the necessary directories:
cd ~
git clone https://github.com/DrewThomasson/VoxNovel.git
docker run -e DISPLAY=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}'):0 \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v "/Users/$(whoami)/VoxNovel:/VoxNovel" \
athomasson2/voxnovel:latest
- XQuartz Configuration: Ensure that XQuartz is configured to allow network clients before attempting to connect.
- Directory Existence: Verify that the directory
/Users/$(whoami)/VoxNovel
exists on your Mac. If not, create it or adjust the volume mount path in the Docker command as needed. - Firewall and Security: If you face connectivity issues, check any firewall settings and security preferences that might block the connections.
๐ช Windows Docker
1. `choco install vcxsrv` - First install VcXsrv and configur it to allow connectionshow to setup VcXsrv
After installing VcXsrv, it typically launches automatically. You can confirm it's running by checking for its icon in the system tray, usually located near the clock in the taskbar. It may also start automatically when you log in to your system.To ensure it's configured to allow connections from Docker containers, follow these steps:
- Right-click on the VcXsrv icon in the system tray.
- Select "XLaunch" to open the configuration wizard.
- In the configuration wizard, select "Multiple windows" and proceed to the next step.
- Choose your preferred settings for display number and screen.
- In the "Extra settings" window, make sure to check the box labeled "Disable access control" to allow connections from Docker containers.
- Complete the configuration by clicking "Finish" and then "Save configuration" when prompted.
With these settings, VcXsrv should be running and configured to allow connections from Docker containers. You can now proceed with running your Docker commands requiring GUI support.
`
๐ง Linux
## Single command Ubuntu install:(Do not use if you already have miniconda installed.)yes | wget -O - https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Ubuntu-install.sh | bash
-This will also create a desktop shortcut to run Voxnovel as well.
sudo apt-get install calibre
sudo apt-get install ffmpeg
conda create --name VoxNovel python=3.10
conda activate VoxNovel
git clone https://github.com/DrewThomasson/VoxNovel.git
cd VoxNovel
pip install bs4
pip install styletts2
pip install tts==0.21.3
pip install booknlp
pip install -r Ubuntu_requirements.txt
python -m spacy download en_core_web_sm
For non Latin-based Languages TTS support (Optional)
Install Mecab for (Non Latin-based Languages tts support)(Optional):
- Ubuntu:
sudo apt-get install -y mecab libmecab-dev mecab-ipadic-utf8
(For non Latin-based Languages tts support)(Optional)
python -m unidic download
pip install mecab mecab-python3 unidic
Steam Deck) (x86_64 Arch Linux)
sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.sh | sudo sh /dev/stdin
also download it from the discovery store or flatpac I did both on my steam deck
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
restart the terminal(close out and open a new window)
conda create --name VoxNovel python=3.10
conda activate VoxNovel
git clone https://github.com/DrewThomasson/VoxNovel.git
cd VoxNovel
sudo pacman -S espeak-ng
(make sure you have pacman fully working, there should be stuff online for the steam deck) guide on getting pacman wokring on steam deck(https://www.reddit.com/r/SteamDeck/comments/t8al0i/install_arch_packages_on_your_steam_deck/) you might have to reset the keys if something goes wrong with that: Resetting all the keys
Remove or reset all the keys installed in your system by removing the /etc/pacman.d/gnupg directory (as root) and by rerunning pacman-key --init followed by pacman-key --populate to re-add the default keys.
15. pip install styletts2
16. pip install tts==0.21.3
17. pip install booknlp
18. pip install -r SteamDeck_requirements.txt
19. pip3 install spacy
20. python3 -m spacy download en_core_web_sm
21. sudo pacman -S calibre
22. mv ~/miniconda3/envs/VoxNovel/lib/libstdc++.so.6 ~/miniconda3/envs/tts/lib/libstdc++.so.6.bak
pip install mechanize
pip install bs4
pip install css_parser
Intel mac
brew install calibre
brew install ffmpeg
conda create --name VoxNovel python=3.10
conda activate VoxNovel
git clone https://github.com/DrewThomasson/VoxNovel.git
cd VoxNovel
pip install styletts2
pip install tts==0.21.3
pip install booknlp
9.pip install -r MAC-requirements.txt
pip install spacy
11.python -m spacy download en_core_web_sm
For non Latin-based Languages TTS support (Optional)
Install Mecab for (Non Latin-based Languages tts support)(Optional):
- macOS:
brew install mecab
,brew install mecab-ipadic
(For non Latin-based Languages tts support)(Optional)
python -m unidic download
pip install mecab mecab-python3 unidic
Apple Silicon Mac (Tested on 2020 M1 pro 8gb ram)
brew install calibre
(You might have to also install it manually from their site if this doesn't work)brew install ffmpeg
conda create --name VoxNovel python=3.10
conda activate VoxNovel
git clone https://github.com/DrewThomasson/VoxNovel.git
cd VoxNovel
pip install tensorflow-macos
(Also optionalpip install tensorflow-metal
But so far I haven't gotten gpu speedup yetpip install styletts2
pip install tts==0.21.3
pip install --no-dependencies booknlp
pip install transformers==4.30.0
pip install tensorflow
pip install -r MAC-requirements.txt
pip install ebooklib bs4 epub2txt pygame moviepy spacy
python -m spacy download en_core_web_sm
For non Latin-based Languages TTS support (Optional)
Install Mecab for (Non Latin-based Languages tts support)(Optional):
- macOS:
brew install mecab
,brew install mecab-ipadic
(For non Latin-based Languages tts support)(Optional)
python -m unidic download
pip install mecab mecab-python3 unidic
Windows 11
Because of BookNLP Windows issues, all of this will be run in WSL (don't worry, it's still easy).Easy Windows 11 install
-
In your PowerShell, paste:
wsl --install
to install WSL. (You might be prompted by your system to enable virtualization in your BIOS if it's available, as it is needed to run WSL on Windows.)
-
After setting your username and password, open WSL and paste this command for a single command install:
yes | wget -O - https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Ubuntu-install.sh | bash
-
(Optional only for Nvida graphics cards Do not run this command if you don't have a Nvidia graphics card) Install the NVIDIA CUDA toolkit (required for Nvidia GPU acceleration):
sudo apt install nvidia-cuda-toolkit
-
Make sure you are in the VoxNovel conda environment:
conda activate VoxNovel
-
Navigate to the VoxNovel folder (if not already there):
cd VoxNovel
-
Congrats! VoxNovel should run in WSL now!
python gui_run.py
python headless_voxnovel
If you have trouble with the WSL environment:
-
List all WSL environments:
wsl --list --verbose
-
Remove a specific WSL environment (e.g., Ubuntu):
wsl --unregister <distro_name>
-
Reinstall WSL:
wsl --install
To launch WSL anytime you need to run this program, you can use the search bar in Windows to find and launch "WSL" or run:
wsl
InDepth Windows 11 install
1. In your PowerShell, paste `wsl --install` to install WSL.-
After you set the username and password, you'll now be able to open WSL as an application. This will allow you to run Ubuntu software. You'll now want to open WSL if you closed it and paste these commands.
(This will install Calibre for ebook conversion)
cd ~
sudo apt update
sudo apt upgrade
sudo apt install calibre
sudo apt install ffmpeg
(This will install Miniconda) Mini conda code was taken from here:https://docs.conda.io/projects/miniconda/en/latest/
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
(This will make sure Miniconda is initialized)
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
-
Close out of your current WSL window and reopen it to be able to use Miniconda.
conda create --name VoxNovel python=3.10
conda activate VoxNovel
cd ~
git clone https://github.com/DrewThomasson/VoxNovel.git
cd VoxNovel
pip install styletts2
pip install tts==0.21.3
pip install booknlp
pip install -r Ubuntu_requirements.txt
pip install spacy
python -m spacy download en_core_web_sm
sudo apt install espeak-ng
You can access the files on your WSL Ubuntu in Windows File Explorer by putting this into the address: \\wsl.localhost\Ubuntu\home\
For non Latin-based Languages TTS support (Optional)
Install Mecab for (Non Latin-based Languages tts support)(Optional):
sudo apt-get install -y mecab libmecab-dev mecab-ipadic-utf8
(For non Latin-based Languages tts support)(Optional)
python -m unidic download
pip install mecab mecab-python3 unidic
python gui_run.py
python headless_voxnovel
Running with Low VRAM (4 GB)
- Turns out once you set the device it stays like that for the full program.
- So, I've split the program into two Python programs: one CPU and one GPU. I've tested this on my (4GB VRAM GPU) and this solution works. at least on my end I really hope it works on your end. ๐
To run the provided scripts on your system, follow these steps in order:
-
Book Processing (CPU Only):
- Script: 1CPU_Book_processing.py
- This script handles the task of only processing the book using BookNLP, specifically forcing it to run on the CPU.
- Run with
python 1CPU_Book_processing.py
-
Audio Generation (GPU Only):
- Script: 2GPU_Audio_generation.py
- This script is dedicated to only generating audio with the GPU and should be run after completing the book processing with
1CPU_Book_processing.py
. - Run with
python 2GPU_Audio_generation.py
Upon running a mini test with an epub file using the above setup, the following performance metrics were observed:
Testing on done with the mini epub file located in the Example_working_files.zip
Task | Configuration | Time (Seconds) |
---|---|---|
Book Processing | GPU only (GeForce GTX 980), 4GB VRAM, 32GB RAM, Intel i7-8700K | 2.922 |
Audio Generation | GPU only (GeForce GTX 980), 4GB VRAM, 32GB RAM, Intel i7-8700K | 128.48 |
Book Processing | CPU only, 32GB RAM, Intel i7-8700K | 4.964 |
Audio Generation | CPU only, 32GB RAM, Intel i7-8700K | 391.4227 |
To Run the auto program
This means all you do is select the book and all the voices will be auto assigned and generated for you.python auto_noGui_run.py
.epub, .pdf, .mobi, .txt, .html, .rtf, .chm, .lit, .pdb, .fb2, .odt, .cbr, .cbz, .prc, .lrf, .pml, .snb, .cbc, .rb, and .tcr,
- (Best results are from using epub or mobi for auto chapter detection)
๐ Folders used by the program
/Final_combined_output_audio: This is where all of your chapter audio files will be put in order of chapter num
/output_audiobooks: This is where all of your m4b audiobook files will be stored
/Working_files: Holds all of the working files used by the program while activly running.
- /Working_files/temp_ebook: Holds all of the individual extracted chapter txt files from the ebook.
/tortoise: Holds all the sample voice files
GUI Part 1 (BookNLP Processor)
-"Process File" button: Click and it'll ask you to select a ebook file.GUI Part 2 (Coqui TTS GUI)
- Select TTS Model Dropdown: This selects the TTS model that will be used for voice cloning.
- Include fast Voice Models Checkbox: (Fast generate at cost of audio quality) Click this to be able to see every other model and singular voices supported by Coqui TTS.
- It will update the "Select TTS Model" Dropdown for voice cloning models to also include (List of values to be added).
- It will update the Dropdown for voices to select for each character to also include (List of values to be added).
- Make all audio generate with Narrator voice Checkbox: This will make every character's audio be generated with the voice you have selected for the Narrator when you click the "Generate audio" button.
- Clone new voice Button: Click this to add a new voice you can clone (make sure you have a reference audio file on hand).
- Add Fine-tuned Xtts model to voice actor Button: If you have a folder containing all the parameters of a fine-tuned Xtts model of a specific voice, then you can click this to make that voice actor clone with that fine-tuned Xtts model, to provide much better voice cloning results.
- Character voices Dropdowns: These are the dropdowns for selecting the Voice Actor (and the Accent of each character if using XTTS).
- (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
- When you select a voice It will play the audio sample of that voice, if it's a fast voice model voice and a refrence audio does not exist, then it will generate one to play.
- (2): The Accents available to select from for this character. (Optional, Default is English).
- (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
- Chapter Delimiter Field: Will change the default chapter delimiter (The string that's used to identify chapters).
- Silence Duration in milliseconds (ms) Field: This will change the amount of milliseconds in between each combined chunk of audio.
- Select TTS Language Dropdown: This will let you select the default Accent used for every character which has not had the Accent manually selected for.
- Loading bar: Will give an approximate amount of time left. (Estimate, you probably won't see accurate predictions until it's been running for 5 min).
- Annotated book preview Block: This will show the entirety of the book with each character's lines color-coded.
- You can click on a line while the audiobook is being generated to hear what that generated line sounds like. But only if the line has already had audio generated for it; if not, it'll play nothing.
- Load Book Button: Clicking this will reload the color-coded annotated book view, it will just randomize the selected colors for each character's lines.
- Generate Audio Button: Will start generating the full audiobook.
- Select random voices Button (Will only be visible if the "include fast Voice Models" checkbox is checked): Will Select an auto-gender-inferred fast model voice for every character except for the narrator's voice.
GUI Part 3 (Book Viewer)
-It's hard to explain its more of a playground if you mess around with it then you should get how it works. But it can be used to fine tune the audiobook -Close out of the window when your done with it.- Free and entirely locally run
- Supports all ebook file formats by using calibre
- Can run on CPU or CUDA GPU
- Autoselects a starting estimated voice by pronouns per character
- Supports all models in Coqui TTS and all voices in the models
- Easily create a new voice actor in seconds through voice cloning in GUI
- Can play audio by clicking on the text in the book viewer in GUI
- Ability to regenerate specific lines if they came out weird
- Ability to add custome fine tuned models for specific voice with the click of a button in GUI
- Outputs a single file output as m4b to include all the metadata(chapters book image ect)
- Supports STYLETTS2 as a model you can select from for voice cloning for WICKED FAST speed (even on cpu)
- Includes 26 default voices for cloning
- Low vram option(Details in readme)
- Gui Docker image (Graphical interface-no sound in gui yet)
- Headless version of docker image
- Headless version of VoxNovel
- A Google Colab using the headless version
- Ability to change the character for a line if incorrectly attributed by booknlp
- Make it so that all the included voices and models already have their premade own demo voices
- Make it so that the demo audio for the cloned voices is not their reference audio but what their voices sound like generated
- Using whisper transcriptions to cut hallucinations out of generated audio
- Incorporating local model to generate sound effects when a book discribes a location or sound effect
-@sidharthrajaram (for his Styletts2 pip install he created, I couldn't of added styletts2 without him. :) ) (https://github.com/sidharthrajaram/StyleTTS2)