macrocosm-os / prompting Goto Github PK

View Code? Open in Web Editor NEW

68.0 7.0 38.0 6.29 MB

SN1: An incentive mechanism for internet-scale conversational intelligence

Home Page: https://bittensor.com

License: MIT License

Python 96.09% Shell 3.89% Makefile 0.02%

ai blockchain llm network cryptocurrency

prompting's Introduction

Bittensor SN1

The Incentivized Internet

Discord • Network • Research

This repository is the official codebase for Bittensor Subnet 1 (SN1) v1.0.0+, which was released on 22nd January 2024. To learn more about the Bittensor project and the underlying mechanics, read here..

Introduction

This repo defines an incentive mechanism to create a distributed conversational AI for Subnet 1 (SN1).

Validators and miners are based on large language models (LLM). The validation process uses internet-scale datasets and goal-driven behaviour to drive human-like conversations.

Installation

This repository requires python3.9 or higher. To install it, simply clone this repository and run the install.sh script.

git clone https://github.com/opentensor/prompting.git
cd prompting
bash install.sh

If you are running a miner, you will also need to uninstall uvloop.

pip uninstall uvloop -y

If you are running a validator, logging in to Hugging Face is required:

huggingface-cli login

You also need to accept the License Agreement for the LMSYS-Chat-1M dataset: https://huggingface.co/datasets/lmsys/lmsys-chat-1m

Compute Requirements

To run a validator, you will need at least 62GB of VRAM.
To run the default huggingface miner, you will need at least 62GB of VRAM.

It is important to note that the baseminers are not recommended for main, and exist purely as an example. Running a base miner on main will result in no emissions and a loss in your registration fee. If you have any questions please reach out in the SN1 channel in the Bittensor Discord.

How to Run

You can use the following command to run a miner or a validator.

python <SCRIPT_PATH>
    --netuid 1
    --subtensor.network <finney/local/test>
    --neuron.device cuda
    --wallet.name <your wallet> # Must be created using the bittensor-cli
    --wallet.hotkey <your hotkey> # Must be created using the bittensor-cli
    --logging.debug # Run in debug mode, alternatively --logging.trace for trace mode
    --axon.port # VERY IMPORTANT: set the port to be one of the open TCP ports on your machine

where SCRIPT_PATH is either:

neurons/miners/openai/miner.py
neurons/validator.py

For ease of use, you can run the scripts as well with PM2. Installation of PM2 is: On Linux:

sudo apt update && sudo apt install jq && sudo apt install npm && sudo npm install pm2 -g && pm2 update

Example of running an Openai miner on Main:

pm2 start neurons/miners/openai/miner.py --interpreter python --name openai_miner -- --netuid 1  --subtensor.network finney --wallet.name my_wallet --wallet.hotkey my_hotkey --neuron.model_id gpt-3.5-turbo-1106 --axon.port 8091

Running with autoupdate

You can run the validator in auto-update mode by using pm2 along with the run.sh bash script. This command will initiate two pm2 processes: one for auto-update monitoring, named s1_validator_update, and another for running the validator itself, named s1_validator_main_process.

pm2 start run.sh --name s1_validator_autoupdate -- --wallet.name <your-wallet-name> --wallet.hotkey <your-wallet-hot-key>

Note: this is not an end solution, major releases or changes in requirements will still require you to manually restart the processes. Regularly monitor the health of your validator to ensure optimal performance.

Testnet

We highly recommend that you run your miners on testnet before deploying on main. This is give you an opportunity to debug your systems, and ensure that you will not lose valuable immunity time. The SN1 testnet is netuid 61.

In order to run on testnet, you will need to go through the same hotkey registration proceure as on main, but using testtao. You will need to ask for some in the community discord if you do not have any.

To run:

pm2 start neurons/miners/openai/miner.py --interpreter python3 --name openai_miner -- --netuid 61 --subtensor.network test --wallet.name my_test_wallet --wallet.hotkey my_test_hotkey --neuron.model_id gpt-3.5-turbo-1106 --axon.port 8091

Limitations

Important: vLLM currently faces a notable limitation in designating a specific GPU for model execution via code. Consequently, to employ a particular CUDA device for your model's operations, it's necessary to manually adjust your environment variable CUDA_VISIBLE_DEVICES. For instance, setting export CUDA_VISIBLE_DEVICES=1,2 will explicitly define the CUDA devices available for use.

Resources

The archiecture and methodology of SN1 is complex, and as such we have created a comprehensive resource to outline our design. Furthermore, we have strict requirements for how miners should interact with the network. Below are the currently available resources for additional information:

prompting's People

Contributors

Stargazers

Watchers

prompting's Issues

Complement validator data with miners key in wandb

Validators from the community mentioned that they are getting blacklisted by some miners. It would be valuable for the validator logs to have this information as it can accelerate analytics. The implementation could be easily deployed by selecting the coldkeys and hotkeys from the axons at the forward step.

Logging: Solid schema to facilitate downstream maintenance

Recently the log schema was changed by the addition of new tasks, which created some maintenance work on dowstream tasks of extraction. We faced the same issue on previous versions of S1, where the solution of having a typed schema (EventSchema) facilitated the progression of the schema of our logs, hence facilitating maintanence.

Proposed solution:
We should add a typed schema to easily normalize and version the data structure we log in wandb.
https://github.com/opentensor/prompting/blob/5bc2e3c159b6ad1f3e14806b1b2003c172df1f4c/prompting/utils/logging.py#L106-L119

Update bittensor version to bittensor>=6.9.3

We are using an older version of bittensor that could bring integration issues with the chain in prod (as it already did on testnet). The recommended version from philanthrope was bittensor>=6.9.3, the latest version is 6.10.0 and the current version on prompting repo is 6.6.0.

Improve wikipedia retrieval mechanism

Replace manual wikipedia wrapper that we have implemented with wikipedia package

https://pypi.org/project/wikipedia/

Expand date qa task to be more challenging

An idea for improving date qa task would be to expand the source material beyond wikipedia date pages of which there are only 365, which is susceptible to lookup.. Instead of using wiki date pages we could probably just scrape wiki articles and detect dates in a more generic way.

There's a couple steps:

Detect date in an article -> reference
Extract context -> query

this will possibly require generation, unlike current dateQA, because the sentence will need some processing. See the current article of the day in wiki. Theres actually a panel which contains the date in a readily available format on the right - we could work with that

Update Readme to discourage use of the baseminer

The baseminers are not competitive on the network, and it would be useful to new miners for our repo to reflect this.

TypeError: Miner.init() got an unexpected keyword argument 'config'

While running the miner script getting the following error:

TypeError: Miner.__init__() got an unexpected keyword argument 'config'

Script:

yusuf@miner:~/tao-mining/prompting$ python3 neurons/miners/zephyr/miner.py --netuid 1 --wallet.name m2 --wallet.hotkey m2h --neuron.model_id HuggingFaceH4/zephyr-7b-beta --axon.port 8890

Any python guru here knows why?

Kill operation on node deregistration

Both validators and miners should stop operating whenever the node gets deregistered. The previous text-prompting repo handles that at the validator side, we should ensure this behaviour is enabled by default on the new repo, both for miners and validators

Async Priority Function not behaving as intended.

The priority function defined in the base miner does not behave as intended because of async keyword.

Add black in CI pipeline

It would be nice to have a step on our CI/CD pipeline to check if the repo is blacked, which would force us to properly format the code on each PR, leading to a uniform formatting of the repo.

UnknownSynapseError: Synapse name 'NeuronInfoSynapse' not found.

I see the trust scocre of my miner fluctuating. it seems like a falling score might be related to this occasional error?

2024-03-07 14:02:18.678 | ERROR | UnknownSynapseError: Synapse name 'NeuronInfoSynapse' not found. Available synapses ['Synapse', 'PromptingSynapse']

2024-03-07 14:01:42.552 |      DEBUG       | axon     | --> | 2052 B | PromptingSynapse | 5HbLYXUBy1snPR8nfioQ7GoA9x76EELzEq9j7F32vWUQHm1x | 213.173.105.9:38102  | 200 | Su>
2024-03-07 14:01:44.606 |       INFO       | Miner running:: network: local | step: 369772 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:01:48.395 |      DEBUG       | axon     | <-- | 1221 B | PromptingSynapse | 5Hddm3iBFD2GLT5ik7LZnT3XJUnRnN8PoeCFgGQgawUVKNm8 | 64.247.206.99:50214 | 200 | Suc>
2024-03-07 14:01:48.396 |      DEBUG       | 📧 Message received, forwarding synapse: required_hash_fields=['messages'] roles=['user'] messages=["Can you provide me with in>
2024-03-07 14:01:48.396 |      DEBUG       | 💬 Querying zephyr: Can you provide me with information about the theme explored in thomas hunt's poetry as presented in the po>
2024-03-07 14:01:49.612 |       INFO       | Miner running:: network: local | step: 369772 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:01:54.617 |       INFO       | Miner running:: network: local | step: 369772 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:01:55.156 |       INFO       | HuggingFaceLLM generated the following output:
Certainly! The collection "The Poetical Works of Leigh Hunt and Thomas Hood, edited by Scott" published in 1889, showcases the poetry of two British Romantic writers, Leigh>
Hunt's poetry in this collection often portrays the natural world as a source of spiritual and emotional solace, as well as a reflection of the human condition. In "The Lig>
Another recurring theme in Hunt's poetry in this collection is the idea of beauty and its transformative power. In "The Hunchback in the Garden," Hunt explores the relation>
2024-03-07 14:01:55.156 |      DEBUG       | ✅ Served Response: Certainly! The collection "The Poetical Works of Leigh Hunt and Thomas Hood, edited by Scott" published in >
Hunt's poetry in this collection often portrays the natural world as a source of spiritual and emotional solace, as well as a reflection of the human condition. In "The Lig>
Another recurring theme in Hunt's poetry in this collection is the idea of beauty and its transformative power. In "The Hunchback in the Garden," Hunt explores the relation>
2024-03-07 14:01:55.165 |      DEBUG       | axon     | --> | 2224 B | PromptingSynapse | 5Hddm3iBFD2GLT5ik7LZnT3XJUnRnN8PoeCFgGQgawUVKNm8 | 64.247.206.99:50214  | 200 | Su>
2024-03-07 14:01:55.166 |      DEBUG       | axon     | <-- | 1195 B | PromptingSynapse | 5F4tQyWrhfGVcNhoqeiNsR6KjD4wMZ2kfhLj4oHYuyHbZAc3 | 104.171.202.244:33094 | 200 | S>
2024-03-07 14:01:55.167 |      DEBUG       | 📧 Message received, forwarding synapse: required_hash_fields=['messages'] roles=['user'] messages=["Can you tell me when princ>
2024-03-07 14:01:55.167 |      DEBUG       | 💬 Querying zephyr: Can you tell me when prince willem became the first constitutional monarch in the netherlands? I'm too lazy>
2024-03-07 14:01:57.091 |       INFO       | HuggingFaceLLM generated the following output:
Prince Willem-Alexander became the first constitutional monarch of the Netherlands on April 30, 2013, upon the abdication of his mother, Queen Beatrix. As a constitutional >
2024-03-07 14:01:57.092 |      DEBUG       | ✅ Served Response: Prince Willem-Alexander became the first constitutional monarch of the Netherlands on April 30, 2013, upon >
2024-03-07 14:01:57.098 |      DEBUG       | axon     | --> | 1428 B | PromptingSynapse | 5F4tQyWrhfGVcNhoqeiNsR6KjD4wMZ2kfhLj4oHYuyHbZAc3 | 104.171.202.244:33094  | 200 | >
2024-03-07 14:01:59.622 |       INFO       | Miner running:: network: local | step: 369774 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:04.628 |       INFO       | Miner running:: network: local | step: 369774 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:09.633 |       INFO       | Miner running:: network: local | step: 369774 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:11.930 |      DEBUG       | axon     | <-- | 1169 B | PromptingSynapse | 5HK5tp6t2S59DywmHRWPBVJeJ86T61KjurYqeooqj8sREpeN | 207.189.112.60:45158 | 200 | Su>
2024-03-07 14:02:11.931 |      DEBUG       | 📧 Message received, forwarding synapse: required_hash_fields=['messages'] roles=['user'] messages=["Can you provide a brief su>
2024-03-07 14:02:11.931 |      DEBUG       | 💬 Querying zephyr: Can you provide a brief summary of jarmo lehtinen's career in the tech industry, highlighting his most sign>
2024-03-07 14:02:14.638 |       INFO       | Miner running:: network: local | step: 369774 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:18.668 |       INFO       | HuggingFaceLLM generated the following output:
Sure, Jarmo Lehtinen is a highly respected figure in the tech industry with over 30 years of experience. Here are some of his most significant accomplishments and contribut>
1. Co-founder of Nokia's Symbian operating system: In the late 1990s, Lehtinen co-founded Symbian, a consortium that developed the Symbian OS for mobile devices. The operat>
2. Key player in the development of Nokia's smartphones: Lehtinen played a pivotal role in the development of Nokia's smartphones, which helped the company maintain its pos>
3. Contributor to open-source projects: Lehtinen has been a strong advocate for open-source software and has contributed to several projects, including the Linux kernel and>
4
2024-03-07 14:02:18.669 |      DEBUG       | ✅ Served Response: Sure, Jarmo Lehtinen is a highly respected figure in the tech industry with over 30 years of experience. He>
1. Co-founder of Nokia's Symbian operating system: In the late 1990s, Lehtinen co-founded Symbian, a consortium that developed the Symbian OS for mobile devices. The operat>
2. Key player in the development of Nokia's smartphones: Lehtinen played a pivotal role in the development of Nokia's smartphones, which helped the company maintain its pos>
3. Contributor to open-source projects: Lehtinen has been a strong advocate for open-source software and has contributed to several projects, including the Linux kernel and>
4
2024-03-07 14:02:18.677 |      DEBUG       | axon     | --> | 2162 B | PromptingSynapse | 5HK5tp6t2S59DywmHRWPBVJeJ86T61KjurYqeooqj8sREpeN | 207.189.112.60:45158  | 200 | S>
2024-03-07 14:02:18.678 |      ERROR       | UnknownSynapseError: Synapse name 'NeuronInfoSynapse' not found. Available synapses ['Synapse', 'PromptingSynapse']
2024-03-07 14:02:18.681 |      DEBUG       | axon     | --> | 108 B | Synapse | None | None:None  | 404 | Synapse name 'NeuronInfoSynapse' not found. Available synapses ['S>
2024-03-07 14:02:19.644 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:24.649 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:29.655 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:34.660 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:39.665 |       INFO       | Miner running:: network: local | step: 370289 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:44.670 |       INFO       | Miner running:: network: local | step: 375259 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:49.675 |       INFO       | Miner running:: network: local | step: 380306 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:11.931 |      DEBUG       | 💬 Querying zephyr: Can you provide a brief summary of jarmo lehtinen's career in the tech industry, highlighting his most sign>
2024-03-07 14:02:14.638 |       INFO       | Miner running:: network: local | step: 369774 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:18.668 |       INFO       | HuggingFaceLLM generated the following output:
Sure, Jarmo Lehtinen is a highly respected figure in the tech industry with over 30 years of experience. Here are some of his most significant accomplishments and contribut>
1. Co-founder of Nokia's Symbian operating system: In the late 1990s, Lehtinen co-founded Symbian, a consortium that developed the Symbian OS for mobile devices. The operat>
2. Key player in the development of Nokia's smartphones: Lehtinen played a pivotal role in the development of Nokia's smartphones, which helped the company maintain its pos>
3. Contributor to open-source projects: Lehtinen has been a strong advocate for open-source software and has contributed to several projects, including the Linux kernel and>
4
2024-03-07 14:02:18.669 |      DEBUG       | ✅ Served Response: Sure, Jarmo Lehtinen is a highly respected figure in the tech industry with over 30 years of experience. He>
1. Co-founder of Nokia's Symbian operating system: In the late 1990s, Lehtinen co-founded Symbian, a consortium that developed the Symbian OS for mobile devices. The operat>
2. Key player in the development of Nokia's smartphones: Lehtinen played a pivotal role in the development of Nokia's smartphones, which helped the company maintain its pos>
3. Contributor to open-source projects: Lehtinen has been a strong advocate for open-source software and has contributed to several projects, including the Linux kernel and>
4
2024-03-07 14:02:18.677 |      DEBUG       | axon     | --> | 2162 B | PromptingSynapse | 5HK5tp6t2S59DywmHRWPBVJeJ86T61KjurYqeooqj8sREpeN | 207.189.112.60:45158  | 200 | S>
2024-03-07 14:02:18.678 |      ERROR       | UnknownSynapseError: Synapse name 'NeuronInfoSynapse' not found. Available synapses ['Synapse', 'PromptingSynapse']
2024-03-07 14:02:18.681 |      DEBUG       | axon     | --> | 108 B | Synapse | None | None:None  | 404 | Synapse name 'NeuronInfoSynapse' not found. Available synapses ['S>
2024-03-07 14:02:19.644 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:24.649 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:29.655 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:34.660 |       INFO       | Miner running:: network: local | step: 369775 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:39.665 |       INFO       | Miner running:: network: local | step: 370289 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:44.670 |       INFO       | Miner running:: network: local | step: 375259 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:49.675 |       INFO       | Miner running:: network: local | step: 380306 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:50.167 |       INFO       | resync_metagraph()
2024-03-07 14:02:54.682 |       INFO       | Miner running:: network: local | step: 380798 | uid: 894 | trust: 0.579 | emission 0.001
2024-03-07 14:02:57.189 |       INFO       | resync_metagraph()
2024-03-07 14:02:59.697 |       INFO       | Miner running:: network: local | step: 380799 | uid: 894 | trust: 0.480 | emission 0.000
2024-03-07 14:03:04.727 |       INFO       | Miner running:: network: local | step: 380799 | uid: 894 | trust: 0.480 | emission 0.000
2024-03-07 14:03:05.173 |       INFO       | resync_metagraph()
2024-03-07 14:03:09.822 |       INFO       | Miner running:: network: local | step: 380800 | uid: 894 | trust: 0.480 | emission 0.000

Add support for streaming

It would be very nice to have our validators and miners supporting streaming communication, this could bring a lot of value whenever we pivot to deploy a dapp on top of sn1

Create bash setup script to accelerate miner deployment on generic cloud machines

We should try to accelerate the experience of people that run on generic machines such as digital ocean by automating basic setup commands such as

install pip
automate dependencies install

Expand personas

Moods

terse
playful
hyped
foul
celebratory
indignant

Tones

skeptical
apologetic
unapologetic
nonchalant
eager
confused

Add generic 🤗 HF miner

The zephyr miner is quite dynamic already, I believe that with a bit of effort, we can transform it into a generic HF miner, which would facilitate the experiments and mining experience a lot.

The model id and extra model params could be passed as parameters to the miner, which will ingest into the hf pipeline that is implemented

Extend max completion length

Ensure that math challenges contain necessary information

There are edge cases where math challenges modify or otherwise corrupt the underlying math problem. We can address this by validating the challenge. Once flagged, the options are to regenerate the challenge or to reject the task altogether.

Two ways to detect inconsistent math challenges:

check that all numeric values in the query are also present
each math task context specifies all essential keywords (including numbers) that must be present

Raise exception if port already in use or is otherwise not open

To address a common pain-point for miners and validators we should check if the axon port is:

open on the machine
already in use

In both cases, the process should terminate (loudly).

Improve Math Task

Some math challenges are low quality or rely on unreasonably large computations.

Minimize challenge hallucinations

Below is a sample of a hallucination challenge. This should be treated somehow so miners don't get polluted challenges, either by improving the base model or by adding some railguards against noisy challenges

Sample:

Assistant, can you provide a brief summary of some notable military victories where the odds were heavily against the winning side, specifically from arabic-language sources? I am particularly interested in learning about these triumphs as they can offer valuable insights into the strategies and tactics employed by forces operating under challenging circumstances. Your assistance would be greatly appreciated as i strive to gather more information on this fascinating topic. Thank you in advance for your help. Best regards,

[your name]

p.s. If you need any additional context or background information, please do not hesitate to let me know. I want to ensure that your responses are as accurate and detailed as possible.Assistant, can you provide a brief summary of some notable military victories where the odds were heavily against the winning side, specifically from arabic-language sources? I am particularly interested in learning about these triumphs as they can offer valuable insights into the strategies and tactics employed by forces operating under challenging circumstances. Your assistance would be greatly appreciated as i strive to gather more information on this fascinating topic. Thank you in advance for your help. Best regards,

[your name]

p.s. If you need any additional context or background information, please do not hesitate to let me know. I want to ensure that your responses are as accurate and detailed as possible.Assistant, can you provide a brief summary of some notable military victories where the odds were heavily against the winning side, specifically from arabic-language sources? I am particularly interested in learning about these triumphs as they can offer valuable insights into the strategies and tactics employed by forces operating under challenging circumstances. Your assistance would be greatly appreciated as i strive to gather more information on this fascinating topic. Thank you in advance for your help. Best regards,

[your name]

p.s. If you need any additional context or background information, please do not hesitate to let me know. I want to ensure that your responses are as accurate and detailed as possible.Assistant, can you provide a brief summary of some notable military victories where the odds were heavily against the winning side, specifically from arabic-language sources? I am particularly interested in learning about these triumphs as they can offer valuable insights into the strategies and tactics employed by forces operating under challenging circumstances. Your assistance would be greatly appreciated as i strive to gather more information on this fascinating topic. Thank you in advance for your help. Best regards,

[your name]

p.s. If you need any additional context or background information, please do not hesitate to let me know. I want to ensure that your responses are as accurate and detailed as possible.Assistant, can you provide a brief summary of some notable military victories where the odds were heavily against the winning side, specifically from arabic-language sources? I am particularly interested in learning about these triumphs as they can offer valuable insights into the strategies and tactics employed by forces operating under challenging circumstances. Your assistance would be greatly appreciated as i strive to gather more information on this fascinating topic. Thank you in advance for your help. Best regards,

[your name]

p.s. If you need any additional context or background information, please do not hesitate to let me know. I want to ensure that your responses are as accurate and detailed as possible.

Cleaner Pipeline should remove \n

Can see a \n in the message here. Remove!

2.3.0 - Post-release analysis

explore incentive landscape (p1) @p-ferreira @bkb2135
- create a generic script to facilitate future development
load all current runs and look for repetitive challenges (p2) @bkb2135
- create a generic script to facilitate future development
ensuring that validators fail gracefully (p3) @dbobrenko
- Reference run: https://wandb.ai/opentensor-dev/alpha-validators/runs/9qwcfbit?nw=nwuserpedromacrocosmos
- PR: #243

Prevent miners from weight setting

This is causing unnecessary strain on the endpoints of bittensor and does not bring value to the network.

Increase efficiency of wiki context creation

At present, 40-50% of wiki articles are rejected because they do not contain sections with greater than our minimum number of words. This introduces extra latency as we sequentially download additional articles until finding a suitable one.

We should instead download batches of pages concurrently so that we don't need to wait for sequential processing times.

Remove unecessary imports for miners

Importing prompting is throwing errors for miners because of the angle model and it's required imports. It may be beneficial to not import all of prompting for the spec version.

Add pm2 in install script

It would be useful to have the pm2 install commands defined in the README into the install.sh script in order to streamline the setup process

Update Baseminers

Baseminers should allow for multi-turn conversation, and with this, there should be an update to their readme.

Stop Redundant Reference Generation

Currently, we generate references in the task, as well as in the async reference generation process

Stop Validators from Sending Empty Challenges

For wiki-based tasks, validators send empty challenges ~1.2% of the time.

Bug on math generator

I have faced the following issue several times after trying to create a math task:

Current behavior:
Math tasks occasionally throw an exception (image above) on instantiation (usage e.g. task = MathTask(llm_pipeline=llm_pipeline, context=dataset.next()).)

Expected behavior:
Math tasks should be created consistently without any issues.

config parsing error in Zephyr mining py file

i am running the command from cmd: python neurons/miners/zephyr/miner.py --wallet.name myTaoCold --wallet.hotkey myTaoHot --subtensor.network local --netuid 1 --logging.debug True --neuron.model_id HuggingFaceH4/zephyr-7b-beta --neuron.system_prompt "Hello, I am a chatbot. I am here to help you with your questions." --neuron.max_tokens 64 --neuron.do_sample True --neuron.temperature 0.9 --neuron.top_k 50 --neuron.top_p 0.95 --wandb.on True --wandb.entity sn1 --wandb.project_name miners_experiments --neuron.load_quantized True
I keep getting the error TypeError: Miner.init() got an unexpected keyword argument 'config' in line 50

If i remove the default config argument then i get the error AttributeError: 'NoneType' object has no attribute 'load_quantized'

Update logging to use python rather than loguru

According to this commit, bittensor does not use loguru anymore. It would be nice to adopt this on prompting itself, following best practices of python.

Fix date qa inconsistencies

The default behaviour of date qa is to pass url-like wikipedia page titles (January_01) to the api rather than space-separated (January 1). This is a bug that I introduced because in the pre-context implementation we always passed the page title via url params. This, combined with recent changes to the default behaviour wiki api (specifically redirect=True and auto_suggest=True) cause the page not to be found because of the incorrect formatting, which triggers a redirect.

>>> wiki.page(title="November_20", pageid=None, auto_suggest=True, redirect=True)
<WikipediaPage 'November 2'>
>>> wiki.page(title="January_11", pageid=None, auto_suggest=True, redirect=True)
<WikipediaPage 'January 1'>
>>> wiki.page(title="January_12", pageid=None, auto_suggest=True, redirect=True)
<WikipediaPage 'January 1'>
>>> wiki.page(title="January_22", pageid=None, auto_suggest=True, redirect=True)
<WikipediaPage 'January 12'>

With the proposed change of redirect=False and auto_suggest=False we get the correct page:

>>> wiki.page(title="November_20", pageid=None, auto_suggest=False, redirect=False)
<WikipediaPage 'November 20'>
>>> wiki.page(title="January_11", pageid=None, auto_suggest=False, redirect=False)
<WikipediaPage 'January 11'>
>>> wiki.page(title="January_12", pageid=None, auto_suggest=False, redirect=False)
<WikipediaPage 'January 12'>
>>> wiki.page(title="January_22", pageid=None, auto_suggest=False, redirect=False)
<WikipediaPage 'January 22'>

But there are also instances where the page cannot be found at all. For this reason, the title also needs to be changed from date.strftime("%B_%d") (January_01) to date.strftime("%B %-d") (January 1) .

Update bittensor version to bittensor>=6.9.3

We are using an older version of bittensor that could bring integration issues with the chain in prod (as it already did on testnet). The recommended version from philanthrope was bittensor>=6.9.3, the latest version is 6.10.0 and the current version on prompting repo is 6.6.0.

Improve overall code documentation

Self descriptive, low priority, we should have better documentation at main functions that we use.

Error: TextGenerationPipeline.call() missing 1 required positional argument: 'text_inputs'

Raised by zephyr when trying to run an inference on most recent branch.

AttributeError: 'ZephyrMiner' object has no attribute 'set_weights'

Hi. i am new to bittensor and installed everything acording to documentation. The miner is crashing for me

ubuntu22
apt install docker.io -y
apt install docker-compose -y
docker-compose up -d mainnet-lite
ubuntu-drivers autoinstall

git clone https://github.com/opentensor/prompting.git
cd prompting
sudo apt-get install -y python3-pip
python3 -m pip install -r requirements.txt
python3 -m pip install -e .

python3 neurons/miners/zephyr/miner.py --netuid 1 --subtensor.network local --wallet.name default --wallet.hotkey default --logging.debug  --neuron.model_id HuggingFaceH4/zephyr-7b-beta

/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (2.2.1) or chardet (4.0.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
2024-03-05 21:19:33.191 |       INFO       | Logging path: /home/user/.bittensor/miners/default/default/netuid1/miner
2024-03-05 21:19:33.199 |       INFO       | 
wallet:
  name: default
  hotkey: default
  path: ~/.bittensor/wallets/
subtensor:
  network: local
  chain_endpoint: wss://entrypoint-finney.opentensor.ai:443
  _mock: false
logging:
  debug: true
  trace: false
  record_log: false
  logging_dir: ~/.bittensor/miners
axon:
  port: 8091
  ip: '[::]'
  external_port: null
  external_ip: null
  max_workers: 10
netuid: 1
neuron:
  device: cuda
  epoch_length: 100
  events_retention_size: 2 GB
  dont_save_events: false
  log_full: false
  name: miner
  model_id: HuggingFaceH4/zephyr-7b-beta
  load_quantized: false
  system_prompt: You are a helpful AI assistant. You answer questions, summarize documents,
    and debug code. You are always straight to the point and honest.
  max_tokens: 256
  temperature: 0.7
  top_k: 50
  top_p: 0.95
  stop_on_forward_exception: false
  full_path: /home/user/.bittensor/miners/default/default/netuid1/miner
mock: false
no_background_thread: true
wandb:
  'off': false
  offline: false
  notes: ''
  'on': false
  entity: opentensor-dev
  project_name: alpha-miners
blacklist:
  force_validator_permit: false
  allow_non_registered: false
config: null
strict: false
no_version_checking: false
no_prompt: false

2024-03-05 21:19:33.200 |       INFO       | Setting up bittensor objects. 
2024-03-05 21:19:33.203 |       INFO       | Connected to local network and ws://127.0.0.1:9944.
2024-03-05 21:19:40.938 |       INFO       | Wallet: wallet(default, default, ~/.bittensor/wallets/)
2024-03-05 21:19:40.938 |       INFO       | Subtensor: subtensor(local, ws://127.0.0.1:9944)
2024-03-05 21:19:40.938 |       INFO       | Metagraph: metagraph(netuid:1, n:1024, block:2503448, network:local)
2024-03-05 21:19:40.942 |       INFO       | Running neuron on subnet: 1 with uid 515 using network: ws://127.0.0.1:9944
2024-03-05 21:19:40.942 |     WARNING      | You are allowing non-validators to send requests to your miner. This is a security risk.
2024-03-05 21:19:41.049 |       INFO       | Attaching forward function to miner axon.
2024-03-05 21:19:41.052 |       INFO       | Axon created: Axon([::], 8091, 5Di4b9gGtjksRmsB4TCdkJphKwH23uBEdTcbduJZ6YYbiHmh, stopped, ['Synapse', 'PromptingSynapse'])
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 638/638 [00:00<00:00, 5.96MB/s]
model.safetensors.index.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 23.9k/23.9k [00:00<00:00, 136MB/s]
model-00001-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.89G/1.89G [00:07<00:00, 263MB/s]
model-00002-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.95G/1.95G [00:07<00:00, 246MB/s]
model-00003-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.98G/1.98G [00:08<00:00, 241MB/s]
model-00004-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.95G/1.95G [00:07<00:00, 256MB/s]
model-00005-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.98G/1.98G [00:07<00:00, 257MB/s]
model-00006-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.95G/1.95G [00:07<00:00, 257MB/s]
model-00007-of-00008.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1.98G/1.98G [00:07<00:00, 263MB/s]
model-00008-of-00008.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 816M/816M [00:03<00:00, 265MB/s]
Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:58<00:00,  7.34s/it]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:18<00:00,  2.33s/it]
generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 111/111 [00:00<00:00, 373kB/s]
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.43k/1.43k [00:00<00:00, 9.09MB/s]
tokenizer.model: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 493k/493k [00:00<00:00, 33.5MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.80M/1.80M [00:00<00:00, 4.98MB/s]
added_tokens.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42.0/42.0 [00:00<00:00, 391kB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 168/168 [00:00<00:00, 1.43MB/s]
2024-03-05 21:21:06.089 |      DEBUG       | Starting miner in background thread.
2024-03-05 21:21:06.095 |      DEBUG       | Started                       
2024-03-05 21:21:06.098 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:06.125 |       INFO       | resync_metagraph()            
2024-03-05 21:21:11.109 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:16.128 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:17.653 |       INFO       | Serving miner axon Axon([::], 8091, 5Di4b9gGtjksRmsB4TCdkJphKwH23uBEdTcbduJZ6YYbiHmh, stopped, ['Synapse', 'PromptingSynapse']) on network: wss://entrypoint-finney.opentensor.ai:443 with netuid: 1
2024-03-05 21:21:17.654 |      DEBUG       | Checking axon ...             
2024-03-05 21:21:17.675 |      DEBUG       | Serving axon with: AxonInfo(5Di4b9gGtjksRmsB4TCdkJphKwH23uBEdTcbduJZ6YYbiHmh,45.153.35.155:8091) -> local:1
2024-03-05 21:21:21.139 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:26.145 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:31.150 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:36.156 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:41.161 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:46.166 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:51.170 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:21:56.175 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:01.178 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:06.182 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:11.187 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:13.642 |      DEBUG       | Axon served with: AxonInfo(5Di4b9gGtjksRmsB4TCdkJphKwH23uBEdTcbduJZ6YYbiHmh,45.153.35.155:8091) on local:1 
2024-03-05 21:22:13.646 |       INFO       | Miner starting at block: 2503461
2024-03-05 21:22:13.667 |       INFO       | resync_metagraph()            
2024-03-05 21:22:16.215 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:21.231 |       INFO       | Miner running:: network: local | step: 0 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:22.368 |       INFO       | resync_metagraph()            
2024-03-05 21:22:26.421 |       INFO       | Miner running:: network: local | step: 1 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:31.468 |       INFO       | Miner running:: network: local | step: 1 | uid: 515 | trust: 0.375 | emission 0.001
2024-03-05 21:22:33.152 |      ERROR       | Error during mining           'ZephyrMiner' object has no attribute 'set_weights'
Traceback (most recent call last):
  File "/home/user/prompting/prompting/base/miner.py", line 123, in run
    self.sync()
  File "/home/user/prompting/prompting/base/neuron.py", line 123, in sync
    self.set_weights()
AttributeError: 'ZephyrMiner' object has no attribute 'set_weights'
2024-03-05 21:22:33.155 |      DEBUG       | None                          
2024-03-05 21:22:36.474 |     WARNING      | Ending miner...               
2024-03-05 21:22:36.476 |      DEBUG       | Stopping miner in background thread.
2024-03-05 21:22:36.476 |      DEBUG       | Stopped

Automatic Detection of Dates in Date Scoring

Currently we are using a regex function to capture dates because datetime objects cannot contain all of the years found in the wikipedia date sections. This requires some specificity in the format of the date received, and we probably want to look at an automatic way to extract dates in a variety of formats, perhaps by using the dateutil.parser.parse() method

Overall mining improvements

1. Miners should stop main thread on exception

Miners are running the mining process in a background thread, which may fail and keep the main thread running.
The expected behaviour is similar to what was implemented on the validator, where on background thread fails, the whole process stops

2. Fix `--openai.model_name` on OpenAI miners

Apparently this parameter is not being overwritten by the cli command. Ensure that this parameter gets overwritten if user pass it to the model

3. Miner config to wandb

For some reason the miner config is not being logged to wandb. The config should be logged into wandb and be visible at the overview page

Bump bittensor version

Rare hanging bug when waiting for wikipedia

Was only reported once, but caused a 10h hang for a validator.

10|s1_vali | 2024-02-16 10:44:49.761 |       INFO       | 📋 Selecting task... from ['summarization', 'qa', 'debugging', 'math', 'date_qa'] with distribution [0.5, 0.5, 0.0, 0.0, 0.0]
10|s1_vali | 2024-02-16 10:44:49.761 |       INFO       | 📋 Creating summarization task...
10|s1_vali | 2024-02-16 20:22:38.092 |     SUCCESS      | Validator killed by keyboard interrupt.

We will address this by running api calls in background threads with timeouts.

Ping miners and punish absentees

Failed to create math task. (, TypeError("'float' object cannot be interpreted as an integer"), ). Skipping to next task.

Occurs pretty infrequently but should be looked at

Ensure support for multiple GPUs

The validator should be able to run in a given gpu device.

Potential source of issue:
https://github.com/opentensor/prompting/blob/303f3b4c490c1f5a3164cba191156f66e33b42e3/prompting/utils/config.py#L71

Integrate miners, validators and bittensor with grafana

Ideally we would have oot analytics integration with an open-source observability tool such as grafana. Bittensor can be in theory easily integratred with grafana, which could be leveraged.

Fix Miner Package/Imports

Our Baseminers, though only examples, should be functional.

Reduce penalty weight

The current penalty weight of 1.0 effectively limits the max. score to 0.8 because there is always overlap between the challenge and completion texts. This was desirable at initial launch because we wanted to disincentivize echo mining, however such strategies are not viable at all any more.

Also, we don't want 'gaming' the penalty model to become profitable at the expense of producing high quality reference answers.

My recommendation is that we reduce the penalty to 0.5 for qa and summarization

Miner logging - add message sender data

Miners should be able to log information of the message sender, such as uid, hotkey, coldkey, etc.

Update autopilot script

Currently, the autopilot script only updates if the difference from current to latest version is 1, else it asks the user to manually update. It would be interesting to increase this gap to diff > 0, as validators continue to be in older version due to custom use cases.

It would also be ideal to refactor the bash script to python to facilitate future maintenance.

Improve get_random_uids tests

We should also test that the uids function raises a specific warning/error under certain conditions.

len(uids)<k should log a bt.logging.warning
len(uids)=0 should raise a ValueError

Copied from #142