technion-kishony-lab / data-to-paper Goto Github PK

data-to-paper is a framework that systematically navigates the power of AI through a complete end-to-end scientific research, starting from raw data and concluding with comprehensive, transparent, backward-traceable, human-verifiable scientific papers (example).

Implementation. Towards this goal, data-to-paper systematically guides interacting LLM and rule-based agents through the conventional scientific path, from annotated data, through creating research hypotheses, conducting literature search, writing and debugging data analysis code, interpreting the results, and ultimately the step-by-step writing of a complete research paper. Tracking information flow through these steps, the process creates transparent and highly-traceable manuscripts, where results, methodology and data are programmatically linked (demo).

The data-to-paper framework is created as a research project to understand the capacities and limitations of LLM-driven scientific research, and to develop ways of harnessing LLM to accelerate research while maintaining, and even enhancing, key scientific values, such as transparency, traceability and verifiability, and while allowing scientist to oversee and direct the process (see also: living guidelines).

The data-to-paper framework is described in the following pre-print:

• Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay and Roy Kishony, "Autonomous LLM-driven research from data to human-verifiable research papers", arXiv:2404.17605

• Data-chained manuscripts. The process creates transparent and verifiable manuscripts, where results, methodology and data are programmatically linked (all numeric values can be click-traced back to the code lines that created them).
• Field agnostic. We strive to make the framework as general as possible, so that it can be used across different fields of research.
• Open-goal or fixed-goal research. data-to-paper can be used to autonomously raise and test a hypothesis, or to test a specific pre-defined user-provided hypothesis.
• Coding guardrails. Standard statistical packages are overridden with multiple guardrails to minimize common LLM coding errors.
• Human-in-the-loop. A GUI app allows the user to oversee the process, and to intervene at each research step.
• Record & Replay. The entire process is recorded, including all LLM responses, Human feedback, and literature search retrievals, allowing for transparent replay.

See INSTALL.md for installation instructions.

1. Install data-to-paper (INSTALL).
2. Run data-to-paper: python data_to_paper/data_to_paper/scripts/run.py
3. This will open a startup dialog that will allow you to specify your own project, or to reproduce example projects (data-to-paper/projects)
4. Click "Start" to start the run, with human overseeing and feedback.
5. At the end of the process, a pdf of the manuscript will be created in the project folder.

We ran data-to-paper on the following test cases:

• Health Indicators (open goal). A clean unweighted subset of CDC’s Behavioral Risk Factor Surveillance System (BRFSS) 2015 annual dataset (Kaggle). Here is an example Paper created by data-to paper.

Try out: python run.py diabetes

• Social Network (open goal). A directed graph of Twitter interactions among the 117th Congress members (Fink et al). Here is an example Paper created by data-to paper.

Try out: python run.py social_network

• Treatment Policy (fixed-goal). A dataset on treatment and outcomes of non-vigorous infants admitted to the Neonatal Intensive Care Unit (NICU), before and after a change to treatment guidelines was implemented (Saint-Fleur et al). Here is an example Paper created by data-to paper.

Try out: python run.py npr_nicu

• Treatment Optimization (fixed-goal). A dataset of pediatric patients, which received mechanical ventilation after undergoing surgery, including an x-ray-based determination of the optimal tracheal tube intubation depth and a set of personalized patient attributes to be used in machine learning and formula-based models to predict this optimal depth (Shim et al). Here is an example Paper created by data-to paper.

We defined three levels of difficulty for the research question for this paper.

1. easy: Compare two ML methods for predicting optimal intubation depth
   Try out: python run.py ML_easy

2. medium: Compare one ML method and one formula-based method for predicting optimal intubation depth
   Try out: python run.py ML_medium

3. hard: Compare 4 ML methods with 3 formula-based methods for predicting optimal intubation depth
   Try out: python run.py ML_hard

We invite people to try out data-to-paper with their own data and are eager for feedback and suggestions. It is currently designed for relatively simple research goals and simple datasets, where we want to raise and test a statistical hypothesis.

We also invite people to help develop and extend the data-to-paper framework in science or other fields.

Disclaimer. By using this software, you agree to assume all risks associated with its use, including but not limited to data loss, system failure, or any other issues that may arise, especially, but not limited to, the consequences of running of LLM created code on your local machine. The developers of this project do not accept any responsibility or liability for any losses, damages, or other consequences that may occur as a result of using this software.

Accountability. You are solely responsible for the entire content of created manuscripts including their rigour, quality, ethics and any other aspect. The process should be overseen and directed by a human-in-the-loop and created manuscripts should be carefully vetted by a domain expert. The process is NOT error-proof and human intervention is necessary to ensure accuracy and the quality of the results.

Compliance. It is your responsibility to ensure that any actions or decisions made based on the output of this software comply with all applicable laws, regulations, and ethical standards. The developers and contributors of this project shall not be held responsible for any consequences arising from using this software. Further, data-to-paper manuscripts are watermarked for transparency as AI-created. Users should not remove this watermark.

Token Usage. Please note that the use of most language models through external APIs, especially GPT4, can be expensive due to its token usage. By utilizing this project, you acknowledge that you are responsible for monitoring and managing your own token usage and the associated costs. It is highly recommended to check your API usage regularly and set up any necessary limits or alerts to prevent unexpected charges.

Here are some other cool multi-agent related projects:

• LangChain
• AutoGen
• AutoGPT
• MetaGPT

And also this curated list of awesome-agents.