learning-and-intelligent-systems / llm4pddl Goto Github PK

View Code? Open in Web Editor NEW

36.0 36.0 3.0 4 MB

Large language models for PDDL domains

Python 0.87% Shell 0.01% Common Lisp 3.34% PDDL 95.79%

llm4pddl's People

Contributors

Stargazers

Watchers

Forkers

aaronhd dtrycode danielqujun

llm4pddl's Issues

make breaking vs continuing after an invalid action a flag

and try both

make sure that dressed problems are different enough

analyze the problem sizes for the pyperplan benchmark tasks

change analyze results node counting to average over only tasks solved by all approaches

Could you please add an instruction to help us understand how to use the codebase?

It is hard for me to figure out how to run it successfully
Thank you!

change the llm cache file names to not use hashing

I'm running into a weird case where hashing on my macOS laptop is different from hashing on the ubuntu cluster, even with PYTHONHASHSEED = 0 set on both. we should just change the way that LLM cache files are constructed to not use the built-in hash.

data augmentation: only keep tasks whose plans use all the operators

autogressive prompting + name randomization doesn't update prompt correctly

add manual training problems

to see whether careful prompt engineering can succeed

fix analyze_results.py in case where experiment_id is not defined, or different from standard format

try blocks where the names of the blocks have integers

and put them in order for the goal

add training index offset

to allow for prompting on a more complicated problem

remove `get_pyperplan_benchmark_task` and `get_custom_benchmark_task`

try blocks with data from manually designed generalized policy

main question will be whether the model is capable of reasoning through what is on the bottom.

create AB testing script

save and load datasets

logs don't get written when locally launching configs

always randomize the names of objects

to prevent "overfitting" in cases where the object names are the same in the train and test problems

add option to do multiple queries for multi-* approaches

rather than one query with num_completions set.

this matters because of the token rate limit

add cached problem_str and domain_str to Task and refactor accordingly

to save from redundant file reading

consider removing domains that have per-problem domain files

these are: airplane, openstack, parcprinter, psr-small

implement timeout for all of planning, and for LLM

right now, the timeouts are only for searching within planning, not for grounding

Potentially unneeded queries being made when importing tokenizer

implement PG3 baseline

if we continue to lean in the generalized planning direction

but note that PG3 has privileged access to the operators

compare against FD with hff and gbfs

currently the node results are favoring pyperplan, which is weird. see if pyperplan and FD match when the algorithm is the same.

switch parentheses format in prompts

currently we're doing ontable(b1) but we should be doing (ontable b1). probably won't make a difference, but good to be consistent with standard PDDL. we'll have to check if this makes a difference.

consider switching eval problems to be hardest ones, rather than next-easiest ones

would lead to more stability in testing, and could potentially should a greater difference between ours and baselines