Type of Issue

The documentation should talk about -

1. datasets folder should be fetched and updated before starting with the app and perhaps point to it
2. The README of template tool should mention auto_complete is in it
3. Can I wipe out the commands.txt entirely ? I think this file should be blank / user should create from scratch
4. What is args in python ~/droidlet/tools/data_processing/txt_to_JSON.py [args] ?
5. I think step #1 and #4 should be run before running the backend and frontend ?
6. Nit to rename : ~/droidlet/tools/data_processing/txt_to_JSON.py to something
7. We need to pretty print the dictionaries
8. The location for where to run python ~/droidlet/tools/data_processing/txt_to_JSON.py should be mentioned
9. Fix the path for annotations_dir_path in the script in #8
10. "point" inside "dance" doesn't work for autocomplete
11. Once we are over or under the index, either wipe out the command field or show a prompt saying "you are done annotating" . Right now, it keeps showing the last message and incrementing the count.
12. We should not track the files : backend/command_dict_pairs.json and backend/commands.txt.
13. when json validation fails, show a pop up.
14. Rename autocomplete_annotation.txt to high_pri_commands.txt in S3

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Detailed description of the requested feature/documentation ask/bug report.

Current Behavior

Description of current behavior or functionality.

Expected Behavior

Description of the expected behavior or functionality. In case of feature request, please add input and expected output.

Steps to reproduce

Please add steps to reproduce the bug here along with the stack trace.

Links to any relevant pastes or documents

Please post links to any relevant documents that might contain any extra information.

Checklist

Use this checklist if you are using this issue as a tracker:

• Task 1
• Task 2
• Task 3 ...

Type of Issue

Like https://github.com/facebookresearch/droidlet/blob/main/craftassist/test.sh. Useful for quickly trying out tests locally

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Description

When inst_seg is removed, we want to force the agent the perceive that area at next step.

I think it's already (or partially?) supported since maybe_remove_inst_seg is called only when a block changes, and when a block changes, this function maybe_remove_block_from_memory will also be called inside which the surrounding area is added to areas_to_perceive.

Did I miss anything?

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask
• Test

Description

Test "where is the flower" (or any relevant object) in minecraft and see if the agent also points to it. I made a change where pointing is now a default action for a specific path in get_memory_handler - specifically when the attribute type is a location for the object being queried.

More details at #17

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

don't default to excluding self; otherwise improve handling

Type of Issue

Currently "point at the cube" doesn't work. The agent from the logs does : Sending chat: /point 3 65 4 3 65 4 but we don't see the execution in game.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Description

Detailed description of the requested feature/documentation ask/bug report.

Current Behavior

Description of current behavior or functionality.

Expected Behavior

Description of the expected behavior or functionality. In case of feature request, please add input and expected output.

Steps to reproduce

Please add steps to reproduce the bug here along with the stack trace.

Links to any relevant pastes or documents

Please post links to any relevant documents that might contain any extra information.

Checklist

Use this checklist if you are using this issue as a tracker:

• Task 1
• Task 2
• Task 3 ...

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Description

https://github.com/facebookresearch/droidlet/blob/main/craftassist/servermgr/deploy.sh
not python/servermgr any more

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Combination bugfixes and code cleanup of agent memory. In part to support sending diffs of agent memory database to dashboard, and in part just to clean up old junk, and fix some outstanding junk.

Subissues:

• Several memory tables are not derived from main memory table; in particular Triples
• lots of raw _db_writes in code. make a wrapper that logs deletes, forces update of updated_time, etc.
• clean up deletes more generally

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

It takes about 1.5 minutes between :

2021-01-14 00:58:27,802 [nsp_dialogue_manager.py:102 - __init__() INFO]: using model_dir=agent/models/semantic_parser/ttad_bert_updated/
initializing decoder with params Namespace(alpha=0.8, batch_size=48, data_dir='agent/datasets/annotated_data/', decoder_config_name='bert-base-uncased', decoder_learning_rate=1e-05, decoder_warmup_steps=1000, dtype_samples='[["annotated", 0.6], ["templated", 0.35], ["templated_filters", 0.05]]', encoder_dropout=0.0, encoder_learning_rate=0.0, encoder_warmup_steps=1, examples_per_epoch=-1, hard=True, lambda_span_loss=0.5, model_name='caip_parser', node_label_smoothing=0.0, num_decoder_layers=6, num_epochs=8, num_highway=2, optimizer='adam', optional_identifier='', output_dir='', param_update_freq=1, pretrained_encoder_name='distilbert-base-uncased', rephrase_proba=-1.0, span_label_smoothing=0.0, train_encoder=False, tree_to_text=False, tree_voc_file='craftassist/agent/models/semantic_parser/ttad_bert_updated/caip_test_model_tree.json', word_dropout=0.0)

and

2021-01-14 00:59:54,888 [craftassist_agent.py:151 - init_inventory() INFO]: Initialized agent inventory

Current Behavior

Description of current behavior or functionality.

Expected Behavior

Description of the expected behavior or functionality. In case of feature request, please add input and expected output.

Steps to reproduce

This is in flat world

cd craftassist
python agent/craftassist_agent.py

Type of Issue

The Fill command in the spec and JSON schema should allow a triple at the action level for has_block_type.
Example command:

fill the entire room with stone

{'dialogue_type': 'HUMAN_GIVE_COMMAND', 
 'action_sequence': [
    {
      'action_type':    'FILL', 
     'reference_object': {
          'filters': {
              'triples': [
                 {'pred_text': 'has_name', 'obj_text': [0, [3, 3]]}]},     
          'text_span': [0, [2, 3]]}, 
    'triples': [
            {'pred_text': 'has_block_type', 
             'obj_text': [0, [5, 5]]}]
    } ]
}

Flagging this here for fix so this doesn't cause regressions.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

https://github.com/facebookresearch/droidlet/blob/main/locobot/robot/launch_pyro.sh

A few issues

• $ip is read incorrectly
• maybe use a better alternative to xterm (xterm uses cut buffers and not the standard X11 clipboard selection by default)
• rviz doesn't show up on the remote computer when launch_pyro.sh is run through ssh

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

In https://github.com/facebookresearch/droidlet/tree/main/locobot#getting-started we already assume that Docker's network interface was setup to be bridging / sharing.

The docker0 bridge would put the host at 172.17.0.1 and the docker launched program at 172.17.0.2.
These need to be documented in that README

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

The docs around individual perception modules are not available on the documentation / website.

For example, all of the handlers here appear nowhere on the docs website: https://github.com/facebookresearch/droidlet/tree/main/locobot/agent/perception/handlers

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

When create an attention node, these table columns get written: TABLE_COLUMNS = ["uuid", "x", "y", "z", "type_name", "ref_type"]. type_name field is filled with attender's eid. It 's name field which should be filled, right?

Check out this code snippet: https://github.com/facebookresearch/droidlet/blob/main/base_agent/memory_nodes.py#L530

This issue is found during deeper inspection into shadow memory data.

Type of Issue

Change pan tilt key bindings for pan tilt from WASD to something else

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Data scripts require GNU systems, eg. readlink -f is not supported on mac. Update data scripts to support Mac users for local development. This is useful for dataset updates and annotation tools.

Current Behavior

All scripts under tools/data_scripts fail when run on Mac OS.

Expected Behavior

Seamless installation and deployment that is platform agnostic.

Checklist

• Extend fetch scripts for parser models and datasets
• Extend deploy scripts for parser models and datasets (may need to install GNU tar on mac)
• Extend hash comparison check

cc @kavyasrinet @snyxan @anuragprat1k @soumith

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Description

File "/home/locobotm/djan/locobot/agent/../../base_agent/core.py", line 23, in step
    self.perceive()
  File "/home/locobotm/djan/locobot/agent/../../base_agent/loco_mc_agent.py", line 206, in perceive
    v.perceive(force=force)
  File "/home/locobotm/djan/locobot/agent/perception/perception.py", line 176, in perceive
    rgb_depth = self.vision.input()
  File "/home/locobotm/djan/locobot/agent/perception/handlers/core.py", line 31, in __call__
    return self.handle(*input)
  File "/home/locobotm/djan/locobot/agent/perception/handlers/input.py", line 25, in handle
    rgb_depth = self.agent.mover.get_rgb_depth()
  File "/home/locobotm/djan/locobot/agent/locobot_mover.py", line 221, in get_rgb_depth
    pts_in_cam = np.multiply(self.uv_one_in_cam, depth)
ValueError: operands could not be broadcast together with shapes (3,230400) (307200,)

Steps to reproduce

1. copied the locobot/robot folder onto the locobot
2. executed the commands in ./launch_pyro.sh separately in 4 terminals (there is a bug in the launch file which I'll fix outside this issue) to launch the pyro4 env
3. ran locobot_agent.py

Model Stats

Validation accuracies:

• annotated: 60.3%
• templated: 96%
• templated filters: 95.9%

Dataset changes since last push: none

Config changes:
Finetuning encoder now (previously frozen), withlr=10^-5 and warmup=1000, same as decoder side.
Encoder dropout set to 0.01. No word dropout, no label smoothing. Other configs unchanged.

Sample outputs

Fixed from internal issue #1472

>>> get_beam_tree("what is the name of the thing closest to you")
{'dialogue_type': 'GET_MEMORY', 'filters': {'output': {'attribute': 'NAME'}, 'argval': {'ordinal': 'FIRST', 'polarity': 'MIN', 'quantity': {'attribute': {'linear_extent': {'source': {'reference_object': {'special_reference': 'SPEAKER'}}}}}}}}

Previous parse:
{'dialogue_type': 'GET_MEMORY', 'filters': {'triples': [{'pred_text': 'has_tag', 'obj_text': 'name of the thing'}], 'output': 'TAG', 'memory_type': 'REFERENCE_OBJECT', 'location': {'reference_object': {'special_reference': 'AGENT'}, 'relative_direction': 'NEAR'}}}
Fixed from internal issue #1472

>>> get_beam_tree("what colour is the carpet")
{'dialogue_type': 'GET_MEMORY', 'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': [0, [4, 4]]}], 'output': {'attribute': 'COLOUR'}}}
>>>

Previous parse:
{'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'filters': {'output': {'attribute': 'COLOUR'}, 'triples': [{'pred_text': 'has_name', 'obj_text': 'carpet'}]}}]}

Fixed from internal issue #1472

>>> get_beam_tree("make two circles behind me")
{'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'BUILD', 'location': {'reference_object': {'special_reference': 'SPEAKER'}, 'relative_direction': 'BACK'}, 'schematic': {'triples': [{'pred_text': 'has_name', 'obj_text': [0, [2, 2]]}], 'repeat': {'repeat_key': 'FOR', 'repeat_count': [0, [1, 1]]}}}]}

Previous parse:
{'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'BUILD', 'location': {'reference_object': {'special_reference': 'SPEAKER'}, 'relative_direction': 'BACK'}, 'repeat': {'repeat_key': 'FOR', 'repeat_count': 'two'}, 'schematic': {'triples': [{'pred_text': 'has_name', 'obj_text': 'circle'}], 'repeat': {'repeat_key': 'FOR', 'repeat_count': 'two'}}}]}

Fixed from internal issue #1472

>>> get_beam_tree("point to the cube")
{'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'DANCE', 'dance_type': {'point': {'location': {'reference_object': {'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': [0, [3, 3]]}]}}}}}}]}
>>>

Previous parse:
{'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'MOVE', 'location': {'reference_object': {'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'cube'}]}}}}]}

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)

Current Behavior

When I ran pip install -r requirements.txt, it errored out while installing spacy's dependency thinc with the error:

l/miniconda3/envs/droidlet/include/python3.6m -c include/_cuda_shim.cu -o build/temp.linux-x86_64-3.6/include/_cuda_shim.o -arch=sm_30 --ptxas-options=-v -c --compiler-options '-fPIC'
  nvcc fatal   : Value 'sm_30' is not defined for option 'gpu-architecture'

The reason is that this old version of spacy/thinc aren't yet supporting CUDA 11.

We should upgrade the requirement if possible.

Expected Behavior

SpaCy/ThinC get installed smoothly

Steps to reproduce

On a machine with CUDA 11 activated, do pip install -r requirements.txt from the droidlet root directory.

Type of Issue

Write a design doc for the dashboard and add it to the repo.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask
• Code Cleanup

Current

It is literally a stub

Expected

Just use the base class instead, and save some lines / indirections

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Right now running the test_model_script fails on the transformers version. Creating this issue to check this again and make sure we can run this with the version of transformers being used in the agent env

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

There are dependency conflicts in our requirements, and packages frozen on old versions.
This is affecting our env setup and will be a pain point for new users.

Current Behavior

Example of error from pip install:

(droidlet_test_env) [~/droidlet]$ pip install mypy-extensions==0.4.1             *[rebeccaqian_inference_script_fix]

Looking in indexes: https://pypi.org/simple, http://100.97.64.150
Collecting mypy-extensions==0.4.1
  Using cached mypy_extensions-0.4.1-py2.py3-none-any.whl (3.6 kB)
Installing collected packages: mypy-extensions
  Attempting uninstall: mypy-extensions
    Found existing installation: mypy-extensions 0.4.3
    Uninstalling mypy-extensions-0.4.3:
      Successfully uninstalled mypy-extensions-0.4.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mypy 0.800 requires mypy-extensions<0.5.0,>=0.4.3, but you have mypy-extensions 0.4.1 which is incompatible.
Successfully installed mypy-extensions-0.4.1

Expected Behavior

A new user should be able to create a new env and install requirements relatively smoothly. This is something that we will need to maintain regularly.

Steps to reproduce

conda create --name [NEW_ENV]
conda activate [NEW_ENV]
pip install -r requirements.txt

Type of Issue

A list of common commands here: https://docs.google.com/document/d/1lu87-HCqd9NFZo2dryBEv5DtFQyc8dabPxWwgAHC1t0/edit needs to be annotated with ground truth parses.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Right now both locobot agent and craftassist agent files implement

@sio.on("sendCommandToAgent")
def send_text_command_to_agent(sid, command):

We should move dashboard_chat as well as handler to loco_mc_agent instead

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask
• Refactor

Hello,
It seems that I am kind of here too early.
I am just wondering is it currently possible to run the craftassist locally right now?
The step where it says:

"make"

What should I do exactly here?
If it is coming up shortly in the next few commits I'll just be back again.
If I can run this successfully on my PC locally, I'll be making a video tutorial of how to install it for others too.

Thank you guys for the amazing work!

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)

Feature Description

This is a bit hard to solve, but detectron2 dependency in locobot/requirements.txt hardcodes to use CUDA 10.2 binaries.
Find a way to make this CUDA version agnostic (or upgrade to CUDA 11) and fix that link

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)

Feature Description

Add servermgr deployment to workflow whenever a commit is pushed to main.

Current Behavior

One has to deploy manually.

Expected Behavior

Deployment happens automatically.

Type of Issue

Write documentation for all the subinterpreters (FilterInterpreter, ReferenceObjectInterpreter, ReferenceLocationInterpreter, FacingInterpreter, ComputeLocations, AttributeInterpreter, ConditionInterpreter) explaining what they do. Document design choices, assumptions made, so new users of droidlet can understand them.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

This class should be documented and surfaced onto the droidlet docs website: https://github.com/facebookresearch/droidlet/blob/main/locobot/agent/locobot_mover.py#L28

Currently it's not there.

Type of Issue

Add a unit test that generates parser health (with ground truth querying) for common must have commands in : https://docs.google.com/document/d/1lu87-HCqd9NFZo2dryBEv5DtFQyc8dabPxWwgAHC1t0/edit

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Detailed description of the requested feature/documentation ask/bug report.

Current Behavior

Description of current behavior or functionality.

Expected Behavior

Description of the expected behavior or functionality. In case of feature request, please add input and expected output.

Steps to reproduce

Please add steps to reproduce the bug here along with the stack trace.

Links to any relevant pastes or documents

Please post links to any relevant documents that might contain any extra information.

Checklist

Use this checklist if you are using this issue as a tracker:

• Task 1
• Task 2
• Task 3 ...

Type of Issue

Add tests like https://github.com/facebookresearch/droidlet/blob/main/craftassist/test/test_get_memory.py and https://github.com/facebookresearch/droidlet/blob/main/craftassist/test/test_put_memory.py for the locobot agent

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask
• Better Engineering

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Looks like we have two wikis floating around internally - https://www.internalfb.com/intern/wiki/Droidlet/ and https://www.internalfb.com/intern/wiki/Locobot_Assistant/. Let's merge these into one.

Reported by @snyxan

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

We catch exceptions in a myriad of ways

• catching but raising another exception anyway with a generic message (for eg https://github.com/facebookresearch/droidlet/blob/main/base_agent/dialogue_objects/get_memory_handler.py#L165)
• silently consuming it (for eg https://github.com/facebookresearch/droidlet/blob/main/base_agent/base_util.py#L89)
• print the error (https://github.com/facebookresearch/droidlet/blob/main/base_agent/save_and_fetch_commands.py#L19)

Standardize how to do exception handling. Do we want to just print a message to the log? Do we want to send the error message and maybe even the stack trace back to the dashboard?

Scoping this to base_agent for a limited scope.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Currently Memory2D component in dashboard is not connected to craftassist backend. We'll need to connect that

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

This feature would add a new React component to the droidlet dashboard that can show the state of the agent's memory with a text based filter. Once this is in, this can evolve into a SQL browser that loads the schema from backend and lets developers query the current memory from the frontend component.

Please feel free to use any open sourced libraries as you see fit.

In the backend, this is where the state of the memory is being communicated through a socket event.

To add your component:

1. Add the React component js and css files here
2. Add the component to index.js
3. Manage and set state of component here
4. Add new dependencies to this

Current Behavior

This is the current state of the dashboard with a 2D grid view of memory :

Expected Behavior

A new pane next to the "Memory 2D" pane that helps visualize the state of memory at backend and eventually allows developers to write SQL queries and query them against the memory from this pane.

Checklist

Use this checklist if you are using this issue as a tracker:

• Task 1
• Task 2
• Task 3 ...

Type of Issue

Consolidate https://github.com/facebookresearch/droidlet/blob/main/locobot/test/all_test_commands.py and
https://github.com/facebookresearch/droidlet/blob/main/craftassist/test/all_test_commands.py

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask
• Better Engineering

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Asks

Details

1. Add docstrings to describe how interpret_point_target works. Specifically,

• why is a 6-tuple returned?
• what is the +1 added to loc[1]?

2. Add docstrings to compute_location_heuristic

• What does it do?
• rel_dir is None for pointing (since the action dictionary output by the nsp does not have a relative_direction field. How does this work for pointing?

Type of Issue

KeyError in BLOCK_DATA.
On a high level, looks like it has to do with format of bid_to_name

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Steps to reproduce

On craftassist:

what is that (point at a mushroom block)
point at the mushroom

Links to any relevant pastes or documents

2021-02-04 00:22:08,498 [nsp_dialogue_manager.py:249 - get_logical_form() INFO]: logical form after grammar fix "{'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'DANCE', 'dance_type': {'point': {'location': {'reference_object': {'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'mushroom'}]}}}}}}]}"
2021-02-04 00:22:08,498 [nsp_dialogue_manager.py:169 - handle_logical_form() INFO]: logical form post-coref "ab1d6b83eec9f74f9af677a72e7437ab0f73294fabb004750e094025ddd69f5e:f7b4bf1ab5dc40329165553c6c70532b" -> {'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'DANCE', 'dance_type': {'point': {'location': {'reference_object': {'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'mushroom'}]}}}}}}]}
2021-02-04 00:22:08,690 [sql_memory.py:198 - update_recent_entities() INFO]: update_recent_entities [<base_agent.memory_nodes.LocationNode object at 0x7f6dbfb2a150>]
2021-02-04 00:22:08,706 [sql_memory.py:198 - update_recent_entities() INFO]: update_recent_entities [<InstSeg Node @ [(8, 63, 3)] with tags ['red mushroom', 'red', 'mushroom'] >]
2021-02-04 00:22:08,760 [loco_mc_agent.py:195 - task_step() INFO]: Starting task <class 'tasks.Point'>
2021-02-04 00:22:08,760 [craftassist_agent.py:281 - send_chat() INFO]: Sending chat: /point 8 64 3 8 64 3
2021-02-04 00:22:09,122 [mc_memory_nodes.py:180 - create() INFO]: Added block object 77e063e5fc4a410b8fa4ce1b79394739 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:09,319 [mc_memory_nodes.py:180 - create() INFO]: Added block object 76b438f651f949e192c87bf21bb47eae with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:09,695 [mc_memory_nodes.py:180 - create() INFO]: Added block object 455c90f7eaa746bd87a0f03420b79c12 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:10,055 [mc_memory_nodes.py:180 - create() INFO]: Added block object 748ee0b648ad4162b2fafb880507c0ab with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:10,416 [mc_memory_nodes.py:180 - create() INFO]: Added block object 9a0a200f059a4977ad2dd4ce330478a0 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:10,763 [mc_memory_nodes.py:180 - create() INFO]: Added block object 7ab902d87df74cc1a756d10f73787998 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:11,252 [mc_memory_nodes.py:180 - create() INFO]: Added block object 70437697cd954d739eb06583158cbbd3 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:11,627 [mc_memory_nodes.py:180 - create() INFO]: Added block object 258366bbaca342929bcc318dded0c21e with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:12,071 [mc_memory_nodes.py:180 - create() INFO]: Added block object 2da17b4a757547d4bde7dd18df18de68 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:12,525 [mc_memory_nodes.py:180 - create() INFO]: Added block object 7addb87831874a01b2e68831079f7b86 with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:12,626 [mc_memory_nodes.py:180 - create() INFO]: Added block object 87e53dc2f9de41e99d9afb2d0882d8ec with 1 blocks, Counter({(95, 4): 1})
2021-02-04 00:22:12,630 [loco_mc_agent.py:153 - handle_exception() ERROR]: Default handler caught exception, db_log_idx=268784
Traceback (most recent call last):
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/core.py", line 18, in start
    self.step()
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/loco_mc_agent.py", line 177, in step
    super().step()
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/core.py", line 23, in step
    self.perceive()
  File "agent/craftassist_agent.py", line 211, in perceive
    v.perceive(force=force)
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/heuristic_perception.py", line 557, in perceive
    type_name = BLOCK_DATA["bid_to_name"][idm]
KeyError: ((7, 64, 0), (5, 0))
Traceback (most recent call last):
  File "agent/craftassist_agent.py", line 371, in <module>
    sa.start()
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/core.py", line 20, in start
    self.handle_exception(e)
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/loco_mc_agent.py", line 172, in handle_exception
    raise e
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/core.py", line 18, in start
    self.step()
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/loco_mc_agent.py", line 177, in step
    super().step()
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/../../base_agent/core.py", line 23, in step
    self.perceive()
  File "agent/craftassist_agent.py", line 211, in perceive
    v.perceive(force=force)
  File "/private/home/kavyasrinet/droidlet/craftassist/agent/heuristic_perception.py", line 557, in perceive
    type_name = BLOCK_DATA["bid_to_name"][idm]
KeyError: ((7, 64, 0), (5, 0))

Checklist

Use this checklist if you are using this issue as a tracker:

• Task 1
• Task 2
• Task 3 ...

Type of Issue

Data format update requested

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Currently the high_pri_commands.txt file under ground_truth has json.dumps for the dictionaries. I suggest we update the data format for all files under ground truth to have json format instead of string so we can get rid of ast.literal_eval altogether and use json.loads.

Type of Issue

Noticing that the 1 teleop command on the locobot agent can get executed 3-6 times. Is this some quirk of the dashboard key handler? I tried a quick hack of setting the commands array to empty after they are emitted but that didn't fix this.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Expected Behavior

Each teleop command only gets executed once.

Steps to reproduce

Play with the teleop commands on the locobot agent. I encountered this while playing with physical locobot.

Type of Issue

• We had codecov integrated in the internal repo but this got dropped at some point. Add the badge back to the main read me.
• Also it seems like codecov reports are not being uploaded at https://github.com/facebookresearch/droidlet/blob/main/.circleci/config.yml#L141. Fix it.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

List of commands:

• go back 1 step
• go forward 1 step
• go left 1 step
• go right 1 step
• ....

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Right now memory2D fetches from perception component of Locobot here. We should refactor that to fetch directly from memory state here.

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Refactor request
• Documentation Ask

Type of Issue

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Description

Currently running :

make

inside the craftassist repo fails to make client because of a mismatch in gcc version. We need to get this fixed soon for everyone to migrate to external.

Current Behavior

Description of current behavior or functionality.

Expected Behavior

make should successfully make cuberite and agent

Steps to reproduce

cd craftassist
make

Type of Issue

Sequence of move commands are broken

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask

Type of Issue

Write tests for data scripts to avoid regressions with model / dataset downloads

Select the type of issue:

• Bug report (to report a bug)
• Feature request (to request an additional feature)
• Tracker (I am just using this as a tracker)
• Documentation Ask