When this extension is run by a single user without collaboration model enables in JupyterLab, it is a bit confusing to see the anonymous planet name for yourself. I was thinking of how we can improve this for the user and attempt to use their real name or username. It looks like os.getlogin() can return this information. This would probably be an upstream contribution to JupyterLab itself though, but I think it would really help the user experience for single users.

When this is run in collaboration mode, I think it would be helpful to show the user which user they are in the chat, so instead of showing something like "Anonymous Callisto" show "Anonymous Callisto (me)" for just your username. I think the awareness API of JupyterLab can be to see if the user in the current user.

Problem

The %%ai magic command is in the top-level jupyter_ai package.

Proposed Solution

Move the magic command logic to jupyter_ai_magic, a new package.

Right now we are hard coding the models we are using for the Chat UI. We need to make those configurable.

Description

When using a Python kernel, I cannot use ChatGPT via the %%ai magic to analyze TypeScript code. The curly braces {} are being interpreted by the magic command, even if they are in a Python """ blockquote block.

Reproduce

1. Create a new notebook.
2. Run a cell containing %load_ext jupyter_ai.
3. Run a cell with TypeScript in it. (See example below)
4. See error (See example below.

Example input:

%%ai chatgpt
Analyze the following code, tell me what it does, and tell me something I can do to improve it.

"""
private _addToolbar(model: ICellModel): void {
  const cell = this._getCell(model);

  if (cell) {
    const toolbarWidget = new Toolbar();
    toolbarWidget.addClass(CELL_MENU_CLASS);

    const promises: Promise<void>[] = [];
    for (const { name, widget } of this._toolbar) {
      toolbarWidget.addItem(name, widget);
      if (
        widget instanceof ReactWidget &&
        (widget as ReactWidget).renderPromise !== undefined
      ) {
        (widget as ReactWidget).update();
        promises.push((widget as ReactWidget).renderPromise!);
      }
    }

    // Wait for all the buttons to be rendered before attaching the toolbar.
    Promise.all(promises)
      .then(() => {
        toolbarWidget.addClass(CELL_TOOLBAR_CLASS);
        (cell.layout as PanelLayout).insertWidget(0, toolbarWidget);

        // For rendered markdown, watch for resize events.
        cell.displayChanged.connect(this._resizeEventCallback, this);

        // Watch for changes in the cell's contents.
        cell.model.contentChanged.connect(this._changedEventCallback, this);

        // Hide the cell toolbar if it overlaps with cell contents
        this._updateCellForToolbarOverlap(cell);
      })
      .catch(e => {
        console.error('Error rendering buttons of the cell toolbar: ', e);
      });
  }
}
"""

Error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 get_ipython().run_cell_magic('ai', 'chatgpt', 'Analyze the following code, tell me what it does, and tell me something I can do to improve it.\n\n"""\nprivate _addToolbar(model: ICellModel): void {\n  const cell = this._getCell(model);\n\n  if (cell) {\n    const toolbarWidget = new Toolbar();\n    toolbarWidget.addClass(CELL_MENU_CLASS);\n\n    const promises: Promise<void>[] = [];\n    for (const { name, widget } of this._toolbar) {\n      toolbarWidget.addItem(name, widget);\n      if (\n        widget instanceof ReactWidget &&\n        (widget as ReactWidget).renderPromise !== undefined\n      ) {\n        (widget as ReactWidget).update();\n        promises.push((widget as ReactWidget).renderPromise!);\n      }\n    }\n\n    // Wait for all the buttons to be rendered before attaching the toolbar.\n    Promise.all(promises)\n      .then(() => {\n        toolbarWidget.addClass(CELL_TOOLBAR_CLASS);\n        (cell.layout as PanelLayout).insertWidget(0, toolbarWidget);\n\n        // For rendered markdown, watch for resize events.\n        cell.displayChanged.connect(this._resizeEventCallback, this);\n\n        // Watch for changes in the cell\'s contents.\n        cell.model.contentChanged.connect(this._changedEventCallback, this);\n\n        // Hide the cell toolbar if it overlaps with cell contents\n        this._updateCellForToolbarOverlap(cell);\n      })\n      .catch(e => {\n        console.error(\'Error rendering buttons of the cell toolbar: \', e);\n      });\n  }\n}\n"""\n')

File /local/home/brgrange/micromamba/envs/docchat/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2430, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2428 with self.builtin_trap:
   2429     args = (magic_arg_s, cell)
-> 2430     result = fn(*args, **kwargs)
   2432 # The code below prevents the output from being displayed
   2433 # when using magics with decodator @output_can_be_silenced
   2434 # when the last Python token in the expression is a ';'.
   2435 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File /local/home/brgrange/micromamba/envs/docchat/lib/python3.11/site-packages/jupyter_ai/magics.py:201, in AiMagics.ai(self, line, cell)
    199 # interpolate user namespace into prompt
    200 ip = get_ipython()
--> 201 prompt = prompt.format_map(FormatDict(ip.user_ns))
    203 # configure and instantiate provider
    204 ProviderClass = get_provider(provider_id)

ValueError: unexpected '{' in field name

Expected behavior

TypeScript code was analyzed by the AI.

Problem

I would like ENTER to send a message in the Jupyter AI chat UI, but it adds a new line.

Proposed Solution

Provide an option to switch the shortcut behavior.

When the option is disabled, ENTER adds a new line, and SHIFT+ENTER sends the message.

When the option is enabled, ENTER sends the message, and SHIFT+ENTER adds a new line.

Additional context

If #85 is fixed, the hints should reflect the current preference.

macOS UI conventions treat ENTER (typically on the numeric keypad) and RETURN (typically in the same row as the middle row of letters) as two separate keys. On Windows and Linux, even though the keys on the numpad and in the letter-key row send different scan codes, UIs tend to treat them identically.

Slack's "Advanced" preferences pane has options to change this behavior, and to handle code blocks bounded by ``` specially:

Problem

There is no check currently in the extension to make sure that the chat provider keys are present. This causes the lab instance to hang.

Description

When using --format code (#43), some generative AIs like ChatGPT may surround the generated code with triple-backticks. This code cannot be run as generated.

The Chat UI has significant improvements that can be made:

• Shift + Enter should send the current message (fixed by #70)
• The text input should have placeholder text like "Ask Jupyter AI anything..."
• The "Send" button should be filled (i.e. blue with a white icon, instead of transparent with a blue icon)
• The "Send" button should have helper text that appears on hover saying "Send (Shift + Enter)"

Problem

Users cannot run image models using Stable Diffusion using Jupyter AI, because no existing output accommodates images.

Proposed Solution

Add a --format image option for the %%ai magic command.

If the AI model outputs a URL, -f image should cause the output cell to display the image by remotely loading that URL as an inline image, either using markdown ![alt text](image URL) syntax or HTML <img alt="alt text" src="image URL" /> syntax.

If the AI model outputs binary data, and -f image is specified, the output cell should display the binary data returned. If the HTTP call to the model's API specifies a MIME type in its response headers, that MIME type should be used to decode the binary data.

Update the example notebooks and documentation with information about -f image.

Additional context

The option should apply a prompt template (#42) that should tell the AI model to output an image only, without explanatory text.

• If user is already at the bottom of the overflow container, make sure the viewport stays anchored to the bottom upon receipt of a new message
• If the user is in the middle of a overflow container, show a badge "New message" at the bottom of the viewport that informs users that a new message has arrived
  • This badge should, on click, scroll to the bottom of the overflow container
• Do this in a way that supports most browsers

It's fairly difficult to implement intelligent scrolling in a correct and browser-independent manner. The first point is achievable via some overflow-anchor hacks, but these don't work on Safari.

Problem

If a user just ran a cell and received an error that's captured in the Err array (#41), they might want to explain it quickly.

Proposed Solution

Add a magic command such as %%explainerror that takes an AI model as a parameter, and that sends a prompt to an AI model to explain the most recent error encountered.

Description

The special variable Err, introduced in a fix for #41, is not yet present in the docs.

Expected behavior

Add an example of how Err[n] is used to the docs.

Description

When using Python 3.10 on a Mac laptop on Apple Silicon (M1 Pro), I get the following error after regenerating my hatch env and trying to start JupyterLab:

2023-04-17 10:57:22,884	ERROR services.py:1195 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-04-17_10-57-21_738793_69190/logs/dashboard.log'
2023-04-17 10:57:22,884	ERROR services.py:1196 -- Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-04-17_10-57-21_738793_69190/logs/dashboard.log'
Traceback (most recent call last):
  File "/Users/jweill/Library/Application Support/hatch/env/virtual/jupyter-ai-monorepo/YuaVGges/jupyter-ai-monorepo/lib/python3.10/site-packages/ray/_private/services.py", line 1167, in start_api_server
    with open(dashboard_log, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2023-04-17_10-57-21_738793_69190/logs/dashboard.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jweill/Library/Application Support/hatch/env/virtual/jupyter-ai-monorepo/YuaVGges/jupyter-ai-monorepo/lib/python3.10/site-packages/ray/_private/services.py", line 1178, in start_api_server
    raise Exception(err_msg + f"\nFailed to read dashboard log: {e}")
Exception: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-04-17_10-57-21_738793_69190/logs/dashboard.log'
2023-04-17 10:57:23,226	INFO worker.py:1538 -- Started a local Ray instance.
[2023-04-17 10:57:23,411 E 69190 732428] core_worker.cc:179: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

Description

While installing jupyter-ai monorepo in a new conda environment jlpm setup:dev is not working with error message below

yarn run v1.21.1
$ lerna run setup:dev --stream --concurrency=1
/bin/sh: lerna: command not found
error Command failed with exit code 127.

Reproduce

1. Create new conda environment
2. Follow installation instructions from README.md until jlpm setup:dev fails

Expected behavior

jlpm setup:dev is working when following installation instructions README.md

Context

Operating System and version: MacOS 12.6.3
Browser and version: Chrome version 109.0.5414.87 (Official Build) (arm64)
JupyterLab version: head-of-master (WIP 4.0.0)

Description

Reproduce

I setup an SageMaker endpoint within AWS classic notebook instance. And I am able to query it through regular way.

When I call %%ai sagemaker-endpoint:sm_llm Write some JavaScript code that prints "hello world" to the console.

It says:
ValidationError: 2 validation errors for SmEndpointProvider content_handler field required (type=value_error.missing) __root__ Could not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. (type=value_error)

Here is my definition of sm_llm:
`
parameters = {
"max_length": 200,
"num_return_sequences": 1,
"top_k": 250,
"top_p": 0.95,
"do_sample": False,
"temperature": 1,
}

class ContentHandler(ContentHandlerBase):
content_type = "application/json"
accepts = "application/json"

def transform_input(self, prompt: str, model_kwargs = {}) -> bytes:
    input_str = json.dumps({"text_inputs": prompt, **model_kwargs})
    return input_str.encode('utf-8')

def transform_output(self, output: bytes) -> str:
    response_json = json.loads(output.read().decode("utf-8"))
    return response_json["generated_texts"][0]

content_handler = ContentHandler()

sm_llm =SagemakerEndpoint(
endpoint_name=MODEL_CONFIG["huggingface-text2text-flan-t5-xxl"]["endpoint_name"],
region_name="us-west-2",
model_kwargs=parameters,
content_handler=content_handler
)
`

Expected behavior

Context

• Operating System and version:
• Browser and version:
• JupyterLab version:

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Paste the output from your browser Javascript console here, if applicable.

I'm not sure if this a a feature request or question about whether you might expect, or are hoping to see, support for other models being submitted as PRs to this repo or as plugin modules that can be added from third party installers.

Recent days have seen powerful hints that the days of running LLMs on the desktop have arrived sooner than we might have expected ( https://simonwillison.net/2023/Mar/11/llama/ ). Image generaators are also available for local use, either bundled as an an application or as a local web service. For example, automatic111/stable-diffusion-webui provides quite a complex UI which suggests their might be quite a rich API available that jupyter-ai could hook into.

So as a feature request, I guess my issue is: it would be useful to have modules for stable diffusion running locally and for one of the new "portable" LLMs running locally.

(I also wonder about jupyterlite in-browser exetnensions, eg wrapping something like https://xenova.github.io/transformers.js/ ?)

As a question about third party plugins, where eg there is a module shipped to support the stable--diffusion-web-ui shipped from the automatic111 org, how would that work?

Possibly related to this, I note that jupyterlite, which originally baked in pyolite kernel, now seems to be re-engineering so that all kernels are in their own repo, and jupyterlite is just an "empty" container. Would that be a more scaleable approach for this project?

Problem

Currently, some AI models have aliases. For example, gpt2 is an alias for huggingface_hub:gpt2. These are defined in a constant in our Python code:

To add a new alias, one would have to modify the file above and make a new release of Jupyter AI for everyone. There are no aliases for local usage.

Proposed Solution

Add a command %%ai alias <name> <value> to create custom aliases. These aliases persist for as long as the kernel remains running.

Add a command such as %%ai alias list that shows users which aliases they have defined. %%ai list (#28) may also display these aliases.

Update the example notebooks and documentation with information about this new command.

Problem

Users reading the docs don't know about the /generate (formerly /autonotebook; #90), /ask or /learn (#94) commands.

Proposed Solution

Update the user docs to explain how these commands work, with examples.

Because the underlying format is actually text, let's rename the magic format from raw to text.

Problem

We subscribe to Azure Open AI Services, which have the same models available.

Proposed Solution

Add an integration for Azure Open AI Services either by further configuring the Open AI, or as a 1st class integration.

Additional context

Problem

Running jlpm setup:dev clobbers the existing playground/config.py file, deleting any configuration a developer may have specified.

Proposed Solution

Don't do that.

Problem

There is no command I can run to get a list of models I can use with the %%ai magic command.

Proposed Solution

Add a command such as %%ai list that lists all supported models.

Backend APIs that support

1. Get config that returns the current chat configuration with selected chat model, embedding model and any api keys.
2. Update config that let's the UI update any part of the configuration.
3. Delete config that let's the UI clear the configuration, so user can start over.

Problem

Inside the monorepo, magic command logic is intermingled with other code in the jupyter_ai package.

Proposed Solution

Move the magic command logic to a new package, jupyter_ai_magic.

I note the release of a new ChatGPT API from OpenAI — https://openai.com/blog/introducing-chatgpt-and-whisper-apis — and a simple Python wrapper for it from @simonw: https://til.simonwillison.net/gpt3/chatgpt-api

Presumably, this is something that could be accommodated as a plugin or extension of some sort to jupyter-ai?

Problem

When users customize the output format using the -f/--format option to the magic command, the prompt may still cause the AI to output in a format that the magic command can't display. For example, if the user asks for HTML markup, the AI model may generate explanatory text that is not in HTML format.

Proposed Solution

For each supported output format, write a prompt template. Apply the prompt template to the specified prompt before sending it to the AI model.

In the examples below, the prompt template is indicated in bold.

%%ai anthropic:claude-v1.2 -f html
Create a square using SVG with a black border and white fill.

… would send:

Create a square using SVG with a black border and white fill. The output should be in HTML format only, with no text before or after.

%%ai chatgpt -f math
Generate the 2D heat equation

… would send:

Generate the 2D heat equation

The output should be in LaTeX surrounded by $$. Do not include an explanation.

Backend API that returns list of providers and models they support, so that the UI can show the user options to select a chat provider.

Problem

The docs do not mention the chat UI at all.

Proposed Solution

Add instructions, with screen shots, about how to use the chat UI to the user docs.

Right now, the /generate command in the Chat UI only generates notebooks. We should use agents and tools to allow this to create other file types, like Python scripts. Could also extend it to other file types as well (JSON, CSV, HTML, image, etc.) but let's start with Python scripts.

It's time to get documentation set up on ReadTheDocs. The boilerplate can be generated with sphinx-quickstart docs, as described in the documentation. However, there will need to be a few additional changes to this:

• Use myst-parser to accept Markdown source
• Use pydata-sphinx-theme

Please review this for inspiration: jupyter-server/jupyter-scheduler#311

Description

• 0.1.1 : https://github.com/jupyterlab/jupyter-ai/actions/runs/4337886839
• 0.1.0 : https://github.com/jupyterlab/jupyter-ai/actions/runs/4332077868

It looks like for some reason jupyter releaser bump-version is not bumping the patch number, but only on release commits?

Problem

When the notebook author wants a generative AI model to generate code, the output is in markdown or raw (text) format, so the notebook author cannot run the generated code as they can run other code in their notebook.

Proposed Solution

Add a --format code format option. When the user runs a %%ai magic command with this option, Jupyter AI inserts a new code cell using ipylab that contains the source code produced.

Update the example notebooks and documentation to include information about --format code.

Additional context

The --format code option should apply a prompt template to ensure that only source code, with no plain-text explanations, get output. See #42.

Perhaps it is because this is a brand-new project, but there is not enough information for learn what this is supposed to do, and not enough for me to get started using, testing, or contributing to this.

I see several subrepositories, but those have no descriptions either, only links to themselves and the top-level repo. It looks like just a project template with some project stub repos. It is not (yet) for Python users, because almost all of the code is written in Typescript. (https://github.com/jupyterlab/jupyter-ai/tree/main/packages)

Can someone create a statement of goals, status, and initial user documentation?

That will help people learn how to use it and possibly contribute to it.

Thank you.

Rich Lysakowski

Problem

Jupyter notebook authors would like to pass an error generated by their code to a large language model (LLM) so that they can understand the error better.

1. User runs code
2. Code throws an error
3. User wants to ask the LLM what the error means, without copying and pasting the error into a new prompt

Proposed Solution

Create a special Err list variables, with the same indexes as the In and Out variables provided by IPython. When a user runs code in cell In[n] that raises an uncaught exception, information about that exception is captured, as a string, in Err[n], and still displayed in the output area.

The README is often the first thing users see when discovering a new project. It's imperative that we make it as friendly as possible. Here are steps that can be taken to do so:

• Remove "check release" badge from top of the README
  • This is redundant since GitHub shows CI badges at the top of a repo anyways
• Rewrite the README to have the following format
  • Catchy one-liner summarizing Jupyter AI's intent and what solution it offers
  • Screenshot that covers all of the interfaces Jupyter AI offers
  • One paragraph explanation of Jupyter AI that elaborates on the preceding one-liner
• Remove contributor docs from README and direct contributors to online docs

Right now the code blocks in the chat UI are styled using react markdown. This uses a quite different visual style than it typical in markdown code blocks in Jupyter (different background, syntax highlighting, typography, etc.). We should figure out how to use the standard styling of Jupyter markdown code blocks. We may need to build our own extension to react markdown for this.

We should remove the old code in the repo (AI modules, etc.).

As we add more commands (/clear, /autonotebook, /filesytem, etc.) users will need better integrated help. I propose that we create a /help command that provides that to users in a chat reply.

Problem

The current build does not catch any python syntax errors, missing imports etc. during the build. Additionally, there is no python lint support in the current setup. Due to both of these missing features, contributors will have to manually observe and fix syntax errors in python.

Solution

• Add linting support to build
• Add static analyzers for example mypy to aid discovery of syntax errors, imports etc.

Problem

Relative timestamps take up substantial space and, especially when a message was sent less than a minute ago, can be distracting with their frequent updates.

Proposed Solution

Use HH:mm format, in either 12-hour or 24-hour time, for timestamps in chat messages.

Additional context

We may be able to use the browser's locale for time formatting.

Description

With a fresh Conda environment and after removing my local Hatch environment, when I attempt to run jlpm dev-install using tip-of-main, I get the following error:

@jupyter-ai/core: ERROR: Could not find a version that satisfies the requirement ray==2.2.0 (from jupyter-ai) (from versions: none)
@jupyter-ai/core: ERROR: No matching distribution found for ray==2.2.0

Attempting to run pip install ray==2.2.0 from my conda env (outside of Hatch) also fails:

(jupyterlab-0416) (hostname):jupyter-ai jweill$ pip install ray==2.2.0
ERROR: Could not find a version that satisfies the requirement ray==2.2.0 (from versions: none)
ERROR: No matching distribution found for ray==2.2.0

Context

• Operating System and version: macOS 12.6.4 "Monterey" for Apple Silicon (M1 Pro)

For quick usage, the chat UI panel as a side panel makes a lot of sense. However, for some usage cases, that starts to be a ton of content for a side panel. I believe we should allow the user to open the chat UI as a full main area panel. We could do this by allowing them to move it between the side panels and main area ("or") or by allowing them to have it in both places if they want ("and"). The second is probably easier given that could just show a card on the launcher.

Problem

We need an example notebook to show how the magic %%ai command works.

Proposed Solution

Add an example notebook demonstrating the magic command.

Backend API that returns list of embedding model providers, so the UI can show these options to the user.

Problem

Upon initial load, the chat panel is completely blank, with no placeholder text to indicate what it is. (See jupyterlab/jupyterlab#13988 for a related issue in JupyterLab, since fixed.)

Proposed Solution

Provide either a placeholder that would appear in an otherwise totally blank chat panel, or have the chatbot provide an initial message to suggest that you can send it a message.

Additional context

See also #55, which suggests a placeholder for the chat area's message box.

Problem

Users must write custom Python code in this repository to add a new LangChain chain or AI model.

Proposed Solution

Add a %ai register command that registers a LangChain chain with a new name.

Registered LangChain chains should show up when the user runs %ai list.

Modify the example notebooks and documentation to refer to this new command.

Problem

Especially to users more familiar with other chat programs, it is not safe to assume that a user will know to press SHIFT+ENTER to send a message, as ENTER adds a new line.

Proposed Solution

When the user has typed anything into the chat UI text box, provide a hint below the text box stating that SHIFT+ENTER will send the message, and/or that ENTER adds a new line.

Additional context

When I have typed information into Slack's message box, using default settings on macOS, I see "Shift + Return to add a new line" below the text box: