Problem

There are several ARC tasks where pixels have to be counted.

Solution

Show a grid between the pixels.

Show width x height under each image.

Hide/show button for the grid.
Where should this button be placed in the UI?
Maybe a toggle grid button in the footer of the editor overview page.

Problem

Currently you have to refer to a task using the exact same text as the filename. It doesn't allow for differences in lowercase/uppercase. If the name isn't exact, then no tasks are found.

Currently only thumbnails of the tasks are shown. There are no caption under the thumbnail.

If you know its name, it's not possible to locate a task in the task gallery.

Solution

A search page, or search box.

Allow for fuzzy matching, typos, uppercase/lowercase.

Problem

If you disable all the training tasks, then none of the tags are task tagging shows any tasks.
https://neoneye.github.io/arc/?filter=-training

The task tagging is only for the training tasks.

Solution

Manually tagging of the 400 evaluation tasks.

Problem

On desktop with big display, the undo button is shown in the bottom bar.

However on the iPad there is no undo button in the bottom bar. Despite being plenty of room for it.

Currently the user have to open the tool: paint menu, to get to the undo button.

When I see new user use the editor, they don't realize that the there is an undo button.

Solution

Show the undo button next to the palette.

Maybe eliminate the redo button.

Problem

Currently there is no undo, so it's costly making mistakes.

I'm curious to how does humans solve ARC task, having the entire history of actions may provide insight into how humans reason about ARC tasks.

Solution

Record the history of actions. And all the parameters for each action.

• A move left with a selection rectangle, take the selection rectangle as parameters.
• A set pixel takes x, y, color.

Investigate how to store the history.

Problem

Currently the ARC_TAGS contains hardcoded tags, only for the ARC dataset.

Solution

Make it easier to show status for your own solver.

Problem

As a single user, the number of files I'm generating are huge.

I had imagined that I could send the history files by email and add them to a database. However it's probably going to be more files than what I can manually process.

Solution

Backend.

Questions:

• What language? rust or python.
• What database? sqlite.
• User management?

Problem

The index.html page shows Initializing... for around 60 seconds. And then it shows the thumbnails.

For the ARC dataset there are 800 tasks. At the moment all 800 thumbnails are being rendered. On a slow device this takes forever.

Rendering thumbnails for every task causes a long startup time.

Solution

Render thumbnails in a less resource hungry way.

Investigate ways to render thumbnails.

Problem

The Editor -> Overview page already make use of window.devicePixelRatio.

However the Editor still use the old code, and thus display a grid with 2px wide grid lines.

This is in particular visible on my Android phone, and in my desktop browsers. Surprisingly it seems to be narrow grid lines on my iPhone/iPad.

Solution

Only show 1px wide grid lines. Make use of the window.devicePixelRatio inside the updateDrawCanvas() function.

Problem

Trying on various iOS devices (iPhone, iPad). I often enter a zoom mode, that I didn't intended.

When my friends try solving a puzzle they often end up in a zoomed in mode, where they are stuck.

Solution

Suppress the double tap behavior.

Problem

Detailed tasks can be near impossible to see with the grid visible

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=3eda0437

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=5c2c9af4

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=50846271

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=f9d67f8b

Solution

Experiment and find something that works.

Problem

The ARC palette is not friendly for colorblind people.

Solution

Make a settings page where the color theme can be changed.

Store the settings in local storage.

Problem

There is now too much noise in the task gallery.

Desktop

Mobile page 1

Mobile page 2

Solution

Ideas for simplifying are welcome.

Can the Settings + filter tags + current tool + current dataset, be combined into a side panel or overlay panel?

Problem

Currently the index.html shows all the tasks in the dataset. It's impossible to find anything.

On loda-lang the tristate filter is pretty useful for narrowing the list of items. I miss this kind of filtering.

Solution

The 3 states are: Plus, Neutral, Minus:

• Plus - only show tasks that has this label.
• Neutral - don't care about this label.
• Minus - don't show any task with this label.

Ideas for labels:

• same input output size
• solved by icecuber
• solved by gpt4
• correct prediction
• incorrect prediction
• easy
• medium
• hard
• ambiguous test output

In the top of index.html show all the available labels.

Extend all the bundle json gz files with labels.

Or provide a csv file with task-ids;status, so it’s possible to see what tasks parapraxis and icecuber are solving, not solving.

Problem

The 1D-ARC are shown as 1 unit wide tasks. Making the details barely visible.

https://neoneye.github.io/arc/?dataset=1D-ARC

Solution

Using 2 unit wide thumbnails may help.

Determine the aspect ratio of the task by counting pixels.

width = Sum over pairs(max(input.width, output.width))
height = max of pairs(input.height + output.height)
aspect = width / height

Inspect what aspect ratios the different kind of tasks yields.

Pick a good aspect ratio threshold.

Problem

Feedback from one of my friends:
quickly jump between input and output, for the current drawing.
While pressing a “Input” button, the input gets shown, and releasing it, and you get back to the drawing.

Currently the overview page, shows a miniature input image. And shows the editor taking up the full screen estate.

Trying to compare what you are editing with a input thumbnail, may be counter intuitive for new users.

Seeing the input image with the same size as the output image. This may straightforward when the input size and output size is the same. However if the input image is different than the output size, then the pixel size of the input image will have to be smaller than the pixel size of the output image, and zoom will have to be considered. I don't want to deal with the zoom problem.

I don't want more buttons in the UI.

Solution

I'm thinking about what to do with this issue.

Maybe I need a proof of concept.

Problem

On a mobile device, there is plenty of space between the images, but this images are too compressed.
https://neoneye.github.io/arc/edit.html?dataset=ARC&task=5d2a5c43

Solution

Maximize the cellsize and minimize the gap between the images.

Problem

Sometimes I want to see the solution.

Solution

In the Overview bottom bar, insert a Reveal button, that while pressed, shows the solution.

Problem

I have seen this in Firefox 121.0.1. I think I have experienced it in other desktop browsers.

The user clicks the Overview button with the mouse.

While editing the user makes a selection, copies the selection, and paste the selection.

Use Enter key to confirm the paste.

For some reason the Overview button has keyboard focus. The paste is confirmed, however the UI jumps from the editor to the overview page. It's supposed to stay on the edit page.

Solution

Investigate how keyboard focus works. How to suppress this behavior.

Problem

Open this url in a desktop browser: Chrome, Edge, Firefox, Safari.
https://neoneye.github.io/arc/edit.html?dataset=arc-dataset-diva&task=landport_cw_86_42_75_11

When grid is disabled, the <tr> with id task-overview-table-row0 is shown correct. It shows: 1 2 3 1.

When grid is enabled, the <tr> with id task-overview-table-row0 is not shown. It is supposed to show: 1 2 3 1. I guess it's overflowing.

It seems like calcCellSize() isn't computing a proper cell size that can fit within the visible viewport.

Solution

Debugging, tweaking.

Problem

I have a web server on localhost where I can inspect what is going on in my ARC solver.

Tapping on a thumbnail and jumping directly to my web server, that will save lots of copy pasting of urls.

Solution

A settings page where you can specify the destination urls to jump to, and the formatting of the url.

http://localhost:1234/inspect?dataset=ARC&task=73251a56

Problem

Feedback from my friends
the tools panel is too overwhelming

I tried to probe what seemed to be the problem. It's just noise with the many buttons: move left/right/up/down, flip, rotate, undo, redo.

I suggested a simple and an advanced mode, that the tools panel can be opened in. And my friend seemed like it could work.

The simple only have these tools: Pencil, Resize, Start over.

Solution

Proof of concept.

Problem

VSCode has Option+ArrowUp or Option+ArrowDown.

It duplicates the row and places the cursor either up or down depending on if ArrowUp or ArrowDown was used.

It's a keyboard shortcut I use several times a day.

It may be handy for duplicating columns with Option+ArrowLeft and Option+ArrowRight.

Solution

Try it out.

Problem

My own way of seeing what is going on in my ARC solver, has so far been logging images to a web console. It's hard to determine, did I solve this task better in the past. How similar is my prediction to the target image.

Inside the task gallery; Ideally being able to see the predictions on the thumbnail, this will greatly help debugging.

Solution

Suggestions are welcome.

Problem

Inspired by The ARC Game
https://volotat.github.io/ARC-Game/

I enjoy solving puzzles I have never solved before.

However I find it boring solving puzzles I already have solved.

Solution

Track what tasks the user have solved in localStorage or IndexedDB.

Filter options:

• Only show unsolved tasks in the Task gallery.
• Only show solved tasks in the Task gallery.

Visualization of solved or unsolved status for each task thumbnail.

Problem

On my a touch screen, it's too easy to make a text selection of non-text content. And the user gets presented for the iOS text action menu, with copy/search. This menu causes confusion when I show it to my friends.

Solution

Disable text select mode.

Problem

Undoing set pixel actions happens one pixel at a time, so it takes a long time to undo the latest drawing operation.

Solution

The entire history of the set pixels actions must be preserved, since it's needed for replaying the solution.

Idea: assign a unique id to the actions that belong to the same batch.

Problem

This is related to #28

It's rare that I trigger it myself. However I have seen this several times, when I have my friends try out solving a task. So it is a serious problem.

If you tap on one of the test pairs, then the drawing canvas takes up the entire screen estate, so there is no way to get out of the edit mode. The only way to get out is page reload and loss of data.

Recreate the problem

• On an iPhone with tiny display. Also happens on iPad.
• Open ARC-Interactive in Safari.
• Go to the overview page.
• Zoom in.
• Tap on one of the test pairs.
• Now the UI is stuck in edit mode. No way to zoom out.

Solution

Is it possible to detect if the UI has been zoomed?

Reset the zoom when switching from overview to editor.

Inspired by the ARC-eyetracking project by Leah Bakst + Michael Pascal.

You rate your own confidence before pressing Submit.

Store this confidence rating in the history log.

Eventually when pressing the Submit button, then show a div that does a playback of all the actions, with a confidence slider and a Submit confirm button.

Problem

There are 2 gifs at the moment. A few screenshots with explanation may be helpful.

Solution

I like the HexWalk readme. That will be something like this:

![ARC-Interactive editor overview](screenshots/arcinteractive_editor_overview1.png)

I like the SWE-Agent readme that has a top banner.

Problem

On an iPad+desktop, there are too many thumbnails in the task gallery.

On my desktop, the thumbnails are too small.

The current size is always 80px tall, and width is either 160px or 320px. I will be nice to have 3 different sizes.

Solution

3 sizes: small, medium, large.

Either on the settings page, or in the navigation bar show a zoom parameter.

Save the zoom parameter in local storage.

Problem

The thumbnails have lots of unnecessary space.

Solution

Hypothesis: Experiment with center of mass in the center of the thumbnail, maybe that will look nicer.

Trim unused space.

Problem

ARC has 400 evaluation tasks that you are not supposed to look at.

Solution

Make it more visual that the current task is evaluation.

Problem

The segmented control doesn't look like a segmented control.

Solution

Investigate how to make a nice segmented control.

Problem

This task on an iPhone. The left most image is truncated. The right most image is truncated.

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=27a28665

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=253bf280

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=f3e62deb

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=d5c634a2

Solution

When there are too many pairs, so the images cannot fit in, then make the area horizontal scrollable.

Problem

I'm not using devicePixelRatio for the lazy loaded <img> elements.

I think the thumbnails can show more details if they are rendered using devicePixelRatio.

I made one false start on fixing this issue.

Solution

Replace <img> with <canvas>. This is non-trivial.

Compare the new proof of concept with the existing solution. Are the new thumbnails more detailed, if so, then continue with this solution.

Problem

ARC-Interactive

New users may think the number can be used for the math, which it cannot.

Solution

Hide the integer number above each of the pairs.

Alternative replace the integers with A-Z.

Or make the number appear while hovering with the mouse.

Problem

The rounded square buttons have browser specific styling, and looks terrible on small devices.

The svg symbols are not inside the buttons.

Solution

Investigate why the svgs aren't centered.

Investigate how to get rid of the browser styling of these buttons.

Problem

ARC-Interactive the tools panel is hidden behind the tools: paint button.
Possibly harming discoverability of the UI.

On mobile phones, there is no room for a tool palette so that's fine.
On big displays where there is plenty of empty space, it may make sense to show a tool palette.

In other paint programs there is usually a palette with tools.

Solution

Determine if there is sufficient room for a tool palette.

Possible migrate the entire tool menu panel so it's always visible on big displays.

Problem

The palette is difficult to interact with.

The pick test pair, is difficult to interact with.

The Apple pencil on iPad works ok with the <canvas> elements and <button>, but doesn't work well with the onclick handlers.

Solution

Replace onclick handlers with something that works more smoothly with Apple Pencil.

Problem

I had to change a few pixels in an existing task in the train pairs.

I had to open the file in a text editor, and find the right pixel, and open the file several times in the original ARC editor to confirm that I had changed the right pixel. Not a smooth work flow.

Solution

• Edit any image inside a task.
• Download the result as a json file.
• Add new train / test pairs.
• Delete new train / test pairs.
• Change order of pairs.

Problem

On the editor page. Tap the "Tasks" button. The task gallery should appear.

When the editor has been opened in a separate tab, and clicking Tasks then nothing happens.

When the editor has been opened with a direct url, then there is nothing in the history, and clicking Back doesn't do anything.

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=18419cfa

Changing the Tasks button to link to the index.html is not the solution, since it discards the vertical scroll position.

Solution

Modify edit.html, so its Tasks button always jump to the index.html page.

If the user previously visited index.html, then preserve the vertical scroll position.

I guess the ugly solution is something like this.

On the index.html, when opening the editor, provide an url parameter ala edit.html?from=index, so that the editor knows that the history.back() function is available. Immediately after the edit.html page has opened with the url parameter from=index, then remove the url parameter, so it doesn't get bookmarked.

When the edit.html gets opened without the from=index url parameter, then link to the index.html page.

Alternative ideas are welcome.

Problem

Currently it shows the input image, as the output image. Despite the user haven't drawn a single pixel. This may be confusing.

Solution

Show a questionmark until the user have made some changes to the image.

Problem

When I open this on my mobile phone, the thumbnail is blank. The grid is enabled.
https://neoneye.github.io/arc/edit.html?dataset=ARC&task=281123b4
https://neoneye.github.io/arc/edit.html?dataset=ARC&task=0934a4d8

It seems like the cellsize was zero.

Solution

• Set a minimum cellsize to 1.
• If the aspect ratio is too wide/tall, so the cellsize drops below 1, then overdraw on the neighboring pixel.
• Maybe render to a temporary image.
• Maybe downscale the image to fit inside the container.

Problem

The edit.html assumes that the user has visited the index.html page beforehand, and that the tasks have been fetched.

However nothing happens when opening a direct link to the editor, without ever having visited the ìndex.html`.

https://neoneye.github.io/arc/edit.html?dataset=ARC&task=18419cfa

Solution

In the edit.js fetch the json.gz for the dataset, and store it in IndexedDB.

Problem

I had to make some changes to an existing task, and wanted to verify that I had made the right changes.

However there is no way to upload a file and inspect the task. So I had to use the original ARC editor.

Solution

Add a browse for existing file button.

Problem

In the major browsers:

• The typical resolutions are 1280x720, 1920x1080, and slighly higher when using external camera.
• The max frame rate seems to be 30 fps. I have not seen faster fps.
• A higher frame rate + higher resolution is needed.

I tried WebGazer, but with only 30 fps and low resolution, the accuracy wasn't good.

Solution

There are VR headsets with cameras that records eye movements.

• Is it possible to get access to this data?
• How accurate is this data?

Problem

The left side and right side of wide thumbnails, are not aligned with the normal thumbnails.

Solution

Make the thumbnails aligned.

Problem

I wonder how does humans solve ARC task without any help.

Currently the UI shows several hints, that may be helpful.

Most ARC datasets have names that explains what the solution is.

The named CleanUp2, reveals it's about cleaning up stuff.

https://neoneye.github.io/arc/edit.html?dataset=ConceptARC&task=CleanUp2

Solution

On the edit.html, don't show the name in the: url, title, heading.

Regarding the url. Maybe base64 encode the name.

Problem

Currently ARC-Interactive moves the content around confined inside the selection rectangle, pixels wraps around.

While trying out O2ARC I noticed that the way the selection rectangle gets moved around is different than in ARC-Interactive.

In O2ARC the x, y position of the selection rectangle gets moved, while the content of the rectangle is preserved. It's unclear what happens to the pixels outside the selection rectangle. It would make sense if the pixels gets moved to the other side of the selection rectangle.

Solution

Try it out.

I'm curious to if this is more intuitive, than the way ARC-Interactive currently does moving around of content inside the selection rectangle.