hackthedeep / teamgoop Goto Github PK

Even though there probably won't be more than 2-3 other people working on these annotations it'd be nice to remove some barriers to collaboration.

Eg: user A working on microtubules and user B working on membrane.

To that end, it'd be useful if .goop files could be merged -- perhaps it's easier to think of this as adding one file to another?

What if the 2d and 3d views were side-by-side so you could edit both simultaneously?

In order to be able to do final geometry tweaks / lighting / rendering / animation, we'll need to be able to export from the application into some sort of 3d interchange format.

For Milestone 1.0, we'll need a basic format that can be loaded by other 3d applications. The choices are:

• Use OBJ (Looks like there's a way to embed different meshes in one obj described here.
• Custom format based, say, on json. (Con: need to write a reader for this in maya / blender etc)
• Use fbx / abc which maya can read but for which I doubt there's an easy way to author from javascript
• Write out a custom .ma which maya can read + embed controls (gets us pretty deeply in bed with maya specifically though).

I think we should go the .obj route as a starting point if we can get that mesh separation to work!

Lets not be overwrought about this since it's a small for-fun project but we should have some guidlines / at least a linter file so our code contributions are from the same universe.

• Especially for Jesse as a .js noob (Reference point: this)

The heart of the 2d aspects of our application is the main "runloop" which does the following:

• Draw image slider on bottom of interface (allowing choice of other slices)
• Get current slice selection and find / display all linked images w/ applied xforms (for 1.0 will just be one image)
• Apply any slice onion skinning
• Draw any visible shapes based on the color of their type
• Apply any shape onion skinning

Choose any arbitrary slice location (in 2d) and create a 3d locator in that position in 3d space. Could be handy for 3d alignment / bounding.

Corresponding to the 2D draw loop in #10, we'll need to build a 3D loop:

• Display structures (with visibility restrictions)
• zooming / tumbling / 'frame up' controls
• Option to display micrograph slices w/ low opacity?
• Allow selection of points / shapes w/ inspector
• Allow selection operations (move point along slice? Select related etc?)

Does it make sense to use three.js for this?

Our annotation interface must be able to:

• Draw points / shapes
• Editing, moving, rotating points / shapes
• Deleting points / shapes
• Undoing(!)
• Linking different shapes together in the event of overhangs / forks

This is, in my brain, heavily inspired by the 'rotoscope' node in nuke which is shown in tons of youtube videos eg this.

We also need to be able to adjust shape properties:

• Link shapes to structures (with means of changing that link if it's incorrect)
• Shapes will either be dot-based (tubes) or freeform-based (all other cellular structures) so there must be a means to set the radius size of the shape.

Create a way to export a "bundle" of a .goop AND the required images so that it's even easier to collaborate with someone else. Not sure this is super desirable anymore but we talked about it at some point.

Add ability to create 3d primatives and sculpt / distort them into position to sculpt more esoteric shapes.

Perhaps allow use of existing locators (#17) for alignment / bounding of these shapes.

Groups are persistent collection of structures that can be used for bulk operations. By creating a Group class and creating UI to add items to a group / manipulate the group over a certain number of slices we might be able to speed up certain manipulations!

Supplement #16 with some computer vision tech to automate the alignment of higher magnification areas.

Now that we have a much better idea the way forward vis-a-vis these tickets after our meeting with Aaron, we'll need some new designs to work from. Open questions:

• Given that the user is going to be switching between Slices, Images, Shapes and Structures I feel like we might want to represent those objects visually so that we know which inspector to display.

• We need inspectors for:

  • Project settings?
  • Images
  • Slices
  • Shapes
  • Structures

• We also have the following visual modes we need some UI to switch between:

  • view slice + shape (2D)
  • view structure (3D)

• We need some visibility controls for hiding / showing with a lot of granularity

• We need spline / point controls for manipulating shapes ala nuke; probably hasn't changed much since the hackathon.

Use some sort of image analysis to detect and offer to skip importing "bad" images eg where there are folds / tears that make it unreliable / unusable.

Do this by detecting wild changes in the histogram relative to neighbors?

Use computer vision / magic to offer sane auto-white balancing as a one-click starting point. Basically offering to choose default values for brightness / contrast via histogram analysis.

We need to be able to save / load our project files so work can be saved / sent to others for collaboration

• Use JSON? Seems like obvious way to serialize javascript instances / data!
• Use .goop instead of .json though?
• At load time, check image paths -- if missing for now just error.

Leverage some smart denoising algorithms out there and offer them as options alongside brightness / contrast adjustments to help see things more clearly in the micrographs.

For our geometry creation we need to create polygons from drawn shapes. We'll want to do this by lofting the curves together into a 3d form with volume.

We need to account for:

• Shapes with different numbers of points (resample before loft to maximum point count)
• Shapes with multiple non-contiguous sections per slice (eg: overhangs / forks; this might make lofting tricky)

• Survey electron community tools / boilerplate and pick the most appropriate starting point
• Commit this boilerplate on newly cleaned "master" branch
• Instructions on how to run electron locally for us non javascript people (npm? node? commands?)

If this thing really takes off, it might be easier than expected to convert this application into a full blown web app.

Things to consider:

• Logins
• File security
• Hosting
• Store / log activity history
• One installation of the app should host multiple project so we'd need some UI for selecting projects.
• @mentions / (e-mail) notifications?

On the plus side it'd be really easy to serve created structures for download / web view via this platform!

Whichever export options we don't end up implementing for #13 lets consider adding them to the app!

Also to possible include:

• .blend (blender)
• Web based three.js viewer w/ cheesy sci-film overlay graphics?
• Prepare for 3d printing (I actually think it would be better / easier to just import the model into any of the preexisting 3d printing software which is designed to solve this exact problem...)

Use some fancy computer vision libraries to automatically create shapes based on click targets and some brightness / contrast threshold. Something like:

For Release 2, it'd be awesome to replace the need for photoshop and do compositing of multiple photograms into a single slice.

This breaks the faulty Milestone 1.0 assumption of one image per slice.

This would require:

• Multiple images could be imported / transformed / layered eg: multiple images at different magnifications.
• Perhaps non-linear transformations ala gridwarps?
• might want to go back to scope and replace an image with a higher resolution image
• might be hard to know how to automatically update shapes if image is tweaked in a complex way; maybe we just show a flag that indicates that the source slice has changed so the user knows to update their shapes...

On project load if an image is missing, offer the user the chance (via some UI) to relink images.

Critical for collaboration but possible to work around by find/replace on .json/.goop file?

I've assigned @mgoetschel because there's some design work here for sure but also some code that'll need to be written (that someone else should pick up). We haven't discussed a relinking ui but there should be examples from other software that "links" to media.

For this application to work at all, we need to be able to:

• Load any number of files from outside the app into our project.
• Ensure this also works if done after work has been done eg adding more images to a project later
• Store paths to images in .goop file so they can be loaded at project open time without needing to embed the images in our project file.
• Try to create best-guess metadata per image based on the filename / Aaron's supplied regex but offer a chance to override.

Big concern

Can we load 16 bit tiffs into an electron app? Chromium doesn't support tiffs natively I think so we could try this bit of javascript but I dunno if it supports 16 bit tiffs.

Here are some ideas we can "jump" between 2d and 3d modes using some clever contextual options:

• Select a point, right click, open in 3d?
• Select by type
• Select shape by type on this slice

Define core classes to be used by our application.

Use OOP conventions, right?

Here's a list of the classes with the imagined fields we'll need:

Image:

• Path to the file
• Image metatdata (grid, image number, date recorded, magnification...)
• Transformations (spatial and image-based)

Slice:

As of Milestone 1.0, A single image.

• Index
• Depth in space
• Descriptive name (so we can display it with something more useful than "Slice 7"?)

Shape:

Drawn on top of Slices to represent contributions of structures on this slice.

• List of points / vectors. Not necessarily contiguous (eg: a fork)
• End fudge distance (positive or negative)

Structure:

An actual 3d structure. The combination of many shapes.

• Start / end depth (better to use depth than index so if more images are added it doesn't change!)
• Collection of included shapes
• Interpolation method for reconstruction (linear, cubic etc)
• Start cap length, end cap length

Project:

A project file. Contains multiple shapes / structures / slices / images

• Species name
• Descriptive Project name (not just species, right? There might be a few reconstructions of the same species...)
• Preferences
• Regex (for now hard coded but useful to store per project or per user if we want to show to other scientists)
• List of Types (Mitochondria / tube type A, tube type B etc), each with a color and whether it is shape-based or single-point based.