jpmorganchase / mosaic Goto Github PK

Document how each of the 3 existing Mosaic Sources can be configured and used.

Show how the Mosaic CLI can be used to add a source to the mosaic config

Examples

1. git-repo source github
2. git-repo source BB
3. local folder source
4. http source

The Sidebar and TOC are scrolled when the page scrolls. On long pages this results in them being off screen and unusable e.g. this page --> https://mosaic-mosaic-dev-team.vercel.app/mosaic/quick-start/publish-site-to-aws

Another consequence of this is the TOC highlight to indicate what section you are on is broken.

If you click on a tab in the app header, and then click on a child page in the sidebar, the tab in the header is no longer highlighted.

With plugin errors no longer closing sources it would be good to find a way to surface plugin errors to the site somehow.

Perhaps a mosaic dev tools à la https://tanstack.com/query/v4/docs/react/devtools
Perhaps a new middleware
Perhaps we write a plugins-report.json file into the Mosaic global filesystem.

Right now, the plugin errors are grouped by source. But this isn't something a regular user would know about. It would be more intuitive to show all plugin errors for a namespace (which may contain multiple sources). The SourceManager should be able to handle this.

At present, when there is an error thrown by a source, the source is closed.

Now that schedules have been implemented we can add in source retries.

Configuration

Extend the schedule schema to introduce retry config:

• retryEnabled
• Number of times to retry before failing
• exponential backoff strategy

A few accessibility issues with the TOC component:

• every item has the same aria-level (which I think should be inline with the heading level)
• the items are <p> when the should be <a>
• not accessible via the keyboard (tabindex is -1)

When you give a page an alias, the alias works fine and the page content etc is visible. However, the app header content is removed so it looks empty.

I think it might be related to the shared config being lost somehow

At present, frontmatter in pages can be referenced using a ref or tag.

It would be good if you could also reference content on pages.

1. break up the content into blocks to allow fine grained control over what content is pulled in.
2. Build an API around getting the content - getSection('heading-name'), getContentBlock('name')

When generating a snapshot or serving certain sets of docs, the following error is sometimes thrown:
"TypeError: Cannot read properties of undefined (reading 'childNodes')".

We are beginning to add code examples to the mosaic docs now and these examples would be much clearer if the markdown and rendered preview were accessible on the page.

The new React doc use sandpack for this so that might be a good starting point.

Errors generated by plugins result in sources being closed.

This presents a problem whereby an issue on 1 page will destroy the source and therefore all pages that are OK are unavailable. To the user of a mosaic site you will see the 404 page everywhere.

We should attempt to handle this better so that broken pages are 404s (or show the error) but good pages are still available.

Given 2 index pages with shared config, 1 of which is in a parent directory of the other, then the child shared config should be merged with the parent.

This is best illustrated by the fact that most pages define a sharedConfig for header/footer info on the site root index file. Without merging, any other index page that specifies a shared config will override the root shared config. This removes heading links etc which is not great.

A workflow should execute in it's own worker thread to prevent any issues serving pages

If 2 or more sources use the same repo then there is an error thrown when cloning:

[Mosaic] Source Error: Command 'git clone <repo url> --no-checkout --origin=origin' failed: 128. fatal: destination path '<repo project-name>' already exists and is not an empty directory.

Basically, this is a timing issue at heart.

1. both sources start up
2. check for a previous clone which doesn't exist
3. clone the repo
4. find out that the other source has created the repo directory first
5. throw error

We have logic that checks for a previous clone but it does not work across sources. A git source is only aware of what it has done.

Workaround

Schedule the sources to start with different delays. Delayed sources won't attempt a clone and will re-use the earlier cloned repo.

Refs and Aliases are powerful features of mosaic that can be used by content authors. The ref and alias plugins are always included in the mosaic config due to the key role they play.

Document and provide examples on how a content author can make use of them.

The Mosaic packages require a more thorough test plan in order to have confidence in the tool.

As part of this issue it would be good to begin the process (ordered by priority)

1. Encourage new Pull Requests to have tests with a preference towards e2e tests when this is available
2. Create a Playwright test Suite for e2e testing of the mosaic site. This will likely be triggered by a github action so that the e2e tests are integrated in the CI/CD workflow.
3. Improve the test coverage for existing code
4. Move away from Jest to Vitest as the test framework used. See the Migration Guide

Document each of the Mosaic Plugins and describe the role of plugins in Mosaic architecture.

Overview

1. Plugin Lifecycle
2. Plugin runner
3. Main Process vs Child Process
4. Plugins enabled by default (refs and aliases)
5. How to add a plugin to mosaic config?

Per Plugin
How to configure them?
What features to they provide?
What frontmatter is used/added/updated by the plugin

Currently, font loading is working via a custom Document with link elements being injected into the head https://github.com/jpmorganchase/mosaic/blob/main/packages/site-components/src/Document.tsx

This affects the initial paint time and slows down the site loading. Ideally, Mosaic should rely on next/font here, e.g. jpmorganchase/salt-ds#1509

When using the Fragment or PropsTable Plugins, a \ is being added to the beginning of lines that have components on them. This causes an MDX compilation failure and the page doesn't load.

Issue first observed in this PR --> jpmorganchase/salt-ds#1889

Screenshot of broken source MDX

Dev Notes

Both of these plugins make use of remark to process the page content so that could be the source...

After this commit: 6511d7c generating snapshots doesn't work with mosaic and failed with:

@jpmorganchase/mosaic-site:serve: 8080 Error resolving ref(s) for page '/mosaic/author/aliases.mdx'. Error reading file "d:/mosaic/author/" in 'd:/mosaic/author/'
@jpmorganchase/mosaic-site:serve: 8080 Error: Error: Error: Plugin 'D:\Work\mosaic\packages\plugins\dist\$RefPlugin.js' threw an exception running `$beforeSend`. See below:
@jpmorganchase/mosaic-site:serve: 8080   JSONParserError: Error reading file "d:/mosaic/author/"
@jpmorganchase/mosaic-site:serve: 8080     at D:\Work\mosaic\node_modules\@apidevtools\json-schema-ref-parser\dist\lib\parse.js:109:23
@jpmorganchase/mosaic-site:serve: 8080     at Generator.throw (<anonymous>)
@jpmorganchase/mosaic-site:serve: 8080     at rejected (D:\Work\mosaic\node_modules\@apidevtools\json-schema-ref-parser\dist\lib\parse.js:29:65)
@jpmorganchase/mosaic-site:serve: 8080     at runNextTicks (node:internal/process/task_queues:60:5)
@jpmorganchase/mosaic-site:serve: 8080     at process.processImmediate (node:internal/timers:447:9)
@jpmorganchase/mosaic-site:serve: 8080     at Object.$beforeSend (file:///D:/Work/mosaic/packages/core/dist/plugin/index.js:66:23)
@jpmorganchase/mosaic-site:serve: 8080     at runNextTicks (node:internal/process/task_queues:60:5)
@jpmorganchase/mosaic-site:serve: 8080     at process.processImmediate (node:internal/timers:447:9)
@jpmorganchase/mosaic-site:serve: 8080     at async file:///D:/Work/mosaic/packages/core/dist/worker/Source.worker.js:38:9
@jpmorganchase/mosaic-site:serve: 8080 Error: [Mosaic] Source 'site#c05241dc' threw an exception. See below:
@jpmorganchase/mosaic-site:serve: 8080     at EventEmitter.<anonymous> (file:///D:/Work/mosaic/packages/core/dist/PullDocs.js:81:27)
@jpmorganchase/mosaic-site:serve: 8080     at Object.onceWrapper (node:events:629:26)
@jpmorganchase/mosaic-site:serve: 8080     at EventEmitter.emit (node:events:526:35)
@jpmorganchase/mosaic-site:serve: 8080     at file:///D:/Work/mosaic/packages/core/dist/Source.js:124:57
@jpmorganchase/mosaic-site:serve: 8080     at EventEmitter.autoUnsubscribeHandler (file:///D:/Work/mosaic/packages/core/dist/WorkerSubscription.js:34:13)
@jpmorganchase/mosaic-site:serve: 8080     at EventEmitter.emit (node:events:514:28)
@jpmorganchase/mosaic-site:serve: 8080     at #onError [as error] (file:///D:/Work/mosaic/packages/core/dist/WorkerSubscription.js:61:23)
@jpmorganchase/mosaic-site:serve: 8080     at ConsumerObserver.error (D:\Work\mosaic\node_modules\rxjs\dist\cjs\internal\Subscriber.js:124:33)
@jpmorganchase/mosaic-site:serve: 8080     at Subscriber._error (D:\Work\mosaic\node_modules\rxjs\dist\cjs\internal\Subscriber.js:84:30)
@jpmorganchase/mosaic-site:serve: 8080     at Subscriber.error (D:\Work\mosaic\node_modules\rxjs\dist\cjs\internal\Subscriber.js:60:18)
@jpmorganchase/mosaic-site:serve: 8080 [Mosaic] Source 'site#c05241dc' closed

Some content may need to be protected or differ depending on the role of the logged in user.

TODO: add more detail

Create a source that can pull in data generated by typedoc

Markdown plugin -- https://github.com/tgreyuk/typedoc-plugin-markdown/tree/master/packages/typedoc-plugin-markdown#readme

Currently if you use the FragmentPlugin and the fragment page has references to metadata, such as {meta.data.title} it will show the host page's title. This should be changed so that it references the metadata from the fragment page. Currently we would just advise that fragments do not contain such references.

Example:

If you edit have a host page referencing a content-fragment.mdx page that looks like:

---
title: Content Fragment
---
#### {meta.title}

You should see the title of that section on the host page be Content Fragment but what you get is the host page's title.

One for a follow up

Originally posted by @DavieReid in #354 (comment)

Currently, where a directory contains an index.mdx file, Mosaic will ensure that the corresponding URL ends in index. For example:

• Source: foo/bar/index.mdx
• URL: https://example.com/foo/bar/index

Requests to the URL https://example.com/foo/bar will be redirected to the corresponding https://example.com/foo/bar/index URL.

It would be nice if there was an option to change the behaviour so that it becomes:

• Source: foo/bar/index.mdx
• URL: https://example.com/foo/bar
• https://example.com/foo/bar/index would be a permanent redirect to https://example.com/foo/bar

This is the behaviour we would like for the Salt site as:

• We get more succinct URLs (easier to read, share verbally, guess, etc.)
• It's more inline with how out-of-the-box Next.js behaves and indeed many other site generation tools where index.md|html|whatever files tend to be served up at URLs whose path omits the index part.

I suggest that this becomes an opt-in option though (perhaps via mosaic.config.js?), so that other Mosaic-based sites don't suddenly have all their URLs change upon upgrading.

Note that the non-index-suffixed URLs would also need to be reflected in any auto-generated links for site navigation, breadcrumbs, etc.

Given this source:

{
      // disabled: process.env.NODE_ENV !== 'development',
      modulePath: '@jpmorganchase/mosaic-source-local-folder',
      namespace: 'mosaic', // each site has it's own namespace, think of this as your content's uid
      options: {
        rootDir: '../../docs', // relative path to content
        prefixDir: 'mosaic', // root path used for namespace
        extensions: ['.mdx'] // extensions of content which should be pulled
      }
    }

it should work without the prefixDir option.

When using the InsertLink functionality of the content editor plugin, the Dialog closes when clicking into the link edit form

Mosaic uses MDX2 to create documents and has support for github flavoured markdown.

Provide a detailed example of how to author a page using MDX and showcase what mosaic components can be used.

Full component documentation will follow at a later date so for now it is sufficient to document the most commonly used components e.g. callout, tile

Issue

When deploying a mosaic site and snapshot to vercel, the snapshot directory is not included in the code deployed to serverless functions.

This results in the site showing the 404 page for all content and the following is displayed in the vercel logs

ENOENT: no such file or directory, realpath '/var/task/<site-name>/public/snapshots/latest/<page-name>.mdx'

Cause

After investigation, it appears that unless a file is somehow included in the source code, as described here then vercel will not include those files in the build output.

Why does the mosaic site itself work?

The site middleware package includes code that resolves the files when using snapshot mode.

Breadcrumbs are a feature that need to have a holistic view of the site pages in order for the breadcrumbs to be generated successfully.

Consider:

• Source A contains info for multiple products under the route /home/products
• Source B contains info for a single product

Expected

The breadcrumbs of Source B correctly have breadcrumbs home > products > product

Actual

The breadcrumbs of Source B only have product

Root Cause

Breadcrumbs are created in the $afterSource lifecycle event at which point the plugin does not have visibility of the global filesystem.

The Frame

The frame can be thought of the components that side around the main page content. The furniture so to speak. For Mosaic this includes:

• Header
• Footer
• Left Sidebar (a vertical navigation) - navigate to any page
• Right Sidebar (table of contents) - navigate within pages
• Breadcrumbs - show where you are and allow quick navigation to a previous page
• Doc Paginator - navigate to next/prev page

How to set a Page Layout

It is set in the frontmatter of a page using the layout field. If no layout is provided then the FullWidth layout is used by default.

Existing Layouts

The table below details he existing layouts and what frame components they use.

Observations

1. Every layout has a header.
2. If a layout does not have a sidebar or a TOC then it is full width. The only exception is theDetailContentOnly layout.
3. Edit/FulllWidth/Landing are very similar.
4. ProductDiscover and DetailContentOnly are identical.
5. ProductPreview and Landing are identical.
6. The names of the layouts are not clearly describing what frame components will be used by the page.

Future Design

1. Built in Header

Build in the Header component into the Base Layout. It is always needed and I can't see a design that wouldn't have the header in it. This simplifies the logic for all layouts.

2. Frame frontmatter

Instead of stating the name of the layout in the frontmatter, you can choose to enable/disable frame components. This gives users flexibility to show/hide the frame components at will.

---
title: A Title
description: A description
frame:
   footer: true
   sidebar: true
   toc: true
   breadcrumbs: true
   paginator: true
---

If you choose to have sidebar:false and toc:false then the page content is full width.
There is no option to disable a header.

Benefits

• Maximum flexibility.
• Less terminology to learn. Don't need to know the difference between for example a ProductDiscover and ProductPreview layout.
• I think/hope this will result in a simpler implementation. A layout becomes a component that reads the config from the frontmatter and acts accordingly. This will improve extensibility when teams potentially add their own layouts.
• Reduces redundancy.

Downsides

• Even if it's not clear what the layouts do it is very easy to set the layout and change it in the frontmatter if needed.
• We would need to find a way to migrate from the existing system to the new system.

This is likely to be a massive refactor.

We would want to do this to take advantage of the speed improvements associated with React Server Components. The app dir is also the general direction next js is moving and we should look to align. See the roadmap.

Off the top of my head the areas this is likely to touch are:

1. layouts
2. loading indicators
3. site components
4. site middleware
5. site store

Create a new Quick Start Guide for creating a new mosaic site.

Probably best to create a new generator that does not add mosaic specific sources but instead adds some sample docs.

Prerequisites

yarn v1
Node v 16+

Steps

1. yarn init - to create a package.json
2. yarn add @jpmorganchase/mosaic-create-site -W
3. yarn mosaic-create-site init
4. yarn mosaic-create-site create -i -o site
5. cd site
6. yarn build
7. yarn serve

NOTES

When adding to an existing monorepo as a new site package, had to run prettier on the generated site.

When using a git repo source, the repo is cloned and a worktree created for the branch specified by the source in the mosaic config file.

Now lets say you create a new branch in git and update the mosaic config file so that the source uses this new branch. When you spin up mosaic, it will attempt to reuse the previous clone and create a worktree for the new branch.

But the previous clone has no knowledge of the new branch as we did not do a fetch. This breaks the source as the worktree creation fails.

This may only be an issue that manifests when developing locally and using git repos as sources.

Workaround

The workaround is to remove the .tmp directory so that the clone is removed and the repo cloned again on next start-up.

Ideal Solution

We should do a fetch even when reusing a cloned repo so that we have the latest info about the repo.

We currently use Prism for code highlighting. It doesn't seem to work very well and I have heard a lot about Shiki recently that is supposed to be better.

I think this rehype plugin would do the job rehype-pretty-code

There are a few instances when you may want to flag a source as disabled, to prevent mosaic from loading docs from that source. You can do this with a namespace but it's a little unintuitive and may have routing implications.

1. local development where the developer wishes to prefer the local source over a remote source e.g. github
2. internal sources vs external sources.

When you navigate to a section on a page and that page is using the Table of Contents (TOC) component the following happens:

Actual

1. the page is briefly at the section linked
2. the page scrolls to the top
3. first section in the TOC is highlighted
4. you are now lost and need to scroll back down the page to find the linked section (if you even know what it is)

Expected

1. The linked section is highlighted in the TOC
2. The linked section is in view (at the top of the page if the page is long enough to do so)

Mosaic generates a sitemap.xml plugin using the SiteMapPlugin.

However, when you try to access the sitemap you get an error saying Unsupported file type

The width of the content, vertical nav and TOC changes between pages.

As part of moving DP Mosaic design language to Salt DS, I'm categorising components that currently being used on DP Mosaic into 3parts.
1. Using Salt component : The component replaced to Salt component. Ex) Button, layout, icon, link, Tooltip
2. Mosaic specific component: The component designed for DP Mosaic using Salt DS.(Currently based on Mosaic Design language but will update it to Salt DS) Ex) Audio, video player, Accordion, callout, card, feature.. and so on
3. To be replaced to salt component: The component building in progress in Salt team. Currently using Mosaic component on the site, to be replaced to Salt component once it’s built. Ex) App header, Nav item, Tabs, List…


Our goal would be replace the DP Mosaic components to Salt DS components and keep Salt DS foundation styles include typography, color, and shadow.
 For the components that designed only specific for DP Mosaic, also will be based on Salt DS, there won’t be DP Mosaic specific styles or specs.



All the related information will be documented on this Figma file

Currently Type ramp update and component categorising in progress.

ACTUAL RESULTS
When creating a 1 page (or more) file-system containing just an index.mdx, you have to specify

frameOverrides:
    header:

(or equivalent) to ensure a sharedConfig file exists.

Without this entry, then you get 404 errors.

EXPECTED RESULTS

A 404 error is not thrown.
If no sharedConfig is defined then it's not an error.

As part of fixing this error we should understand what the minimum frontmatter for a page is.
In theory it could be

---
---

Use https://arethetypeswrong.github.io/ to verify that we are shipping the correct package.json config to ensure types are configured correctly.

We only support node 18 and ESM so any result for CJS or older node versions is irrelevant.

On initial page load, images and icons on the page are huge and take up a significant portion of the page.

It appears Salt styling is not applied to the SSR'd page and so when it first loads, prior to salt styles loading on the client, we have giant images/icons.

The table of contents should indent headings depending on the level to make the page hierarchy clear to users.

    A request to an unknown targetDir causes `mosaic serve` to exit with `Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client`. 

We should probably log the error out like [PULLDOCS] ${targetDir} not found...etc

Originally posted by @tambo in #33 (comment)

We noticed on the Salt site, that the sitemap.xml files generated by Mosaic are incorrect (jpmorganchase/salt-ds#1559):

Description

The sitemap.xml files generated by Mosaic are incorrect. The contents of <loc> needs to be the full URL to the page (see https://www.sitemaps.org/protocol.html), but instead seems to have the path to the source MDX file.

Steps to reproduce

Check the sitemap.xml file being generated by Mosaic. For example, visit https://www.saltdesignsystem.com/sitemap.xml

Expected behavior

The contents of sitemap.xml should conform to the specification.

Remote sources that pull content need to do so on a schedule. It would be good to:

1. Provide a section in the mosaic config that allows users to define a schedule that will apply to all sources
2. but allow users to override the "global" schedule for individual sources
3. standardise the names e.g. initial delay, checkIntervalMins

The git source has onCommitChange function has a default sync but no real way to override this from the plugin options.

The http source has a checkInterval option https://github.com/jpmorganchase/mosaic/blob/main/packages/source-http/src/index.ts#L9

The OpenApiComponent renders the react-swagger component without any styling.

Root Cause

I suspect the root cause is the fact that esbuild is told to exclude node_modules when bundling the open API component
https://github.com/jpmorganchase/mosaic/blob/main/scripts/bundle.js#L36

The package.json file also mentions a dist/index.css file which is incorrect. Should be dist/styles.css.

Expected Result

Adding the following CSS import to _app.tsx will apply swagger styling

import '@jpmorganchase/mosaic-open-api-component/index.css';

Workaround

Add the following CSS import to _app.tsx

import 'swagger-ui-react/swagger-ui.css';

It is important for content authors to have an understanding of:

1. In mosaic terminology what is the "frame"
2. what layouts mosaic provides out of the box,
3. what frame components are shown in each layout
4. how a page can be configured to use a specific layout
5. how a consumer can override the configuration of the frame components

1. If you are trying to reference data on the same page, you need to use quotes
2. When content updates, Mosaic crashes stating there is a circular ref.

When viewing a page with a Table of Contents visible in the right sidebar, clicking on an item should highlight that item.