open-contracting / extension-explorer Goto Github PK

For example, the codelist page can briefly explain what purpose codes and codelists serve (similar to
text prepared for OC4IDS), what they are, and how they're implemented in OCDS. For closed codelists, can describe that they provide titles and descriptions to the enum of array fields, with an example.

On the schema page, a brief paragraph can clarify the use of terms like 'fields', and refer to JSON Schema (similar to text prepared for OC4IDS).

The extensions list page has very little text. We can repeat a brief explainer of what extensions are and why they matter.

Check what the OCDS documentation has for these concepts (if missing, there might be an opportunity to improve in OCDS 1.1.5).

I use the extension explorer when I want to check something in an extension's schema or documentation.

Sometimes I also want to check back to the actual Github repository for the extension, for example to look at the commit history or to get the url to check the repository out for editing.

I've just discovered I can get there via the 'Maintained by...' link, but it would be easier if there was a direct link.

We set up temporary hosting for this, to enable good user feedback sessions to go ahead. But it is definitely temporary hosting, and while there are many other tasks still to get right before a public launch we should discuss the plan for permanent hosting.

I've been looking into this and trying some code ....

So the suggestion is to use Travis to build the static site and then to push that static site to a simple web server, in a manner similar to what we do for the standard.

The Travis build will run in a separate repository ( https://github.com/open-contracting/extension-explorer-build ??? ), and will run once a day on a cron job.

You can see an experimental set of scripts at https://github.com/odscjames/test-extensions-explorer-site These do everything except upload to a web server (that would be set up for master branch only).

You can see a build for these at https://travis-ci.org/odscjames/test-extensions-explorer-site/builds/428595924 - it lists all the files it has built.

Why Travis?

Easy to set up, all the user access for actions like "rebuild" are already done, state and errors of process are public for all to see.

Why not the https://github.com/open-contracting/extension-explorer repo?

Because we wanted to keep builds for this repository deterministic, so that if a developer is working on a new branch and starts to get errors they know the error is something they have done and not another change outside the repository that is hard or impossible to track down.

It would be very difficult or maybe impossible to cleanly add deploy stuff to this repo without that coming in (as the deploy stuff would need the data from extensions-data-collector and then you lose the deterministic aspect).

So it may be possible, but it would start to make the Travis process quite complicated. The only benefit of putting the deploy in here is that you would automatically get the website redeploying everytime you merged to master instead of up to 24 hours later, but as we don't anticipate that many urgent changes we don't think that's important - and if there are any important changes it's possible to go to Travis and push "rebuild" on the last build to get a deploy anyway.

Which server?

Just an Apache Linux Open Data Services one, similar to https://raw.githubusercontent.com/OpenDataServices/opendataservices-deploy/master/open-contracting-standard-deploy.sh - I haven't looked into the details of this yet but don't anticipate any issues here.

Posting for comments and feedback .....

The schema reference page shows the types for each field. If the field is a $ref it shows the definition name as the type.
These should be links to either the core docs or to definitions that live within the page.

The explorer currently marks core extensions as recommended and shows the extensions' authors.

However, these subtle clues might not make it clear to users that there is a substantial difference in the quality and stability of extensions.

We can consider a more expressive banner to communicate this, on extensions' documentation pages.

This could be multiple locations. This is similar to #17 (which was for fields).

For example, https://extensions.open-contracting.org/en/extensions/budget_and_spend/master/

e.g. with a colored badge

It would be great to have ability to search extensions.

An MVP version might be a search on the user-interface layer of the full library (see #20) of extensions, based just on titles and descriptions.

A more developed version might integrate extensions into the updated OCDS docs search, so that there is unified search across the two which:

• Returns OCDS core documentation responses first

• Shows extension results in a tab / below

so that we're prioritising the core schema and docs as the answer to people's questions, but then providing access to extensions when the core schema doesn't cover what users need.

On http://localhost:5000/en/extension/lots/v1.1/

I found this while trying to get Frozen Flask working

I would like to propose that the extension explorer rename ‘Core’ extensions to ‘Recommended’ extensions, and that it include a version of the following text at the top of the page.

The main Open Contracting Data Standard covers data that is both widely available, and useful to a wide range of different users.

Our recommended extensions describe how to publish:

• Additional data that exists in some countries, but not others due to the way contracting processes work, but that should always be published when it exists (e.g. Lots);

• Additional data that describes important features of a contracting process, but which may not be possible to publish in all countries, or for all kinds of contracting process (e.g. Bid details);

• Additional data that has widespread value to users, but that often requires publishers to make specific plans to collect this data, or to collect it in a structured way (e.g. Lots).

These extensions are maintained alongside the core standard, and go through the same governance process as OCDS, in order to make sure they represent the needs of a wide community.

We encourage all users of OCDS to consider whether these extensions could be used with their data, and where they lack the data needed to use an extension, to consider whether this could be collected in future. “

At the bottom of the page we could include:

Note: Recommended extensions were previously called ‘Core Extensions’. The name was updated in 2018 to better reflect their purpose. “

The description above would work for all current core extensions with the exception of “Milestone documents” for which either:

(a) We move it to be a community extension (not strictly allowed without a 1.2 I think, but we might be able to justify doing that in a bug-fix release)

(b) We add some special handling for this case, and add a section to the bottom of the ‘Recommended extensions’ page where ‘Milestone documents’ would display, with the following heading and description:

Extensions covering deprecated fields

When certain features of OCDS are deprecated, we may in some cases maintain extensions that ‘add back’ those features. These are provided so that users declaring use of these extensions do not receive data validation errors when using the deprecated properties. These use of these extensions is not recommended unless you have previously been using the fields and data structures they specify.

In tandem with #19 I would like to suggest we rename ‘Community extensions’ to ‘Extension Library’ and:

• Include all extensions in here, but visually mark ‘recommended’ (i.e. core) extensions in some way.

• Visually mark on the cards the GitHub repo an extension is from (i.e. to show when they are open-contracting maintained or not)

• Include the pre-amble below at the top of the page:

The extensions library contains extensions developed by the OCDS team, and by members of the OCDS community.

Many extensions have been developed to meet a particular publisher or user need. They are included in the extension library if they are likely to be relevant to more than one publisher, user or use-case.

The reason for this suggestion is that, whilst we want to draw particular attention to recommended extensions somewhere, we don't want people who are browsing extensions to miss finding a core extension that could have been useful to them.

Using visual cues, rather than silos, to indicate the extent to which an extension is 'recommended' / 'strongly encouraged', or the extent to which it can be trusted as reliable, may help us in future.

Running find . -name '+*' and find . -name '-*', I can't find an extension that does this, but if/when one does, the code probably doesn't handle it. (I wrote this note sometime in 2018 or 2019, so I assume the code doesn't handle it.)

This will allow linking to the Extension Explorer from other documentation, without having to pin to a specific patch version of an extension.

Conceptually, each version of OCDS should refer to a minor version of an extension. For example, OCDS 1.1.4 might refer to version 4.5 of ocds_ipsum_extension (a fictional core extension). If a new patch version of ocds_ipsum_extension is released after 1.1.4, users should be led to use that patch version (e.g. 4.5.2), not the patch version that was available when 1.1.4 was released (e.g. 4.5.1). This allows bugfixing core extensions without releasing a new version of OCDS.

In practice, given that we want the extension explorer to be a static site, one option is to create redirect pages at URLs like /extensions/ipsum/latest/v4.5/, which would redirect to the latest patch version at /extensions/ipsum/v4.5.2/. To author the redirect page, we'd determine a list of minor versions (one of which would be v4.5) and look for the most recent version that matches >=v4.5,<v4.6. (Along the same lines, we could have /extensions/ipsum/latest/ to redirect to its latest version.)

This can be broken up into individual issues, but I wanted to capture some things that had been discussed as potential follow-up to the creation of an extension explorer.

• Delete the building of extensions.json from the extension registry (needed for extensionlist directive)
• Update the standard and profiles documentation to refer to specific versions of extensions in the extensions explorer, instead of including extension documentation and extension lists
• Move extensions' readme to docs/ folder, and replace the root readme with some boilerplate that e.g. provides context about OCDS and extensions, and links to the extensions explorer
• Update the extension template to match above (as not all extensions will be in the explorer, we want the repository page of such extensions to display the docs)
• Update "documentationUrl" in (if appropriate) extension.json, standard_extension_template, extension-schema.json, Reviewing community extensions.
• Fix the broken links to OCDS for PPPs in the risk allocation extension open-contracting-extensions/public-private-partnerships#165

Stop using GitHub URLs for extensions, where possible:

• extension.json: documentationUrl
• ocds_transactions_relatedMilestone_extension: README.md

Discussion on how we implement the website.

e.g. the person starts typing and a dropdown lists extensions whose titles or descriptions match.

I created a python3 virtual environment, activated it and installed requirements.txt but I'm getting errors trying to run the following command in README.md:

Command:

ocdsextensionregistry generate-data-file --locale-dir ../ocds-extensions-translations/locale/ > extension_explorer/data/extensions.json

Error:

exception "No module named 'recommonmark'" prevented loading of ocdsextensionregistry.cli.commands.generate_data_file module
exception "No module named 'recommonmark'" prevented loading of ocdsextensionregistry.cli.commands.generate_pot_files module
usage: ocdsextensionregistry [-h] {download} ...
ocdsextensionregistry: error: argument subcommand: invalid choice: 'generate-data-file' (choose from 'download')

I pip installed recommonmark and tried running the command again which resulted in the following error:

FileNotFoundError: [Errno 2] No translation file found for domain: 'additionalContactPoint/master/schema'

An intital wireframe of how a core extensions may look can be found in:

https://open-contracting.github.io/extension-explorer/wireframes/core_extension_0.html
(the fonts are not working due to limitations in typekit)

e.g. https://extensions.open-contracting.org/en/extensions/implementationStatus/master/
https://extensions.open-contracting.org/en/extensions/guarantees/master/

Can maybe warn and remove the tables?

The description of these extensions (in extension.json) can also be abbreviated (or perhaps autolink the URL).

If new versions of extensions are added to the registry, the explorer fails to build due to missing MO files in https://github.com/open-contracting/ocds-extensions-translations

A workaround would be to:

• Run ocdsextensionregistry generate-pot-files build/locale to create POT files
• Create empty PO files, if PO files are missing, using a loop similar to this one
• Run sphinx-intl build -d locale to create MO files

• Integrate content from CRM-3987
• Describe multilingual support (the schema page links to a #multilingual-support anchor)
  • Can link to http://standard.open-contracting.org/latest/en/schema/reference/#language
• Uncomment the three occurrences of the 'documentation' link
• Ensure the content covers https://www.open-contracting.org/tag/extensions/ and links to the schema style guide
• Add content relevant to passing extension review (see internal process document)
  • Consider which of the review criteria to add to the readmes of the extension registry or extension template
• Explain that extension authors just need their extension versions added to the registry to appear in the explorer
• Decide whether to move content from the extension template to either the explorer or the standard documentation
  • This content might be split into multiple pages.
  • If moved, we can replace the extension template readme with sample headings and expand on what makes good documentation

At the moment the schema reference page just gives the list of all the fields within all the definitions that exist in the extension.

It would be useful to also display what paths (i.e jsonpointer paths) within the schema showing where those definitions live.

Follow up issue to #47. Copying relevant comments:

• Add content relevant to passing extension review (see internal process document)
• Consider which of the review criteria to add to the readmes of the extension registry or extension template

I've integrated the criteria from the 'quick checks' section of the process note, but left the 'in-depth checks'. Potentially, we could add the criteria under the 'individual files' subheading to the extension registry readme.

I'm not sure which heading you're referring to. Do you mean under each subheading under this heading in the template's repo? https://github.com/open-contracting/standard_extension_template#extension-repository-structure

Ah sorry, I was referring to the headings in the process note. I've integrated the quick checks, but left the in depth checks. My suggestion was to add the content from the individual files subheading to the extension registry readme - I hadn't thought about where exactly in the readme to put it, though.

Right - I don't think it belongs in the registry repo, if the goal is for publishers to author good quality extensions (whether registered or not). I thought maybe it'd work better in the extension template repo (though if we move content out of there, then it probably makes sense to put it in the Extension Explorer).

Prototype: https://github.com/open-contracting/editor-tools/issues

This issue is intended to collate comments about the design and minor layout changes to the extension-explorer website.
This supersedes issue #1.

Any issue about functionality or about what kind of things we display should be raised in individual separate issues.

I got very confused looking at the pagination extension because the schema page shows lots of deprecated fields.

I think we should hide or collapse them, or at least indicate which fields are deprecated.

IMHO? Agree/disagree?

Doing so creates a lot of meaningless commits as we update it to the latest data, and also we end up with a lot of data here that will make the git repo much larger than it needs to be.

(I understand this was probably done for project kick off, which is fine - I just wanted to note it so we pick this up later.)

Instead we should set something up to automatically (or at least with a few well documented commands) include the other repository and build the data.

What we set up probably depends on how we want to deploy this website, so leaving for discussion with @kindly

They are only 30% and 28% translated. (Spanish is 83% and moving up.)

Follow-up to open-contracting-archive/toucan#12

There is some commented-out code suggesting the JSON Schema validation properties that I intended to add support for. In addition to those, I can't remember if required fields are indicated.

We can compare the functionality / presentation to the sphinxcontrib-jsonschema Sphinx directive.

For a reason I forget, I noted the PPP extension as a good test.

Some of mine that should be in there:

• Frozen-Flask

I'll start with three user stories:

• AS A data publisher following a link from OCDS 1.1, I WANT TO be linked to the version of an extension that is relevant to me without needing to think about it SO THAT I use the correct extension version.”

• AS A policy maker concerned with our ‘compliance’ with open contracting I WANT TO know exactly which core extensions we should be using SO THAT I’m not at risk of trying to hit a moving target”

• “AS A developer implementing a commitment to publish to OCDS 1.1 I WANT TO have an extension selector that lets me get the code I need for all the 1.1 versions of OCDS core extensions SO THAT I don’t risk using the wrong version of each extension”

At the moment the display of versions feels quite confusing (reflecting some underlying confusion in our versioning approach I think).

In theory:

• For core extensions, the standard pins a particular version. If I link from OCDS 1.1.3 -> extension explorer, I should be shown the list of core extensions as pinned in OCDS 1.1.3;

• For community extensions I should generally be taken to the the latest version compatible with the version of OCDS I am working with, and only then offered chance to check other versions;

Suggestions

• We make sure the OCDS documentation at http://standard.open-contracting.org/latest/en/extensions/ lists the recommended extensions, and the particular versions of these ‘pinned’ by that version of OCDS, linking to that specific version of the docs;

• We make sure it possible to link to the extension selector, with it pre-filtered to display only core extensions, and using the versions pinned for a specific OCDS version (e.g. 1.1.3) [NOTE: It is not inevitable that OCDS version 1.1.3 will always pin an extension version 1.1.3; When we OCDS 1.2, it could continue to pin to extension versions 1.1.3 if those versions have not changed. This maintains the semantic versioning property of the extension version number).

• We add a bit of documentation (possibly just tooltip style) to version information to explain “OCDS extensions are expected to follow semantic versioning principles, using the major (1.x.x) number to indicate backwards compatibility, and the minor and patch number to indicate new features and bug fixes respectively.”

Additional suggestions

• We add a drop-down to the top of the screen to select which version of OCDS you are currently dealing with (and allow this to be set via parameter / cookie etc.)

• We use something dynamic to then highlight which versions of the extension are compatible with that version of OCDS (using the compatibility fields from extension.json)

• We rename ‘master’ to ‘latest’ - as ‘latest’ is a more familiar label to users (ReadTheDocs does this for example… - well, sort of, it takes the default branch from GitHub as latest, even if that’s not called master - we might want to do the same)

Mock up here:

(Note: as I go through this, I wonder if with the suggestion in #20 that we include all extensions in the ‘library’, and visually mark ‘recommended’ extensions, whether we even need separate ‘core’ and ‘community’ sections. We can handle listing which extensions are ‘core’/‘recommended’ for a particular version of OCDS within the OCDS documentation itself, just linking out to the explorer as required).

Related: #44

With usual merge protections, etc

Want to check with @kindly before doing this.

https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

It's presently the extension author's responsibility to declare their extension's compatibility with minor versions of OCDS in extension.json.

However, we might anticipate that not all authors will update their extension.json when a new major/minor version of OCDS is released (even if the new version requires no other changes to their extension).

However, we would still like to be able to list extensions that are known to be compatible with a version of OCDS, even if the extension itself doesn't declare it as compatible.

There are a variety of potential solutions, but just describing the desire for now.

For example, the bid extension's readme refers to Bid, Bids.details and bidStatistics.csv among others. We can attempt to automatically link these to the definition, field or codelist.

https://github.com/open-contracting/extension-explorer#manual-tests

Follow-up to open-contracting/ocds-extensions#78

We used to do this with Travis cron jobs. There's a similar feature for GitHub Workflows:

https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#onschedule

On the tech check-in call today, we discussed a possible feature of listing groups / collections / packs of extensions, which might include: profile extensions (e.g. PPP), thematic extensions (e.g. budgetary), etc. – and, if we were to remove 'core extensions' from the main navigation per #21, we could have such a page to list core extensions.

I assume these pages would be configured within this repository. One option is to have a YAML file for each listing (or one YAML file for all listings), with the URL slug and title of the page, and a list of extension names (I only describe this option to give a sense of how I think it could work). Profiles might need a special case.

https://docs.google.com/spreadsheets/d/1HEwlpagffcUV9E3xyhp-Bi4Wpgy5HxI0iGTVALpST_Y/edit#gid=0

The pagination extension patches release-schema.json, release-package-schema.json and record-package-schema.json but only the fields in release-schema.json are shown on the schema page.

We should display fields from all schemas patched by an extension.

For new extensions.

On our call earlier this week, @duncandewhurst mentioned regularly using the docson schema viewer when interpreting the schema.

Being able to view a copy of the schema with any set of extensions applied would potentially by very useful indeed.

Implementation
An ideal implementation might allow that:

• A user can see in docson which fields have been added by the extension, possibly by virtue of colour coding;

• This is visible as part of each individual extension page;

• It is possible from the extension selector to explore what the schema will look like with a collection of extensions applied to it;

Issues to consider

• We may need the ability to see how the extension looks against different versions of OCDS in future (an MVP could merge just against 1.1, but when 1.2 is out, would need to cover both)

Possible implementation approach

I believe we have modified docson in the past and found that possible, so I'd be thinking that we have a process that:

• Grabs the extension schema;
• Modifies this to insert some property to each object or rifled in the extension to include the name of the extension, and a flag that this is an extended field;
• Merge this into the OCDS schema;
• Update docson so that it outputs class information about the presence of an 'isExtended' flag in each object/field, and the name of the extension;
• Update CSS so that this shows up

I had a quick look at https://github.com/open-contracting/extension-explorer/blob/gh-pages/extension_explorer/__init__.py

It's possible for a community extension to be promoted to a core extension. So, to have more or less permanent URLs, core and community should not be part of the path to an extension.

Based on the context shared about core extension versioning, a core extension's version should come after the extension's ID in the URL, since core extensions are not guaranteed to be released as a series.

So, I figure the sitemap would look like, for the routes configured thus far:

/
/<lang>/
/<lang>/extensions/<extension>/<version>/

I used an /extensions/ path component, to avoid conflict between extension names and whatever other pages we add to the sitemap.