It currently writes an extension.json file instead of an entry.json file, and doesn't follow entry-schema.json.

13 community extensions have the category ppp, but can be given more precise categories like planning, budget, etc. We may want to review whether any are worth surfacing outside the PPP profile.

This is a spin off from #90

Note this includes validation that can be done on extensions.csv only - there is additional validations we can do on other files but that's a different issue.

Currently there is the following validations applied to extensions.csv

• id is unique
• repository_url starts with https://github.com/

What other validations could be applied?

• id is all alphanumeric or underscore only
• repository_url is of pattern https://github.com/XXXX/YYYYY
• a category is set
• a category is set and it is one of an approved list only (could store approved list in new file categories.csv?)
• core is a string which is a boolean value "true", "1", "false", etc - so leaving blank or writing "cat!" would fail (currently blank would be silently changed to false)

As for the method of applying validation, currently validations is done by throwing an exception in ocdsextensionregistry/validate.py or in the validate_extension_registry_data_only method of ocdsextensionregistry.models.ExtensionModel. In ocdsextensionregistry/tests/test_validate.py there are pytests that are already run by Travis.

We could develop new validations directly in Python, or we could also try the technique of converting a line to a JSON and then using JSON Schema. I suspect that in many cases Python would be the simpler or more powerfull route but I note that both can by tested by pytest in the same way as current validations. The important thing is to agree on the criteria - as long as there are some tests I'm not to fussed about the technique.

In an entry.json file we should be able to specify multiple versions of extension.

An option on how this can be done is to allow the entry.json file to contain a list of different entry objects. The first in the list could be considered the latest one but each entry can contain its own semver and a list of versions of the standard its compatible with.
i.e

[
{"name": "bids",
 "version": "3",
 "url" : "http://some.urlr.2/extension.json",
 "compatibility": ["1.2"]
  .....
},
{"name": "bids",
 "version": "2.1",
 "url" : "http://some.urlr.2.1/extension.json",
 "compatibility": ["1.1"]
  .....
}
]

The metrics extension supports publication of forecasts (planning stage), targets (tender stage), agreed performance targets (award and contract stage) and results (implementation stage).

After reading the README on https://github.com/open-contracting/ocds_metrics_extension, here is what I gathered I should do if i was to model the monthly physical and financial progress agreed upon:

At the planning stage, data must be as follow

{
  "forecasts": [
    {
      "id": "completionPercent",
      "title": "Construction progress (percent)",
      "description": "Percent completion of the construction",
      "observations": [
        {
          "period": {
            "startDate": "2017-01-01T00:00:00Z",
            "endDate": "2017-02-31T23:59:59Z"
          },
          "measure": "25",
          "unit": {
            "name": "percent",
            "id": "P1",
            "scheme": "UNCEFACT"
          }
        },
        {
          "period": {
            "startDate": "2017-02-01T00:00:00Z",
            "endDate": "2017-03-31T23:59:59Z"
          },
          "measure": "30",
          "unit": {
            "name": "percent",
            "id": "P1",
            "scheme": "UNCEFACT"
          }
        }
      ]
    }
  ]
}

at the tender, award and contract stage, the key "forecasts" must become "targets"

at the implementation stage, it becomes metrics

{
  "metrics": [
    {
      "id": "completionPercent",
      "title": "Construction progress (percent)",
      "description": "Percent completion of the construction",
      "observations": [
        {
          "period": {
            "startDate": "2017-01-01T00:00:00Z",
            "endDate": "2017-02-31T23:59:59Z"
          },
          "measure": "15",
          "unit": {
            "name": "percent",
            "id": "P1",
            "scheme": "UNCEFACT"
          }
        },
        {
          "period": {
            "startDate": "2017-02-01T00:00:00Z",
            "endDate": "2017-03-31T23:59:59Z"
          },
          "measure": "20",
          "unit": {
            "name": "percent",
            "id": "P1",
            "scheme": "UNCEFACT"
          }
        }
      ]
    }
  ]
}

When doing reporting or analytics, I can use the same ID , in this case "completionPercent" to track how well things unfolded at each of the stages.

So my questions are:

• is my assumption correct about the key name changing across the stages? at planning, key is "forecasts" , at implementation it is "metrics", in the rest of the stages it is "targets"

• is this the best or recommended way to model the data? is there a way to avoid repeating the "unit" block for example?

From #85 (but also related to #97):

Yes, .... author and authorEmail... ... . In terms of need: Whereas, for our extensions, people may know to contact the OCDS helpdesk, for government-published extensions, there is likely no way to contact that government except through GitHub issues, which may not be monitored, based on my experience working on dozens of government repositories via GitHub.

If this is the need, how about adding a explicit support option to https://github.com/open-contracting/standard_extension_template/blob/master/extension.json ? It doesn't guarantee that people will monitor support channels, but this would make people think about the fact support might be needed and allow them to specify their preferred channels. We can highlight these in generated documentation.

Eg, something like:

{
	"support": {
		"email": "[email protected]",
		"web": "https://github.com/open-contracting/extension_xxxxxxxxxxxxxxxx/issues"
	}

}

(Email or web or both)

(Maybe even allow language flexibility, but maybe start with this and think about language later)

These fields should be optional to start with to avoid breaking existing extensions, but after we had added to all the current ones we could even make them required.

In some of the PPPs or some of the "big contracts" there are different types of certificate involved:

• certificate that the bidder is qualified to participate
• certificate of completion or certificate of compliance

I haven't seen it in a real life example, but I imagine that sometimes items used need to be certified (think ISO or UL or green energy certification) - or in case of services, people provided them need to certified professionals.

Have you looked at having certificate part of the core extensions? do you think it should be developed separately?

Create extensions documentation website to:

• Render Sphinx directives on all (including community) extensions
• Build the docs of each version of each extension
• Render the information in extension.json (compatibility, etc.)

(Optionally, we can explore how to integrate GitHub issues for specific extensions to this website. However, I think this suggestion was made prior to issues being collected in ocds-extensions.)

Once this website exists, we can consider removing core extensions from the standard’s website – instead linking to this website – to avoid duplication.

It is missing a t, "documentaion_url": "https://github..."

We need a schema to document what each of the extension registry properties means

This might also belong in https://github.com/open-contracting/extension_tester

We've been thinking a bit about tests that could be useful, including:

(1) Does the extension provide an id property for each object it introduces (required vs. recommended?)

(2) Does the example data provided in the extension repo validate against the extension

(3) Does a schema with the extension provided still validate our sample data (we might need to create better sample data for this)

Changes to system of entries:

• Consider a single CSV for all entries, instead of directories of entry.json files (harder to navigate, review, analyze, etc.)
• Consider removing fields that are duplicating extension.json: description, documentationUrl, name, dependencies, compatibility (i.e. leaving category, core, url, slug, version)
• Based on the above, update entry-schema.json (which may be used to validate a CSV, like codelist-schema.json does), README.md, compile.py
• Update compile.py to validate uniqueness of "slug"
• Delete new.py

Additional changes to entries:

• Add enum for "category" in entry-schema.json (based on extensionlist directive) #99
• Add "downloadUrl" to entry-schema.json for link to ZIP file
• Add "author" and "authorEmail" #97

Changes to resources that depend on current registry layout:

• Update "Reviewing an extension registry entry" in https://docs.google.com/document/d/1bRhVVkuTPXw6acE2opKD-Yj80y0lvAsfAqDjG632nNY/edit

From #47

I don't know if it's too late to change this?

See comment "This loop is temporary only so docs work while transistioning onto new format". Also, not sure whether our intention is to continue to only collect the first entry in each entry.json during compilation.

It links to open-contracting/standard#377 when it should be https://github.com/open-contracting/ocds_budget_breakdown_extension

Can an extension go from inactive to active, or do extensions only ever get permanently 'retired' by being rendered inactive? Basically, in what cases would an extension be inactive?

Expanding on open-contracting/ocds-extensions#51

Hi!
Please considerer to add a new extension "DateCreated" to the registry.
This is the link:
https://chilecompracl.visualstudio.com/Documentaci%C3%B3n%20OCDS/_git/ocds_contractDateCreated_extension?path=%2FREADME.md&version=GBmaster
It was build to indicate the date of Contract creation because there may be a diference between the creation and the release date.
It was validate with @yolile of OCDS HelpDesk who's helping us in the implementacion of the standard in ChileCompra.

Thanks in advance!!

This spun off from #85

• Add to https://github.com/open-contracting/standard_extension_template/blob/master/extension.json and the appropriate JSON Schema. (This way extension authors can edit these values themselves - changes probably don't require approval.)
• Change compile scripts in this repository to read values and add them to generated output.
• Update documentation on creating new extensions

Should the format allow multiple authors, something like:

{
	"authors": [{
			"name": "James",
			"email": "[email protected] "
		},
		{
			"name": "Bob",
			"email": "[email protected]"
		}
	]
}

Fields fully optional so existing extensions aren't suddenly marked as broken. Also do we want to support people giving a name but no email?

• Sync each entry.json with referenced extension.json
• Sync entry-schema.json with extension-schema.json (mainly, the field order and constraints) and sync the README.md documentation of the two schemas
• Add this new script to the readme as a maintenance task (e.g. before compilation)

Common fields:

• compatibility
• name
• description
• documentationUrl
• dependencies
• version

entry-schema.json has the following fields which are not in extension-schema.json, because these should be determined by the registry, not by the extension (which can be a community extension):

• core
• category
• slug
• url (can technically be included in extension-schema.json as a self-referential URL)

extension-schema.json has the following fields which are not in entry-schema.json:

• codelists

From #47

Replaces the contents of each branch with what's in master, changing version numbers in url and documentationUrl where appropriate. We would thereafter only interact with branches through this build script.

Follow-up to open-contracting-extensions/public-private-partnerships#142 cc @duncandewhurst

The documentation url is currently https://raw.githubusercontent.com/open-contracting/ocds_budget_projects_extension/master/, which gives a 404 as it is missing the actual doc (eg, README.md). Anyway, it should be https://github.com/open-contracting/ocds_budget_projects_extension instead.

I assume we don't want all PR branches to get deployed…

• signatories: Change to "ppp" (currently "contract")
• performance_failures: Change to "ppp" (currently "implementation")

As discussed on #90, we are going to de-duplicate values between the extension_registry and the repository for each extension. Below is a report in the differences between the 2 sides (master branch in both cases).

The report covers name, description and documentation fields. We are however ignoring most differences in the documentation field as most of them are simply a "/" or a "/blob/master/README.md" at the end so fundamentally they point to the same place anyway. The one exception is below.

It is noted that in the past, for these 3 fields, the value from the Registry would have been used. Therefore it might be that the easy answer to this is "in each case, use the values from the registry". But this should be read to double check.

I can make the changes if someone confirms to me what changes should be made.

Id: signatories

Name Different!
Registry: Signatories
Extension: ocds_contract_signatories

Description Different!
Registry: Adds an array of organizationReferences to the contract section OCDS to support explicit declaration of the signatories for a contract.
Extension: Adds an array of OrganizationReference's to the contract section OCDS to support explicit declaration of the signatories for a contract.

Please make https://github.com/open-contracting/ocds_contract_signatories_extension/blob/master/extension.json have the correct value

Id: documentation

Name Different!
Registry: Documentation
Extension: Document Details

Description Different!
Registry: Extending the documents block with page numbers, access details and authorship information
Extension: This extension captures additional details about documentation, as identified in consultation for the OCDS PPP extension. 

Please make https://github.com/open-contracting/ocds_documentation_extension/blob/master/extension.json have the correct value

Id: transactions_relatedMilestones

Name Different!
Registry: Transactions - Related Milestones
Extension: ocds_transactions_relatedMilestone_extension

Please make https://github.com/open-contracting/ocds_transactions_relatedMilestone_extension/blob/master/extension.json have the correct value

Id: extendsContractID

Name Different!
Registry: Contract extensions via supplementary contract: extendsContractID
Extension: Contract extensions as contracts: extendsContractID

Please make https://github.com/open-contracting/ocds_extendsContractID_extension/blob/master/extension.json have the correct value

Id: requirements

Description Different!
Registry: The requirements extension is based on the EU's Core Criterion and Core Evidence Vocabulary (CCCEV) model for communicating criteria and responses. The extension is designed to allow procuring entities or buyers to express criteria, relating to either items being procured or bidders themselves, as structured data.
Extension: https://github.com/open-contracting/standard/issues/223

Please make https://github.com/open-contracting/ocds_requirements_extension/blob/master/extension.json have the correct value

Id: tariffs

Name Different!
Registry: Tariffs
Extension: ocds_tariffs_extension

Description Different!
Registry: Capturing details of tariffs, tolls and user fees associated with a contract.
Extension: The tariffs extension introduces support for disclosure of structured data on the user fees associated with a public private partnership project.

Please make https://github.com/open-contracting/ocds_tariffs_extension/blob/master/extension.json have the correct value

Id: performance_failures

Description Different!
Registry: This extension introduces a performance failures array to the implementation section of OCDS, based on the performance failures reporting table defined in the framework for disclosure in PPPs.
Extension: Adds fields to the implementation section to allow disclosure of an array of contracting performance failures. Based on the performance failures reporting table defined in the World Bank Framework for Disclosure in PPPs.

Please make https://github.com/open-contracting/ocds_performance_failures/blob/master/extension.json have the correct value

Id: additionalContactPoint

Description Different!
Registry: For including extra contact points on each organization. Particularly useful when each contact point deals only in a particular language or group of languages.
Extension: For including extra contact points on each organization. Particularly used when each contact point deals only in a particular language or group of languages.

Please make https://github.com/open-contracting/ocds_additionalContactPoints_extension/blob/master/extension.json have the correct value

Id: location

Description Different!
Registry: Allow the point of delivery for a given item or the site of works to be completed to be indicated in both the tender, award and contract stage.
Extension: Communicates the location of proposed or executed contract delivery.

Please make https://github.com/open-contracting/ocds_location_extension/blob/master/extension.json have the correct value

Id: risk_allocation

Name Different!
Registry: Risk Allocation
Extension: ocds-riskAllocation-extension

Please make https://github.com/open-contracting/ocds-riskAllocation-extension/blob/master/extension.json have the correct value

Id: qualification

Name Different!
Registry: Qualification
Extension: ocds_qualification_extension

Description Different!
Registry: For providing information on a pre-qualification stage.
Extension: Draft extension for PPPs to add a qualification section between planning and tender. Currently this is another tender object. See discussion in https://github.com/open-contracting/public-private-partnerships/issues/36

Please make https://github.com/open-contracting/ocds_qualification_extension/blob/master/extension.json have the correct value

Id: partyScale

Description Different!
Registry: For classifying organisations as micro, sme or large.
Extension: For classifying organizations as micro, sme or large.

Please make https://github.com/open-contracting/ocds_partyDetails_scale_extension/blob/master/extension.json have the correct value

Id: charges

Description Different!
Registry: The charges extension is used to record details of the total charges that are estimated or applied to users or government during the operation of a Public Private Partnership contract. 
Extension: The charges extension is used to record details of the **total** charges that are estimated or applied to users or government during the operation of a Public Private Partnership contract.

Please make https://github.com/open-contracting/ocds_charges_extension/blob/master/extension.json have the correct value

Id: shareholders

Name Different!
Registry: Shareholders
Extension: ocds-shareholders-extension

Description Different!
Registry: Draft shareholders extension for ppp extension
Extension: The shareholders extension can be used to attach company ownership details to each party in the parties array.

Please make https://github.com/open-contracting/ocds-shareholders-extension/blob/master/extension.json have the correct value

Id: budgets_and_projects

Name Different!
Registry: Budget and projects
Extension: Budget and Project Updates

Description Different!
Registry: This extension separates planning information on budgets and projects, in line with a proposal for OCDS upgrades, and to support the PPP extension.
Extension: This extension introduces a projects block in 'planning' which can be used to give total project value, and provide project-level information, alongside existing fields in the planning/budget section for details of the budget allocated to this specific contracting process.

Please make https://github.com/open-contracting/ocds_budget_projects_extension/blob/master/extension.json have the correct value

Id: metrics

Description Different!
Registry: The metrics extension supports publication of forecasts (planning stage), targets (tender stage), agreed performance targets (award and contract stage) and results (implementation stage)..
Extension: The metrics extension supports publication of forecasts (planning stage), targets (tender stage), agreed performance targets (award and contract stage) and results (implementation stage).

Please make https://github.com/open-contracting/ocds_metrics_extension/blob/master/extension.json have the correct value

Id: ppp

Name Different!
Registry: PPP Extension
Extension: OCDS for PPPs (Alpha)

Description Different!
Registry: This extension collects together a number of schema additions required by the OCDS for PPPs implementation profile. Some of these may later be factored out into their own extesions.
Extension: This extension collects together a number of schema additions required by the OCDS for PPPs implementation profile. Some of these may later be factored out into their own extensions.

documentationUrl Different!
Registry: http://ocds-for-ppps.readthedocs.io/
Extension: http://standard.open-contracting.org/profiles/ppp/latest/en/

Please make https://github.com/open-contracting/public-private-partnerships/blob/master/extension.json have the correct value

Id: buyer_per_contract

Description Different!
Registry: In some cases, a single contracting process involves multiple buyers (for example, a framework that different organisations can buy from). In these cases, the multiple buyers extension can be used to specify a distinct buyer for each contract.
Extension: In some cases, a single contracting process involves multiple buyers (for example, a framework that different organizations can buy from). In these cases, the multiple buyers extension can be used to specify a distinct buyer for each contract. 

Please make https://github.com/open-contracting/ocds_multiple_buyers_extension/blob/master/extension.json have the correct value

Id: finance

Name Different!
Registry: Finance
Extension: Financing

Please make https://github.com/open-contracting/ocds_finance_extension/blob/master/extension.json have the correct value

Id: budget_breakdown

Name Different!
Registry: Budget breakdown
Extension: Budget Breakdown

Description Different!
Registry: For providing a detailed budget breakdown.
Extension: Providing a breakdown of budgets by year and source

Please make https://github.com/open-contracting/ocds_budget_breakdown_extension/blob/master/extension.json have the correct value

This was suggested by @kindly. Should the name of the file for the extension be added to the url in registry/extension_name/extension.json, so that instead of this:

https://raw.githubusercontent.com/open-contracting/ocds_charges_extension/master/

we will have this?:

https://raw.githubusercontent.com/open-contracting/ocds_charges_extension/master/extension.json

Notice that https://raw.githubusercontent.com/open-contracting/ocds_charges_extension/master/ is in fact broken as no document is specified at that location.

Party Scale and scale are the same extension under different names in the registry

The standard (and ppp) repo has:

addons:
  apt:
    packages:
    - lftp

Because its deploy script uses ltfp.

However, this repo doesn't have that block in .travis.yml, despite using lftp in its deploy script.

I think we should add that block for consistency. It's curious that deployment works – Travis doesn't list lftp as a default package.

The following would be appropriate for OCDS 1.2.

Issues

• How to handle multiple versions of extensions, e.g. package management, dependency issues
• How to publish both 1.1 and 1.2 versions of an extension (not just in different branches of the registry)

Possible solutions

• Add version number to extension.json files #31

Notes

• Right now, we use branches of extension_registry for generating the documentation correctly for each version of OCDS. If the registry lived outside the main OCDS documentation, we wouldn't need to carefully tag and branch as much.
• On the call we discussed some timing issues around updating CoVE and extension_registry at the same time. extension_registry has a new format for extensions.json (and extensions.js), but we can't change those files without breaking other things: standard docs, extension_tester, ppp (apply_extensions.py). CoVE can already support the new and old formats. No one uses these files outside OCP/ODS (except possibly Development Gateway).

@kindly @edugomez @duncandewhurst

From #90:

@jpmckinney

The new extensions.csv works for repositories whose releases (versions) can be determined via the version control system. However, extensions are allowed to be uploaded to e.g. an FTP server (imagine a folder of ZIP files for extensionA.1.0.zip, extensionA.1.1.zip, extensionB.1.1.zip, etc.) One solution is for each row of the CSV to be a release, rather than an extension; the URL would have to point to the specific release, and a Version column would have to be added for the version number. This would cause repetition of Id, Category, and Core across releases of a single extension. To address that, we can either add a consistency test, or have Id, Category and Core in one file, and Id, URL and Version in another file. The first file would be managed by standard authors who decide the Category and Core, whereas the management of the second file could be more distributed, as it will pretty much only be appended to via PR by extension authors.

The new output files have columns for each release of the standard for which there are extensions released. While core extensions follow the standard's versioning, community extensions can use any version numbers they like. They can have a 1.1.3-alpha, 1.1.3-rc.1, 5.3, 11 – anything, really. So, we may need to, as above, have one release per row. This will duplicate some information, but as this file is generated, that duplication won't come with data integrity risks.

@odscjames

I would favour the second file approach - have a “extensionversions.csv” with columns like Id, Version, Reference.

I think the problem you identify of repetition is an important one to avoid. Also this way, the number of columns that might not be relevant is reduced which makes for files that are easier to edit. And as you note, different people will tend to be editing different files so that cuts down on possible conflicts.

Inactive (but used by PPP):

• https://github.com/open-contracting/ocds_documentation_extension

Not in registry:

• https://github.com/open-contracting/api_extension

Why is this the case?

Cleanup of above:

• Markdown linting
• adding a .travis.yml file
• adding an Apache license file
• adding an Issues section to the readme

Done

Not in registry:

• https://github.com/open-contracting/ocds_parties_extension
• https://github.com/open-contracting/ocds_related_process_extension
• https://github.com/open-contracting/ocds_socialwitness_extension

Links to extension repositories and some extension documentation on the community extensions page of the OCDS V1.1 documentation lead to a 400: Invalid request error.

Fixing this looks like a case of updating the entries in the extension registry from links such as:

https://raw.githubusercontent.com/open-contracting/ocds_budget_projects_extension/master/

to

https://github.com/open-contracting/ocds_budget_projects_extension

Just checking that these aren't specified as "raw.githubusercontent.com" type links for a reason, before I go through and update them?

This would allow a front-end application to compare between the pinned version of an extension they are using, and the latest version to see if more up-to-date versions are available.

Can we add https://github.com/open-contracting-extensions/ocds_contractRegister_extension to the extension with category 'implementation'.

If required, we can make a 1.0 release of the extension for this after the pull request at open-contracting-extensions/ocds_contract_completion_extension#5 is merged.

Once the work on declaring the consolidated PPP extension is complete, publishers will declare the PPP profile using the following URL in the extensions key in the release package metadata:

http://standard.open-contracting.org/profiles/ppp/{{version}}/en/_static/consolidatedExtension/extension.json

In view of the changes to the extension registry:

1. Will it be possible to register the extension at a non-Github URL?
2. Does the PPP build need to create a ZIP of the extension?

It says open-contracting-extensions/public-private-partnerships#36 when it should be https://github.com/open-contracting/ocds_qualification_extension

https://github.com/dgcpcompraspublicas/ocds_partyDetails_gender_extension

As we're maintaining multiple branches that should only differ by version number, it becomes very demanding to keep them in sync if we include scripts, documentation, JSON Schema, etc.

FYI @timgdavies

{"id": "http://example.com", ...

... completely seperate from the version of the OCDS standard
... compatable with (use compatability ...

Remove extension_registry_git_ref from:

• sphinxcontrib-opencontracting
• documentation-support
• standard_profile_template
• standard
• profiles

Update the Handbook's instructions for:

• profiles
• releases of core extensions

If we were to stick to stick to the concept of extensions as re-usable blocks... does it make sense to try to find another technical term for describing random/additional data fields without a theme?

The suggestions i have so far instead of "OCDS extension" are:

• "OCDS country add-on" (which can be translated into what has been added to the OCDS from a country)
• "OCDS country module" (each data in the end needs to be displayed and processed... usually it would entail adding a specific module to an application that uses OCDS data... so we could just go with module)

Do you think it is worth it to have a brainstorming sessions on terminology for this? does this overcomplicate things? should we just stick to extensions even if they are random fields and not really re-usable?

At the moment if you want to use bids extension (https://github.com/open-contracting/ocds_bid_extension) you also need to declare requirements extension (https://github.com/open-contracting/ocds_requirements_extension), however this dependency appears to be a bug and should be removed.