open-contracting / european-union-support Goto Github PK
View Code? Open in Web Editor NEWSupport scripts for TED mapping
License: BSD 3-Clause "New" or "Revised" License
Support scripts for TED mapping
License: BSD 3-Clause "New" or "Revised" License
Definition:
The criteria (or criterion) will (only) be used to select the candidates to be invited for the second stage of the procedure (if a maximum number of candidates was set).
Shall I understand it as:
The criteria or criterion will only be used to select the candidates to be invited for the second stage of the procedure if a maximum number of candidates was set. (all parenthesis removed)
or:
The criteria or criterion will be used to select the candidates to be invited for the second stage of the procedure, and possibly for the rest of the procedure. If a maximum number of candidates was set, they will only be used to select the candidates to be invited for the second stage.
or something else?
Any idea @JachymHercher?
In F02 "Contract notice", the guidance for IV.2.1 to link to the previous release(s) states:
Discard. The previous publication concerning this procedure is the OCDS release with the same ocid as this release and with the nearest earlier date to this release.
Problem: a Prior Information Notice can contain several contracts, each one generating a different ocid
. There is apparently no way from F02 to determine which ocid
to link back to.
I also don't know how we determine the ocid
in F02.
Until now, in the mapping files, I have consistently expressed boolean values as 'true' or 'false', using single quotes.
However, `true` and `false`, with back ticks, are more appropriate.
BT-04 (Procedure identifier)
The European Public Procurement Procedure Identifier, a unique identifier of a procurement procedure. Including this identifier in all published versions of this notice (e.g. published on TED, national publication portals, regional publication portals) allows unique identification of procurement procedures around the EU.
This BT is not present at planning stage (notices 1-9) and only appears from the tendering stage. That means we can't add it in the ocid from the start, unless the procedure starts with a tender notice.
On solution could be to have two ocid: one for the planning stage and one for the rest of the procedure. They would be linked via .relatedProcesses
. However, as there can also be a process split between the tender and the award stages (certain lots having their own procedure), we may end up with data that is hard to analyse.
BT-651 "Subcontracting Tender Indication" is an array of codes (values).
Description:
The information about subcontracting that must be indicated in the tender.
In OCDS, the subcontracting extension has tender.subcontracting.description
, but it matches better with BT-554 Subcontracting Description.
We could cram BT-651 Subcontracting Tender Indication in tender.subcontracting.description
, asking implementers to append the label or labels from the codelist.
Or we add a new field to the subcontracting extension.
Related to #75
Currently, the guidance CSV only has information down to the BT-notice number pair (BT, notice number, notice type, BT description, datatype + standard forms information).
Since many developers probably rely on Xpath in their code, we should add eForms xpath to the guidance CSV.
We must keep in mind that previous study of the xpath-BT cardinality has revealed that BTs can, on average, be bound to many different Xpaths. This high cardinality of xpaths per BT is mainly due to the duplication of xpaths that describe organisations. A few corner cases are due to data modelling decisions, and seldom exceed 4 xpath per BT.
Definition:
Whether tenderer-related information can be supplemented even after the submission deadline.
Basically, whether the information of a tenderer can me modified or added after the submission deadline/tendering period.
I'm not sure whether an existing field would fit (submissionMethodDetails
?), or if we need a new one.
It's bound to a short codelist.
A lot of BT that can describe the whole procedure can also describe individual lots. This is very much visible in the XML, where the element that describes the general information of a procedure (cac:ProcurementProject
) is also a child of cac:ProcurementProjectLot
that describes a lot, bringing all its own children elements.
Rough XML structure of an eForms document:
<[form type]>
<cac:ProcurementProject>
general information on the procedure...
</cac:ProcurementProject>
<cac:ProcurementProjectLot>
<cac:ProcurementProject>
general information on lot 1...
</cac:ProcurementProject>
other information on lot 1
</cac:ProcurementProjectLot>
<cac:ProcurementProjectLot>
<cac:ProcurementProject>
general information on lot 2...
</cac:ProcurementProject>
other information on lot 2
</cac:ProcurementProjectLot>
...
</[form type]>
For example, the two possible xpaths for BT-24 (Description) and their respective mapping in OCDS:
/*/cac:ProcurementProject/cbc:Description
=> tender.description
/*/cac:ProcurementProjectLot/cac:ProcurementProject/cbc:Description
=> Lot.description
From a quick scan in the mapping data, BT that have an xpath that starts with/*/cac:ProcurementProject
almost always have also an xpath that starts with /*/cac:ProcurementProjectLot/cac:ProcurementProject
. On the other hand, many BT only have an xpath that starts with /*/cac:ProcurementProjectLot
, meaning the XML schemas of eForms are rather centered on lots.
@jpmckinney What's the status of the thinking around the extension of the Lot object in OCDS? I think we're going to need it for data such as:
Belarus draft data includes an additional productCountry
field in contract/items
for the country where goods were created. Helpdesk feedback suggested renaming to countryOfOrigin
.
eForms enables the publication of statistics through the BT pair BT-759 Received Submissions Count / BT-760 Received Submissions Type (nb of submissions, type of submissions (codelist)).
Example:
<efac:ReceivedSubmissionsStatistics>
<efbc:StatisticsCode listName="received-submission-type">tenders</efbc:StatisticsCode>
<efbc:StatisticsNumeric>5</efbc:StatisticsNumeric>
</efac:ReceivedSubmissionsStatistics>
<efac:ReceivedSubmissionsStatistics>
<efbc:StatisticsCode listName="received-submission-type">t-sme</efbc:StatisticsCode>
<efbc:StatisticsNumeric>4</efbc:StatisticsNumeric>
</efac:ReceivedSubmissionsStatistics>
<efac:ReceivedSubmissionsStatistics>
<efbc:StatisticsCode listName="received-submission-type">t-oth-eea</efbc:StatisticsCode>
<efbc:StatisticsNumeric>0</efbc:StatisticsNumeric>
</efac:ReceivedSubmissionsStatistics>
<efac:ReceivedSubmissionsStatistics>
<efbc:StatisticsCode listName="received-submission-type">t-no-eea</efbc:StatisticsCode>
<efbc:StatisticsNumeric>0</efbc:StatisticsNumeric>
</efac:ReceivedSubmissionsStatistics>
<efac:ReceivedSubmissionsStatistics>
<efbc:StatisticsCode listName="received-submission-type">t-esubm</efbc:StatisticsCode>
<efbc:StatisticsNumeric>5</efbc:StatisticsNumeric>
</efac:ReceivedSubmissionsStatistics>
The OCDS bid extension has codes for most of the eForms codes, but the following eForms codes have no straight match yet (current mapping table):
For the micro/small/med codes, we could suggest implementers that they sum them up and add them as smeBids
in OCDS.
label:reverse
to extract label keyslabel:copy
task to copy across to F17-F18Standard forms contained information that don't explicitly exist anymore in eForms, such as (not exhaustive):
/*/ext:UBLExtensions/ext:UBLExtension/ext:ExtensionContent/efext:EformsExtension/efac:NoticeSubType/cbc:SubTypeCode
). Providing a mapping table (each notice is bound to a single directive) would enable the extraction of this information.Since we have an OCDS data structure to receive this information, do we provide guidance to help implementers extract this information from the eForms XML, in spite of the fact it's not explicitly stored in the XML?
Potential resources on this topic:
EU green public procurement criteria: https://ec.europa.eu/environment/gpp/eu_gpp_criteria_en.htm
EU library of use cases: https://ec.europa.eu/environment/gpp/case_group_en.htm
In output/mapping/eForms/BT-sfLevel.csv, certain rows have several notice numbers, such as 01,02,03
, but we want granular rows so that we can then add form types based on single notice numbers. This way, we also enable the publication of guidance per notice number.
We consequently want to explode those rows rows in as many rows.
This BT is bound the Main activity code list.
This list contains many values that:
@JachymHercher Is it safe to assume that only the COFOG values will be used with BT-10, since it applies to public organizations? That would simplify the guidance as I'm not sure what sort of code scheme the other values come from.
Merci !
In F01, II.2.3 "Place of performance" can be repeated, but only one .deliveryAddress
is allowed in the Location extension.
Shall we add a .deliveryAddresses
array in Location or a EU specific extension?
The Standard Form Level is the shared key we will use to script the mapping the existing SF guidance with the eForms Xpath. In order to anticipate mapping duplicates, it would be good to see if we have non-1:1 relationships in the mapping chain:
OCDS guidance for SF => SF Xpath => SF level => eForms BT => eForms Xpath => OCDS guidance for eForms
eForms adds more details to the prizes that are awarded at a contest. The BTs are grouped in BG-44 Prize.
From official examples:
<cac:AwardingTerms>
<cac:Prize>
<cbc:RankCode>1</cbc:RankCode>
<cbc:ValueAmount currencyID="EUR">10000</cbc:ValueAmount>
<cbc:Description languageID="FRA">Une indemnité forfaitaire d'un montant identique à celui versé aux concurrents non retenus sera payée à l'équipe lauréate sous forme d'avance sur le marché de maîtrise d'œuvre dès l'issue du concours; cette somme étant à valoir sur le montant des honoraires à percevoir ultérieurement au titre du marché de maîtrise d'œuvre.</cbc:Description>
</cac:Prize>
<cac:Prize>
<cbc:RankCode>>= 1</cbc:RankCode>
<cbc:ValueAmount currencyID="EUR">10000</cbc:ValueAmount>
<cbc:Description languageID="FRA">Les trois concurrents (candidats admis à concourir) recevront une indemnisation forfaitaire maximale de 10 000 EUR HT. Pour les prestations fournies, sous réserve de la recevabilité de leurs prestations au regard du règlement du concours et du respect du programme. Les indemnisations sont forfaitaires, conformément aux dispositions de l'article R. 2172-4 du code de la commande publique, l'acheteur, sur proposition du jury, se réserve le droit, dans le cas d'un projet qu'il jugerait incomplet ou dont les prestations seraient non conformes par rapport au règlement de concours et ou au programme, de supprimer totalement ou partiellement l'indemnité.</cbc:Description>
</cac:Prize>
</cac:AwardingTerms>
Currently, the design contest extension only has .prizes.description
. We may need an array of Prize objects.
Figuring out the structure of an eForms XML document and how the data is distributed into XML blocks may help structuring the guidance in a handy way for implementers.
I'll use the can_24_FRA_comments.xml example document from the eForms-SDK.
In XPATHs provisional release v. 1.0.docx, the following BT-Xpath mappings have unclear semantics:
/*/cac:ProcurementProjectLot/cac:TenderingTerms/cac:TenderRecipientParty/cbc:EndpointID
=> BT-18 (BT-509)/*/cac:ProcurementProjectLot/cac:ProcurementProject/cac:ContractExtension/cbc:OptionsDescription
=> BT-54 (BT-53)We can assume that the presence of BT-54 (Options Description) and the corresponding Xpath imply that BT-53 (Options), of datatype indicator, is true
. If cbc:OptionsDescription
is absent, BT-53 is false
. In this case, BT-53 wouldn't have its own Xpath. However, other BT such as BT-94 (Recurrence) are indicators and have their own Xpath (/*/cac:ProcurementProjectLot/cac:TenderingTerms/cbc:RecurringProcurementIndicator
).
Throughout this issue, comments for which the relevant updates to guidance.yaml
are done are marked with a ✔️
The same way we did for standard forms, in order to avoid repetitions, we need to document the guidance that applies to any eForms document.
The schema of this extension has changed a lot and the corresponding mappings must be updated.
eForms has a code list for organization roles and subroles. On top of these roles, buyers can be (or not) 'contracting entity' via the BT-740 (Buyer Contracting Entity) boolean.
Definition of 'buyer' in the eForms regulation:
Buyer refers to a contracting authority, a contracting entity, a defence contractor, an international organisation, or an organisation awarding a contract subsidized by a contracting authority; unless the above mentioned are an association of organisations that is not an organisation in itself, in which case each individual organisation is considered a 'buyer'.
Definition of 'contracting authorities' in the directive 2014/24/EU:
contracting authorities means the State, regional or local authorities, bodies governed by public law or associations formed by one or more such authorities or one or more such bodies governed by public law.
I couldn't find an official definition of 'contracting entity'.
'contracting entity' is not mentioned in related issues (open-contracting/standard#825, open-contracting/standard#571)
I'm not sure how to tie BT-740 back to OCDS roles (buyer, procuringEntity):
False
)?True
)?Any idea @JachymHercher?
Definition:
Advanced or qualified electronic signature or seal (as defined in Regulation (EU) No 910/2014) is required.
@JachymHercher I'm not sure what this BT is about: is it the regular electronic signature that applies in virtually any electronic procedure, or is it something more advanced?
In a nutshell:
electronicSignature: true/false
, or would it be too broad?BT-17 Submission Electronic: required
necessarily imply that BT-744 Submission Electronic Signature: true
?eForms uses xsd:date
datatype to express dates that don't require a time. According to the XML Schema recommendation, an xsd:date
is the date portion of an xsd:dateTime
object, with the possibility to add an indication of the timezone.
Valid examples of xsd:date
:
Question: is this datatype fully compatible with OCDS date, or does a time portion must be added? I see that in the EU standard forms guidance dates (e.g. "20191127") are mapped as-is, without conversion.
Ideally we should require a conversion of the dates both in eForms and SF guidance so that they match the OCDS date format: YYYY-MM-DDThh:mm:ssTZ
What do you think @jpmckinney?
For the description of the procedure in standard forms, we wrote the following guidance:
Map to the lot's
.description
F12, F13: Instead, map totender.description
@jpmckinney Do you remember why design contests (notice and result notice) had special guidance in this case? They are not alotted?
In OCDS we support the following procurement categories:
With the possibility to add additional procurement categories via an open codelist.
In eForms, they support the same values + 'combined' ("Provision of any combination of services, supplies and/or works").
How do we reflect that in the mapping?
Can we leave tender.mainProcurementCategory
empty and
tender.additionalProcurementCategories
?tender.additionalProcurementCategories
?I think I saw somewhere that there were discussions around a 'combined' option in OCDS.
@jpmckinney The European Commission has updated the TED XML schema with a few new elements https://webgate.ec.europa.eu/fpfis/wikis/pages/viewpage.action?pageId=876970615
Quoting the blog post:
These 4 elements have been added to R2.0.9.S05:
There will be no change to the R.2.0.8 (defence).
Do we intend to stick to the latest release of the TED XML schemas?
@BenCCS Does your team plan to make use of these new elements?
@JachymHercher The description of eDelivery Gateway URL in the annex to the eForms regulation:
The organisation's uniform resource locator for exchange of data and documents.
It's not the buyer's profile, it's an interface to exchange documents securely using eID (part of the CEF infrastructure):
eDelivery is a building block that provides technical specifications and standards, installable software and ancillary services to allow projects to create a network of nodes for secure digital data exchange. By building with eDelivery, public and private organisations from different sectors can easily create a safe and interoperable channel to transfer documents and data among each other over a public or private network.
(https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/eDelivery)
Do you think the eDelivery system is specific to Europe for document interchange? If specific, I would add and document it in the EU OCDS extension. Otherwise, probably in the Communication extension.
Thanks!
We should clarify in guidance how groups of lots translate from eForms to OCDS.
Groups of lots exist in eForms to:
In order to track the progress of the mapping, we need the following stats:
Hi, I have the XML files for the Norwegian Doffin. It's the same as TED. And I well make JSON files of it.
I'm trying to run code with ours data (not mine) but i dont find the docs folder
for i in 01 02 03 04 05 06 07 08 12 13 14 15 20 21 22 23 24 25; rake table LANGUAGE=EN FILES=F$i > path/to/european-union/docs/forms/F$i.md; end
for i in 01 02; rake table LANGUAGE=EN FILES=MOVE FORM=T$i > path/to/european-union/docs/forms/T$i.md; end
When i run
for i in 01 02 03 04 05 06 07 08 12 13 14 15 20 21 22 23 24 25; rake table LANGUAGE=EN FILES=F$i > /home/bjorn/source/european-union-support/output/content/F$i.md; end
I get
index of key in data:
6
omit: icar_noticeref
ignore: internets
enumerations: maintype_ministry
additional: false
rake aborted!
unexpected key: nutscode
/home/bjorn/source/european-union-support/tasks/table.rake:210:in block (2 levels) in <top (required)>' /home/bjorn/source/european-union-support/tasks/table.rake:59:in
each'
/home/bjorn/source/european-union-support/tasks/table.rake:59:in `block in <top (required)>'
Tasks: TOP => table
(See full trace by running task with --trace)
Not needed for this project, but may be valuable to the community.
The initial mapping stats indicate only 61 eForms Xpath don't have OCDS guidance. That's surprisingly low given the number of new terms introduced in eForms.
The following link is used in the documentation (e.g. F14) : https://op.europa.eu/en/web/eu-vocabularies/tedschemas, but it's broken.
Here is the right one: https://op.europa.eu/en/web/eu-vocabularies/e-procurement/tedschemas
I believe that we haven't really mapped VI.3 "Additional description". tender.description
is mapped to something else.
In eForms there is a BG "Additional information", with an "Additional information" field.
F23 and F25 have fields that look similar:
/AWARD_CONTRACT/AWARDED_CONTRACT/INFO_ADD_VALUE
), "Any other details relevant to the value of the concession accord to Art. 8(3) of the directive" (this section is about the method to calculate the value of a concession)/OBJECT_CONTRACT/CALCULATION_METHOD
) "Method used for calculating the estimated value of the concession"I've mapped CALCULATION_METHOD
to award.concession.valueCalculationMethod
(the name may be improved), and I wonder whether INFO_ADD_VALUE
could also map to it. They seem semantically very close.
Could both F23 and F25 be filled in a concession procedure, and could INFO_ADD_VALUE
and CALCULATION_METHOD
clash?
Definition:
Information about a number linked to the selection criteria (or criterion) used for selecting the candidates to be invited for the second stage of the procedure.
It contains the following BTs:
This convoluted, multi-purpose data structure enables the expression of any numeric feature of selection criteria.
These semantics could be added to the selection criteria extension. Do we keep the same structure? It would ease the eForms mapping, but might not work for non-EU procedures.
In standard forms, we mapped the information about weight to the criterion's .description
.
For the standard forms, for each form, we have used the Xpath as the unit of guidance:
Level | Xpath | Guidance |
---|---|---|
VI.4.2 | Body responsible for mediation procedures /COMPLEMENTARY_INFO/ADDRESS_MEDIATION_BODY |
Add a party, and add ‘mediationBody’ to its .roles |
For the eforms guidance, for each form type (Planning, Result, etc.), I'm leaning toward using the BT as the unit of guidance and group the matching Xpath with it:
BT | Xpath | Guidance |
---|---|---|
BT-510 | Street name//cac:PostalAddress/cbc:StreetName (#49 (comment)) |
Map to the party's .address.streetAddress |
Most BT have a single Xpath, and multi-Xpath BTs seem to have a logic that deserve to be treated as a whole. The information we currently doesn't tell us whether a given BT can have a different logic from a form type or a notice to another.
eForms introduces new roles (I have crossed the roles that are not new):
This topic isn't new: open-contracting/standard#1180 cc @JachymHercher
BT-65 "Subcontracting Tender Obligation" is an array of codes (values).
Description:
An obligation to be met by the tenderer concerning subcontracting.
In OCDS, the subcontracting extension has tender.subcontracting.description
, but it rather matches better with BT-554 Subcontracting Description.
We could cram BT-65 Subcontracting Tender Obligation in tender.subcontracting.description
, asking implementers to append the label or labels from the codelist.
Or we add a new field to the subcontracting extension.
Related to #74
The BidsStatistics
may be used to publish statistics about amount of money. In that case, the value
field must be completed with an optional currency
field (see II.1.7 "Currency" in F03).
In eForms, organization have a main role that is similar to OCDS roles (winner, buyer, etc.) and it can be refined with a subrole via BT-770. Possible values:
We could create an OCDS extension that adds a .subrole
array field to the Party object, with an open codelist including the codes above (rephrased with OCDS terms).
On top of the duration in days/months/years and the start and end dates, eForms offers the possibility to express duration according to a wide code list. OCDS has no equivalent.
Users have the possibility to use this field as a replacement for the other types of durations.
Since it's tightly bound to a EU code list, I suggest we add it to the EU extension.
BT-1252 Direct Award Justification Previous Procedure Identifier enables to link a procedure that justifies the use of a direct procedure.
An identifier of a previous procedure that justifies the use of a procedure which allows directly awarding contracts, i.e. justifying a procedure that does not require publishing a call for competition in the Official Journal of the European Union.
This BT would typically work well with the tender.relatedProcesses
array, but I'm not sure the existing codes fit the bill.
Is a new code necessary?
Currently, when the guidance has several steps, it looks like this:
"Add 'electronicSubmission' to
tender.submissionMethod
, and map totender.submissionMethodDetails
If to the following address is selected, this results in a loss of structure. (WARNING #27)"
To improve readability and make the separation of steps more systematic (no hesitation between commas or line breaks), we could consistently use line breaks and number the steps. The numbering should be automatic and not require renumbering when we remove an intermediate step.
The steps above would look like this:
"Add 'electronicSubmission' to
tender.submissionMethod
Map to
tender.submissionMethodDetails
f to the following address is selected, this results in a loss of structure. (WARNING #27)"
Although we have created many extensions to cover the fields of standard forms, eForms introduces fields that require more fields to be created, and possibly new extensions:
.languages
array.)deliveryLocation.description
.communication.invitationToConfirmInterest.startDate
.nonElectronicRationale
in Submission terms.multipleTendersPolicy
in Submission termsWhen guidance is validated for a specific context (eForms xpath + BT + ?), it should spread to all other rows that match the same context, and have these rows tagged as validated.
To do:
*
to reduce repetition.Miscellaneous
black
on Python scriptsI'm also putting notes from a call with Colin here, in case relevant in future.
eForms documentation
fields/fields.json
was added more recently and might be useful.examples/notices
, can_24
means "Contract Award Notice EU/2014/24".output/mapping
directory
eforms-guidance.json
, each JSON element is a form-BT pair.
status
values can be:
comments
.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.