Open-source description language for privacy to declare data types and data behaviors in your tech stack in order to simplify data privacy globally. Supports GDPR, CCPA, LGPD and ISO 19944.
Home Page: https://ethyca.github.io/fideslang
License: Creative Commons Attribution 4.0 International
Although users have generally not complained about this, I think most don’t realize that the account category exists and why it should be used. The intent of account is to handle the common case where data used to manage a user’s account (e.g. an email address used to sign-in) may need to be processed differently than data used for non-account purposes (e.g. an email address used to contact them for support).
However, this distinction seems to not be intuitive to early users, and in fact the lack of questions about it seems to imply that it’s just not noticed! Therefore, it might be more intuitive and useful if we had two sibling subcategories, e.g. ...account.email and ...contact.email for this purpose.
Alternatively, it may be easier to remove it entirely. The distinction between account and contact, for example, might be better modeled by applying two different data_use annotations (e.g. provide vs. improve).
Now, we have data categories like:
After, the account category would be moved to a subcategory of user:
Alternatively, the account category could be deleted entirely and migrated to use user.provided.identifiable.contact equivalents instead.
A description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
As a data protection officer, I would like to be able to see the data owner. On the mouse over event, include the person or position information. The field must be of type string allowing the insertion of certain names such as:
John Ave
just name
Chief Finance Officer
just role
John Ave <[email protected]>
name + email
Chief Finance Officer <[email protected]>
role + email
Data Categories, Data Uses, Data Subjects, Data Qualifiers must contain owners. All levels and categories must have owners.
The filling of the field is optional and when it is empty, it should not appear.
As Is:
To Be:
Because this functionality is optional, backwards compatibility will be possible. I would like to start the conversation about technical debt and how this functionality will be implemented.
See ya
matches on the policy page has as typo causing it to be displayed as matches enum  r
Add any other context about the problem here.
Ethyca internal requirements doc: https://ethyca.atlassian.net/wiki/spaces/PM/pages/2351661057/Data+Map+UI+-+Data+Source+and+Lineage
An accompanying ticket to ethyca/fides#654, we are looking to extend fideslang to include the third_country_transfers attribute.
TBD, need to determine how we want this to work in ethyca/fides#654, and will add additional details here
We're adding as much documentation as we possible can to the pydantic models themselves for fideslang, and it's only going to get harder and more cumbersome to try to manage updating two separate places whenever we make a fideslang model change.
I'd like each of the data types to have a nice markdown table generated purely from the pydantic sauce and included as part of our docs
Using some kind of automated docs builder, it may exist
Users should be able to build a taxonomy that includes dbt models as datasets
There should be a flag/config value that allows the user to specify that they would like to load datasets from dbt model files and include them as part of the taxonomy.
The alternative would be to generate static fides dataset files from the dbt files, but this seems redundant. And would leave the potential for the files to get out of sync. If we always parse from the source dbt files, we can never get out of sync.
This step needs to happen at Taxonomy parse time and will probably exist as part of fideslang.
Currently in fidesctl, the human-readable column names for a data map are defined in a constant. This seems to be misaligned with the rest of our codebase which relies heavily on pydantic models to define and maintain structure and attribution (i.e. descriptions) of data across the fides ecosystem.
Add human-readable values, utilizing the alias attribute, to be used by other fides products.
The use case that inspires this issue built a constant (dict) with Pandas DataFrame columns mapped to a human-readable equivalent, which is currently passed as the first set of records to a data map item. See fidesctl#779
This could yet be challenging due to some required separation when using a DataFrame to join resources to construct data map items.
Creating at least one privacy declaration without a name causes validation errors
In a python repl...
>>> from fideslang import System, PrivacyDeclaration
>>> p = PrivacyDeclaration(data_categories=[], data_use="test", data_subjects=[])
>>> p2 = PrivacyDeclaration(data_categories=[], data_use="test", data_subjects=[])
>>> s = System(fides_key="s", system_type="s" privacy_declarations=[p, p2])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for System
privacy_declarations
'<' not supported between instances of 'NoneType' and 'NoneType' (type=type_error)
Doing just one privacy declaration works okay, perhaps because this sort doesn't need to compare
Be able to make multiple privacy declarations without names without problems
If applicable, add screenshots to help explain your problem.
There's a number of ways we could fix this, considering name is being deprecated:
data_use insteadname to ""I'm not sure what the best way is, open to suggestions!
Now that fideslang lives here full-time, the docs probably need to be revisited and updated to match the true state of fideslang.
There might be some useful docs still in fidesctl for fideslang, but they might be gone by the team this ticket gets worked on so git history will come in handy
These changes were also recently made to Fideslang and published as version 1.1.0 - https://ethyca.atlassian.net/wiki/spaces/EN/pages/2345599010/Fideslang+1.1.0+Planning
The ticket for that work is here: https://github.com/ethyca/fideslang/issues/58
The history of the data_files/ directory suggests that it is quite outdated and could lead to confusion. These should likely be removed or at least re-generated if still required
n/a
fideslang to be imported or viewed from the documentation website
n/a
Found as part of #62
Extending taxonomy to adequately describe controller/processor of data types for a given system's behaviors relative to data processing.
Two prior approaches have been investigated:
Requires additional evaluation after next major release on 11/3/2021
Originally posted by JasonMWhite October 26, 2021
A common issue that we face is the need to know which data we hold as a controller vs as a processor. Depending on the role in which we are acting, we can have different requirements for retention and permitted uses.
This can be more than a simple controller/processor split. There are also situations where we could be a co-controller, or there could be several independent controllers.
Have you thought of how you might extend the data subject taxonomy to include a controllership dimension?
We want to enable dbt model parsing, and having a config section where values could be configured would make this much easier.
A new section, similar to the existing cli, api, etc. that allows us to configure values for dbt
A description of any alternative solutions or features you've considered.
The initial look should be something like:
[dbt]
models_dir = "models/"Systems can send data to themselves, so is it useful to be able to list themselves as system_dependencies? Right now it is not possible. system_dependencies is deprecated, but egress and ingress are replacing it and do allow self referencing. Making this issue as a place to discuss the pros/cons
Related to https://github.com/ethyca/fidesctl-plus/issues/193
This is a simple one, but I’ve found that every time we explain the provide data use, we invariably want to say something like “…the provide use is the most common reason to collect user data, and it’s because you need their data to provide your product or service.” On top of these, we use the noun system in many places in fideslang (including as a top-level category in data_category) where it has a different meaning, so I feel it'd be more intuitive to not use the "system" terminology in the data_use taxonomy at all.
Now, we have data uses like:
After, we’d just rename these:
A description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
As part of ethyca/fides#3036, we'll be making a proper db table to store the privacy_declarations of Systems. Since we've already got pydantic models for PrivacyDeclarations, by updating the System, PrivacyDeclaration and DataUse pydantic models (defined here in fideslang) to have orm_mode set to True, this would minimize the amount of code changes in fides that we'd need to implement this update; instead we can rely mainly on pydantic's orm model mapping to treat the new PrivacyDeclaration DB (ORM) records similarly to how they were being handled before as JSON blobs that were validated against the pydantic models
class Config:
orm_mode = True
to the System, PrivacyDeclaration and DataUse pydantic models defined in fideslang.
- The reason we'll be updating the `DataUse` model is because we'll add a 'hard' FK reference from `PrivacyDeclaration` to `DataUse` records, rather than relying on a 'soft' string-matched reference that existed previously
Instead of making this update, we could update more of our code in fides to translate DB/ORM instances of PrivacyDeclarations into JSON blobs as they'd been previously, so that they can be validated properly by the pydantic model. IMO, going that route would introduce more customized logic that would be fragile to future changes.
The proposed updates here would be a more "out of the box" solution that should be more maintainable and less error prone. Probably significantly less code too!
Another alternative: could/should we create a base pydantic class for our fideslang pydantic models that defaults to orm_mode = True? we may want to update more of our pydantic models in a similar fashion moving forward, so this could save us some updates. I'd need to do a bit more investigation to look at potential downsides/regressions from this broader change...
We may look to make similar changes moving forward, since we may want to store more of our fideslang pydantic models in their own DB tables. How can we make this as future proof as possible while also not blowing things up? See base class suggestion above.
For ethyca/fides#3315, we've discussed leveraging the meta attribute on the System model/class to pass in some arbitrary metadata. For that use case, it could be useful to specify nested JSON objects (ideally of arbitrary depth), rather than the "flat" (i.e. depth of 1) objects/dictionaries that are required currently based on the type annotation of Optional[Dict[str, str]].
@TheAndrewJackson also noted (in ethyca/fides#3315) that this was a requirement he needed for the webscanner demo.
Could we allow for nested JSON objects of arbitrary depth for the meta attributes (of System and probably of Dataset too, for consistency)?
e.g.
{
"saas_config": {
"type":"stripe",
"icon":"***",
}
}
or
{
"web_scan": {
"domains": [
{
"url": "***",
"cookies": [
"***",
"***",
"***",
]
}
]
}
}
I think this may be as simple as updating the type annotations to Optional[Dict[str, Any]]?
There may be some unforeseen implications of making this change -- is it possible that any dependent code (either in our, i.e. Ethcya's, code base or anyone else who uses fideslang) is depending on the stricter typing that's in place, and if so, would this change be disruptive?
Additionally, it would be good to understand the original motivation behind the stricter typing, if that was a deliberate decision.
If we judge that updating the type annotation on the meta property to be more permissive is not a good path forward, then we could create another property that allows for nested dictionaries (of arbitrary length) for arbitrary metadata. This feels pretty messy, though - especially considering we already have a fidesctl_meta attribute (that holds more strictly-typed system metadata)...
This is motivated by the work to more explicitly tie Systems to saas connector types. See ethyca/fides#3315 for some more detail on that use case.
Generating datasets from dbt models is great, but generally not super useful unless we are attributing them to a specific system. This would create a feature that allows us to do that
We could add a config value like default_system that will automatically attribute dbt models to a certain system at runtime.
Additionally, we could have a special key/value pair in the dbt models yaml parser that can assign models to specific systems on a per-model basis
Add any other context or screenshots about the feature request here.
When I am creating or editing a data use on a system, I need to be able to provide one or more cookie names that correspond to that data use.
Add an optional PrivacyDeclaration.cookies field which should be an optional list of strings.
There might be times when we want to assign different dbt models to different systems. We should allow the user to do this via model-level metadata
The dbt model parser would look for a special metadata field at the model leve called something like system that would tell fidesctl which system to assign the dataset to
Using a generic, catch-all default system for all datasets
Add any other context or screenshots about the feature request here.
When I click on the links on both the README, and policy pages I get 404s. For example on the README Learn more about [Data Categories in the taxonomy reference now](https://ethyca.github.io/fideslang/data_categories/). goes to a 404.
Add any other context about the problem here.
Reticketed from https://github.com/ethyca/fideslang/issues/94, which handled this for Datasets.
We want to more broadly unify some of the -ctl and -ops fields. Ops wasn't really concerned with updating Systems or Organizations in the past, but as we move to share models in between these products, it would be good to share the underlying attributes.
fides_meta instead of fidesctl_metafides_meta instead of fidesctl_metameta fields to Organization and System.Are there other models we need to consider?
Note:
Downstream fides writes system information to fidesctl_meta. This will need to switch to fides_meta. Likewise, downstream database tables will need to rename the fidesctl_meta attribute.
A description of what the problem is.
Ex. I'm always frustrated when [...]
A description of what you want to happen.
A description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
This library contains types, but when using it through pip install fideslang the installing library doesn't know about the types resulting in a mypy no type hints or library stubs error. This can be fixed by adding a py.typed file. After that installing libraries will know types are present and remove this error.
pip install fideslangfrom fideslang.models import FidesKeymypy: Skipping analyzing "fideslang.models": found module but no type hints or library stubsThe package should alert the installing package to the presence of types.
N/A
joint_controller and data_protection_impact_assessment are listed in the docs as array fields, but they are just object fields.
https://ethyca.github.io/fideslang/resources/system/
For example, data_protection_impact_assessment is not an array, but rather this: https://github.com/ethyca/fideslang/blob/main/src/fideslang/models.py/#L597-L618
The documentation should be updated to not describe these fields as arrays.
The fides project has extended the fideslang models to included extra metadata for processing DSRs. Most notably the FidesopsMeta model. All of the changes made to fideslang models in the dataset.py should be upstreamed into the fideslang project. This is to standardize on what DSR metadata should look like.
Acceptance Criteria
fidesops_meta to fides_metaWe want to include dbt models as viable sources of dataset information.
We need to write a parser for fideslang that allows us to create fideslang Dataset objects from standard dbt model yml files.
A description of any alternative solutions or features you've considered.
For this first iteration, we won't be implementing this into any existing commands, but we should have tests that prove the parser works.
The light mode in particular of the fideslang docs is using the old ethyca brand colors.
Appropriate updates can be pulled from either fidesops or fidesctl.
The dataset_references field on PrivacyDeclaration resources and the system_dependencies field on System resources were deprecated in #85 (see ethyca@36638c3 and ethyca@6e03b27). These fields should be removed.
While writing tests for https://github.com/ewdurbin/dbml-to-fides, I grabbed .fides/db_dataset.yml as a fixture.
When testing some transformations, I discovered that this file contains duplicates for fidesuser, fidesuserpermissions, and client tables with conflicting information (namely data_categories).
$ egrep ' - name: fidesuser$| - name: fidesuserpermissions$| - name: client$' .fides/db_dataset.yml
- name: client
- name: fidesuser
- name: fidesuserpermissions
- name: client
- name: fidesuser
- name: fidesuserpermissionsI would have expected that something validates a Fides dataset to have non-conflicting keys entries for each member of a collection
n/a
n/a
n/a
See engineering ticket: https://github.com/ethyca/fideslang/issues/58
Usages and diagrams both need updating, as well as the visual taxonomy.
There isn't a clear way to label PII data that is tied to a business.
There are instances where data can be associated with a business rather than an individual. For example, a phone number could be for a business but it would get categorized as user.contact.phone_number.
Add a business branch to the root node of the data categories tree.
Currently, it seems like the system category is the fallback.
Right now you are able to add two DatasetCollections of the same name to a Dataset. The name field should be enforced to be unique, since databases wouldn't allow i.e. two schemas of the same name.
There should be a validator that prevents the second DatasetCollection from being added
More context for why this came up in ethyca/fides#718 (comment)
With Pydantic 1.10.0 The tests are failing.
@pytest.mark.unit
def test_find_referenced_fides_keys_1():
test_data_category = DataCategory(
name="test_dc",
fides_key="key_1.test_dc",
description="test description",
parent_key="key_1",
)
expected_referenced_key = {"key_1", "key_1.test_dc", "default_organization"}
referenced_keys = relationships.find_referenced_fides_keys(test_data_category)
> assert referenced_keys == set(expected_referenced_key)
E AssertionError: assert {'default_organization', 'key_1.test_dc'} == {'key_1', 'default_organization', 'key_1.test_dc'}
E Extra items in the right set:
E 'key_1'
E Full diff:
E - {'key_1', 'default_organization', 'key_1.test_dc'}
E ? ---------
E + {'default_organization', 'key_1.test_dc'}Tests should pass
If applicable, add screenshots to help explain your problem.
I have opened a PR to specify pydantic < 1.10.0 for now until we can figure out what the issue is. #79
We want to let users "tag" their resources to eventually make filtering and reporting easier
We should add a tags field to the FidesBase model found here as type List[str]
Adding this to the generic meta field instead
Add any other context or screenshots about the feature request here.
The identifiable / nonidentifiable categories tend to confuse users; especially engineers who are attempting to classify a given field. We received this feedback from Momentive, Twitter, Ikea, and Walmart
A description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
https://github.com/ethyca/fideslang/pull/109 had a workaround for a pydantic bug. we should consider reverting this workaround in due course, if possible, just to clear out tech debt.
will the underlying bug(s) have been addressed at a certain point so that we can revert the change?
keep the workaround in place
keeping track of tech debt 😄
The following proposed changes to fideslang follow months of review and feedback from users. Most of these changes are due to confusion when implementing causing hesitation.
identifiable, nonidentifiable, derived, and provided subcategories
provide.system to provide.serviceaccount to a subcategory of user
user.account.email, user.contact.emailImplementing these changes will provide some simplicity when selecting a DataCategory and help reduce the barrier to entry.
Maintaining the existing taxonomy as backwards compatible.
Some decision made around this change:
fideslangThe visual taxonomy is missing the user.biometric category. Make sure these categories appropriately align with the docs.
"In particular the docs refer to data categories like user.biometric and user.biometric_health but on the visualization there’s user.biometric_health and user.credentials.biometric_credentials.
Can we double check that both/either are correct."
We need a way to track directionality of data flowing between systems
Adding an ingress and an egress field to the system resources
More alternatives and discussion took place here (internal only)
Current Definition: yaml system: - fides_key: demo_analytics_system name: Demo Analytics System description: A system used for analyzing customer behaviour. system_type: Service privacy_declarations: - name: Analyze customer behaviour for improvements. data_categories: - user.contact - user.device.cookie_id data_use: improve.system data_subjects: - customer data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified dataset_references: # This goes away - demo_users_dataset Proposed Definitionyaml system: - fides_key: demo_analytics_system name: Demo Analytics System description: A system used for analyzing customer behaviour. system_type: Service privacy_declarations: - name: Analyze customer behaviour for improvements. data_categories: - user.contact - user.device.cookie_id data_use: improve.system data_subjects: - customer data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified ingress: # This is added - fides_key: marketing_system type: system # An Enum of [system|dataset] action: process # An as-of-yet undefined Enum egress: # This is added - fides_key: customer_dataset type: dataset action: persist
Note that the main changes here: 1. The removal of the dataset_references field 2. The addition of the ingress and egress fields
Design decisions to consider: 1. Ingress and Egress are defined on a per-declaration basis 2. This design does not let users specify which taxonomy types are involved in the egress/ingress, instead inheriting those values from the privacy declaration they are attached to 3. The enum values for action will need to be defined (process? persist? Read/write?) 4. No delineation between internal/external ingress/egress (external systems should be defined the same as internal systems, so this seems like an ok omission)
Update Fideslang data uses as described here: https://ethyca.atlassian.net/wiki/spaces/PM/pages/2643132432/One+set+of+data+uses+to+rule+them+all+privacy+regulations#%5CuD83D%5CuDDD2-Proposed-Changes
Changes break down into five sections:
Row 5 is deliberately removed from the scope of this work. We will continue to refine this and look at including it in a future update.
Acceptance Criteria
Added the is_default column, ready to cut a patch release
The fideslang module is designed to be the foundation for all fides projects but is currently living in fidesctl. It should be put here to facilitate more intentional changes to the language itself and cleaner releases and version pinning of the language.
This repo becomes the python project for fideslang, which gets uploaded to pypi and versioned just like everything else.
Keeping this module in fidesctl
This will simplify the fidesctl repo somewhat and give us a docs site to heavily describe the language.
To be consistent with the other taxonomies, Data Subjects should be hierarchical to be more descriptive in declaring the types of data subjects that a piece of software can hold.
Corresponding ticket to ethyca/fides#3358 which fixes the problem in fides. But we should fix it also here in fideslang to keep it compatible in its own right with python < 3.10.
fideslang in a python env < 3.103.10 should be able to easily leverage fideslangIf applicable, add screenshots to help explain your problem.
See ethyca/fides#3358 for more context
Now that PrivacyDeclarations have their own DB table/records, we should allow their id to be exposed on API responses so that it can be referenced by other operations (e.g. custom field references)
Optional[str] field of id in the PrivacyDeclaration modelInstead of updating fideslang here, we could instead extend the PrivacyDeclaration model class in fides to include the additional field, and it would be used specifically in response payloads. This may be a bit more targeted, but it's a lot more code changes and i think it will be hard to follow, so i prefer this option here.
I believe this will be needed to allow the fidesplus FE to be able to add custom fields that reference particular PrivacyDeclarations. without this id being returned in our API responses, there's no way that the FE will be able to know the id of particular PrivacyDeclaration.
When a System resource that includes egress and/or ingress DataFlows are exported to YAML using write_manifests(), the type field renders incoherently. Ex:
type: !!python/object/apply:fideslang.models.FlowableResources
- systemThe type field should be written as a literal string value. Ex:
type: systemRather than assert that type parameters must be a member of the FlowableResources Enum, assert that type is a literal str, and add a validator that asserts it's inclusion in the FlowableResources Enum.
Once #43 is implemented, we would like to enforce that each system has at least one "tag"
Add a pydantic validator to make sure that the "tag" isn't left empty for any systems
A description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
When pip installing fideslang the types are not being included even though they are present so when using the package types have to be ignored.
Types should be included.
If applicable, add screenshots to help explain your problem.
Add any other context about the problem here.
The taxonomy visualization image is not working and hasn't been for a few days. It's very helpful to see. https://ethyca.github.io/fideslang/
Describe what you'd like to see documented, and link the page if applicable
Add any other context about the problem here.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.