Database design about data HOT 25 CLOSED

I think it'd be a good idea to add a severity in the format, Arachni currently uses high, medium, low, informational.

from data.

Also, for the references, it may be better if desc is changed to title or if we have a mandatory title and an optional description.

In addition, caps in the keys seem weird, it'd be more consistent if we use lowercases for everything and underscores only instead of mixing - and _.

from data.

Added severity to the JSON format by updating the issue

from data.

Changed all to lower case with underscores

from data.

Changed desc to title, I don't believe we need desc+title for references, IMHO with title it's enough, but if you guys believe we should have both, we can add "desc"

from data.

A few more suggestions, maybe rename solution to fix so that it'll match fix_effort (or vice versa) or better yet have:

{
    "fix": {
      "guidance": "Blah blah",
      "effort": 50
    }
}

So that we can keep a coherent relationship between the fields.
That would also allow us to get platform specific in the future by extending the schema.

from data.

Why not use yaml? Yaml has multiline support, and you could probably even use yaml-frontmatter format as well.

For eg, at backdoor, we store the challenge information in yaml as follows:

---
tags:
    - binary
    - medium
creator: vampire
score: '70'
title: 'Hidden Flag - Medium'
flag: This is the flag
---
n00b became depressed when 'Pro' found the flag in his binary in a matter of seconds.
This time he hid the flag a little more securely.
See if you can still find it: [file](http://hack.bckdr.in/HIDE-MEDIUM/hide_medium)

The lower portion is in markdown, and any such files render with the metadata in github perfectly.

from data.

@Zapotek Agreed on the fix stuff, added to spec in issue text

from data.

@captn3m0 I like yaml, i'm 100% sure it can be used to store our database and solves the multi-line issue that json has. Anything else we need to know about yaml? Any other extra feature we might take into account to consider migrating to that format?

from data.

Well, some YAML parsers aren't really safe (looking at Ruby).

from data.

@andresriancho If references only need a URL and a title we can use the URL as the key and the title as the value like in Arachni. And better make the title mandatory I think.

from data.

Agreed on the mandatory title (changed) but don't agree on the URL as key, this:

  "references": [
      {"url": "http://foo.com/xss", "title": "First reference to XSS vulnerability"},
    ]

is more readable for humans than:

  "references": [
      {"http://foo.com/xss": "First reference to XSS vulnerability"},
    ]

And we want humans editing these JSON files easily, quickly, without second-doubting what they should enter in each section/field

from data.

PS: Also added the tags which were present in the arachni json files

from data.

• Most yaml parsers run in safe mode by default these days.
• All json is automatically valid yaml as well, which means you could just rename files and they will be parsed as yaml fine even now

Other than this (and the high readability), I cant think of any other benefits.

from data.

To accommodate fix guidance for different programming languages:

  "fix": {
      "guidance": "A very long text explaining how to fix XSS vulnerabilities",
      "effort": 50
    },

Or

  "fix": {
      "guidance": {"general": "A very long text explaining how to fix XSS vulnerabilities in general",
                          "django": "Change the view / template code using django stuff"}
      "effort": 50
    },

from data.

Maybe the good way is to put guidance for different languages in different files, like you already described for languages support?

from data.

I believe that would just complicate things and make it difficult to find the right file to edit 👎 for me.

from data.

Usually, languages have json support out of the box ( "encoding/json" in Go or "json" in Python) and don't have yaml.
Also, it's easy to write bad yaml than json, but it's not so important in our case.

from data.

👎 for yaml then

from data.

I want to specify:

1. Is description required and must contain at least 30 chars?
2. What is a minimum and maximum length for title and tags? (I think about 2 and 255)

from data.

Is description required and must contain at least 30 chars?

Yes, I set it to an arbitrary 30 chars, but if you see it's too low set it to more.

What is a minimum and maximum length for title and tags? (I think about 2 and 255)

Title feel like 4-255
Tags feel like 2-255

from data.

If we gonna use

  "fix": {
      "guidance": {"general": "A very long text explaining how to fix XSS vulnerabilities in general",
                          "django": "Change the view / template code using django stuff"}
      "effort": 50
    },

than effort also depends on different programming languages/cms/frameworks

from data.

Well, the effort is the aprox number of minutes required to fix the issue, I doubt that fixing a vuln in django takes 1 minute and then it will take 38 in php. We could either:

• Associate it with the general/django/etc. and most likely have the same aprox effort for all
• Make it independent and accept the mistake we might have in some cases

I'm +0 on this, you take the decision and let me know how it goes.

from data.

What do you think if we add index file for search purpose?

{
"tags": {
    "tag_name1": [1, 2], // array of vulnerabilities ids for the tag
}
// ...
}

The file might be generated automatically.

from data.

Might be useful, please create a different ticket for it

from data.