def "create route, ASN pw supplied, then inet pw supplied"() {

-In the ack message summary the 'Create' count is not incremented

-In the detail it says:
SUCCEEDED: [route] 192.168.0.0/16AS100

but should say 'Create SUCCEEDED.....'

-No notif is sent for this successful creation

Non-existing lookup renders a 500 error code response and stacktrace. Ex: http://rest.db.ripe.net/lookup/ripe/mntner/ninjaa-mnt

curl "http://rest-test.db.ripe.net/search?source=test&query-string=test-dbm-mnt" gives an xml response where some elements have the attribute xmlns= .

creating an assignment with webupdates
the final screen shows a success message and the object that has just been created

but sometimes it seems it does the query too fast and does not find the assignment just created and shows the parent allocation object instead

this is very confusing to users

(1) Remove the unused xsi (XML Schema Instance) namespace.
(2) The xlink namespace needs to be declared on the whois-resources (root) element (currently attached to the object element).

query works with or without the ':1000' it finds the object:

puppy:denis:~ $ whois -rBG fdf8:f53b:82e4::/50
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

inet6num: fdf8:f53b:82e4:1000::/50
netname: dens_test
descr: testing inet6num operations
country: EU #Country is really world wide
admin-c: DI1029-RIPE
tech-c: DI1029-RIPE
status: ASSIGNED
notify: [email protected]
remarks: reverse engineering update ops
remarks: added for RIPE NCC DB Group testing 20121024
Hierarchical objects will be created and deleted
during this testing over the next few days
mnt-by: AARDVARK2-MNT
mnt-lower: AARDVARK2-MNT
mnt-routes: AARDVARK2-MNT
mnt-domains: AARDVARK2-MNT
geoloc: 52.37559917665907 4.888916015625
language: EN
changed: [email protected] 20121024
source: RIPE

% This query was served by the RIPE Database Query Service version 1.66.3 (WHOIS4)

but update fails to find it:

inet6num: fdf8:f53b:82e4:1000::/50
netname: dens_test
descr: testing inet6num operations
country: EU #Country is really world wide
admin-c: DI1029-RIPE
tech-c: DI1029-RIPE
status: ASSIGNED
notify: [email protected]
remarks: reverse engineering update ops
remarks: added for RIPE NCC DB Group testing 20121024
Hierarchical objects will be created and deleted
during this testing over the next few days
mnt-by: AARDVARK2-MNT
mnt-lower: AARDVARK2-MNT
mnt-routes: AARDVARK2-MNT
mnt-domains: AARDVARK2-MNT
geoloc: 52.37559917665907 4.888916015625
language: EN
changed: [email protected] 20121024
source: RIPE
delete: testing finished
override: denis,xxx,testing finished

Delete FAILED: [inet6num] fdf8:f53b:82e4::/50

inet6num: fdf8:f53b:82e4::/50
***Info: Value fdf8:f53b:82e4:1000::/50 converted to fdf8:f53b:82e4::/50
netname: dens_test
descr: testing inet6num operations
country: EU #Country is really world wide
admin-c: DI1029-RIPE
tech-c: DI1029-RIPE
status: ASSIGNED
notify: [email protected]
remarks: reverse engineering update ops
remarks: added for RIPE NCC DB Group testing 20121024
Hierarchical objects will be created and deleted
during this testing over the next few days
mnt-by: AARDVARK2-MNT
mnt-lower: AARDVARK2-MNT
mnt-routes: AARDVARK2-MNT
mnt-domains: AARDVARK2-MNT
geoloc: 52.37559917665907 4.888916015625
language: EN
changed: [email protected] 20121024
source: RIPE

***Error: Object [inet6num] fdf8:f53b:82e4::/50 does not exist in the database

Either via port43 query or rest api, the route object is not uniquely addressable.

Need to add a search key that allows to address a route object specifically, e.g. '10.0.0.0/24AS123'.

We should not use real data in the objects that we pre-load into the TEST DB or allow users to change the core elements. In the object TEST-NCC-HM-MNT we use the email address [email protected]. But test.de is an existing website. In this case it would be better to use [email protected].

Also TEST-NCC-HM-MNT has mnt-by set to TEST-DBM-MNT which also has a public password. They should both have mnt-by set to TEST-ROOT-MNT to protect this data.

The whois-resources XML schema at: http://apps.db.ripe.net/whois/xsd/whois-resources.xsd is out of date, and needs to be updated to describe the changes in the XML request / response.

Currently, an error response (e.g. 400, 500 responses) includes error text as plaintext or html. We should include an error object in the XML or JSON response, so that application developers can check for an error condition and obtain the error text programatically.

• Include an error object in the XML and JSON response, so we can include error text (and possible error code).
• Include the error object in a streaming response, if an unexpected error occurs (e.g. client hits ACL limit).
• Set the correct content-type in the response for an error (XML or JSON).

the REST API currently depends on the Jersey Bundle, which in turn pulls in a lot of dependencies. Investigate decomposing the dependency into include only what's necessary (jersey-server, jersey-client, jersey-servlet). (This change previously caused 500 server error responses in production, investigate why).

Also investigate the related issue of "duplicate classes" warning from shade plugin during maven build.

def "create route, mnt-by, parent inet and ASN pw supplied"() {

The pending code seems to have broken the basic notif model for route objects. Even with an object submitted with all passwords and is created successfully without queuing the notif message is NOT sent to the "mnt-nfy:" address of the MNTNER.

In the legacy code a DOMAIN object could be deleted by the MNTNER in the "mnt-domains:" of the parent address object

This functionality was missed in the new software. It now prevents legacy space holders from deleting DOMAIN objects in some situations.

Doing a query with --resource shows no output from sources that have no information. We should not show these individual %ERROR: 101 per source. But if no sources have a result we should show one %ERROR: 101.

puppy:denis:~ $ whois --resource -rG 193.0.0.3
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

%ERROR:101: no entries found
%
% No entries found in source AFRINIC-GRS.

%ERROR:101: no entries found
%
% No entries found in source APNIC-GRS.

%ERROR:101: no entries found
%
% No entries found in source ARIN-GRS.

%ERROR:101: no entries found
%
% No entries found in source JPIRR-GRS.

%ERROR:101: no entries found
%
% No entries found in source LACNIC-GRS.

%ERROR:101: no entries found
%
% No entries found in source RADB-GRS.

inetnum: 193.0.0.0 - 193.0.7.255
netname: RIPE-NCC
descr: RIPE Network Coordination Centre
org: ORG-RIEN1-RIPE
descr: Amsterdam, Netherlands
remarks: Used for RIPE NCC infrastructure.
country: NL
admin-c: JDR-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
status: ASSIGNED PI
source: RIPE
mnt-by: RIPE-NCC-MNT
mnt-lower: RIPE-NCC-MNT
changed: [email protected] 20110217
changed: [email protected] 20130319

route: 193.0.0.0/21
descr: RIPE-NCC
origin: AS3333
mnt-by: RIPE-NCC-MNT
changed: [email protected] 19960812
changed: [email protected] 19970822
changed: [email protected] 19970822
changed: [email protected] 19980102
changed: [email protected] 19980225
changed: [email protected] 19980720
changed: [email protected] 20000908
changed: [email protected] 20080910
source: RIPE

% This query was served by the RIPE Database Query Service version 1.65.10 (WHOIS2)

In the output below we say:
% Contact information will not be shown.
% Abuse contact for 'AS3333' is '[email protected]'

$ whois -K as3333
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
% Only primary keys will be visible.
% Contact information will not be shown.

as-block: AS3209 - AS3353

% Abuse contact for 'AS3333' is '[email protected]'

aut-num: AS3333

% This query was served by the RIPE Database Query Service version 1.60.2 (WHOIS3)

We need to write a migration guide to assist Developers to migrate from the beta REST API to the new API.

The old API is now deprecated, and will be turned off at some stage in the future.

A lookup / search request to the REST API will always get a 404 response, even if the object exists, if the client has been denied access due to excessive querying of personal data (ACL denied).

Need support in the REST API to specify update against a specific version, e.g.

PUT http://rest.db.ripe.net/ripe/mntner/ripe-dbm-mnt?version=25

So that we would avoid accidental overwrites.

To help users take advantage of this feature, we also should return the version with each object (like tags).

Since this would put a huge load on the database, we should simpligy the current VersionDAO massively, and simply use sequence_id as version. To further simplify things, we should add sequence_id in RpslObject, next to object_id, to make things easy to access and perfectly clear.

• by default it searches exact matching prefix in route tree;
• hierarchical lookup flags can override this behavior (e.g. -L 10.0.0.0/24AS123 would return all less specific routes with origin: AS123)

... so that when running in parallel, we don't get random failures depending on test execution order

def "modify inetnum add remarks:"() {

groovy e2e test output for the notifications looks like this:

<<<<<
>>>>> RECEIVE NOTIFICATION

subject: Notification of RIPE Database changes

----

This is to notify you of changes in RIPE Database or
object authorisation failures.

I can't see from this output which notification goes to which address. It needs to print out subject: and to:

def "modify ROLE with abuse-mailbox, ref by admin-c & tech-c, no ref by abuse-c, remove abuse-mailbox, org has abuse-c"()

ORGANISATION object has:
abuse-c: role1-ripe
admin-c: role2-ripe
tech-c: role2-ripe

role1-ripe and role2-ripe both have an abuse-mailbox attribute
Try to modify role2-ripe and remove abuse-mailbox
This fails with error:
***Error: There is an organisation referencing role Abuse Role's abuse-mailbox

But no organisation is referencing THIS role's abuse-mailbox. The organisation references the OTHER role's abuse-mailbox

It should be possible to remove the abuse-mailbox from any role object that is not referenced 'itself' by any abuse-c attribute

(side note - if the organisation does not have an abuse-c then this modify works)

def "modify inetnum, add remarks: with syntax error, no notifs sent"() {

I tried to modify an object with a : missing

inetnum:      192.168.200.0 - 192.168.200.255
netname:      RIPE-NET1
descr:        /24 assigned
country:      NL
admin-c:      TP1-TEST
tech-c:       TP1-TEST
notify:       [email protected]
status:       ASSIGNED PI
mnt-by:       RIPE-NCC-HM-MNT
mnt-by:       lir-MNT
changed:      [email protected] 20020101
remarks      just added
source:       TEST

and get the error:

The following paragraph(s) do not look like objects
and were NOT PROCESSED:

This SHOULD be recognised as an object with a syntax error:
***ERROR: unrecognised attribute 'remarks'

Optional attribute in ARIN AS object was ignored. Now it is expanded into a set of independent aut-num objects.

(in NotificationSpec)
def "create inetnum"() {

in the test we have these methods:
def notif = notificationFor "[email protected]"
notif.subject =~ "Notification of RIPE Database changes"

and in the test output I see:

RECEIVE NOTIFICATION

subject: Notification of RIPE Database changes

but when there are several notifications I can't see which message went to who. So this output needs to include the To:

RECEIVE NOTIFICATION

to: [email protected]
subject: Notification of RIPE Database changes

Also when notifications don't work as expected the test just hangs trying to get the message from the address I specified in the test. When it does not come I don't know where it went. There is a method to test if there are no more messages, noMoreMessages()

I would like a method to list all available messages (if this is possible)
listAllMessages()
which would just print out a list of To: addresses

def "create route, parent inet pw supplied"() {
We should follow the route creation auth flowchart to determine which mntners we send notifs of pending auth to:
http://www.ripe.net/images/routecreationIPspacecheck.gif

So if looking at less specific inetnum, use mnt-routes if found, else use mnt-lower if found, else use mnt-by.

But an exception is when we get to mnt-by don't send notifs to power mntners.

When looking at aut-num objects send notifs to mnt-by but exclude power mntners

def "modify person add reference to ORGANISATION with ref-nfy"() {

Modifying a person or role object still adds:

The old object can be seen in the history using the query options --list-versions and --show-version 1 FP1-TEST

but we don't show history or person/role objects. So this message should not be put out for person/role modifies

The address space used for the ripe meeting is classed as 'ripe address space'. So where our business rules only allow things to be done from ripe address space, about 500 people become ripe employees during a ripe meeting. The following /21 and IPv6 /48 (to get from OPS) should be excluded from ripe address space as far as our business rules are concerned.

inetnum: 193.0.28.0 - 193.0.31.255
netname: RIPENCC-MEETING-INFRASTRUCTURE
descr: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
remarks: RIPE NCC Training Services & RIPE Meetings
remarks: This space is used as infrastructure space during RIPE meetings
country: NL
admin-c: JDR-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
status: ASSIGNED PA
mnt-by: RIPE-NCC-MNT
mnt-routes: RIPE-NCC-MNT
mnt-domains: RIPE-NCC-MNT
changed: [email protected] 20120316
source: RIPE

We have data like this in the DB:

puppy:denis:~ $ whois -rBG 2a03:a300::/32
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

inet6num: 2a03:a300::/32
netname: KZ-KAZTRANSCOM-20110412
descr: JSC "Kaztranscom"
country: KZ
org: ORG-OA9-RIPE
admin-c: KTC1-RIPE
tech-c: KTC1-RIPE
status: ALLOCATED-BY-RIR
mnt-by: RIPE-NCC-HM-MNT
mnt-lower: MNT-ALCO-2K
mnt-routes: MNT-ALCO-2K
notify: [email protected]
changed: [email protected] 20110412
source: RIPE

route6: 2a03:a300:800::/32
descr: JSC "KazTransCom"
origin: AS35104
mnt-by: MNT-ALCO-2K
changed: [email protected] 20110923
source: RIPE

% This query was served by the RIPE Database Query Service version 1.65.10 (WHOIS2)

The ROUTE object has the extra ':800' in the pkey that is wrong, but was accepted by the legacy code.

We have the problem now of how to handle this data. We do not want to 'fix' all this data in the DB. Instead we want to present it in a query, properly formatted - without the extra :800. But if someone submits an update using the properly formatted prefix we want to match that to this incorrect object and 'fix' the pkey as part of the update.

def "change notify in object"() {

Modify an existing object that has dates on the changed line. The submitted new object has no date, leaving the software to generate it. You get this as the diff in the notification:

@@ -8,3 +8,3 @@
referral-by: MOD-MNT
-changed: [email protected] 20120901
+changed: [email protected]
source: TEST # Filtered

It has done the diff on the submitted object, not the 'final' object after generation of any attribute (part) values. This looks like I have removed the date.

when you make a query using -b an then click on the results to do a lookup, you get an error page:
https://apps.db.ripe.net/whois/lookup//inetnum/80.96.196.0-80.96.196.255.html

it seems the 'ripe' folder is missing from the url between /lookup//inetnum

without the -b flag I get the following link:

https://apps.db.ripe.net/whois/lookup/ripe/inetnum/80.96.196.0-80.96.196.255.html

Looking at the audit logs all I see is the credential that was used to authenticate an update. The purpose of these logs was to be able to retrospectively follow what happened when an update was processed in real time. Re-running an update, if at all possible, may not follow the same path.

When and why was this changed?

The new REST API does not include any object in the create response. This is necessary when creating a person or role object, so the nic-hdl can be identified.

The locator link in the REST API Create response (/whois-resources/link@xlink:href) is incorrect.

For example, here is the response when creating a person in the test database (the path /whois/ is incorrectly included in the link).

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<whois-resources xmlns:xlink="http://www.w3.org/1999/xlink">
<link xlink:type="locator" xlink:href="http://rest-test.db.ripe.net/whois/test?password=emptypassword"/>
<objects>
    <object type="person">
        <link xlink:type="locator" xlink:href="http://rest-test.db.ripe.net/test/person/PP1-TEST"/>
        <source id="test"/>
        <primary-key>
            <attribute name="nic-hdl" value="PP1-TEST"/>
        </primary-key>
        <attributes>
            <attribute name="person" value="Pauleth Palthen"/>
            <attribute name="address" value="Singel 258"/>
            <attribute name="phone" value="+31-1234567890"/>
            <attribute name="mnt-by" value="TEST-DBM-MNT" referenced-type="mntner">
                <link xlink:type="locator" xlink:href="http://rest-test.db.ripe.net/test/mntner/TEST-DBM-MNT"/>
            </attribute>
            <attribute name="nic-hdl" value="PP1-TEST"/>
            <attribute name="remarks" value="updated"/>
            <attribute name="changed" value="[email protected] 20120101"/>
            <attribute name="source" value="TEST"/>
        </attributes>
    </object>
</objects>
</whois-resources>

We want to add tags to the split files. Do them in the same way as in a query output, separated from the object and with a reference to the object pkey, but use '#' instead of '%' as the comment line starter.

def "modify organisation, change org-type OTHER to LIR"() {

gives the errors:

***Error: Adding or removing a RIPE NCC maintainer requires administrative
authorisation

***Error: This org-type value can only be set by administrative mntners

even though the update has the RS mntner and auth

We currently seem to be doing an exact match query based on the contents of the published stats file. However, when RS allocates consecutive blocks to the same user, they are listed separately in the stats file, but merged into a single DB object. For example:

In stats:
80.253.128.0/20
80.253.144.0/20

In DB:
inetnum: 80.253.128.0 - 80.253.159.255

This allocation in the DB should be tagged as a ripe-registry-resource, but it is not tagged at all.

def "create object with syncupdate, fail auth error"() {

Submitting an update by syncupdates causes GetResponse to timeout looking for the notification emails:

java.lang.AssertionError: Unable to get message for: [email protected]: Condition net.ripe.db.whois.update.mail.MailSenderStub$GetResponse was not fulfilled within 30 seconds.

With mailupdates it works OK

A ticket from Mike in RS:

Dear DB,

RS is using override and pgpupdate to update objects.
Both of them can not normally update objects with extended ASCII characters.

pgpupdate just doesn't work for them, an update results in invalid signature

override does work, but after an update all extended ASCII characters are
replaced with question marks.

For example, I updated ORG-PA33-RIPE recently and "ArmИe" is now "Arm?e"

Could this be addressed?

Thanks!

Mike

We want to find all INETNUM/INET6NUM objects in the DB that:

-are not tagged with either ripe-registry-resource or ripe-user-resource
-do not have the netname NON-RIPE-NCC-MANAGED-ADDRESS-BLOCK

list all pkeys with netname

Discuss with OPS about changing the remarks: in the object below (and IPv6 object) to make it clear these addresses are used by all attendees at a RIPE Meeting.

Find out if it is used at other meetings and also by TS (as current remark implies).

inetnum: 193.0.28.0 - 193.0.31.255
netname: RIPENCC-MEETING-INFRASTRUCTURE
descr: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
remarks: RIPE NCC Training Services & RIPE Meetings
remarks: This space is used as infrastructure space during RIPE meetings
country: NL
admin-c: JDR-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
status: ASSIGNED PA
mnt-by: RIPE-NCC-MNT
mnt-routes: RIPE-NCC-MNT
mnt-domains: RIPE-NCC-MNT
changed: [email protected] 20120316
source: RIPE

Lookup links in the response from rest-test.db.ripe.net should point back to rest-test, and not to rest.db.ripe.net.

For example:

curl http://rest-test.db.ripe.net/lookup/test/mntner/TEST-DBM-MNT

Returns the following:

<?xml version="1.0"?>
<whois-resources>
    <objects>
        <object xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" type="mntner">
        <link xlink:type="locator" xlink:href="http://rest.db.ripe.net/lookup/test/mntner/TEST-DBM-MNT"/>
        <source id="test"/>
        <primary-key>
            <attribute name="mntner" value="TEST-DBM-MNT"/>
        </primary-key>
        <attributes>
            <attribute name="mntner" value="TEST-DBM-MNT"/>
            <attribute name="descr" value="Mntner for TEST DBM objects."/>
            <attribute name="admin-c" value="AA1-TEST" referenced-type="person-role">
                <link xlink:type="locator" xlink:href="http://rest.db.ripe.net/lookup/test/person-role/AA1-TEST"/>
            </attribute>
            <attribute name="tech-c" value="AA2-TEST" referenced-type="person-role">
                <link xlink:type="locator" xlink:href="http://rest.db.ripe.net/lookup/test/person-role/AA2-TEST"/>
            </attribute>
            <attribute name="auth" value="MD5-PW" comment="Filtered"/>
            <attribute name="remarks" value="**********************************************************************"/>
            <attribute name="remarks" value="Password is &quot;emptypassword&quot; without the quotes."/>
            <attribute name="remarks" value="**********************************************************************"/>
            <attribute name="mnt-by" value="TEST-ROOT-MNT" referenced-type="mntner">
                <link xlink:type="locator" xlink:href="http://rest.db.ripe.net/lookup/test/mntner/TEST-ROOT-MNT"/>
            </attribute>
            <attribute name="referral-by" value="TEST-ROOT-MNT" referenced-type="mntner">
                <link xlink:type="locator" xlink:href="http://rest.db.ripe.net/lookup/test/mntner/TEST-ROOT-MNT"/>
            </attribute>
            <attribute name="remarks" value="This is an automatically created object."/>
            <attribute name="source" value="TEST" comment="Filtered"/>
        </attributes>
        <tags/>
    </object>
</objects>
</whois-resources>

The xlink:href attribute in all elements is incorrectly pointing to rest.db.ripe.net.

remove app.db & /whois/ prefix

The whois-resources root element does not need to be included in the JSON response, it is one unnecessary level which always needs to be traversed.

The response currently looks like:

{
"whois-resources": {
"objects": {
...
}
}
}

Instead, the response should be simplified to:

{
"objects": {
"object": [
{ ... } ]
}
...
}

Please extend api to allow for 'person-role'

In #1088988, Henriette reported that when webupdates tries to parse the acknowledgement from syncupdates, it fails on some errors and merges them on a single line.

The format of the REST API response should be consistent, pretty print (or not) ALL responses (this may be more difficult to do with streamed objects).