In some cases it is desirable to see what the request type, URL, parameters and headers are.

Compare it to what Hibernate does when printing its SQL.

The server works at a certain time, whereas the server JOSS runs on probably has another time.

Make sure that when JOSS makes a call to the server that involves the calculation of time from the server (such as for temp URLs), that the Swift server time is used, not the JOSS server time.

Currently, Object.getName() returns the full name, including a pseudo-hierarchy.

The following name:

scripts/custom/some.js

In this case getName() returns this exact name.

It desirable to be able to return the following as well (in the case of the example):

• scripts/custom
• some.js

This functionality will also be necessary to save a pseudo-hierarchy object using the downloadAsFile method.

Pushing and pulling has consequences for the files that already exist in the target folder, yet do not exist in the source folder -- they get deleted.

However, if this folder happens to be your project folder, this is undesirable.

Also, files in the source directory which are marked as .gitignore, are not feasible candidates for syncing, so they must be skipped.

Hook into the .gitignore principle, enable it by default and allow it to be turned off.

Currently, JOSS has Object.getPublicUrl(), which constructs a URL for an object. If the host is set, it will use this (pretty) host, else it will use the (non-pretty) ObjectStore URL.

JOSS must support the following:

• getPublicUrl(); uses publicHost or else ObjectStore URL
• getPrivateUrl(); uses privateHost or else ObjectStore URL
• getUrl(); uses privateHost if the container is private and privateHost is set, else publicHost (if set), else ObjectStore URL

AccountConfig will change:

• remove host
• add publicHost
• add privateHost

Considered keeping host for backwards compatibility reasons, but rejected.

Replace the mocking frameworks Mockito / PowerMock with JMockit.

Hi,when authenticating account ,it respones HTTP Status code: 500. then,i check the keystone.log get this error info:
(root): 2013-11-12 15:57:45,592 ERROR get_version() got an unexpected keyword argument 'auth'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 184, in call
result = method(context, **params)
TypeError: get_version() got an unexpected keyword argument 'auth'
what can i do ?

Make sure that Website shows it URL as well.

In this case, always show the official, ugly URL and put the Container name as the first value of the host.

Ie, this:

a32c0e5f920a4dbc967e50dd2a4e3957.objectstore.eu/joss

Becomes:

joss.a32c0e5f920a4dbc967e50dd2a4e3957.objectstore.eu

This is a name which can be used as a CNAME reference.

http://docs.rackspace.com/files/api/v1/cf-devguide/content/TempURL_File_Name_Overrides-d1e213.html

Don't forget URL encoding

I have an issue with tenantName.
Accordingly to this spec http://docs.openstack.org/api/quick-start/content/ , swift will return 400 if I have both tenantId and tenantName specified.
I specify only tenantName.

I found the issue in the code. But I don't understand how this jar works for other people, because I didn't see any bug in the issues section?

Maybe I'm wrong

I am uses a third party library (Amazon's SDK) which puts encryption keys in the metadata when you store the object. This is an implementation of envelope encryption. When the object is retrieved it uses the keys to decrypt the file (it is a little bit more complicated than that, but that will suffice).

When the library sets a metadata entry it will be in the form of x-amz-iv, x-amz-key, and x-amz-matdesc. Joss then proceeds to change these names to X-Amz-Iv, X-Amz-Key, and X-Amz-Matdesc. When the sdk goes to get the encryption keys it asks for the metadata by the names it stored it as: x-amz-iv, x-amz-key, and x-amz-matdesc which don't exist and so the decryption fails.

Why is Swift changing my metadata names? Is there a way to turn this off?

This is happening in org.javaswift.joss.headers.Metadata. The constructor like looks like:

this.name = capitalize(name);

The real ObjectStore / network / Internet / you-name-it can have quirks. In order to simulate this quirky behaviour, it would be handy if JOSS could sometimes throw an error on calling a method on the mock ObjectStore.

The AccessImpl object and its children must be resilient to new fields, not failing when they are added.

Internal URL must be assumed to be only accessible from a certain IP-range, probably excluding the range JOSS users are working on.

Public URL has the same REST API functionality as internal URL, so is the recommended way to go according to CloudVPS.

Implement ASAP.

The spec allows for versioning of containers: http://docs.openstack.org/developer/swift/overview_object_versioning.html

Versioning would need a few additional methods on both the Container and the StoredObject

StoredObject:

1. addVersioning - Sends a X-Versions-Location header - I personally feel it would be useful to have Joss create a versioning container for you and to auto name it but this is up for debate. If this is not the case you would probably pass the container you want to use for versioning as a parameter
2. isVersioned
3. removeVersioning - Sends a X-Remove-Versions-Location header

There are a few commands for StoredObjects that go along with versioning:

1. getPreviousVersion
2. listVersions
3. deleteAllVersions - It would probably be convenient to have a delete all versions command since delete on a versioned container only deletes the most recent update.

I can issue a pull request if that would be helpful, but I would like to hear some comments before I put anything to code.

Especially with the new host names, replacing those with a proper domain is essential.

It is desirable if JOSS can be instructed to return URLs with a custom host, instead of always returning the public URL that Keystone returns.

Note that mock Swift already makes use of a custom URL mode. Combine the two?

Introduction

It is possible to upload a file directly from the browser to the ObjectStore using FormPost. This is the code in the HTML:

<form action="https://0fef7424e0f44779abf6e98d0bae9874.objectstore.eu/installables" method="POST"
        enctype="multipart/form-data">
    <input type="hidden" name="redirect" value="https://sis.42.nl/list" />
    <input type="hidden" name="max_file_size" value="104857600" />
    <input type="hidden" name="max_file_count" value="10" />
    <input type="hidden" name="expires" value="1375255337" />
    <input type="hidden" name="signature" value="c264da6ff7910a6a161e2da4b546c3aeee1a3d9a" />
    <input type="file" name="file1" /><br />
    <input type="submit" />
</form>

Technical background

JOSS must be capable of generating the signature for FormPost on the basis of the following parameters:

• path (same mechanism as TempURL)
• redirect
• max_file_size
• max_file_count
• expires (same mechanism as TempURL)

I got it working in a test setup with this configuration:

path: /v1/AUTH_0fef7424e0f44779abf6e98d0bae9874/installables
redirect: https://sis.42.nl/list
max_file_size: 104857600
max_file_count: 10
expires: 1375255337
signature: c264da6ff7910a6a161e2da4b546c3aeee1a3d9a

This python code helps in determining the signature:

import hmac
from hashlib import sha1
from time import time
print '============'
path = '/v1/AUTH_0fef7424e0f44779abf6e98d0bae9874/installables'
print 'path: '+path
redirect = 'https://sis.42.nl/list'
print 'redirect: '+redirect
max_file_size = 104857600
print 'max_file_size: '+str(max_file_size)
max_file_count = 10
print 'max_file_count: '+str(max_file_count)
expires = int(time() + 86400)
print 'expires: '+str(expires)
key = '********'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
    max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()
print 'signature: '+signature

Note that the official documentation places a '/' at the end of path. This doesn't work.

Reading

• http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost
• more information on Temp URL (path and expires): #32

When AccountConfig is set in Tomcat, it has problems with the fluent interface (ie, returning this, instead of void).

Do the following:

• make sure AccountConfig is what it was before, it no fluent interface
• by default init a new AccountConfig object in AccountFactory
• set a fluent interface on AccountFactory for all relevant properties

If you send the following metadata field to the ObjectStore:

X-Container-Meta-small-caps-text: value

It creates the following:

X-Container-Meta-Small-Caps-Text: value

The best solution would be to have JOSS already capitalize the first character and all characters after a dash.

This would instantly fix the mismatch (for this issue) between the mock and the real ObjectStore as well.

Keystone pretty quickly moved to v2.0 auth requests, but tempauth [1] never supported it, and with alternative swift auth systems like swauth [2] - v1.0 auth requests have always been the lowest common denominator.

I've seen a few swift clients that have options to support v1.0 and v2.0 auth requests (e.g. swiftly [3], python-swiftclient [4])

I wonder how or where @robert-bor would like to see something like that added to JOSS?

1. https://github.com/openstack/swift/blob/master/swift/common/middleware/tempauth.py
2. https://github.com/gholt/swauth
3. https://github.com/gholt/swiftly/blob/master/swiftly/client/standardclient.py#L205
4. https://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L265

exists() gets the Container/StoredObject information object. When this entity does not exist, it returns a 404.

Make sure that no ERROR is logged if exists is called and this leads to a 404.

At trystack.org initially you do not have a tenant ID/name. If you place an authentication call to Keystone without providing a tenant, you will not get the applicable endpoints. You will have to go through a number of stages to retrieve the tenant and pass this in the auth call

1. send the auth call with username/password
2. use the authentication token to get the tenant list
3. send the auth call (again), this time also with the tenant ID and/or name

Step 1

curl -k -X 'POST' -v http://x86.trystack.org:5000/v2.0/tokens 
-d '{"auth":{"passwordCredentials":{"username": "********", "password":"********"} }}' 
-H 'Content-type: application/json'

Step 2

Replace "[INSERT_AUTH_TOKEN_HERE]" with the token ID found in the response of the previous call

curl -H "X-Auth-Token: [INSERT_AUTH_TOKEN_HERE]" http://x86.trystack.org:5000/v2.0/tenants

Step 3

Replace "[INSERT_TENANT_ID_HERE]" with the tenant ID found in the response of the previous call
Replace "[INSERT_TENANT_NAME_HERE]" with the tenant name found in the response of the previous call

curl -k -X 'POST' -v http://x86.trystack.org:5000/v2.0/tokens 
-d '{"auth":{"passwordCredentials":{"username": "********", "password":"********"}, 
"tenantId":"[INSERT_TENANT_ID_HERE]", 
"tenantName":"[INSERT_TENANT_NAME_HERE]" }}' 
-H 'Content-type: application/json'

In practice, there is probably only one tenant per username/password combination. If this is the case, JOSS should be able to auto-determine the tenant from the array of tenants and place the authentication again with the found tenant.

It would be good for JOSS to synchronize watches with the server at the same pace a token expires, ie 24 hours.

The main use case for this is to be able to pass a configurable host when JOSS is in mock mode.

This configurable host could then be a service method tapping directly into the mocked content, so that it looks like a resource from a static resource, but actually is not.

Also apply URLEncoder.encode to the container/object names

This can be either done by immediately saving the metadata to the object as well (less reliable, but cheaper), or by invalidating the data so a new fetch is done (more relliable, but costs a call).

HttpClient does not have a timeout by default. It would be good if JOSS would support a way to set a default timeout on the HttpClient.

http://stackoverflow.com/questions/9734384/default-timeout-for-httpcomponent-client
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html

From Folsom onward, Swift no longer deletes metadata that is not explicitly passed.

Make sure that the mock situation reflects this reality.

Ie, this:

Screen Shot 2013-07-02 at 08.16.52.png

becomes:

https://0fef7424e0f44779abf6e98d0bae9874.objectstore.eu/installables/Screen%2520Shot%25202013-07-02%2520at%252008.16.52.png

and must be:

https://0fef7424e0f44779abf6e98d0bae9874.objectstore.eu/installables/Screen%20Shot%202013-07-02%20at%2008.16.52.png

Original question posed by ctrl-alt-dev

The mock ObjectStore is all in-memory and therefore reacts a lot faster than the real ObjectStore. To have a situation which resembles the real one, please add to JOSS the option to have a small delay on every call to the mock ObjectStore.

Currently, metadata must be set with a Map<String,Object>. Make sure that it can also be set per record that you want to add, with each set action resulting in a separate call.

Similarly, metadata must be fetchable per individual record as well.

When you call list() on Container and StoredObject, you only get the first page (10k elements) of results. The other pages will have to be fetched using the pagination principle.

Make sure that list() hides this pagination detail. It must assemble all underlying pages and return them as one.

Send a Keep-Alive header on every call to make sure the connection does not have to be rebuild from scratch

Adopt Erik's classloader hack:

public class EndPoint {

    public java.lang.String adminURL;

    public java.lang.String region;

    public java.lang.String internalURL;

    public java.lang.String publicURL;

    public String id;
}

public class Metadata {

    public boolean is_admin;

    public List<String> roles;

}

public class AccessImpl implements Access {

// ... snip...

    public Metadata metadata;

public class Tenant {

    public boolean enabled;

    public String id;

    public String name;

    public String handle;

    public String description;
}

Make sure JSON ignores the (new) extra fields.

https://github.com/openstack/swift/blob/master/swift/common/middleware/tempauth.py

Basic syntax:

curl -v -H 'X-Storage-User: VOLUME:USERNAME' -H 'X-Storage-Pass: PASSWORD' URL

Note that the URL looks like this: http://:8080/auth/v1.0

The response will have two interesting headers:

• X-Storage-Url
• X-Auth-Token

Inspired on #49 and the latest RedHat test drive.

It must be possible for JOSS to check whether a signature is a valid one and whether the expiry date has expired.

This allows for mock mode to be used with temp URLs.

Add a new property to AccountConfig called containerCaching. This property is true by default.

Any instantiated Container instance is tracked by its Account object, ie calls are routed through a ContainerCache object.

Make sure Account has a method to clean the ContainerCache.

It is now passed as part of the authenticate call, which is not the right place

A normal Container is created privately by default.

A Website must be created publicly.

In a Main class, when the program terminates it must clean up the ObjectDeleter (whose function is to look after DeleteAt / DeleteAfter objects) as well.

Currently, the mock mechanism directly uses the underlying mock object store.

However, it is desirable if Client / Account / Container / StoredObject have logic that is largely similar, whereas the commands differentiate.

The Mock Object Store needs to be abstracted out of the various Mock classes of the core entities.

Commands will get their own mock and impl packages.

Might be handy to have factories to instantiate the mock/impl commands.

Advice Koert:

• 2xx range -> success
• 4xx range -> failure
• 5xx range -> retryable

Combine this with having a number of convenience classes for setting up the ranges

Requirement

Content in private container must currently be served through the application. It is desirable that the application server can serve a 3xx HTTP response with a valid temporary URL (TempURL) to the object.

The implementation must be hidden by the API. Possibly by making this part of the getXXXURL() family.

My notes

Before TempURLs can be created, the account must first be passed the key for the hashes. This is done in the following way:

curl -v -X POST -H "X-Auth-Token: [AUTH_TOKEN]" -H "X-Account-Meta-Temp-Url-Key: [YOUR_PASSWORD]" [SWIFT_URL]

The digest must be created out of:

• the method
• expires
• object path

Note that the object path contains the path after the host!

Let's take an example

• method: GET
• expires: 2737152115
• object path: /v1/AUTH_a32c0e5f920a4dbc967e50dd2a4e3957/secret/hum3.png

String plainText = "GET\n2737152115\n/v1/AUTH_a32c0e5f920a4dbc967e50dd2a4e3957/secret/hum3.png";

This body must be SHA1 hashed in base-16 notation. For the example the result will be:

e9dab41e3ec1ae2d6c9dfb794bc9cb401757992c

The Fetch URL is the same as usual, except that you add an extra parameter (containing two sub-parameters, semi-colon delimited) to it:

• temp_url_sig=e9dab41e3ec1ae2d6c9dfb794bc9cb401757992c
• temp_url_expires=2737152115

The full URL now becomes:

https://a32c0e5f920a4dbc967e50dd2a4e3957.objectstore.eu/secret/hum3.png?temp_url_sig=e9dab41e3ec1ae2d6c9dfb794bc9cb401757992c;temp_url_expires=2737152115

The ObjectStore does the same composition with method, expires and object path and also creates a hash out of it, using the password (X-Account-Meta-Temp-Url-Key) set in the Account. It then compares this hash with the temp_url_sig. If the hashes match, the first test is passed. It then checks whether the time has expired. If it has not, it can now return the content.

Hash code

This codes behaves in a similar way to Python's hmac.new():

    public static String getHmacMD5(String privateKey, String input) throws Exception{
        SecretKeySpec keySpec = new SecretKeySpec(privateKey.getBytes(), "HmacSHA1");
        Mac mac = Mac.getInstance("HmacSHA1");
        mac.init(keySpec);
        byte[] hashBytes = mac.doFinal(input.getBytes());
        return Hex.encodeHexString(hashBytes);
    }

Caveats

CloudVPS currently works with a host that hides the /v1/AUTH_Account. Account is also replaced with a base-16 identifier. This value must still be passed as the full object path for the hash. Koert has said he will look into the possibility of just adding the regular object path, which would make a really nice solution.

Links

http://docs.rackspace.com/files/api/v1/cf-devguide/content/Create_TempURL-d1a444.html
http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.tempurl
https://www.hpcloud.com/learn/controlled-access-object-store (alternative approach)
http://www.rackspace.com/blog/rackspace-cloud-files-how-to-create-temporary-urls/

When no tenantId and/or tenantName is passed, the access object returned by Keystone will contain an empty serviceCatalog element.

The serviceCatalog element is supposed to be an array. This is true as long as the serviceCatalog has elements.

However, when the serviceCatalog has no elements, this can be different:

• in some cases it becomes a map (curly brackets)
• in some cases it becomes an array (square brackets)

Jackson understandably cannot deal with this ambiguity.

The concept solution is to have a different child class for Access that is only used when no tenant is supplied. In this case, the serviceCatalog is not added as a serializable field.

Example with an empty map:

{
    "access": {
        "token": {
            "expires": "2013-05-13T10:06:25Z",
            "id": "********"
        },
        "serviceCatalog": {},
        "user": {
            "username": "********",
            "roles_links": [],
            "id": "********",
            "roles": [],
            "name": "********"
        }
    }
}

Example with an empty array:

{
    "access": {
        "token": {
            "issued_at": "2013-05-12T10:09:31.816711",
            "expires": "2013-05-13T10:09:31Z",
            "id": "********"
        },
        "serviceCatalog": [],
        "user": {
            "username": "********",
            "roles_links": [],
            "id": "********",
            "roles": [],
            "name": "********"
        },
        "metadata": {
            "is_admin": 0,
            "roles": []
        }
    }
}

HEAD calls in the new Swift return SC_OK instead of SC_NO_CONTENT.

JOSS must be able to deal with this and the old situation.

It must be possible for downloadAsFile() to determine its own file name on the basis of a root directory only, similar to how the class Website now works.

Suppose the following object gets the order to downloadAsFile():

scripts/custom/some.js

If it is literally written to this path, it must be able to create the folders it requires.

The download instructions must allow for the possibility to retain (default) or remove the folder structure for an object on downloading to a local file.

The structure of Website must hook into this structure to keep redundancy at a minimum.

For example:

• allowReauthenticate
• allowCaching

httpClient is not a good choice, because it is somewhat harder to configure this in context.xml, especially with its quirky configurability

Trystack.org uses tenant ID (tenantId) instead of tenant name (tenantName).

Make sure that tenant ID is supported for the authenticate call as well.

Note: this will lead to a change in AccountConfig! tenant will be renamed to tenantName.

This does not work:

https://og.cloudvps.com:443/v1/AUTH_bfo000042-test/Amersfoort?marker=3235-aaaaadoublecheck"éé-te-dadsfa-small.jpg&limit=9999&format=json

Make sure all query parameters are URL encoded.

Currently, URLs are not properly encoded, leading to problems with URLs containing special characters.

Support Swift's Static Website: http://docs.openstack.org/api/openstack-object-storage/1.0/content/Create_Static_Website-dle4000.html

Have the following functionality in place:

• setIndex; determines the index file in the container
• setError; determines the error file in the container
• syncDirectory; synchronize a complete directory to the ObjectStore, ie remove deleted, update changed and add new objects