The ckanext-s3archive from ckan

ckanext-s3archive's Introduction

Description

This extension archives all resource files stored in the local CKAN filestore into S3. The files are removed locally and just hosted on s3. When a user requests these files they are redirected to a temporary S3 url for download. Only authorized users are allowed to get these urls and the urls expire after 30 mins for security.

Installation

To install this package, from your CKAN virtualenv, run the following from your CKAN base folder (e.g. pyenv/):

pip install -e git+https://github.com/ckan/ckanext-s3archive#egg=ckanext-s3archive

Then activate it by setting ckan.plugins = s3archive in your main ini-file.

Compatability

CKAN 2.2+

Configuration

Config options should be placed in main section of the CKAN config file:

ckanext.s3archive.access_key = <access_key>
ckanext.s3archive.secret_key = <secret_key>
ckanext.s3archive.bucket = <bucket>

ckan.plugins = <current_plugins> s3archive

Usage

The following paster command will archive all the files to S3

paster --plugin=ckanext-s3archive s3archive archive -c <path to config file>

This command can be put in a cron job and is safe to be run regularly but is recommended that it is run about once a day. This is to give plenty time for the last job to finish before rerunning. If doing a large migration then it is probably best to run the command outside the cron for the first run.

ckanext-s3archive's People

Stargazers

Watchers

ckanext-s3archive's Issues

Support for SSE

I'd really like to see SSE enabled as an option. It should be as easy as adding encrypt_key=True to uploader.py.

I'll clone and pull if needed.

The README has the pip install wrong

I have added a pull request, to fix the changes.

Jon

Receiving namespace package error on install

Obtaining ckanext-s3archive from git+https://github.com/ckan/ckanext-s3archive#egg=ckanext-s3archive
Updating ./src/ckanext-s3archive clone
Complete output from command python setup.py egg_info:
running egg_info
writing requirements to ckanext_s3archive.egg-info/requires.txt
writing ckanext_s3archive.egg-info/PKG-INFO
writing namespace_packages to ckanext_s3archive.egg-info/namespace_packages.txt
writing top-level names to ckanext_s3archive.egg-info/top_level.txt
writing dependency_links to ckanext_s3archive.egg-info/dependency_links.txt
writing entry points to ckanext_s3archive.egg-info/entry_points.txt
warning: manifest_maker: standard file '-c' not found

error: Namespace package problem: ckanext.s3archive is a namespace package, but its
__init__.py does not call declare_namespace()! Please fix it.
(See the setuptools manual under "Namespace Packages" for details.)

Recommend Projects