Giter VIP home page Giter VIP logo

weshare.click's Introduction

weshare.click

Weshare.click (or weshare for short) is a simple file-sharing application that you can self-host on AWS and use as an alternative to file-transfer services such as WeTransfer or Dropbox Transfer.

The first version of this application was built from scratch by Eoin and Luciano on a series of live streams as part of the AWS Bites show. The full playlist (6 episodes / ~8 hours) is available on YouTube.

The idea was to showcase how to build a real application using AWS, Serverless, and Node.js. The application is still minimal but usable.

What are the use cases?

Imagine that you have a file you want to share with others, or even with yourself on another device. You don't want to use Google Drive, Dropbox, or any of the public cloud services, perhaps because they are blocked in some way. This codebase will allow you to deploy your own, branded file-sharing service!

Features

  • Upload a file using REST APIs or a CLI application
  • Ability to use a custom domain
  • Authentication (OAuth 2.0 & OAuth 2.0 device flow)

The feature set is still minimal, but it gives you a fully functional and usable backend.

If there's any feature that you'd like to see here, please submit an issue or, even better, a PR.

Current Architecture

The current architecture makes use of the following services:

  • Lambda for the backend
  • API Gateway for the API
  • Cognito User Pool for authentication
  • S3 for file storage (using pre-signed URLs)
  • Route53 and ACM to manage your custom domains and certificates

A high-level architecture diagram is available in this repository.

The CLI authentication uses the OAuth 2.0 device authentication flow, which is not supported by Cognito by default, so we built a shim on top of the built-in Cognito OAuth 2.0 code flow. A diagram that showcases how that works is available.

Installation guide

Requirements

  • Your AWS account
  • The AWS CLI configured with the right credentials to deploy to your AWS account
  • Node.js (v16 or higher)
  • Serverless framework (v3 or higher)
  • Your custom domain (easier if already registered with Route53)
  • A bash-compatible environment (Tested on macOS but it should also work on Linux and Windows with subsystem for Linux)
  • jq: optional but useful if you need to run some of the suggested CLI commands below

1. Get the code & dependencies

The first step is to get the code locally by cloning this repository:

git clone [email protected]:awsbites/weshare.click.git
cd weshare.click

Now you can run the following script to download the necessary dependencies for every package:

./setup.sh

2. Configuration

Before deploying your weshare instance you need to provide some configuration.

To do that you have to create a file named config.cjs at the root of the project with the following content:

// @ts-check
'use strict'

const { defineConfig } = require('./weshare.cjs')

exports.config = defineConfig({
  // region: 'eu-west-1', // inferred from AWS_REGION or DEFAULT_AWS_REGION (or 'eu-west-1' if not set)
  // stage: 'dev', // the name of the stage to deploy to (e.g. 'dev', 'prod')
  // serviceName: 'weshare', // the name of the service (for resource naming)
  domain: '' // <-- ADD YOUR DOMAIN NAME HERE (e.g. 'files.weshare.click' or 'weshare.click')
})

Note: If you don't like copy pasting you can instead run mv config.cjs~sample config.cjs)

You only need to specify the domain name but you can also change the default region and the stage.

3. Deployment

To deploy all the stacks you can run:

./deploy.sh

Warning: The first deployment will need some manual intervention. The deployment will create a new Route 53 hosted zone. You will need to make sure that the DNS are propagated correctly to that new Hosted Zone. This is something that needs to be done DURING THE FIRST DEPLOYMENT. In fact, the deployment will also create a certificate in ACM and it will try to validate it based on resolving some DNS on that hosted zone. Until ACM is able to validate the domain, your deployment will be pending. See below how to manually configure the Hosted zone.

Configure NS records for the new hosted zone

When the new hosted zone is created it gets an NS record that contains 4 different name servers. You will need to update your domain configuration (in your domain provider site) to point to the nameserver to the 4 name servers in this record.

If you need a one-liner on how to get the name servers:

export DOMAIN='example.com' # <-- replace with your actual domain name
aws route53 get-hosted-zone --id $(aws route53 list-hosted-zones-by-name --dns-name $DOMAIN | jq -r .HostedZones[0].Id) | jq -r .DelegationSet.NameServers[]

If you are using a subdomain (e.g. files.example.com) and you are managing the top-level domain (e.g. example.com) with a Hosted Zone in Route53, you can configure a name server delegation with:

export TOP_LEVEL_DOMAIN='example.com'
export DOMAIN='files.example.com'
export HOSTED_ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name $TOP_LEVEL_DOMAIN | jq -r .HostedZones[0].Id)
export NAMESERVERS=$(aws route53 get-hosted-zone --id $HOSTED_ZONE_ID | jq '.DelegationSet.NameServers | map({Value: .})')
export COMMAND=$(cat << EOF
{
  "Comment": "delegating subdomain to hosted zone",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "${DOMAIN}",
        "Type": "NS",
        "TTL": 300,
        "ResourceRecords":  ${NAMESERVERS}
      }
    }
  ]
}
EOF
)
aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch "${COMMAND}"

Of course, if this looks like too much code, you can do the same on the AWS web console! ๐Ÿ˜Ž

If you know a better way to streamline the first deployment, please create an issue or a PR!

4. Create users in the Cognito User pool

To be able to login into weshare, you need to create some users first. You can do this either from the AWS web console or programmatically.

Here's how to add a new user to the user pool from the AWS CLI:

export USER_POOL_NAME="weshare-user-pool-dev" # <-- update if you changed the `serviceName` or the `stage` in the config
export USERNAME="[email protected]" # <-- replace with the username you want to add (needs to be an email)
export TEMP_PASS="ChangeMeSoon(1234)" # <-- replace with the temporary password
export USER_POOL_ID=$(aws cognito-idp list-user-pools --max-results 60 | jq -r ".UserPools[] | select(.Name | contains(\"${USER_POOL_NAME}\")) | .Id")
aws cognito-idp admin-create-user --user-pool-id "${USER_POOL_ID}" --username "${USERNAME}" --temporary-password "${TEMP_PASS}"

If all went well you should receive an email with your temporary password. After the first login, you'll be requested to change the password.

Usage

  1. Install the weshare CLI with npm i -g weshare
  2. Login using weshare login
  3. Upload a file with weshare <filepath>

For more info use weshare --help

Uninstall

To remove this app from this account you need to:

  1. Delete all the files from your files bucket and the bucket itself
  2. Run ./remove.sh to remove all the stacks
  3. Clean up DNS-related changes (e.g. delegation from another hosted zone or name server configuration in your domain provider)

Contributing

Everyone is very welcome to contribute to this project. You can contribute just by submitting bugs or suggesting improvements by opening an issue on GitHub.

License

Licensed under MIT License. ยฉ Luciano Mammino, Eoin Shanaghy.

weshare.click's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

weshare.click's Issues

Limit number of downloads

It might be useful to have an optional feature to limit the amount of downloads.

Potential benefits include

  • Limiting the blast radius of denial of wallet attacks
  • Enabling "self-destructing" download links that are valid for one download only, e.g. for sending sensitive information

Thoughts on how to implement that

  • Use DynamoDB table item attributes to track current and max downloads counters
  • Alternative: Store current and max downloads counters as object metadata on each uploaded file. Would alleviate need for separate database but would add more latency since S3 would effectively act as a database. The biggest downside I see for that option: in order to change object metadata you need to create a copy of that object.

Specify date when download URL expires

It might be useful to have an option to specify an expiration date for a download URL, i.e. users would no longer be able to download a file associated with a URL once its date has expired.

Potential benefits

  • Users can increase the security of their generated download URLs by issuing short-lived ones, making it harder to guess and exploit them
  • Tracking an expiration date could be used with a clean-up routine to automatically delete files whose download URLs have expired (๐Ÿ‘‰ #8)

Thoughts on how to implement that

  • Use DynamoDB table item attribute to track expiration date
  • Alternative: Store expiration date as object metadata on each uploaded file. Would alleviate need for separate database but would add more latency since S3 would effectively act as a database.

Refresh token automatically if token is expired

This requires the following information:

  • grant_type: refresh_token
  • refresh_token
  • client_id
  • client_secret
  • scope: openid
  • audience: (same value as the client_id)

Teoretically a user could refresh a token by themselves by performing the following request:

curl -v -d 'grant_type=refresh_token&refresh_token=XXXX&client_id=XXX&client_secret=XXX&scope=openid&audience=XXX' -H 'Content-Type: application/x-www-form-urlencoded' 'https://<cognito_user_pool_url>/oauth2/token'

But since a user won't have access to the client_secret this is something that will need to be done server side and there needs to be an API exposed to the user to do that...

Bug: UserPooldParameter and UserPoolClientIdParameter in backend stack do not update on change

In backend/serverless.yml we currently define the following parameter:

resources:
  Parameters:
    UserPoolIdParameter:
      Type: AWS::SSM::Parameter::Value<String>
      Default: /weshare/${sls:stage}/userPoolId
    UserPoolClientIdParameter:
      Type: AWS::SSM::Parameter::Value<String>
      Default: /weshare/${sls:stage}/userPoolClientId

These parameters are only populated at the first deployment and are not kept up to date anymore.

If a user redeploys the user pool for any reason, these parameters should change and the backend stack should be able to reference the new values.

This will result in a permanently failed authentication and the inability to upload files.

A possible solution might be to not rely on CloudFormation parameters but use SSM values directly.

A current workaround is to manually update the API Gateway Authorizer for the createShare endpoint and specify there the new values.

Add installation instructions and make project more configurable

The current README does not provide any useful information on how to install the project on a given account.

Also, we need to make the project configurable making sure the user can easily specify their custom domain and avoid potential conflicts with values that need to be unique (e.g. Cognito user pool domain)

Make custom domain name optional

Right now it is necessary to have a custom domain which makes the first setup unnecessarily complicated and it makes it hard for people to just try to install weshare.

It might be good to figure out how to use AWS-generated domain names (through API Gateway URLs) by default.

This requires a bit of research and there might be some significant pain points:

  • We currently assume there's only 1 base URL and we rely on that to build the other URLs (${base}/auth and ${base}/share). If we cannot rely on a custom domain we would need to either merge the auth and the backend stack to have the same API Gateway or figure out a way to support multiple base URLs. Similarly, the prefixes /auth and /share are currently the result of using custom domain mappings with API Gateway. We would need to either recreate them as actual paths under the API gateways or make the CLI smarter to use these prefixes or not depending on whether a custom domain is being used or not.
  • Doing conditional resources in CloudFormation is notoriously painful...

Protect download link with password

It might be useful to have an option to protect a download URL with a password.

Potential benefits

  • Users can more confidently share sensitive information by not only sharing a URL but also requiring a password in order to download the file

Thoughts on how to implement that

  • Use DynamoDB table item attribute to store download password
  • Alternative: Store download password as object metadata on each uploaded file. Would alleviate need for separate database but would add more latency since S3 would effectively act as a database. Also not sure about potential unwanted security implications of storing passwords that way
  • Caveat: This would probably require some kind of web frontend for users to type in the download password

Switch to POST pre-signed urls for upload

By switching to a POST pre-signed URL for upload we can allow the client to upload an arbitrary stream of bytes. This should help us to make the CLI more generic and make it possible to support the following use cases:

  • Upload generic input from stdin (which in turn allows dynamic pipelines like on-the-fly gzip compression or encryption)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.