Giter VIP home page Giter VIP logo

gcs-connector's Introduction

Google Cloud Storage Connector

This repo contains the Uploader app that is deployed in Cloud Run. To run the whole Storage connector, you will also need the Trigger function to be deployed in Cloud Function.

GCS Connector uploads files on WizyVision, on every files that will be uploaded in the source GCS bucket.

gcs connector

Setup and Requirements

  1. Google Cloud project
  2. WizyVision account
  3. Cloud Storage bucket
  4. Enable the APIs required
gcloud services enable \
    cloudfunctions.googleapis.com \
    pubsub.googleapis.com \
    cloudbuild.googleapis.com \
    artifactregistry.googleapis.com \
    containerregistry.googleapis.com \
    run.googleapis.com \
    --quiet
  1. Grant the pubsub.publisher role to the Cloud Storage service account. This will allow the service account to publish events when images are uploaded into the bucket.
SERVICE_ACCOUNT="$(gsutil kms serviceaccount -p GOOGLE_CLOUD_PROJECT_ID_HERE)"

gcloud projects add-iam-policy-binding GOOGLE_CLOUD_PROJECT_ID_HERE \
    --member="serviceAccount:${SERVICE_ACCOUNT}" \
    --role='roles/pubsub.publisher'
  1. Install docker for Manual installation.

Installation

Run on Google Cloud

Adding Permissions

After successful deployment (applies for both using Cloud Run Button and Manual Installation), navigate to GCP IAM & Admin and find the default compute service account created.

Screen Shot 2022-12-21 at 9 39 25 AM

Add these roles to the default service account:

  • Storage Object Viewer
  • Cloud Run Developer

Manual installation

  1. Clone this repository.
git clone https://github.com/sephdiza/gcs-connector.git
  1. Create a file named .env at the root of this folder. Copy the contents of placeholder.env and set the values.
# GCP
GOOGLE_CLOUD_PROJECT="<INPUT_YOUR_PROJECT_NAME_HERE>"

# Path where you put the service account file
GOOGLE_APPLICATION_CREDENTIALS_PATH="./secrets/service-account.json"

# WizyVision
# The application API key
WV_BEARER_API_KEY="<INPUT_WIZYVISION_APP_BEARER_API_KEY_HERE>"

# Public API url of your account
WV_PUBLIC_API_URL="https://mywizyvisiondomain.eu.wizyvision.app/api/v1/public/files"

# Comma-separated list of Tag IDs to set for the uploaded files
TAG_IDS=""

# Privacy to set for the uploaded files, Standard if this is left empty
PRIVACY_ID=""

# Header and token to be authenticated to call the Uploader service
UPLOADER_SERVICE_AUTH_HEADER="Wizdam-Dev-Csc-Token"
UPLOADER_SERVICE_AUTH_TOKEN=""

# Slack used for notification when errors are encountered. Can be left blank
SLACK_BEARER_TOKEN=""
SLACK_CHANNEL_ID=""
  1. Build the image.
docker build -t IMAGE_NAME .
  1. Push the image to the container registry.
docker tag IMAGE_NAME gcr.io/PROJECT_NAME_HERE/IMAGE_NAME
docker push gcr.io/PROJECT_NAME_HERE/IMAGE_NAME 
  1. Deploy to cloud run.
gcloud run deploy IMAGE_NAME \
  --project PROJECT_NAME \
  --image IMAGE_URL \
  --region REGION \
  --allow-unauthenticated \
  --memory=256Mi \
  --max-instances=10
  1. Take note of the URL of the service after successfully deploying. This will be needed in the Trigger service deployment.

Screen Shot 2023-01-06 at 9 57 03 AM

Testing the Connector

Note: The Trigger function should already be deployed

  1. Upload files on the GCS bucket
  2. In WizyVision webapp, you should see the file uploaded there with the correct tags and privacy set during the deployment

gcs-connector's People

Contributors

sephdiza avatar imajean avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.