- GCP Account
- Development environment with following commands:
- Docker
- how to install: https://docs.docker.com/get-docker
gcloud
- how to install: https://cloud.google.com/sdk/docs/install
gsutil
- how to install: https://cloud.google.com/storage/docs/gsutil_install
- Docker
- Prepare your configurations
- Run
virtuoso
container on your local and Load your rdf data - Upload
virtuoso.db
to GCS viagsutil
command - Create and Auto-Start your instance via
gcloud
command - Congratulations ๐๐๐ฅณ๐๐๐ฅณ๐
- Setup on GCP
- Install
gcloud
andgsutil
command - Reserve static external IP address
- Get your domain
- Edit
.env
file
how to install `gcloud` and `gsutil` command
Other distributions here:
for Ubuntu:
sudo apt-get install apt-transport-https ca-certificates gnupg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-sdk
gcloud init
cf. https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address
Referring to .env.example
, rewrite your .env
more info ...
Mandatory:
USERNAME
: your usernameUSER_EMAIL
: your email (for letsencrypt)SERVER_NAME
: your domain (e.g. your-doma.in)GCP_PROJECT_NAME
: your project nameGCE_*
: instance preferences your want to createGCS_BUCKET_NAME
: your bucket name
Optional:
Parameters_NumberOfBuffers
&Parameters_MaxDirtyBuffers
: virtuoso performance tuningTOKEN_LINE
: enable notifycation cf. https://notify-bot.line.me
virtuoso official documentation: http://docs.openlinksw.com/virtuoso/rdfperfloading/
- Put the RDF data under
data/
- Config from
.env
:source .env
- Run script:
docker run --rm -v $(pwd)/data:/data -v $(pwd)/script:/script -u "$(id -u $USERNAME):$(id -g $USERNAME)" python:3.10-alpine python /script/configureSQL.py ttl --origin https://$SERVER_NAME
- Start virtuoso container:
docker-compose up -d virtuoso
- confirm your server online:
docker-compose logs
-
virtuoso_container | HH:MM:SS Server online at
$PORT_VIRTUOSO_ISQL
(pid 1)
- Load RDF data on virtuoso:
nohup docker exec -i virtuoso_container isql $PORT_VIRTUOSO_ISQL -U dba -P $PASSWORD_VIRTUOSO < ./script/initialLoader.sql &
nohup $@ &
runs commands ($@
) in the background and independently.- To check the completion of the process, please refer to
nohup.out
.
- Create a named bucket with the command
gsutil mb gs://<bucketname>
- You can use the
-p
option to link it to your project
- Copy data from the local to the GCS:
gsutil cp src_url dst_url
source .env
gsutil mb -p $GCE_PROJECT_NAME gs://$GCS_BUCKET_NAME
gsutil cp ./.virtuoso/virtuoso.db gs://$GCS_BUCKET_NAME
For variable GCE_CREATE_ARGS
, check the description in .env.example
carefully and read the GCP documentation much carefully before executing commands below.
source .env
gcloud compute instances create $GCE_CREATE_ARGS
Learn more
When we create an instance with gcloud
, the following arguments are given:
GCE_CREATE_ARGS="$GCE_INSTANCE_NAME \
--project $GCE_PROJECT_NAME \
--zone $GCE_ZONE \
--machine-type $GCE_MACHINE_TYPE \
--tags $GCE_TAGS \
--create-disk $GCE_CREATE_DISK \
--metadata-from-file user-data=$PWD/gcp/cloud-config.yml,NGINX_CONFIG=$PWD/nginx/default.conf.template,DOTENV=$PWD/.env,COMPOSE_FILE=$PWD/docker-compose.yml,startup-script=$PWD/gcp/startup.sh \
--metadata google-logging-enabled=true,cos-metrics-enabled=true,USERNAME=$USERNAME \
--address $GCE_STATIC_IP_ADDRESS \
--shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring"
In this case, metadata-from-file
and metadata
are used to send various files to GCP.
cloud-config.yml
: Read only once when the instance is createdstartup.sh
: Loaded every time in start up (from a stopped).env
,default.conf.template
,docker-compose.yml
are read from the metadata server and saved as a file- Also synchronize data with
gsutil
and launch containers withdocker-compose
.
Congratulations !!
Open the GCP project page and check that the instance has been successfully launched.
- with GitHub Actions: for example, if there is a change under
data/
in the repository, it should trigger the virtuoso container to start โ load the data โ sync with GCS