Skip to content

Latest commit

 

History

History
206 lines (147 loc) · 7.61 KB

development.md

File metadata and controls

206 lines (147 loc) · 7.61 KB

Development Guide

Open Saves is a service that platform agnostic cloud storage & save solution that runs within Kubernetes.

Install Prerequisites

To build Open Saves you'll need the following applications installed.

  • Git
  • Go
    • On Windows, you need to make sure that the environment variable GOBIN is left undefined to work around an issue with building the gazelle tools. Otherwise you are going to get an error when trying to build/run bazel run //:gazelle update or bazel build //:buildifier
  • Docker including the post-install steps.
  • Protobuf
  • Go support for Protocol Buffers

You can install required go modules to compile protos by running:

go get -u \
  golang.org/x/lint/golint \
  github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway \
  github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger \
  github.com/golang/protobuf/protoc-gen-go

Optional Software

It's recommended that you install Go using their instructions because package managers tend to lag behind the latest Go releases.

Get the Code

# Create a directory for the project.
mkdir -p $HOME/workspace
cd $HOME/workspace
# Download the source code.
git clone https://github.com/googleforgames/open-saves.git
cd open-saves

Typically for contributing you'll want to create a fork and use that but for purpose of this guide we'll be using the upstream/main.

Building

In order to build the main Open Saves server, simply invoke the Makefile in the root project directory by running

make

from command line. This will compile protos and build binaries. Ouput binaries will be placed under the build/ directory.

Updating Go build dependencies

We use Go modules to manage dependencies. If you have changes in dependencies (i.e. adding a new source file), add the package to go.mod.

Running tests

In order to run the test, use

make test

Running a simple gRPC server/client

To stand up the gRPC server, there's a lightweight wrapper around the server code that lives in cmd/server/main.go. To start this, run

./build/server -cloud=gcp -project="<your GCP project>" \
  -bucket="gs://<your-bucket>" -cache="<redis-server-address>:6379"

You should see an output like the following

$ ./build/server -cloud=gcp -project="your-project" -bucket="gs://your-bucket" -cache="localhost:6379"
INFO[0000] Instantiating Open Saves server on GCP
INFO[0000] starting server on tcp :6000

To test the server is actually running, there is a sample gRPC client usage in examples/grpc-client/main.go. While the server is running, run

go run examples/grpc-client/main.go -address=localhost:6000 -insecure=true

Starting cache store

If using Redis for your cache store, you will need to pass in a path to Redis instance when starting the cache store. As an example, if in a development environment, you can run the following command.

./build/server -cloud=gcp -bucket="gs://your-bucket" -cache="localhost:6379"

If using Memorystore for Redis, you will get a private IP. In this case, to test locally, you would need to use port forwarding from a Compute Engine instance. The following commands illustrates how to create a new Compute Engine instance and forward port 6379 for the Redis host at 10.0.0.3 to localhost:6379.

gcloud compute instances create redis-forwarder --machine-type=f1-micro
gcloud compute ssh redis-forwarder -- -N -L 6379:10.0.0.3:6379

Setting up Google Cloud

You need to set up Cloud Firestore in Datastore mode (Datastore) and Cloud Storage to run the current version of Open Saves.

Cloud Firestore in Datastore mode

Cloud Firestore in Datastore mode (Datastore) is primarily used to manage metadata of Open Saves. Smaller blob data (usually up to a few kilobytes) could also be stored in Datastore.

Please follow the quick start guide to set up a database in Datastore mode. You may choose whichever region you like, however, it can only be specified one and cannot be undone. Google Cloud Platform currently allows only one Datastore database per project, so you would need to create a new project to change the database location.

Cloud Storage

Cloud Storage is used to store large blobs that don't fit in Datastore.

A sample Terraform configuration file is found at deploy/terraform/gcp/blobstore.tf. You'll need to change the bucket names as necessary as Cloud Storage bucket names are global resources. Alternatively, you may choose to create buckets with the Cloud Console. Refer to the Cloud Storage Documentation to learn more about creating buckets.

Updating the Cloud Build builder image

Note: You need permissions to the GCP project to make changes.

We use Cloud Build to run build tests for the GitHub repository. It is occasionally necessary to update the base image to upgrade to a new Go version, add a new build dependency, etc.

The base image is built using scripts/open-saves-builder-base.Dockerfile. You can build a new image locally by running scripts/build-open-saves-builder-base.sh.

After building the image, you can push it to Cloud Container Repository by running

docker push gcr.io/triton-for-games-dev/open-saves-builder-base:testing

Then, change the image tag in scripts/cloudbuild.Dockerfile from latest to testing, run

gcloud builds submit .

in the top directory, and make sure the tests still pass.

After verifying, you can revert the change in cloudbuild.Dockerfile, merge the changes to the main branch on GitHub, and tag and push the new image as latest by running

docker tag open-saves-builder-base:latest gcr.io/triton-for-games-dev/open-saves-builder-base:latest
docker push gcr.io/triton-for-games-dev/open-saves-builder-base:latest

Updating public images

Cloud Build detects changes to the main branch and kicks off the build and deployment of the open-saves-server:testing and open-saves-collector:testing images to Google Container Registry. This is done by a Cloud Build Trigger with the Github App, and the file that configures this is cloudbuild_update_images.yaml, using the substitution variable TAG_NAME = testing. A similar trigger tags images with :latest when new releases are detected, currently any new tags pushed.

These are not to be confused with cloudbuild.yaml which configures continuous integration when pull requests are opened.

IDE Support

Open Saves is a standard Go project so any IDE that understands that should work. We use Go Modules which is a relatively new feature in Go so make sure the IDE you are using was built around Summer 2019. The latest version of Visual Studio Code supports it.

Contributing to the project

Check out How to Contribute before contributing to the project.