How to sync data between Google Cloud Storage and Filestore?

Key concepts: Google Cloud Storage, Filestore, Storage Transfer Service

• Google Cloud Storage- GCPs service for blob storage.
• Filestore instances are fully managed file servers on Google Cloud that can be connected to multiple client types within the same network/VPC/IP range. In short, GCPs NFS service.
• Storage Transfer Service (STS)- Quickly and securely transfer data to, from, and between cloud and on-premises storage systems.

Why was the concept of Network File System (NFS) introduced?

The objective was to provide transparent, remote access to file systems on the same network via a client server

Pre-requisites: GCP Account, GCP Project (Role: Owner), Basic understanding of cloud infrastructure

Gotchas- All resources being spun up should be in the same region and VPC

Steps: (I won’t be getting into the machine configs/bucket types as they are subjective to your requirements)

1. Create a Service Account with the following permissions

Permissions: Owner, Project IAM Admin, Role Administrator, Pub/Sub Editor, Cloud Filestore Editor, Storage Object Admin, Storage Transfer Admin, Storage Transfer Agent

2. Create a GCS bucket: Multi-regional, Standard class, Uniform Access Control

3. Upload dummy files 🙂

4. Create a filestore instance

5. Create a VM instance (Same Region, Same VPC)

I used e2-standard-32 for this POC purposes, to have a good performance

Attach the Service Account that was created to this VM

6. SSH into the instance (Each command will followed by its purpose)

Perform the following commands

• Using a Vim editor to create a new file
• copy the service account key data (which was stored locally)
• Paste it to this new file and save it locally to this VM

Install Docker

Authorize the Service Account to gcloud

Install NFS

• Make a local directory to map to the Filestore file share
• /usr/local/my_dir is the name of the local POSIX directory for the Compute Engine VM instance

Mount the file share associated with the Filestore instance

Confirm the mount point

7. Create a Storage Transfer Service Job

Source: GCS

Destination: POSIX filesystem

GCS Path

Create an agent pool

Authorize the Service Account for all STS features

Set a frequency for the Transfer Jobs (Hourly, Daily, Weekly)

8. Monitor/Track Transfer Status

Important Links:

Filestore overview | Google Cloud

Requirements for file system transfers | Cloud Storage Transfer Service Documentation | Google…

Create and delete service account keys | IAM Documentation | Google Cloud

Transfer large data sets from Cloud Storage to Filestore | Google Cloud

Commands