Pipelines PGFS with Google Cloud Storage

Overview: Google Cloud Storage

PGFS uses the gs: prefix to indicate an Google Cloud Storage bucket.

The general syntax for using GCS is this:

select pgfs.create_storage_location(
               'storage_location_name',
               'gs://bucket_name',
               credentials => '{}'::JSONB
       );

The credentials argument in JSON format offers the following settings:

See the Google Cloud documentation for more information on how to manage service account keys.

These options can also be set up via the equivalent environment variables to facilitate authentication in managed environments such as Google Kubernetes Engine.

Example: private GCS bucket

SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images',
                                    credentials => '{"google_service_account_key_file": "/var/run/gcs.json"}'
       );

Example: authentication in GKE

Ensure that the GOOGLE_APPLICATION_CREDENTIALS or the GOOGLE_SERVICE_ACCOUNT_KEY_FILE environment variable is set on your PostgreSQL pod. Then, PGFS will automatically pick them up:

SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images');