Moving data between projects  |  Firestore  |  Google Cloud (2023)

This page describes how to use the managed import and export featuresto move Firestore data from oneproject to another. This can be useful for setting up a development environmentor as part of permanently migrating an app to another project.The example on this page demonstrates how to export data from asource project and then import that data into a destination project. Moving databetween projects involves the following steps:

1. Create a Cloud Storage bucket to hold the data from your source project.
2. Export the data from your source project to the bucket.
3. Give your destination project permission to read from the bucket.
4. Import the data from the bucket into your destination project.

Before you begin

Before you can use the managed export and import service, you must complete thefollowing tasks:

1. Enable billing for both your source project and destination project. Only Google Cloud projects with billing enabled can use the export and import functionality.

2. Make sure your account has the necessary IAM permissions in your source project and destination project. If you are a project owner for both projects, your account has the required permissions. Otherwise, the following IAM roles grant the necessary permissions for Firestore export and import operations:

   Owner, Cloud Datastore Owner, or Cloud Datastore Import Export Admin

   A project owner can grant one of these roles by following the steps in Grant access.

3. Set up the gcloud command-line tool and connect to your project in one of the following ways:

   • Access gcloud from the Google Cloud console using Cloud Shell.

     (Video) What's your #cloud #migration #strategy?

     Start Cloud Shell

     Make sure gcloud is configured for the correct project:

     See Also

     The R’s of Migration - DZoneData Migration Software: Which Solution Fits Your Project BestMove and rename buckets  |  Cloud Storage  |  Google CloudHow to transfer your data to Google Cloud | Google Cloud Blog

     gcloud config set project [SOURCE_PROJECT_ID]

   • Install and initialize the Google Cloud SDK.

4. Set up indexes in your new project. The composite indexes should match between the source and destination projects. Indexes should be set up first to avoid having to process each document multiple times.

Export data from the source project

Export your data by creating a Cloud Storage bucket for yourFirestore export files and starting an export operation.

Create a Cloud Storage bucket

Create aCloud Storage bucket in the same location as your Firestoredatabase. To view your database location, see yourproject location setting.You cannot use a Requester Pays bucket for export and import operations.

If your Cloud Storage bucket is not in yoursource project, you must give the source project'sdefault service account access to the bucket. Each Google Cloud projecthas an automatically created default service account with the name PROJECT_ID@appspot.gserviceaccount.com. Firestoreexport operations use this default service account to authorize Cloud Storagebucket operations. To give thedefault service account access to your source bucket, grant it theStorage Adminrole.

You can grant this role with thegsutil toolavailable in Cloud Shell:

Start Cloud Shell

gsutil iam ch serviceAccount:[SOURCE_PROJECT_ID]@appspot.gserviceaccount.com:admin \gs://[BUCKET_NAME]

You can also grant this role in the Google Cloud console.

Disable write operations (optional)

If your app continues to write to your database while you perform an exportoperation, you might not capture all of those writes in your export files. Toexport data from a consistent state, disable writes to your database by updatingyour security rules and halting any Admin SDK operations.

1. Update security rules

   In the Firestore Rules tab of the console, update your source project security rules to deny all writes. For example:

    // Deny write access to all users under any conditions service cloud.firestore { match /databases/{database}/documents { match /{document=**} { allow write: if false; } // Reads do not affect export operations // Add your read rules here } }

2. Halt writes from Admin SDKs

   (Video) Migrating from Cloud Datastore to Cloud Firestore (Part 1: app migration)

   Security rules do not stop writes coming from privileged server environments created using a Firebase Admin SDK or a Google Cloud Server Client Library. Make sure to halt write operations from your admin servers by shutting down or updating your servers.

Begin an export operation

Use the gcloud firestore export command to exportdata from your source project. You can export all your data or onlyspecific collections. Replace [SOURCE_BUCKET] with the name of yourCloud Storage bucket:

Export all data

gcloud firestore export gs://[SOURCE_BUCKET] --async

Export specific collections

gcloud firestore export gs://[SOURCE_BUCKET] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async

Take note of your export operation's outputURIPrefix as you will usethis later on. By default, Firestore adds a pre-fix to your exportfiles based on a timestamp:

outputUriPrefix: gs://[SOURCE_BUCKET]/2019-03-05T20:58:23_56418

As the export operation runs, you can use the firestore operations listcommand to view your operation's progress:

gcloud firestore operations list

Import data into the destination project

Next, give the destination project access to yourFirestore data files and start an import operation.

Give the destination project access to your data files

Before you can begin an import operation, you mustmake sure your destination project can access your Firestoredata files.

Move data files to a local bucket

If your source bucket location is different from theFirestore location of your destination project, you must moveyour data files to a Cloud Storage bucket in the samelocation as your destination project.

Move your data files to another Cloud Storage bucket by following the stepsin Moving and Renaming Buckets.For all the following steps, use this new bucket as the [SOURCE_BUCKET].

Give the project service account access to your source bucket

If your source bucket is not in your destination project, then you must give thedestination project's default service account access to your source bucket. Thedefault service account is named[DESTINATION_PROJECT_ID]@appspot.gserviceaccount.com. Togive the default service account access to your source bucket, grant it theproper permissions to access the bucket.

You can grant the necessary roles with thegsutil toolavailable in Cloud Shell:

Start Cloud Shell

gsutil iam ch serviceAccount:[DESTINATION_PROJECT_ID]@appspot.gserviceaccount.com:legacyBucketReader,legacyObjectReader \gs://[SOURCE_BUCKET]

You can also grant this role in the Google Cloud console.

Begin an import operation

Before starting the import operation, make sure gcloud is configuredfor the correct project:

gcloud config set project [DESTINATION_PROJECT_ID]

Use the gcloud firestore import command to import the data inyour source bucket into your destination project:

gcloud firestore import gs://[SOURCE_BUCKET]/[EXPORT_PREFIX] --async

Where [EXPORT_PREFIX] matches the pre-fix in your export operation'soutputUriPrefix. For example:

gcloud firestore import gs://[SOURCE_BUCKET]/2019-03-05T20:58:23_56418 --async

As the export operation runs, you can use the firestore operations listcommand to view your operation's progress:

gcloud firestore operations list