Firestore Indexes Got Out Of Sync

Community

I hang around Firebase Developers and Expo Developers a fair amount.

White Label App

I have a "white label" solution that is backed by Firebase. I stand up a different Firebase project for each of my customers, and each project is meant to have "identical code, slightly different configuration". By "configuration" I mostly mean data-in-a-database, though some minor differences exist at the code level (app name, icon graphic, colour scheme, Firebase configuration values).

Use of scripts and automation makes keeping the customer projects in sync almost effortless. The effort goes into writing the scripts and in not allowing myself to "manually adjust" any of the Firebase projects. If a change is to be made, it will be made in scripts and applied to all of the customer projects so that they are all running "the same code".

Firestore Indexes

Firestore is a fantastic NoSQL document store database. It is performant and extremely scalable. One of the tools it uses to remain performant is indexes. Firestore automatically generates indexes for each individual field in your Documents so single-field queries "just work". For more complex queries, you need to generate a compound index.

Good news! Firestore will let you know when it needs a compound index in order to run a specific query. Firestore will throw an exception and provide a URL that simply needs to be clicked on to generate the missing index!

Bad news.....if your app is deployed to a project that doesn't have that index...then your app will get an exception and the user of your app will typically get a "white screen" or some other misbehaviour coming from the BUG that you have released. (It is a bug because it is misbehaviour that you failed to test for....)

Index Configuration

I do a lot of my automated setup and maintenance using the Firebase CLI, which is installed as the firebase-tools NPM package:

$ npm install -g firebase-tools

The CLI tool has dozens of features, such as initializing a new configuration repository for a Firebase project, creating initial Cloud Functions from a default template, deploying web content to Firebase Hosting, setting and getting configuration parameters, secrets and security rules, and Firestore index definitions.

Bad On Me

One area that I allowed myself to get into was fixing "white screens" in my app as quickly as I could. A user reports an issue, I reproduce it, I get the URL to create the index from my browser's JS Console that will FIX THE ISSUE, and I click it.

I have allowed myself to get in the habit of "clicking the link" to create the missing Firestore index, but then fail to remember to use the CLI to update my firestore.indexes.json file and then deploy it to all the other projects. Frequently users of one project have a workflow or a use-case that users in the other projects do not, so complaints from one project don't happen in other projects...and life gets busy.

Over four years and 14 projects, I have a "skew" in the indexes that are running across the projects. We are no longer running "the same code".

Unskewing the Skew

I'm in the process of a major update to my app. It involves updates to many parts of the code, and in doing so I have stumbled across this skew. Firestore indexes in each of my projects are not identical. They are mostly identical, but each project has a handful of "unique indexes".

The solution is to merge the differing configuration into a single comprehensive firestore.indexes.json file.

Here's how I approached it:

I wrote a (Bash) shell script to get the index definitions from each of my Firebase projects and put them into a file in the directory temp-indexes

#!/bin/bash
set -e

# the list of Firebase project names, one per line
ALL_PROJECTS="ab-calgary
bc-vancouver
mb-winnipeg
ns-halifax
on-toronto
qc-montreal"

mkdir -p temp-indexes

for i in $ALL_PROJECTS; do
  echo "project: ${i}"
  firebase -P ${i} firestore:indexes
  mv firestore.indexes.json temp-indexes/${i}.json
done

For each of the index JSON files, I "flattened" their definition with the following JavaScript function:

import fs from 'fs';

const filename = process.argv[2];
const fileContent = fs.readFileSync(filename, 'utf-8');
const jsonData = JSON.parse(fileContent);

/* the jsonData has the structure:
  {
    "indexes": [
      {
        "collectionGroup": "users",
        "queryScope": "COLLECTION",
        "fields": [
          {
            "fieldPath": "email",
            "order": "ASCENDING"
          },
          {
            "fieldPath": "address.city",
            "order": "ASCENDING"
          },
        ]
      },
      ....
    ]
  */

//  Convert each element of the indexes[] into a one-line string of format:
//         "collectionGroup,fieldPath:order:arrayConfig,fieldPath:order:arrayConfig,..."

jsonData.indexes
  .map((index) => {
    const fieldStrings = index.fields.map(
      (field) => `${field.fieldPath}:${field.order || ''}:${field.arrayConfig || ''}`,
    );
    return `${index.collectionGroup},${fieldStrings.join(',')}`;
  })
  .forEach((line) => console.log(line));

and now I could convert each of the JSON files from the command line with:

$ cd temp-indexes

$ for i in * ; do                                                                
      node ../flatten_indexes.js $i > ${i}.flat
  done

$ cat * | sort | uniq > all.flat

The file temp-indexes/all.flat is the comprehensive list of indexes from all of the projects, but in a "flattened" format. Now we want to "unflatten" the indexes with the following script:

import fs from 'fs';

const filename = process.argv[2];
const fileContent = fs.readFileSync(filename, 'utf-8');

const retVal = {indexes: [], fieldOverrides: []};

const lines = fileContent.split('\n');
lines.forEach((line) => {
  const cols = line.split(',');
  const name = cols.shift();

  const newIndex = {
    collectionGroup: name,
    queryScope: 'COLLECTION',
    fields: [],
  };

  cols.forEach((col) => {
    const [fieldPath, order, arrayConfig] = col.split(':');

    if (order && order.length > 0) {
      newIndex.fields.push({
        fieldPath,
        order,
      });
    } else if (arrayConfig && arrayConfig.length > 0) {
      newIndex.fields.push({
        fieldPath,
        arrayConfig,
      });
    } else {
      console.error(`Invalid line: ${line}`);
      process.exit(1);
    }
  });
  retVal.indexes.push(newIndex);
});

console.log(JSON.stringify(retVal, null, 2));

which we can run as:

$ node ./unflatten_indexes.js ./all.flat > ../firestore.indexes.json

We now have our updated firestore.indexes.json file that we should check into our version control system and then deploy to each of our projects using the firebase deploy command (or firebase deploy --only firestore)