Meteor lags ? Aggregates to the rescue !

My app has 4 main collections in mongo :

• A user has many weeks and clients

• Each week has many orders, one per client, where the order content is stored (can be empty if the client didn't order that week).

Initially, my documents were quite dry, I was doing the "joins" work on the client. It got laggy though, so I moved the joins logic to the meteor server, with hooks (matb33:collection-hooks) in every directions to keep denormalized fields in sync.

It was a pain though, really hard to follow what was going on and to debug.

There's also potential for chain reactions and never ending hooks loops.

But mostly, it was buggy in edge cases.

So I'm migrating away from those hooks, and creating an aggregation pipeline per collection (week, client, order) that computes all the dynamic fields for some given ids, and returns the documents where the value is different, plus some additional ids to check in the next pass.

It's not yet in production, but after this rewrite, my e2e tests are passing.

Meteor uses mongodb 4.2 in development server. I didn't feel like forcing an update to 4.4, which brings in two very useful features for my use case (the $function operator and the ability to $merge to the collection that you are aggregating on).

So I've been $writing:{ $some: {$nested: {$queries:'It's a pain''}}} . I'm back to junior level of mastery in this new paradigm, where "It works" comes as a surprise. I'm sure i'll rewrite those ugly aggregations at some point, but for now they seem to do the job.

One nice approach to using aggregation pipelines in meteor is to declare them in a separate js file, and then have a simple, non-meteor script that runs the aggregation and logs the results.

I use nodemon to watch the files, and get a very fast feedback loop, without the need to wait for the meteor server to recompile. Great when you're learning as you go.

I start the script like this (-r esm allows me to use modern imports)

nodemon -e js  -r esm tests/scripts/mongofun.js

And the script itself is very simple. It just runs the query

tests/scripts/mongofun.js :

const mongoUrl='mongodb://127.0.0.1:3001/meteor'

async function actual_aggregation(db) {
  const result = await db
    .collection('Orders')
    .aggregate([
      {
        $match: {_id:'<some test id>'}
      }, 
      ...the_pipeline_imported_from_the_meteor_source
    ])
    .toArray();
  console.log(result);
}

const mongodb = require('mongodb');
const client = new mongodb.MongoClient(mongoUrl, {
  useUnifiedTopology: true
});

client.connect().then(
  async () => {
    const start = Date.now();
    await author_aggregation(client.db());
    console.log('took ' + (Date.now() - start) + ' ms');
  },
  e => console.error(e)
);

Anyway, that's all for today :)