💾 Using a Custom DataStore

Nodeblocks includes MongoDB examples by default, but any storage engine can be used for the features it supports when it exposes the collection-like interface those handlers expect. This flexibility allows you to use SQL databases, Redis, flat files, or even in-memory storage for testing.

📦 Required Packages: The core-profile example imports Express, the SDK, and Ramda:

npm install express @nodeblocks/backend-sdk ramda
npm install --save-dev @types/node @types/express @types/ramda

📋 Required Interface

Your custom datastore must implement these core methods used by most CRUD handlers:

• insertOne(doc) → { insertedId, acknowledged }
• findOne(filter) → the document or null
• find(filter, { skip?, limit? }?) → a cursor-like object with toArray(): Promise<Record[]>
• countDocuments(filter) → Promise<number>
• updateOne(filter, update) → { matchedCount, modifiedCount }
• deleteOne(filter) → { deletedCount }

Some SDK features also call additional methods such as updateMany, deleteMany, findOneAndUpdate, or MongoDB change-stream watch. Features may also use MongoDB operators such as $push and $pull; implement equivalent behavior when you expose those routes.

Below, we'll implement a JSON file adapter that demonstrates the core contract and show how to integrate it with a composed subset of profile features.

---

1️⃣ Implement the Adapter

datastore/jsonFileDataStore.ts

import {promises as fs} from 'fs';

interface DbRecord {
  id: string;
  [key: string]: unknown;
}
interface QueryFilter {
  id?: string;
  [key: string]: unknown;
}
interface FindOptions {
  limit?: number;
  skip?: number;
}
interface UpdateOperation {
  $set?: Record<string, unknown>;
}

export function createJsonFileDataStore(dbFile: string) {
  return {
    /* Create */
    async insertOne(doc: DbRecord) {
      const records = await readAll();
      records.push(doc);
      await writeAll(records);
      return {insertedId: doc.id, acknowledged: true};
    },

    /* Read single */
    async findOne(query: QueryFilter) {
      const records = await readAll();
      return records.find(r => match(r, query)) ?? null;
    },

    /* Read many */
    find(query: QueryFilter = {}, {skip = 0, limit}: FindOptions = {}) {
      return {
        async toArray() {
          const records = await readAll();
          const matches = records.filter(r => match(r, query));
          return limit === undefined ? matches.slice(skip) : matches.slice(skip, skip + limit);
        },
      };
    },

    /* Count */
    async countDocuments(query: QueryFilter = {}) {
      const records = await readAll();
      return records.filter(r => match(r, query)).length;
    },

    /* Update */
    async updateOne(query: QueryFilter, update: UpdateOperation) {
      const records = await readAll();
      const idx = records.findIndex(r => match(r, query));
      if (idx === -1) {
        return {matchedCount: 0, modifiedCount: 0, acknowledged: true};
      }
      const nextRecord = {...records[idx], ...update.$set};
      const modifiedCount = JSON.stringify(records[idx]) === JSON.stringify(nextRecord) ? 0 : 1;
      records[idx] = nextRecord;
      await writeAll(records);
      return {matchedCount: 1, modifiedCount, acknowledged: true};
    },

    /* Delete */
    async deleteOne(query: QueryFilter) {
      const records = await readAll();
      const idx = records.findIndex(r => match(r, query));
      if (idx === -1) {
        return {deletedCount: 0, acknowledged: true};
      }
      records.splice(idx, 1);
      await writeAll(records);
      return {deletedCount: 1, acknowledged: true};
    },

    /* Optional bulk methods when a service needs them */
    async updateMany(query: QueryFilter, update: UpdateOperation) {
      const records = await readAll();
      let matchedCount = 0;
      let modifiedCount = 0;
      const nextRecords = records.map(record => {
        if (!match(record, query)) return record;
        matchedCount += 1;
        const nextRecord = {...record, ...update.$set};
        if (JSON.stringify(record) !== JSON.stringify(nextRecord)) modifiedCount += 1;
        return nextRecord;
      });
      await writeAll(nextRecords);
      return {matchedCount, modifiedCount, acknowledged: true};
    },

    async deleteMany(query: QueryFilter) {
      const records = await readAll();
      const nextRecords = records.filter(r => !match(r, query));
      const deletedCount = records.length - nextRecords.length;
      await writeAll(nextRecords);
      return {deletedCount, acknowledged: true};
    },
  };

  /* ------------------------------------- */

  function match(record: DbRecord, query: QueryFilter) {
    return Object.entries(query).every(([k, v]) => record[k] === v);
  }

  async function readAll(): Promise<DbRecord[]> {
    try {
      const raw = await fs.readFile(dbFile, 'utf8');
      return JSON.parse(raw);
    } catch {
      return [];
    }
  }

  async function writeAll(records: DbRecord[]) {
    await fs.writeFile(dbFile, JSON.stringify(records, null, 2));
  }
}

Why These Methods?

Nodeblocks handlers call collection-like methods. If your storage technology can implement the method names, return shapes, query operators, and cursor behavior used by your selected features, you can use it without modifying service code.

---

2️⃣ Using the Custom DataStore

Because services use dependency injection, you pass your custom datastore into a service. profileService is typed for MongoDB Collection values and registers profile follow/like routes that this adapter does not support. Compose only the core profile features below instead:

import express from 'express';
import {partial} from 'ramda';
import {features, middlewares, primitives} from '@nodeblocks/backend-sdk';
import {createJsonFileDataStore} from './datastore/jsonFileDataStore';

const {nodeBlocksErrorMiddleware} = middlewares;
const {compose, defService} = primitives;
const {createProfileFeature, getProfileFeature, findProfilesFeature, editProfileFeature, deleteProfileFeature} =
  features;

const profilesDataStore = createJsonFileDataStore('./profiles.json');
const identitiesDataStore = createJsonFileDataStore('./identities.json');

const profileCoreService: primitives.Service = (dataStores, configuration) =>
  defService(
    partial(
      compose(createProfileFeature, getProfileFeature, findProfilesFeature, editProfileFeature, deleteProfileFeature),
      [{dataStores, configuration}],
    ),
  );

express()
  .use(
    profileCoreService(
      {
        profiles: profilesDataStore,
        identities: identitiesDataStore,
      },
      {
        authSecrets: {
          authEncSecret: 'your-encryption-secret',
          authSignSecret: 'your-signing-secret',
        },
        identity: {
          typeIds: {
            admin: '100',
            guest: '000',
            regular: '001',
          },
        },
      },
    ),
  )
  .use(nodeBlocksErrorMiddleware())
  .listen(8089, () => console.log('Server running'));

That's it—the core profile service will now persist its data through your adapter instead of MongoDB.

Note: This adapter supports the core profile routes and pagination only. Do not add profile follow/like features with this implementation: they use MongoDB's $push and $pull operators. Avatar-bearing profiles also need a fileStorageDriver for URL normalization. Other services may require additional methods or query semantics. Implement exactly the collection behavior used by your selected features.

For a fully custom domain service with a single collection, see Creating a Custom Service and inject { reviews: jsonFileDataStore } (or similar) into your own service factory.

---

3️⃣ Implementation Checklist

When implementing a custom datastore, ensure:

1. Method signatures match those expected by the handlers (insertOne, findOne, etc.)
2. Return types contain the required fields (insertedId, modifiedCount, etc.)
3. Async support - methods that perform I/O must return Promises; find() may synchronously return a cursor whose toArray() returns a Promise
4. Error handling - implement proper error handling to prevent crashes
5. Extra methods and operators - add updateMany / deleteMany / findOneAndUpdate, MongoDB-style update operators, or watch() when using features that call them

When these conditions are met, you gain complete freedom to use flat files, SQL databases, cloud functions, or even in-memory mocks for testing.

---

🧪 Testing with Mock DataStore

For unit tests, you can create a simple in-memory implementation that follows the same interface:

export const memoryDataStore = {
  _data: [] as any[],
  async insertOne(doc) {
    this._data.push(doc);
    return {insertedId: doc.id, acknowledged: true};
  },
  async findOne(q) {
    return this._data.find(r => r.id === q.id) ?? null;
  },
  find(q = {}, {skip = 0, limit} = {}) {
    return {
      toArray: async () => {
        const matches = this._data.filter(r => Object.entries(q).every(([k, v]) => r[k] === v));
        return limit === undefined ? matches.slice(skip) : matches.slice(skip, skip + limit);
      },
    };
  },
  async countDocuments(q = {}) {
    return this._data.filter(r => Object.entries(q).every(([k, v]) => r[k] === v)).length;
  },
  async updateOne(q, u) {
    const idx = this._data.findIndex(r => Object.entries(q).every(([k, v]) => r[k] === v));
    if (idx === -1) return {matchedCount: 0, modifiedCount: 0, acknowledged: true};
    this._data[idx] = {...this._data[idx], ...u.$set};
    return {matchedCount: 1, modifiedCount: 1, acknowledged: true};
  },
  async deleteOne(q) {
    const before = this._data.length;
    this._data = this._data.filter(r => !Object.entries(q).every(([k, v]) => r[k] === v));
    return {deletedCount: before - this._data.length, acknowledged: true};
  },
};

This mock datastore is perfect for testing because it:

• Requires no I/O - all operations happen in memory
• Follows the same interface - can be swapped with any real datastore
• Provides fast tests - no database setup or cleanup required

---

🔧 Common Use Cases

Development and Testing

• JSON files - Simple setup, human-readable data
• In-memory storage - Fast tests, no persistence needed
• SQLite - File-based SQL database, no server required

Production Environments

• PostgreSQL/MySQL - Robust, scalable relational databases
• MongoDB - Document database with rich querying
• Redis - High-performance caching and session storage

Cloud and Serverless

• DynamoDB - AWS managed NoSQL database
• Firestore - Google Cloud document database
• CosmosDB - Azure multi-model database

---

✅ Summary

• Interface compliance - Implement the required methods with correct signatures
• Dependency injection - Pass your datastore when creating services
• Correct collection keys - Match each service's expected datastore shape (profiles, identities, etc.)
• Storage flexibility - Use any storage technology that can implement the interface
• Testing support - Create in-memory mocks for fast, reliable unit tests
• Production ready - Switch between development and production datastores without code changes

This abstraction gives you complete freedom to choose the right storage solution for your needs while maintaining compatibility with Nodeblocks services.