Skip to content

8.15 #15426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 73 commits into from
May 16, 2025
Merged

8.15 #15426

Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
3fbefcc
add support for encrypted schemas
baileympearson Feb 27, 2025
19c0132
Merge pull request #15310 from mongodb-js/NODE-6506-add-support-for-e…
vkarpov15 Mar 17, 2025
db8eef7
Add support for encrypted models and discriminators
baileympearson Mar 17, 2025
f1c986c
misc test cleanups
baileympearson Mar 18, 2025
682fb11
update setup script
baileympearson Mar 18, 2025
cda0e26
misc cleanup
baileympearson Mar 18, 2025
edfbdfa
documentation
baileympearson Mar 18, 2025
1a48b98
doc comments
baileympearson Mar 18, 2025
065bc99
use object fromentries
baileympearson Mar 18, 2025
a8f37eb
fix lint in markdown file
baileympearson Mar 18, 2025
53d31a6
throw errors when discriminators have duplicate keys
baileympearson Mar 19, 2025
888c0a8
Remove table testing in favor of hard-coded it blocks
baileympearson Mar 19, 2025
5d3b51f
consistent capitalization in docs
baileympearson Mar 19, 2025
2c02d45
add missing configuration
baileympearson Mar 19, 2025
ecb3f7c
Add support for encrypted discriminators
baileympearson Apr 2, 2025
b7016de
add test for default mongoose connection
baileympearson Apr 2, 2025
14a40ff
add test for default mongoose connection
baileympearson Apr 2, 2025
59af7cf
fix discriminator logic
baileympearson Apr 9, 2025
2fc58da
Merge pull request #15320 from mongodb-js/schema-maps-auto-generate
vkarpov15 Apr 15, 2025
b739793
Merge branch 'master' into csfle
vkarpov15 Apr 22, 2025
db05164
new-test-setup-poc
baileympearson Apr 17, 2025
6c9b5f6
asdf
baileympearson Apr 17, 2025
aee89e8
cleanups
baileympearson Apr 21, 2025
8a2c374
misc fixes
baileympearson Apr 21, 2025
08169cd
make tests run on branch
baileympearson Apr 22, 2025
92bb79b
Merge pull request #15366 from mongodb-js/use-devtools-cluster-tooling
vkarpov15 Apr 22, 2025
a0e4bdd
Add a property to .
baileympearson Apr 11, 2025
a66b479
add documentation
baileympearson Apr 11, 2025
d3bed74
remove .only()
baileympearson Apr 11, 2025
d53600e
last comment of Val's
baileympearson Apr 15, 2025
a9ace50
asd
baileympearson Apr 15, 2025
25ee432
fix documentation lint
baileympearson Apr 18, 2025
523fdd3
Comments
baileympearson Apr 22, 2025
ff411c1
wait for model init
baileympearson Apr 22, 2025
da9f98e
increaste timeouts
baileympearson Apr 22, 2025
db72dbf
Fix CI?
baileympearson Apr 22, 2025
3777b5b
fix tests
baileympearson Apr 22, 2025
1e342ff
obtain CE from mongoose instance, not global driver config
baileympearson Apr 23, 2025
7634218
Merge pull request #15356 from mongodb-js/NODE-6894
vkarpov15 Apr 28, 2025
ce4c3d5
Fix: Add strictFilter option to findOneAndUpdate with tests (#14913)
maazahmed-mangotech May 5, 2025
eab7e49
comments
baileympearson May 5, 2025
460626a
add support for UUUID
baileympearson May 5, 2025
f9e7684
fix non-determinisic FLE test
baileympearson May 5, 2025
411a655
fix lint
baileympearson May 5, 2025
699c82d
Update lib/schema.js
vkarpov15 May 5, 2025
6fc22b6
Merge branch 'csfle' into csfle-review-comments
vkarpov15 May 5, 2025
df21dee
Merge pull request #15404 from mongodb-js/csfle-review-comments
vkarpov15 May 5, 2025
d2c1820
misc final comments
baileympearson May 6, 2025
02f2521
lint and section title
baileympearson May 7, 2025
387c957
Merge pull request #15407 from mongodb-js/csfle-review-comments-2
vkarpov15 May 7, 2025
19fa17b
Add requireFilter for all methods with comprehensive tests, fix befor…
muazahmed-dev May 8, 2025
b826ca2
Merge branch 'Automattic:master' into fix-findOneAndUpdate-empty-filter
muazahmed-dev May 8, 2025
b107906
fix(model): make bulkSave() rely on document.validate() to validate d…
vkarpov15 May 9, 2025
aa64400
Merge branch 'Automattic:master' into fix-findOneAndUpdate-empty-filter
muazahmed-dev May 9, 2025
35018b6
Remove Array.isArray check, add // empty object validation in require…
muazahmed-dev May 9, 2025
458ea69
Merge branch 'fix-findOneAndUpdate-empty-filter' of https://github.co…
muazahmed-dev May 9, 2025
8157b25
types: stricter projection typing with 1-level deep nesting
vkarpov15 May 12, 2025
4ab2e4b
Update query.js
vkarpov15 May 12, 2025
919592d
Merge pull request #15402 from muazahmed-dev/fix-findOneAndUpdate-emp…
vkarpov15 May 12, 2025
e5dcf43
style: fix lint
vkarpov15 May 12, 2025
6049bd6
feat(error): set `cause` to MongoDB error `reason` on ServerSelection…
vkarpov15 May 13, 2025
c289cb4
Merge branch 'master' into vkarpov15/gh-15410
vkarpov15 May 13, 2025
d338909
fix: make bulkSave() continue to use validateSync() for backwards compat
vkarpov15 May 13, 2025
6fbf7d1
Update test/model.test.js
vkarpov15 May 13, 2025
a7e7ce5
Merge branch '8.15' into csfle
vkarpov15 May 13, 2025
7609cb6
Merge pull request #15390 from Automattic/csfle
vkarpov15 May 13, 2025
b4c437a
Merge pull request #15420 from Automattic/vkarpov15/gh-15416
vkarpov15 May 14, 2025
39c9b18
refactor: remove unnecessary awaits in tests
vkarpov15 May 14, 2025
b4d34f4
Merge pull request #15415 from Automattic/vkarpov15/gh-15410
vkarpov15 May 14, 2025
7bb3864
Update types/index.d.ts
vkarpov15 May 14, 2025
f6545db
types: remove incorrect callback type signature for projection
vkarpov15 May 14, 2025
8699cd6
types: avoid including document methods like $assertPopulated Project…
vkarpov15 May 14, 2025
33a22b5
Merge pull request #15418 from Automattic/vkarpov15/gh-15327
vkarpov15 May 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ module.exports = {
'!.*',
'node_modules',
'.git',
'data'
'data',
'.config'
],
overrides: [
{
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/encryption-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
branches: ['master']
pull_request:
branches: [ 'master' ]
branches: [ 'master', 'csfle' ]
workflow_dispatch: {}

permissions:
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,4 @@ notes.md
list.out

data
*.pid
fle-cluster-config.json
138 changes: 134 additions & 4 deletions docs/field-level-encryption.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,141 @@ The resulting document will look similar to the following to a client that doesn

You can read more about CSFLE on the [MongoDB CSFLE documentation](https://www.mongodb.com/docs/manual/core/csfle/) and [this blog post about CSFLE in Node.js](https://www.mongodb.com/developer/languages/javascript/client-side-field-level-encryption-csfle-mongodb-node/).

Note that Mongoose does **not** currently have any Mongoose-specific APIs for CSFLE.
Mongoose defers all CSFLE-related work to the MongoDB Node.js driver, so the [`autoEncryption` option](https://mongodb.github.io/node-mongodb-native/5.6/interfaces/AutoEncryptionOptions.html) for `mongoose.connect()` and `mongoose.createConnection()` is where you put all CSFLE-related configuration.
Mongoose schemas currently don't support CSFLE configuration.
## Automatic FLE in Mongoose

## Setting Up Field Level Encryption with Mongoose
Mongoose supports the declaration of encrypted schemas - schemas that, when connected to a model, utilize MongoDB's Client Side
Field Level Encryption or Queryable Encryption under the hood. Mongoose automatically generates either an `encryptedFieldsMap` or a
`schemaMap` when instantiating a MongoClient and encrypts fields on write and decrypts fields on reads.

### Encryption types

MongoDB has two different automatic encryption implementations: client side field level encryption (CSFLE) and queryable encryption (QE).
See [choosing an in-use encryption approach](https://www.mongodb.com/docs/v7.3/core/queryable-encryption/about-qe-csfle/#choosing-an-in-use-encryption-approach).

### Declaring Encrypted Schemas

The following schema declares two properties, `name` and `ssn`. `ssn` is encrypted using queryable encryption, and
is configured for equality queries:

```javascript
const encryptedUserSchema = new Schema({
name: String,
ssn: {
type: String,
// 1
encrypt: {
keyId: '<uuid string of key id>',
queries: 'equality'
}
}
// 2
}, { encryptionType: 'queryableEncryption' });
```

To declare a field as encrypted, you must:

1. Annotate the field with encryption metadata in the schema definition
2. Choose an encryption type for the schema and configure the schema for the encryption type

Not all schematypes are supported for CSFLE and QE. For an overview of supported BSON types, refer to MongoDB's documentation.

### Registering Models

Encrypted schemas can be registered on the global mongoose object or on a specific connection, so long as models are registered before the connection
is established:

```javascript
// specific connection
const GlobalUserModel = mongoose.model('User', encryptedUserSchema);

// specific connection
const connection = mongoose.createConnection();
const UserModel = connection.model('User', encryptedUserSchema);
```

### Connecting and configuring encryption options

Field level encryption in Mongoose works by generating the encryption schema that the MongoDB driver expects for each encrypted model on the connection. This happens automatically when the model's connection is established.

Queryable encryption and CSFLE require all the same configuration as outlined in the [MongoDB encryption in-use documentation](https://www.mongodb.com/docs/manual/core/security-in-use-encryption/), except for the schemaMap or encryptedFieldsMap options.

```javascript
const keyVaultNamespace = 'client.encryption';
const kmsProviders = { local: { key } };
await connection.openUri(`mongodb://localhost:27017`, {
// Configure auto encryption
autoEncryption: {
keyVaultNamespace: 'datakeys.datakeys',
kmsProviders
}
});
```

Once the connection is established, Mongoose's operations will work as usual. Writes are encrypted automatically by the MongoDB driver prior to sending them to the server and reads are decrypted by the driver after fetching documents from the server.

### Discriminators

Discriminators are supported for encrypted models as well:

```javascript
const connection = createConnection();

const schema = new Schema({
name: {
type: String, encrypt: { keyId }
}
}, {
encryptionType: 'queryableEncryption'
});

const Model = connection.model('BaseUserModel', schema);
const ModelWithAge = model.discriminator('ModelWithAge', new Schema({
age: {
type: Int32, encrypt: { keyId: keyId2 }
}
}, {
encryptionType: 'queryableEncryption'
}));

const ModelWithBirthday = model.discriminator('ModelWithBirthday', new Schema({
dob: {
type: Int32, encrypt: { keyId: keyId3 }
}
}, {
encryptionType: 'queryableEncryption'
}));
```

When generating encryption schemas, Mongoose merges all discriminators together for all of the discriminators declared on the same namespace. As a result, discriminators that declare the same key with different types are not supported. Furthermore, all discriminators for the same namespace must share the same encryption type - it is not possible to configure discriminators on the same model for both CSFLE and Queryable Encryption.

## Managing Data Keys

Mongoose provides a convenient API to obtain a [ClientEncryption](https://mongodb.github.io/node-mongodb-native/Next/classes/ClientEncryption.html)
object configured to manage data keys in the key vault. A client encryption can be obtained with the `Model.clientEncryption()` helper:

```javascript
const connection = createConnection();

const schema = new Schema({
name: {
type: String, encrypt: { keyId }
}
}, {
encryptionType: 'queryableEncryption'
});

const Model = connection.model('BaseUserModel', schema);
await connection.openUri(`mongodb://localhost:27017`, {
autoEncryption: {
keyVaultNamespace: 'datakeys.datakeys',
kmsProviders: { local: '....' }
}
});

const clientEncryption = Model.clientEncryption();
```

## Manual FLE in Mongoose

First, you need to install the [mongodb-client-encryption npm package](https://www.npmjs.com/package/mongodb-client-encryption).
This is MongoDB's official package for setting up encryption keys.
Expand Down
65 changes: 65 additions & 0 deletions lib/drivers/node-mongodb-native/connection.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const pkg = require('../../../package.json');
const processConnectionOptions = require('../../helpers/processConnectionOptions');
const setTimeout = require('../../helpers/timers').setTimeout;
const utils = require('../../utils');
const Schema = require('../../schema');

/**
* A [node-mongodb-native](https://github.com/mongodb/node-mongodb-native) connection implementation.
Expand Down Expand Up @@ -320,6 +321,20 @@ NativeConnection.prototype.createClient = async function createClient(uri, optio
};
}

const { schemaMap, encryptedFieldsMap } = this._buildEncryptionSchemas();

if ((Object.keys(schemaMap).length > 0 || Object.keys(encryptedFieldsMap).length) && !options.autoEncryption) {
throw new Error('Must provide `autoEncryption` when connecting with encrypted schemas.');
}

if (Object.keys(schemaMap).length > 0) {
options.autoEncryption.schemaMap = schemaMap;
}

if (Object.keys(encryptedFieldsMap).length > 0) {
options.autoEncryption.encryptedFieldsMap = encryptedFieldsMap;
}

this.readyState = STATES.connecting;
this._connectionString = uri;

Expand All @@ -343,6 +358,56 @@ NativeConnection.prototype.createClient = async function createClient(uri, optio
return this;
};

/**
* Given a connection, which may or may not have encrypted models, build
* a schemaMap and/or an encryptedFieldsMap for the connection, combining all models
* into a single schemaMap and encryptedFields map.
*
* @returns the generated schemaMap and encryptedFieldsMap
*/
NativeConnection.prototype._buildEncryptionSchemas = function() {
const qeMappings = {};
const csfleMappings = {};

const encryptedModels = Object.values(this.models).filter(model => model.schema._hasEncryptedFields());

// If discriminators are configured for the collection, there might be multiple models
// pointing to the same namespace. For this scenario, we merge all the schemas for each namespace
// into a single schema and then generate a schemaMap/encryptedFieldsMap for the combined schema.
for (const model of encryptedModels) {
const { schema, collection: { collectionName } } = model;
const namespace = `${this.$dbName}.${collectionName}`;
const mappings = schema.encryptionType() === 'csfle' ? csfleMappings : qeMappings;

mappings[namespace] ??= new Schema({}, { encryptionType: schema.encryptionType() });

const isNonRootDiscriminator = schema.discriminatorMapping && !schema.discriminatorMapping.isRoot;
if (isNonRootDiscriminator) {
const rootSchema = schema._baseSchema;
schema.eachPath((pathname) => {
if (rootSchema.path(pathname)) return;
if (!mappings[namespace]._hasEncryptedField(pathname)) return;

throw new Error(`Cannot have duplicate keys in discriminators with encryption. key=${pathname}`);
});
}

mappings[namespace].add(schema);
}

const schemaMap = Object.fromEntries(Object.entries(csfleMappings).map(
([namespace, schema]) => ([namespace, schema._buildSchemaMap()])
));

const encryptedFieldsMap = Object.fromEntries(Object.entries(qeMappings).map(
([namespace, schema]) => ([namespace, schema._buildEncryptedFields()])
));

return {
schemaMap, encryptedFieldsMap
};
};

/*!
* ignore
*/
Expand Down
1 change: 1 addition & 0 deletions lib/drivers/node-mongodb-native/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@
exports.BulkWriteResult = require('./bulkWriteResult');
exports.Collection = require('./collection');
exports.Connection = require('./connection');
exports.ClientEncryption = require('mongodb').ClientEncryption;
1 change: 1 addition & 0 deletions lib/error/serverSelection.js
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ class MongooseServerSelectionError extends MongooseError {
this[key] = err[key];
}
}
this.cause = reason;

return this;
}
Expand Down
47 changes: 47 additions & 0 deletions lib/helpers/model/discriminator.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,51 @@ const CUSTOMIZABLE_DISCRIMINATOR_OPTIONS = {
methods: true
};

/**
* Validate fields declared on the child schema when either schema is configured for encryption. Specifically, this function ensures that:
*
* - any encrypted fields are declared on exactly one of the schemas (not both)
* - encrypted fields cannot be declared on either the parent or child schema, where the other schema declares the same field without encryption.
*
* @param {Schema} parentSchema
* @param {Schema} childSchema
*/
function validateDiscriminatorSchemasForEncryption(parentSchema, childSchema) {
if (parentSchema.encryptionType() == null && childSchema.encryptionType() == null) return;

const allSharedNestedPaths = setIntersection(
allNestedPaths(parentSchema),
allNestedPaths(childSchema)
);

for (const path of allSharedNestedPaths) {
if (parentSchema._hasEncryptedField(path) && childSchema._hasEncryptedField(path)) {
throw new Error(`encrypted fields cannot be declared on both the base schema and the child schema in a discriminator. path=${path}`);
}

if (parentSchema._hasEncryptedField(path) || childSchema._hasEncryptedField(path)) {
throw new Error(`encrypted fields cannot have the same path as a non-encrypted field for discriminators. path=${path}`);
}
}

function allNestedPaths(schema) {
return [...Object.keys(schema.paths), ...Object.keys(schema.singleNestedPaths)];
}

/**
* @param {Iterable<string>} i1
* @param {Iterable<string>} i2
*/
function* setIntersection(i1, i2) {
const s1 = new Set(i1);
for (const item of i2) {
if (s1.has(item)) {
yield item;
}
}
}
}

/*!
* ignore
*/
Expand Down Expand Up @@ -80,6 +125,8 @@ module.exports = function discriminator(model, name, schema, tiedValue, applyPlu
value = tiedValue;
}

validateDiscriminatorSchemasForEncryption(model.schema, schema);

function merge(schema, baseSchema) {
// Retain original schema before merging base schema
schema._baseSchema = baseSchema;
Expand Down
Loading