mapReduce¶

Definition定义¶

mapReduce¶

The mapReduce command allows you to run map-reduce aggregation operations over a collection.mapReduce命令允许您在集合上运行map-reduce聚合操作。

Aggregation Pipeline as Alternative聚合管道作为替代方案

Aggregation pipeline provides better performance and a more coherent interface than map-reduce, and map-reduce expressions can be rewritten using aggregation pipeline operators, such as $group, $merge, etc.聚合管道提供了比map-reduce更好的性能和更一致的接口，并且可以使用聚合管道运算符（如$group、$merge等）重写map-reduce表达式。

For map-reduce expressions that require custom functionality, MongoDB provides the $accumulator and $function aggregation operators starting in version 4.4. 对于需要自定义功能的map-reduce表达式，MongoDB从4.4版开始提供$accumulator和$function聚合运算符。These operators provide users with the ability to define custom aggregation expressions in JavaScript.这些操作符使用户能够在JavaScript中定义自定义聚合表达式。

For examples of aggregation alternatives to map-reduce operations, see Map-Reduce Examples. 有关映射减少操作的聚合替代方案的示例，请参阅Map-Reduce示例。See also Map-Reduce to Aggregation Pipeline.另请参见对聚合管道的Map-Deduce。

Syntax语法¶

The mapReduce command has the following syntax:语法如下所示：

db.runCommand( {
     mapReduce: <string>,
     map: <string or JavaScript>,
     reduce: <string or JavaScript>,
     finalize: <string or JavaScript>,
     out: <output>,
     query: <document>,
     sort: <document>,
     limit: <number>,
     scope: <document>,
     jsMode: <boolean>,
     verbose: <boolean>,
     bypassDocumentValidation: <boolean>,
     collation: <document>,
     writeConcern: <document>,
     comment: <any>
} )

collation: {
   locale: <string>,
   caseLevel: <boolean>,
   caseFirst: <string>,
   strength: <int>,
   numericOrdering: <boolean>,
   alternate: <string>,
   maxVariable: <string>,
   backwards: <boolean>
}

var mapFunction = function() { ... };
var reduceFunction = function(key, values) { ... };

db.runCommand(
               {
                 mapReduce: <input-collection>,
                 map: mapFunction,
                 reduce: reduceFunction,
                 out: { merge: <output-collection> },
                 query: <query>
               }
             )

Requirements for the map Functionmap函数的要求¶

The map function is responsible for transforming each input document into zero or more documents. It can access the variables defined in the scope parameter, and has the following prototype:

function() {
   ...
   emit(key, value);
}

The map function has the following requirements:

• In the map function, reference the current document as this within the function.
• The map function should not access the database for any reason.
• The map function should be pure, or have no impact outside of the function (i.e. side effects.)
• The map function may optionally call emit(key,value) any number of times to create an output document associating key with value.
• In MongoDB 4.2 and earlier, a single emit can only hold half of MongoDB’s maximum BSON document size. MongoDB removes this restriction starting in version 4.4.
• Starting in MongoDB 4.4, mapReduce no longer supports the deprecated BSON type JavaScript code with scope (BSON type 15) for its functions. The map function must be either BSON type String (BSON type 2) or BSON type JavaScript (BSON type 13). To pass constant values which will be accessible in the map function, use the scope parameter.
  The use of JavaScript code with scope for the map function has been deprecated since version 4.2.1.

The following map function will call emit(key,value) either 0 or 1 times depending on the value of the input document’s status field:

function() {
    if (this.status == 'A')
        emit(this.cust_id, 1);
}

The following map function may call emit(key,value) multiple times depending on the number of elements in the input document’s items field:

function() {
    this.items.forEach(function(item){ emit(item.sku, 1); });
}

Requirements for the reduce Function¶

The reduce function has the following prototype:

function(key, values) {
   ...
   return result;
}

The reduce function exhibits the following behaviors:

• The reduce function should not access the database, even to perform read operations.
• The reduce function should not affect the outside system.
• MongoDB will not call the reduce function for a key that has only a single value. The values argument is an array whose elements are the value objects that are “mapped” to the key.
• MongoDB can invoke the reduce function more than once for the same key. In this case, the previous output from the reduce function for that key will become one of the input values to the next reduce function invocation for that key.
• The reduce function can access the variables defined in the scope parameter.
• The inputs to reduce must not be larger than half of MongoDB’s maximum BSON document size. This requirement may be violated when large documents are returned and then joined together in subsequent reduce steps.
• Starting in MongoDB 4.4, mapReduce no longer supports the deprecated BSON type JavaScript code with scope (BSON type 15) for its functions. The reduce function must be either BSON type String (BSON type 2) or BSON type JavaScript (BSON type 13). To pass constant values which will be accessible in the reduce function, use the scope parameter.
  The use of JavaScript code with scope for the reduce function has been deprecated since version 4.2.1.

Because it is possible to invoke the reduce function more than once for the same key, the following properties need to be true:

• the type of the return object must be identical to the type of the value emitted by the map function.
• the reduce function must be associative. The following statement must be true:

  reduce(key, [ C, reduce(key, [ A, B ]) ] ) == reduce( key, [ C, A, B ] )

• the reduce function must be idempotent. Ensure that the following statement is true:

  reduce( key, [ reduce(key, valuesArray) ] ) == reduce( key, valuesArray )

• the reduce function should be commutative: that is, the order of the elements in the valuesArray should not affect the output of the reduce function, so that the following statement is true:

  reduce( key, [ A, B ] ) == reduce( key, [ B, A ] )

Requirements for the finalize Function¶

The finalize function has the following prototype:

function(key, reducedValue) {
   ...
   return modifiedObject;
}

The finalize function receives as its arguments a key value and the reducedValue from the reduce function. Be aware that:

• The finalize function should not access the database for any reason.
• The finalize function should be pure, or have no impact outside of the function (i.e. side effects.)
• The finalize function can access the variables defined in the scope parameter.
• Starting in MongoDB 4.4, mapReduce no longer supports the deprecated BSON type JavaScript code with scope (BSON type 15) for its functions. The finalize function must be either BSON type String (BSON type 2) or BSON type JavaScript (BSON type 13). To pass constant values which will be accessible in the finalize function, use the scope parameter.
  The use of JavaScript code with scope for the finalize function has been deprecated since version 4.2.1.

out Options¶

You can specify the following options for the out parameter:

out: <collectionName>

out: { <action>: <collectionName>
        [, db: <dbName>]
        [, sharded: <boolean> ]
        [, nonAtomic: <boolean> ] }

out: { inline: 1 }

Required Access¶

If your MongoDB deployment enforces authentication, the user executing the mapReduce command must possess the following privilege actions:

Map-reduce with {out : inline} output option:

• find

Map-reduce with the replace action when outputting to a collection:

• find,
• insert,
• replace

Map-reduce with the merge or reduce actions when outputting to a collection:

• find,
• insert,
• update

The readWrite built-in role provides the necessary permissions to perform map-reduce aggregation.

Restrictions¶

MongoDB drivers automatically set afterClusterTime for operations associated with causally consistent sessions. Starting in MongoDB 4.2, the mapReduce command no longer support afterClusterTime. As such, mapReduce cannot be associatd with causally consistent sessions.

Map-Reduce Examples¶

In the mongo shell, the db.collection.mapReduce() method is a wrapper around the mapReduce command. The following examples use the db.collection.mapReduce() method:

Create a sample collection orders with these documents:

db.orders.insertMany([
   { _id: 1, cust_id: "Ant O. Knee", ord_date: new Date("2020-03-01"), price: 25, items: [ { sku: "oranges", qty: 5, price: 2.5 }, { sku: "apples", qty: 5, price: 2.5 } ], status: "A" },
   { _id: 2, cust_id: "Ant O. Knee", ord_date: new Date("2020-03-08"), price: 70, items: [ { sku: "oranges", qty: 8, price: 2.5 }, { sku: "chocolates", qty: 5, price: 10 } ], status: "A" },
   { _id: 3, cust_id: "Busby Bee", ord_date: new Date("2020-03-08"), price: 50, items: [ { sku: "oranges", qty: 10, price: 2.5 }, { sku: "pears", qty: 10, price: 2.5 } ], status: "A" },
   { _id: 4, cust_id: "Busby Bee", ord_date: new Date("2020-03-18"), price: 25, items: [ { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" },
   { _id: 5, cust_id: "Busby Bee", ord_date: new Date("2020-03-19"), price: 50, items: [ { sku: "chocolates", qty: 5, price: 10 } ], status: "A"},
   { _id: 6, cust_id: "Cam Elot", ord_date: new Date("2020-03-19"), price: 35, items: [ { sku: "carrots", qty: 10, price: 1.0 }, { sku: "apples", qty: 10, price: 2.5 } ], status: "A" },
   { _id: 7, cust_id: "Cam Elot", ord_date: new Date("2020-03-20"), price: 25, items: [ { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" },
   { _id: 8, cust_id: "Don Quis", ord_date: new Date("2020-03-20"), price: 75, items: [ { sku: "chocolates", qty: 5, price: 10 }, { sku: "apples", qty: 10, price: 2.5 } ], status: "A" },
   { _id: 9, cust_id: "Don Quis", ord_date: new Date("2020-03-20"), price: 55, items: [ { sku: "carrots", qty: 5, price: 1.0 }, { sku: "apples", qty: 10, price: 2.5 }, { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" },
   { _id: 10, cust_id: "Don Quis", ord_date: new Date("2020-03-23"), price: 25, items: [ { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" }
])

db.orders.aggregate([
   { $group: { _id: "$cust_id", value: { $sum: "$price" } } },
   { $out: "agg_alternative_1" }
])

db.orders.aggregate( [
   { $match: { ord_date: { $gte: new Date("2020-03-01") } } },
   { $unwind: "$items" },
   { $group: { _id: "$items.sku", qty: { $sum: "$items.qty" }, orders_ids: { $addToSet: "$_id" } }  },
   { $project: { value: { count: { $size: "$orders_ids" }, qty: "$qty", avg: { $divide: [ "$qty", { $size: "$orders_ids" } ] } } } },
   { $merge: { into: "agg_alternative_3", on: "_id", whenMatched: "replace",  whenNotMatched: "insert" } }
] )

Output¶

If you set the out parameter to write the results to a collection, the mapReduce command returns a document in the following form:

{ "result" : "map_reduce_example", "ok" : 1 }

{
    "result" : <string or document>,
    "timeMillis" : <int>,
    "counts" : {
        "input" : <int>,
        "emit" : <int>,
        "reduce" : <int>,
        "output" : <int>
    },
    "ok" : <int>,
}

If you set the out parameter to output the results inline, the mapReduce command returns a document in the following form:

{
    "results" : [
       {
          "_id" : <key>,
          "value" :<reduced or finalizedValue for key>
       },
       ...
    ],
    "ok" : <int>
}

{
    "results" : [
       {
          "_id" : <key>,
          "value" :<reduced or finalizedValue for key>
       },
       ...
    ],
    "timeMillis" : <int>,
    "counts" : {
       "input" : <int>,
       "emit" : <int>,
       "reduce" : <int>,
       "output" : <int>
    },
    "ok" : <int>
}

mapReduce.result¶

For output sent to a collection, this value is either:

• a string for the collection name if out did not specify the database name, or
• a document with both db and collection fields if out specified both a database and collection name.

mapReduce.results¶

For output written inline, an array of resulting documents. Each resulting document contains two fields:

• _id field contains the key value,
• value field contains the reduced or finalized value for the associated key.

mapReduce.timeMillis¶

Available for MongoDB 4.2 and earlier only

The command execution time in milliseconds.

mapReduce.counts¶

Available for MongoDB 4.2 and earlier only

Various count statistics from the mapReduce command.

mapReduce.counts.input¶

Available for MongoDB 4.2 and earlier only

The number of input documents, which is the number of times the mapReduce command called the map function.

mapReduce.counts.emit¶

Available for MongoDB 4.2 and earlier only

The number of times the mapReduce command called the emit function.

mapReduce.counts.reduce¶

Available for MongoDB 4.2 and earlier only

The number of times the mapReduce command called the reduce function.

mapReduce.counts.output¶

Available for MongoDB 4.2 and earlier only

The number of output values produced.

mapReduce.ok¶

A value of 1 indicates the mapReduce command ran successfully. A value of 0 indicates an error.

In addition to the aforementioned command specific return fields, the db.runCommand() includes additional information:

• for replica sets: $clusterTime, and operationTime.
• for sharded clusters: operationTime and $clusterTime.

See db.runCommand Response for details on these fields.

Additional Information¶

• Troubleshoot the Map Function
• Troubleshoot the Reduce Function
• db.collection.mapReduce()
• Aggregation