#StackBounty: #javascript #node.js #mongodb #express.js #mongoose Basic REST API for manipularing a MongoDB collection, using Node, Exp…

Bounty: 100

I’m practicing back-end programming and NodeJS. As an exercise, I’m attempting to build a REST API for a MongoDB collection. I’m also learning to use the Express and Mongoose middle-wares, so that’s what we’ll use for the server and database respectively. Also practicing async / await to handle promises.

The requirements for this basic REST API and exercise are:

  • Support get and delete on individual resources.
  • Support get and post on the resource collection.
  • Apply generalization and separation of concerns.
  • Protect against Mongo injection.
  • Use async / await to handle promises.

This is the current working implementation:


app.js

const express = require('express')
const mongoose = require('mongoose')
const morgan = require('morgan')

const songRouter = require('./routes/song-router.js')

const mongurl = 'mongodb://localhost:27017/library'
const port = 3000

const app = express()
app.use(morgan('combined'))
app.use('/songs', songRouter)

mongoose.connect(mongurl, () => {
  console.log(`n    >> Mongoose connected to ${mongurl}`)
})

app.listen(port, () => {
  console.log(`n    >> Node listening to port ${port}`)
})

models/song-model.js

const mongoose = require('mongoose')

const song = {
  name: {
    type: String,
    required: true
  },

  author: {
    type: String,
    required: true
  },

  key: String
}

const options = {
  timestamps: true
}

const schema = new mongoose.Schema(song, options)

module.exports = mongoose.model('song', schema)


routes/song-router.js

const express = require('express')

const control = require('../controllers/song-control.js')

const router = express.Router()
router.use(express.json())

router
  .route('/')
  .get(control.getAll)
  .post(control.postOne)

router
  .route('/:songId')
  .get(control.getOne)
  .delete(control.deleteOne)

module.exports = router


controllers/song-control.js (version 1, without generalization)

const songModel = require('../models/song-model.js')

exports.getAll = async (req, res, nxt) => {
  try {
    const allSongs = await songModel.find({})
    res.status(200).json(allSongs)
  } catch (err) {
    nxt(err)
  }
}

exports.getOne = async (req, res, nxt) => {
  try {
    const oneSong = await songModel.findById(req.params.songId)
    res.status(200).json(oneSong)
  } catch (err) {
    nxt(err)
  }
}

exports.postOne = async (req, res, nxt) => {
  try {
    const postedSong = await songModel.create(req.body)
    res.status(200).json(postedSong)
  } catch (err) {
    nxt(err)
  }
}

exports.deleteOne = async (req, res, nxt) => {
  try {
    const deletedSong = await songModel.findByIdAndDelete(req.params.songId)
    res.status(200).json(deletedSong)
  } catch (err) {
    nxt(err)
  }
}


controllers/song-control.js (version 2, first attempt at generalization)

const songModel = require('../models/song-model.js')

exports.getAll = buildMongoFunction('find')

exports.getOne = buildMongoFunction('findById', true)

exports.postOne = buildMongoFunction('create', false)

exports.deleteOne = buildMongoFunction('findByIdAndDelete', true)

function buildMongoFunction (funName, argIsParam) {
  return async (req, res, nxt) => {
    const arg = argIsParam ? req.params.songId : req.body

    try {
      const reply = await songModel[funName](arg)
      res.status(200).json(reply)
    } catch (err) {
      nxt(err)
    }
  }
}



I’m looking forward to all kinds and types of feedback: style, bugs, anti-patterns, ways to do this more concise / maintainable / redeable, conventions, best practices; whatever you think can be improved, please share.

I have some specific questions, but please feel free to ignore these and comment on something else!

  • The generalization of controllers/song-control.js feels hacky. Is there a better way to implement the generalization of that pattern? How’d you do it?
  • How well are these concepts being applied: generalization, separation of concerns? Would you separate responsibilities even further? Or are they too separated? Can something be further generalized?
  • How well is async / await being used?
  • Should I sanitize inputs? Or is enforcing Mongoose models and schemas protection enough against Mongo injections?
  • Seems that Mongoose queries do not return promises. Is the async / await code here doing any actual asynchronous job?
  • What would you recommend doing in a different way?


Get this bounty!!!

#StackBounty: #mysql #node.js #mongodb #mongoose #sequelize.js mongoose Schema to sequelize model

Bounty: 50

I made one app with mongodb (mongoose as ODM) but now I want to work with MySQL (work obligation) so I took Sequelize module for that, but I really don’t understand how to convert my userSchema to user model with all its méthodes (I’m working with passportJs for authentication, so I have some methods that I’m using for example setpassword …)

Here my userSchema (mongoose) that works perfectly.

var mongoose = require('mongoose');
var crypto = require('crypto');
var jwt = require('jsonwebtoken');
var validator = require('node-mongoose-validator');
var Schema = mongoose.Schema;

var userSchema = new Schema({
    name: {
      type: String,
      maxlength: 50
    },
    mail: {
      type: String,
      required: true,
      maxlength: 50,
      index: {
        unique: true
      }
    },
    hash: String,
    salt: String,
    {
      collection: "user"
    }
);

userSchema.methods.setPassword = function(password) {
  this.salt = crypto.randomBytes(16).toString('hex');
  this.hash = crypto.pbkdf2Sync(password, this.salt, 1000, 64).toString('hex');
};

userSchema.methods.validPassword = function(password) {
  var hash = crypto.pbkdf2Sync(password, this.salt, 1000, 64).toString('hex');
  return this.hash === hash;
};

userSchema.methods.generateJwt = function() {
  var expiry = new Date();
  expiry.setDate(expiry.getDate() + 7);

  return jwt.sign({
    _id: this._id,
    mail: this.mail,
    name: this.name,
    exp: parseInt(expiry.getTime() / 1000),
  }, process.env.JWT_SECRET); // secret code from .env
};

module.exports = mongoose.model('user', userSchema);

and here what I’ve tried with sequelize:

 var crypto = require('crypto');
    var jwt = require('jsonwebtoken');

    var User = sequelize.define('user', {
      name: Sequelize.STRING,
      mail: Sequelize.STRING,
      hash: Sequelize.STRING,
      salt: Sequelize.STRING


    });

    User.methods.setPassword = function(password) {
      this.salt = crypto.randomBytes(16).toString('hex');
      this.hash = crypto.pbkdf2Sync(password, this.salt, 1000, 64).toString('hex');
    };

    User.methods.validPassword = function(password) {
      var hash = crypto.pbkdf2Sync(password, this.salt, 1000, 64).toString('hex');
      return this.hash === hash;
    };

    User.methods.generateJwt = function() {
      var expiry = new Date();
      expiry.setDate(expiry.getDate() + 7);

      return jwt.sign({
        _id: this._id,
        mail: this.mail,
        name: this.name,
        exp: parseInt(expiry.getTime() / 1000),
      }, process.env.JWT_SECRET); // DO NOT KEEP YOUR SECRET IN THE CODE!
    };


module.exports = User;

I did not test that because I need to develop one other part, but I need to know that do you think about that, I feel that its full of errors

Thank you in advance


Get this bounty!!!

#StackBounty: #node.js #mongoose how can I customize the output of validation errors in a mongoose schema?

Bounty: 100

I get this syntax from errors when I do an insert like this:

enter image description here

I get this error syntax in the err property. In this case the documento attribute and thecorreo attribute are not unique.
in the same err.message a text string is built where it includes the fields and the errors for which in this case the registry insertion could not be done. How can I do something like this, with the previous example so that the output is like this:

     {
err: {
.
.
.
          “message”: “documento debe ser unicon correo debe ser unicon”
     }
}

this is my full code:

    const mongoose = require('mongoose');
    const uniqueValidator = require('mongoose-unique-validator');

    let Schema = mongoose.Schema;

    let usuarioSchema = new Schema({
        nombres: {
            type: String,
        },
        documento: {
            type: String,
            unique: true,
            required: [true, 'documento debe ser único']
        },
        correo: {
            type: String,
            unique: true,
            required: [true, 'Correo  debe ser único']
        },
        ultimo_inicio_sesion: {
            type: Date,
            default: null
        },
        contrasena: {
            type: String,
            required: [true, 'Contraseña es necesario']
        },
        rol:
        {
            type: String
        }

    })


    usuarioSchema.methods.toJSON = function () {

        let user = this;
        let userObject = user.toObject();

        return userObject;
    }


    usuarioSchema.plugin(uniqueValidator, { message: '{PATH} debe de ser único' });

    module.exports = mongoose.model('usuarios_', usuarioSchema);

How can do it?


Get this bounty!!!

#StackBounty: #javascript #node.js #mongodb #mongoose #async-await Mongoose pass data out of withTransaction helper

Bounty: 50

Introduction

Hey there,

I am trying to pass out data from the mongoose withTransaction callback. Right now, I am using the following code which implements callbacks:

const transactionSession = await mongoose.startSession()
        await transactionSession.withTransaction(async (tSession) => {
            try {
                // MARK Transaction writes & reads removed for brevity

                console.log("Successfully performed transaction!")
                cb(null, "Any test data")

                return Promise.resolve()
            } catch (error) {
                console.log("Transaction aborted due to error:", error)
                cb(error)

                return Promise.reject()
            }
        })

    } catch (error) {
        console.log(error)
        return cb(error)
    }

At the moment, I am using a callback to pass out data from the withTransactioncallback:

cb(null, "Any test data")

However, the problem is that naturally the callback is executed first, before the Promise.resolve() is returned. This means, that (in my case) a success response is sent back to the client before any necessary database writes are committed:

// this is executed first - the callback will send back a response to the client
cb(null, "Any test data")

// only now, after the response already got sent to the client, the transaction is committed.
return Promise.resolve()

Why I think this is a problem:

Honestly, I am not sure. It just doesn’t feel right to send back a success-response to the client, if there hasn’t been any database write at that time. Does anybody know the appropriate way to deal with this specific use-case?

I thought about passing data out of the withTransaction helper using something like this:

const transactionResult = await transactionSession.withTransaction({...})

I’ve tried it, and the response is a CommandResult of MongoDB, which does not include any of the data I included in the resolved promise.

Summary

Is it a problem, if a success response is sent back to the client before the transaction is committed? If so, what is the appropriate way to pass out data from the withTransaction helper and thereby committing the transaction before sending back a response?

I would be thankful for any advice I get.


Get this bounty!!!

#StackBounty: #node.js #mongodb #mongoose #nosql How to use aggregate in mongoosejs with this example

Bounty: 50

I am here to ask a question about mongo aggregate function to achieve this example.

Scenario

I have 3 Mongo Schema i.e House, family and educations which as :

House: {
_id: ObjectId,
state: Number,
houseNumber: Number
}

Family: {
    _id: ObjectId,
    houseId: ObjectId,//ref: house
    name: String,
    gender: String
}

Education: {
    _id: ObjectId,
    familyId: ObjectId,//ref: Family
    level: String, //might be one of ["primary","secondary","higher_secondary"]
}

Expected Output:

{
    state1: {
        primary: {
            male: 3,
            female: 4   
        },
        secondary: {
            male: 4,
            female: 8
        }
    },
    state2: {
        primary: {
            male: 5,
            female: 4   
        },
        secondary: {
            male: 4,
            female: 6
        }
    }
}

I want to group all the education level by gender and then ward.

What I did:

I am newbie in mongo world and recently shifted from sql to no-sql. I had done this:

let edu = await Education.find({level: "primary"}).populate({
      path: "family",
      match: {gender: "male"},
      select: "house",
      populate: {
        path: "house",
        match: {state: 1},
        select: "_id"
      }
    });
let count = (await edu.filter(each => !isEmpty(each.family) && !isEmpty(each.family.house)).length) || 0;

By doing this I get count of male member who has studied primary from state 1. but I cannot afford to call this function one by one for each data.

As requestd the sample data are:

house = [
  {
    _id: AA1,
    state: 1,
    houseNumber: 101
  },
  {
    _id: AA2,
    state: 1,
    houseNumber: 102
  },
  {
    _id: AA3,
    state: 2,
    houseNumber: 201
  }
];

family = [
  {
    _id: BB1,
    houseId: AA1, //ref: house
    name: "John",
    gender: "male"
  },
  {
    _id: BB2,
    houseId: AA1, //ref: house
    name: "Solena",
    gender: "female"
  },
  {
    _id: BB3,
    houseId: AA2, //ref: house
    name: "Efrain",
    gender: "male"
  },
  {
    _id: BB4,
    houseId: AA3, //ref: house
    name: "Naruto",
    gender: "male"
  }
];


education = [
  {
    _id: CC1,
    familyId: AA1, //ref: Family
    level: "primary"
  },
  {
    _id: CC2,
    familyId: AA2, //ref: Family
    level: "secondary"
  },
  {
    _id: CC3,
    familyId: AA3, //ref: Family
    level: "primary"
  },
  {
    _id: CC4,
    familyId: AA4, //ref: Family
    level: "secondary"
  }
];

P.S expected output is not relevant output to the sample data. And ObjectId has been replaced with some unique reference.

Any lead from here guyz?


Get this bounty!!!

#StackBounty: #node.js #angular #mongodb #express #mongoose Server-side pagination using ngx-pagination

Bounty: 200

I got the ngx-pagination module working with all the listings in the GET, but I want the pagination to work server-side too, but I’m unsure how to implement it further than what I have. I’m looking at the documentation for ngx-pagination, but I’m a little bit confused. Here’s what I have.

html

<body [ngClass]="[(this.isOpen && this.mobile) || (this.isOpen && this.tablet) ? 'hideContent' : 'showContent']">
    <div class="loading">
        <!-- <mat-spinner class="loader" *ngIf="isLoading"></mat-spinner> -->

        <ngx-spinner id="loadingIcon" *ngIf="isLoading" type="cog" size="large" color="#3071a9">


            <p class="loadingTitle">Loading...</p>
        </ngx-spinner>

    </div>

    <div class="spacing"></div>
    <div class="container">
        <div class="row no-gutters"
            *ngIf="!this.isOpen && this.mobile || this.isOpen && !this.mobile || !this.isOpen && !this.mobile">
            <div class="class col-md-7"></div>

        </div>

        <!-- /|slice:0:show -->
        <!--; let i = index-->
        <div class="row"
            *ngFor="let auction of posts | paginate: { itemsPerPage: 10, currentPage: p, totalItems: this.posts.count }">
            <div class="col-md-12 col-centered">

                <div class="listingCard" [@simpleFadeAnimation]="'in'">

                    <div class=container>

                        <div class="row">
                            <div class="col-md-3">

                            </div>
                            <div class="col-md-6">
                                <div id="title">{{auction.title}}</div>
                            </div>

                        </div>

                    </div>



                </div>
            </div>

        </div>

    </div>

    <pagination-controls (pageChange)="p = $event"></pagination-controls>

</body>

</html>

<!DOCTYPE html>
<html>

<head>
</head>

<body [ngClass]="[(this.isOpen && this.mobile) || (this.isOpen && this.tablet) ? 'hideContent' : 'showContent']">
    <div class="loading">
        <!-- <mat-spinner class="loader" *ngIf="isLoading"></mat-spinner> -->
        <ngx-spinner id="loadingIcon" *ngIf="isLoading" type="cog" size="large" color="#3071a9">

            <p class="loadingTitle">Loading...</p>
        </ngx-spinner>
    </div>
    <div class="spacing"></div>
    <div class="container">
        <div class="row no-gutters"
            *ngIf="!this.isOpen && this.mobile || this.isOpen && !this.mobile || !this.isOpen && !this.mobile">
            <div class="class col-md-7"></div>
        </div>
        <!-- /|slice:0:show -->
        <!--; let i = index-->
        <div class="row"
            *ngFor="let auction of posts | paginate: { itemsPerPage: 10, currentPage: p, totalItems: this.posts.count }">
            <div class="col-md-12 col-centered">
                <div class="listingCard" [@simpleFadeAnimation]="'in'">
                    <div class=container>
                        <div class="row">
                            <div class="col-md-3">
                            </div>
                            <div class="col-md-6">
                                <div id="title">{{listing.title}}</div>
                            </div>
                        </div>
                    </div>
                    =
                </div>
            </div>
        </div>
    </div>
    <pagination-controls (pageChange)="p = $event"></pagination-controls>
</body>

component

 p: number = 1;

ngOnInit(){
    this.submitListingService.getListings(this.postsPerPage, this.currentPage);
    this.listingService
      .getPostUpdateListener()
      .pipe(takeUntil(this.destroy))
      .subscribe((postData: { listing: Listing[]; postCount: number }) => {
        this.isLoading = false;
        this.totalPosts = postData.postCount;
        this.posts = postData.listing;
        this.filteredPosts = postData.listing;
      });
}

angular service

getListings(postsPerPage: number, currentPage: number) {
    let listings = "Get Listings";
    const params = new HttpParams().set("listings", listings);
    const queryParams = `?pagesize=${postsPerPage}&page=${currentPage}`;
    this.http
      .get<{ message: string; posts: any; maxPosts: number }>(
        "http://localhost:3000/api/listings" + queryParams,
        { params }
      )
      .pipe(
        map(postData => {
          return {
            posts: postData.posts.map(post => {
              return {
               title: post.title,                   
                id: post._id
              };
            }),
            maxPosts: postData.maxPosts
          };
        })
      )
      .pipe(takeUntil(this.destroy))
      .subscribe(transformedPostData => {
        this.posts = transformedPostData.posts;
        this.postsUpdated.next({
          listing: [...this.posts],
          postCount: transformedPostData.maxPosts
        });
      });
  }

app.js

app.get("/api/listings", (req, res, next) => {
  Post.find({ auctionEndDateTime: { $gte: Date.now() } })
      .populate("creator", "username")
      .then(documents => {
        req.params.Id = mongoose.Types.ObjectId(req.params.Id);
        res.status(200).json({
          message: "Auction listings retrieved successfully!",
          posts: documents
        });
      });
});


Get this bounty!!!

#StackBounty: #node.js #angular #mongodb #express #mongoose GET not returning sent message. Only inbox items

Bounty: 300

When a user sends a message, it generates a messageTrackingId. Right now it $unwinds the creatorName as a unique returned value in inbox. I want only one user entry. No duplicates of the same user. Currently though they can send multiple messages if the other user hasn’t responded generating new messageTrackingIds as a result. How can I make the initial sent message appear in the inbox as well so that I can use that messageTrackingId instead of generating new ones? I’ve been stuck on this for awhile so I appreciate any help.

app.get

app.get("/api/messages", (req, res, next) => {
  query = {};
  inbox = false;
  messageId = false;
  if (req.query.recipientId) {
    query = { recipientId: req.query.recipientId };
    inbox = true;

    Messages.aggregate(
      // Pipeline
      [
        {
          $lookup: {
            from: "users", // other table name
            localField: "creator", // name of users table field
            foreignField: "_id", // name of userinfo table field
            as: "creatorName" // alias for userinfo table
          }
        },
        { $unwind: "$creatorName" },
        {
          $match: {
            recipientId: { $eq: req.query.recipientId }
          }
        },

        // Stage 1
        {
          $group: {
            _id: "$messageTrackingId",
            message: { $addToSet: "$message" },
            recipientId: { $addToSet: "$recipientId" },
            creator: { $addToSet: "$creator" },
            messageTrackingId: { $addToSet: "$messageTrackingId" },
            creatorName: { $addToSet: "$creatorName.instagramName" },
            creationDate: { $addToSet: "$creationDate" }
          }
        },

        // Stage 2
        {
          $project: {
            _id: 1,
            message: { $arrayElemAt: ["$message", 0] },
            recipientId: { $arrayElemAt: ["$recipientId", 0] },
            creator: { $arrayElemAt: ["$creator", 0] },
            messageTrackingId: { $arrayElemAt: ["$messageTrackingId", 0] },
            creatorName: { $arrayElemAt: ["$creatorName", 0] },
            creationDate: { $arrayElemAt: ["$creationDate", 0] }
          }
        }
      ]
    )
      //.populate('creator', 'instagramName')

      .then(documents => {
        if (res.subject === "Test") {
          console.log("Nice");
        }
        if (inbox === false && messageId === false) {
          res.status(200).json({
            message: "User's Sent Messages Retrieved!",
            posts: documents
          });
        }
        if (inbox === true) {
          res.status(200).json({
            message: "User's Inbox Retrieved!",
            posts: documents
          });
        }
        if (messageId === true) {
          res.status(200).json({
            message: "Message Chain Retrieved!",
            posts: documents
          });
        }
      });

    //   .exec((err, locations) => {
    //     if (err) throw err;
    //     console.log(locations);
    // });
  } else if (req.query.creator) {
    query = { creator: req.query.creator };
    inbox = false;
    Messages.find(query)
      .populate("creator", "instagramName")
      .then(documents => {
        if (inbox === false && messageId === false) {
          res.status(200).json({
            message: "User's Sent Messages Retrieved!",
            posts: documents
          });
        }
        if (inbox === true) {
          res.status(200).json({
            message: "User's Inbox Retrieved!",
            posts: documents
          });
        }
        if (messageId === true) {
          res.status(200).json({
            message: "Message Chain Retrieved!",
            posts: documents
          });
        }
      });
  } else if (req.query.messageId) {
    query = { messageTrackingId: req.query.messageId };
    console.log(req.query.messageId);
    messageId = true;
    console.log("MESSAGE ID IS TRUE");
    Messages.find(query)
      .populate("creator", "instagramName")
      .then(documents => {
        if (inbox === false && messageId === false) {
          res.status(200).json({
            message: "User's Sent Messages Retrieved!",
            posts: documents
          });
        }
        if (inbox === true) {
          res.status(200).json({
            message: "User's Inbox Retrieved!",
            posts: documents
          });
        }
        if (messageId === true) {
          res.status(200).json({
            message: "Message Chain Retrieved!",
            posts: documents
          });
        }
      });
  }
});

app.post

app.post("/api/messages", checkAuth, (req, res, next) => {
  console.log("Made It")
  messagingTrackingIDValue = "";

  const messaging = new Messages({
    creator: req.userData.userId,
    recipient: req.body.recipient,
    recipientId: req.body.recipientId,
    message: req.body.message,
    //message: req.body.message,
    messageTrackingId: req.body.messageTrackingId,
    creatorName: req.userData.username,
    creationDate: req.body.creationDate
  });

  //saves to database with mongoose
  messaging.save().then(result => {
    if (result.creator !== messaging.creator) {
    } else if (result.creator === req.userData.userId) {
    }
    console.log(result);
    res.status(201).json({
      message: "Message Sent Successfully!",
      postId: result._id
    });
  });
});

angular service

  sendMessage(
    recipient: string,
    message: string,
    creationDate: Date,
    recipientId: string,
    creatorName: string,
    messageTrackingId: string
  ) {
    const messaging: Messages = {
      id: null,
      recipient: recipient,
      message: message,
      creationDate: creationDate,
      creator: null,
      recipientId: recipientId,
      creatorName: creatorName,
      messageTrackingId: messageTrackingId
    };

    this.http
      .post<{ message: string; messagingId: string; creator: string }>(
        "http://localhost:3000/api/messages",
        messaging
      )
      .subscribe(responseData => {
        console.log(responseData);
        const id = responseData.messagingId;
        messaging.id = id;

        console.log("Message sent successfully!");

        //   window.location.reload();
        //  this.posts.push();
        //  this.postsUpdated.next([...this.posts]);
      });
  }




  replyToMessage(
    recipient: string,
    message: string,
    creationDate: Date,
    recipientId: string,
    creatorName: string,
    messageTrackingId: string
  ) {
    const messaging: Messages = {
      id: null,
      recipient: recipient,
      message: message,
      creationDate: creationDate,
      creator: null,
      recipientId: recipientId,
      creatorName: creatorName,
      messageTrackingId: messageTrackingId
    };

    this.http
      .post<{ message: string; messagingId: string; creator: string }>(
        "http://localhost:3000/api/messages",
        messaging
      )
      .subscribe(responseData => {
        console.log(responseData);
        const id = responseData.messagingId;
        messaging.id = id;

        console.log("Message sent successfully!");
      });
  }







  getMessages(recipientId: string) {
    return this.http
      .get<{
        message: string;
        posts: any;
        maxPosts: number;
        messageList: string;
      }>("http://localhost:3000/api/messages?recipientId=" + recipientId)
      .pipe(
        map(retrievedData => {
          return {
            posts: retrievedData.posts.map(post => {
              return {
                creator: post.creator,
                recipientId: post.recipientId,
                creationDate: post.creationDate,
                messageTrackingId: post.messageTrackingId,
                creatorName: post.creatorName,
                id: post._id
              };
            }),
            maxPosts: retrievedData.maxPosts
          };
        })
      );
  }

Here’s an example of the recipient replying to message so sender gets messageTrackingId to use

First message and then reply message. Since the recipient replied, the sender has the messageTrackingId to use for next message to same user.

Made It
{ _id: 5e0674ddd55aae5294370870,
  creator: 5df0014e25ee451beccf588a,
  recipient: 'joe',
  recipientId: '5df00d08c713f722909c99c1',
  message: 'This is the initial message',
  messageTrackingId: '3cb3f5bb-5e17-49a7-8aca-4a61ddd1d847',
  creatorName: 'andy',
  creationDate: 2019-12-27T21:17:17.155Z,
  __v: 0 }
Made It
{ _id: 5e067529d55aae5294370872,
  creator: 5df00d08c713f722909c99c1,
  recipient: 'andy',
  recipientId: '5df0014e25ee451beccf588a',
  message: 'This is the reply message',
  messageTrackingId: '3cb3f5bb-5e17-49a7-8aca-4a61ddd1d847',
  creatorName: 'joe',
  creationDate: 2019-12-27T21:18:33.947Z,
  __v: 0 }

If recipient never replies and sender sends another message this happens:

Made It
{ _id: 5e06756bd55aae5294370873,
  creator: 5df00d08c713f722909c99c1,
  recipient: 'andy',
  recipientId: '5df0014e25ee451beccf588a',
  message: 'This is the first message',
  messageTrackingId: '2077a8e6-844c-4639-a4fa-7aee0b8beaf4',
  creatorName: 'joe',
  creationDate: 2019-12-27T21:19:39.217Z,
  __v: 0 }
Made It
{ _id: 5e06757cd55aae5294370874,
  creator: 5df00d08c713f722909c99c1,
  recipient: 'andy',
  recipientId: '5df0014e25ee451beccf588a',
  message: 'This is another message to same user.',
  messageTrackingId: 'feeb0e20-432e-4c9a-9f59-45913c194edc',
  creatorName: 'joe',
  creationDate: 2019-12-27T21:19:56.257Z,
  __v: 0 }


Get this bounty!!!

#StackBounty: #node.js #mongodb #mongoose #latency Why are mongodb queries to a localhost instance of mongo so much faster than to a cl…

Bounty: 100

I’m using this code to run the tests outlined in this blog post.

(For posterity, relevant code pasted at the bottom).

What I’ve found is that if I run these experiments with a local instance of Mongo (in my case, using docker)

docker run -d -p 27017:27017 -v ~/data:/data/db mongo

Then I get pretty good performance, similar results as outlined in the blog post:

finished populating the database with 10000 users
default_query: 277.986ms
query_with_index: 262.886ms
query_with_select: 157.327ms
query_with_select_index: 136.965ms
lean_query: 58.678ms
lean_with_index: 65.777ms
lean_with_select: 23.039ms
lean_select_index: 21.902ms
[nodemon] clean exit - waiting 

However, when I switch do using a cloud instance of Mongo, in my case an Atlas sandbox instance, with the following configuration:

CLUSTER TIER
M0 Sandbox (General)
REGION
GCP / Iowa (us-central1)
TYPE
Replica Set - 3 nodes
LINKED STITCH APP
None Linked

(Note that I’m based in Melbourne, Australia).

Then I get much worse performance.

default_query: 8353.110ms
query_with_index: 8114.474ms
query_with_select: 3603.191ms
query_with_select_index: 4609.637ms
lean_query: 8455.082ms
lean_with_index: 7885.048ms
lean_with_select: 4209.963ms
lean_select_index: 3798.596ms

I get that obviously there’s going to be some round trip overhead between my computer and the mongo instance, but I would expect that to add 200ms max.

It seems that that round trip time must be being added multiple times, or something completely else that I’m not aware of – can someone explain just what it is that would cause this to blow out?

A good answer might involve doing an explain plan, and explaining that in terms of network latency.

The test code:

(async () => {
  try {
    await mongoose.connect('mongodb://localhost:27017/perftest', {
      useNewUrlParser: true,
      useCreateIndex: true
    })

    await init()

    // const query = { age: { $gt: 22 } }
    const query = { favoriteFruit: 'potato' }

    console.time('default_query')
    await User.find(query)
    console.timeEnd('default_query')

    console.time('query_with_index')
    await UserWithIndex.find(query)
    console.timeEnd('query_with_index')

    console.time('query_with_select')
    await User.find(query)
      .select({ name: 1, _id: 1, age: 1, email: 1 })
    console.timeEnd('query_with_select')

    console.time('query_with_select_index')
    await UserWithIndex.find(query)
      .select({ name: 1, _id: 1, age: 1, email: 1 })
    console.timeEnd('query_with_select_index')

    console.time('lean_query')
    await User.find(query).lean()
    console.timeEnd('lean_query')

    console.time('lean_with_index')
    await UserWithIndex.find(query).lean()
    console.timeEnd('lean_with_index')

    console.time('lean_with_select')
    await User.find(query)
      .select({ name: 1, _id: 1, age: 1, email: 1 })
      .lean()
    console.timeEnd('lean_with_select')

    console.time('lean_select_index')
    await UserWithIndex.find(query)
      .select({ name: 1, _id: 1, age: 1, email: 1 })
      .lean()
    console.timeEnd('lean_select_index')
    process.exit(0)
  } catch (err) {
    console.error(err)
  }
})()


Get this bounty!!!

#StackBounty: #express #mongoose #uuid Mongoose and array of ref UUID's does not convert

Bounty: 50

When using the library mongoose-uuid, I am able to setup UUID types for my schemas, so when I read the data it is in string (utf-8) format and when I save the data it is in UUID ObjectID BSON type 4 format. This works great with top level or flat direct values and ref definitions in my schema. However, when I have a UUID’s in an array of ref’s in a schema, the array saves to the database correctly, However when it is presented it is in its raw type. Based on the example below you can see scope_id is presented in the right format but the entitlements are not.

Here are the versions I am using:
mongoose-uuid – 2.3.0
mongoose – 5.5.11

I have tried modifying the library (mongoose-uuid) by changing the getter and converting the value, however, when I do so, it works when presenting but fails when it saves to the database. This is most likely due to the fact that the value is converted or casted before saving to the database.

Here is an example schema

    {
      "code": {
        "type": String,
        "required": true
      }, 
      "scope_id": {
        "type": mongoose.Types.UUID,
        "ref": "scopes"
      },
      "entitlements": [{
        "type": mongoose.Types.UUID,
        "ref": "entitlements"
      }]
    }

Example actual response

{
    "entitlements": [
        "zMihi1BKRomM1Q41p7hgLA==",
        "ztOYL7n1RoGA6aoc0TcqoQ=="
    ],
    "code": "APPUSR",
    "scope_id": "b8f80c82-8325-4ffd-bfd7-e373a90e7c45",
    "id": "32e79061-e531-45ad-b934-56811e2ad713"
}

Expected Response

{
    "entitlements": [
        "ccc8a18b-504a-4689-8cd5-0e35a7b8602c",
        "ced3982f-b9f5-4681-80e9-aa1cd1372aa1"
    ],
    "code": "APPUSR",
    "scope_id": "b8f80c82-8325-4ffd-bfd7-e373a90e7c45",
    "id": "32e79061-e531-45ad-b934-56811e2ad713"
}


Get this bounty!!!

#StackBounty: #javascript #node.js #mongoose #array.prototype.map Can not add element to object with map()

Bounty: 50

I have a nodejs express application with an api to return data from a mongodb database. This is my mongoose model:

const bookingSchema = new mongoose.Schema({
  timestamp: {
    type: Date,
    default: Date.now,
    required: true
  },
  tags: {
    type: [String],
    required: true
  },
  amount: {
    type: Number,
    required: true
  },
  type: {
    type: String,
    required: true,
    enum: ['expense', 'income']
  }
})

When I’m calling the api with the path /api/bookings/listbymonth/2019/1 this function inside the backend gets called:

const bookingsListByMonth = (req, res) => {
  const year = ("0000" + req.params.year).slice(-4)
  const month = ("0000" + req.params.month).slice(-2)
  const dateOfMonth = `${year}${month}01`
  const start = moment(dateOfMonth).startOf("month")
  const end = moment(dateOfMonth).endOf("month")

  bookingMongooseModel
    .find({
      timestamp: {
        $gt: start,
        $lt: end
      }
    })
    .sort({ timestamp: 1 })
    .exec((err, bookings) => {
      if (!bookings) {
        return res
          .status(404)
          .json({
            "message": "booking not found"
          })
      } else if (err) {
        return res
          .status(404)
          .json(err)
      }
      res
        .status(200)
        .json(processBookings(bookings));
    })
}

Instead of simply returning the json data, I want to preprocess the data and make a nice timestamp and currency field. That’s why the json data runs through an additional processBookings function. For testing I tried to add another field timestamp2: 123:

const processBookings = (bookings) => {
  console.log("Bookings unsorted: n" + bookings + "n")

  const mainTags = [
    "Essen",
    "Essen gehen",
    "Notwendiges",
    "Luxus",
  ]

  let bookingsProcessed = []

  mainTags.forEach((tag) => {
    let singleTagBookings = bookings.filter(
      booking => booking.tags.includes(tag)
    )

    singleTagBookings.map((item) => {
      item.timestamp2 = "123"
      return item
    })

    let message = null;
    if (singleTagBookings.length === 0) {
      message = "No bookings found";
    }

    bookingsProcessed.push({
      name: tag,
      bookings: singleTagBookings,
      message: message
    })
  });

  console.log("Bookings sorted:")
  bookingsProcessed.forEach((item) => {
    console.log(item)
  })

  return bookingsProcessed
}

The objects in the bookings array should have another property timestamp2: "123", but they don’t. Here’s the output:

Bookings unsorted: 
{ tags: [ 'Luxus', 'voluptatem', 'atque', 'qui', 'sunt' ],
  _id: 5cb2c9e1ff6c9c6bef95f56f,
  timestamp: 2019-01-06T08:53:06.945Z,
  amount: 68.02,
  type: 'expense',
  __v: 0 },{ tags: [ 'Essen gehen', 'ut', 'unde', 'et', 'officiis' ],
  _id: 5cb2c9e1ff6c9c6bef95f56e,
  timestamp: 2019-01-09T20:35:06.411Z,
  amount: 33.77,
  type: 'income',
  __v: 0 }

Bookings sorted:     
{ name: 'Essen', bookings: [], message: 'No bookings found' }
{ name: 'Essen gehen',
  bookings: 
   [ { tags: [Array],
       _id: 5cb2c9e1ff6c9c6bef95f56e,
       timestamp: 2019-01-09T20:35:06.411Z,
       amount: 33.77,
       type: 'income',
       __v: 0 } ],
  message: null }
{ name: 'Notwendiges',
  bookings: [],
  message: 'No bookings found' }
{ name: 'Luxus',
  bookings: 
   [ { tags: [Array],
       _id: 5cb2c9e1ff6c9c6bef95f56f,
       timestamp: 2019-01-06T08:53:06.945Z,
       amount: 68.02,
       type: 'expense',
       __v: 0 } ],
  message: null }

As in the comments suggested I tried to use let bookings = [ {tags: ["Essen"]}]; as test data. Here it works. The output is:

Bookings unsorted: 
[object Object]

Bookings sorted:
{ name: 'Essen',
  bookings: [ { tags: [Array], timestamp2: '123' } ],
  message: null }
{ name: 'Essen gehen',
  bookings: [],
  message: 'No bookings found' }
{ name: 'Notwendiges',
  bookings: [],
  message: 'No bookings found' }
{ name: 'Luxus', bookings: [], message: 'No bookings found' }

So I guess it has something to do with my mongoose model restricting to add any additional field. However If I put

console.log("EXTENSIBLE " + Object.isExtensible(bookings))
res
  .status(200)
  .json(processBookings(bookings));

into my bookingsListByMonth function I get:

EXTENSIBLE true

So in theory I should be able to add something to the bookings object?

As a workaround I added the timestamp2 field to my mongoose model:

const bookingSchema = new mongoose.Schema({
  timestamp: {
    type: Date,
    default: Date.now,
    required: true
  },
  timestamp2: {
    type: String,
    default: null
  },
  tags: {
    type: [String],
    required: true
  },
  amount: {
    type: Number,
    required: true
  },
  type: {
    type: String,
    required: true,
    enum: ['expense', 'income']
  }
})

This works, however it adds an additional useless data field into my database. How can I modify the bookings json object returned from the mongodb? If I can’t modify it because it is a mongoose model, how can I make a copy that is editable?


Get this bounty!!!