Here’s my User Schema:

const userSchema = new mongoose.Schema({
    username: {
        type: String,
        required: true,
        unique: true,
        trim: true,
        lowercase: true,
        minLength: 3,
        maxLength: 30
    },
    password: {
        type: String,
        required: true,
        minLength: 6,
        select: false
    },
    firstName: {
        type: String,
        required: true,
        trim: true,
        maxLength: 50
    },
    lastName: {
        type: String,
        required: true,
        trim: true,
        maxLength: 50
    },
    friends: [{
        type: mongoose.Schema.Types.ObjectId,
        ref: "User",
        sparse: true
    }]
});

And here’s my Signup route (although I have a feeling this isn’t relevant to the question):

userRouter.post('/signup', async (req, res) => {
    const validatedData = signupValidator.safeParse(req.body);

    if (validatedData.success) {
        const {username, firstName, lastName} = validatedData.data;
        const possibleUser = await UserModel.find({username: username})

        if ( possibleUser.length > 0 ) {
            res.json({msg: "User already exists"});
            console.log("user exists. signup aborted.");
            return
        }

        try {
            const hashedPassword = await bcrypt.hash(req.body.password, 10);

            const newUser = await UserModel.create({ username, firstName, lastName, password: hashedPassword });
            
            await AccountModel.create({user: newUser._id, balance: 1 + Math.random()*10000});
            
            const token = jwt.sign({ userId: newUser._id }, JWT_SECRET);
            
            res.status(200).json({
                msg: "User created successfully",
                token: token
            });

        } catch (err) {
            res.json({msg: "error occured during signup", error: err});
            return
        }

    } else {
        res.json({ msg: "Invalid inputs "});
        return
    }
});

I signup one user, no problems. I try to signup another, boom. Everything blows up. I get the following error:

MongoServerError: E11000 duplicate key error collection: paytm-users.users index: friends_1 dup key: { friends: undefined }

After going through some stack overflow questions and the documentation, I have gathered that the issue lies with the friends field, which is null for the first user, and then null for the second one also, which seems to make MongoDB really, really angry. I have encountered the words ‘unique index’ and ‘sparse index’ frequently in the depths of my despair, but haven’t understood fundamentally what it is that is pissing off mongoDB so much, or how to implement this supposed saviour that is ‘sparse index’. Which index is it that I am ‘sparse’-ing?

If you’ve come so far, you may as well be kind and answer my deeper questions:
More fundamentally, Why can’t two separate documents have a field that is similar (by virtue of being empty)? Also, nowhere in my schema have I mentioned that the friends field should be unique. So what is it that is causing the friends field of any document within the collection to be unique ACROSS THE WHOLE COLLECTION?

Khajiit has questions, if you have kindness.

Thank you and god bless you.