Sort fix

[mikehquan19]

Overview

Remove manual course sorting to avoid

Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting

Instead, programmatically create indexes to the courses collection on upload so the query's results can be sorted based on expected order. Note that indexes are idempotent, so on every upload, if index already exists, it will not be created again.

Over Nebula-API, course query should explicitily have expected ordering:

opts := options.Find().
    SetSort(bson.D{
        {Key: "subject_prefix", Value: 1},
        {Key: "course_number", Value: 1},
        {Key: "catalog_year", Value: 1},
    })

SetSort defines the ordering required by the query; it does not necessarily mean Mongo will perform an in-memory sort.

If the index satisfies the requested ordering, Mongo can read the results directly in index order and avoid in-memory sort. Otherwise, it can retrieve the matching documents first and then perform in-memory sort.

Use explain("executionStats") to verify whether it uses index order or performs in-memory sort. If the plan has IXSCAN stage and not SORT stages, it uses index order. Otherwise, it has performed blocking in-memory sort on the filtered docs.

Example:

db.courses.explain("executionStats").find({ 
     credit_hours: "3"
}).sort({ 
     subject_prefix: 1, 
     course_number: 1, 
     catalog_year: 1,
})

Note

I added unique indexes on course, which verfied that courses' compound indexing on subject_prefix, course_number, catalog_year.