Measuring performance is a difficult task. All the benchmarks on this page are synthetic in the sense that they test one function repeatedly. Measurements in live systems are much harder to produce reliably. So take the following data with a grain of salt. That being said, a stated feature of the sortedcontainers Module is performance so it would be negligent not to produce this page with comparisons.
The source for all benchmarks can be found under the “tests” directory in the files prefixed “benchmark.” Measurements are made from the min, max, and average of 5 repetitions. In the graphs below, the line follows the average and at each point, the min/max displays the bounds. Note that the axes are log-log so properly reading two different lines would describe one metric as “X times” faster rather than “X seconds” faster. In all graphs, lower is better. Measurements are made by powers of ten: 10, 100, 1,000, 10,000, and 100,000.
Measurements up to 100,000,000 elements have been successfully tested but are impractical for publishing. Only a couple implementations (including sortedcontainers) are capable of handling so many elements. The major limiting factor at that size is memory. Consider the simple case of storing Python’s integers in a SortedList. Each integer object requires ~36 bytes so a hundred million elements will require about four gigabytes of memory. If the implemenation adds significant overhead then most systems will run out of memory. For all datasets which may be kept in memory, sortedcontainers is an excellent choice.
A good effort has been made to find competing implementations. Six in total were found with various list, set, and dict implementations.
Several competing implementations were omitted because they were not easily installable or failed to build.
The most similar module to sortedcontainers is skiplistcollections given that each is implemented in Python. But as is displayed below, sortedcontainers is several times faster and provides a richer API. Often the pure-Python implementation in sortedcontainers is faster even than the C-implementation counterparts. Where it lacks, performance is generally on the same magnitude.
Because sortedcontainers is implemented in pure-Python, its performance depends directly on the Python runtime. SortedContainers was primarily developed, tested and benchmarked on CPython 2.7, specifically build:
Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32
Therefore, for each benchmark below two graphs are shown. The first compares the performance of sortedcontainers running on CPython 2.7 against competing implementations. The second displays a performance comparison of sortedcontainers running on the CPython 2.7, CPython 3.4 and PyPy 2.2 runtimes. In general CPython 3.4 is about half the speed of CPython 2.7. The PyPy 2.2 runtime displays much more variability due to its JIT-ed nature. Once the just-in-time compiler optimizes the code, performance is often several to tens of times faster.
A couple final notes about the graphs below. Missing data indicates the benchmark either took too long or failed. The set operations with tiny, small, medium, and large variations indicate the size of the container involved in the right-hand-side of the operation: tiny is exactly 10 elements; small is 10% of the size of the left-hand-side; medium is 50%; and large is 100%. The sortedcontainers module uses a different algorithm based on the size of the right-hand-side of the operation for a dramatic improvement in performance.
Graphs comparing SortedList performance.
Graphs comparing SortedDict performance.
Graphs comparing SortedSet performance.