Some days ago, we released the new version 2.19 of the privacyIDEA authentication system. As explained in the release notes, we worked on improving the performance of authentication requests and managed to reduce the time needed to handle one authentication request by up to 71%! If such claims make you suspicious, we totally understand your concerns. This is why we will explain our benchmark approach in this blog post, so you can see for yourself where our numbers come from.
The first question we asked ourselves was: What exactly do we want to find out? As we worked a lot on optimizing the LDAP resolver, we first wanted to know if privacyIDEA 2.19 by itself processes authentication requests faster than privacyIDEA 2.18.1. The next question was whether performance can be further improved by enabling the new user cache feature of privacyIDEA 2.19. Here, we wanted to differentiate between a worst-case scenario with an empty cache and a best-case scenario with an already-populated cache. For each scenario, we wanted to get an idea of the time that privacyIDEA needs to handle an incoming authentication request. Having cleared our objective, the next step was creating a suitable lab environment that resembles the real world as closely as possible to run our benchmarks in.
Our lab environment
We created a lab environment as follows. First, we set up an Univention Corporate Server and added 1000 users to its directory, which we simply called user000 to user999, as well as a privacyIDEA service account. In the same network, we prepared two Ubuntu 16.04 virtual machines and installed privacyIDEA 2.18.1 on the first and privacyIDEA 2.19-dev5 on the second machine. On both instances, we added a realm with a LDAP resolver connecting to the Univention Corporate Server via LDAPS.
In order to keep our environment as close to the real world as possible, we would need to enroll HOTP or TOTP tokens for our 1000 users. However, we ultimately decided against it. For one, we suspected that the time spent calculating and checking the next OTP value on the server is relatively small in comparison to the time spent communicating with the LDAP server. Enrolling real OTP tokens would also require us to keep track of secrets and counter values on our benchmark client, which would complicate our setup a lot. To keep things simple, we instead decided to enroll one simple password (SPASS) token for each user.
Our benchmarking approach
Now, we had set up two privacyIDEA instances with 1000 tokens. The next question was: How exactly do we now measure the performance of one authentication request? We decided to settle on the following approach: One benchmark consists of 2000 successful authentication requests, performed one after another for the users user000 to user999. This means that each user is authenticated twice during one benchmark. For each authentication request, we measured the time from sending the request until receiving the response using a simple benchmarking script in Python based on python-requests. We copied the script to the virtual machines and performed all authentication requests against https://localhost in order to exclude the network delay from our measurements. Running the script then produces 2000 measurements of response time, of which we computed the median response time.
We decided to measure the response times for the following scenarios:
- Scenario #1: Authentication against privacyIDEA 2.18.1
- Scenario #2: Authentication against privacyIDEA 2.19, with the user cache feature disabled
- Scenario #3: Authentication against privacyIDEA 2.19, with the user cache enabled and initially empty
- Scenario #4: Authentication against privacyIDEA 2.19, with an already-populated user cache. This means
that the user cache contains valid 1000 entries, one for each user fromuser000to
user999.
The scenarios 2, 3 and 4 were carried out on the privacyIDEA 2.19 machine. For scenarios 3 and 4, we enabled the user cache with a timeout of one day (corresponding to 86400 seconds).
Our results
Now, we had everything in place to start our benchmarks! In total, running the benchmark for all four scenarios took roughly one hour and we obtained the following results.
| scenario# | description | median response time |
|---|---|---|
| #1 | privacyIDEA 2.18.1 | 716ms |
| #2 | privacyIDEA 2.19, disabled user cache | 306ms |
| #3 | privacyIDEA 2.19, enabled but initially empty user cache | 268ms |
| #4 | privacyIDEA 2.19, enabled and populated user cache | 203ms |
Interesting! According to our measurements, an update to privacyIDEA 2.19 alone seems to reduce the median response time by roughly 57% (Scenario #2), even without enabling the user cache. This speedup can probably be attributed to some performance improvements in the LDAP resolver (see issues 655 and 664).
Furthermore, if the user cache is enabled and fully populated (Scenario #4), the median response time is reduced by another 33%. In comparison to privacyIDEA 2.18.1, this corresponds to a reduction by 71%. Of course, this models a best-case scenario in which the LDAP server does not need to be queried any more at all. This may not be the case in the real world, e.g. if an otppin=userstore policy is enabled.
Scenario #3 is quite interesting, as the user cache is initially empty and is subsequently populated during the first 1000 authentication requests. For the second round of 1000 authentications, privacyIDEA can rely on the user cache instead of querying the LDAP server. We can also observe this if we plot our measurements:
The horizontal axis denotes our measurements and the vertical axis gives the median response time in milliseconds. We can observe that the median response time drops dramatically after the thousandth measurement, from which we can conclude that the user cache does have a measurable positive impact on the performance of authentication requests.
Of course, our benchmark is not perfect and leaves room for improvement due to multiple reasons. Firstly, the measurements also include the round-trip time between LDAP server and privacyIDEA instance, which significantly depends on the network setup. Secondly, we have only enrolled SPASS tokens and no real OTP tokens. Thirdly, we have not performed any concurrent requests and cannot, for example, say anything about the maximum number of authentication requests per second. Finally, two authentication requests by the same user are several minutes apart. If the same user sends two authentication requests during the timespan configured by the cache timeout option of the LDAP resolver (which defaults to 2 minutes), privacyIDEA queries an in-memory cache, which may be even faster than the query to the local database performed by the user cache.
However, we believe that our benchmark shows that privacyIDEA 2.19 improves the performance of authentication requests quite significantly even without the user cache. Additionally, enabling the user cache may bring significant performance improvements in case a large number of users are expected to send authentication requests over a large timespan. Finally, we noticed that the LDAP connection in our test setup is quite fast (a LDAP search takes just unter 30 milliseconds), so the user cache may provide an even better speedup in case of slower LDAP servers or connections. You are welcome to try it out for yourself! If you have any further questions, pleask ask them on our community site.
Start the discussion at community.privacyidea.org