Attention: all iOS users need to update to Coursicle version 1.0.5 or later to fix this issue. Click here to update.
This week we discovered and fixed a very elusive bug that had been affecting some of our iOS users: under very specific circumstances, users would not receive a notification that a class they needed to take had changed to "Open" (has an open seat), but they did receive the subsequent notification, usually minutes later, that the class had changed to "Closed" (has no open seats).
We had received a few scattered reports of this issue from students since launching the iOS app in March, but each time after consulting our server's logs and talking with the student we concluded that the student must have accidentally dismissed the notification or their phone had been powered off when we sent the notification. It wasn't until a student wrote in this week claiming that 4 times in a row they did not receive an "Open" notification that we began to consider that the issue wasn't our server not sending the notification to the student, but it might be the student's phone that wasn't displaying the notification as it should.
Through close work with this student and another who had reported the issue, we began to narrow in on what was happening.
We figured out that the student's phone actually was receiving and displaying these notifications, but due to a programming error, the notification would disappear from the lock screen ~0.5 seconds after it was first displayed. Unless the student happened to be staring at their phone at the very moment they got this notification, they would completely miss it.
Quite a lot has to happen exactly right for this bug to manifest itself, and it takes a bit of background on how the Coursicle app and iOS works to understand when and why it happens and why it was predominantly affecting "Open" notifications.
High-Level Technical Explanation
We have a bit of code in the app that dismisses all notifications from notification center and removes the badge (the little number on the app's icon indicating how many notifications have been sent since the app was last opened). This code executes when the app is launched and when the app is re-opened so that users don't have to manually dismiss each notification they receive.
The push notifications we send to users' phones also include some data that asks iOS to wake up the Coursicle app in the background and update the class whose status changed to the new status. We do this so that when the user opens the app after seeing the notification, the class already has the correct status listed instead of there being a 1-2 second delay while the app syncs with our server to get the new status. This is an example of "Background App Refresh", which is a setting you may have seen in Coursicle's entry in the "Settings" app. Something important to know about Background App Refresh: to save power, iOS does not wake up apps in the background every time these particular push notifications are received. iOS may decide to wait a couple of minutes or hours before waking up the app in the background so it can refresh its content, or iOS may decide not to wake it up at all, based on a variety of factors such as how much battery and cellular data apps running in the background have been consuming recently, whether the phone is in "Low Power Mode", and of course whether the user has disabled Background App Refresh for the app.
The last bit of background needed to understand this bug is that iOS sometimes terminates apps that are running in the background in order to free up more system resources (namely, RAM) for apps that are currently in use by the user. This could happen hours, days, or weeks after last using an app and happens silently without the user knowing. In fact, apps that are terminated by iOS in this way still show up in the app switcher (the view that comes up when you double tap the home button). Apps are also terminated when a user swipes up on them in the app switcher, but there's one key difference: when the user terminates an app, the app is not eligible to be woken up in the background by a push notification whereas an app terminated by iOS is eligible.
Putting it all together, we can explain the bug: if the user received a push notification from Coursicle while the Coursicle app was in an iOS-terminated state (that is, not user terminated) and iOS determined that apps on the user's phone hadn't been consuming too much battery and cellular data recently, then it would launch the Coursicle app in the background so the app could update the class with the new status. The issue was that our code that clears all of Coursicle's notifications was executing anytime the app was launched from a terminated state regardless of whether the user was opening the app or if iOS was waking it in response to receiving a notification. So, the notification would display for a fraction of a second while the Coursicle app was being launched in the background, and then as soon as it was done launching our clear-all-notifications code would execute and the notification would disappear. Since the notification was displayed for only tenths of a second, many users were missing it entirely.
In most cases, these notifications that were disappearing were "Open" notifications, because users are most commonly being sent "Open" notifications after long periods of waiting, during which they're usually not opening the app. And the longer the app is not used, the more likely it is that iOS has decided to terminate the app during that time, thus satisfying a primary requirement for this bug to occur.
All of these conditions made it very hard to know there even was a bug: only users who rarely open the Coursicle app, who are active enough on their phone that iOS determines it needs to terminate the Coursicle app, and who don't quit the Coursicle app themselves after using it were affected. And even among these users, iOS may not decide to wake up our app for every notification, and in these cases, the bug wouldn't be encountered. Finally, some of these users may have noticed that they weren't getting the occasional "Open" notification, but because it's likely they were receiving most "Open" notifications they didn't write in to tell us because they couldn't discern a pattern to tell us about.
On our side, it was difficult for us to figure out what the issue was, since the actual problem was very different from the symptom most users noticed: users would write in saying that they didn't receive an "Open" notification but that they did get the corresponding "Closed" notification, so we were just checking our server's logs to make sure that we did indeed send them an "Open" notification. In fact, the "Open" notifications were being received but were disappearing far too quickly for the user to see. And any notification could have been affected, it was just that "Open" notifications were affected disproportionately.
Code-Level Technical Details
Originally, we thought perhaps since we set
content-available: 1 (which tells iOS that it should consider waking up our app in the background so it can sync its data with our server) in a push notification payload that also has the
badge fields set, that iOS was accidentally considering our push notification to be "silent" and was thus throttling its delivery to the user semi-randomly (i.e. based on battery, network, and data budgets that are shared by all apps on the user's phone and are only reset every 24 hours). After some more investigation and a discussion with Apple, we realized that these notifications are delivered without throttling. Although one using this configuration should note that while the push notification's alert is displayed immediately to the user when the notification is received by the phone, iOS may still delay the delivery of the push notification to the app running in the background (that is, the typical throttling of Background App Refresh still applies).
As explained in the High-Level Technical Explanation, the bug ended up being that we were dismissing all notifications whenever the app was launched regardless of whether the app had been launched by the user or by the system in response to a push notification with
content-available: 1. Specifically, we had set
applicationIconBadgeNumber = 0 in
didFinishLaunchingWithOptions, which is called anytime the app starts up from a terminated state, even when it's being launched in the background in response to a push notification. We did this because we wanted to dismiss all notifications anytime the user opened the Coursicle app, and putting
applicationIconBadgeNumber = 0 in
applicationWillEnterForeground, something we had done earlier, wasn't handling the case when the app is launched from a terminated state. The fix was simple: we made it so we only set
applicationIconBadgeNumber = 0 in
applicationDidBecomeActive. This is what we should have done in the first place.