Obscure Problems and Gotchas

This has been a very obscure gotcha that I finally overcame recently. It will only have relevance to people who are using PagerDuty on Samsung phones. If you don’t know what PagerDuty is, it’s just a program to get people on a team who are responsible for the welfare of a system to take care of unexpected problems. Those of us with grey hair on our bodies know what an actual pager was back in the day. Some of us even used pagers. Therefore, this is just a virtual pager, same idea, just far more convenient as you can hook it up via push notifications, SMS, phone call, and email to name a few things.

Samsung phones is very broad statement

Yes this is a broad statement, but it’s because all Samsung phones use the same stock operating system from the factory. I am talking about Android with the One UI layer over the top of it. It’s why they work so well (unlike Google’s garbage nexus line🖕. Nexus 4, 5, & 6 were all planned obsolescence pieces of garbage. Yes I am still bitter 😡).

Therefore, for argument’s sake let’s just talk about my phone which is the Samsung Galaxy 22 plus. There is a very good possibility this problem doesn’t persist across all devices, but what I am describing here is definitely a software problem, not a hardware problem which is why I am confident in my deduction.

Obscure problem

• I have revised my PagerDuty settings like a thousand times and still, when I was getting paged in the middle of the night to address a production issue my Samsung’s DND settings were blocking PagerDuty even though I clearly setup PagerDuty as an exception to that rule.
• I made absolutely sure to include PagerDuty’s special contact card on my phone. I have deleted it and re-added it. I make sure it stays synchronized and yet even though that contact is marked as a favorite, it will not break through DND settings.
• I have uninstalled PagerDuty, I have reinstalled PagerDuty, I have reviewed all of the settings too many times. I have reviewed PagerDuty’s documentation. This is not a PagerDuty problem, this is a Samsung problem.
• Essentially I have determined we have entered a situation where there is an impenetrable shield, even though there are instructions to use a spear that can pierce through anything.

Gotcha

The gotcha is simply that the DND settings are too good. They are bugged, but man they sure did stop PagerDuty from waking me up 🤷. Great for my sleep routine, bad for my career.

Proof

It took a while to diagnose this problem because:

1. It was hard to know it wasn’t working. I found out by people saying, “Man that incident last night was terrible, why weren’t you there?”, then I said, “What incident?” – not a good way to find out things aren’t working. I am grateful for my understanding coworkers.
2. I tested it out easily when I had time by doing the following:
   • Put your phone on DND.
   • Create a test service in PagerDuty, make yourself the only responder to not bug anyone else.
   • Send an email to the service to trigger a page.
   • Watch your phone not go off, you will see the notification, but your phone will stay silent.

Work around

Luckily there is a way to reproduce the same effects of the built in DND settings. For this section understand that mode and routine are two different things in the Samsung ecosystem.

Samsung phone routine

1. Turn off DND, make sure the schedule is shut off. Don’t let your phone automatically enter this mode.
2. Familiarize yourself with Samsung’s routines by going to:
   • Settings > Modes and Routines > Routines (at the bottom right)
3. Create a new Routine
4. Name it clearly, something like “My DND” so you don’t get it confused with the actual DND mode.
   • I named mine “My Soft DND”, because it can be broken through.
5. The setup is completely up to you, but here’s what I did:
   • IF Time Period, every day, between [21:00 today] and [09:00 the following day]
   • THEN Sound mode and volume:
     • Sound mode = mute
     • Ringtone = 0%
     • Notifications = 0%
     • System = 0%
6. Finally there is the Set actions section which returns everything back to normal when this routine ends. Therefore, no need to create a counter-routine to this one. Very convenient and simple.

Look below for a screenshot from my phone:

PagerDuty Settings

I am not going to go over all of the settings here because PagerDuty has plenty of instructions on this already. The only thing I am going to call out is you have to make sure that your PD settings raise your volume to %100 when it pages you. This is how you break through the soft DND you just setup.

Happy pages, may you lose all the sleep 👍