Table of Contents

Building Alerting Rotations

Morgan Wynnes Updated by Morgan Wynnes

Read Time: 2 mins

Alerting rotations are used to define periods of time in which network alerts are sent to certain users within Sonar. A network alert would be triggered as a result of a network event that occurs. Network events are considered to be when a device moves into a warning state, a down state, or begins flapping. Alerting rotations can be used to setup an on call rotation for specific users, during which they would be alerted about any network events that happen.

Device States Overview

Device State

What This Means

ooo

Good

The device is working normally

ooo

Warning

The device has either: violated an ICMP threshold defined in a monitoring template, an SNMP query has failed due to a configuration error (i.e. an invalid OID), or an SNMP OID threshold has been violated

ooo

Down

The device has failed to respond to ICMP (if ICMP monitoring is enabled), or it has failed to respond to SNMP (if SNMP OIDs are configured)

Flapping

The device is moving between states in a quick sequence

Creating an Alerting Rotation

Alerting rotations are set up in two different pieces; first, you need to configure the alerting rotation itself, and then you need to define which day(s)/time(s) of the week the alerting rotation should be active on. An in-depth overview on setting up both of these pieces is provided below.

1. Alerting Rotation Setup

To set up an alerting rotation:

  1. Navigate to the Alerting Rotations screen by clicking Settings menu > Monitoring > Alerting Rotations
  2. Click "Create Alerting Rotation"
  3. You will be presented with the Create Alerting Rotation window, where you can set the details for this rotation - an explanation of the fields you will see here and how to use them is provided below:
  • Enabled: This is what determines if the alerting rotation is currently being used, or if it is disabled.
  • Name: A descriptive name for the rotation.
  • Start: The date this rotation should start.
    This can be set for a future date, but the rotation will not be applied and users will not start receiving notifications until that date.
  • Users: The users that are part of this alerting rotation and who will therefore receive notifications of the network events.
  • Repetitions: If Infinite Repetitions is checked, this rotation will repeat forever; if unchecked, this rotation will repeat until the Repetitions value is met and will then stop.
  • Weeks Between Repetitions: This is how often the rotation repeats; for example, if 0 it will repeat every week, if 1 it will repeat every other week, etc.
  • Warning Minutes Before First Alert: The number of minutes a device can be in warning state before notifications are sent. This is ignored if the device is in warning due to violating an SNMP threshold, as the thresholds themselves have a time parameter.
  • Warning Minutes Between Successive Alerts: The amount of time in minutes that should elapse before the next alert is sent out for a device in warning state.
    If you are only looking to receive alerts for the initial state changes, set this to a significantly higher number, such as 999999.
  • Down Minutes Before First Alert: The number of minutes a device must be in down state before an alert is sent out.
  • Down Minutes Between Successive Alerts: The amount of time in minutes that should elapse before the next alert is sent out for a device in down state.
    If you are only looking to receive alerts for the initial state changes, set this to a significantly higher number, such as 999999.
  • Accounts: If All Accounts is checked, this rotation will be applied to all accounts; otherwise, you can specify which account types and/or groups this rotation should be applied to through the Account Types and Account Groups fields.
  • Network Sites: If All Network Sites is checked off, this rotation will be applied to all network sites; otherwise, you can specify which particular network site(s) this rotation should be applied to through the Network Sites field.
  • Inventory Models: If All Inventory Models is selected, this rotation will be applied to all inventory models; otherwise, you can specify which particular inventory model(s) this rotation should be applied to through the Inventory Models field.

2. Alerting Rotation Days

Once you have set up an alerting rotation, you can set up the days that the alerting rotation should be active for. To set up alerting rotation days:

  1. Navigate to the Alerting Rotation Days screen by clicking Settings menu > Monitoring > Alerting Rotation Days
  2. Click "Create Alerting Rotation Day"
  3. You will be presented with the Create Alerting Rotation Day window, where you can define the day and start/end times for the rotation. An overview of the fields contained within this window is provided below:
  • Alerting Rotation: Any alerting rotations that have been configured will appear in the list here - select the alerting rotation that you wish to define the days for.
  • Day: The day that this alerting rotation should be active on.
  • Start Time: The time that this particular alerting rotation should begin.
  • End Time: The time that this particular alerting rotation should end at.
You can specify only certain times in particular days of the week, or you can set it up to run all day everyday. To set up the alerting rotation to run 24x7, add each day as an alerting rotation day and set the start time as 00:00 and the end time as 23:59.

Alerting Rotation Notifications

Currently, notifications for alerting rotations are sent out via email. There is a preconfigured triggered email and associated email message within Sonar V2 instances titled "Inventory Item Status Updates" - this is the email that will automatically be sent out as a notification. By default, this email will be enabled, but this setting can be changed through the email's associated "Edit" menu within the Triggered Emails screen of your instance.

In order for the alerting rotation triggered email to fire as a notification, the following conditions must be met:

1. The alerting rotation must be configured
2. The rotation must have at least one alerting rotation day set for it
3. There must be at least one user actively assigned to the rotation
4. The Inventory Item Status Updates triggered email must be enabled

The preconfigured email message for the Inventory Item Status Updates email is housed under the Email Messages screen of your instance. To view the content of the email message, simply click on the message bubble icon next to the email, and then click on "View" from the panel that appears below - the email message content will appear in a new window. If you need to modify the message, you can do so by clicking on "Edit" for the email message.

For more information on triggered emails and how to modify them, check out our Triggered Emails: Setup article here.

Claiming a Device & Suppressing Alerts

Although the main use of alerting rotations is to be notified of any network events, there are certain situations where you might want to suppress further alerts from being sent out - this can be done in two ways:

1. Claiming a Device

When an alert is generated and sent to a user, they have the ability to suppress further alerting for the item by claiming that device. Claiming a device is done individually for each inventory item, and can be done by clicking "Claim", as seen in the image above. Once a user claims an item, the user will be displayed under the Claimed By column, and alerting for the item will be suppressed until the item is unclaimed, or the monitoring for the item returns to a Good state.

2. Suppress Alerts

Through the "Suppress Alerts" button you can halt all alerting rotation notifications from being sent out. Clicking this button will open a new window where you will specify the date and time you would like to suppress alerting until - this will effectively prevent everyone in the system from receiving any configured alerts until the specified date/time, at which point the notifications will resume. Typically, you would only use this button when you have some sort of global monitoring outage, and want to suppress notifications as you are currently aware and working to resolve the issue.

Alerting Rotation Example Build Out

Let's say that we have 3 technicians (Jerry, George, and Jackie) dedicated for our company's after-hours network monitoring, operating in a rotating on-call basis. Each technician covers one week at a time, working Monday through Friday between the hours of 5 pm to 8 am the following morning (so each technician works one week and then is off for the following two weeks). Below we will walk through setting up the alerting rotations for each technician, as well as the alerting rotation days to accommodate our scenario.

Part 1: Configuring the Alerting Rotations

Since these technicians are all responsible for the same job, most of the alerting rotation settings are the same between them. All of the alerting rotations are enabled so that they are ready to be activated on our specified start dates, and are set to repeat infinitely with 3 weeks in between each repetition. With 3 weeks in between repetitions, we can stagger each technician's start date by 1 week, to create the weekly on-call rotation. Our alerts are also for monitoring network sites only, including all inventory models, and do not include customer account equipment. Since our main concern is with an item moving into a down state, we have configured the 'minutes before first alert' and 'minutes between successive alerts' fields accordingly.

For Jerry, we have given his rotation a descriptive name, selected him as the user, and set his start date as November 2nd, 2020. This means that Jerry's rotation will be the first to start out of the 3 technicians, so he will be on-call from November 2nd through to November 6th, at which point George's rotation week will begin.

George's rotation has been given a descriptive name, his instance user has been selected, and he has a start date of November 9th, 2020; so George will start his first rotation the week of November 9th - 13th. After his rotation week ends on the 13th, his next rotation will not be until November 30th - December 4th, since there are 3 weeks between repetitions.

The rotation for Jackie has also been named accordingly, has her user selected, and a start date of November 16th, 2020. This means Jerry will be on-call the first week (November 2nd - 9th), George will be on-call the second week (November 9th - 13th), and Jackie will be on-call the third week (November 16th - 20th). Once Jackie's rotation week is complete, she will not be on-call again until December 7th - 11th.

Once the technicians' alerting rotations have been saved, they will appear within the alerting rotations table. With these configured rotations, the flow will be as follows:

  • November 2nd - 6th 2020: the first week of the on-call rotation, which Jerry will be on-call for
  • November 9th - 13th, 2020: the second week of the on-call rotation, which George will be on-call for
  • November 16th - 20th, 2020: the third week of the on-call rotation, which Jackie will be on-call for
  • November 20th - 27th, 2020: the on-call rotation will start over, since we have it set to infinite repetitions and there are 3 weeks between each repetition; Jerry will be on-call for his 2nd week

Part 2: Adding the Alerting Rotation Days

Now that we have our alerting rotations configured, we need to define the days and times of the week that they will be active for. Since our technicians operate outside of business hours, we need to set up the alerting rotation days to cover 5 pm to 8 am the following morning.

To account for these hours, we will need to set up 2 alerting rotation days for each day of the week - one to cover the time between midnight and 8 am of the day, and one for 5 pm to midnight of the same day. Therefore, we will need to set up 10 alerting rotation days for each technician to accommodate this schedule.

Starting with Jerry's rotation, we will set up 2 alerting rotation days for the first day, Monday:

  1. One alerting rotation day for Monday from midnight to 8 am
  1. And one alerting rotation day for Monday from 5 pm to midnight

Once we have finished creating Jerry's alerting rotation days for the remaining days (Tuesday through Friday), we will repeat this process for George and Jackie as well.

When we are finished setting up all of our alerting rotation days for George and Jackie, our on-call alerting rotation schedule will now be complete and take effect starting Monday, November 2nd, 2020, repeating infinitely.

How did we do?

Pollers: General Overview, Deployment Strategy, Build Out & Setup

Contact