Security Awareness: Metrics, or How I Learned to Keep My Job
I've spent several years doing awareness things. All the things, and all the stuff. Reactively sending memos about phishing emails. Hanging up posters. Writing (internal) blog posts. Typing up newsletters. Emailing newsletters. Emailing reminders. Emailing policy reminders. Lots of "Don't do this, do this" statements. *sigh*
But no one cared. I wanted to prove to my leadership that security awareness programs are crucial, and maaaaaybe I can get a little budget to do more stuff. So I began to consider the viewpoint of my leaders. I wasn't telling them a story.
So, what was my story? How were all the "things" I was doing providing value? What data was I collecting, if any? I decided to start over (not at a new company, just, with my program). No one wonder they didn't appreciate what I was doing. I needed awareness of the awareness.
I needed a security awareness program plan, first and foremost. I needed to know what that program plan would focus on. I needed to know what risks were important to my leadership. I could then pinpoint behaviors that needed to change, and then my metrics began presenting themselves. Which in turn allowed my program plan to form. Which made my leadership smile. Which was a win.
OKAY SO WHAT METRICS?? As a CISO/CTO/CIO/Director/Analyst etc., I would want to know how we are measuring. Some ideas to consider:
- Phishing training: ditch the click rate. It's vanity. Focus on report rate. The more users you have reporting, the stronger your...
- Incident Response rate: You want to provide unfathomable value to your team and leadership? Show that awareness efforts to change behavior (see above) creates efficiency for your IR team. Better and more data in means they can respond quicker. More in = quicker response...Show percentage change in time to respond.
- Compliance Training completion rate: ugh. You need to report on this. But do not get caught up on this. You are in one of two boats: 98%+, or noonereallycares and you can't force people to do it. If you're not in a regulated industry, focus on a metric that shows the increase in completion amongst poor performing departments. Showing a larger percentage increase like this is more powerful that showing a small 1-2% increase in overall completion.
- Hours Spent Learning: I began using this metric because it was not expected...and leadership loved it. I tracked everyone FTE hour spent at an event, a live training, online training, etc. This is mostly for October's NCSAM, but when you can provide a number like "500+ hours spent learning about cyber risks", that's a hell of a baseline and people will remember that.
- Helpdesk Tickets/Calls: Find a way to track this. Some great ones are: lost/stolen devices, malware infections/mo, etc. Especially if you are doing a campaign focusing on reporting incidents, track the months up to it for a baseline and then show the percentage increase. That right there is good ole behavior change numbers.
- Intranet/Newsletter clicks/stats: avoid this metric if you can. I was managing an intranet site for a 100k+ userbase and we were averaging maaaaaaybe 700 visits per month. That's not a metric you want to report on, unless your point is to stop doing blog posts. No one is reading your blog posts/newsletters except your security peers (maybe). Instead, create a reason for users to visit the site - and track that change in site visits. Maybe you have decided to use funny, engaging videos to supplement your off-the-shelf training content, track those views and visits. But, stop doing newsletters - mmkay?
- Gamification: Okay, my feelings on this stuff are biased, so I'll be blunt. If you are going to track a metric you think is from a gamification initiative - track overall participation first, and then represent the results by clarifying adoption rate over several months. Gamification is an ever changing, unknown blackhole of a time suck. If you are implementing Game Based Learning, make sure to tell the true story, not just the story of those who participated.
The first five listed are the real game changers for programs, the latter are just my way of motivating programs into Security Awareness 2.0.
There are a ton of other ways to track really cool metrics, and they should all show positive behavior change. Got a unique way you track it in your program? Let us know! Tweet at us!
So, what's your story?