Meaningful Metrics in Information Security

René „Lynx“ Pfeiffer

DeepSec In-Depth Security Conference

whoami

• 🆔 René „Lynx“ Pfeiffer
• ℹ Senior Systems Administrator
• ℹ DeepSec In-Depth Security Conference organisation team
• ☢ Study of theoretical physics
• 🕸 Internet user since 1992
• ⏳ 30+ years of experience with software development, computing platforms, and systems administration
• 👨‍🏫 20+ years teaching at University of applied Sciences (UAS) Technikum Wien

Metrics?

• Metric / metrics is / are used in engineering, science, and business.
• Possible uses
  • Performance indicators
  • Measure of some property or observable attribute
  • Can be more of a set of values / properties
• „a set of numbers that give information about a particular process or activity“ (Cambridge Business English Dictionary)

Measurement

Source: SI base units

How to measure?

• Measuring == comparing (includes counting!)
• Observable events / attributes may not be independent
• Measurement uses units = measured quantity with numerical value
• Physics uses 7 SI base units - everything else is derived
• All measurements have errors!

Statistics

• Statistics = mathematical statistics
  • Descriptive statistics - describes/summarises collected data and properties
  • Inferential statistics - draws conclusions from data
• Sensible measurements requires use of statistics.
• Statistics requires sensible data.
• Good statistical results require many samples.
• Single events carry no statistical meaning!

Data Science = Mathematical Statistics + Code + fancy Hardware + lots of Trial and Error 😏

Graphs without Meaning

This is just a drawing with no meaning! There is no data that backs this drawing!

So what?

Not everything that can be counted or measured is worth counting or measuring.

Lines of Code (1)

„…The way that I work, I try to get out there and I try to get six pages a day.…“ – Stephen King

Lines of Code (2)

We don't know Elon's daily goal in terms of written pages, lines, or words.

Key Performance Indicators (KPIs)

• Origin in finance to track budgets and profits
• KPIs focus on strategic business goals
• KPIs live on invented or normalised scores (usually percentages or counters)
• Choice of KPIs determine organisation's future
  • Bug detection KPI leads to hunt for easy-to-find bugs
  • Attack detection KPI leads to more rules on the IDS/IPS
  • Tickets closed KPI leads to quick fixes or simple workarounds
  • Commit LOC KPI leads to layout changes & more whitespaces
• Applies to IT security policies, too!

Time is your Friend

Properties of Time

• Timestamps are easily available
• Timestamps set chronological order of events
• Synchronisation problems can be bypassed by
  • logical clock algorithms
  • monotonic counters
• Durations can be derived from timestamps
  • Useful for reviews
  • Often used as a „default metric value“
  • Essential for planning and training

How to measure IT Security?

• Work with inventories:
  • Accounts
  • Versions (of components)
  • Vulnerabilities (present) - don't use this for gamification / negative goal!
  Look for unknown devices / components / accounts!
• Time to detect / fix / recover / correct / …
• Telemetry data (CPU, network, storage, …)
• Use case / context specific data - the hard & individual part

Measuring is a Process

Cyber Resiliency Metrics, Measures of Effectiveness, and Scoring

Things to remember

• Less (data) is more - and cheaper.
• How useful is the information?
• How will the information be used? Who will use it?
• What are the costs of aquiring the metrics?
• KPIs: Why are they demanded?
• Use metrics as a complement to judgement.
• Even the best metrics can be corrupted or subverted.
• „When a measure becomes a target, it ceases to be a good measure…“ (Marilyn Strathern)

Questions?

Source

• Cyber Resiliency Metrics, Measures of Effectiveness, and Scoring
• ISO/IEC 27004:2016
• Measurement Guide for Information Security: Volume 1 — Identifying and Selecting Measures (NIST SP 800-55 Vol. 1)
• Quantities and Units for Software Product Measurements (NISTIR 8289)
• Technical performance metrics of a security operations center
• To Improve Cyber Resilience, Measure It
• Using Quality Metrics and Security Methods to Predict Software Assurance

Document Classification

This document is classified as TLP:CLEAR. Data and documents classified as TLP:CLEAR are public. Recipients can spread this to the world, there is no limit on disclosure. Sources may use TLP:CLEAR when information carries minimal or no foreseeable risk of misuse, in accordance with applicable rules and procedures for public release. Subject to standard copyright rules,. This classification requires no protection for transfer and storage of the information [Standard Operations Procedure (SOP) Information Classification 0002].

If you require assistance or permission for sharing the document, please contact the author.

See the TRAFFIC LIGHT PROTOCOL (TLP) description for more details (part of FIRST Standards Definitions and Usage Guidance, www.first.org).

About the Author

René Pfeiffer was born in the year of Atari's founding and the release of the game Pong. Since his early youth he started taking things apart to see how they work. He couldn't even pass construction sites without looking for electrical wires that might seem interesting. The interest in computing began when his grandfather bought him a 4-bit microcontroller with 256 byte RAM and a 4096 byte operating system, forcing him to learn Texas Instruments TMS 1600 assembler before any other programming language.

After finishing school he went to university in order to study physics. He then collected experiences with a C64, a C128, two Commodore Amigas, DEC's Ultrix, OpenVMS and finally GNU/Linux on a PC in 1997. He is using Linux since this day and still likes to take things apart und put them together again. Freedom of tinkering brought him close to the Free Software movement, where he puts some effort into the right to understand how things work – which he still does.

René is a senior systems administrator, a lecturer at the University of Applied Sciences Technikum Wien and FH Burgenland, and a senior security consultant. He uses all the skills in order to develop security architectures, maintain/improve IT infrastructure, test applications, writing technical documents, and to analyse security-related attributes of applications, networks (wired/wireless, components), (cryptographic algorithms), protocols, servers, cloud platforms, and more indicators of modern life. If you are interested in putting your security to the test, please let me know.