What is External Risk Monitoring? It's All in the Data

By Jonas Edgeworth, VP - Engineering, RiskIQ

Jonas Edgeworth, VP - Engineering, RiskIQ

We now live in a business environment where enterprise commerce, workflows, and assets have shifted outside of their traditional security perimeter. To operate more efficiently and communicate better with customers, businesses are moving into the cloud, SaaS, mobile, and social channels. But as these offerings move further into the mainstream of our personal and professional lives, we must remember that with new digital channels comes all new risk.

Intentional or not, business units choose these external services without fully considering the security implications, circumventing centralized controls for immediacy of execution. For instance, they may purchase technical services from cloud providers or move enterprise assets and their associated data beyond the purview (and sight) of the CSO’s office. In fact, many of the mobile apps you come across were developed and published to several hundred app stores without any intermediate controls.

"A comprehensive WHOIS database shows how domains are related based on registration information."

Security programs must adapt to this ever-increasing attack surface to successfully protect their customers and employees. Managing this growth is more and more difficult given the variety of avenues that support it, all of which can lead to significant visibility gaps for the traditional Information Security organization.

With reduced resources and a shortage of trained security professionals, security teams responsible for tackling these challenges are being asked to do more with less—and many feel like they are working with the deck stacked against them. Without specialized tools and knowledge, threats that originate outside the traditional perimeter can’t be mitigated on time or at the source—there are simply too many signals and not enough context to identify and prioritize incidents and to take action against them.

Imagine finding a compromised website serving up malware which appears to be owned by your organization. It’s hosted by Amazon Web Services (AWS) and uses a privacy protected domain registration via GoDaddy. You’re the one responsible for finding the person in your organization who stood up the website so you can begin mitigating the incident. How long would that take? For most security professionals, by the time you addressed it, it would be too late.

Embrace The New World Order

Recently, Forrester, inc released The Forrester Wave™: Digital Risk Monitoring, Q3 2016. Like most categories established by Forrester, the creation of the Digital Risk Monitoring (DRM) shows the writing on the wall: solutions that focus on security outside the firewall are now table stakes for CISOs. But the complexity of protecting this attack surface should not discourage security teams from tackling these challenges head on. Security teams need to be empowered to gain oversight over these new channels without having to act as a gatekeeper for critical business initiatives.

How To Meet The Demands Of Ever-Shifting Threat Landscape

To address this dissolving perimeter, organizations have to step back and think about how to solve the more abstract problem: discovering new digital assets or threats and detecting changes related to them. And the only way to solve this problem is to leverage a very data-centric approach.

For instance, to discover a new host on a blacklisted IP address, you need access to massive amounts of DNS traffic in a Passive DNS database. A comprehensive WHOIS database shows how domains are related based on registration information. To find rogue mobile apps, we needed to build a repository of every mobile app that resides across the hundreds of app stores on the internet. Each of these data sets is a piece to a larger puzzle—with them in hand, discovering a new asset or detecting a new potential threat becomes easier by orders of magnitude.

Picking The Right Tool For The Job - Web Scale Data Sets Require Big Data Architectures

Given these massive data sets, my team of engineers at RiskIQ had to decide how to best ingest, process, persist, and analyze these data sets at scale with a quick turnaround for our customers. What scale? We process just shy of two billion HTTP requests and twenty billion raw Passive DNS data records a day.

The key to solving this was to leverage technologies and architectures that can grow with the data. The beauty is that data engineering at scale has largely been addressed through technologies that make up the Hadoop ecosystem—Hive, HDFS, MapR, and Spark all play crucial roles in extracting core intelligence from our datasets, from basic curation and aggregation to building sophisticated statistical models. Now, what would have taken days, weeks or even months to execute with more antiquated technical architectures can be executed in hours.

A Flexible Platform Allows For Rapid Prototyping And Deployment Of Products

The power of these tools lies in their flexibility to meet the needs of a multitude of use-cases. Remember what I said about it all being a part of a big puzzle? For example, a model developed by our Data Science team to detect phishing can be quickly repurposed to solve many seemingly unrelated use-cases: scams, counterfeiting, defacement, application service identification to name a few. Similarly, our Security Research team can perform forensic research by leveraging Hive to quickly backtest detection rules over a roughly a petabyte of web crawl data. This flexibility allows us to quickly react to new threats as they arise, and allow us to test and validate novel use-cases.

As the interconnection between these data sets becomes more intrinsic and the number of datasets grows, an adaptable architecture is paramount to maintain the visibility needed to manage an ever-evolving threat landscape. 

New Editions