In today’s rapidly evolving cyber landscape, Chief Information Security Officers (CISOs) face unprecedented challenges. The surge in sophisticated cyber threats demands robust strategies for cyber resilience, risk management, compliance, and collaboration—all while being prepared for unexpected crises. How can organisations fortify their defenses against unforeseen failures? What lessons can we glean from past incidents to strengthen our future strategies?
This article delves into practical approaches for integrating best practices, leveraging continuous monitoring, and enhancing communication with vendors and internal teams. These insights aim to bolster your defences and guide informed decisions when collaborating with cybersecurity software vendors.
The Critical Need for Endpoint Security
Endpoint security is a cornerstone of any organisation’s cybersecurity strategy. With the increasing number of devices accessing corporate networks, protecting these endpoints is more crucial than ever. Here are the core benefits:
- Enhanced Protection: Defends against a wide range of cyber threats, including malware, ransomware, and phishing attacks.
- Improved Compliance: Helps meet regulatory requirements and industry standards, reducing the risk of fines and legal issues.
- Increased Productivity: Minimises downtime caused by security incidents, ensuring smooth business operations.
- Better Visibility: Offers comprehensive insights into endpoint activities, allowing for effective threat detection and response.
- Reduced Risk: Lowers the likelihood of successful attacks and data breaches, protecting sensitive information and maintaining customer trust.
The Role of the Kernel in Endpoint Security
You might wonder why security solutions leverage kernel drivers. Given the design of operating systems and the need to combat modern attackers effectively, kernel drivers are pivotal for several reasons:
Visibility and Enforcement of Security-Related Events
Kernel drivers provide system-wide visibility and early threat detection. They enable capabilities like system event callbacks and filter drivers to monitor file operations. For instance, they can monitor and intercept suspicious file activities in real-time, preventing malware from executing.
Performance Enhancement
Kernel drivers improve performance, especially for high-throughput network activities. They offer significant benefits, allowing security vendors to optimise performance to match or exceed that of user-mode operations. For example, a network security tool can use kernel drivers to analyse large volumes of traffic efficiently without impacting system performance.
Tamper Resistance
Operating in kernel mode offers enhanced tamper resistance, ensuring software cannot be disabled by malware or malicious insiders. Kernel-mode drivers load early in the boot process, providing an additional layer of security against sophisticated attacks aiming to disable security software before it can protect the system.
Best Practices for Selecting and Managing Endpoint Security Solutions
To mitigate risks associated with Endpoint Detection and Response (EDR) agents, CISOs should consider the following best practices:
Limit Kernel Mode Operations
Choose endpoint security agents designed to operate primarily in user mode. This approach maintains application isolation and protects the system from crashes and data corruption. Ensure that interactions with kernel mode are minimal and restricted to essential functions like data collection, prevention, and anti-tampering. For Windows systems, ensure that communication between user and kernel mode components adheres to best practices, minimising and controlling kernel interactions.
Controlled Update Processes
Select vendors that support a phased rollout approach for updates. Begin with a small subset of systems to ensure stability and performance before wider deployment. Vendors should provide the ability to control update deployment, including enabling or disabling updates at different organisational levels.
Utilize Modern Frameworks
Opt for vendors that move away from outdated kernel extensions and utilise modern frameworks like eBPF (Extended Berkeley Packet Filter) for Linux and Apple’s Endpoint Security Framework (ESF) for macOS. These frameworks reduce the attack surface, improve performance, and align with industry best practices by allowing safe code execution in user space.
Require Transparency and Trust from Vendors
Demand clear communication from your vendors regarding agent behaviour, updates, and incident responses. This includes detailed release notes, version information, and auditing details for each update. Transparency about changes and their purposes builds trust and fosters better preparation and response to potential issues.
Learning from Past Incidents: The July 19 Global IT Outage
On July 19, 2024, a Rapid Response Content update for the CrowdStrike Falcon sensor caused widespread disruptions for systems running Windows 7 and above. The update, published at 04:09 UTC, led to kernel instability and Blue Screen of Death (BSOD) loops on systems online between 04:09 and 05:27 UTC. Approximately 8.5 million devices were affected globally. Mac and Linux hosts were not impacted, nor were Windows hosts that were offline or did not connect during this period.
The update aimed to gather telemetry on new threat techniques observed by CrowdStrike. However, a defect in the Rapid Response Content caused an out-of-bounds memory read, leading to crashes. This incident became one of the largest IT outages in history.
Other Notable Incidents
These types of disruptions aren’t unprecedented. Here are some other significant incidents:
- McAfee Antivirus Update (2010): A McAfee antivirus update falsely identified a critical Windows XP system file as malware, leading to widespread malfunctions, reboot loops, and loss of network access.
- Symantec Endpoint Protection Update (2012): An update conflicted with third-party software, causing system crashes on Windows XP machines.
- Webroot Antivirus Update (2017): Webroot mistakenly flagged essential Windows system files as malware, leading to significant disruptions as critical files were quarantined.
These incidents highlight the importance of rigorous testing, controlled rollouts, and the risks associated with operating in kernel mode, where errors can have system-wide impacts.
Key Lessons and Strategies for Improvement
The recent incidents underscore several critical areas where the industry must maintain vigilance:
Comprehensive Testing Procedures
Conduct thorough testing of updates and new features in real-world environments. This includes compatibility testing with all critical third-party applications to prevent system crashes and ensure smooth updates.
Enhanced Content Validation
Implement additional validation checks to ensure updates are robust and free from defects. This reduces the likelihood of false positives and widespread disruptions.
Balanced Agent Architecture
Design security agents to operate primarily in user mode, limiting kernel mode interactions to essential functions. This approach reduces the attack surface and minimizes the risk of system-wide failures.
Strengthened Error Handling
Develop robust error-handling mechanisms to manage and mitigate errors gracefully. This includes implementing design patterns capable of isolating faults and allowing systems to degrade gracefully instead of crashing.
Controlled Rollout Processes
Adopt a phased rollout strategy, starting with a small subset of systems to monitor impact and performance before wider deployment. Avoid “big bang” rollouts, uncontrolled, unregulated, and automated upgrades that can lead to widespread issues.
Enhanced Monitoring and Feedback
Implement continuous monitoring of agent and system performance during and after deployment. Real-time monitoring and anomaly detection systems can swiftly catch and rectify errors.
Customer Control and Transparency
Provide customers with greater control over update delivery and deployment. Ensure that all updates are configurable, documented, and controlled by the customer, enhancing trust and collaboration.
Mitigating Risks with Endpoint Protection Agents
Integrating endpoint protection agents with complex operating systems and other security measures can introduce risks. Here are some best practices to mitigate these risks:
- Sequential Changes: Implement updates one at a time to easily identify and resolve any issues that arise.
- Controlled Testing: Test updates on a select group of devices to evaluate their impact before full deployment.
- Phased Rollouts: Gradually extend updates across different segments after successful testing to ensure stability.
- Strategic Updates: Carefully manage and monitor live updates, especially on critical business devices, to balance immediate protection with potential risks.
- Configurable Updates: Ensure all updates are configurable, documented, and controlled by the customer.
Assessing Endpoint Security Maturity
Evaluating your organization’s endpoint security maturity involves a structured framework that assesses the implementation of best practices and compares approaches to mature models. Key indicators include:
Granular Control and Flexibility
Ensure your solution offers detailed control over updates and security measures. This allows adaptation to specific security needs and minimises business disruptions.
Robust Communication and Collaboration
Maintain effective communication channels and collaborative practices with stakeholders. This promotes a coordinated and informed response to security threats.
Automated and Adaptive Security
Leverage advanced technologies like AI and machine learning for real-time threat detection and response. A mature solution continuously learns and adapts to new threats.
Comprehensive Incident Response Plans
Develop well-defined and regularly tested incident response plans, including steps for containment, recovery, and clear communication protocols.
Integration with Business Continuity Planning
Ensure security measures support overall organisational resilience and maintain critical operations during disruptions.
Continuous Improvement Culture
Foster a culture of regular review and updates of security practices based on incident lessons, threat landscape changes, and technological advancements.
Resilient Design Patterns
Implement design patterns capable of isolating faults and allowing systems to degrade gracefully instead of crashing. This reduces the risk of critical failures like BSODs.
By focusing on these indicators, CISOs can enhance their organization’s security posture and foster continuous improvement.
Conclusion
Trust in software vendors is vital for effective communication and rapid issue resolution. As CISOs, our commitment to continuous improvement and proactive security measures is crucial to safeguarding our organisations in an increasingly hostile cyber environment.
Building and maintaining trust in our systems, teams, and vendors is essential for navigating the complex landscape of cybersecurity successfully. As Warren Buffett aptly said:
“Trust is like the air we breathe—when it’s present, nobody really notices; when it’s absent, everybody notices.”
By prioritising trust and implementing the best practices outlined above, we can ensure our cybersecurity efforts are both effective and resilient, helping us stay ahead of evolving threats and unexpected scenarios. For robust endpoint solutions, see our Endpoint Protection solution powered by SentionalOne.
Originally published by Chris Boehm. Read the original article at SentinelOne Blog.