One year and almost 10,000 incident reports later, the 2022 VOID Report is here! After scrutinizing an entire year’s worth of incidents one thing is crystal clear: Resilience saves time. Taking the time to understand how to better respond when something green turns red—learning from the people, the processes, and the systems—will make the next incident smoother. Because, yes, there will always be the next incident.
Today we’re releasing a Prowler Training Course to help people start using Prowler and get up to speed quickly. The course is free, and consists of a series of short, actionable videos where I walk you through the basics of Prowler, how to run it and what some sample outputs look like, and then how to write your own custom checks and some advanced features like sending your results directly to Security Hub.
We recommend sharing the article with friends and colleagues who are not familiar with the VOID or the findings of the 2021 report. It is a quick intro to some of the most important findings of the report.
The Kafka producer-cluster-consumer system is a great way to funnel high volumes of messages with spectacular reliability and order from place-to-place within your product’s architecture. However, it isn’t without its drawbacks. As with any system, some of those drawbacks are by circumstance while others are by design. The protocol for reassigning topic partitions when a […]
Today we’re announcing Prowler Pro, an enterprise version of Prowler, a well-known open source security tool for AWS. Verica and I joined forces back in late 2021 to make Prowler better, and develop a new product that extends on that solid foundation. Prowler Pro has the best of Prowler plus continuous monitoring, automated deployment for multi-account, personalized support, and visualization of your data via helpful dashboards with custom alerting and reporting.
While we’ve all been getting by with virtual conferences and events, it’s just not the same as the real thing. We’re thrilled to see in-person learning and networking make a comeback, and we are ready to see your faces! Here’s what we’re up to and where you can join us in the next couple of months:
DISH Network recently came on board as a Verica customer, using our Continuous Verification Platform (CVP) to help ensure the reliability of the Kubernetes and Kafka systems that will power their new 5G network. That was a whole lot of technology terms, but what we’re really thrilled about is that we’ll be helping make something […]
“Companies can plow all their resources into building and testing apps fit for millions of users, but they can never truly know how their software will behave until it’s pushed out into the wild.” Paul Sawers, VentureBeat We made it official. For the past few years, we’ve been quietly working on our Continuous Verification Platform […]
The internet stood still when Facebook, Instagram, and WhatsApp or Amazon Web Services went down for hours. That may have seemed like a minor inconvenience to American users of those services. Elsewhere, it crippled essential communication for billions of people across the world. Commercial software is the engine that powers many parts of our lives. Recent outages show that […]
At the end of January, 2021, a group of Reddit users organized what’s called a “short squeeze.” They intended to wreak havoc on hedge funds that were shorting the stock of a struggling brick and mortar game retailer called GameStop. They were coordinating to buy more stock in the company and drive its price further […]
The final pattern we’ll cover is the lack of “near miss” incident reports. We classify a near miss as an incident that the organization noted had no noticeable external or customer impact, but still required intervention from the organization (often to prevent an actual public incident from eventually happening). As of today, there are only nine near-miss reports in the VOID—not even a half of a percent of the total entries. While this isn’t a surprising aspect of the reports in the VOID, we call it out here as an important opportunity to establish a baseline and track how it changes over time.
Roughly a quarter of the VOID incident reports (26%) either identify a specific “root cause” or explicitly claim to have conducted a Root Cause Analysis (RCA). We consider these data preliminary, however, given the incomplete nature of the overall dataset. As we continue to add more reports, we’ll track this and see if it changes.
We’re specifically looking into RCA because, like MTTR, it is appealing in its decisiveness and apparent simplicity, but it too is misleading.
Software organizations tend to value measurement, iteration, and improvement based on data. These are great things for an organization to focus on; however, this has led to an industry practice of calculating and tracking Mean Time to Resolve, or MTTR. While it’s understandable to want to have a clear metric for tracking incident resolution, MTTR is problematic for a couple of reasons.
Whenever we read a company’s public incident report, there are so many questions that ultimately go unanswered. As most people familiar with incident management and analysis know, there’s plenty of information that doesn’t make it into the public writeups of a software incident.
So we thought: what if we could ask the people involved in those incidents to take us back to their experience, and answer some of those unanswered questions? That’s the idea behind the VOID podcast.
When it comes to outages, we only seem to hear the bad news. In reality, the people who run these systems resolve these issues fairly quickly the majority of the time. They know this, but we haven’t had the data to show this, until now. This finding, that over half of software-related incidents are resolved in under two hours, comes from the Verica Open Incident Database (VOID). The VOID makes public software-related incident reports available to everyone, raising awareness and increasing understanding of software-based failures in order to make the internet a more resilient and safe place.
The potential for catastrophic outcome is a hallmark of complex systems. It is impossible to eliminate the potential for such catastrophic failure; the potential for such failure is always present by the system’s own nature. Richard Cook, How Complex Systems Fail Today, we’re announcing a project that represents a small step on a long road […]
Our vision was to interview leaders in the Chaos Engineering field in a cross between the styles of Between Two Ferns and The Office as awkwardly as possible—we think we’ve succeeded.
I’ve been friends with James [Wickett] and his wife for a while and he asked me if I would be interested in the role. I worked with James at one of my previous jobs around 10 years or so ago, so I knew that I liked working with him and that he was a really good guy. I was able to trust that what he said about Verica was legit. He said the people here were smart and caring, and that it wasn’t a culture that beat you down. There are a lot of those types of cultures in workplaces today, but Verica is a place that you could feel like you’re being built up. I really like that. It’s a great place to work. I was saying the other day that even if I were to win the lottery, I’d still want to work here.
Chaos Engineering is grounded in empiricism, experimentation over testing, and verification over validation. But not all experimentation is equally valuable. The principles of Chaos Engineering extend to a “gold standard” captured in a set of advanced principles.
The Birth of Chaos The beginning of Chaos Engineering goes back to 2008 when Netflix moved from the datacenter to the cloud. The move didn’t go as planned. The thinking at the time was that the datacenter locked them into an architecture of single points of failure, like large databases, and vertically scaled components. Moving […]
Karthik Gaekwad Austin, TexasAt Verica for: 8 months Ed Note: This is an ongoing series about what a day in the life of various Verica folks looks like. In this post, we chat with Karthik, one of our engineers on the Kubernetes module team. In addition to being Verica’s resident Cloud Native expert, Karthik is an […]
Ed note: This post presumes you have some familiarity with Chaos Engineering, and are considering whether you can start experimenting with it at your organization. If you’re not familiar with Chaos Engineering, here’s a great post to get you up to speed. Chaos Engineering is often characterized as “breaking things in production,” which lends it […]
Randall HansenVP, User Experience “I live in the burning wreckage of Portland, Oregon”At Verica for: 2 years Ed Note: This is an ongoing series about what a day in the life of various Verica folks looks like. In this post, we chat with Randall Hansen, VP of User Experience. Randall’s super power is seeing the world […]
This is an ongoing series about what a day in the life of various Verica folks looks like. In this post, we chat with Chantell Nichols, a Senior Software Engineer on the Platform team. Chantell is a dedicated teammate with an attitude that makes everyone believe anything is possible. She has an insatiable thirst for learning that motivates and inspires everyone around her.
“The growth of complexity in society has got ahead of our understanding of how complex systems work and fail. And when such complexity fails, we still apply simple, linear, componential ideas as if that will help us understand what went wrong.” —Sidney Dekker, Drift Into Failure1 Rapid technological innovation has presented businesses unique opportunities to […]
Ed note: This is the second in a multi-part series about the free Security Chaos Engineering report from O’Reilly. This guest post is from Kelly Shortridge, co-author of the report, and originally was posted on the Capsule8 blog. It is re-posted here with her permission. Production is the remunerative reliquary where we can realize value […]
Ed note: This is the first in a multi-part series about the free O’Reilly report co-authored by Aaron Rinehart and Kelly Shortridge. Information security is broken. Our users, our customers, our world, are entrusting us with more and more of their life and we are failing to keep that trust. Year after year the same […]
“Each person deserves a day away in which no problems are confronted, no solutions searched for. Each of us needs to withdraw from the cares which will not withdraw from us.” —Maya Angelou When I joined Verica I was told that the policy on vacation time was unlimited PTO. No sick days to be concerned […]
I was hired at Netflix to lead the Traffic Team in early 2015. A few weeks later I was also asked to charter a Chaos Engineering Team. At the time, Chaos Engineering was essentially a program called Chaos Monkey with a few supporting blog posts. I wanted to get a feel for what our engineers […]
Co-Founder and CEO Casey Rosenthal and Co-Founder and CTO Aaron Rinehart of Verica join Enterprise Security Weekly to talk about Security Chaos Engineering. In this episode, Casey and Aaron discuss how companies of all sizes in different verticals are adopting Chaos Engineering and how Continuous Verification is now gaining adoption. The discussion digs deep into […]
You have probably heard the phrase “practice makes perfect”, but let’s take that thought up and out. It is really the experience of practice that is more interesting! Consider a jazz band you’ve heard or seen live, or really any type of music where improvisation plays an important role. Great bands work together effortlessly, and when things go […]
The Verica crew had a fun time with the fine folks down in Austin, TX at the city’s first DevSecOps Days event. Going to Austin during the holiday season never results in a White Christmas, but this year Austin delivered on some great security and devops talks that should make you cheery all year long. […]
For the 2019 OWASP Global AppSec DC event, which is the largest application security conference on the planet, I brought the MEASURE framework in the form of a talk. One of the goals of the talk was to bring together the worlds of Safety and Security. I believe that Safety fits in the MEASURE framework for […]
The start of DevOps was simple enough: get some developers and operations folks together and build something. With together being the keyword. The concept of DevOps came from this notion that operations needed to take part in the Agile movement. Many articles chronicle the history of how Patrick Debois and Andrew Clay Shafer coined the term and […]
You may know us at Verica from our ground-breaking work in the fields of chaos engineering and continuous verification, both from our history at Netflix and UnitedHealth Group and from the articles we have on this very blog, but you probably don’t know the team behind the company. Well, this is a little insight into […]
Many people use the words Verification and Validation interchangeably, which risks the ability to focus on system-level behaviors that correspond to business value. We prefer to use definitions inspired by the field of Operations Research Management. Verification is finding congruence between what you expect from a system and the actual output. Validation is finding congruence between an […]
One of my favorite topics of discussion within the domain of Availability is mythology. Not dragons and unicorns, which would be undeniably cool, but myths in the sense of made-up stories we tell ourselves to explain things that we don’t understand. There are many things that we as an industry tell ourselves about the nature […]
Root Cause Analysis (RCA), a common practice throughout the software industry, does not provide any value in preventing future incidents in complex software systems. Instead, it reinforces hierarchical structures that confer blame, and this inhibits learning, creativity, and psychological safety. In short, RCA is an inhumane practice. Fortunately, there are healthy alternatives to RCA. I’ll […]
Three years from now, if you are pushing code into a serious production environment, it will go through a Continuous Verification (CV) pipeline. The software industry’s transition into complex systems is accelerating. The humans designing, building, and operating these complex systems are no longer capable of understanding how all of the pieces fit together. It’s […]