Security for AI: The New Wave of Startups Racing to Secure the AI Stack

In the race to adopt generative AI, every enterprise grapples with a common concern: security. 

The AI models at the core of the modern AI stack are of particular concern. They require high volumes of sensitive enterprise data, employ self-learning that is difficult to control, and are deployed in environments that introduce human error. Meanwhile, cybercriminals (armed with equally sophisticated technology) generate new threats at unprecedented rates. The proliferation of AI has increased the surface area vulnerable to cyber attacks; LLMs make an attractive target.

Securing these models using current tools is impossible; as a result, enterprise buyers are ruled by caution, and the arc of adoption has failed to keep up with the hype. (Menlo’s report on enterprise adoption of AI underscored this point: Before they can deploy these models at scale, buyers want assurance that data exchanges are secure and their open-source models are safe.) 

The complexity of the challenge and sheer scale of the opportunity has inspired a wave of security innovation. Below, we provide a snapshot of the current market, identify where Menlo Ventures will invest, and highlight the promising companies paving the way for safe and scalable deployments.

GenAI, the New Threat Vector

AI models are increasingly becoming the target of cyber attacks. Just last November, OpenAI confirmed they suffered a DoS attack that impacted their API and ChatGPT traffic and caused multiple outages. Foundational model players like Anthropic* and OpenAI have communicated the need to secure their model weights from model theft, which occurs through compromised credentials or supply chain attacks. 

While in use, LLMs are vulnerable to prompt injections, insecure output handling, sensitive information disclosure, and insecure plugin design (Source: OWASP). During the 2023 Black Hat convention, cybersecurity experts openly demoed a synthetic ChatGPT compromise, wherein indirect prompt injections modified the chatbot to persuade users to divulge sensitive information. In other applications, prompt injections can manipulate LLMs to produce malware, fraud (e.g., phishing emails), or issue unwanted API calls. 

LLMs are also vulnerable to attacks while in development. To demonstrate, Mithril Security poisoned an open-source GPT-J-6B model on Hugging Face to generate fake news in response to a specific set of prompts. Until announcing it, Mithril’s manipulated model went unnoticed, and was available for enterprises to incorporate and deploy. While the example is illustrative, the message is clear: Exploited LLMs can cause widespread damage, be difficult to detect, and be even harder to resolve.

Thankfully, cybersecurity and AI experts are joining forces to tackle these challenges head-on.

The Time to Invest Is Now: Massive Opportunities in Governance, Observability, and Security 

We divide the emerging technologies into three categories, governance, observability, and security, and believe adoption will follow in that order. However, some protection measures are more pressing than others. Model consumption threats, because they expose models to outside actors, are an imminent vector that enterprise buyers must consider. Emerging AI firewalls and guardrails will need to placate these concerns on the same vector. More advanced attacks, like prompt injections, will also be top of mind for operators.

Governance solutions like Cranium and Credo help organizations create a catalog of AI services, tools, and owners for both internally developed and third-party solutions. They assign risk scores across safety and security measures and help assess business risks. Understanding AI usage across the organization is the first step toward observing and protecting LLM models. 

Observability tools, whether broad tools for model monitoring like Helicone or security use-case-specific tools like CalypsoAI, enable organizations to aggregate logs on access, inputs, and outputs to detect misuse and provide full auditability of the solution stack. 

Security solutions in the space focus on providing trust boundaries in model building and consumption. For both internal and external models, model consumption boundaries require rigorous control. We at Menlo are especially excited about AI Firewall providers like Robust Intelligence, Lakera, and Prompt Security, who moderate input and output validity, protect against prompt injections, and detect PII/sensitive data. Similarly, companies like Private AI and Nightfall help organizations identify and redact PII data from inputs and outputs. Importantly, enterprises must continuously monitor the effects of threats and attacks on LLM models and perform continuous red teaming. Companies like Lakera and Adversa aim to automate red teaming activities to help organizations investigate the robustness of their guardrails. On top of this, threat detection and response solutions like Hiddenlayer and Lasso Security work to detect anomalous and potentially malicious behavior attacking LLMs. 

There are many ways to construct models, from licensing third-party models to fine-tuning or training custom models. Anyone fine-tuning or custom building LLMs must feed the model large quantities of business/proprietary data, which may contain sensitive information such as financial data, healthcare records, or user log data. Federated learning solutions like DynamoFL and FedML address security needs by training local models on local data samples without needing to centralize and exchange data, only exchanging parameters. Tonic and Gretel can also address the issue by generating synthetic data to remove the worry of feeding in sensitive data to LLMs. PII identification/redaction solutions like Private AI or Kobalt Labs help identify and redact sensitive information from LLM data stores. When companies are building on top of open-source models that can have thousands of vulnerabilities, pre-production code scanning solutions, like the ones Protect AI provides, are paramount. Lastly, production monitoring tools like Giskard continuously seek out, identify, and prioritize vulnerabilities while models are already in production. 

It’s important to note that the velocity of development in this space is faster than ever. While companies may enter the market in one segment (e.g., start in building AI firewalls), they are quickly expanding their feature set to span across the market map (e.g., to DLP, vulnerability scanning, observability, etc.). 

Menlo has long invested in pioneering cybersecurity companies such as Abnormal Security, BitSight, Obsidian Security, Signifyd, and Immersive Labs. We’re eager to invest in teams with deep expertise in AI infrastructure, governance, and security who are tackling the ever-challenging, ever-changing cyber threat landscape—particularly as attacks on AI models become more prolific. If you are a founder building innovative security solutions for AI, we would love to connect with you.

*Backed by Menlo Ventures