Agents Everywhere: The Flow Of Information - Negative Star Innovators Blog

It is our thesis that every single human will eventually direct AI agents across both digital and physical domains in their personal and working lives. Even physical work will be automated by robots and orchestrated by agents that humans direct. Many agents will operate persistently with little or no continued human supervision, delegating tasks and exchanging information with other agents long after a person has set them up. These agents will execute work and exchange information at a scale humans cannot manually manage or comprehend. There will simply not be enough human time to understand the massive amount of information being exchanged by these agents. Every additional agent and interaction increases the surface area for information to flow to third parties, with dangerous implications.

We believe agents will mediate every aspect of everyday life by joining calls, searching past recordings, managing messages, retrieving documents, operating cameras, purchasing, controlling robots, and representing people in negotiations. Often, each party will have its own agent. In a job interview, the employer’s agent may transcribe and analyze the conversation while the candidate’s agent retrieves relevant experience, assesses the employer’s claims and advises the candidate. In a contract negotiation, each party’s agent can review the proposed terms against its own priorities, flag risks and propose revisions before the people involved approve the final agreement.

Even if you do not use agents yourself, information you share with other people and organizations can still be processed by agents you do not control. Those agents will be able to retain and analyze this data, derive new information from it, and act on the results.

What Information Can Leak

The information that can leak and be used against us goes far beyond Personally Identifiable Information (PII) and credentials.

Confidential records and access credentials
Information that is private by nature or can be used to access accounts, assets, identities, or private communications. This includes PII, passwords, authentication tokens, phone numbers, addresses, account and payment details, financial and legal records, private communications, and customer account and transaction records. These risks apply to individuals and organizations alike and extend well beyond conventional definitions of PII.
Private context, plans and know-how
Information whose sensitivity comes from what it reveals about intentions, capabilities, methods, relationships, or accumulated knowledge. For businesses, this includes product strategy, proprietary methods, research, non-public pricing strategies and commercial terms, supplier relationships, sourcing strategies, negotiated terms, proprietary source code, customer intelligence, internal documents, and operational know-how. For individuals, this includes private projects, unpublished creative work, hobbies, interests and leisure activities, personal and professional plans, family, social and professional relationships, personal obligations and routines, and other private context whose disclosure could enable manipulative marketing, targeted scams, reputational or relationship harm, loss of bargaining power, or threats to their livelihood, autonomy or safety.
Sensitive human data and inferences
Biometric, physiological and behavioral data and inferences drawn from them can be used to identify us, infer our mental and physical health, preferences and dislikes, and build models predicting how we are likely to behave in particular circumstances. For instance, researchers have demonstrated that standard video cameras can capture minute physiological changes to measure physical health metrics such as heartbeats, blood pressure, blood oxygen saturation, respiratory rate, heart-rate variability, and age without any physical contact [1][2][3]. In the realm of mental and cognitive health, “voice biomarkers” extracted from brief audio recordings can indicate if you have dementia, depression, insomnia and anxiety by analyzing subtle auditory features like speech rate, pitch, zero-crossing rates, and vocabulary choice [4][5][6][7]. Wi-Fi and RF radar signals can detect our precise body positions, sleep stages, respiration rates, and heartbeats by measuring how radio frequencies bounce off the human body, even through walls [8][9][10]. Human gait provides another sensitive data stream: walking patterns can be used to identify individuals, gait speed is associated with survival in older adults, and gait features can indicate neurological disorders, cognitive load, fatigue and sleep deprivation [11][12][13][14][15][16]. Furthermore, modalities like physiological eye-tracking (assessing pupil dilation, fixation duration, and gaze patterns) can evaluate our cognitive load, reveal task-specific problem-solving strategies and aspects of task performance, and indicate our preferences and dislikes [17][18][19].
Physical-world operations and sensor data
Information captured by robots, vehicles, cameras, microphones, and other physical sensors. This includes video, audio, spatial maps, precise locations, home or facility layouts, safety events, machine telemetry, household and movement patterns, and the operational processes that reveal how a business or household actually functions. Exposure can create privacy, safety, and physical-security risks, not just conventional data-security risks.

These categories overlap, and information that appears harmless in isolation can become sensitive when linked with other data, accumulated over time, or used to derive new inferences.

How Leaked Information Can Be Used Against You

How Confidential Records and Credentials Can Be Used Against You

Passwords, authentication tokens and account details can provide immediate access to accounts, assets and private communications. Other records may not provide immediate access but can expose legal disputes, financial pressures, customer relationships, health information or private conversations.

For an individual, this can mean identity theft, fraud, blackmail or the exposure of private life. For a business, it can mean customer harm, regulatory liability, operational disruption and loss of trust. A leaked record can also become the starting point for a larger attack, one document may reveal account names, internal systems, relationships or security practices that make subsequent attacks more effective.

The Business Case: Your Data and Accumulated Context Are Your Moat

Your data is your moat. Losing control of it can expose your business, strengthen competitors and erode your differentiation. If frontier AI companies train on your business data, what they learn may improve the results they provide to everyone, including your competitors. Knowledge accumulated inside your business can therefore become a capability available across the market. AI providers may also expand into their customers’ markets. Claude Design could compete with Figma, while Personal Finance in Claude could compete with financial-advice and personal-finance services. You could end up paying a future competitor while supplying the context that helps it build your replacement.

AI is making it cheaper to start and run a business and to build products. As product development and operations become easier, markets will fill with similar offerings competing for scarce attention. In that environment, your data and operational know-how are among the few moats that can remain.

The risk is not limited to deliberate reuse by an AI provider. Once information leaves your control, it may be exposed through a security breach, retained longer than expected, shared carelessly with employees, contractors or third parties, or disclosed through legal processes such as subpoenas, court orders, discovery or governments’ backdoors. It may also be combined with other datasets to reveal more than you intended to share.

The Individual Case: Relationships, Routines and Manipulation

For individuals, the harm is not limited to identity theft or the exposure of a private document. Accumulated context can reveal family and social relationships, personal obligations, hobbies, routines, intentions, constraints, future plans and our most personal psychological conditions. It can show who is connected to whom, who trusts whom, who depends on whom and what pressures someone is under. This is already happening but AI has changed the economics which we discuss later.

Marketing companies can use this information to produce highly individualized and manipulative advertising. Scammers can use the same context to make fraudulent requests and impersonations more convincing. A scammer who knows someone’s family relationships, current, future plans and communication patterns could use that information to make a deepfake impersonation of a family member appear credible.

This context can also weaken a person’s negotiating position. An employer, buyer, client or other counterparty may learn how urgently someone needs a job or sale, what alternatives they have, or which constraints are shaping their decisions. Private routines and relationship information can also create risks to autonomy and safety, including manipulation, stalking, coercion and unwanted contact.

No single fact needs to look highly sensitive for the accumulated picture to become dangerous.

How Sensitive Human Data Can Be Used Against You

Consider what would happen to a company’s stock price if its CEO showed signs of mental distress, psychiatric disorders, and/or illness during an earnings call that AI could detect by monitoring the CEO’s voice and body. The result could still be disastrous if the AI produced a false positive.

Another example is that companies could determine which imagery and topics appeal to you, which people you are sexually attracted to (right down to individual features), what you dislike seeing and which communication styles you find most persuasive. They could then manipulate you based on that information for instance by using generative AI video to create millions of customized ads for millions of individuals (a customized video for each individual) and deliver each one at the moment that person is most susceptible to making a purchase.

A job interviewer reviewing your video call may be able to estimate in real time how desperate you are for a job and use that inference to justify a lowball offer. A health insurance company could change your premiums based on a phone conversation without ever informing you why its risk assessment changed.

The inference does not have to be conclusive for someone to act on it. Nor does the affected person need to know that the inference was made. In most cases the affected person would never know. The better approach is to prevent biological and psychological inferences from being generated or shared from videos, audio and other data streams in the first place.

Physical-World Exposure: Safety, Security and Operational Processes

Physical-world data reveals more than the objects appearing in a camera feed. Facility maps can expose entrances, restricted areas and security coverage. Robot and machine telemetry can reveal production rates, capacity, maintenance problems, operating schedules and details of proprietary processes. Safety events can show where systems fail and where physical vulnerabilities exist.

In a household, spatial maps, cameras, microphones and movement patterns can reveal the layout of a home, when people are present, what personal sensitive items are present, which rooms they use and how they move through their daily routines. Information collected to perform a useful physical task can therefore become information about occupancy, vulnerability and behavioral analysis.

As agents gain control over robots, vehicles and other physical systems, leaked information will not only describe the physical world. It may also reveal how to influence the systems acting within it.

Why AI Changes the Economics of Information Exploitation

AI allows the analysis of more information and the inference of more information than was ever possible before. What’s more, AI can act on information by itself. For example, it can automatically send messages, target advertisements, generate content, approve or reject requests, set fees amount, initiate transactions and control robots without waiting for a person to review each decision.

In the past, limited human time and cost meant that most available information could not be used to full effect. Only when the potential return justified the cost did we see highly effective analysis and action, such as nation-state intelligence agencies spying on one another or corporate espionage between high-value companies.

The cost of mass analysis and the highly individualized use of information via AI is dropping fast. The cost of humans’ time is no longer the bottleneck. The bottleneck will be compute and compute will continue to become cheaper and more powerful over time.

This changes the economics of exploitation. Information that was previously too expensive to analyze, combine or act upon can now be processed continuously and at scale. An actor no longer needs to justify the cost of assigning a person to investigate you or your company.

Exploiters Don’t Need to Be Right All the Time

An inference does not need to be correct 100% of the time to be useful. It only needs to provide a statistical advantage.

An advertiser that can slightly improve its estimate of when someone is susceptible to an offer can benefit across millions of impressions. A scammer can profit even if most attempts fail. An employer, insurer or competitor may benefit from a probabilistic assessment that is wrong in many individual cases but useful on average.

False positives do not eliminate the risk. A false inference about illness, distress, vulnerability, dishonesty or intention can still affect a decision, a negotiation or a person’s reputation. The person being assessed may never know which inferred information was used, or have a meaningful opportunity to challenge it.

Where Agent Information Leaks

Information can flow to third parties when agents send context to external frontier AI providers, when locally running agents communicate with the outside world, when agents exchange information with other agents in a business or customer context, and even when an agent communicates directly with an external human.

The same categories of information can appear in prompts, documents, conversation histories, tool calls, workflow logs and agent-to-agent messages. An agent message can contain credentials, strategic knowledge, human inferences or a physical map just as easily as a conventional file can.

External disclosure also does not require an agent to send an entire document. An agent may reveal sensitive information through a short query, a request for assistance, a summary, a tool parameter or an answer given to another party.

Local Inference Does Not Eliminate Information Leakage

Local inference does not eliminate this problem. It just changes where the model runs, not whether information still needs to leave.

Local inference will increasingly be used as its cost decreases. However, frontier models will continue to be provided by AI companies in the cloud because training and serving the most capable models requires expensive compute beyond most people’s budgets. Cloud provision also allows providers to control access to new models and observe how those models perform in real-world use.

Even where the primary model runs locally, agents may still need to communicate with external tools, services, customers, robots and other agents. Local inference is therefore an important control, but it is not a complete solution.

Agents Need Internal Access Boundaries

Agents will also need internal boundaries. Employees do not receive unrestricted access to every company system and document; agents likewise should only receive and share the information required for their role and task.

An agent assisting with customer support should not automatically inherit access to source code, legal records or executive communications. An agent coordinating a robot does not necessarily need access to employee records. One agent’s need for information does not mean that every downstream agent needs the same information.

These boundaries must also govern agent-to-agent communication. Information should not propagate simply because another agent requests it. Access should be scoped to the task, limited to the necessary information and constrained by explicit rules about what may be retained or shared.

Information Governance for an Agentic World

This is ultimately a data governance problem. Static policies written for documents and databases will not be enough when information moves continuously through prompts, tools, agents and physical systems. Governance must also account for agents that, if given sufficient permissions, can change configurations, tools or other parts of the operating environment intended to constrain them.

Organizations and individuals will need ways to determine which information an agent may access, which information must be redacted before leaving a trust boundary, which agents may receive it, and what records exist of those disclosures. They will also need controls for information that becomes sensitive through aggregation or inference rather than through its original classification.

There is also an independence problem. AI providers have legitimate goals of their own which are improving their models, enforcing safety and abuse policies, managing capacity and cost, meeting legal obligations, making a profit and expanding into new products. Those goals are not identical to protecting every customer’s competitive advantage or operational continuity. Providers can also (and do) change enforcement practices at short notice.

Enterprise agreements, no-training commitments and compliance controls can reduce risk, but they do not eliminate this dependency. Organizations will also use multiple models, providers and local inference. They need an independent privacy and governance layer that remains under their control and applies consistent policy across that mix, allowing them to use the best available intelligence without exposing more sensitive context than necessary or relying on each provider to enforce the same rules. An agreement with an AI provider does not stop your data leaking via security breaches, data being shared with contractors and employees, court orders, or governments’ backdoors.

Privacy and restriction of information will become critical infrastructure used by all companies and individuals. The objective is not to prevent agents from using information. It is to ensure that increasingly powerful agents can only use the information they actually need without every interaction becoming an uncontrolled transfer of knowledge to the outside world.

Agent Redactor: The First Step Toward Verifiable Control

To build this multi-modal privacy infrastructure, we have to start where agents are bleeding information most heavily today. That is text, prompts, and tool calls that are sent to AI providers.

Agent Redactor is an open-source local proxy that sits directly between AI agents and their LLM endpoints. Using an on-device AI model, deterministic rules, and keyword matching, it intercepts and redacts sensitive context from outbound requests before they ever leave your machine. It then seamlessly restores the original values in the responses so your workflows continue to function uninterrupted. It currently been tested to work on Claude Code, Codex, Hermes, OpenClaw, and OpenCode. It should be able to work on other agenetic frameworks as well but it hasn’t been tested.

Today, Agent Redactor addresses the most immediate and practical leakage path which is written context sent from agents to external model providers. But our vision extends far beyond text sent to an AI provider.

Agent Redactor is the foundational step in a much larger vision. Our ultimate goal is to build the exact multi-modal control layer described in this thesis. One capable of governing modalities such as voice, video, physical-world telemetry, and biometric inferences before they are exposed to third-party agents, AI providers, or any untrusted environment. We are building the architecture for a future where you can fully leverage the power of agents without giving up the data that makes your business defensible and your private life your own.

We believe Agent Redactor will be disruptive by shifting information control from the promises of AI providers to a local, verifiable layer governed by the user.

This is a massive undertaking and it requires an open, collaborative approach. The future we described is arriving quickly.

The window to secure our data is right now, before the agentic ecosystem becomes closed and controlled by a few major players. We invite you to check out Agent Redactor. Test it in your workflows, share your feedback, or contribute directly to the codebase. We must demonstrate the demand for verifiable privacy and embed these controls into the system now, before agent architectures become permanently closed off. Join us in building the privacy infrastructure for the agentic age.