← Back to blog

Beyond Model Evals and System Prompts: Platform Harness Tools

August 20, 2026

This is an investigation into AI chat implementations by several of the big AI players with distinctly different platform capabilities: Meta AI, Grok, Snap MyAI, ChatGPT, and Perplexity¹. Specifically the tools and data that these platforms have enabled their platform AIs with. The probes were completed in July 2026. Findings suggest a core set of tools (web search, memory) while also demonstrating that platforms are experimenting with varying degrees of integration (and perhaps, also, investment).

Motivation: Investigating AI Platform Specialties

Consumer AI products wrap models with a harness, augmented system prompts, and dynamic tools. There has been extensive research and evaluation on the core models, especially via task suites (MMLU, GSM8K, HumanEval, HellaSwag, etc).² There has also been considerable investigation into the system prompts being used (GitHub users asgeirtj, elder-plinius, and x1xhlol)³. But there has been less focus on what is dynamically loaded into platform AI chat products, especially functions and platform-specific data access.

What’s so interesting about dynamically loaded prompts and tools? Isn’t this just boring plumbing? The agent harness changes the model behavior. SWE-bench recognizes this and publishes results as model and agent pairs.⁴ Garry Tan advocates for “fat skills and fat data” as the “brain”, integrated with the model by a thin harness.⁵ He argues the real leverage is in the skills and data available to the model. Not only can the harness make the model smarter, by deeper reasoning or access to verified information, but it also grants the agent capabilities. Taking an action and reading the result is often the smartest available next step. A big part of why agents are so good at coding is that they can run the code, then run the tests against it.

This investigation uncovers how some of the big players are approaching this from a consumer chat lens within their platforms.

Experiment: Probing Popular Platforms

This investigation takes some handpicked platforms with an emphasis on social media players: ChatGPT, Meta AI, Grok, Snap MyAI, and Perplexity. I also included the GPT API as a “direct model” comparison baseline. I started out with a large number of prompts, and did some playing to see what proved interesting before settling on the 24 I used (the prompts). This is not a comprehensive assessment of what these platforms are capable of, but just a single user’s perspective of their behavior, hoping to offer some insight into their intentions as well as what might work well in building a harness. Some results were surprising (ChatGPT has 50+ direct functions!) others less so (Meta AI will use any Meta data it has on you).

These prompts fell under ten categories:

• Self-disclosure
• Context injection
• Tool surface
• Render layer
• Behavioral/policy
• Memory & state
• Multimodal
• Non-obvious/technical details
• Platform fingerprint
• Owner & platform bias

This was all accomplished manually through the platforms’ primary chat interfaces using personal accounts. The goal was to capture “real world” behavior, if a little anecdotal. I’ve grouped the more interesting findings into four categories here, but all prompts and outputs are available in this repo, with additional notes on methodology: https://github.com/codiak/platform-ai-probe

Results: Tool Discovery

Generally speaking, giving an agent a tool is the inclusion of instructions and calling details as part of the system context. We’ll call this “direct tool access”. There is also the concept of lazy loading tools which consists of either including a descriptive name with the ability to pull calling details, or otherwise augmenting what the LLM has access to based on the user’s prompt. Some of the platforms were unwilling to list all of their functions, but they all provided a count which appeared stable across multiple runs:

Function Counts

1. ChatGPT (web) - 53
2. Claude Web - 38 (bonus)
3. Meta AI - 28
4. Perplexity Deep Research - 21
5. Grok on X - 10
6. Perplexity - 4
7. Snap MyAI - “0”

Digging deeper into what kind of tools were available, OpenAI and Perplexity shared a lot of similar tools such as web access, memory, and loading skills. Grok and Meta both exposed unique platform tools such as searching for posts and displaying those posts inside the chat. Here are some notes on where platforms differed:

Meta: Denied access to private messages and view history, but was able to access “your joined Facebook groups (44 total),” a post/reel scan “back to 2004,” and “a semantic search of your own content.” Would not provide exact function names, but willingly described them.

Grok on X: Clearly has no access to private X content such as messages, bookmarks, and even liked posts. In many cases it did not even know my account handle (despite enabling “Personalize Grok using X”). It did however have full advanced search across X, and it appears to be quite good and fast at using it. One unique tool was “view_x_video” which exposes videos as text.

Perplexity: Plenty of searching, citation, and display tools, although oddly the agent does not appear fully aware. Even when explicitly told not to search, Perplexity always searches and lists citations. In a similar vein, when asked about live weather the agent denied being able to show live weather, while also displaying a live AccuWeather widget. It seems like Perplexity is making some rule-based choices for the agent.

ChatGPT: Exposed a flexible set of tools for accomplishing a broader set of tasks including running code, setting up recurring tasks, and lazy loading other skills/tools. Compared to most of the other platforms, ChatGPT’s tool set appears oriented towards calling and integrating with external systems (third parties, your filesystem, etc). It was also the only platform to note “ask for structured clarification” as a tool.

Snap MyAI: Snap’s model is severely limited in what it is aware of or able to do. For example, even though it can generate an image, it does not know that until you ask it to generate one. And even then, it will not provide any details on the generation capability. This suggests they are dynamically handling function calling outside of the main chat history, or they have very rigid controls around what makes it into a chat message.

For full outputs, the “system prompt” probe⁶ and “tool enumeration” probe⁷ revealed the most about available tools.

Results: Localization, Personalization, and Other Dynamic Context

Since an LLM is “frozen” in time based on when it was last trained, there is a bare minimum of dynamic system prompting to get it to behave. For one, every platform prompts the model with the current date and time, as well as the knowledge cutoff^8,9. For these results, we are looking at what is always added as general context, in contrast to what might be included for a particular task via a tool call or other means. Most platforms include the user’s name, account handle(s), general location, and device/app information^10,11,12. Grok appears to lean on memory to accomplish this, unless you explicitly allow “personalization with X”.

A few interesting inclusions:

• Perplexity specifically includes “the current president of the USA is Donald Trump.” - this has been noted by other prompt leak projects, and it was specifically a patch for a widely reported bug¹⁰.
• Grok on x.com reports the time in UTC, while Grok on grok.com reports the time in the user’s local time. This suggests the two sites diverge in more ways than one, since this is a core prompt difference⁹.
• Grok also shared its context in markdown format, suggesting it does not internally use the OpenAI-style JSON wrapping format for input.
• The GPT API, while the vanilla baseline, does inject some context: knowledge cutoff (2024-06), a current date, output channel names (analysis, commentary, final), and a numeric “oververbosity” setting⁹.

A few style guidelines were uncovered:

• Meta includes a “no em-dashes” rule⁹.
• Grok is made to stick to KaTeX formatting⁶.
• Perplexity is supplied a response structure and requires citations¹³.

And personalization wise beyond the user’s name or display name:

• Perplexity includes key bits of personal information from memory in each conversation’s context, such as sex and date of birth¹².
• ChatGPT specifically includes account metrics for some reason: account age in weeks, activity cadence, average conversation depth, and average message length¹⁰.

Results: Memory reading and writing

Most platforms have a pretty straightforward memory integration, where the agent can discretely save a detail, and reference it later. I tested this in a simple way: asking it to remember something in one session, and asking it to recall in a new session¹⁴. I was able to differentiate when the AI used a memory tool versus searching previous conversations pretty easily, which seem to be used interchangeably by most. Here’s how it played out:

In summary, most platforms have found both a memory tool and a conversation search tool to be useful. And it’s becoming clear Grok on X is just “tacked on”.

Results: Platform data retrieval

Grok and Meta AI have public data tools accessible to them, and as a result are quite useful as research agents for those platforms. Grok has access to the full X advanced search capabilities. Meta goes even further and is able to query across your liked posts and saved posts, as well as general public content searching. Grok does not support accessing any private/semi-private data, including liked posts, but it can of course access your public posts.

Platform Semantic Search

Giving an agent general search capability provides a more interesting tool than just a search box. Both Grok and Meta AI are quite adept at querying across their platforms for content, and applying appropriate filters. They will run several searches in quick succession to find what you are looking for. But what makes both even more useful for digging into the platform is their access to semantic search. Meta AI has semantic search across posts on Instagram, Facebook, and Threads, and Grok has the same for X content. Neither of these tools are accessible via the common interface (at least not directly).

Grok Semantic Search

Of all the tools I turned up across these platforms, this was the one I found most interesting. x_semantic_search has no equivalent in the X product UI. Embedding-based retrieval with usernames and exclude_usernames filters, from_date/to_date, and a min_score_threshold (default 0.18). X’s advanced search already covers account scoping and date ranges. Semantic matching, with a min_score_threshold you can tune, is uniquely available to Grok^6,13,7.

Meta AI Semantic Search

Meta AI’s search tool supported a large number of parameters^15,7,17:

• semantic_queries
• authors
• key_celebrities
• author_ids
• author_id_file
• platform
• content_type
• location
• since
• until
• ranking_intent
• num_results_per_page
• page_number
• verbosity

I’ve bolded a few of the more intriguing ones, such as key_celebrities and location which expose entity recognition. While not as powerful as the short-lived Facebook Graph Search (which supported queries such as “find mutual friends who like Radiohead and are under 40 years old”), this enables much more than the normal platform search (“What posts of Taylor Swift in New York City have there been recently?”). As noted earlier, these queries run across Instagram, Facebook, and Threads.

Other Platforms

ChatGPT and Perplexity differ in platform integration, and it changes the nature of their products. Both support connectors that point outward to external data sources, while the social media platform AIs point inwards to their specialized and exclusive data.

Platform Knowledge at Training

A social media company training an AI has access to all their own data which is ripe to include in the training. So it would seem the resulting models would have a deeper understanding of their respective platforms. In practice, it can be challenging to tease out these differences. All the frontier LLMs are trained on a massive amount of web data, and so you don’t get something quite as representative of the platform. That is, the massive dataset is likely to dilute the platform-exclusive data.

That said, I did a lightweight test and challenged each AI with writing a platform-optimized post about lemonade. Each platform AI held its own, and oriented pretty well to the respective platforms. They all have an unmistakable LLM-speak, but Grok and Meta did seem to cater to their platforms decently. Their closing lines capture the essence of their outputs, and also the tendency to say “drop a comment/post”. Is “drop” the ultimate or most typical engagement optimization?

Combined text outputs for this test are here, for comparison: https://github.com/codiak/platform-ai-probe/blob/main/outputs/P32.md

And as a follow up prompt, I tested the image generation ability of each. Meta and Snap both determined that a vertical image would be platform-optimal.

Snapshot of Agent Development

More than anything, this investigation has turned into an interesting snapshot of the state of platform AI in July 2026. This selection of platforms produced a nice spectrum of varying platform-integration degrees, with a bare API as the baseline:

One thread this investigation pulls on is how AIs will become more specialized, whether by augmentation, training, or both. Perplexity has hardwired its implementation to retrieve sources, even when explicitly told not to^19,20. This is a clear product decision to make the model analytical and grounded in references. Meta AI leans on my platform activity to make personalized recommendations. When asked for the best social app, it referenced my activity and posts to make a recommendation, and landed on Threads, keeping it in the Meta family²¹. Grok on X is oriented the other way, using the public graph instead of personal data. Its semantic search across the public conversation leads to a platform-oriented consensus, biased by its influential figures^6,10,19.

For social media research (academic, marketing, design, etc), Grok and Meta AI are already powerful tools to add to your toolkit. But proceed with caution, there are biases (Meta’s Threads promotion²¹, Grok’s ideology¹⁹) and fabrications (Meta pretending to check posts¹⁴) to watch out for.

As someone who works with coding agents regularly, the defining aspect of this snapshot is that all of these platform AIs are read-only. Even the most integrated, Grok and Meta AI, cannot post, reply, DM, follow, or delete anything. And it’s not a technical gap: ChatGPT, by contrast, has connectors that take real actions on tools I choose. The platforms are purposefully limiting what a user can accomplish with these AIs. It ultimately means they are not agents, because they cannot act on your behalf inside the platform. It’s just a conversational interface.

Investigation repository: https://github.com/codiak/platform-ai-probe