LangSmith Integration

Overview

Import traces and feedback from your LangSmith projects. Converra extracts the system prompt and a readable transcript so you can optimize based on real production behavior.

What Gets Imported

Converra imports multi-turn conversational traces from LangSmith:

• Traces: Agent execution traces with 2+ user messages (deduped by trace ID)
• Prompt: The first system prompt found in the trace (used to create/dedupe a Converra Prompt)
• Transcript: A flattened, readable conversation built from LLM inputs/outputs
• Root feedback: LangSmith feedback attached to the root trace (if present)
• Basic LLM params: model + sampling params when available

Note

Tool calls are captured and displayed in conversation details. Full tool schema preservation for replay is on the roadmap.

What Gets Skipped

Converra automatically filters out traces that aren't conversational:

Trace Type	Example Use Case	Action
No system prompt	Embeddings, simple completions	Skipped
Single-turn (1 user message)	Summarization, classification, extraction	Skipped
Multi-turn (2+ user messages)	Chatbots, support agents, copilots	Imported

Why? Converra optimizes conversational AI through multi-turn simulation testing. Single Q&A API calls (like "summarize this document") don't benefit from this approach because:

• Our simulation has personas ask follow-up questions — irrelevant for single-turn tasks
• Our evaluation scores conversation quality, not task accuracy
• These traces create noise without providing optimization value

If you primarily have single-turn use cases, contact us — task-based evaluation is on our roadmap.

How It Works

1. Connect - Paste your LangSmith API key
2. Select project - Choose which LangSmith project to import from
3. Enable sync - Turn on continuous sync or import manually
4. Review prompts - Prompts are deduped by content hash and shown in Converra for optimization

Continuous Sync

Once connected, Converra can automatically import new traces from your LangSmith project.

Configuration

Setting	Options	Default
Sync Frequency	Hourly, Every 6 hours, Daily	Daily
Preferred Time	Any hour (UTC)	Midnight UTC

How It Works

• Converra tracks the last imported trace timestamp (watermark)
• Each sync fetches only new traces since the last import
• Duplicates are automatically skipped (by trace ID)
• Insights are generated in the background after import

Manual Import

You can also trigger imports manually at any time from the Integrations page. This is useful for:

• Initial bulk import before enabling continuous sync
• Importing after a sync was paused
• Testing the connection

Compatibility

Converra auto-detects your tracing architecture and applies the right import strategy.

Supported Architectures

Architecture	Description	Status
Simple chatbot	Single agent with native messages	✅ Supported
Multi-agent (per-trace)	Multiple LLM runs in a single trace	✅ Supported
Multi-agent (session-based)	Traces grouped by inputs.session_id	✅ Supported
Embedded history	Conversation in inputs.history or <conversation_history> tags	✅ Supported

Supported LLM Formats

Converra auto-detects and parses trace formats from all major providers (OpenAI, Anthropic, LangChain) — both inputs and outputs. No configuration needed.

Agent Role Detection

For multi-agent systems, Converra automatically classifies agents as routers or responders based on their names and roles.

Not Yet Supported

• thread_id in metadata (use inputs.session_id instead)
• Custom session field locations

Have a setup that's not working? Contact us and we'll help.

Multi-Agent Support

LangSmith projects often contain multi-agent traces (Router → SupportBot → RefundHandler).

Converra discovers all LLM runs in a trace and groups them into agent systems:

• Agent Discovery - See which prompts appear together in traces
• Path Visualization - View the actual execution paths (e.g., Router → Support → Escalation)
• Per-Prompt Scores - Each prompt in the system gets its own performance metrics
• Prompt Type Classification - Conversational prompts vs single-shot (classification, extraction, etc.)

Navigate to the Agents page to see discovered agent systems and their performance.

Polly (LangSmith Assistant)

Polly is great for debugging and prompt editing inside LangSmith. Converra is complementary: we turn traces into measurable, gated improvements you can ship with confidence.

Getting Started

1. Request early access to enable the connector
2. Go to Integrations in the Converra app
3. Click Connect LangSmith
4. Follow the setup wizard

API Key Types

LangSmith supports two types of API keys:

Personal API Key (Recommended)

The simplest option. Go to Settings → Personal → API Keys in LangSmith and create a new key. This works immediately without additional configuration.

Organization-Scoped API Key

If you need organization-level access control, you can use an org-scoped API key:

1. Create an org-scoped API key in Settings → Organization → API Keys
2. Paste the key in Converra

Converra automatically detects your workspace when using org-scoped keys. If you have multiple workspaces, you can manually specify which one to use in Advanced options by entering the Workspace ID (found in Settings → Workspaces).

Full Resync

By default, Converra performs incremental syncs — only importing new traces since the last watermark. You can also trigger a full resync that reimports existing sessions.

When to Use Full Resync

• After updating Converra's trace parsing (to get improved transcript extraction)
• If you suspect conversations were missed during a previous sync
• After re-enabling a paused integration

How It Works

1. Go to Integrations in the Converra app
2. Click Sync on your LangSmith connection
3. Choose Full resync to reimport all conversations within the lookback window

Full resync reimports existing sessions and updates their transcripts without creating duplicates (deduplication by trace ID still applies).

Large Projects

For projects with many traces, Converra handles large lookback queries efficiently. You can safely set lookbackDays: 90 with maxTraces: 2000 without timeouts.

FAQ

Do I need to change my LangSmith instrumentation?

No. If you already trace runs and collect feedback in LangSmith, Converra imports what you have. You can optionally add more detail for better replay fidelity.

How often does Converra sync?

You choose: hourly, every 6 hours, or daily (default). Syncs are incremental—only new traces since the last import are fetched. You can also trigger manual imports anytime.

What about Langfuse?

Langfuse is now supported! See the Langfuse Integration guide. The workflow is similar to LangSmith, with a few differences (Basic Auth, project-scoped API keys, region selection).