Skip to content

fix: AnthropicLLM uses deprecated completions API, broken for Claude 3+ and drops full conversation history #2564

Description

@immuhammadfurqan

Bug Description

application/llm/anthropic.py uses the legacy Claude 2-era text completions API:

# lines 47-52
completion = self.anthropic.completions.create(
    model=model,
    max_tokens_to_sample=max_tokens,
    prompt=f"{self.HUMAN_PROMPT} {prompt}{self.AI_PROMPT}",
)

This API is not supported by any Claude 3/3.5/4 model. Additionally both _raw_gen and _raw_gen_stream only extract messages[0] and messages[-1], silently discarding the entire multi-turn conversation history between them. All other LLM providers pass the full messages list.

Expected Behavior

AnthropicLLM should use client.messages.create() / client.messages.stream() (the modern Messages API), pass the full conversation history, and correctly handle system messages via the system= parameter.

Proposed Fix

  • Remove deprecated HUMAN_PROMPT/AI_PROMPT imports and instance variables
  • Rewrite _raw_gen to use anthropic.messages.create(), return response.content[0].text
  • Rewrite _raw_gen_stream to use anthropic.messages.stream(), yield text deltas
  • Add _split_messages() helper to separate system prompt from conversation turns

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions