Skip to content

How-To Guides

Creating a New Delphyne Project

To start a new Delphyne project, we recommend taking the following steps:

  1. Ensure Delphyne is properly installed.
  2. Create a new folder for your project.
  3. Add an initially empty delphyne.yaml file in this folder.
  4. Ensure that your local or global VSCode settings are right.
  5. Create a Python file and define a strategy inside it.
  6. Register this file in the modules section of delphyne.yaml.
  7. To benefit from typechecking, ensure that Pyright is running in strict mode.
  8. Optionally, create a prompts folder for storing Jinja prompt templates.
  9. Create a demonstration file with extension .demo.yaml.
  10. As your project matures, consider adding tests along with a pyproject.toml file.

Running an Oracular Program

There are mainly two ways to run oracular programs:

  • Using StrategyInstance.run_toplevel: As demonstrated in the Overview, one can manually create a policy environment and extract a search stream from a pair of a strategy instance and of a policy. This can be done within any Python script, but by default offers no support for exporting logs and traces, displaying progress, caching LLM requests, handling interruption, etc... Enabling all these features requires substantial additional setup.
  • Running a command: Instead, a command file can be created that specifies a strategy instance, a policy, some search budget along with extra information. The specified command can be launched from within VSCode or from the shell, using the Delphyne CLI (e.g. delphyne run my_command.exec.yaml --cache --update).

Debugging an Oracular Program

Here are some various tips for debugging oracular programs:

  • Strategies can be debugged before associated policies are defined by writing demonstrations, for which the Delphyne VSCode extension provides rich feedback.
  • For VSCode to stop at breakpoints in strategies while evaluating demonstrations, the Delphyne server must be started with a debugger attached. See the box below for how to do this. Instructions for killing and starting Delphyne servers is available here.
  • Debugging messages can be included in strategy trees using the message function.
  • When running oracular programs via commands, the command output features various useful debugging information by default, in the form of policy logs (including a log of all LLM requests, parsing errors, etc...) and of an inspectable trace.
  • If request caching is enabled (by specifying the cache_file argument in the command file or passing the --cache option in the CLI), then any run of a command can be replayed identically with a debugger attached (see information below), unless a nondeterministic policy is used (e.g. most policies that use multiple threads).

Attaching a Debugger to the Delphyne CLI

For debugging purposes, it is useful to attach a Python debugger to the Delphyne CLI. To do so, we recommend defining the following debug-delphyne alias:

alias debug-delphyne='python -m debugpy --listen 5678 --wait-for-client -m delphyne'

In addition, you should add the following to your .vscode/launch.json file.

"name": "Attach",
"type": "debugpy",
"request": "attach",
"connect": {
    "host": "localhost",
    "port": 5678
    }
}
You can then run the Delphyne CLI with a debugger attached by simply substituting delphyne with debug-delphyne. For example, debug-delphyne serve launches a Delphyne server with a debugger attached. Note that the server does not immediately start after running this command, which waits for the user to launch the Attach debugging profile from inside VSCode.

Tuning an Oracular Program

Policies often feature many hyperparameters that must be tuned for the associated oracular program to perform well. The Delphyne standard library defines an Experiment class for running an oracular program on a set of different hyperparameter combinations. It supports the use of multiple workers, allows interrupting and resuming experiments, retrying failed attempts, and caching all LLM requests for replicability.

For a usage example, see examples/find_invariants/experiments.

Replaying a configuration in a debugger

Whether an experiment configuration (i.e. a run that corresponds to a specific hyperparameter setting) succeeds or fails with an exception, it can be replayed within a debugger as follows:

alias debug-python='python -m debugpy --listen 5678 --wait-for-client'
debug-python experiment_cli.py replay <config_name>

Here, experiment_cli.py is the experiment script that calls Experiment.run_cli. See instructions for launching a debugger client on VSCode side.

Writing a Conversational Agent

A common pattern for interacting with LLMs is to have multi-message exchanges where the full conversation history is resent repeatedly. LLMs are also sometimes allowed to request tool calls. This pattern is implemented by the interact strategy from Delphyne's standard library. For usage examples, see:

  • examples/find_invariants/baseline.py: real-world example from the oracular programming paper
  • tests/example_strategies.py:propose_article: simple example also involving tool calls

Vertical vs Horizontal LLM Pipelines

Delphyne supports the bidirectional integration of two complementary kinds of agents: vertical agents, where a specialized program orchestrates calls to LLMs, and horizontal agents, where an LLM orchestrates calls to tools. Delphyne supports the implementation of horizontal agents via its interact strategy, and allows these agents to invoke tools that are themselves implemented as oracular programs -- whether vertical or horizontal.

Performing Expensive Computations in Strategies

For efficiency and replicability reasons, strategies must not directly perform expensive and possibly nondeterministic computations (e.g. a call to an external SMT solver with a wall clock timeout). In such cases, the Compute effect should be used. See the reference page for details and explanations. For example usage, see examples/find_invariants/abduct_and_branch.py and the associated demonstration file.

Maximizing input token caching and preserving reasoning state

In multi-turn conversations with reasoning models, the Chat Completions API used by default by standard_model cannot fully take advantage of input token caching because it does not return reasoning items that can be resent in later requests within the same conversation, thereby repeatedly invalidating the cache.

In contrast, the OpenAI Responses API returns encrypted reasoning items alongside model answers, which can be resent. In addition to improving caching behavior, resending reasoning tokens can save output tokens or even improve performance in some cases.

To leverage the Responses API and resend reasoning tokens, you can pass the following additional arguments to
standard_model:

model = dp.standard_model("gpt-5",
    api_type="responses", 
    use_reasoning_cache=True,
    convert_user_feedback_to_tool=False)

The use_reasoning_cache argument enables a cache that associates each conversation prefix with the hidden reasoning tokens returned for that prefix by the Responses API. Because of this implementation choice, reasoning tokens are a policy detail that never appear in demonstrations or even in replay caches (LLMCache).

The convert_user_feedback_to_tool argument is provided to work around a current limitation of the OpenAI Responses API, which only supports persisting the KV cache across tool calls following an assistant message, and not across ordinary user feedback messages.

When convert_user_feedback_to_tool is set to True, feedback messages are not presented to the API as user messages but are converted into pairs of tool-call and tool-result messages. For this conversion to be possible, feedback messages must be tagged, as enabled by setting the tag_user_feedback_messages flag when calling the few_shot prompting policy:

pp = dp.few_shot(model, tag_user_feedback_messages=True)

All the features described in this section are currently only supported for OpenAI models.