Evaluating Demonstrations

Evaluation Feedback

DemoFeedback

DemoFeedback = StrategyDemoFeedback | QueryDemoFeedback

Feedback sent by the server for each demonstration in a file.

QueryDemoFeedback `dataclass`

Feedback sent by the server for a standalone query demonstration.

Attributes:

Name	Type	Description
`kind`	`Literal['query']`	Always "query".
`diagnostics`	`list[Diagnostic]`	Global diagnostics.
`answer_diagnostics`	`list[tuple[int, Diagnostic]]`	Diagnostics attached to specific answers.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class QueryDemoFeedback:
    """
    Feedback sent by the server for a standalone query demonstration.

    Attributes:
        kind: Always "query".
        diagnostics: Global diagnostics.
        answer_diagnostics: Diagnostics attached to specific answers.
    """

    kind: Literal["query"]
    diagnostics: list[Diagnostic]
    answer_diagnostics: list[tuple[int, Diagnostic]]

StrategyDemoFeedback `dataclass`

Feedback sent by the server for each strategy demonstration.

Attributes:

Name	Type	Description
`kind`	`Literal['strategy']`	Always "strategy".
`trace`	`Trace`	The resulting browsable trace, which includes all visited nodes.
`answer_refs`	`dict[TraceAnswerId, DemoAnswerId]`	A mapping from answer ids featured in the trace to the position of the corresponding answer in the demonstration. This mapping may be partial. For example, using value hints (e.g., `#flag_value`) forces the demonstration interpreter to create answers on the fly that are not part of the demonstration.
`saved_nodes`	`dict[str, TraceNodeId]`	Nodes saved using the `save` test instruction.
`test_feedback`	`list[TestFeedback]`	Feedback for each test in the demonstration.
`global_diagnostics`	`list[Diagnostic]`	Diagnostics that apply to the whole demonstration (individual tests have their own diagnostics).
`query_diagnostics`	`list[tuple[DemoQueryId, Diagnostic]]`	Diagnostics attached to specific queries.
`answer_diagnostics`	`list[tuple[DemoAnswerId, Diagnostic]]`	Diagnostics attached to specific answers.
`implicit_answers`	`dict[ImplicitAnswerCategory, list[ImplicitAnswer]]`	Implicit answers that were generated on the fly and that can be explicitly added to the demonstration, grouped by category. The dictionary should have no empty value: each mentioned catefory should have at least one implicit answer.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class StrategyDemoFeedback:
    """
    Feedback sent by the server for each strategy demonstration.

    Attributes:
        kind: Always "strategy".
        trace: The resulting browsable trace, which includes all visited
            nodes.
        answer_refs: A mapping from answer ids featured in the
            trace to the position of the corresponding answer in the
            demonstration. This mapping may be **partial**. For example,
            using value hints (e.g., `#flag_value`) forces the
            demonstration interpreter to create answers on the fly that
            are not part of the demonstration.
        saved_nodes: Nodes saved using the `save` test instruction.
        test_feedback: Feedback for each test in the demonstration.
        global_diagnostics: Diagnostics that apply to the whole
            demonstration (individual tests have their own diagnostics).
        query_diagnostics: Diagnostics attached to specific queries.
        answer_diagnostics: Diagnostics attached to specific answers.
        implicit_answers: Implicit answers that were generated on the fly
            and that can be explicitly added to the demonstration,
            grouped by category. The dictionary should have no empty
            value: each mentioned catefory should have at least one
            implicit answer.
    """

    kind: Literal["strategy"]
    trace: Trace
    answer_refs: dict[TraceAnswerId, DemoAnswerId]
    saved_nodes: dict[str, TraceNodeId]
    test_feedback: list[TestFeedback]
    global_diagnostics: list[Diagnostic]
    query_diagnostics: list[tuple[DemoQueryId, Diagnostic]]
    answer_diagnostics: list[tuple[DemoAnswerId, Diagnostic]]
    implicit_answers: dict[ImplicitAnswerCategory, list[ImplicitAnswer]]

TestFeedback `dataclass`

Feedback returned by the demo interpreter for a single test.

The test is considered successful if no diagnostic is a warning or an error. Most of the time, and even when unsuccessful, a test stops at a given node, which can be inspected in the UI and which is indicated in field node_id.

Attributes:

Name	Type	Description
`diagnostics`	`list[Diagnostic]`	List of diagnostics for the test.
`node_id`	`TraceNodeId \| None`	Identifier of the node where the test stopped.

Source code in src/delphyne/analysis/feedback.py

@dataclass
class TestFeedback:
    """
    Feedback returned by the demo interpreter for a single test.

    The test is considered successful if no diagnostic is a warning or an
    error. Most of the time, and even when unsuccessful, a test stops at
    a given node, which can be inspected in the UI and which is
    indicated in field `node_id`.

    Attributes:
        diagnostics: List of diagnostics for the test.
        node_id: Identifier of the node where the test stopped.
    """

    diagnostics: list[Diagnostic]
    node_id: TraceNodeId | None

ImplicitAnswer `dataclass`

An implicit answer that is not part of the demonstration but was generated on the fly.

The VSCode extension then offers to add such answers explicitly in the demonstration. This is particularly useful for handling Compute nodes in demonstrations.

Attributes:

Name	Type	Description
`query_name`	`str`	Query name.
`query_args`	`dict[str, object]`	Arguments passed to the query.
`answer_mode`	`str \| None`	Answer mode.
`answer_content`	`str \| object`	Answer content, as raw text or as a JSON value for structured output.
`answer_structured`	`bool`	Whether the answer is structured.
`answer_tool_calls`	`Sequence[ImplicitAnswerToolCall]`	Associated tool calls.
`answer_justification`	`str \| None`	Justification for the answer.
`comment`	`str \| None`	An optional comment that can be added to provide context, to be logged in the extension's output channel.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class ImplicitAnswer:
    """
    An implicit answer that is not part of the demonstration but was
    generated on the fly.

    The VSCode extension then offers to add such answers explicitly in
    the demonstration. This is particularly useful for handling
    `Compute` nodes in demonstrations.

    Attributes:
        query_name: Query name.
        query_args: Arguments passed to the query.
        answer_mode: Answer mode.
        answer_content: Answer content, as raw text or as a JSON value
            for structured output.
        answer_structured: Whether the answer is structured.
        answer_tool_calls: Associated tool calls.
        answer_justification: Justification for the answer.
        comment: An optional comment that can be added to provide
            context, to be logged in the extension's output channel.
    """

    query_name: str
    query_args: dict[str, object]
    answer_mode: str | None
    answer_content: str | object
    answer_structured: bool
    answer_tool_calls: Sequence[ImplicitAnswerToolCall]
    answer_justification: str | None
    comment: str | None

DemoAnswerId

DemoAnswerId = tuple[int, int]

A (query_id, answer_index) pair that identifies an answer in a demo.

DemoQueryId

DemoQueryId = int

Index of the query in the queries section of a demo.

Diagnostic

Diagnostic = tuple[DiagnosticType, str]

A diagnostic gathers a type (i.e. severity) and a message.

DiagnosticType

DiagnosticType = Literal['error', 'warning', 'info']

Diagnostic type.

Browsable Traces

Trace `dataclass`

A browsable trace.

Raw traces contain all the information necessary to recompute a trace but are not easily manipulated by tools. In comparison, these offer a more redundant but also more explicit view. This module provides a way to convert a trace from the former format to the latter.

Attributes:

Name	Type	Description
`nodes`	`dict[TraceNodeId, Node]`	A mapping from node ids to their description.

Info

A browsable trace features answer identifiers, for which a meaning must be provided externally. For example, the demonstration interpreter also produces a mapping from answer ids to their position in the demonstration file. In addition, commands like run_strategy return a raw trace (core.traces.Trace) in addition to the browsable version, which maps answer ids to their actual content.

Source code in src/delphyne/analysis/feedback.py

@dataclass
class Trace:
    """
    A browsable trace.

    [Raw traces][delphyne.core.traces.Trace] contain all the information
    necessary to recompute a trace but are not easily manipulated by
    tools. In comparison, [these][delphyne.analysis.feedback.Trace]
    offer a more redundant but also more explicit view. This module
    provides a way to convert a trace from the former format to the
    latter.

    Attributes:
        nodes: A mapping from node ids to their description.

    !!! info
        A browsable trace features answer identifiers, for which a
        meaning must be provided externally. For example, the
        demonstration interpreter also produces a mapping from answer
        ids to their position in the demonstration file. In addition,
        commands like `run_strategy` return a raw trace
        (`core.traces.Trace`) in addition to the browsable version,
        which maps answer ids to their actual content.
    """

    nodes: dict[TraceNodeId, Node]

Node `dataclass`

Information about a node.

Attributes:

Name	Type	Description
`kind`	`str`	Name of the node type, or `Success`.
`success_value`	`ValueRepr \| None`	The success value if the node is a success leaf, or `None` otherwise.
`summary_message`	`str \| None`	A short summary message (see the `Node.sumary_message` method).
`leaf_node`	`bool`	Whether the node is a leaf node
`label`	`str \| None`	A label describing the node, which can be useful for writing node selectors (although there is currently no guarantee that the label constitutes a valid selector leading to the node). Currently, the label shows all node tags, separated by "&".
`tags`	`list[str]`	The list of all tags attached to the node.
`properties`	`list[tuple[Reference, NodeProperty]]`	List of node properties (attached queries, nested trees, data fields...). Each property is accompanied by a pretty-printed, local space reference.
`actions`	`list[Action]`	A list of explored actions.
`origin`	`NodeOrigin`	The origin of the node in the global trace.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Node:
    """
    Information about a node.

    Attributes:
        kind: Name of the node type, or `Success`.
        success_value: The success value if the node is a success leaf,
            or `None` otherwise.
        summary_message: A short summary message (see the
            `Node.sumary_message` method).
        leaf_node: Whether the node is a leaf node
        label: A label describing the node, which can be useful for
            writing node selectors (although there is currently no
            guarantee that the label constitutes a valid selector
            leading to the node). Currently, the label shows all node
            tags, separated by "&".
        tags: The list of all tags attached to the node.
        properties: List of node properties (attached queries, nested
            trees, data fields...). Each property is accompanied by a
            pretty-printed, local space reference.
        actions: A list of explored actions.
        origin: The origin of the node in the global trace.
    """

    # TODO: Make node labels into valid selectors that can be used with
    # the `at` instruction in demonstration tests.

    kind: str
    success_value: ValueRepr | None
    summary_message: str | None
    leaf_node: bool
    label: str | None
    tags: list[str]
    properties: list[tuple[Reference, NodeProperty]]
    actions: list[Action]
    origin: NodeOrigin

NodeOrigin

NodeOrigin = (
    Literal["root"]
    | tuple[Literal["child"], TraceNodeId, TraceActionId]
    | tuple[Literal["nested"], TraceNodeId, TraceNodePropertyId]
)

Origin of a node.

A node can be the global root, the child of another node, or the root of a nested tree.

Action `dataclass`

An action associated with a node.

Attributes:

Name	Type	Description
`ref`	`Reference`	Pretty-printed local reference for the action.
`hints`	`list[str] \| None`	If the trace results from executing a demonstration, this provides the list of hints that can be used to recover the action through navigation. Otherwise, it is `None`. Note that this is not identical to `ref.with_hints`. Both could plausibly be shown in the UI but the former is more concise.
`related_success_nodes`	`list[TraceNodeId]`	List of related success nodes. A related success node is a node whose attached value was used in building the action. Indeed, in the VSCode extension's Path View, we get a sequence of actions and for each of them the list of success paths that were involved in building that action.
`related_answers`	`list[TraceAnswerId]`	List of related answers. A related answer is an answer to a local query that is used in building the action. Storing this information is useful to detect useless answers that are not used in any action.
`destination`	`TraceNodeId`	Id of the child node that the action leads to.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Action:
    """
    An action associated with a node.

    Attributes:
        ref: Pretty-printed local reference for the action.
        hints: If the trace results from executing a demonstration,
            this provides the list of hints that can be used to recover
            the action through navigation. Otherwise, it is `None`. Note
            that this is not identical to `ref.with_hints`. Both could
            plausibly be shown in the UI but the former is more concise.
        related_success_nodes: List of related success nodes. A related
            success node is a node whose attached value was used in
            building the action. Indeed, in the VSCode extension's Path
            View, we get a sequence of actions and for each of them the
            list of success paths that were involved in building that
            action.
        related_answers: List of related answers. A related answer is an
            answer to a local query that is used in building the action.
            Storing this information is useful to detect useless answers
            that are not used in any action.
        destination: Id of the child node that the action leads to.
    """

    ref: Reference
    hints: list[str] | None
    related_success_nodes: list[TraceNodeId]
    related_answers: list[TraceAnswerId]
    value: ValueRepr
    destination: TraceNodeId

NodeProperty

NodeProperty = Data | NestedTree | Query

Description of a node property (see NodePropertyId).

Data `dataclass`

Generic property that displays some data.

Attributes:

Name	Type	Description
`kind`	`Literal['data']`	Always "data".
`content`	`str`	string representation of the data content.

Source code in src/delphyne/analysis/feedback.py

@dataclass
class Data:
    """
    Generic property that displays some data.

    Attributes:
        kind: Always "data".
        content: string representation of the data content.
    """

    kind: Literal["data"]
    content: str

NestedTree `dataclass`

A nested tree.

Attributes:

Name	Type	Description
`kind`	`Literal['nested']`	Always "nested".
`strategy`	`str`	Name of the strategy function that induces the tree.
`args`	`dict[str, ValueRepr]`	Arguments passed to the strategy function.
`tags`	`list[str]`	Tags attached to the space induced by the tree.
`node_id`	`TraceNodeId \| None`	Identifier of the root node of the nested tree, or `None` if it is not in the trace (i.e., the nested tree hasn't been explored).

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class NestedTree:
    """
    A nested tree.

    Attributes:
        kind: Always "nested".
        strategy: Name of the strategy function that induces the tree.
        args: Arguments passed to the strategy function.
        tags: Tags attached to the space induced by the tree.
        node_id: Identifier of the root node of the nested tree, or
            `None` if it is not in the trace (i.e., the nested tree hasn't
            been explored).
    """

    kind: Literal["nested"]
    strategy: str
    args: dict[str, ValueRepr]
    tags: list[str]
    node_id: TraceNodeId | None  # None if the subtree hasn't been explored

Query `dataclass`

Information about a query.

Attributes:

Name	Type	Description
`kind`	`Literal['query']`	Always "query".
`name`	`str`	Name of the query.
`args`	`dict[str, object]`	Query arguments, serialized in JSON.
`tags`	`list[str]`	Tags attached to the space induced by the query.
`answers`	`list[Answer]`	All answers to the query present in the trace.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Query:
    """
    Information about a query.

    Attributes:
        kind: Always "query".
        name: Name of the query.
        args: Query arguments, serialized in JSON.
        tags: Tags attached to the space induced by the query.
        answers: All answers to the query present in the trace.
    """

    kind: Literal["query"]
    name: str
    args: dict[str, object]
    tags: list[str]
    answers: list[Answer]

Answer `dataclass`

An answer to a query.

Attributes:

Name	Type	Description
`id`	`TraceAnswerId`	Unique answer identifier.
`value`	`ValueRepr`	Parsed answer value.
`hint`	`tuple[] \| tuple[str] \| None`	If the trace results from executing a demonstration (vs running a policy with tracing enabled), then `hint` is either `()` if the answer corresponds to the default answer and `(l,)` if the answer is labeled with `l`. Otherwise, it is `None`.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Answer:
    """
    An answer to a query.

    Attributes:
        id: Unique answer identifier.
        value: Parsed answer value.
        hint: If the trace results from executing a demonstration (vs
            running a policy with tracing enabled), then `hint` is
            either `()` if the answer corresponds to the default answer
            and `(l,)` if the answer is labeled with `l`. Otherwise, it
            is `None`.
    """

    id: TraceAnswerId
    hint: tuple[()] | tuple[str] | None
    value: ValueRepr

Reference `dataclass`

A reference to a space or to a value.

Several human-readable representations are provided:

Attributes:

Name	Type	Description
`with_ids`	`str`	A pretty-printed, id-based reference.
`with_hints`	`str \| None`	A pretty-printed, hint-based reference. These are typically available in the output of the demonstration interpreter, but not when converting arbitrary traces that result from running policies.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Reference:
    """
    A reference to a space or to a value.

    Several human-readable representations are provided:

    Attributes:
        with_ids: A pretty-printed, id-based reference.
        with_hints: A pretty-printed, hint-based reference. These are
            typically available in the output of the demonstration
            interpreter, but not when converting arbitrary traces that
            result from running policies.
    """

    with_ids: str
    with_hints: str | None

ValueRepr `dataclass`

Multiple representations for a Python object.

We allow providing several representations for Python objects: short, one-liner string descriptions, detailed descriptions, JSON representation... All of these can be leveraged by different tools and UI components.

Attributes:

Name	Type	Description
`short`	`str`	A short representation, typically obtained using the `str` function.
`long`	`str \| None`	A longer, often multi-line representation, typically obtained using the `pprint` module.
`json`	`object`	A JSON representation of the object.
`json_provided`	`bool`	Whether a JSON representation is provided (the JSON field is `None` otherwise). This is not always the case since not all Python objects can be serialized to JSON.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class ValueRepr:
    """
    Multiple representations for a Python object.

    We allow providing several representations for Python objects:
    short, one-liner string descriptions, detailed descriptions, JSON
    representation... All of these can be leveraged by different tools
    and UI components.

    Attributes:
        short: A short representation, typically obtained using the
            `str` function.
        long: A longer, often multi-line representation, typically
            obtained using the `pprint` module.
        json: A JSON representation of the object.
        json_provided: Whether a JSON representation is provided (the
            JSON field is `None` otherwise). This is not always the case
            since not all Python objects can be serialized to JSON.
    """

    short: str
    long: str | None
    json_provided: bool
    json: object

TraceAnswerId

TraceAnswerId = int

Global answer id, as set by core.traces.Trace.

TraceActionId

TraceActionId = int

Index of an action within a given node.

TraceNodePropertyId

TraceNodePropertyId = int

Index of a property within a given node. A property is an element that can be listed in the UI, which is either an attached query, a nested tree or some data.

Demonstration Interpreter

evaluate_demo

evaluate_demo(
    demo: Demo,
    context: DemoExecutionContext,
    *,
    extra_objects: dict[str, object],
    answer_database_loader: AnswerDatabaseLoader,
    load_implicit_answer_generators: ImplicitAnswerGeneratorsLoader,
) -> DemoFeedback

Evaluate a query or strategy demonstration.

This is the main entrypoint of the demonstration interpreter.

Attributes:

Name	Type	Description
`demo`		The demonstration to evaluate.
`context`		The execution context in which to resolve Python identifiers.
`extra_objects`		Additional objects that can be resolved by name (with higher precedence).
`load_implicit_answer_generators`		Load the implicit answer generators (e.g. including the one handling `Compute` nodes). This function is allowed to raise exceptions, which are then reported as errors.

Returns:

Type	Description
`DemoFeedback`	A feedback object containing the results of the evaluation.

Warning

This function creates an ObjectLoader internally and is therefore not thread-safe.

Source code in src/delphyne/analysis/demo_interpreter.py

def evaluate_demo(
    demo: dm.Demo,
    context: DemoExecutionContext,
    *,
    extra_objects: dict[str, object],
    answer_database_loader: dp.AnswerDatabaseLoader,
    load_implicit_answer_generators: ImplicitAnswerGeneratorsLoader,
) -> fb.DemoFeedback:
    """
    Evaluate a query or strategy demonstration.

    This is the main entrypoint of the demonstration interpreter.

    Attributes:
        demo: The demonstration to evaluate.
        context: The execution context in which to resolve Python
            identifiers.
        extra_objects: Additional objects that can be resolved by name
            (with higher precedence).
        load_implicit_answer_generators: Load the implicit answer
            generators (e.g. including the one handling `Compute`
            nodes). This function is allowed to raise exceptions, which
            are then reported as errors.

    Returns:
        A feedback object containing the results of the evaluation.

    !!! warning
        This function creates an `ObjectLoader` internally and is
        therefore not thread-safe.
    """
    if isinstance(demo, dm.StrategyDemo):
        feedback, _ = evaluate_strategy_demo_and_return_trace(
            demo,
            context,
            extra_objects=extra_objects,
            answer_database_loader=answer_database_loader,
            load_implicit_answer_generators=load_implicit_answer_generators,
        )
        return feedback
    else:
        return evaluate_standalone_query_demo(
            demo, context, extra_objects=extra_objects
        )

DemoExecutionContext `dataclass`

Demonstration Execution Context.

Attributes:

Name	Type	Description
`strategy_dirs`	`Sequence[Path]`	A list of directories in which strategy modules can be found, to be added to `sys.path`.
`modules`	`Sequence[str]`	A list of modules in which python object identifiers should be resolved. Modules can be part of packages and so their name may feature `.`.

Source code in src/delphyne/analysis/demo_interpreter.py

@dataclass(frozen=True, kw_only=True)
class DemoExecutionContext:
    """
    Demonstration Execution Context.

    Attributes:
        strategy_dirs: A list of directories in which strategy modules
            can be found, to be added to `sys.path`.
        modules: A list of modules in which python object identifiers
            should be resolved. Modules can be part of packages and so
            their name may feature `.`.
    """

    strategy_dirs: Sequence[Path]
    modules: Sequence[str]

    def with_root(self, root: Path) -> "DemoExecutionContext":
        return DemoExecutionContext(
            strategy_dirs=[root / p for p in self.strategy_dirs],
            modules=self.modules,
        )

ObjectLoader

Utility class for loading Python objects.

Demonstration and command files may refer to Python identifiers that need to be resolved. This is done relative to an execution context (DemoExecutionContext) that specifies a list of directories to be added to sys.path, along with a list of modules.

An exception is raised if an object with the requested identifier can be found in several modules.

Source code in src/delphyne/analysis/demo_interpreter.py

class ObjectLoader:
    """
    Utility class for loading Python objects.

    Demonstration and command files may refer to Python identifiers that
    need to be resolved. This is done relative to an execution context
    (`DemoExecutionContext`) that specifies a list of directories to be
    added to `sys.path`, along with a list of modules.

    An exception is raised if an object with the requested identifier
    can be found in several modules.
    """

    def __init__(
        self,
        ctx: DemoExecutionContext,
        extra_objects: dict[str, object] | None = None,
        reload: bool = True,
    ):
        """
        Attributes:
            ctx: The execution context in which to resolve Python
                identifiers.
            extra_objects: Additional objects that can be resolved by
                name (with higher precedence).
            reload: Whether to reload all modules specified in the
                execution context upon initialization. Setting this
                value to `True` makes `ObjectLoader` not thread-safe
                (also, multiple instances must not be used in an
                overlappaping way within a single thread).

        Raises:
            ModuleNotFound: a module could not be found.
        """
        self.ctx = ctx
        self.extra_objects = extra_objects if extra_objects is not None else {}
        self.modules: list[Any] = []
        with _append_path(self.ctx.strategy_dirs):
            for module_name in ctx.modules:
                try:
                    module = __import__(module_name)
                    if reload:
                        module = importlib.reload(module)
                    self.modules.append(module)
                except AttributeError:
                    raise ModuleNotFound(module_name)

    def find_object(self, name: str) -> Any:
        """
        Find an object with a given name.

        If the name is unqualified (it features no `.`), one attempts to
        find the object in every registered module in order. If the name
        is qualified, one looks at the specified registered module.

        Raises:
            ObjectNotFound: The object could not be found.
            AmbiguousObjectIdentifier: The object name is ambiguous,
                i.e. it is found in several modules.
        """
        if name in self.extra_objects:
            return self.extra_objects[name]
        comps = name.split(".")
        assert comps
        if len(comps) == 1:
            # unqualified name
            cands: list[object] = []
            modules_with_id: dict[int, list[str]] = defaultdict(list)
            for module in self.modules:
                if hasattr(module, name):
                    obj = getattr(module, name)
                    modules_with_id[id(obj)].append(module)
                    cands.append(obj)
            if len(modules_with_id) > 1:
                ambiguous = [ms[0] for ms in modules_with_id.values()]
                raise AmbiguousObjectIdentifier(name, ambiguous)
            if cands:
                return cands[0]
        else:
            # qualified name
            module = ".".join(comps[:-1])
            attr = comps[-1]
            if hasattr(module, attr):
                return getattr(module, attr)
        raise ObjectNotFound(name)

    def load_and_call_function(self, name: str, args: dict[str, Any]) -> Any:
        """
        Load and call a function by wrapping a call to `find_object`.
        """
        f = self.find_object(name)
        args = tp.parse_function_args(f, args)
        return f(**args)

    def load_strategy_instance(
        self, name: str, args: dict[str, Any]
    ) -> dp.StrategyComp[Any, Any, Any]:
        """
        Load and instantiate a strategy function with given arguments.

        Raises:
            ObjectNotFound: If the strategy function cannot be found.
            AmbiguousObjectIdentifier: If an ambiguous name is given.
            StrategyLoadingError: If the object is not a strategy function
                or if the arguments are invalid.
        """
        f = self.find_object(name)
        try:
            args = tp.parse_function_args(f, args)
            comp = f(**args)
            assert isinstance(comp, dp.StrategyComp), (
                f"Object {name} is not a strategy function."
                + " Did you forget to use the @strategy decorator?"
            )
            return cast(Any, comp)
        except Exception as e:
            raise StrategyLoadingError(str(e))

    def load_query(
        self, name: str, args: dict[str, Any]
    ) -> dp.AbstractQuery[Any]:
        """
        Load a query by name and instantiate it with given arguments.

        Raises:
            ObjectNotFound: if the query cannot be found.
            AmbiguousObjectIdentifier: if an ambiguous name is given.
            AssertionError: if the object is not a query.
        """
        obj = self.find_object(name)
        assert issubclass(obj, dp.AbstractQuery), (
            f"Object {name} is not a query type."
        )
        q = cast(type[dp.AbstractQuery[Any]], obj)
        return q.parse_instance(args)

init

__init__(
    ctx: DemoExecutionContext,
    extra_objects: dict[str, object] | None = None,
    reload: bool = True,
)

Attributes:

Name	Type	Description
`ctx`		The execution context in which to resolve Python identifiers.
`extra_objects`		Additional objects that can be resolved by name (with higher precedence).
`reload`		Whether to reload all modules specified in the execution context upon initialization. Setting this value to `True` makes `ObjectLoader` not thread-safe (also, multiple instances must not be used in an overlappaping way within a single thread).

Raises:

Type	Description
`ModuleNotFound`	a module could not be found.

Source code in src/delphyne/analysis/demo_interpreter.py

def __init__(
    self,
    ctx: DemoExecutionContext,
    extra_objects: dict[str, object] | None = None,
    reload: bool = True,
):
    """
    Attributes:
        ctx: The execution context in which to resolve Python
            identifiers.
        extra_objects: Additional objects that can be resolved by
            name (with higher precedence).
        reload: Whether to reload all modules specified in the
            execution context upon initialization. Setting this
            value to `True` makes `ObjectLoader` not thread-safe
            (also, multiple instances must not be used in an
            overlappaping way within a single thread).

    Raises:
        ModuleNotFound: a module could not be found.
    """
    self.ctx = ctx
    self.extra_objects = extra_objects if extra_objects is not None else {}
    self.modules: list[Any] = []
    with _append_path(self.ctx.strategy_dirs):
        for module_name in ctx.modules:
            try:
                module = __import__(module_name)
                if reload:
                    module = importlib.reload(module)
                self.modules.append(module)
            except AttributeError:
                raise ModuleNotFound(module_name)

find_object

find_object(name: str) -> Any

Find an object with a given name.

If the name is unqualified (it features no .), one attempts to find the object in every registered module in order. If the name is qualified, one looks at the specified registered module.

Raises:

Type	Description
`ObjectNotFound`	The object could not be found.
`AmbiguousObjectIdentifier`	The object name is ambiguous, i.e. it is found in several modules.

Source code in src/delphyne/analysis/demo_interpreter.py

def find_object(self, name: str) -> Any:
    """
    Find an object with a given name.

    If the name is unqualified (it features no `.`), one attempts to
    find the object in every registered module in order. If the name
    is qualified, one looks at the specified registered module.

    Raises:
        ObjectNotFound: The object could not be found.
        AmbiguousObjectIdentifier: The object name is ambiguous,
            i.e. it is found in several modules.
    """
    if name in self.extra_objects:
        return self.extra_objects[name]
    comps = name.split(".")
    assert comps
    if len(comps) == 1:
        # unqualified name
        cands: list[object] = []
        modules_with_id: dict[int, list[str]] = defaultdict(list)
        for module in self.modules:
            if hasattr(module, name):
                obj = getattr(module, name)
                modules_with_id[id(obj)].append(module)
                cands.append(obj)
        if len(modules_with_id) > 1:
            ambiguous = [ms[0] for ms in modules_with_id.values()]
            raise AmbiguousObjectIdentifier(name, ambiguous)
        if cands:
            return cands[0]
    else:
        # qualified name
        module = ".".join(comps[:-1])
        attr = comps[-1]
        if hasattr(module, attr):
            return getattr(module, attr)
    raise ObjectNotFound(name)

load_and_call_function

load_and_call_function(name: str, args: dict[str, Any]) -> Any

Load and call a function by wrapping a call to find_object.

Source code in src/delphyne/analysis/demo_interpreter.py

def load_and_call_function(self, name: str, args: dict[str, Any]) -> Any:
    """
    Load and call a function by wrapping a call to `find_object`.
    """
    f = self.find_object(name)
    args = tp.parse_function_args(f, args)
    return f(**args)

load_strategy_instance

load_strategy_instance(name: str, args: dict[str, Any]) -> StrategyComp[Any, Any, Any]

Load and instantiate a strategy function with given arguments.

Raises:

Type	Description
`ObjectNotFound`	If the strategy function cannot be found.
`AmbiguousObjectIdentifier`	If an ambiguous name is given.
`StrategyLoadingError`	If the object is not a strategy function or if the arguments are invalid.

Source code in src/delphyne/analysis/demo_interpreter.py

def load_strategy_instance(
    self, name: str, args: dict[str, Any]
) -> dp.StrategyComp[Any, Any, Any]:
    """
    Load and instantiate a strategy function with given arguments.

    Raises:
        ObjectNotFound: If the strategy function cannot be found.
        AmbiguousObjectIdentifier: If an ambiguous name is given.
        StrategyLoadingError: If the object is not a strategy function
            or if the arguments are invalid.
    """
    f = self.find_object(name)
    try:
        args = tp.parse_function_args(f, args)
        comp = f(**args)
        assert isinstance(comp, dp.StrategyComp), (
            f"Object {name} is not a strategy function."
            + " Did you forget to use the @strategy decorator?"
        )
        return cast(Any, comp)
    except Exception as e:
        raise StrategyLoadingError(str(e))

load_query

load_query(name: str, args: dict[str, Any]) -> AbstractQuery[Any]

Load a query by name and instantiate it with given arguments.

Raises:

Type	Description
`ObjectNotFound`	if the query cannot be found.
`AmbiguousObjectIdentifier`	if an ambiguous name is given.
`AssertionError`	if the object is not a query.

Source code in src/delphyne/analysis/demo_interpreter.py

def load_query(
    self, name: str, args: dict[str, Any]
) -> dp.AbstractQuery[Any]:
    """
    Load a query by name and instantiate it with given arguments.

    Raises:
        ObjectNotFound: if the query cannot be found.
        AmbiguousObjectIdentifier: if an ambiguous name is given.
        AssertionError: if the object is not a query.
    """
    obj = self.find_object(name)
    assert issubclass(obj, dp.AbstractQuery), (
        f"Object {name} is not a query type."
    )
    q = cast(type[dp.AbstractQuery[Any]], obj)
    return q.parse_instance(args)

ImplicitAnswerGenerator

ImplicitAnswerGenerator = Callable[
    [AnyTree, AttachedQuery[Any]], tuple[ImplicitAnswerCategory, Answer] | None
]

A function that optionally maps a tree node along with a query within this node to an implicit answer for the query. This is useful in particular for supporting Compute nodes in demonstrations.

Evaluating Demonstrations

Evaluation Feedback

DemoFeedback

QueryDemoFeedback dataclass

StrategyDemoFeedback dataclass

TestFeedback dataclass

ImplicitAnswer dataclass

DemoAnswerId

DemoQueryId

Diagnostic

DiagnosticType

Browsable Traces

Trace dataclass

Node dataclass

NodeOrigin

Action dataclass

NodeProperty

Data dataclass

NestedTree dataclass

Query dataclass

Answer dataclass

Reference dataclass

ValueRepr dataclass

TraceAnswerId

TraceActionId

TraceNodePropertyId

Demonstration Interpreter

evaluate_demo

DemoExecutionContext dataclass

ObjectLoader

__init__

find_object

load_and_call_function

load_strategy_instance

load_query

ImplicitAnswerGenerator

QueryDemoFeedback `dataclass`

StrategyDemoFeedback `dataclass`

TestFeedback `dataclass`

ImplicitAnswer `dataclass`

Trace `dataclass`

Node `dataclass`

Action `dataclass`

Data `dataclass`

NestedTree `dataclass`

Query `dataclass`

Answer `dataclass`

Reference `dataclass`

ValueRepr `dataclass`

DemoExecutionContext `dataclass`

init