Evaluating Demonstrations

Evaluation Feedback

DemoFeedback

DemoFeedback = StrategyDemoFeedback | QueryDemoFeedback

Feedback sent by the server for each demonstration in a file.

QueryDemoFeedback `dataclass`

Feedback sent by the server for a standalone query demonstration.

Attributes:

Name	Type	Description
`kind`	`Literal['query']`	Always "query".
`diagnostics`	`list[Diagnostic]`	Global diagnostics.
`answer_diagnostics`	`list[tuple[int, Diagnostic]]`	Diagnostics attached to specific answers.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class QueryDemoFeedback:
    """
    Feedback sent by the server for a standalone query demonstration.

    Attributes:
        kind: Always "query".
        diagnostics: Global diagnostics.
        answer_diagnostics: Diagnostics attached to specific answers.
    """

    kind: Literal["query"]
    diagnostics: list[Diagnostic]
    answer_diagnostics: list[tuple[int, Diagnostic]]

StrategyDemoFeedback `dataclass`

Feedback sent by the server for each strategy demonstration.

Attributes:

Name	Type	Description
`kind`	`Literal['strategy']`	Always "strategy".
`trace`	`Trace`	The resulting browsable trace, which includes all visited nodes.
`answer_refs`	`dict[TraceAnswerId, DemoAnswerId]`	A mapping from answer ids featured in the trace to the position of the corresponding answer in the demonstration. This mapping may be partial. For example, using value hints (e.g., `#flag_value`) forces the demonstration interpreter to create answers on the fly that are not part of the demonstration.
`saved_nodes`	`dict[str, TraceNodeId]`	Nodes saved using the `save` test instruction.
`test_feedback`	`list[TestFeedback]`	Feedback for each test in the demonstration.
`global_diagnostics`	`list[Diagnostic]`	Diagnostics that apply to the whole demonstration (individual tests have their own diagnostics).
`query_diagnostics`	`list[tuple[DemoQueryId, Diagnostic]]`	Diagnostics attached to specific queries.
`answer_diagnostics`	`list[tuple[DemoAnswerId, Diagnostic]]`	Diagnostics attached to specific answers.
`implicit_answers`	`dict[ImplicitAnswerCategory, list[ImplicitAnswer]]`	Implicit answers that were generated on the fly and that can be explicitly added to the demonstration, grouped by category. The dictionary should have no empty value: each mentioned catefory should have at least one implicit answer.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class StrategyDemoFeedback:
    """
    Feedback sent by the server for each strategy demonstration.

    Attributes:
        kind: Always "strategy".
        trace: The resulting browsable trace, which includes all visited
            nodes.
        answer_refs: A mapping from answer ids featured in the
            trace to the position of the corresponding answer in the
            demonstration. This mapping may be **partial**. For example,
            using value hints (e.g., `#flag_value`) forces the
            demonstration interpreter to create answers on the fly that
            are not part of the demonstration.
        saved_nodes: Nodes saved using the `save` test instruction.
        test_feedback: Feedback for each test in the demonstration.
        global_diagnostics: Diagnostics that apply to the whole
            demonstration (individual tests have their own diagnostics).
        query_diagnostics: Diagnostics attached to specific queries.
        answer_diagnostics: Diagnostics attached to specific answers.
        implicit_answers: Implicit answers that were generated on the fly
            and that can be explicitly added to the demonstration,
            grouped by category. The dictionary should have no empty
            value: each mentioned catefory should have at least one
            implicit answer.
    """

    kind: Literal["strategy"]
    trace: Trace
    answer_refs: dict[TraceAnswerId, DemoAnswerId]
    saved_nodes: dict[str, TraceNodeId]
    test_feedback: list[TestFeedback]
    global_diagnostics: list[Diagnostic]
    query_diagnostics: list[tuple[DemoQueryId, Diagnostic]]
    answer_diagnostics: list[tuple[DemoAnswerId, Diagnostic]]
    implicit_answers: dict[ImplicitAnswerCategory, list[ImplicitAnswer]]

TestFeedback `dataclass`

Feedback returned by the demo interpreter for a single test.

The test is considered successful if no diagnostic is a warning or an error. Most of the time, and even when unsuccessful, a test stops at a given node, which can be inspected in the UI and which is indicated in field node_id.

Attributes:

Name	Type	Description
`diagnostics`	`list[Diagnostic]`	List of diagnostics for the test.
`node_id`	`TraceNodeId \| None`	Identifier of the node where the test stopped.

Source code in src/delphyne/analysis/feedback.py

@dataclass
class TestFeedback:
    """
    Feedback returned by the demo interpreter for a single test.

    The test is considered successful if no diagnostic is a warning or an
    error. Most of the time, and even when unsuccessful, a test stops at
    a given node, which can be inspected in the UI and which is
    indicated in field `node_id`.

    Attributes:
        diagnostics: List of diagnostics for the test.
        node_id: Identifier of the node where the test stopped.
    """

    diagnostics: list[Diagnostic]
    node_id: TraceNodeId | None

ImplicitAnswer `dataclass`

An implicit answer that is not part of the demonstration but was generated on the fly.

The VSCode extension then offers to add such answers explicitly in the demonstration. This is particularly useful for handling Compute nodes in demonstrations.

Attributes:

Name	Type	Description
`query_name`	`str`	Query name.
`query_args`	`dict[str, object]`	Arguments passed to the query.
`answer_mode`	`str \| None`	Answer mode.
`answer_content`	`str \| object`	Answer content, as raw text or as a JSON value for structured output.
`answer_structured`	`bool`	Whether the answer is structured.
`answer_tool_calls`	`Sequence[ImplicitAnswerToolCall]`	Associated tool calls.
`answer_justification`	`str \| None`	Justification for the answer.
`comment`	`str \| None`	An optional comment that can be added to provide context, to be logged in the extension's output channel.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class ImplicitAnswer:
    """
    An implicit answer that is not part of the demonstration but was
    generated on the fly.

    The VSCode extension then offers to add such answers explicitly in
    the demonstration. This is particularly useful for handling
    `Compute` nodes in demonstrations.

    Attributes:
        query_name: Query name.
        query_args: Arguments passed to the query.
        answer_mode: Answer mode.
        answer_content: Answer content, as raw text or as a JSON value
            for structured output.
        answer_structured: Whether the answer is structured.
        answer_tool_calls: Associated tool calls.
        answer_justification: Justification for the answer.
        comment: An optional comment that can be added to provide
            context, to be logged in the extension's output channel.
    """

    query_name: str
    query_args: dict[str, object]
    answer_mode: str | None
    answer_content: str | object
    answer_structured: bool
    answer_tool_calls: Sequence[ImplicitAnswerToolCall]
    answer_justification: str | None
    comment: str | None

DemoAnswerId

DemoAnswerId = tuple[int, int]

A (query_id, answer_index) pair that identifies an answer in a demo.

DemoQueryId

DemoQueryId = int

Index of the query in the queries section of a demo.

Diagnostic `dataclass`

A diagnostic message shown in the editor.

Source code in src/delphyne/analysis/feedback.py

@dataclass(frozen=True)
class Diagnostic:
    """
    A diagnostic message shown in the editor.
    """

    severity: DiagnosticType
    message: str
    tags: Sequence[DiagnosticTag] = ()

DiagnosticType

DiagnosticType = Literal['error', 'warning', 'info']

Diagnostic type.

Browsable Traces

Trace `dataclass`

A browsable trace.

Raw traces contain all the information necessary to recompute a trace but are not easily manipulated by tools. In comparison, these offer a more redundant but also more explicit view. This module provides a way to convert a trace from the former format to the latter.

Attributes:

Name	Type	Description
`nodes`	`dict[TraceNodeId, Node]`	A mapping from node ids to their description.

Info

A browsable trace features answer identifiers, for which a meaning must be provided externally. For example, the demonstration interpreter also produces a mapping from answer ids to their position in the demonstration file. In addition, commands like run_strategy return a raw trace (core.traces.Trace) in addition to the browsable version, which maps answer ids to their actual content.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Trace:
    """
    A browsable trace.

    [Raw traces][delphyne.core.traces.Trace] contain all the information
    necessary to recompute a trace but are not easily manipulated by
    tools. In comparison, [these][delphyne.analysis.feedback.Trace]
    offer a more redundant but also more explicit view. This module
    provides a way to convert a trace from the former format to the
    latter.

    Attributes:
        nodes: A mapping from node ids to their description.

    !!! info
        A browsable trace features answer identifiers, for which a
        meaning must be provided externally. For example, the
        demonstration interpreter also produces a mapping from answer
        ids to their position in the demonstration file. In addition,
        commands like `run_strategy` return a raw trace
        (`core.traces.Trace`) in addition to the browsable version,
        which maps answer ids to their actual content.
    """

    nodes: dict[TraceNodeId, Node]
    spaces: dict[TraceSpaceId, tuple[TraceNodeId, TraceNodePropertyId]]

Node `dataclass`

Information about a node.

Attributes:

Name	Type	Description
`kind`	`str`	Name of the node type, or `Success`.
`success_value`	`ValueRepr \| None`	The success value if the node is a success leaf, or `None` otherwise.
`summary_message`	`str \| None`	A short summary message (see the `Node.sumary_message` method).
`leaf_node`	`bool`	Whether the node is a leaf node
`label`	`str \| None`	A label describing the node, which can be useful for writing node selectors (although there is currently no guarantee that the label constitutes a valid selector leading to the node). Currently, the label shows all node tags, separated by "&".
`tags`	`list[str]`	The list of all tags attached to the node.
`properties`	`list[tuple[Reference, TraceSpaceId \| None, NodeProperty]]`	List of node properties (attached queries, nested trees, data fields...). Each property is accompanied by a pretty-printed, local space reference.
`actions`	`list[Action]`	A list of explored actions.
`origin`	`NodeOrigin`	The origin of the node in the global trace.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Node:
    """
    Information about a node.

    Attributes:
        kind: Name of the node type, or `Success`.
        success_value: The success value if the node is a success leaf,
            or `None` otherwise.
        summary_message: A short summary message (see the
            `Node.sumary_message` method).
        leaf_node: Whether the node is a leaf node
        label: A label describing the node, which can be useful for
            writing node selectors (although there is currently no
            guarantee that the label constitutes a valid selector
            leading to the node). Currently, the label shows all node
            tags, separated by "&".
        tags: The list of all tags attached to the node.
        properties: List of node properties (attached queries, nested
            trees, data fields...). Each property is accompanied by a
            pretty-printed, local space reference.
        actions: A list of explored actions.
        origin: The origin of the node in the global trace.
    """

    # TODO: Make node labels into valid selectors that can be used with
    # the `at` instruction in demonstration tests.

    kind: str
    success_value: ValueRepr | None
    summary_message: str | None
    leaf_node: bool
    label: str | None
    tags: list[str]
    properties: list[tuple[Reference, TraceSpaceId | None, NodeProperty]]
    actions: list[Action]
    origin: NodeOrigin

NodeOrigin

NodeOrigin = (
    Literal["root"]
    | tuple[Literal["child"], TraceNodeId, TraceActionId]
    | tuple[Literal["nested"], TraceNodeId, TraceNodePropertyId]
)

Origin of a node.

A node can be the global root, the child of another node, or the root of a nested tree.

Action `dataclass`

An action associated with a node.

Attributes:

Name	Type	Description
`ref`	`Reference`	Pretty-printed local reference for the action.
`hints`	`list[str] \| None`	If the trace results from executing a demonstration, this provides the list of hints that can be used to recover the action through navigation. Otherwise, it is `None`. Note that this is not identical to `ref.with_hints`. Both could plausibly be shown in the UI but the former is more concise.
`related_success_nodes`	`list[TraceNodeId]`	List of related success nodes. A related success node is a node whose attached value was used in building the action. Indeed, in the VSCode extension's Path View, we get a sequence of actions and for each of them the list of success paths that were involved in building that action.
`related_answers`	`list[TraceAnswerId]`	List of related answers. A related answer is an answer to a local query that is used in building the action. Storing this information is useful to detect useless answers that are not used in any action.
`destination`	`TraceNodeId`	Id of the child node that the action leads to.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Action:
    """
    An action associated with a node.

    Attributes:
        ref: Pretty-printed local reference for the action.
        hints: If the trace results from executing a demonstration,
            this provides the list of hints that can be used to recover
            the action through navigation. Otherwise, it is `None`. Note
            that this is not identical to `ref.with_hints`. Both could
            plausibly be shown in the UI but the former is more concise.
        related_success_nodes: List of related success nodes. A related
            success node is a node whose attached value was used in
            building the action. Indeed, in the VSCode extension's Path
            View, we get a sequence of actions and for each of them the
            list of success paths that were involved in building that
            action.
        related_answers: List of related answers. A related answer is an
            answer to a local query that is used in building the action.
            Storing this information is useful to detect useless answers
            that are not used in any action.
        destination: Id of the child node that the action leads to.
    """

    ref: Reference
    hints: list[str] | None
    related_success_nodes: list[TraceNodeId]
    related_answers: list[TraceAnswerId]
    value: ValueRepr
    destination: TraceNodeId

NodeProperty

NodeProperty = Data | NestedTree | Query

Description of a node property (see NodePropertyId).

Data `dataclass`

Generic property that displays some data.

Attributes:

Name	Type	Description
`kind`	`Literal['data']`	Always "data".
`content`	`str`	string representation of the data content.

Source code in src/delphyne/analysis/feedback.py

@dataclass
class Data:
    """
    Generic property that displays some data.

    Attributes:
        kind: Always "data".
        content: string representation of the data content.
    """

    kind: Literal["data"]
    content: str

NestedTree `dataclass`

A nested tree.

Attributes:

Name	Type	Description
`kind`	`Literal['nested']`	Always "nested".
`strategy`	`str`	Name of the strategy function that induces the tree.
`args`	`dict[str, ValueRepr]`	Arguments passed to the strategy function.
`tags`	`list[str]`	Tags attached to the space induced by the tree.
`node_id`	`TraceNodeId \| None`	Identifier of the root node of the nested tree, or `None` if it is not in the trace (i.e., the nested tree hasn't been explored).

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class NestedTree:
    """
    A nested tree.

    Attributes:
        kind: Always "nested".
        strategy: Name of the strategy function that induces the tree.
        args: Arguments passed to the strategy function.
        tags: Tags attached to the space induced by the tree.
        node_id: Identifier of the root node of the nested tree, or
            `None` if it is not in the trace (i.e., the nested tree hasn't
            been explored).
    """

    kind: Literal["nested"]
    strategy: str
    args: dict[str, ValueRepr]
    tags: list[str]
    node_id: TraceNodeId | None  # None if the subtree hasn't been explored

Query `dataclass`

Information about a query.

Attributes:

Name	Type	Description
`kind`	`Literal['query']`	Always "query".
`name`	`str`	Name of the query.
`args`	`dict[str, object]`	Query arguments, serialized in JSON.
`tags`	`list[str]`	Tags attached to the space induced by the query.
`answers`	`list[Answer]`	All answers to the query present in the trace.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Query:
    """
    Information about a query.

    Attributes:
        kind: Always "query".
        name: Name of the query.
        args: Query arguments, serialized in JSON.
        tags: Tags attached to the space induced by the query.
        answers: All answers to the query present in the trace.
    """

    kind: Literal["query"]
    name: str
    args: dict[str, object]
    tags: list[str]
    answers: list[Answer]

Answer `dataclass`

An answer to a query.

Attributes:

Name	Type	Description
`id`	`TraceAnswerId`	Unique answer identifier.
`value`	`ValueRepr`	Parsed answer value.
`hint`	`tuple[] \| tuple[str] \| None`	If the trace results from executing a demonstration (vs running a policy with tracing enabled), then `hint` is either `()` if the answer corresponds to the default answer and `(l,)` if the answer is labeled with `l`. Otherwise, it is `None`.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Answer:
    """
    An answer to a query.

    Attributes:
        id: Unique answer identifier.
        value: Parsed answer value.
        hint: If the trace results from executing a demonstration (vs
            running a policy with tracing enabled), then `hint` is
            either `()` if the answer corresponds to the default answer
            and `(l,)` if the answer is labeled with `l`. Otherwise, it
            is `None`.
    """

    id: TraceAnswerId
    hint: tuple[()] | tuple[str] | None
    value: ValueRepr

Reference `dataclass`

A reference to a space or to a value.

Several human-readable representations are provided:

Attributes:

Name	Type	Description
`with_ids`	`str`	A pretty-printed, id-based reference.
`with_hints`	`str \| None`	A pretty-printed, hint-based reference. These are typically available in the output of the demonstration interpreter, but not when converting arbitrary traces that result from running policies.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class Reference:
    """
    A reference to a space or to a value.

    Several human-readable representations are provided:

    Attributes:
        with_ids: A pretty-printed, id-based reference.
        with_hints: A pretty-printed, hint-based reference. These are
            typically available in the output of the demonstration
            interpreter, but not when converting arbitrary traces that
            result from running policies.
    """

    with_ids: str
    with_hints: str | None

ValueRepr `dataclass`

Multiple representations for a Python object.

We allow providing several representations for Python objects: short, one-liner string descriptions, detailed descriptions, JSON representation... All of these can be leveraged by different tools and UI components.

Attributes:

Name	Type	Description
`short`	`str`	A short representation, typically obtained using the `str` function.
`long`	`str \| None`	A longer, often multi-line representation, typically obtained using the `pprint` module.
`json`	`object`	A JSON representation of the object.
`json_provided`	`bool`	Whether a JSON representation is provided (the JSON field is `None` otherwise). This is not always the case since not all Python objects can be serialized to JSON.

Source code in src/delphyne/analysis/feedback.py

@dataclass(kw_only=True)
class ValueRepr:
    """
    Multiple representations for a Python object.

    We allow providing several representations for Python objects:
    short, one-liner string descriptions, detailed descriptions, JSON
    representation... All of these can be leveraged by different tools
    and UI components.

    Attributes:
        short: A short representation, typically obtained using the
            `str` function.
        long: A longer, often multi-line representation, typically
            obtained using the `pprint` module.
        json: A JSON representation of the object.
        json_provided: Whether a JSON representation is provided (the
            JSON field is `None` otherwise). This is not always the case
            since not all Python objects can be serialized to JSON.
    """

    short: str
    long: str | None
    json_provided: bool
    json: object

TraceAnswerId

TraceAnswerId = int

Global answer id, as set by core.traces.Trace.

TraceActionId

TraceActionId = int

Index of an action within a given node.

TraceNodePropertyId

TraceNodePropertyId = int

Index of a property within a given node. A property is an element that can be listed in the UI, which is either an attached query, a nested tree or some data.

Demonstration Interpreter

evaluate_demo

evaluate_demo(
    demo: Demo,
    *,
    object_loader: ObjectLoader,
    answer_database_loader: AnswerLoader,
    implicit_answer_generators: Sequence[ImplicitAnswerGenerator],
) -> DemoFeedback

Evaluate a query or strategy demonstration.

This is the main entrypoint of the demonstration interpreter.

Attributes:

Name	Type	Description
`demo`		The demonstration to evaluate.
`object_loader`		An object loader that can be used to resolve query and strategy names.
`extra_objects`		Additional objects that can be resolved by name (with higher precedence).
`implicit_answer_generators`		Load the implicit answer generators (e.g. including the one handling `Compute` nodes).

Returns:

Type	Description
`DemoFeedback`	A feedback object containing the results of the evaluation.

Warning

This function creates an ObjectLoader internally and is therefore not thread-safe.

Source code in src/delphyne/analysis/demo_interpreter.py

def evaluate_demo(
    demo: dm.Demo,
    *,
    object_loader: ObjectLoader,
    answer_database_loader: dp.AnswerLoader,
    implicit_answer_generators: Sequence[ImplicitAnswerGenerator],
) -> fb.DemoFeedback:
    """
    Evaluate a query or strategy demonstration.

    This is the main entrypoint of the demonstration interpreter.

    Attributes:
        demo: The demonstration to evaluate.
        object_loader: An object loader that can be used to resolve
            query and strategy names.
        extra_objects: Additional objects that can be resolved by name
            (with higher precedence).
        implicit_answer_generators: Load the implicit answer
            generators (e.g. including the one handling `Compute`
            nodes).

    Returns:
        A feedback object containing the results of the evaluation.

    !!! warning
        This function creates an `ObjectLoader` internally and is
        therefore not thread-safe.
    """
    if isinstance(demo, dm.StrategyDemo):
        feedback, _ = evaluate_strategy_demo_and_return_trace(
            demo,
            object_loader=object_loader,
            answer_database_loader=answer_database_loader,
            implicit_answer_generators=implicit_answer_generators,
        )
        return feedback
    else:
        return evaluate_standalone_query_demo(
            demo, object_loader=object_loader
        )

ObjectLoader

Utility class for loading Python objects.

Demonstration and command files may refer to Python identifiers that need to be resolved. This is done relative to a list of directories to be added to sys.path, along with a list of modules.

An exception is raised if an object with the requested identifier can be found in several modules.

Source code in src/delphyne/analysis/object_loaders.py

class ObjectLoader:
    """
    Utility class for loading Python objects.

    Demonstration and command files may refer to Python identifiers that
    need to be resolved. This is done relative to a list of directories
    to be added to `sys.path`, along with a list of modules.

    An exception is raised if an object with the requested identifier
    can be found in several modules.
    """

    def __init__(
        self,
        *,
        strategy_dirs: Sequence[Path],
        modules: Sequence[str],
        extra_objects: dict[str, object] | None = None,
        initializers: Sequence[str | ObjectLoaderInitializer] = (),
    ):
        """
        Attributes:
            strategy_dirs: A list of directories in which strategy
                modules can be found, to be added to `sys.path`.
            modules: A list of modules in which python object
                identifiers should be resolved. Modules can be part of
                packages and so their name may feature `.`.
            extra_objects: Additional objects that can be resolved by
                name (with higher precedence).
            initializers: A sequence of initialization functions to call
                before any object is loaded. Each element specifies a
                qualified function name, or a pair of a qualified
                function name and of a dictionary of arguments to pass.
                Each initializer function is called at most once per
                Python process (subsequent calls with possibly different
                arguments are ignored).

        Raises:
            ModuleNotFound: a module could not be found.
        """
        self.extra_objects = extra_objects if extra_objects is not None else {}
        self.modules: list[Any] = []
        with _GLOBAL_OBJECT_LOADER_LOCK:
            with _append_path(strategy_dirs):
                for module_name in modules:
                    try:
                        module = importlib.import_module(module_name)
                        self.modules.append(module)
                    except AttributeError:
                        raise ModuleNotFound(module_name)
            for initializer in initializers:
                match initializer:
                    case str() as name:
                        f = self.find_object(name)
                        args = {}
                    case ObjectLoaderInitializer(name, args):
                        f = self.find_object(name)
                if not callable(f):
                    raise TypeError(f"Initializer {name} is not callable.")
                if id(f) not in _GLOBAL_OBJECT_LOADER_EXECUTED_INITIALIZERS:
                    f(**args)
                    # We only count the initializer as executed after a
                    # successful call. This way, if the initializer
                    # raises an exception, the parent command can be run
                    # again after fixing the issue (e.g., modifying
                    # `delphyne.yaml`).
                    _GLOBAL_OBJECT_LOADER_EXECUTED_INITIALIZERS.add(id(f))

    @staticmethod
    def trivial() -> "ObjectLoader":
        """
        Create a trivial object loader that always fails at loading
        objects.
        """
        return ObjectLoader(strategy_dirs=[], modules=[])

    def find_object(self, name: str) -> Any:
        """
        Find an object with a given name.

        If the name is unqualified (it features no `.`), one attempts to
        find the object in every registered module in order. If the name
        is qualified, one looks at the specified registered module.

        Raises:
            ObjectNotFound: The object could not be found.
            AmbiguousObjectIdentifier: The object name is ambiguous,
                i.e. it is found in several modules.
        """
        if name in self.extra_objects:
            return self.extra_objects[name]
        comps = name.split(".")
        assert comps
        if len(comps) == 1:
            # unqualified name
            cands: list[object] = []
            modules_with_id: dict[int, list[str]] = defaultdict(list)
            for module in self.modules:
                if hasattr(module, name):
                    obj = getattr(module, name)
                    modules_with_id[id(obj)].append(module)
                    cands.append(obj)
            if len(modules_with_id) > 1:
                ambiguous = [ms[0] for ms in modules_with_id.values()]
                raise AmbiguousObjectIdentifier(name, ambiguous)
            if cands:
                return cands[0]
        else:
            # qualified name
            module = ".".join(comps[:-1])
            attr = comps[-1]
            if hasattr(module, attr):
                return getattr(module, attr)
        raise ObjectNotFound(name)

    def load_and_call_function(self, name: str, args: dict[str, Any]) -> Any:
        """
        Load and call a function by wrapping a call to `find_object`.
        """
        f = self.find_object(name)
        args = tp.parse_function_args(f, args)
        return f(**args)

    def load_strategy_instance(
        self, name: str, args: dict[str, Any]
    ) -> dp.StrategyComp[Any, Any, Any]:
        """
        Load and instantiate a strategy function with given arguments.

        Raises:
            ObjectNotFound: If the strategy function cannot be found.
            AmbiguousObjectIdentifier: If an ambiguous name is given.
            StrategyLoadingError: If the object is not a strategy function
                or if the arguments are invalid.
        """
        f = self.find_object(name)
        try:
            args = tp.parse_function_args(f, args)
            comp = f(**args)
            assert isinstance(comp, dp.StrategyComp), (
                f"Object {name} is not a strategy function."
                + " Did you forget to use the @strategy decorator?"
            )
            return cast(Any, comp)
        except Exception as e:
            raise StrategyLoadingError(str(e))

    def load_query(
        self, name: str, args: dict[str, Any]
    ) -> dp.AbstractQuery[Any]:
        """
        Load a query by name and instantiate it with given arguments.

        Raises:
            ObjectNotFound: if the query cannot be found.
            AmbiguousObjectIdentifier: if an ambiguous name is given.
            AssertionError: if the object is not a query.
        """
        obj = self.find_object(name)
        assert issubclass(obj, dp.AbstractQuery), (
            f"Object {name} is not a query type."
        )
        q = cast(type[dp.AbstractQuery[Any]], obj)
        return q.parse_instance(args)

init

__init__(
    *,
    strategy_dirs: Sequence[Path],
    modules: Sequence[str],
    extra_objects: dict[str, object] | None = None,
    initializers: Sequence[str | ObjectLoaderInitializer] = (),
)

Attributes:

Name	Type	Description
`strategy_dirs`		A list of directories in which strategy modules can be found, to be added to `sys.path`.
`modules`		A list of modules in which python object identifiers should be resolved. Modules can be part of packages and so their name may feature `.`.
`extra_objects`		Additional objects that can be resolved by name (with higher precedence).
`initializers`		A sequence of initialization functions to call before any object is loaded. Each element specifies a qualified function name, or a pair of a qualified function name and of a dictionary of arguments to pass. Each initializer function is called at most once per Python process (subsequent calls with possibly different arguments are ignored).

Raises:

Type	Description
`ModuleNotFound`	a module could not be found.

Source code in src/delphyne/analysis/object_loaders.py

def __init__(
    self,
    *,
    strategy_dirs: Sequence[Path],
    modules: Sequence[str],
    extra_objects: dict[str, object] | None = None,
    initializers: Sequence[str | ObjectLoaderInitializer] = (),
):
    """
    Attributes:
        strategy_dirs: A list of directories in which strategy
            modules can be found, to be added to `sys.path`.
        modules: A list of modules in which python object
            identifiers should be resolved. Modules can be part of
            packages and so their name may feature `.`.
        extra_objects: Additional objects that can be resolved by
            name (with higher precedence).
        initializers: A sequence of initialization functions to call
            before any object is loaded. Each element specifies a
            qualified function name, or a pair of a qualified
            function name and of a dictionary of arguments to pass.
            Each initializer function is called at most once per
            Python process (subsequent calls with possibly different
            arguments are ignored).

    Raises:
        ModuleNotFound: a module could not be found.
    """
    self.extra_objects = extra_objects if extra_objects is not None else {}
    self.modules: list[Any] = []
    with _GLOBAL_OBJECT_LOADER_LOCK:
        with _append_path(strategy_dirs):
            for module_name in modules:
                try:
                    module = importlib.import_module(module_name)
                    self.modules.append(module)
                except AttributeError:
                    raise ModuleNotFound(module_name)
        for initializer in initializers:
            match initializer:
                case str() as name:
                    f = self.find_object(name)
                    args = {}
                case ObjectLoaderInitializer(name, args):
                    f = self.find_object(name)
            if not callable(f):
                raise TypeError(f"Initializer {name} is not callable.")
            if id(f) not in _GLOBAL_OBJECT_LOADER_EXECUTED_INITIALIZERS:
                f(**args)
                # We only count the initializer as executed after a
                # successful call. This way, if the initializer
                # raises an exception, the parent command can be run
                # again after fixing the issue (e.g., modifying
                # `delphyne.yaml`).
                _GLOBAL_OBJECT_LOADER_EXECUTED_INITIALIZERS.add(id(f))

trivial `staticmethod`

trivial() -> ObjectLoader

Create a trivial object loader that always fails at loading objects.

Source code in src/delphyne/analysis/object_loaders.py

@staticmethod
def trivial() -> "ObjectLoader":
    """
    Create a trivial object loader that always fails at loading
    objects.
    """
    return ObjectLoader(strategy_dirs=[], modules=[])

find_object

find_object(name: str) -> Any

Find an object with a given name.

If the name is unqualified (it features no .), one attempts to find the object in every registered module in order. If the name is qualified, one looks at the specified registered module.

Raises:

Type	Description
`ObjectNotFound`	The object could not be found.
`AmbiguousObjectIdentifier`	The object name is ambiguous, i.e. it is found in several modules.

Source code in src/delphyne/analysis/object_loaders.py

def find_object(self, name: str) -> Any:
    """
    Find an object with a given name.

    If the name is unqualified (it features no `.`), one attempts to
    find the object in every registered module in order. If the name
    is qualified, one looks at the specified registered module.

    Raises:
        ObjectNotFound: The object could not be found.
        AmbiguousObjectIdentifier: The object name is ambiguous,
            i.e. it is found in several modules.
    """
    if name in self.extra_objects:
        return self.extra_objects[name]
    comps = name.split(".")
    assert comps
    if len(comps) == 1:
        # unqualified name
        cands: list[object] = []
        modules_with_id: dict[int, list[str]] = defaultdict(list)
        for module in self.modules:
            if hasattr(module, name):
                obj = getattr(module, name)
                modules_with_id[id(obj)].append(module)
                cands.append(obj)
        if len(modules_with_id) > 1:
            ambiguous = [ms[0] for ms in modules_with_id.values()]
            raise AmbiguousObjectIdentifier(name, ambiguous)
        if cands:
            return cands[0]
    else:
        # qualified name
        module = ".".join(comps[:-1])
        attr = comps[-1]
        if hasattr(module, attr):
            return getattr(module, attr)
    raise ObjectNotFound(name)

load_and_call_function

load_and_call_function(name: str, args: dict[str, Any]) -> Any

Load and call a function by wrapping a call to find_object.

Source code in src/delphyne/analysis/object_loaders.py

def load_and_call_function(self, name: str, args: dict[str, Any]) -> Any:
    """
    Load and call a function by wrapping a call to `find_object`.
    """
    f = self.find_object(name)
    args = tp.parse_function_args(f, args)
    return f(**args)

load_strategy_instance

load_strategy_instance(name: str, args: dict[str, Any]) -> StrategyComp[Any, Any, Any]

Load and instantiate a strategy function with given arguments.

Raises:

Type	Description
`ObjectNotFound`	If the strategy function cannot be found.
`AmbiguousObjectIdentifier`	If an ambiguous name is given.
`StrategyLoadingError`	If the object is not a strategy function or if the arguments are invalid.

Source code in src/delphyne/analysis/object_loaders.py

def load_strategy_instance(
    self, name: str, args: dict[str, Any]
) -> dp.StrategyComp[Any, Any, Any]:
    """
    Load and instantiate a strategy function with given arguments.

    Raises:
        ObjectNotFound: If the strategy function cannot be found.
        AmbiguousObjectIdentifier: If an ambiguous name is given.
        StrategyLoadingError: If the object is not a strategy function
            or if the arguments are invalid.
    """
    f = self.find_object(name)
    try:
        args = tp.parse_function_args(f, args)
        comp = f(**args)
        assert isinstance(comp, dp.StrategyComp), (
            f"Object {name} is not a strategy function."
            + " Did you forget to use the @strategy decorator?"
        )
        return cast(Any, comp)
    except Exception as e:
        raise StrategyLoadingError(str(e))

load_query

load_query(name: str, args: dict[str, Any]) -> AbstractQuery[Any]

Load a query by name and instantiate it with given arguments.

Raises:

Type	Description
`ObjectNotFound`	if the query cannot be found.
`AmbiguousObjectIdentifier`	if an ambiguous name is given.
`AssertionError`	if the object is not a query.

Source code in src/delphyne/analysis/object_loaders.py

def load_query(
    self, name: str, args: dict[str, Any]
) -> dp.AbstractQuery[Any]:
    """
    Load a query by name and instantiate it with given arguments.

    Raises:
        ObjectNotFound: if the query cannot be found.
        AmbiguousObjectIdentifier: if an ambiguous name is given.
        AssertionError: if the object is not a query.
    """
    obj = self.find_object(name)
    assert issubclass(obj, dp.AbstractQuery), (
        f"Object {name} is not a query type."
    )
    q = cast(type[dp.AbstractQuery[Any]], obj)
    return q.parse_instance(args)

ObjectLoaderInitializer `dataclass`

Specification of a function to be called upon creation of an object loader.

Source code in src/delphyne/analysis/object_loaders.py

@dataclass
class ObjectLoaderInitializer:
    """
    Specification of a function to be called upon creation of an object
    loader.
    """

    function: str
    args: dict[str, Any]

ImplicitAnswerGenerator

ImplicitAnswerGenerator = Callable[
    [AnyTree, AttachedQuery[Any]], tuple[ImplicitAnswerCategory, Answer] | None
]

A function that optionally maps a tree node along with a query within this node to an implicit answer for the query. This is useful in particular for supporting Compute nodes in demonstrations.

Object Loader Exceptions

ModuleNotFound `dataclass`

Bases: Exception

Raised by ObjectLoader when a module is not found.

Source code in src/delphyne/analysis/object_loaders.py

@dataclass
class ModuleNotFound(Exception):
    """
    Raised by `ObjectLoader` when a module is not found.
    """

    module_name: str

ObjectNotFound `dataclass`

Bases: Exception

Raised by ObjectLoader when an object cannot be found.

Source code in src/delphyne/analysis/object_loaders.py

@dataclass
class ObjectNotFound(Exception):
    """
    Raised by `ObjectLoader` when an object cannot be found.
    """

    object_name: str

StrategyLoadingError `dataclass`

Bases: Exception

Raised by ObjectLoader when a strategy instance cannot be loaded.

Source code in src/delphyne/analysis/object_loaders.py

@dataclass
class StrategyLoadingError(Exception):
    """
    Raised by `ObjectLoader` when a strategy instance cannot be loaded.
    """

    message: str

AmbiguousObjectIdentifier `dataclass`

Bases: Exception

Raised when attempting to load an object with an ambiguous name.

Attributes:

Name	Type	Description
`identifier`	`str`	the ambiguous identifier.
`modules`	`Sequence[str]`	a list of modules where different objects with the same identifier were found

Source code in src/delphyne/analysis/object_loaders.py

@dataclass(frozen=True)
class AmbiguousObjectIdentifier(Exception):
    """
    Raised when attempting to load an object with an ambiguous name.

    Attributes:
        identifier: the ambiguous identifier.
        modules: a list of modules where different objects with the same
            identifier were found
    """

    identifier: str
    modules: Sequence[str]

Evaluating Demonstrations

Evaluation Feedback

DemoFeedback

QueryDemoFeedback dataclass

StrategyDemoFeedback dataclass

TestFeedback dataclass

ImplicitAnswer dataclass

DemoAnswerId

DemoQueryId

Diagnostic dataclass

DiagnosticType

Browsable Traces

Trace dataclass

Node dataclass

NodeOrigin

Action dataclass

NodeProperty

Data dataclass

NestedTree dataclass

Query dataclass

Answer dataclass

Reference dataclass

ValueRepr dataclass

TraceAnswerId

TraceActionId

TraceNodePropertyId

Demonstration Interpreter

evaluate_demo

ObjectLoader

__init__

trivial staticmethod

find_object

load_and_call_function

load_strategy_instance

load_query

ObjectLoaderInitializer dataclass

ImplicitAnswerGenerator

Object Loader Exceptions

ModuleNotFound dataclass

ObjectNotFound dataclass

StrategyLoadingError dataclass

AmbiguousObjectIdentifier dataclass

QueryDemoFeedback `dataclass`

StrategyDemoFeedback `dataclass`

TestFeedback `dataclass`

ImplicitAnswer `dataclass`

Diagnostic `dataclass`

Trace `dataclass`

Node `dataclass`

Action `dataclass`

Data `dataclass`

NestedTree `dataclass`

Query `dataclass`

Answer `dataclass`

Reference `dataclass`

ValueRepr `dataclass`

init

trivial `staticmethod`

ObjectLoaderInitializer `dataclass`

ModuleNotFound `dataclass`

ObjectNotFound `dataclass`

StrategyLoadingError `dataclass`

AmbiguousObjectIdentifier `dataclass`