reports / dbtjn020iu
codex-claude-council
8 tools · specification 1.2 · requested 2026-10-13 04:39:33 UTC
Server
- Overall
- A4.7definitions 70% · coherence 30%
- Definition quality
- A4.6mean 4.6 · min 4.5 · 8/8 tools
- Coherence
- A4.8
The codex-claude-council server presents a coherent and well-organized tool set for orchestrating multi-model discussions. Naming is consistent, tools are clearly differentiated, and the count is appropriate for the domain. The only minor concern is the close similarity between council_ask and council_debate, but their distinct output purposes mitigate confusion. No shadowing risks are present as all tools have distinct purposes and similar costs.
- 4
- DisambiguationThe tools are mostly distinct: council_ask for a single cross-checked answer, council_debate for the full transcript, council_ask_codex and council_ask_claude for single-model queries, and council_join/council_turn/council_result/council_cancel for interactive and job management. However, council_ask and council_debate are very similar in purpose, differing only in output detail, which could cause some confusion.
- 5
- Naming ConsistencyAll tool names follow a consistent 'council_' prefix with clear verb suffixes: ask, debate, ask_codex, ask_claude, join, turn, result, cancel. This is a highly predictable and consistent naming pattern.
- 5
- Tool Count AppropriatenessWith 8 tools, the set is well-scoped for a council-based multi-model interaction server. Each tool serves a distinct function: initiating, joining, managing, and retrieving results, without unnecessary redundancy.
- 5
- CompletenessThe tool surface covers the full lifecycle of a council interaction: starting a council (ask, debate, ask_codex, ask_claude), joining interactively (join, turn), retrieving results (result), and cancelling (cancel). There are no obvious gaps for the stated purpose.
Tools
council_ask
554445A4.6This is an exemplary tool description that clearly explains the tool's purpose, process, and usage guidelines, while distinguishing it from siblings. It provides rich behavioral transparency beyond the annotations, covering costs, timeouts, and session persistence. The parameter descriptions add significant value over the schema, and the description is complete for the tool's complexity, earning high scores across all dimensions.
- 5
- Purpose Clarity · 25%The description clearly states the tool's purpose: 'Get one cross-checked answer from Codex (ChatGPT) and Claude' with a detailed explanation of the process. It distinguishes itself from siblings by explicitly naming alternatives: 'for a quick single opinion use council_ask_codex or council_ask_claude, and for the whole discussion use council_debate.'
- 5
- Usage Guidelines · 20%The description provides explicit when-to-use guidance: 'Use it for decisions, reviews and questions where a second model's check is worth minutes of waiting.' It also gives clear when-not-to-use guidance by pointing to alternatives for quick single opinions or full discussions. Additionally, it warns about time and quota costs, and mentions the job_id fallback for long-running calls.
- 4
- Behavioral Transparency · 20%Annotations declare readOnlyHint=true and destructiveHint=false, and the description reinforces this by stating 'both models read the project but never change it' and 'in plan mode (nothing is changed).' It also discloses that the tool runs local CLIs on the user's subscriptions, consumes quota, and can take minutes, with a 50-second timeout returning a job_id. It adds session persistence behavior ('Each model keeps its session between calls') and network defaults, which are not in annotations. Minor gap: it doesn't explicitly state that the tool is not idempotent, but the description's process implies it.
- 4
- Parameter Semantics · 15%Schema coverage is 100%, so the baseline is 3. The description adds significant value beyond the schema by explaining the purpose and constraints of each parameter. For example, it clarifies that 'skill' is optional and only pass if the user asks, and that 'workspace' should be passed for project-related questions. It also explains the meaning of 'max_rounds' in the context of the exchange process, and the session_id parameters are described with usage instructions. This goes beyond the schema's basic descriptions, earning a 4.
- 4
- Conciseness & Structure · 10%The description is a single, dense paragraph that front-loads the core purpose and process, then covers usage guidance, costs, and key behaviors. It is efficient with no fluff, but it is quite long and could benefit from breaking into shorter sentences or bullet points for readability. However, every sentence adds value, so it earns a 4.
- 5
- Contextual Completeness · 10%Given the tool's complexity (13 parameters, multiple models, session handling, timeouts), the description is remarkably complete. It covers the process, usage scenarios, cost implications, timeout behavior, session persistence, and parameter-specific guidance. The output schema exists, so return values are not needed. The description fully compensates for the lack of annotations on side effects and provides all necessary context for an agent to use the tool correctly.
- params
- 13 (1 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 1
- bytes
- 5966
- hash
- 9c9a7860338ceac0
council_debate
554455A4.7This is an excellent tool description that clearly differentiates from its sibling council_ask, provides explicit usage guidance, and adds valuable behavioral context beyond the annotations. It is well-structured and complete, making it easy for an agent to select and invoke correctly. The only minor gap is that it doesn't explicitly state the output schema's structure, but the output schema itself covers that.
- 5
- Purpose Clarity · 25%The description clearly states the tool runs the same council as council_ask but returns the full discussion as JSON, listing the components (opening answers, exchanges, verdicts, check, settings). It explicitly distinguishes from council_ask by focusing on the transcript output, which is a specific verb+resource+scope.
- 5
- Usage Guidelines · 20%It provides explicit when-to-use guidance: 'Use it when the user wants to see where the models disagreed or how they converged; use council_ask when only the answer matters, since the transcript is long.' This directly contrasts with the sibling tool and gives clear selection criteria.
- 4
- Behavioral Transparency · 20%Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is covered. The description adds valuable context: it mentions the transcript is long, same cost and timing as council_ask, and that a job_id is returned for council_result if still running after ~50s. It does not contradict annotations, and the added behavioral details (long output, async behavior) go beyond the annotations.
- 4
- Parameter Semantics · 15%Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the overall flow (rounds until agreement, synthesizer writes final answer, other checks) and clarifies the meaning of max_rounds (0 means no limit, default 5) and the skill parameter (installed skills, comma-separated, can be from plugins). This goes beyond the schema's basic descriptions, justifying a 4.
- 5
- Conciseness & Structure · 10%The description is well-structured and front-loaded: it starts with the core purpose, then usage guidance, then cost/timing. It is concise, with no redundant sentences, and every sentence adds value. The length is appropriate for the tool's complexity.
- 5
- Contextual Completeness · 10%Given the tool's complexity (13 parameters, async behavior, multiple models), the description is complete. It covers the output format (JSON transcript), the process (rounds, synthesizer, check), cost/timing, and how to handle long-running jobs (job_id for council_result). The output schema exists, so return values are already documented. The description fully compensates for the lack of explicit output details.
- params
- 13 (1 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 1
- bytes
- 5712
- hash
- 3e999008e669b802
council_ask_codex
554445A4.6This is an excellent tool description that clearly defines the tool's purpose, distinguishes it from siblings, and provides comprehensive usage guidelines. It adds significant behavioral context beyond the annotations, such as timing, session persistence, and plan-mode read-only behavior. The parameter semantics are well-covered with practical guidance, and the description is well-structured despite its length. It sets a high standard for tool descriptions.
- 5
- Purpose Clarity · 25%The description clearly states the tool's purpose: 'Ask Codex (ChatGPT) alone, through the user's Codex CLI signed in with ChatGPT, and return its answer.' It specifies the resource (Codex/ChatGPT) and the action (ask and return answer), and distinguishes it from siblings by explicitly naming council_ask and council_ask_claude as alternatives.
- 5
- Usage Guidelines · 20%The description provides explicit guidance on when to use this tool: 'Use it for a quick second opinion from ChatGPT; use council_ask when the answer should also be cross-checked by Claude, or council_ask_claude for Claude alone.' It also gives context on when to pass workspace ('whenever the question is about the code') and when to omit it, and explains session_id usage. This is comprehensive and actionable.
- 4
- Behavioral Transparency · 20%Annotations already declare readOnlyHint=true and destructiveHint=false, and the description reinforces this by stating 'it reads the project but never changes it' and 'in plan mode (nothing is changed).' It adds useful behavioral details beyond annotations: 'usually answers within a minute (longer at high effort, when it returns a job_id for council_result)' and 'its session carries over between calls.' It also notes that it uses the user's ChatGPT plan, which is important for cost awareness. Minor gap: it doesn't explicitly state that the tool is non-idempotent, but that's not required given the annotations.
- 4
- Parameter Semantics · 15%Schema description coverage is 100%, so the schema already documents all parameters. The description adds value by explaining the purpose of workspace ('Pass it whenever the question is about the code...') and session_id ('to continue in place instead of a new session, so the model keeps that session's memory'), which goes beyond the schema's basic descriptions. It also clarifies the skill parameter's trigger conditions and installation status. This is above the baseline of 3 because it provides practical usage context for key parameters.
- 4
- Conciseness & Structure · 10%The description is well-structured and front-loaded with the core purpose, then covers usage guidance and key behavioral notes. It is somewhat long (about 150 words) but every sentence adds value, covering alternatives, timing, workspace behavior, and session persistence. It could be slightly more concise, but the density of useful information justifies the length.
- 5
- Contextual Completeness · 10%Given the tool's complexity (8 parameters, multiple modes, session handling, and integration with external CLI), the description is remarkably complete. It covers when to use it, how it behaves (read-only, session carryover, timing), and parameter semantics. The output schema exists, so return values are not the description's responsibility. The description effectively compensates for the lack of detailed annotations on parameters and provides all necessary context for an agent to select and invoke the tool correctly.
- params
- 8 (1 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 1
- bytes
- 4041
- hash
- 510f49dc3bd5627d
council_ask_claude
554455A4.7This is an exemplary tool description that excels in all dimensions. It clearly defines the tool's purpose, provides explicit usage guidelines with sibling differentiation, and adds rich behavioral context beyond the annotations. The parameter semantics are well-explained, and the description is concise yet comprehensive. It sets a high standard for MCP tool descriptions.
- 5
- Purpose Clarity · 25%The description clearly states the tool's purpose: 'Ask Claude alone, through the user's Claude Code CLI... and return its answer.' It specifies the resource (Claude Code CLI) and the action (ask), and distinguishes it from siblings by explicitly naming council_ask and council_ask_codex as alternatives. This is a specific verb+resource with clear differentiation.
- 5
- Usage Guidelines · 20%The description provides explicit when-to-use guidance: 'Use it for a quick second opinion from Claude, for example from Codex or ChatGPT; use council_ask when the answer should also be cross-checked by Codex, or council_ask_codex for Codex alone.' It also gives context on when to pass workspace ('Pass it whenever the question is about the code...') and when to omit it ('Omit only for questions unrelated to any project'). This is exemplary usage guidance.
- 4
- Behavioral Transparency · 20%Annotations already declare readOnlyHint=true and destructiveHint=false, and the description reinforces this by stating 'it reads the project in plan mode and never changes it.' It adds valuable context beyond annotations: it mentions the user's Claude plan, typical response time ('usually answers within a minute'), and session carry-over behavior. It also notes that high effort may return a job_id for council_result. The only minor gap is not detailing what happens on errors or timeouts, but the description adds significant behavioral context.
- 4
- Parameter Semantics · 15%Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the purpose of each parameter in context: for example, it clarifies that 'model' should be set 'only when the user asks for a specific one,' and that 'workspace' should be passed 'whenever the question is about the code.' It also explains the 'skill' parameter's trigger conditions and installation status. This goes beyond the schema's basic descriptions, earning a 4.
- 5
- Conciseness & Structure · 10%The description is well-structured and front-loaded: the first sentence states the core purpose, followed by usage guidance, then behavioral details, and finally parameter-specific notes. Every sentence adds value, and it is appropriately sized for a tool with 7 parameters. There is no redundancy or fluff.
- 5
- Contextual Completeness · 10%Given the tool's complexity (7 parameters, multiple siblings, output schema present), the description is remarkably complete. It covers when to use, when not to use, behavioral traits (plan mode, session carry-over, time expectations), and parameter semantics. The output schema exists, so return values are not the description's responsibility. This is a model description for a complex tool.
- params
- 7 (1 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 1
- bytes
- 3823
- hash
- a5a8a8ab9e425a67
council_join
554445A4.6This is an excellent tool description that clearly explains a complex, interactive tool. It distinguishes itself from siblings, provides explicit usage guidelines, and adds behavioral context beyond the annotations. The parameter semantics are well-covered by the schema, and the description adds workflow context. Minor improvements could be made in structure, but overall it is highly effective.
- 5
- Purpose Clarity · 25%The description clearly states the tool's purpose: the host model participates directly in the council conversation rather than spawning a separate session. It explicitly contrasts with council_ask (two separate sessions) and mentions the sibling tools council_turn and council_result, distinguishing it from alternatives. The verb 'Take part' and resource 'council' are specific and unambiguous.
- 5
- Usage Guidelines · 20%The description provides explicit when-to-use guidance: 'Use this when the user wants you to discuss with the other model, or gives the id of their session of the other model; use council_ask instead when the user wants a council of two separate sessions.' It also explains the workflow (answer independently, reply until agreement, write and check final answer) and clarifies that the other model runs a CLI session while the host writes replies in-conversation. This is comprehensive and actionable.
- 4
- Behavioral Transparency · 20%Annotations declare readOnlyHint=true, destructiveHint=false, and openWorldHint=true, which already indicate safe, read-only, and potentially network-accessible behavior. The description adds valuable context: the host model participates directly, the other model runs a CLI session, and the tool returns the council's message for the host to reply to. It also notes that the other model's session may be the user's own (other_session_id), which is a behavioral nuance not captured by annotations. Minor gap: it doesn't explicitly state that the tool is non-idempotent (idempotentHint=false), but the description implies stateful interaction.
- 4
- Parameter Semantics · 15%Schema description coverage is 100%, so the schema already documents all 11 parameters. The description adds value by explaining the overall workflow and how parameters like 'me', 'other_session_id', and 'workspace' fit into the council process. It also clarifies the 'skill' parameter's impact on time and usage, and the 'max_rounds' semantics. However, some parameter details (e.g., 'network' and 'web_search' defaults) are already in the schema, so the description doesn't add much beyond that. Still, it provides context that helps the agent decide when to set each parameter.
- 4
- Conciseness & Structure · 10%The description is a single dense paragraph that front-loads the core purpose and workflow, then covers usage guidelines and parameter context. It is efficient but somewhat long; however, every sentence adds necessary information about the tool's unique behavior and when to use it. The structure could be improved with bullet points, but it remains readable and focused.
- 5
- Contextual Completeness · 10%Given the tool's complexity (11 parameters, multiple sibling tools, and a non-trivial workflow), the description is remarkably complete. It explains the interaction model, the role of the host model, the distinction from council_ask, and the purpose of key parameters. The output schema exists, so return values are not described, which is acceptable. The description covers all necessary aspects for an agent to use the tool correctly, including when to omit parameters and when to set them.
- params
- 11 (1 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 1
- bytes
- 5485
- hash
- b8d2df8cd3e03a7d
council_turn
554355A4.5This is a well-crafted tool description that clearly explains the purpose, usage conditions, and fallback behavior. It effectively distinguishes from sibling tools and provides actionable guidance for an agent. The only minor issue is a potential tension with the readOnlyHint annotation, but the description's clarity compensates for it.
- 5
- Purpose Clarity · 25%The description clearly states the tool's purpose: sending a reply to the council's latest message after a turn is granted. It specifies the action (send your reply), the resource (council's latest message), and the context (after council_join, council_turn, or council_result returned your turn). It also distinguishes from siblings by mentioning council_result as an alternative when the council is not waiting.
- 5
- Usage Guidelines · 20%The description provides explicit when-to-use guidance: use after a turn is granted, and only for councils started with council_join. It also gives clear when-not-to-use instructions: if the council is not waiting, call council_result instead, and if the next step takes longer than ~50s, call council_result with the council_id. This is excellent guidance for an agent to select the correct tool.
- 4
- Behavioral Transparency · 20%The annotations declare readOnlyHint=true, which is somewhat contradictory because sending a turn is a state-changing action, but the description clarifies that the text is passed to the other model as written, implying a side effect. The description adds important behavioral context: the text is passed verbatim, and the tool waits for the next step. It also mentions the timeout behavior (50s) and the fallback to council_result, which goes beyond annotations. However, it does not explicitly address the readOnlyHint contradiction, so a small deduction is warranted.
- 3
- Parameter Semantics · 15%The schema already provides 100% coverage for both parameters, with clear descriptions for each. The tool description adds minimal extra meaning beyond what the schema provides, but it does emphasize that the text must be the complete reply written by the agent, which reinforces the schema's description. Since the schema covers everything, a baseline of 3 is appropriate.
- 5
- Conciseness & Structure · 10%The description is concise and front-loaded, with the core purpose in the first sentence. It packs essential usage guidance, exclusions, and timeout behavior into a few sentences without redundancy. Every sentence adds value, and the structure is logical: purpose, usage conditions, and fallback instructions.
- 5
- Contextual Completeness · 10%Given the tool's moderate complexity, the description is complete. It covers the trigger conditions, the exact behavior (text passed as written), the alternative tool to use when not waiting, and the timeout fallback. The output schema exists, so return values are not needed. The description fully equips an agent to use the tool correctly in all scenarios.
- params
- 2 (2 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 2
- bytes
- 1710
- hash
- dc74f82c3cd5933e
council_result
554455A4.7This is an excellent tool description that clearly explains the purpose, usage, and behavioral nuances. It effectively guides the agent on when to use it versus alternatives, and provides practical details like wait time and retry logic. The description complements the annotations and schema without redundancy, making it a model example.
- 5
- Purpose Clarity · 25%The description clearly states the tool's purpose: to wait for and retrieve the answer from a running council job, identified by a job_id. It explicitly names the source tools (council_ask, council_debate, etc.) and distinguishes it from those by focusing on result retrieval, not initiation.
- 5
- Usage Guidelines · 20%The description provides explicit guidance on when to use this tool: after a council tool returns 'working' status, and it advises against re-asking the same question to avoid duplicate jobs. It also mentions the alternative of omitting job_id to use the most recent job, and notes the wait behavior and retry pattern.
- 4
- Behavioral Transparency · 20%Annotations already indicate readOnlyHint=true and destructiveHint=false, so the description doesn't need to repeat that. It adds valuable behavioral details: the tool waits up to ~50 seconds, returns as soon as something is ready, and may need to be called again if still working. It also clarifies that waiting uses no model quota, which is useful context beyond annotations.
- 4
- Parameter Semantics · 15%The schema already provides a description for job_id, and the description adds context by explaining that it comes from a council tool and that omitting it uses the most recent job. This adds meaning beyond the schema's basic description, though the schema coverage is 100% so the baseline is 3; the extra context justifies a 4.
- 5
- Conciseness & Structure · 10%The description is concise and well-structured, with the main purpose in the first sentence, followed by practical usage details. Every sentence adds value, and it avoids redundancy with the schema or annotations.
- 5
- Contextual Completeness · 10%Given the tool's simplicity (one optional parameter, no nested objects) and the presence of an output schema, the description covers all necessary aspects: what it does, when to use it, how to handle retries, and the cost implication. It is complete for an agent to use effectively.
- params
- 1 (0 required, 100% described)
- annotations
- readOnly=true destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 0
- bytes
- 1616
- hash
- 88b7259e1b13882e
council_cancel
554455A4.7This is an excellent tool description that clearly defines the purpose, provides explicit usage guidelines with an alternative, and transparently discloses the destructive nature of the operation. It is concise yet complete, making it a model example for tool documentation.
- 5
- Purpose Clarity · 25%The description clearly states the tool stops a running council job, enumerates the specific job types it applies to, and explicitly mentions that progress is discarded. It distinguishes itself from siblings by naming the exact council tools it affects and contrasting with council_result.
- 5
- Usage Guidelines · 20%The description provides explicit when-to-use guidance ('when the user asks to stop, or a council is no longer needed') and when-not-to-use ('to check on a job instead, use council_result'). This is a clear alternative and exclusion, making it highly actionable.
- 4
- Behavioral Transparency · 20%Annotations indicate destructiveHint=false, but the description explicitly states that progress is discarded and the answer can no longer be retrieved, which is a critical behavioral disclosure beyond the annotations. It also mentions that CLI processes are ended, adding transparency. However, it doesn't detail side effects like whether the job is permanently deleted or if there are any confirmation steps, but the core destructive nature is well covered.
- 4
- Parameter Semantics · 15%The schema already provides a description for job_id, and the description adds context by explaining that job_id is the one returned by a council tool while still working, and that omitting it uses the most recent job. This adds meaningful usage nuance beyond the schema, though the schema coverage is 100% so the baseline is 3; the extra context justifies a 4.
- 5
- Conciseness & Structure · 10%The description is a single, well-structured sentence that front-loads the action and includes all necessary details without redundancy. It is concise and every clause adds value, from the list of affected tools to the alternative tool reference.
- 5
- Contextual Completeness · 10%Given the tool's simplicity (one optional parameter, no required params, output schema present), the description is complete. It covers the action, the scope, the consequences, and the alternative, leaving no significant gaps for an agent to misuse the tool.
- params
- 1 (0 required, 100% described)
- annotations
- readOnly=false destructive=false idempotent=false openWorld=true
- output schema
- yes
- cost
- 0
- bytes
- 1496
- hash
- e841b2c1e8bef88a
Deterministic findings
No deterministic findings: every hard gate and checklist item passes.
Automate
The same check from a terminal, a CI job or a script, with an API key from your account. Submitting spends one of the account's daily calls; reading the result spends nothing.
The exit code is the verdict, so the command is the gate. A stdio command or a server URL can stand in for the file.
export TDQS_API_KEY=tdqs_...
# exit code 1 below B, so the command is the gate; uvx runs the same command from PyPI
npx mcp-tdqs score --file tools.json --hosted https://tdqs.dev --fail-under BSubmitting answers at once with the report's address. Its score is null until the status is completed, and the overall tier is on the server score.
curl https://tdqs.dev/api/v1/reports \
-H 'Authorization: Bearer tdqs_...' \
-H 'Content-Type: application/json' \
-d @tools.json
# poll the report it names until "status" is "completed"; this one is
curl https://tdqs.dev/api/v1/reports/dbtjn020iu \
-H 'Authorization: Bearer tdqs_...'The package submits the definitions and polls until the report is done; what comes back is the same object the API returns. The scorer itself is exported too, for a model of your own.
import { requestHostedReport } from 'mcp-tdqs';
const report = await requestHostedReport({
apiKey: 'tdqs_...',
hostedUrl: 'https://tdqs.dev',
serverName: 'codex-claude-council',
tools, // the tools/list result
});
if (report.score === null || !['A', 'B'].includes(report.score.serverScore.overallTier)) {
throw new Error(`TDQS ${report.score?.serverScore.overallTier ?? 'failed'}: ${report.url}`);
}The same helper for Python: it submits the definitions, polls until the report is done, and returns the report as a pydantic model. The scorer itself is exported too, for a model of your own.
from tdqs import parse_tool_definitions, request_hosted_report
parsed = parse_tool_definitions(tools_list) # the tools/list result
report = request_hosted_report(
api_key="tdqs_...",
hosted_url="https://tdqs.dev",
server_name="codex-claude-council",
tools=parsed.tools,
)
tier = report.score.server_score.overall_tier if report.score is not None else None
if tier not in ("A", "B"):
raise SystemExit(f"TDQS {tier or 'failed'}: {report.url}")