A real review, start to finish

The paper uses the phrase 'for the first time' in the conclusion. Given the rapidly evolving nature of agentic workflow automation (e.g., AFlow, ADAS, and other NAS-inspired agent frameworks mentioned), claiming priority on the 'agentic supernet' concept is risky and potentially inaccurate without a rigorous historical audit of concurrent preprints.

Suggestion: Remove 'for the first time' and instead frame the contribution as a novel application or synthesis of supernet techniques to the agentic domain.

Definition 3.1 contains a notation error: 'where $\mathbb{M}$ and $\mathbb{M}$ correspond to LLM backbones and the set of available LLMs'. The symbol $\mathbb{M}$ is used twice to represent different (or intended different) things.

Suggestion: Correct the second occurrence to a different symbol, such as $\mathcal{M}_{set}$ or $\mathbb{M}_{pool}$.

The paper refers to 'Dilemma 1' and 'Dilemma 2' in the Introduction as if they were clearly marked headers or list items, but they are embedded in paragraph text, making them difficult to locate during a quick scan.

Suggestion: Explicitly format these as bullet points or bolded sub-headers in the Introduction to improve signposting.

The jump from the high-level 'supernet' concept to the 'textual gradient' method in Section 3.3 feels abrupt. While the distinction between differentiable and non-differentiable parts is clear, the justification for why 'textual gradient' is a viable proxy for complex multi-agent architectures lacks a brief theoretical grounding.

Suggestion: Add a sentence explaining that textual gradients treat the agentic system as a differentiable program, allowing for semantic feedback loops that simulate backpropagation.

The paper introduces 'Agentic Supernet', which is conceptually derived from Neural Architecture Search (NAS) supernets applied to multi-agent workflows. While this adaptation is a novel contribution in the context of LLM agents, it draws heavily from the established NAS paradigm.

Suggestion: Emphasize the unique challenges of applying supernets to discrete, black-box agentic operators compared to traditional neural network layers.

The proposed 'textual gradient' is a heuristic that lacks formal grounding, yet it is presented as a form of backpropagation for black-box operators.

Suggestion: Clarify that this is a heuristic/evolutionary search method rather than gradient-based learning.

Experimental results lack variance reporting (standard deviations/error bars) for the stochastic controller network.

Suggestion: Include metrics across at least 3-5 independent training runs.

The 1:4 train-to-test ratio and the use of the same benchmarks for both optimization and evaluation create a high risk of overfitting.

Suggestion: Use a held-out test set strictly separated from the optimization process.

The paper lacks comparisons against simple, cost-aware heuristic baselines to justify the complexity of the full agentic supernet.

Suggestion: Include a baseline that performs dynamic routing based on simple rules to isolate the value of the learned controller.

The conclusion uses 'for the first time' to describe the contribution, which is inaccurate given the existing literature on NAS-inspired agent frameworks.

Suggestion: Reframe the contribution as a novel application of supernet techniques to the agentic domain.

The evaluation relies on a high number of API calls and model-based metrics (pass@1), which are sensitive to the specific choice of LLM backbones; the cost savings are compelling but the overall system complexity of MaAS itself is significant.

Suggestion: Include analysis on the overhead of the controller network's inference time relative to the total cost savings.

The proposed 'textual gradient' (Eq 12) is fundamentally a heuristic, not a gradient. Using it to 'approximate backpropagation' for discrete, black-box agentic operators is mathematically ill-defined and lacks a formal convergence proof. It is essentially an LLM-based prompt optimization loop mislabeled as gradient-based optimization.

Suggestion: Clarify that this is an evolutionary/heuristic search method rather than gradient-based learning. Re-evaluate the claim that this constitutes 'backpropagation' or 'gradient approximation'.

The claim of 'superior cross-LLM-backbone transferability' is evaluated only on Qwen-2.5-70b and Llama-3.1-70b, which are models structurally and behaviorally similar to the training backbone (gpt-4o-mini). There is no evaluation on models with significantly different architectural biases or capabilities (e.g., smaller parameter models or different architectures), which weakens the claim of generalizability.

Suggestion: Include smaller-scale or non-transformer-based models to empirically support the claim of broad transferability.

The experimental design uses a 1:4 train-to-test ratio for all benchmarks. This is non-standard for LLM benchmarks where models are typically evaluated in zero-shot or few-shot settings on static test sets. There is a high risk of overfitting the 'agentic supernet' to the small test sets, as the 'optimization' occurs on both sets.

Suggestion: Use a held-out test set for evaluation that is strictly separate from any optimization data. Clarify if the 'test' results in tables were used for any hyperparameter tuning or architectural selection.

The paper does not adequately address the 'cold start' problem of the supernet. Since the supernet depends on initial performance feedback to guide the controller, the initial search phase's computational cost and instability are significant practical barriers not discussed.

Suggestion: Acknowledge the instability and computational requirements of the initial training phase.

The paper compares against numerous SOTA baselines but lacks a comparison against a simple 'query-dependent cost-aware' baseline that uses a rule-based or heuristic approach (e.g., length-based or difficulty-based routing without the full neural supernet) to determine if the complex MaAS framework is strictly necessary.

Suggestion: Include a simpler baseline that dynamically switches between standard CoT and more complex tools based on basic heuristic rules to quantify the specific gain provided by the learned supernet vs. dynamic triggering.

Results are reported as point estimates without confidence intervals or variance metrics across different seeds/executions, despite the stochastic nature of the sampling process in the controller network (Eq. 7).

Suggestion: Include standard deviations/error bars over at least 3-5 independent training runs of the MaAS framework.

The ablation study lacks a key check: the impact of the 'early-exit' operator on datasets where complexity is uniformly high. It is unclear if the cost efficiency is purely due to the early-exit or if the supernet is actually learning effective architectures.

Suggestion: Perform an ablation where the search space is restricted to a fixed, 'best-performing' architecture found by the supernet versus the full dynamic supernet to show the benefit of query-dependent sampling.

The training set size for the HumanEval and MATH benchmarks (33 and 119 samples, respectively) is quite small for training a controller network for a supernet, raising questions about potential overfitting to these specific prompts.

Suggestion: Discuss the risk of overfitting or report zero-shot/transfer performance on held-out prompts of similar difficulty to ensure the 'learned' controller isn't just memorizing the training subset.