Delay-corrected Bellman operator + causal attribution for constrained RL contraction proof under unknown stochastic delay [R]
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
这篇题为“延迟校正的贝尔曼算子+因果归因，用于未知随机延迟下的约束强化学习收缩证明[R]”的论文，旨在解决标准约束强化学习（RL）中假设后果是即时的这一局限性。当违规行为是延迟且随机发生时，这种假设就会失效，导致错误的惩罚。然而，该方法目前需要访问环境的结构因果模型（SCM）来生成预训练标签，这限制了其在已知或可合理指定SCM的基准设置之外的适用性。…
Standard constrained RL assumes consequences are immediate and attributable to the current action. This breaks down whenever violations are delayed and stochastic, which is most real-world settings you end up penalizing whatever action happened to precede the observed violation, not the action that caused it.
Working on CCPL (Causal Consequence-Penalized Learning) to address this:
- A delay-corrected Bellman operator using an adaptive effective discount learned from the consequence-delay distribution. Contraction proof holds under unknown stochastic delay.
- An Interventional Consequence Net (ICN), pretrained on structural-causal-model labels, estimating marginal causal contribution per action for attribution rather than penalizing based on temporal proximity.
**Limitations, to be upfront about them:** - The ICN currently requires access to the environment's structural causal model to generate pretraining labels it's not learned end-to-end from observational or interventional data alone. That's a real constraint on applicability outside benchmark settings where the SCM is known or can be reasonably specified.
Open to contributions and collaborators, especially if you work in constrained/safe RL or causal inference feel free to open an issue or reach out directly.