Highlights

• •

  Certainty of evidence assessment can be a time-consuming process

• •

  The GRADE Working Group is developing GRADErater, an official automated tool

• •

  A prototype has been built (https://gradeai.med.up.pt/) and is undergoing testing

Abstract

Objective

The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group is developing GRADErater (GRADE Rating Automation Through Enhanced Reasoning), an official, automated tool for evaluation of the certainty of the evidence (CoE). In this article we describe the principles and methods underlying its development and state how the GRADE Working Group will use automation to rate the CoE of intervention effects.

Principles and methods

We followed the GRADE methods for registering a project group on GRADE and AI. The project group established that the automated tool should be developed according to the following principles: (i) compliance with current GRADE guidance, (ii) transparency, (iii) human oversight, (iv) implementation of decision rules, (v) ease of use, (vi) understandability, and (vii) continuous improvement. We developed a set of decision rules to appraise each domain of the CoE in pairwise and network meta-analysis. These rules were developed based on official GRADE sources and validated by a GRADE experts. Based on these rules, we created a first version (https://gradeai.med.up.pt/) for GRADErater. We are now evaluating this version in terms of (i) its underlying rules, and (ii) its user interface. These assessments will allow for a refinement of the rules and of the first version. The modified version will be appraised and presented to the GRADE Working Group with input from internal and external interest-holders before we seek formal approval. Once launched, the automated tool will be continuously evaluated and refined by incorporating feedback from end-users. We will add new features (including generative AI-based functionalities), integrate this tool with GRADEpro, and develop versions in languages other than English.

Conclusion

This project will follow a transparent methodology to create GRADErater, an official tool endorsed by the GRADE Working Group that will support humans in applying the most current GRADE methods to rate the CoE.

研究亮点

• 证据确定性评价工作耗时较长
• GRADE 工作组正在研发官方自动化工具 GRADErater
• 工具原型已搭建完成（网址：https://gradeai.med.up.pt/），目前处于测试阶段

摘要

研究目的

推荐分级的评估、制定与评价（GRADE）工作组正在开发自动化官方工具 GRADErater（全称：基于增强推理实现 GRADE 评级自动化），用于证据确定性评价。本文阐述该工具研发的底层原理与方法，并说明 GRADE 工作组如何借助自动化技术完成干预效应的证据确定性评级。

原理与方法

本研究遵循 GRADE 工作组关于 GRADE 与人工智能专项项目的立项规范。项目组明确自动化工具的研发需遵循七大原则：（1）完全契合现行 GRADE 操作指南；（2）过程透明可追溯；（3）人工全程监督审核；（4）内置标准化判定规则；（5）操作简易；（6）结果易于理解；（7）持续迭代优化。研究团队制定了一套判定规则，用于评估两两 Meta 分析与网状 Meta 分析中证据确定性的各个评价维度。规则均依据 GRADE 官方文件编制，并经 GRADE 领域专家验证。依托这套规则，团队搭建了 GRADErater 首版原型工具（网址：https://gradeai.med.up.pt/）。现阶段我们从两方面对原型开展评估：一是内置判定规则，二是用户操作界面。评估结果将用于优化判定规则与初代工具。修改后的版本将结合内外部相关方意见完成评审，提交 GRADE 工作组审议，待正式获批后上线。工具正式发布后，我们将持续收集终端用户反馈，对其开展常态化评估与迭代升级；后续还会新增生成式 AI 相关功能、实现与 GRADEpro 工具联动，并开发多语言版本（英语以外语种）。

结论

本项目将依托一套透明规范的研发流程打造 GRADErater，该工具为 GRADE 工作组官方认证工具，可辅助研究者运用最新 GRADE 方法完成证据确定性评级。

https://www.sciencedirect.com/science/article/pii/S0895435626002878

网站截图