As part of our ongoing commitment to transparency and responsible AI development, we are sharing the performance metrics of our de-generative AI models.
These metrics, benchmarked against a diverse dataset comprising both academic and proprietary (privately generated) samples, allow usāand othersāto assess how well our models perform across various tasks and use cases, and provide a clear baseline for future enhancements. This transparency is especially important in the context of degenerative AI, where outputs are often complex, nuanced, and subject to interpretation.
To offer a consistent and interpretable view of model performance, we will report the following standard evaluation metrics. Each of these metrics offers unique insights into model behavior and collectively provide a robust framework for evaluating and comparing degenerative model performance over time.
This metric measures the overall correctness of the model's outputs by calculating the proportion of total predictions that are correct. It is a broad indicator of performance but can be misleading in the presence of class imbalance.
Precision evaluates the proportion of correct positive predictions relative to the total number of positive predictions made. It answers the question: Of all the outputs labeled as positive by the model, how many were actually correct? This is particularly important in scenarios where false positives carry a high cost.
Also known as sensitivity, recall measures the proportion of actual positive cases that the model correctly identified. It reflects the modelās ability to capture all relevant cases and is critical in contexts where missing a true positive is particularly undesirable.
The F1 score is the harmonic mean of precision and recall, providing a single metric that balances both concerns. It is especially useful when there is an uneven class distribution or when a balance between precision and recall is essential.
By making these results available, we invite constructive feedback from the research community, foster shared learning, and ultimately strive for more responsible and effective deployment of degenerative AI technologies.
talk to a human expert
No sales pitch. Just a conversation.
We stand for truth
Gartner does not endorse any vendor, product or service depicted in its researchpublications, and does not advise technology users to select only those vendors with thehighest ratings or other designation. Gartner research publications consist of the opinions ofGartnerās research organization and should not be construed as statements of fact. Gartnerdisclaims all warranties, expressed or implied, with respect to this research, including anywarranties of merchantability or fitness for a particular purpose. GARTNER is a registeredtrademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationallyand is used herein with permission. All rights reserved.