Propose a metric
Every metric starts as a paper. If you’ve found a peer-reviewed paper that defines a useful evaluation methodology, you can propose it.1
Open an issue
Open an issue in the papers repository referencing the paper. Include the title, authors, venue, and a brief summary of the methodology.
2
Community discussion
The community debates the paper’s assumptions, limitations, and how the implementation should map to the methodology.
3
RFC
After consensus, an RFC is opened. Every design decision is documented and traceable to the original discussion.
Build an SDK
Once a metric is approved via RFC, it needs implementations. Each SDK is maintained independently by the community, following the language-agnostic spec derived from the RFC.
Want to maintain an SDK for your favorite language? Read the contribution guide.