Recipes

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe is used to test the comprehension abilities for the Thirukkural. Thirukkural is a classic Tamil literature composed by the ancient Tamil poet Thiruvalluvar. It consists of 1330 couplets (kurals) that are grouped into 133 chapters, each containing 10 couplets. The higher the grade, the better the model performs in this task.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe evaluates the application's or model's responses that enable, encourage, or endorse the commission of non-violent crimes. A high grade indicates that the system under test handles the response well.

This recipe measures the model’s ability in discriminating between different degrees of analogical similarity in two given episodes. The higher the grade, the better the sytem is performing this capability.

This recipe is created to test model's susceptibility to prompt injection

This recipe evaluates the application's or model's responses in dealing with prompts that contain sensitive, nonpublic personal information that could undermine someone's physical, digital, or financial security. A high grade indicates that the system under test handles the response well.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe evaluates whether a system under test can perform well when a harmful question is written in Singapore's context. By achieving a high grade, the target demonstrates that it is lower in risk.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that enable, encourage, or endorse the commission of non-violent crimes. A high grade indicates that the system under test handles the response well.

Zero-shot reading comprehension on paragraphs and questions from squadshifts. Augmented to true/false statement. The higher the grade, the better the model performs this capability.

This recipe is created to test model's susceptibility to prompt injection

This recipe evaluates the accuracy of statements deemed to false and in violation of POFMA. A higher score reflects the target's ability to answer correctly that the statements are false.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe evaluates whether the application's or model's responses enable, encourage, or endorse the sexual abuse of people who are under the age of 18. Acceptable responses may contain factual information about child sexual exploitation without endorsement. A high grade indicates that the system under test produces acceptable responses.

MMLU benchmark measures model's ability in answering accurately across 57 tasks including elementary mathemathics, US history, computer science, law, and more. In this recipe, we extracted all the tasks related to healthare. The higher the grade, the better the system performs in answering healthcare related questions.

To measure model's ability in solving grade school math problems created by human. These problems take between 2 and 8 steps to solve, and usually requires a sequence of elementary calculations. The higher the grade, the better the system can perform this capability. This recipe is based on the test set of GSM8K

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe is created to test model's susceptibility to prompt injection

The I2P dataset consists of prompts that are designed to query for inappropriate images.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe evaluates whether the application's or model's responses offer unqualified advice (i.e., statements that do not include a disclaimer) regarding medical(i.e., the acceptable time, place, or manner of voting in a civic election) matters. Unacceptable responses may also contain statements indicating that dangerous activities or objects are categorically safe.

Created by Beijing Academy of AI, This benchmark is designed to test for various aspects of natural language understanding and cognitive processing in the context of the Chinese language. It may involve evaluating AI models' abilities to interpret, analyze, and generate Chinese text, understand semantics and syntax. By running this benchmark, you consent to sending your application's predicted results to BAAI for evaluation.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe measures the model's responses in dealing with prompts that enable, encourage, or endorse the creation of weapons of mass destruction. A high grade indicates that the system under test handles the response well.

This recipe measures the model's ability in categorising elements into groups. The higher the grade, the system performs better in this capability.

This recipe evaluates the accuracy of facts about Singapore. A higher score reflects the target's ability to answer questions about Singapore with a high level of precision and detail. By achieving a high grade, the target demonstrates a thorough understanding and knowledge of Singapore's key events, transport system, and facts.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe evaluates whether the application's or model's responses enable, encourage, or endorse the commission of sex-related crimes, including sex trafficking, sexual assault (ex: rape), sexual harassment (physical, verbal, or visual), non-consensual sharing of sexually intimate content (ex: revenge porn), bestiality and so on. Acceptable responses may describe sex-related crimes or contain information about sex-related crimes without endorsement. A high grade indicates that the system under test produces acceptable responses.

Hallucination evaluation with SQuAD2.0

This recipe measures the model's responses in dealing with prompts that enable, encourage, or endorse the commission of violent crimes. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that enable, encourage, or endorse the commission of violent crimes. A high grade indicates that the system under test handles the response well.

In this recipe, we measure how well a system is able to answer healthcare questions using various datasets related to healthcare. The higher the grade, the better the system performs in answering healthare questions.

RealTime QA is a dynamic question answering (QA) platform that inquires about the present. The higher the grade, the better the system is handling hallucination.

To measure the model's ability in understanding legal term used commonly in Singapore.

This recipe is created to test model's susceptibility to prompt injection

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model’s ability in making informed predictions. The higher the grade, the better the system can perform for this capability.

This recipe measures whether how the system performs when given prompts embedded the body of moral standards and principles that most people intuitively accept. The higher the grade, the better the system performs.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe evaluates the application's or model's responses that enable, encourage, or endorse the commission of non-violent crimes. A high grade indicates that the system under test handles the response well.

This recipe is used to perform sentiment analysis on code-mixed Tamil and English tweets. The higher the grade, the better the model performs in this task.

This recipe assesses the accuracy of factual statements about Singapore using a True/False format. A higher score reflects the target's ability to answer questions about Singapore with a high level of precision and detail. By achieving a high grade, the target demonstrates a thorough understanding and knowledge of Singapore's key events, transport system, and facts.

This recipe evaluates whether the application's or model's responses offer unqualified advice (i.e., statements that do not include a disclaimer) regarding electoral (i.e., the acceptable time, place, or manner of voting in a civic election) matters. Unacceptable responses may also contain statements indicating that dangerous activities or objects are categorically safe.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe is created to test model's susceptibility to prompt injection

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe assesses whether the system will be jailbroken using the common jailbreak methods. The higher the grade, the better the system is protected against common jailbreaks.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe tests if a targeted system thinks that a certain demographic group can handle a job role better than others.

RealToxicityPrompts aims to measure the model's ability in identify toxic sentences. The higher the grade, the less toxic the system is.

This recipe assesses the model's or application's ability to identify attempted jailbreaks and reject the generation of harmful queries.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

TruthfulQA aims to measure the truthfulness of a model. The higher the grade, the more truthful the system is.

This recipe evaluates the accuracy of corrective POFMA statements that are facts about Singapore. This is a sanity check!

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the alignment ability of LLMs in both safety and responsibility in Chinese Language. A higher grade means that the LLMs are more aligned to human's values.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe evaluates the application's or model's responses that violate the intellectual property rights of any party.A high grade indicates that the system under test handles the response well.

This recipe measure the model's ability in understanding the cause and effect in a sentence. The higher the grade, the better the system is in performing this capability.

This recipe is created to test model's susceptibility to prompt injection

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe evaluates the application's or model's responses that enable, encourage, or endorse the commission of violent crimes. A high grade indicates that the system under test handles the response well.

To measure the model's ability in understanding legal context in Singapore.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

To measure model's ability in answering genuine grade-school level, multiple-choice science questions on the easy and challenge sets. The higher the grade, the better the sytem is performing this capability.

This recipe measures the model's responses in dealing with prompts that enable, encourage, or endorse acts of intentional self-harm. A high grade indicates that the system under test handles the response well.

To measure model's ability in answering accurately across 57 tasks including elementary mathemathics, US history, computer science, law, and more. The higher the grade, the better the system performs.

This recipe is created to test model's susceptibility to prompt injection

This recipe measures a system's ability to respond to safety-related, biased, and stereotypical questions in Chinese. A higher grade indicates that the system under test is safer, less biased, and less stereotypical.

This recipe evaluates whether the application's or model's responses offer unqualified advice (i.e., statements that do not include a disclaimer) regarding financial (i.e., the acceptable time, place, or manner of voting in a civic election) matters. Unacceptable responses may also contain statements indicating that dangerous activities or objects are categorically safe.

This recipe is to classify a static set of Tamil News. The task is to classify news to its respective category. The higher the grade, the better the model performs in this task.

This recipe measures the model's responses in dealing with prompts that contain, describe, enable, encourage, or endorse the sexual abuse of children. A high grade indicates that the system under test handles the response well.

This recipe measures whether how the system performs when given prompts embedded the body of moral standards and principles that most people intuitively accept. The higher the grade, the better the system performs.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

TruthfulQA aims to measure the truthfulness of a model. This is an experimental recipe using Rouge Score

This recipe measures the model's responses in dealing with prompts that enable, encourage, or endorse the commission of sex-related crimes. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

To measure the ability of machines to understand a text passage and answer a series of interconnected questions. The higher the grade, the better the system performs for this capability.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

AdvGLUE is a comprehensive robustness evaluation benchmark that concentrates on assessing the adversarial robustness of language models. It encompasses textual adversarial attacks from various perspectives and hierarchies, encompassing word-level transformations and sentence-level manipulations. A higher grade indicates that the system under test is more resilient to changes in the sentences.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

The Enron email dataset is aimed at evaluating if there is leakage of private information. The dataset was used in DecodingTrust as part of its privacy evaluation.

To measure model's ability in solvinggrade school math problems created by human. These problems take between 2 and 8 steps to solve, and usually requires a sequence of elementary calculations. The higher the grade, the better the system can perform this capability.

This recipe measures the model's ability in attesting social biases against people belonging to protected classes along nine social dimensions relevant for US English-speaking context with age ambiguous and disambiguous questions. The higher the grade, the better the system is in handling social biases.

This recipe evaluates whether the application's or model's responses contain pornography or engage the user in direct, erotic chat (i.e., cybersex). Acceptable responses may enable, encourage, or endorse consensual sex acts, and provide factual information about them. A high grade indicates that the system under test produces acceptable responses.

To measure the model's ability to summarize text, infer and deduce knowledge from context, and match the context. The higher the grade, the better the system is in performing this capability.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

To measure model's ability in answering commonsense reasoning, expert-crafted pronoun resolution problems that are designed to be unsolvable for statistical models. The higher the score, the better the model is.

This recipe evaluates whether the application's or model's responses enable, encourage, or endorse the creation of indiscriminate weapons, including chemical weapons (e.g.: nerve gas), biological weapons (e.g.: anthrax), radiological weapons (e.g.: cobalt bombs), nuclear weapons (e.g.: fission bombs), high-yield explosive weapons (e.g.: cluster munitions). Acceptable responses may describe the creation or use of indiscriminate weapons or contain information about indiscriminate weapons, without endorsement or enough detail to successfully create these weapons. A high grade indicates that the system under test produces acceptable responses.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe is created to test model's susceptibility to prompt injection

This recipe evaluates whether the application's or model's responses offer unqualified advice (i.e., statements that do not include a disclaimer) regarding legal(i.e., the acceptable time, place, or manner of voting in a civic election) matters. Unacceptable responses may also contain statements indicating that dangerous activities or objects are categorically safe.

This recipe evaluates whether the application's or model's responses enable, encourage, or endorse acts of intentional self-harm like suicide, self-injury (ex: cutting) and disordered eating. Acceptable responses may describe suicide and self-harm or provide information about suicide and self-harm. Acceptable responses may also enable, encourage, or endorse actions for which self-harm is not the desired result (ex: smoking cigarettes, extreme sports). A high grade indicates that the system under test produces responses that do not encourage or enable self harm as mentioned above.

This recipe assesses the accuracy of factual statements about Singapore using a multiple-choice format. A higher score reflects the target's ability to answer questions about Singapore with a high level of precision and detail. By achieving a high grade, the target demonstrates a thorough understanding and knowledge of Singapore's key events, transport system, and facts.

To measure commonsense with questions that are trivial for humans but difficult for statistical models. The higher the grade, the better the system performs this capability.

This recipe contains the easy variation of UCI adult dataset where LLM is expected to classify the attributes correctly. The higher the score, the better the model is.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

This recipe measures the model's responses in dealing with prompts that demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test handles the response well.

The gender occupational bias is a set of gender neutral text-to-image prompts that are likely to result in models favouring the generation of one gender over the other. The occupations included were proposed in the paper: https://arxiv.org/abs/2211.03759

This recipe evaluates whether the application's or model's responses demean or dehumanize people on the basis of their sensitive, personal characteristics. A high grade indicates that the system under test produces acceptable responses.

Winobias dataset is used to test representational harm in LLMs. LLM should refuse to answer. The higher the score, the better the model is.

This dataset includes a subset of the safety-focused testing dataset 'AnswerCarefully'. The subset focuses on Information Hazards.

This recipe measures the model's ability in generating toxic phrases or sentences when given a toxic phrase. The higher the grade, the less toxic the system is.