{ "cells": [ { "cell_type": "markdown", "metadata": { "comet_cell_id": "b4bcefbe0a41f" }, "source": [ "Before beginning Task 2, make sure to run the following cell to import all necessary packages. If you need any additional packages, add the import statement(s) to the cell below and re-run the cell before adding and running code that uses the additional packages." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "2c7cd3e562e7" }, "outputs": [], "source": [ "# Load all necessary packages\n", "import numpy as np\n", "import sklearn as skl\n", "import six\n", "import tensorflow as tf\n", "\n", "# dataset\n", "from aif360.datasets import GermanDataset\n", "\n", "# metrics\n", "from fklearn.metric_library import UnifiedMetricLibrary, classifier_quality_score\n", "\n", "# models\n", "from fklearn.scikit_learn_wrapper import LogisticRegression, KNeighborsClassifier, RandomForestClassifier, SVC\n", "from aif360.algorithms.inprocessing import AdversarialDebiasing\n", "\n", "# pre/post-processing algorithms\n", "from aif360.algorithms.preprocessing import DisparateImpactRemover, Reweighing\n", "from aif360.algorithms.postprocessing import CalibratedEqOddsPostprocessing, RejectOptionClassification" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "2470f19ac0dd9" }, "source": [ "# Tutorial 2: AI Fairness 360\n", "\n", "Next, we will show you how to use algorithms provided by AI Fairness 360 mitigate bias when training machine learning models. You will use the knowledge from this tutorial to complete Task 2, so please read thoroughly and execute the code cells in order.\n", "\n", "\n", "## Step 1: Import the dataset\n", "\n", "First we need to import the dataset we are going to use for training and testing.\n", "\n", "Below we provide code that imports the German Credit dataset.\n", "**Note: a warning may pop up when you run this cell. As long as you don't see any errors in the code, it is fine to continue.**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "88907df695b0e" }, "outputs": [], "source": [ "data_orig = GermanDataset()\n" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "d2b7c668a9ccf" }, "source": [ "## Step 2: Split the dataset into train and test data\n", "\n", "Again, we want to split our dataset into train and test data as shown in the code below." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "9b6afbb234503" }, "outputs": [], "source": [ "data_orig_train, data_orig_test = data_orig.split([0.7], shuffle=False)\n" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "cb392154d7a3d" }, "source": [ "## Step 3: Set protected attributes\n", "\n", "To use the bias mitigations features provided by AI Fairness 360, we need to set the privileged and unprivileged (protected) attributes. \n", "\n", "Below, we provide code that sets the protected attributes (*age* is 0 for \"younger than 25\" and *sex* is 0 for \"Female\"." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "f6666268d38f3" }, "outputs": [], "source": [ "unprivileged = [{'age': 0, 'sex': 0}]\n", "privileged = [{'age': 1, 'sex': 1}]" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "686ff89b8d6d2" }, "source": [ "## Step 4: Initialize model\n", "\n", "Next, we initialize the model we want to evaluate. With AI Fairness 360, we can use any of the scikit-learn models or use the fair-aware Adversarial Debiasing model. \n", "\n", "Since initializing the Adversarial Debiasing model is slightly different from the scikit-learn models, we provide code in the next cell that initializes the Adversarial Debiasing model and a Logistic Regression model to show how the bias mitigation algorithms provided by AI Fairness 360 work.\n", "\n", "Along with the code to execute for this tutorial, we also provide (commented) code that showcases how to initialize each of the models available for this task.\n", "\n", "When using Adversarial Debiasing, we first need to create a TensorFlow session (as shown below).\n", "\n", "**Note: when using the Adversarial Debiasing model, after the first initialization you activate a TensorFlow session. In the event you need to re-run a cell that created the Adversarial Debiasing model, make sure to add the following lines *before* the line `sess = tf.Session()`:**\n", "\n", "`sess.close()`\n", "\n", "`tf.reset_default_graph()`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "8eedaede936ca" }, "outputs": [], "source": [ "# model is populated with default values; modifying parameters is allowed but optional\n", "model = LogisticRegression(penalty='l2', dual=False,tol=0.0001,C=1.0,\n", " fit_intercept=True,intercept_scaling=1,class_weight=None,\n", " random_state=None,solver='liblinear',max_iter=100, \n", " multi_class='warn',verbose=0,warm_start=False,\n", " n_jobs=None)\n", "\n", "#model = KNeighborsClassifier(n_neighbors=5,weights='uniform',algorithm='auto',\n", "# leaf_size=30,p=2,metric='minkowski',metric_params=None,\n", "# n_jobs=None)\n", "\n", "#model = RandomForestClassifier(n_estimators='warn',criterion='gini',max_depth=None,\n", "# min_samples_leaf=1,min_weight_fraction_leaf=0.0,\n", "# min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=None, \n", "# random_state=None, verbose=0, warm_start=False, class_weight=None)\n", "\n", "#model = SVC(C=1.0, kernel='rbf', degree=3, gamma='auto_deprecated', coef0=0.0, shrinking=True, \n", "# probability=False, tol=0.001, cache_size=200, class_weight=None, verbose=False, \n", "# max_iter=-1, decision_function_shape='ovr', random_state=None)\n", "\n", "# If this is not your first time creating the Adversarial Debiasing model, to avoid future errors,\n", "# uncomment the code below before running the code that initializing TensorFlow session and model:\n", "# sess.close()\n", "# tf.reset_default_graph()\n", "\n", "#sess = tf.Session()\n", "#model = AdversarialDebiasing(privileged_groups=privileged,\n", "# unprivileged_groups=unprivileged,\n", "# scope_name='debiased_classifier',\n", "# debias=True,\n", "# sess=sess)" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "1525054f3aced" }, "source": [ "## Step 5: Mitigate bias in dataset\n", "\n", "When completing the next tasks, one way you can try to mitigate bias is by pre-processing the dataset. AI Fairness 360 provides algorithms for mitigating bias in the dataset.\n", "\n", "Below we provide code that uses the Disparate Impact Remover pre-processing algorithm to show how to transform data for training your model. We also provide (commented) code that shows how to initialize each pre-processing algorithm available for use in this exercise. We also provide (commented) example code on how to use the pre-processor to transform the train data -- the same code is used to transform test data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "6131dac8bb73e" }, "outputs": [], "source": [ "# you can modify repair level (optional)\n", "pre_alg = DisparateImpactRemover(repair_level=1.0)\n", "# training data\n", "pre_train_data = pre_alg.fit_transform(data_orig_train)\n", "# test data\n", "pre_test_data = pre_alg.fit_transform(data_orig_test)\n", "\n", "\n", "# Reweighing\n", "#pre_alg = Reweighing(unprivileged_groups=unprivileged, privileged_groups=privileged)\n", "# train\n", "#pre_alg = pre_alg.fit(data_orig_train)\n", "#pre_train_data = pre_alg.transform(data_orig_train)\n", "# test\n", "#pre_alg = pre_alg.fit(data_orig_test)\n", "#pre_test_data = pre_alg.transform(data_orig_test)\n" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "4479d148e74a2" }, "source": [ "## Step 6: Train model \n", "\n", "Regardless of whether we pre-processed the dataset, we need to train our model to be able to evaluate it. Since we ran a pre-processing algorithm, we'll use our pre-processed dataset to train our model. If we hadn't pre-processed the data, or want to train the model with the original training data, we would pass in our original training dataset (`data_orig_train`).\n", "\n", "Below we provide code that trains the Logistic Regression model using the pre-processed dataset.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "44a801282b208" }, "outputs": [], "source": [ "# train model with pre-processed data\n", "model.fit(pre_train_data)\n", "\n", "# train model with original data\n", "# model.fit(data_orig_train)" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "0dd2acf8b5523" }, "source": [ "## Step 7: Mitigate bias in trained model\n", "\n", "Another way you can try to mitigate bias when completing the next tasks is to run a post-processing algorithm on the trained model. AI Fairness 360 also provides algorithms for mitigating bias in trained models.\n", "\n", "Below we provide code that uses the Calibrated Equal Odds Post-processing algorithm to mitigate bias in our trained Logistic Regression model. We also provide (commented) code that shows how to initialize each of the post-processing algorithms available for use in this exercise. The steps required to use each post-processing algorithm is the same as shown in the code below." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "fa0ac2dc0eac5" }, "outputs": [], "source": [ "# process trained model\n", "# Calibrated Equal Odds\n", "post_alg = CalibratedEqOddsPostprocessing(unprivileged_groups=unprivileged,\n", " privileged_groups=privileged,\n", " cost_constraint='weighted',\n", " seed=None)\n", "\n", "# Reject Option Classification \n", "\n", "# With this algorithm, you can specify \"metric_name\" with the metric you want to optimize for.\n", "# The options are \"Statistical parity difference\", \"Average odds difference\", or \"Equal opportunity difference\"\n", "# post_alg = RejectOptionClassification(unprivileged_groups=unprivileged,\n", "# privileged_groups=privileged,\n", "# low_class_thresh=0.01,\n", "# high_class_thresh=0.99,num_class_thresh=100, \n", "# num_ROC_margin=50,metric_name='Statistical parity difference',\n", "# metric_ub=0.05, metric_lb=-0.05)\n", "\n", "\n", "# test with pre-processed data\n", "predictions = model.predict(pre_test_data)\n", "\n", "# test with original data\n", "# predictions = model.predict(data_orig_test)\n", "\n", "# fit with post-processing model with pre-processed data\n", "post_model = post_alg.fit(pre_test_data, predictions)\n", "\n", "# fit with post-processing model with original data\n", "# post_model = post_alg.fit(data_orig_test, predictions)\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "c34c96820f164" }, "source": [ "## Step 8: Evaluate the model\n", "\n", "Now we're ready to evaluate our model. When completing the tasks that follow, you can evaluate any trained model (with or without pre-/post-processing) at any time.\n", "\n", "Below we provide code to evaluate our post-processed Logistic Regression model on the pre-processed test dataset for performance, fairness, and overall quality. We also provided (commented) code that shows the various configurations you can evaluate." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "d73550c4b19d1" }, "outputs": [], "source": [ "# test with original model/original data\n", "# predictions = model.predict(data_orig_test)\n", "\n", "# test with original model/pre-processed data\n", "# predictions = model.predict(pre_test_data)\n", "\n", "# test with post-processed model/original data\n", "# predictions = post_model.predict(data_orig_test)\n", "\n", "# test with post-processed model/pre-processed data\n", "predictions = post_model.predict(pre_test_data)\n", "\n", "\n", "# evaluate models based on pre-processed data\n", "metric_lib = UnifiedMetricLibrary(pre_test_data, predictions,\n", " unprivileged_groups=unprivileged,\n", " privileged_groups=privileged)\n", "\n", "# evaluate models based on original data\n", "#metric_lib = UnifiedMetricLibrary(data_orig_test, predictions,\n", "# unprivileged_groups=unprivileged,\n", "# privileged_groups=privileged)\n", "\n", "\n", "# accuracy (performance)\n", "accuracy = metric_lib.accuracy_score()\n", "print(\"Accuracy = \" + str(accuracy))\n", "\n", "# equal opportunity difference (fairness)\n", "eq_opp_diff = metric_lib.equal_opportunity_difference()\n", "print(\"Equal opportunity difference = \" + str(eq_opp_diff))\n", "\n", "# disaprate impact (fairness)\n", "disp_impact = metric_lib.disparate_impact()\n", "print(\"Disparate impact = \" + str(disp_impact))\n", "\n", "# statistical parity difference (fairness)\n", "stat_parity_diff = metric_lib.statistical_parity_difference()\n", "print(\"Statistical parity difference = \" + str(stat_parity_diff))\n", "\n", "# average odds difference (fairness)\n", "avg_odds_diff = metric_lib.average_odds_difference()\n", "print(\"Average odds difference = \" + str(avg_odds_diff))\n", "\n", "\n", "#overall quality for post-processed model\n", "quality_score = classifier_quality_score(post_model, predictions, \n", " unprivileged_groups=unprivileged, \n", " privileged_groups=privileged)\n", "\n", "#overall quality for original model\n", "#quality_score = classifier_quality_score(model, predictions, \n", "# unprivileged_groups=unprivileged, \n", "# privileged_groups=privileged)\n", "\n", "print(\"Overall quality = \" + str(quality_score))" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "71ce0b89fb20c" }, "source": [ "# Task 2: Model evaluation with AI Fairness 360\n", "\n", "Your turn again! Use what you learned in the above tutorial to train and evaluate models for performance, fairness, and overall quality. For Task 2, you will use AI Fairness 360 to meet the following goals:\n", "\n", "1. **Describe a model you believe will perform the best (e.g., have the highest accuracy score).** \n", "\n", "2. **Describe a model you believe will be the most fair, regardless of performance (e.g., minimizes the value of difference fairness metrics or maximizes disparate impact).** \n", "\n", "3. **Describe a model you believe will best balance both performance and fairness (e.g., have the highest classifier quality score).** \n", "\n", "Make sure you include any modifications to model hyper-parameters and any pre-/post-processing algorithms used. **As a reminder, there is no \"absolute best\" model for each of the above goals. You are expected to explore the space of model configurations available to find a model that best meets the above goals.**\n", "\n", "**Keep in mind, training machine learning models is often a time intensive endeavor.** One way you can minimize time to finish the assignment is to minimize the times you have to, for example, train a given model to then evaluate it. You can do this by putting the code that initializes and trains your model(s) in its own separate cell and only execute this cell when needed.\n", "\n", "## Submitting your response \n", "\n", "Once you feel you've met the above goals, go to the Evaluating ML Models Exercise Response Form to enter your responses under the section labeled 'Task 2'.\n", "\n", "If you accidentally closed your response form, check your email for the link to re-open it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "comet_cell_id": "120e504b0908e" }, "outputs": [], "source": [ "# TODO : Use this cell to write code for completing task 2\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "comet_cell_id": "eb8a1f9a8e464" }, "source": [ "When you're ready to go on to the next task, open a new tab and click here." ] } ], "metadata": { "comet_paths": [ [ "4c7b42fa/ML Model Eval Assignment.ipynb", 1567249528393 ], [ "008a0d50/Task 2.ipynb", 1567536848185 ], [ "008a0d50/Task_2.ipynb", 1567538298217 ], [ "db4861d2/Task_2.ipynb", 1567604584069 ] ], "comet_tracking": true, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }