We proposed an automated method to extract concepts from neural networks that explains a specific class, tackling the "What" challenge in explainability. We go beyond understanding where the model looks, delving into what it sees at the precise location of interest. Taking it a step further, we enhance this method by (1) effectively estimating the importance of discovered concepts, and (2) generating localized heatmaps, revealing concept locations within each image.

In this article we demonstrate that all concept extraction methods can be viewed as dictionary learning methods. We leverage this common framework to develop a comprehensive framework for comparing and improving concept extraction methods. Furthermore, we extensively investigate the estimation of concept importance and show that it is possible to determine optimal importance estimation formulas in certain cases. We also highlight the significance of local concept importance in addressing a crucial question in Explainable Artificial Intelligence (XAI): identifying data points classified based on similar reasons.

Since the remarkable work of Chris Olah and the Clarity team at OpenAI, feature visualization techniques have stagnated since 2017, and the methods proposed at the time are very difficult to make work on modern models (e.g. Vision Transformers). In this article, we propose a simple technique to revive feature visualization on modern models. Our method is based on a magnitude constraint, which ensures that the generated images have a magnitude similar to real images while avoiding the need for managing an additional hyperparameter.

In this article, we begin by demonstrating the existence of a trade-off between alignment and accuracy: the more accurate models are on ImageNet (84 state-of-the-art models), the less they are aligned with humans in terms of saliency maps. To address this trade-off, we propose a training routine : the harmonization loss. This routine aligns deep neural networks (DNNs) with human visual strategies (almost down to the human-human leve), even improving categorization accuracy.

In this work, we conducted psychophysics experiments at scale to evaluate the ability of human participants to leverage representative attribution methods for understanding the behavior of different image classifiers representing three real-world scenarios: (1) identifying bias, (2) characterizing novel strategy and (3) understanding failure cases. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varied widely across these scenarios. This suggests a critical need for the field to move past quantitative improvements of current attribution methods.

This paper could be seen as a follow-up to Look at the variance and further improves the efficiency of black-box methods using Hilbert-Schmidt Independence Criterion (HSIC). HSIC measures the dependence between regions of an input image and the output of a model based on kernel embeddings of distributions. Our experiments show that HSIC is up to 8 times faster than the previous best black-box attribution methods while being as faithful. Importantly, we show that these advances can be transposed to efficiently and faithfully explain object detection models such as YOLOv4.

In this work, we propose the first explainability method based on formal method and therefore able to meet certification requirements. Specifically, we introduce EVA (Explaining using Verified perturbation Analysis) -- the first explainability method guarantee to have an exhaustive exploration of a perturbation space. We leverage the beneficial properties of verified perturbation analysis -- time efficiency, tractability and guaranteed complete coverage of a manifold -- to efficiently characterize the input variables that are most likely to drive the model decision. We evaluate the approach systematically and demonstrate state-of-the-art results on multiple benchmarks.

We argue that, when learning a 1-Lipschitz neural network with the dual loss of an optimal transportation problem, the gradient of the model is both the direction of the transportation plan and the direction to the closest adversarial attack. Traveling along the gradient to the decision boundary is no more an adversarial attack but becomes a counterfactual explanation, explicitly transporting from one class to the other. Through extensive experiments on XAI metrics, we find that the simple saliency map method, applied on such networks, becomes a reliable explanation! The proposed networks were already known to be certifiably robust, and we prove that they are also tailored for explainability.

For more than a year, I worked alongside my thesis to implement more than fifty explainability papers. Xplique is the result of this work, it is a library that I develop and maintain. The library is composed of several modules: (1) the Attributions Methods module, (2) The Feature Visualization module, (3) The Concepts module and (4) the Metrics module.

We describe a novel and efficient black-box attribution method which is grounded in Sensitivity Analysis and uses Sobol indices. Beyond modeling the individual contributions of image regions, Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to a neural network's prediction through the lens of variance.

We propose two new measures to evaluate explanations borrowed from the field of algorithmic stability: mean generalizability MeGe and relative consistency ReCo. We conduct extensive experiments on different network architectures, common explainability methods, and several image datasets to demonstrate the benefits of the proposed metrics and show that popular fidelity measures are not sufficient to guarantee trustworthy explanations. Finally, we found that 1-Lipschitz networks produce explanations with higher MeGe and ReCo than common neural networks.