LENS Project

This tool is designed to offer a concise summary of the key concepts (termed as a 'dictionary of features') used by a large vision model across each of the 1,000 ImageNet classes. Each concept is accompanied by an importance score. This demonstration leverages insights from three of my prior projects, illustrating the potential synergy between attribution methods, concept analysis, and feature visualization.

This doctoral thesis advances explainability in computer vision, developing tools to better understand the features used by deep neural networks. It explores attribution methods (saliency maps) and introduces new approaches like Sobol indices and EVA, which provide formal guarantees. The thesis finds current methods insufficient in complex scenarios and proposes aligning models with human reasoning and optimizing models using Human data (Harmonization project) or within constrained spaces (e.g., 1-Lipschitz functions). It further shifts focus from "where" models attend to "what" they perceive using concept-based explainability. The work culminates in a unified framework and interactive exploration for visualizing important concepts in models like ResNet.

This work introduces a metric based on 𝒱-information to quantify feature complexity in deep learning models. Analyzing 10,000 features from an ImageNet-trained model, our study finds a spectrum from simple to complex features, with simpler ones emerging early in training. Interestingly, simpler features are often more important for decision-making and tend to bypass the visual hierarchy via residual connections. Finally, we found that important features follow a Sedimentation process, becoming accessible early and building the foundation for the model’s learning.

We proposed an automated method to extract concepts from neural networks that explains a specific class, tackling the "What" challenge in explainability. We go beyond understanding where the model looks, delving into what it sees at the precise location of interest. Taking it a step further, we enhance this method by (1) effectively estimating the importance of discovered concepts, and (2) generating localized heatmaps, revealing concept locations within each image.

In this article we demonstrate that all concept extraction methods can be viewed as dictionary learning methods. We leverage this common framework to develop a comprehensive framework for comparing and improving concept extraction methods. Furthermore, we extensively investigate the estimation of concept importance and show that it is possible to determine optimal importance estimation formulas in certain cases. We also highlight the significance of local concept importance in addressing a crucial question in Explainable Artificial Intelligence (XAI): identifying data points classified based on similar reasons.

Since the remarkable work of Chris Olah and the Clarity team at OpenAI, feature visualization techniques have stagnated since 2017, and the methods proposed at the time are very difficult to make work on modern models (e.g. Vision Transformers). In this article, we propose a simple technique to revive feature visualization on modern models. Our method is based on a magnitude constraint, which ensures that the generated images have a magnitude similar to real images while avoiding the need for managing an additional hyperparameter.

In this article, we introduce a novel interpretability tool called 'feature accentuation,' which addresses the need to understand both 'where' and 'what' neural network vision models recognize in arbitrary input images. By utilizing image-seeded feature visualization, we offer a comprehensive approach that provides naturalistic visualizations, shedding light on the spatial and semantic facets governing feature responses, ultimately enhancing our grasp of these complex models.

In this article, we begin by demonstrating the existence of a trade-off between alignment and accuracy: the more accurate models are on ImageNet (84 state-of-the-art models), the less they are aligned with humans in terms of saliency maps. To address this trade-off, we propose a training routine : the harmonization loss. This routine aligns deep neural networks (DNNs) with human visual strategies (almost down to the human-human leve), even improving categorization accuracy.

In this work, we conducted psychophysics experiments at scale to evaluate the ability of human participants to leverage representative attribution methods for understanding the behavior of different image classifiers representing three real-world scenarios: (1) identifying bias, (2) characterizing novel strategy and (3) understanding failure cases. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varied widely across these scenarios. This suggests a critical need for the field to move past quantitative improvements of current attribution methods.

We argue that, when learning a 1-Lipschitz neural network with the dual loss of an optimal transportation problem, the gradient of the model is both the direction of the transportation plan and the direction to the closest adversarial attack. Traveling along the gradient to the decision boundary is no more an adversarial attack but becomes a counterfactual explanation, explicitly transporting from one class to the other. Through extensive experiments on XAI metrics, we find that the simple saliency map method, applied on such networks, becomes a reliable explanation! The proposed networks were already known to be certifiably robust, and we prove that they are also tailored for explainability.

This paper could be seen as a follow-up to Look at the variance and further improves the efficiency of black-box methods using Hilbert-Schmidt Independence Criterion (HSIC). HSIC measures the dependence between regions of an input image and the output of a model based on kernel embeddings of distributions. Our experiments show that HSIC is up to 8 times faster than the previous best black-box attribution methods while being as faithful. Importantly, we show that these advances can be transposed to efficiently and faithfully explain object detection models such as YOLOv4.

In this work, we propose the first explainability method based on formal method and therefore able to meet certification requirements. Specifically, we introduce EVA (Explaining using Verified perturbation Analysis) -- the first explainability method guarantee to have an exhaustive exploration of a perturbation space. We leverage the beneficial properties of verified perturbation analysis -- time efficiency, tractability and guaranteed complete coverage of a manifold -- to efficiently characterize the input variables that are most likely to drive the model decision. We evaluate the approach systematically and demonstrate state-of-the-art results on multiple benchmarks.

For more than a year, I worked alongside my thesis to implement more than fifty explainability papers. Xplique is the result of this work, it is a library that I develop and maintain. The library is composed of several modules: (1) the Attributions Methods module, (2) The Feature Visualization module, (3) The Concepts module and (4) the Metrics module.

We describe a novel and efficient black-box attribution method which is grounded in Sensitivity Analysis and uses Sobol indices. Beyond modeling the individual contributions of image regions, Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to a neural network's prediction through the lens of variance.

We propose two new measures to evaluate explanations borrowed from the field of algorithmic stability: mean generalizability MeGe and relative consistency ReCo. We conduct extensive experiments on different network architectures, common explainability methods, and several image datasets to demonstrate the benefits of the proposed metrics and show that popular fidelity measures are not sufficient to guarantee trustworthy explanations. Finally, we found that 1-Lipschitz networks produce explanations with higher MeGe and ReCo than common neural networks.