Stephanie Fu

I am a PhD student at BAIR advised by Trevor Darrell.

Previously, I graduated from MIT with an MEng in computer science (advised by Phillip Isola) and bachelors degrees in computer science and music. During my undergrad, I was fortunate to work on exciting research under Bill Freeman, Yoel Fink, and Phillip Isola.

I am broadly interested in the science behind deep learning and representation learning, especially in the realm of computer vision. Most recently, I have been interested in developing and understanding models with visual intelligence.

In 2019, I co-founded TEDxMIT and helped launch the inaugural conference at MIT CSAIL. Since then, TEDxMIT has brought communities across the Greater Boston area together with three more conferences and adapted to the COVID-19 pandemic by holding a virtual event. Check out the highlight reels from past events and details about the upcoming conference here!

news

Jun 2023	We just released DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data!
Jun 2023	I will be attending UC Berkeley for my PhD in Computer Science, supported by the NSF GRFP and Chancellor’s Fellowship!
May 2023	I was offered Stanford’s Knight-Hennessy Scholarship!

selected publications

Evaluating Multiview Object Consistency in Humans and Image Models

Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O’Connell, Yoni Friedman, Nancy Kanwisher, Joshua B. Tenenbaum, and Alexei A. Efros

2024
A Vision Check-up for Language Models

Pratyusha Sharma, Tamar Rott Shaham, Manel Baradad, Stephanie Fu, Adrian Rodriguez-Munoz, Shivam Duggal, Phillip Isola, and Antonio Torralba

2024
OpenStreetView-5M: The Many Roads to Global Visual Geolocation

Guillaume Astruc, Nicolas Dufour, Ioannis Siglidis, Constantin Aronssohn, Nacim Bouia, Stephanie Fu, Romain Loiseau, Van Nguyen Nguyen, Charles Raude, Elliot Vincent, Lintao XU, Hongyu Zhou, and Loic Landrieu

2024
FeatUp: A Model-Agnostic Framework for Features at Any Resolution

Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feldman, Zhoutong Zhang, and William T. Freeman

2024
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

Stephanie Fu*, Netanel Tamir*, Shobhita Sundaram*, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola

arXiv:2306.09344 2023

Abs Paper Code Website

Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.
Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning

Mark Hamilton, Scott Lundberg, Lei Zhang, Stephanie Fu, and William T Freeman

International Conference on Learning Representations (ICLR) 2021

Abs Paper

Visual search, recommendation, and contrastive similarity learning power technologies that impact billions of users worldwide. Modern model architectures can be complex and difficult to interpret, and there are several competing techniques one can use to explain a search engine’s behavior. We show that the theory of fair credit assignment provides a unique axiomatic solution that generalizes several existing recommendation- and metric-explainability techniques in the literature. Using this formalism, we show when existing approaches violate "fairness" and derive methods that sidestep these shortcomings and naturally handle counterfactual information. More specifically, we show existing approaches implicitly approximate second-order Shapley-Taylor indices and extend CAM, GradCAM, LIME, SHAP, SBSM, and other methods to search engines. These extensions can extract pairwise correspondences between images from trained opaque-box models. We also introduce a fast kernel-based method for estimating Shapley-Taylor indices that require orders of magnitude fewer function evaluations to converge. Finally, we show that these game-theoretic measures yield more consistent explanations for image similarity architectures.
MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval

Mark Hamilton, Stephanie Fu, Mindren Lu, Johnny Bui, Darius Bopp, Zhenbang Chen, Felix Tran, Margaret Wang, Marina Rogers, Lei Zhang, Chris Hoder, and William T. Freeman

In Proceedings of the NeurIPS 2020 Competition and Demonstration Track 06–12 dec 2021

Abs Paper Blog Website

We introduce MosAIc, an interactive web app that allows users to find pairs of semantically related artworks that span different cultures, media, and millennia. To create this application, we introduce Conditional Image Retrieval (CIR) which combines visual similarity search with user supplied filters or “conditions”. This technique allows one to find pairs of similar images that span distinct subsets of the image corpus. We provide a generic way to adapt existing image retrieval data-structures to this new domain and provide theoretical bounds on our approach’s efficiency. To quantify the performance of CIR systems, we introduce new datasets for evaluating CIR methods and show that CIR performs non-parametric style transfer. Finally, we demonstrate that our CIR data-structures can identify “blind spots” in Generative Adversarial Networks (GAN) where they fail to properly model the true data distribution.
Digital electronics in fibres enable fabric-based machine-learning inference

Gabriel Loke, Tural Khudiyev, Brian Wang, Stephanie Fu, Syamantak Payra, Yorai Shaoul, Johnny Fung, Ioannis Chatziveroglou, Pin-Wen Chou, Itamar Chinn, Wei Yan, Anna Gitelson-Kahn, John Joannopoulos, and Yoel Fink

Nature Communications Jun 2021

Abs Paper Blog

Digital devices are the essential building blocks of any modern electronic system. Fibres containing digital devices could enable fabrics with digital system capabilities for applications in physiological monitoring, human-computer interfaces, and on-body machine-learning. Here, a scalable preform-to-fibre approach is used to produce tens of metres of flexible fibre containing hundreds of interspersed, digital temperature sensors and memory devices with a memory density of ~7.6\thinspace\texttimes\thinspace105 bits per metre. The entire ensemble of devices are individually addressable and independently operated through a single connection at the fibre edge, overcoming the perennial single-fibre single-device limitation and increasing system reliability. The digital fibre, when incorporated within a shirt, collects and stores body temperature data over multiple days, and enables real-time inference of wearer activity with an accuracy of 96% through a trained neural network with 1650 neuronal connections stored within the fibre. The ability to realise digital devices within a fibre strand which can not only measure and store physiological parameters, but also harbour the neural networks required to infer sensory data, presents intriguing opportunities for worn fabrics that sense, memorise, learn, and infer situational context.