Welcome!

Hae Jin (Hayley) Song

Ph.D. Candidate in Computer Science

USC Viterbi School of Engineering,
Information Sciences Institute (ISI)

| CV | Github | Pubs | Projects |

I'm a Computer Science PhD student at USC, working with Prof. Laurent Itti at iLab and ISI's VIMAL .

Before starting my PhD, I studied at MIT for my Bachelors and Masters in Electrical Engineering and Computer Science (EECS) with Minor in Mathematics.

• During my Masters, I concentrated on Artificial Intelligence and worked under the joint guidance of Prof. Regina Barzilay and Dr. Julian Straub. My main projects were (1) non-rigid image registration of mammograms for breast cancer detection, and (2) 3D reconstruction of human arms for efficient lymphedema screening. You can find out more about them here.

Along the way, I worked at academic research labs at MIT (CSAIL, Media Lab, McGovern Institute) and INRIA (ILDA lab). I also interned in the industry, at a French robotics startup (Keecker), Apple (6-month COOP) and MathWorks (PhD Research Intern).

Research Interests

Geometric Fingerprints of Generative Models

Broadly, my research interest lies in understanding how complex, high-dimensional information-processing systems (e.g., human intelligence, modern generative models, collective behaviors like traffic patterns) behave and developing efficient algorithms to analyze their characteristics and traces in a principled way, from geometric perspectives. Through this geometric understanding of their properties and internal mechanisms, I aim to improve their behaviors (e.g., steering away from degenerative patterns and biases while aligning to more proper values), by mechanistically intervening in their internal causal pathways. Of many such complex systems, my PhD research focuses on generative models and representation learning algorithms.

Currently, despite the rapid development of Generative A.I., there is still a big gap in our understanding and control over their designs and behaviors. My aim is to fill this gap, by developing a formal framework and efficient algorithms that can represent, analyze and control the behaviors of generative models (e.g., Large Foundation Models (FMs)) in their high-dimensional spaces.

To this end, I approach these problems from geometric perspectives, and I am working on formalizing the theory of (generative) model behaviors using Differential Geometry and Riemannian Manifold. Grounded on this theory, I also work to develop efficient algorithms that can extract geometric signatures of the behaviors and internal representations of large generative models (which live on much higher dimensional spaces than what the current geometric approach has been employed (e.g., 1D signals or 3D images) ), in order to "fingerprint" and characterize them.

My recent work and extensions

I have started on this research endeavor through my work on fingerprinting generative models: In our recent work published in CVPR 24, we proposed a useful theoretical framework to represent model behaviors on a data manifold (i.e., as vector fields on a data manifold constructed from real images). We also formalized, for the first time in the literature, the definition of “artifacts” and “fingerprints” of generative models in a geometric language, and proposed an effective attribution method to study and compare vision generative models. (See more details here.)

• I am excited to extend this work to a larger variety of Foundation Models, including LLMs and SoTA multimodal models, to study their fingerprints and proactively watermark the models.

• I will also look into their internal mechanisms that cause these fingerprints and develop principled methods to intervene and control causal pathways that are responsible for different model behaviors (e.g., image artifacts by Vision GMs or hallucinations in LLMs).

I hope by researching these problems, I will contribute to a generalized theory of generative models and their internal mechanisms, which in turn help shape a safe and responsible integration of Generative A.I. into our society.

Research Thread 2

Another thread of my research lies at the intersection of representation learning and information theory, inspired by the way our perceptual system integrates multimodal sensory inputs via identifying invariant semantics. I am interested in understanding how the semantic information flows while processing observations from multiple modalities, using tools in deep learning and thermodynamic approaches to information flow.

My guiding question is,

How do we extract the shared semantics from observations expressed in vastly different representational forms (eg. images, sounds, written texts), and how do we create/actualize various forms of observations, starting from the semantics we want to communicate?

I approach this question from an information-processing point of view and am developing generative models with disentangled representation to jointly learn the analysis and synthesis processes of multimodal data. My most recent work introduces a generative model with adversarial training that learns spatial semantics from map tiles collected from diverse sources such as satellites, Google Street Map and custom rendering engines.

Currently, I am exploring different ways to understand our proposed model, in particular, by measuring semantic information and studying the flow of information between the latent partitions.

• How we can quantify the amount of the shared semantic information captured by our model?
• If we view each latent partition as a subsystem that constitutes a global system represented by the whole latent space, then we can view the the adversarial discriminator as a demon (like Maxwell's Demon) sitting at the boarder of the latent subsystems.

• From this point of view, (how) does this adversary -- the demon sitting at the boarder of the content and style latent partitions -- achieve the non-equilibrium state of the semantic vs. domain-specific information by "sorting" or "organizing" the information into the correct partition as the training happens?

It's exciting to see how the ideas and tools in thermodynamics can help quantify and visualize this flow of semantic information in our model!

Please see more details on my research in my publications and projects :)

More Details (on My Research Questions)