ASU Research ProfileINR and SIREN guide

Implicit neural representationsPeriodic activationsSIRENComputer graphics

ASU Research Profile: Neural Fields With Periodic Activations

A research explainer for implicit neural representations: what they are, why sine-based networks are different from ReLU networks, how the math works, and where these models show up in graphics, physics, audio, images, and neural rendering.

Research Paper SIREN Source

Core object

f_theta(x)

A neural function that maps coordinates to values.

Key trick

sin activations

Periodic layers make fine detail and derivatives easier to learn.

Main model

SIREN

Sinusoidal Representation Network, introduced for INR signal fitting.

Research use

PDE fields

Continuous functions can represent physics solutions, SDFs, audio, images, and scenes.

Interactive lab

Play with the core ideas

These demos are simplified, deterministic models designed for intuition. They show coordinate queries, frequency learning behavior, derivative fields, and oscillatory physics without needing a GPU or a training loop in the browser.

Coordinate Field Explorer

Activation

Frequency scale6

Phase offset0.35

Probe readout

x0.62

y0.38

f(x, y)-0.036

approx grad(-4.72, 8.19)

Click the field to query the function at a coordinate. A real INR learns the field values through weights; this toy shows how a coordinate function can be sampled anywhere.

Tiered explanation

From intuition to research depth

Each tier explains the same idea with more precision, starting from a simple mental model and ending with the reasons periodic networks behave differently.

5th grader

A picture without a grid

Imagine asking a tiny robot, 'what color is the picture at this exact spot?' Instead of storing every square pixel, the robot learns an answer machine. Give it a location, and it gives back a color, height, sound volume, or other value.

High school

Coordinates in, signal out

An INR treats data as a function. A 2D image is not only a table of pixels; it can be a function from (x, y) to (red, green, blue). A sound can be a function from time to pressure. A 3D shape can be a function from (x, y, z) to distance from the surface.

Undergrad

Why periodic activations matter

ReLU networks are piecewise linear. They can represent complex signals, but training often learns smooth, low-frequency structure first. A SIREN puts sine functions inside the network, so the model naturally has oscillations, smooth higher-order derivatives, and a useful frequency basis for fine detail.

Expert

Function-space inductive bias

Periodic activations change the hypothesis class and the training dynamics. SIRENs compose learned affine maps with sinusoids at every hidden layer, while Fourier-feature MLPs inject fixed or random sinusoidal bases at the input. Both fight spectral bias, but SIRENs keep the periodic basis trainable throughout the depth of the model.

Deeper model

What is actually happening inside an INR

The important design choice is not only the activation function. It is the whole representation: continuous coordinates, signal-specific overfitting, differentiability, initialization, and the frequency bias of the architecture.

Representation

Neural fields, not lookup tables

An INR is a field stored as a neural network. Instead of saving every pixel, voxel, or sample, it saves a function that can be queried at any continuous coordinate.

Frequency

The high-frequency problem

Natural signals often contain edges, transients, textures, and waves. Standard MLPs can fit them in principle, but optimization often learns smooth low-frequency parts first.

Activation

Why sine is more than decoration

Sine activations are smooth, periodic, and differentiable at every order. That matters when the output is judged by gradients, Laplacians, normals, or wave behavior.

Encoding

Fourier features versus SIREN

Fourier features add sinusoidal coordinates before a conventional network. SIREN makes the hidden layers sinusoidal, so the basis can be transformed and recombined throughout depth.

Optimization

Initialization is part of the model

A periodic network can become unstable if early weights create extreme oscillations or vanishing gradients. SIREN's initialization is designed to keep activations and derivatives usable.

Priors

Learning families of functions

A single INR can overfit one signal, but hypernetworks and meta-learning can generate many INRs, which is useful for related PDEs, shapes, images, or scenes.

Workflow

How a SIREN or INR is trained

The same recipe appears across images, audio, geometry, neural rendering, and physics. What changes is the coordinate domain, the target value, and the loss.

1. Choose the domain

Normalize coordinates such as pixels, time, or 3D points into a stable range, often around [-1, 1] or [0, 1].

2. Choose the signal target

Decide whether the network should output color, density, signed distance, audio amplitude, displacement, or a physical state vector.

3. Pick the representation family

Use plain ReLU for smooth simple fields, Fourier features for encoded high frequencies, SIREN for periodic hidden layers and derivative-heavy tasks, or hybrid encodings for large scenes.

4. Sample coordinates

Train on coordinate/value pairs. For PDEs, sample interior points, boundary points, and initial-condition points instead of only observed data.

5. Optimize losses

Combine data loss with derivative losses, PDE residuals, normal constraints, Eikonal terms, or rendering losses depending on the application.

6. Query continuously

After training, evaluate the function at arbitrary coordinates for rendering, interpolation, gradients, normals, or downstream tasks.

Math core

The equations behind the page

The important shift is that data becomes a differentiable coordinate function. Once the model is a function, calculus becomes part of the representation.

Implicit neural representation

f_theta: R^d -> R^k, x |-> y

The model stores a continuous field in weights theta. The input x is a coordinate, and y is a signal value such as RGB, density, pressure, occupancy, or signed distance.

SIREN hidden layer

h_i = sin(omega_i * (W_i h_{i-1} + b_i))

The sine nonlinearity appears throughout the network. The omega term controls initial frequency scale; the paper uses a principled initialization so activations and gradients stay trainable.

Signal fitting

L_data = (1 / N) * sum_i ||f_theta(x_i) - y_i||_2^2

For an image, sample pixel coordinates and train the function to output the correct colors. At test time, query any coordinate, including between original samples.

Derivative supervision

L_grad = sum_i ||grad_x f_theta(x_i) - g(x_i)||_2^2

Because f_theta is differentiable, the loss can supervise gradients, Laplacians, normals, or PDE residuals instead of only target values.

SDF / Eikonal view

||grad_x phi(x)||_2 = 1, phi(surface) = 0

A signed distance field can be learned by enforcing surface constraints and the Eikonal equation. This is why derivative quality matters so much.

Comparison

How this differs from ReLU and other methods

These methods can all represent functions, but they place different assumptions on frequency, smoothness, memory, and derivatives.

Baseline

ReLU MLP

Strong for general learning, but its piecewise-linear structure and low-frequency bias make sharp texture, audio, wavefields, and clean derivatives harder to fit in coordinate-based regression.

Input encoding

Fourier features

Before the MLP sees x, encode x as sin(Bx) and cos(Bx). The bandwidth of B controls how much high-frequency detail the downstream ReLU network can learn.

Hidden encoding

SIREN

Use sin as the activation inside every hidden layer. Frequencies, phases, and compositions are learned through the weights, giving strong signal fitting and derivative behavior.

Explicit storage

Grids and voxels

How it is used

A field can represent pressure, temperature, density, geophysical potential, acoustics, or simulation outputs without committing to one fixed mesh resolution.

Why periodic detail matters

Many measured fields contain waves, boundaries, and multi-scale behavior; derivative access also helps analysis and inverse problems.

Applications

What INRs are used for

INRs show up anywhere the object of interest is better thought of as a continuous field than as a fixed table of samples.

2D signals

Images and video

Map pixel or spacetime coordinates to color. This makes resolution-independent reconstruction, super-resolution experiments, and compression-style overfitting possible.

1D signals

Audio and waveforms

Map time to amplitude. Periodic activations are a natural fit for sounds because audio is built from oscillatory structure across many frequencies.

3D geometry

Signed distance fields

Map a 3D point to distance from a surface. The learned field can produce meshes, normals, collision tests, and smooth geometry queries.

Physics

PDE and PINN solvers

Use derivatives from automatic differentiation to train on equations like Poisson, Helmholtz, wave, and Eikonal constraints.

Neural rendering

NeRF-style scenes

Represent a scene as a coordinate field. NeRF maps spatial location and viewing direction to density and radiance for novel view synthesis.

Meta-learning

Families of functions

Hypernetworks can generate INR weights, letting a model learn a prior over many related signals instead of fitting one function from scratch every time.

Caveats

When this is powerful, and when it is not

Training can be slow because a network may be optimized per signal, per scene, or per PDE instance.

A compact neural field is not automatically faster than a grid lookup; every query requires network evaluation.

Very large or dynamic scenes need acceleration structures, factorization, hash grids, caching, or explicit-neural hybrids.

SIRENs are sensitive to initialization and frequency scale. A bad scale can underfit, overfit, or make gradients unstable.

Compression claims depend on the bitrate target, signal type, entropy coding, architecture, and decode cost.

Continuous does not mean perfectly correct between samples. The learned function interpolates according to its inductive bias.

Derivative accuracy is valuable but fragile; a model can match values while still producing bad gradients.

Neural fields are hard to edit directly because the signal is stored across many coupled weights instead of obvious pixels or voxels.

Glossary

Terms that come up constantly

INR

Implicit Neural Representation; a neural network used as a coordinate-to-signal function.

Neural field

A broad name for continuous fields represented with neural networks or neural data structures.

SIREN

Sinusoidal Representation Network; an INR architecture using sine activations in hidden layers.

Spectral bias

The tendency of many neural networks to learn low-frequency structure before high-frequency detail.

Fourier features

A coordinate encoding that maps inputs through sine and cosine functions before an MLP.

SDF

Signed Distance Field; a function whose value is distance to a surface, negative inside and positive outside.

Eikonal equation

A constraint often written ||grad phi|| = 1 for signed distance functions.

PINN

Physics-Informed Neural Network; a neural network trained with losses derived from physical equations.

NeRF

Neural Radiance Field; a scene function mapping 3D location and view direction to density and radiance.

Hypernetwork

A network that outputs the weights of another network, useful for learning priors over many INRs.

Expert notes

What makes periodic INRs interesting

The research value is not just that sine waves can draw squiggles. The deeper point is that coordinate networks need the right bias for fine-scale signals and derivative-constrained problems.

Spectral bias is the tendency of standard networks to learn low-frequency components before high-frequency detail. In coordinate regression, that can blur textures, audio transients, and geometry.

Fourier features move sinusoidal structure to the input. SIRENs move it into the entire network, so deeper layers can compose phases and frequencies rather than only consuming a fixed encoding.

Derivative fidelity is not a side benefit. It is central for normals, curvature, PDE residuals, wave propagation, and boundary-value problems.

The SIREN initialization is part of the method. A sine network can fail if weights push activations into unstable frequency scales or shrink gradients too aggressively.

Periodic activations also connect to stationary function priors in Bayesian neural networks, which helps explain why they encode translation-aware assumptions in function space.

Reading trail

Sources used for this guide

These are the papers and project pages that shaped the content. The first two are the best entry points for understanding SIRENs and Fourier-feature coordinate networks.

Implicit Neural Representations with Periodic Activation Functions

Sitzmann et al., NeurIPS 2020. SIREN paper and project page.

Fourier Features Let Networks Learn High Frequency Functions

Tancik et al., NeurIPS 2020. Input encodings for overcoming spectral bias.

On the Spectral Bias of Neural Networks

Rahaman et al., ICML 2019. Frequency-dependent learning behavior.

NeRF: Representing Scenes as Neural Radiance Fields

Mildenhall et al., ECCV 2020. Continuous 5D neural fields for novel views.

Signal Processing for Implicit Neural Representations

Xu et al., NeurIPS 2022. Processing continuous signals stored in MLP weights.

Periodic Activation Functions Induce Stationarity

Meronen, Trapp, and Solin, NeurIPS 2021. Function priors from periodic activations.

Physics-informed neural networks

Raissi, Perdikaris, and Karniadakis, Journal of Computational Physics 2019. Neural PDE solvers and inverse problems.

Instant Neural Graphics Primitives

Muller et al., SIGGRAPH 2022. Multiresolution hash encoding for fast neural graphics primitives.

COIN: COmpression with Implicit Neural representations

Dupont et al., ICLR Neural Compression Workshop 2021. Image compression by storing fitted network weights.

Hypernetwork-based Meta-Learning for Low-Rank PINNs

Cho et al., NeurIPS 2023. Related paper on efficient PDE-family approximation.

Where Do We Stand with Implicit Neural Representations?

Technical survey covering INR methods, advantages, applications, and open problems.

DeepSDF: Learning Continuous Signed Distance Functions

Park et al., CVPR 2019. Continuous signed-distance functions for 3D shape representation.

Occupancy Networks

Mescheder et al., CVPR 2019. 3D reconstruction as a continuous occupancy function.

Compression with Bayesian Implicit Neural Representations

Guo et al., NeurIPS 2023. Compressing signals by encoding neural-field weights.

Neural Fields in Robotics

Irshad et al., 2024 survey. Applications in pose estimation, manipulation, navigation, physics, and autonomous driving.