LongBench v2 / 66ee3c6d821e116aacb20f6f / When it comes to large-scale volumetric calcium imaging, which statement is…

Nature Methods
nature methods
https://doi.org/10.1038/s41592-024-02400-9
Article
Self-inspired learning for denoising live-cell 
super-resolution microscopy
Liying Qu1,14, Shiqun Zhao2,14, Yuanyuan Huang1,14, Xianxin Ye2, Kunhao Wang2, 
Yuzhen Liu1, Xianming Liu 
 
 3, Heng Mao4, Guangwei Hu 
 
 5, Wei Chen 
 
 6, 
Changliang Guo 
 
 2, Jiaye He 
 
 7,8, Jiubin Tan9, Haoyu Li 
 
 1,9,10,11, 
Liangyi Chen 
 
 2,12,13 & Weisong Zhao 
 
 1,9,10,11 
Every collected photon is precious in live-cell super-resolution (SR) 
microscopy. Here, we describe a data-efficient, deep learning-based 
denoising solution to improve diverse SR imaging modalities. The method, 
SN2N, is a Self-inspired Noise2Noise module with self-supervised data 
generation and self-constrained learning process. SN2N is fully competitive 
with supervised learning methods and circumvents the need for large 
training set and clean ground truth, requiring only a single noisy frame for 
training. We show that SN2N improves photon efficiency by one-to-two 
orders of magnitude and is compatible with multiple imaging modalities for 
volumetric, multicolor, time-lapse SR microscopy. We further integrated 
SN2N into different SR reconstruction algorithms to effectively mitigate 
image artifacts. We anticipate SN2N will enable improved live-SR imaging 
and inspire further advances.
Fluorescence microscopy of live cells requires gentle imaging condi-
tions and adequate spatiotemporal resolution to record authentic 
biological information, thus the photon budget is usually limited. By 
encoding SR information via specific optics and fluorescent on–off 
indicators, the SR techniques have further strengthened the spatial 
resolution1 and enabled previously unappreciated, intricate structures 
to be observed2–5. However, for a finite number of fluorophores within 
the cell volume, the increase in spatial resolution leads to the rise in 
illumination intensity or exposure time by orders of magnitude to 
maintain the signal-to-noise ratio (SNR)6. Furthermore, any increase 
in spatial resolution must be matched with an increase in temporal 
resolution to prevent motion artifacts7,8. This is particularly challenging 
for live-cell SR imaging to accumulate sufficient photons.
Because of the hardware-limited photon budget, computa-
tionally boosting the SNR is essential to maximize the utilization of 
sensor-collected photons. By modeling the image formation process, 
classical denoising algorithms based on numerical filtering and math-
ematical optimization can remove the noise from fluorescence images 
to a certain extent9–12. However, the corresponding carried assumptions 
are not dependent on the specific content of the imaging data and 
cannot fulfill the optimal performance. Therefore, to capture the full 
statistical complexity of data, the field has witnessed a sudden surge 
Received: 22 January 2024
Accepted: 31 July 2024
Published online: xx xx xxxx
 Check for updates
1Innovation Photonics and Imaging Center, School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, China. 2State Key 
Laboratory of Membrane Biology, Beijing Key Laboratory of Cardiometabolic Molecular Medicine, Institute of Molecular Medicine, National Biomedical 
Imaging Center, School of Future Technology, Peking University, Beijing, China. 3School of Computer Science and Technology, Harbin Institute of 
Technology, Harbin, China. 4School of Mathematical Sciences, Peking University, Beijing, China. 5School of Electrical and Electronic Engineering, 
Nanyang Technological University, Singapore, Singapore. 6School of Mechanical Science and Engineering, Advanced Biomedical Imaging Facility, 
Huazhong University of Science and Technology, Wuhan, China. 7National Innovation Center for Advanced Medical Devices, Shenzhen, China. 8Shenzhen 
Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. 9Key Laboratory of Ultra-precision Intelligent Instrumentation of 
Ministry of Industry and Information Technology, Harbin Institute of Technology, Harbin, China. 10Frontiers Science Center for Matter Behave in Space 
Environment, Harbin Institute of Technology, Harbin, China. 11Key Laboratory of Micro-Systems and Micro-Structures Manufacturing of Ministry of 
Education, Harbin Institute of Technology, Harbin, China. 12PKU-IDG/McGovern Institute for Brain Research, Beijing, China. 13Beijing Academy of Artificial 
Intelligence, Beijing, China. 14These authors contributed equally: Liying Qu, Shiqun Zhao, Yuanyuan Huang. 
 e-mail: weisongzhao@hit.edu.cn


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
each pixel of the camera is sampled independently; (p4) The imaging 
system can be regarded as an equivalent low-pass filter. In this case, 
each SR image is composed of many PSFs multiplied with the corre-
sponding fluorophore brightness at different spatial positions, and 
each PSF occupies at least larger than an area of 2 × 2 pixels12. Consider-
ing this spatial redundancy (p1 and p2), we can directly resample one 
SR image as two subimages by an unusual form of binning we regard as 
‘diagonal resampling’ (Fig. 1a). More specifically, we consider every four 
adjacent pixels (2 × 2) as one unit, and every two diagonal pixels (top 
left and bottom right, magenta color in Fig. 1a; top right and bottom 
left, green color in Fig. 1a and Extended Data Fig. 1a) are averaged as one 
new pixel. Because of the independence of each pixel and the spatial 
symmetry on the two diagonal directions (p2 and p3), we can gener-
ate statistically independent image pairs that share identical details 
but different noise realizations for the N2N configuration (Methods). 
Fundamentally, assuming identical contents inside one smallest unit, 
our method can be seen as a relaxation form of the temporal resam-
pling approach22–24, which requires the entire contents of two frames 
being identical. Compared to other spatial generation methods33–35, 
this resampling in diagonal axes creates the most content-similar 
image pair (Supplementary Fig. 2a), providing a more stable process 
without the need for additional registration or calibration to create 
perfectly matched image content. Notably, although originated from 
the spatial redundancy of SR images, our approach is not sensitive to 
changes in pixel size and can produce stable denoising results even for 
undersampled images (until 260 nm pixel size with 150 nm resolution; 
Extended Data Fig. 2a,b).
Because the produced two subimages are two times smaller, we 
further adapt an interpolation method to rescale them to the original 
structural scale (Methods). Different from the conventional spatial 
methods, we apply a Fourier interpolation to the resulting two sub-
images, which is based on the fact that the optical transfer function 
(OTF) of an SR microscope has only finite support (p4). Padding the 
Fourier-transformed image out of its OTF support with zeros does not 
alter its information content36, and after back-transforming, this pad-
ded image will have doubled pixel numbers in both x and y axes, identi-
cal to the original SR image (Fig. 1a and Extended Data Fig. 1a). Without 
this operation, the network may produce structural artifacts for the 
scale difference between the training and test datasets (Extended Data 
Fig. 2e). On the other hand, the spatial methods (such as bilinear inter-
polation) create more pixels according to the noise-corrupted pixels, 
and it will be problematic when meeting the background areas without 
fluorescence signal (Supplementary Fig. 2b), because these smoothly 
created pixels do not conform to the randomness of noise, which may 
influence the learning process, potentially producing background 
artifacts (Extended Data Fig. 2e).
Self-constrained learning process. Beyond the full form of N2N, we 
further design a self-constrained learning process to constrain the 
training process and denoising variance (Fig. 1a). This is based on a 
simple intuition that the denoising predictions of the same underly-
ing signal should be identical. In our case, the generated data pair has 
matched content but different noise components, thus ideally, the 
corresponding two denoised results should have no differences. Spe-
cifically, the two generated images are successively and individually fed 
into the network, and the two resulting predictions can be calculated 
by the loss function shown in equation (1) to execute the training stage.
1 = ‖ ̃
x1 −x2‖1 + ‖ ̃
x2 −x1‖1 + λ‖ ̃
x1 −
̃
x2‖1,
(1)
where ̃
x represents the network outcome of input x, and || ||1 refers to 
the l1 norm. We used the l1 norm as the distance calculation (Methods). 
The first two terms are the conventional N2N losses, and the last term 
is our self-constrained loss with a constraint weight λ. By enforcing this 
constraint, we found that the low-frequency components manifest a 
of data-driven methods, producing the unprecedented restoration 
results13. After iterative training on a dataset with ground truth (GT) 
labels, deep neural networks (DNNs) can learn the mapping between 
noisy images and their clean counterparts14–16. Intuitively, for super-
vised learning, the collection of abundant content-matched clean 
images is crucial, which is of great challenge to live-cell applications, 
especially under the SR scale.
To denoise images without clean ones, Noise2Noise (N2N)17 learns 
a mapping between pairs of independently degraded versions of the 
same image, and its performance can approach that of supervised 
learning methods. Nevertheless, the need for the twin noisy pairs is 
still against the live-cell SR applications. By leveraging the pixel-wise 
independence of noise, several approximated forms of N2N have 
been developed to denoising without paired data18–21. However, these 
approximations may lead to downgraded performances. To realize the 
full form of N2N configuration, the DeepCAD22 used a temporally inter-
leaved self-supervised data generation process to create the required 
noisy data pairs, by assuming that the two adjacent frames in a video of 
continuous imaging can be considered with the same underlying con-
tent. Unfortunately, although this assumption is routinely satisfied in 
calcium imaging22,23 or other fast-imaging applications24, it is still hard 
to accomplish in live-cell SR techniques considering the commonly 
compromised temporal resolution and increased spatial resolution.
On the other hand, SR imaging usually has a more-than-sufficient 
sampling rate, at least over the Nyquist sampling theorem to protect 
the enriched spatial information. Thus, we create a self-supervised data 
generation strategy based on this spatial redundancy, using a diagonal 
resampling step followed by a Fourier interpolation. Compared to 
the previous resampling methods22–24, this spatially interleaved data 
generation is more universal and stable, and it produces almost no bias 
in our tests. Beyond this spatially self-supervised N2N realization, we 
also develop a self-constrained learning process to further enhance 
the denoising performance and data efficiency. Conceptually, the two 
network predictions of the generated noisy pair will be constrained 
to one identical expectation, and this process shrinks the learning 
and predictive uncertainty, inherently increasing the efficiency and 
effectiveness. Together, our self-inspired N2N (SN2N) reaches or even 
outperforms supervised learning-based denoising methods, especially 
when only a single frame is used for training.
To showcase the broad applicability, we apply SN2N directly to 
data obtained on two commercial spinning-disk confocal-based struc-
tured illumination microscopes (SD-SIM)25,26, two commercial stimu-
lated emission depletion (STED)27,28 microscopes and a high-resolution 
confocal microscope. Here, we show the method enables high-quality, 
long-term, multicolor three-dimensional (3D) live-cell SR imaging 
(five-dimensional (5D) in xyz color time). We also show benefits for 
expansion microscopy (ExM)29. Beyond that, we also integrate our 
SN2N framework into the existing SR reconstruction procedures, 
including iterative deconvolution on SD-SIM and STED, SR optical 
fluctuation imaging (SOFI) reconstruction30,31 and SIM10, effectively 
mitigating the artifacts embedded in the SR images and enhancing the 
photon efficiencies. These organic integrations deliver the fact that 
our SN2N can serve as a practical framework for further strengthening 
the performance of live-cell SR microscopy. With fully open-sourced 
code, we also expect our SN2N will be applicable to other fields for 
random noise removal.
Results
Principle of SN2N
Self-supervised data generation. Before designing the data gen-
eration strategy, we intend to list several physical properties (p) of SR 
microscopy32: (p1) To fulfill the increased spatial resolution, the sample 
rate is routinely finer than the Nyquist theorem. (p2) The point spread 
function (PSF) represents the highest frequency of the system and 
 
is usually with spatial symmetry inside 2 × 2 pixels; (p3) Commonly, 


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
greater propensity of being stable than the high-frequency ones 
(Extended Data Fig. 2g,h). Thus, to avoid over-smoothed results, we 
routinely set the constraint weight as 1. Notably, our SN2N is model 
independent, thus we used the commonly used U-Net37 to highlight its 
ability (Fig. 1a, Extended Data Fig. 1a and Methods).
Data augmentation for low-level tasks. Although there are plenty of 
strategies to increase the training data amount, they are designed for 
high-level classification tasks and not suitable for low-level denoising 
applications38. Accordingly, we develop random patch transformations 
in multiple dimensions (Patch2Patch) to further improve the data 
efficiency (Extended Data Fig. 1b). Specifically, the randomly selected 
regions of interest on each image will be rotated or flipped, or will 
remain still and subsequently interchanged with other regions of inter-
est from the same frame or a different frame from different times or 
experiments (Methods and Supplementary Video 1). This Patch2Patch 
can create more imaging results without changing the inherent noise 
properties and hence it can effectively reduce the required data bulk.
Benchmarking with known structures
Simulation validations. To quantitatively test SN2N’s performance, we 
first validated it on synthetic microtubule imaging data with 150 nm 
resolution and three different Poisson and Gaussian noise levels 
(Methods and Supplementary Fig. 1). Our SN2N solution is superior 
1/5 data
Full data
1/50 data
xy2
Step 3. Self-constrained learning
Self-inspired process
a
d
b
Raw
GT
PURE
ACsN
N2V
Supervised
SN2N (w/o c)
SN2N
c
STD: 48.32
STD: 16.41
STD: 12.40
STD: 15.97
STD: 5.12
STD: 11.87
STD: 5.42
e
f
Steps 1–2. Self-supervised data generation
Self-constraining
Self-supervising
Self-supervising
Step 1. Diagonal resampling
Step 2. Fourier rescaling
=
=
1
2
3
4
+
2 + 3
Net (xy1)
Net (xy2)
iFFT
xy1
xy2
FFT
Padding
1 + 4
Convolution
Skip connection
Upsampling
Downsampling
xy1
Correlation
0.2
0.4
0.6
0.8
1.0
0
Spatial frequency  (µm–1)
0
2.5
5.0
7.5
10.0
10
15
20
25
PSNR
SN2N
k = 1.2
SN2N
(w/o c)
k = 2.9
k = 0
Raw
N2V
k = 2.7
Supervised
k = 4.3
SN2N
(w a)
k = 0.9
Raw
PURE
ACsN
N2V
SN2N
SN2N (w/o c)
Supervised
1.0
0.8
0.6
0.4
0.2
0
SSIM
Level 1
Level 2
Level 3
1/7
Fig. 1 | Workflow and simulation validation of SN2N. a, Overview of SN2N. 
Steps 1–2, self-supervised data generation. The single-frame input image of 
H × W pixels is diagonally resampled to two images of H/2 × W/2 pixels. Then, 
the two resulting images are rescaled back to two images of H × W pixels with a 
Fourier interpolation. Step 3, self-constrained learning process, that is, the two 
rescaled images serve as both inputs and labels, and the corresponding two 
predicted images from a classical U-Net architecture are constrained to minimize 
the difference between them (Methods). b, Validation of SN2N using synthetic 
microtubule structures (Methods). The synthetic structures were convoluted 
with a 150 nm PSF and downsampled two times (pixel size, 32.5 nm) as GT. The 
noisy images were created by further injection of Poisson noise and Gaussian 
noise. From left to right, noisy (top) and GT images (bottom), PURE, ACsN, N2V, 
supervised, SN2N without self-constrained loss (SN2N w/o c) and SN2N denoising 
results. c, Data uncertainty results of b. Ten independent frames of identical 
contents under the same imaging conditions were fed to the trained SN2N 
network, and the STD of the ten resulting predictions was calculated as the data 
uncertainty. Marked numbers are the average values of STD maps. d, FRC analysis 
of the denoising images. e, SSIM values of various denoising methods under 
different noise levels (n = 10). f, PSNR values of networks trained by different 
amounts of data under the level-1 condition (n = 10). Full data dimensions are 
2,048 × 2,048 × 50. k denotes the slope (red lines) of PSNR values along the data 
increment. Error bars indicate the s.e.m. Experiments were repeated ten times 
independently with similar results. Scale bar, 1 µm (b).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
to existing methods in two aspects. First, SN2N is denoising effectively, 
especially for ultralow-SNR conditions. Under the low-SNR condition 
(level 1), classical denoising methods, for example, Poisson unbiased 
risk estimate (PURE)9 and automatic correction of sCMOS-related 
noise (ACsN)11, failed to eliminate the noise and left overly blurred 
underlying structures (Fig. 1b). The approximated form of N2N, 
 
Noise2Void (N2V)19, can reduce the noise, but the predicted images 
from ten repetitive acquisitions are still with large standard deviation 
(STD) values (for example, 15.97 in Fig. 1b,c), reflecting the limited 
performances. Removing the self-constrained learning process (SN2N 
w/o c), the improvement of our method against N2V is relatively limited 
and cannot compete with the supervised learning method using suf-
ficient training data. The full SN2N truly approaches the performance 
of supervised learning with similar STD and even better Fourier ring 
e
STD
20
15
10
5
0
SN2N
SN2N
(w/o c)
N2V
PURE
SD-SIM
SD-SIM
SN2N-1f (full aug)
f
g
l
SN2N-1f 
(full aug)
Supervised-1f
(w/o aug)
GT
SN2N-1f 
(full aug)
SN2N-1f
(w/o aug)
Supervised-1f
(w/o aug)
SN2N-500f
Supervised-
500f
GT
SD-SIM
SD-SIM
SN2N-1f 
(full aug)
GT
h
i
j
k
n
o
Supervised-1f
(w/o aug)
SD-SIM
m
a
SD-SIM
d
Spatial frequency (µm–1)
Correlation
0.2
0.4
0.6
0.8
1.0
0
25
20
15
10
5
0
b
240
210
180
150
120
Distance (nm)
LRQ
0.20
0.25
0.30
0.35
0.40
0.10
0
c
SSIM
0.2
0.4
0.6
0.8
1.0
0
SN2N
SN2N
(w/o c)
N2V
PURE
SD-SIM
Raw
Supervised
-1f 
SN2N-1f
0
0.2
0.4
0.6
0.8
1.0
SSIM
Raw
Supervised
-1f 
SN2N-1f 
0
0.2
0.4
0.6
0.8
1.0
SSIM
Data
uncertainty
Model
uncertainty
STD
4
6
8
10
12
14
16
Supervised (w/o aug)
SN2N (w/o aug)
Supervised-1f (w/o aug)
SN2N-1f (w/o aug)
SN2N-1f
SN2N-500f
Supervised-1f (w/o aug)
SN2N-1f (full aug)
Supervised-1f (w/o aug)
SN2N-1f (full aug)
Supervised-1f (w/o aug)
SN2N-1f (full aug)
Training data amount (frame)
1
500
100
10
SSIM
0.5
0.6
0.7
0.8
0.9
1.0
Exposure time
1×
10×
5×
2×
0.5
0.6
0.7
0.8
0.9
1.0
SSIM
W/o
aug
Full 
aug
P2P
aug
Basic
aug
SSIM
0.5
0.6
0.7
0.8
0.9
1.0
SD-SIM
90
120
150
180
210 240 (nm)
PURE
N2V
100×
exposure
SN2N
SN2N (w/o c)
1/7
100× exposure
PURE
N2V
SN2N (w/o c)
SN2N
Fig. 2 | Systematical evaluations in SD-SIM experiments using known 
structures. a, Benchmarking on the Argo-SIM slide under the SpinSR10 SD-SIM 
system. Representative images are presented below the corresponding intensity 
profiles indicated by the white line. Top row, low-SNR SD-SIM images (left), and 
PURE (middle) and N2V (right) denoising results; bottom row, our SN2N without 
self-constrained loss (w/o c), full SN2N denoising results and high-SNR (with 100× 
exposure) SD-SIM images. Intensity profiles of the double-line pair are displayed 
as insets. b, The LRQ values of images in a. c, Average SSIM values (n = 10). d, FRC 
analysis of images in a. e, The STD of the denoising images predicted from ten 
repetitively collected noisy images. f, Qdot 525 (QD525)-labeled microtubules in 
fixed COS-7 cells imaged by SD-SIM (left) and SN2N (right) trained with one frame 
and full augmentation (‘SN2N-1f (full aug)’). g, Enlarged region enclosed by the 
white boxes in f. From left to right: Raw image under SD-SIM, GT image, supervised 
learning method (‘Supervised-500f’), SN2N results trained with 500 frames 
(‘SN2N-500f’), supervised learning (‘Supervised-1f (w/o aug)’), SN2N (‘SN2N-1f 
(w/o aug)’) trained with one frame and no augmentation, and SN2N trained 
with one frame and full augmentation (‘SN2N-1f (full aug)’). h–j, Average SSIM 
values of different training data amounts (n = 10) (h), different models trained by 
images under different exposure time (n = 10) (i) and different data augmentation 
strategies (n = 10) (j). k, Data and model uncertainties quantified by the STD of 
ten independent frames and ten independently trained models, respectively. 
l,m, Representative denoising data in fixed COS-7 cells of lysosomes labeled with 
LAMP1-EGFP (l) and mitochondria labeled with Tom20–mGold1 (m). n,o, SSIM 
values of imaging results in l (n) and m (o) (n = 10). In the box blots, the center line 
indicates the median, box limits indicate the 25th and 75th percentiles, and the 
whiskers represent the maximum and minimum values; error bars indicate the 
s.e.m. Experiments were repeated ten times independently with similar results; 
scale bars, 1 µm (a and f), 500 nm (g) and 2 µm (l and m).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
correlation (FRC)39, structural similarity (SSIM)40, peak signal-to-noise 
ratio (PSNR) and root-mean-square error (RMSE) metrics (Fig. 1b–e 
and Extended Data Fig. 3b). On the other hand, when examining the 
higher SNR conditions (level 2 and level 3), all methods can achieve 
acceptable denoising results (Fig. 1e and Extended Data Fig. 3a) and 
the improvements are less impressive, in which the results of N2V and 
SN2N without constraint (SN2N w/o c) reach closer to the full SN2N 
(Supplementary Table 1).
Second, SN2N is data efficient and can be trained even on one 
single image. The requirement of large datasets for DNNs to capture 
the accurate data distribution of high noise level is moderated by our 
self-constrained learning process. To test this, we measured the slopes 
of SSIM, PSNR and RMSE metrics of different learning-based methods 
by decreasing the training data size by a factor of 5 and 50 (one frame 
only; Fig. 1f, Extended Data Fig. 3c,d and Supplementary Table 2). As 
the data pool shrunk, the denoising quality of the supervised learning 
method dropped quickly, for example, k = 4.3 for PSNR (Fig. 1f), and this 
variation also dramatically effected the performances of N2V (k = 2.7) 
and SN2N w/o c (k = 2.9). In contrast, it is clearly observed that the full 
SN2N moderated the influence (k = 1.2). Our self-constrained loss helps 
the network to learn the denoising process more efficiently, which is 
further strengthened by the developed Patch2Patch (k = 0.9, SN2N 
with augmentation, SN2N w a). Fundamentally, these two aspects are 
highly correlated with the data and model uncertainties of the DNNs41, 
in which the denoising effectiveness reflects the data uncertainty, and 
data efficiency represents the model uncertainty. Using ten acquisi-
tions’ predictions and ten repetitively trained models’ predictions, we 
measured the data and model uncertainties, respectively. Consistently, 
it can be seen that our SN2N outperforms other methods (Extended 
Data Fig. 3e and Supplementary Figs. 3 and 4). Finally, we applied 
N2V19, parametric probabilistic Noise2Void (PPN2V)42, Self2Self (S2S)43, 
Recorrupted2Recorrupted (R2R)44, Noise2Fast (N2F)33 and our SN2N 
on the single-frame simulation dataset (Supplementary Fig. 5a), and we 
found that SN2N reached the closest to the GT, reflecting the advanced 
denoising ability using a limited data amount.
Experimental evaluation using standard sample under SD-SIM. SR 
confocal microscopy can double the spatial resolution by narrowing 
its pinhole size, but correspondingly the number of photons reaching 
the detector is severely restricted25. Although the photon reassignment 
concept has mitigated this inherent low-SNR condition25, the photon 
efficiency of SR confocal microscopy still needs to be improved for 
capturing fast long-term suborganelle dynamics in multiple dimen-
sions (5D in xyz-color-time), especially for its paralleled version, that 
is, SD-SIM26. Next, we experimentally evaluated our SN2N with GT 
under a commercial SD-SIM system (Olympus SpinSR10 with an sCMOS 
camera; Methods). Compared to the diffraction-limited confocal mode 
(Extended Data Fig. 4d), the integrated optical reassignment module 
and ×3.2 second-stage magnification system of SpinSR10 bring heavily 
reduced SNR conditions. Using a commercial Argo-SIM slide (Meth-
ods), we measured the variance across the vertical straight line, and 
only SN2N could draw the expected brightness profile with minimum 
fluctuations, even smaller than that of image under 100× exposure 
(Fig. 2a). Furthermore, the double-line structures are highlighted by 
SN2N from noise with the highest contrast (Fig. 2a and Supplementary 
Video 2). Based on several metrics including line restoration quality 
(LRQ; Methods)12 according to the known line structures, SSIM against 
the 100× exposure generated GT, FRC and STD values from repetitive 
acquisitions, as well as visual inspection, we found SN2N successfully 
restored real-world collected images with superior stability and quality 
(Fig. 2b–e, Extended Data Fig. 4a and Supplementary Table 3).
Experimental evaluation by biological samples under SD-SIM. To 
examine the broad applicability of SN2N, we applied it on microtubules 
and outer membranes of lysosomes and mitochondria in fixed COS-7 
cells under another commercial SD-SIM system (GATACA Systems, 
Live-SR with an sCMOS camera; Methods) (Fig. 2f,l,m). Generally, the 
ultimate goal of unsupervised learning-to-denoise is to approach the 
similar performance of supervised learning without requiring the 
data pairs and large training set. In this case, we focused on compar-
ing our SN2N against the supervised learning method in the denois-
ing performance and data efficiency (Fig. 2f,l,m), in which the 500× 
exposure result is considered as the GT. When only using one frame, the 
supervised learning method produces obscure structures (Fig. 2g,l,m), 
indicating an underfitting configuration, and the performance grows 
dramatically when increasing the data amount from 1 to 500 frames 
(Fig. 2g,h). In contrast, we found the expansion of the data pool has 
little effect on the denoising performance of SN2N (Fig. 2g,h), and 
the integration of our Patch2Patch data augmentation (Methods) 
helps the one-frame-trained model to approach the performance of 
the 500-frame-trained model (Fig. 2j and Supplementary Table 5). 
Furthermore, by taking advantage of its intrinsic data efficiency and 
further reasonable data augmentation, SN2N exploits the full potential 
of available data, producing superior model and data uncertainties 
(Fig. 2k). Interestingly, we found that the one-frame-trained SN2N 
is less affected by the SNR degradations (Fig. 2i and Supplementary 
Table 6), prompting us to explore the trickier denoising task of multi-
color live-cell SD-SIM data.
RL-SN2N on SD-SIM unlocks fast long-term imaging across 5D
Multicolor live-cell SR imaging. The Richardson–Lucy (RL) deconvo-
lution45,46 has been routinely applied on SD-SIM to enhance the contrast 
and resolution and is especially useful in scenarios containing strong 
out-of-focus signals or requiring precise segmentation. However, 
it is prone to artifacts under live-cell imaging conditions (Fig. 3b). 
Thus, beyond denoising, we integrate the RL deconvolution with our 
SN2N (RL-SN2N; Methods) for simultaneously achieving the artifacts 
removal and resolution enhancement (Fig. 3a,b). To avoid breaking the 
Fig. 3 | Multicolor live-cell SR imaging enabled by RL-SN2N on SD-SIM.  
a, Workflow of RL-SN2N (Methods). In the training stage, RL deconvolution was 
applied after self-supervised data generation to create the RL-SN2N training 
set. In the inference stage, the denoised data are used for segmentation and 
downstream analysis. b, A representative example for dual-color SR imaging 
of mitochondria (green) and ER (magenta) labeled with Tom20–mCherry 
and Sec61β-EGFP in live COS-7 cells under SD-SIM (top left), SD-SIM after RL 
deconvolution (top right, RL SD-SIM), SN2N result (bottom left) and RL-SN2N 
result (bottom right), alongside the enlarged region of the yellow dashed box.  
c, A representative example for four-color imaging of the mitochondria (green), 
ER (gray), lysosomes (red) and GA (blue) labeled with MitoTracker Deep Red 
FM, Sec61β-EGFP, Lamp1–mCherry and Golgi-BFP in live COS-7 cells under raw 
SD-SIM (right) and RL-SN2N (left). d, The yellow dashed box in c is enlarged and 
shown at seven time points under different configurations. From top to bottom: 
Raw SD-SIM, RL-SN2N, RL SD-SIM segmentation (by Otsu hard threshold) and 
RL-SN2N segmentation (by Otsu hard threshold) results. The lines in different colors 
indicate different interaction events (E1–E4). e, Segmentation results of the 
mitochondria (orange, by Mitonet) and ER (by ERnet) under SD-SIM (left), and 
RL-SN2N (right). The ERnet segmentations contain tubules (cyan), sheets (yellow) 
and sheet-based tubules (magenta, SBTs). f, Trajectories of lysosomes exhibiting 
directed motion (green), free diffusion motion (blue) and confined motion (red), 
and ER tubules (black). g, Distribution of the estimated α values of lysosomes 
versus their temporal average distances to ER tubules. h, Average mitochondrial 
diameters (n = 100; Methods). i, Average numbers of events surpassing the MOC 
threshold (>0.26) of Lys–Mito (n = 46). DM (green), directed motion; FDM (blue), 
free diffusion motion; CM (red), confined motion. In the box blots, the center line 
indicates the median, box limits indicate the 25th and 75th percentiles, and the 
whiskers represent the maximum and minimum values; error bars indicate the 
s.e.m. Experiments were repeated ten times independently with similar results; 
scale bars, 2 µm (c–e), and 5 µm (b).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
a
c
d
Segmentation
E1: Lys-GA with Mito; E2/E3/E4: Lys-GA with ER
13.6 s
19.5 s
32.3 s
49.3 s
60.3 s
63.7 s
83.3 s
RL-SN2N
SD-SIM
RL-SN2N 30.0 s
SD-SIM 30.0 s
RL-SN2N
RL SD-SIM
b
e
f
g
h
i
Step 3. Self-constrained
learning
RL-SN2N
Steps 1–2. Self-supervised data generation
Step 1. RL
deconvolution
Noisy data
Data pair
RL data pair
Step 1. Diagonal resampling
& Fourier ×2
Step 2. RL deconvolution
xy1
xy2
xy1
Net (xy1)
Net (xy2)
xy2
Step 3. Organelle
segmentation
Step 2. SN2N
inference
Trained model
Training stage
Step 4. Downstream
analysis
Denoised data
Statistics
Smart algorithm
Segmentation
Inference stage
SD-SIM
RL-SN2N
Number of events
50
60
70
DM FDM CM
Directed motion 
Confined motion 
ERnet tubules
Free diffusion
Mito diameter (µm)
Lys-Mito
ER-Mito
0
0.5
1.5
1.0
Contacted
Not contacted
Mitonet
ERnet SBTs
ERnet sheets
ERnet tubules
0
0.4
0.8
1.2
1.6
Mean distances (µm)
MSD coefficient (α)
0
0.1
0.2
0.3
0.4
Confined motion 
Free diffusion 
Directed motion 
ER-Lys
Lys-Mito
RL-SN2N
SN2N
RL SD-SIM
SD-SIM


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
RL-SN2N
SD-SIM
SD-SIM
Nucleus
3D rendering
x
yz
~46 µm
~46 µm
~6 µm
0 h 10 min
0 h 30 min
0 h 35 min
0 h 40 min
0 h 50 min
0 h 55 min
1 h 5 min
1 h 0 min
1 h 10 min
0 h 0 min
SD-SIM
RL-SN2N
a
2D-UNet
3D-UNet
b
c SD-SIM
Segmetation
e
f
g
yz
SD-SIM
RL-SN2N
SD-SIM
RL-SN2N
SD-SIM
RL-SN2N
5 µm
0 µm
(z)
xz
Segmentation
RL-SN2N
5 µm
0 µm
(z)
RL-SN2N
1 h 0 min
1 h 10 min
1 h 0 min
1 h 10 min
SD-SIM
500 nm
y
z x
7.6 µm
7.6 µm
Convolution
Downsampling
Upsampling
Skip connection
h
1 h 50 min
2 h 55 min
i
1 h 10 min
ER
Mito
d
39 min
33 min
3 min
0 min
2.8 µm
0.4 µm
(z)
yz
xz
xz
yz
yz
xz
xz
yz
0 min
RL-SN2N
xz
yz
12 min
xz
yz
27 min
SD-SIM
xz
yz
xz
57 min
yz
xz
42 min
yz
xz
27 min
yz
5 µm
0 µm
(z)


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
pixel-wise noise independence, the deconvolution is executed after 
the self-supervised data generation step during the training stage, 
and at the inference phase, the trained SN2N model is fed with the 
directly deconvolved images. We first validated the performance of 
RL-SN2N using the Argo-SIM slide, and its LRQ contrast substantially 
outperformed SN2N (Extended Data Fig. 4b,c).
Also, we expect the resulting high-quality SR images can 
empower precise segmentation, facilitating automated analysis at 
suborganelle-level precision from massive data (see the prototype 
in Fig. 3a and Extended Data Fig. 5a). The first representative case is 
dual-color imaging of endoplasmic reticulum (ER) and outer mito-
chondrial membrane (OMM) labeled live COS-7 cells (Supplementary 
Video 3). We can observe that our RL-SN2N effectively extracts the 
fluorescence signal from the noisy background with further enhanced 
contrast (Extended Data Fig. 5b,c), hence many relative movements 
between OMM and ER can be dissected. Predictably, the direct hard 
threshold segmentation (Methods) on the low-SNR data produced 
highly broken structures and strong false positives. Our RL-SN2N mini-
mized such noise-induced segmentation artifacts, allowing inspection 
of fast mitochondrial fission events near the ER–mitochondria contact 
sites (Extended Data Fig. 5c).
Next, we extended the application to four-color live-cell imaging 
of lysosomes (Lys), Golgi apparatus (GA), mitochondrial matrix protein 
(Mito) and ER (Fig. 3c and Supplementary Video 4). Consistently, our 
RL-SN2N provided plausible reconstructions for monitoring multiple 
organelle interactions from obscure captures. The increases in SNR 
and contrast were stable during time-lapse SR imaging (Fig. 3d), which 
led to distinct observations of Lys–GA (GA being surrounded by Lys) 
with Mito (event 1, E1) and Lys–GA with ER (E2–E4) interactions. In 
these events, we found that Mito or ER settled in a circle could anchor 
onto a moving Lys–GA and be pulled out for spatial movements as 
it followed the trajectory of Lys–GA (segmentations in Fig. 3d)47,48. 
Furthermore, we applied two recently developed learning-based seg-
mentation methods, that is, the ERnet49 and Mitonet50, on our ER and 
Mito data (Methods). Although better than the hard threshold segmen-
tations, we found these well-trained networks were still influenced by 
the noise conditions (Fig. 3e and Extended Data Fig. 5d–g). On the other 
hand, empowered by our RL-SN2N, ERnet could precisely draw the 
ER topology of tubules, sheets and sheet-based tubules (Fig. 3e). The 
trajectories of Lys and spatial masks of ER tubules (Methods and Fig. 3f) 
enabled us to examine the correlation between motions of lysosomes 
and the ER network. By calculating the mean square displacement 
(MSD) of Lys and their distances to ER, the Lys with confined motion 
behaviors were mostly located adjacent to ER48 (Fig. 3g and Extended 
Data Fig. 5h,i,k). Mitonet-generated masks allowed us to calculate the 
Mander’s overlap coefficient (MOC) of Lys with the nearest Mito for 
identifying potential functional sites of Lys–Mito contact sites24, that 
is, MOC > 0.26 as a potential event (Methods and Extended Data Fig. 5j), 
in which the Lys with directed motion behaviors exhibited a relatively 
larger number of events compared to the free diffusion and confined 
motions (Fig. 3i). We also quantified the mitochondrial diameters at 
the ER–Mito or Lys–Mito contact sites (Methods and Fig. 3h), reflecting 
the fact that the constricted loci in Mito preferentially associated with 
the ER–Mito contact48. Together, in the absence of tedious manual pro-
cesses, our RL-SN2N on SD-SIM system simplified the dissection of the 
synergy of different organelles at a suborganelle scale (Extended Data 
Fig. 5a). Comparatively, N2V19, PPN2V42, S2S43, R2R44, N2F33, DeepCAD22, 
DeepSeMi21 and SRDTrans34 cannot provide valid denoising results from 
data under such ultralow-SNR conditions (Supplementary Fig. 6a).
3D extension for volumetric SR imaging across 5D. With roots in the 
confocal microscopy, SD-SIM can perform 3D SR imaging with further 
enhanced axial contrast by the 3D deconvolution. To fully utilize the 
axial information, we extend the RL deconvolution and U-Net from 
its 2D37 version to its 3D mode (Fig. 4a and Extended Data Fig. 4f,g). 
By our RL-SN2N, 3D mitochondrial networks were visualized, and, as 
expected, it revealed tori cross-sections of OMM genuinely, which 
were ambiguous in the raw SD-SIM results (Fig. 4b and Supplementary 
Video 5). Various types of OMM structures, that is, from tubular to a 
series of different structures including fragments, small vesicles and 
spheroids, can be resolved by our RL-SN2N (Fig. 4d). The segmentation 
also helps to highlight the hollow structures of OMM networks, which 
are hardly distinguishable in the raw images (Fig. 4c). Furthermore, 
benefiting from SNR and contrast reinforcements, we can capture the 
Fig. 4 | 3D RL-SN2N on SD-SIM unlocks fast long-term imaging across 5D.  
a, The extension of 2D U-Net (top) to its 3D mode (bottom). b–d, 3D OMM network 
imaging of Tom20–mCherry-labeled live COS-7 cells. b, Color-coded volumes of 
raw SD-SIM (top) and RL-SN2N (bottom). The color-coded axial views (yz and xz 
planes) indicated by the yellow dashed lines are provided alongside. Magnified  
view of yellow boxed regions on xz section is shown at the bottom left of the  
images. c, 3D rendering views of the white boxed region in b under raw SD-SIM 
(1st column), segmentation of SD-SIM (2nd column), SN2N (3rd column) and 
segmentation of SN2N (4th column). d, 2D slices under raw SD-SIM (top) and  
RL-SN2N (bottom). e,f, 4D imaging of OMM network (Tom20–mCherry) in live COS-
7 cells. e, Representative color-coded volumes and their xz and yz cross-sections 
at six time points. f, Magnified views and their xz and yz cross-sections of the white 
boxed region in e under RL-SN2N at four time points. The red and white arrows 
indicate the mitochondrial fission and before fission, respectively. g, 5D imaging 
of mitochondria (green, mGold-Mito-N-7), ER (magenta, DsRed-ER) and nucleus 
(blue, SPY650-DNA) in live COS-7 cells. Representative 3D rendering views of the 
cell mitosis process at ten time points under RL-SN2N, except for the first (left part) 
and the last views being under raw SD-SIM. h, Magnified views from yellow boxes in 
g under raw SD-SIM (left) and RL-SN2N (right). i, Two representative time points of 
mitochondria (green) and ER (magenta) after mitosis. Experiments were repeated 
five times independently with similar results; scale bar, 5 µm (b and i), 1 µm (c, d and 
h), 10 µm (e) and 2 µm (f).
Fig. 5 | SN2N and RL-SN2N permit long-term live-cell STED imaging. a, Results 
of live HeLa cells labeled with SiR-tubulin under STED (center pixel, STED 
depletion laser power as 50%, left) and its SN2N result (STED-SN2N, right). Data 
from ref. 52 (Methods). b, Magnified views of the white boxed region in a under 
different imaging configurations. FRC-measured resolution values are labeled. 
c, A sketch of different pixel assembly strategies: center pixel (‘Cent. pix. ‘), 5 × 5 
sum and APR. d, Fluorescence intensity profiles along the white arrow in b of 
confocal/STED (center pixel, left) and their SN2N results (right). e, FRC analysis 
of the images in b. f, Average FWHM values (n = 6, positions). g, STED snapshots 
of live COS-7 cells labeled with SiR-Tubulin (top), LifeAct-EGFP (middle) and 
Sec61β–EGFP (bottom) under a commercial STED microscope (Leica; Methods). 
h, Three representative time points of STED (left) and SN2N (right) counterparts 
of magnified views of the white-boxed regions in g. i,j,m, A representative 
example of PKMO-labeled live COS-7 cells imaged under different conditions 
using another commercial STED microscopy (Abberior). i, Three STED frames 
under high depletion power (86%) and long duration time (100 μs per pixel). 
 j, Representative frames of STED (left) and SN2N results (right) at high depletion 
power (86%) and short duration time (10 μs per pixel). k, Photobleaching analysis 
of STED images used in i, j and m. l, Fluorescence profiles along the white arrow 
in j. m, Long-term imaging of STED (left) and RL-SN2N (right) at low depletion 
power (41%) and short duration time (10 μs per pixel). n, Magnified views of the 
white boxed regions in m. o,p, Representative montages of the mitochondrial 
fusion (o) and fission (p) events. The yellow and blue arrows highlight the regions 
of mitochondrial fusion and fission, respectively, while the white arrows indicate 
the moments before events. In the box blots, the center line indicates the median, 
box limits indicate the 25th and 75th percentiles, and the whiskers represent the 
maximum and minimum values; error bars indicate the s.e.m. Experiments were 
repeated five times independently with similar results; scale bar, 2 µm (a), 1 µm 
(b, g and h) and 500 nm (i and m–o). a.u., arbitrary units.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
SN2N
243 nm
Confocal
267 nm
Center pixel
STED-SN2N
Center pixel
STED
STED 50%
APR
Cent. pix.
5 × 5 sum
a
b
c
SN2N
d
e
f
0
1.0
Intensity (a.u.)
Distance (nm)
Center pixel confocal/STED
0
1.0
Distance (nm)
Intensity (a.u.)
STED power (%)
FWHM (nm)
STED (%)
Spatial frequency (µm–1)
0
2.5
12.5
5.0
7.5
10.0
Correlation
g
i
j
k
l
m
n
RL-SN2N
o
0 min 0 s
1 min 40 s
STED
SN2N
RL-SN2N
STED 0%
STED 8%
STED 16%
STED 30%
STED 50%
STED
Center pixel
148 nm
SN2N
Center pixel
91 nm
STED -ISM
166 nm
STED-ISM
173 nm
SN2N
Center pixel
101 nm
STED
Center pixel
155 nm
SN2N
Center pixel
104 nm
STED-ISM
203 nm
STED
Center pixel
193 nm
STED -ISM
206 nm
SN2N
Center pixel
109 nm
STED
Center pixel
210 nm
Confocal
Center pixel
216 nm
SN2N
Center pixel
118 nm
ISM
5 × 5 APR
213 nm
STED 0%
h
Center pixel confocal/STED-SN2N
0
1
High power
long duration
High power
short duration
Low power
short duration
7 min 55 s
3 min 45 s
5 min 50 s
0 min 0 s
0 min
STED
SN2N
3 min
6 min
3 min
SN2N
STED
0 min
6 min
STED
SN2N
6 min
3 min
SN2N
STED
0 min
1 min 40 s
3 min 45 s
7 min 55 s
0 min 0 s
0 min 0 s
2 min 0 s
2 min 30 s
3 min 45 s
4 min 10 s
18 min 0 s
16 min 15 s
16 min 40 s
24 min 35 s
Intensity (a.u.)
Distance (nm)
A
0
0.2
0.4
0.6
0.8
1.0
300
0
250
200
150
100
50
76 nm
Intensity (a.u.)
Time (min)
0
0.2
0.4
0.6
0.8
1.0
0
10
20
30
Center pixel 
confocal/STED
Center pixel 
SN2N
SN2N
STED
5 × 5 APR
5 × 5 APR
5 × 5 APR
5 × 5 APR
5 × 5 sum
5 x 5 sum
300
0
250
200
150
100
50
300
0
250
200
150
100
50
0
20
40
60
80
50
100
150
200
250
300
0
ISM/STED-ISM
Center pixel confocal/STED
Center pixel SN2N
Confocal/STED
Low power short duration
High power short duration
High power long duration
3 min 45 s
3 min 45 s
1 min 40 s
1 min 40 s
7 min 55 s
0 min 0 s
0 min 0 s
0 min 0 s
0 min 0 s
1 min 40 s
3 min 45 s
5 min 50 s
7 min 55 s
STED
STED
SN2N
SN2N
80
0
5
8
12
16
19
30
50
p
1/7


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
four-dimensional (4D) OMM network dynamics across hours (Fig. 4e, 
Supplementary Fig. 10 and Supplementary Video 5). In our RL-SN2N 
results, the kiss-and-run events happening in 3D are correctly identi-
fied (Fig. 4f), which might be misinterpreted as the fissions in 2D slices 
under low-SNR and contrast conditions. Finally, we extended our test 
to the challenging 5D SR imaging, revealing the dynamics of ER, Mito 
and nuclei during the entire cell mitosis process in all three dimensions 
over a long duration of 3 h (Fig. 4g, Supplementary Fig. 11 and Supple-
mentary Video 6). Empowered by RL-SN2N, we clearly monitored the 
segregation process (Fig. 4h) and the interactions (Fig. 4i) of dense 
ER–Mito networks.
Additionally, according to the simulations, we found our 
approach is not sensitive to pixel size (Extended Data Fig. 2a). Thus, 
our success in SD-SIM equipped with an sCMOS camera (~38 nm pixel) 
made us wonder whether our SN2N and RL-SN2N can be applied to 
diffraction-limited SD-confocal microscopy (~65 nm pixel) or SD-SIM 
with an EMCCD camera (~94 nm pixel) (Methods). On the Argo-SIM 
slice, our SN2N removed the readout noise of SD-confocal images 
(Extended Data Fig. 4d,e). Likewise, by applying RL-SN2N (with further 
2× upsampling) on two-color whole-cell volumes from the EMCCD 
SD-SIM, we successfully recorded two intermediate phases of cell mito-
sis, in which the 3D mitochondrion and nuclear protein distributions 
came into presence from background noise (Extended Data Fig. 6).
Empowering long-term live-cell STED imaging
Similarly to confocal SR microscopy, the increase in spatial resolution of 
STED microscopy from depletion brings dramatically decreased SNR51. 
Because the depletion is driven by de-excitation through stimulated 
emission, enlarging the depletion laser power will inherently decrease 
the emission brightness of fluorescence labels and cause adverse effects 
such as photobleaching and phototoxicity, preventing long-term moni-
toring of samples51,52. Thus, the SN2N extraction of structures from few 
photons gives the possibility to tackle the contradiction for imaging live 
cells with both high spatial resolution and SNR. First, we systematically 
evaluated SN2N’s denoising performances under different depletion 
powers (Fig. 5a,b and Supplementary Fig. 12), in which the data52 were 
collected from a custom STED setup incorporating a SPAD array detec-
tor with 5 × 5 elements for adaptive pixel-reassignment (APR; Methods). 
By assembly of SPAD signals collected from the center pixel, the spatial 
resolution of STED is further improved but most photons would be 
dropped. From ref. 52, the 5 × 5 APR STED results, or equivalently the 
STED image scanning microscopy (STED-ISM), can fully exploit the 
collectible photons to preserve SNR (Fig. 5c). Differently, without these 
collected photons from the circumjacent 24 pixels, our SN2N effectively 
recovered the live microtubule structures from only photons collected 
from the center pixel, especially under a high depletion power (Fig. 5b). 
The two-peak draws of microtubule intersections, FRC curves and 
full width at half maximum (FWHM) measurements also highlight the 
increment process of spatial resolution without sacrificing SNR after 
SN2N denoising (Fig. 5d–f).
It is straightforward to apply our SN2N on a commercial STED 
system (Leica, TCS SP8 STED 3X; Methods) for time-lapse imaging of 
microtubule-, actin- and ER-labeled COS-7 cells (Fig. 5g,h and Supple-
mentary Video 7), routinely enabling high-quality live-cell SR imaging. 
Beyond that, we also recorded the long-term dynamics of mitochon-
drial cristae (PK Mito Orange, PKMO53 labeled) using another com-
mercial STED system (Abberior Instruments, STEDYCON; Methods) 
under various imaging conditions (Supplementary Video 7). To acquire 
high-quality STED images in situ, the high depletion power and long 
pixel duration time were applied and hereby instantly extinguished 
the fluorescence signal before five frames of recording (Fig. 5i). The 
high depletion power with short pixel duration time could delay the 
bleaching effects without loss of spatial resolution but create noisy 
captures, in which our SN2N effectively restored this SNR degrada-
tion (Fig. 5j,k). The measured cristae-to-cristae distance reflects the 
achievable resolution of ~76 nm (Fig. 5l). Further turning down the 
depletion laser power will offer significantly less susceptibility to pho-
tobleaching but short of spatial resolution maximization (Fig. 5k,m). 
Fortunately, our RL-SN2N can lift the dropped resolution in the absence 
of amplified photobleaching (Fig. 5m,n). Short exposure and low illu-
mination energy facilitated us to record the perplexing locomotion of 
cristae during mitochondrial fusion (Fig. 5o) and fission (Fig. 5p) over 
half an hour. In contrast, conventional STED produced noisy SR images 
under the same conditions (Fig. 5m).
Improving reconstruction efficiency of SOFI
SOFI30 can routinely break the diffraction limit by exploiting the natural 
temporal fluctuations of fluorescence emissions under optical systems 
in their native states. However, the statistical uncertainty of reconstruc-
tions from short sequences may dramatically affect image continuity 
and homogeneity, which leads it to generally requiring hundreds of raw 
images (~1,000 frames for 2nd order) to preserve structural integrity31. 
To increase its reconstruction efficiency, we integrated our SN2N solu-
tion into the SOFI reconstruction pipeline (Fig. 6a). Specifically, the raw 
image sequence is calculated by the nth-order cumulant (core SOFI), and 
the resulting image is followed by the RL-SN2N procedure (Methods). 
Using SIM as a reference, the 2nd-order SOFI-SN2N is first validated on 
a wide-field microscope (Methods). We found that the strong snowflake 
artifacts in conventional 20-frame SOFI were effectively eliminated, and 
the original microtubule structures were highlighted with high axial 
contrast (Fig. 6b, Extended Data Fig. 7a and Supplementary Video 8). 
 
The two-peak analyses, FRC metrics and FWHM measurements all 
demonstrate the massively improved temporal resolvability and effec-
tively increased spatial resolution of SOFI-SN2N (Fig. 6b–d). Next, we 
extended our SOFI-SN2N to different orders of cumulants executed 
on a commercial SD-confocal microscope (Fig. 6e–h and Methods). 
Consistently, both visual examination and FRC analysis exhibited that 
the 2nd-order SOFI-SN2N can produce artifact-free results from only 
20 frames (Fig. 6e–g). On the other hand, the increase of resolution 
from higher-order cumulants brings a cost of more frames needed, and 
our SOFI-SN2N enables efficient 3rd-order and 4th-order SOFI recon-
structions from 50 frames and 100 frames, respectively (Fig. 6f and 
Extended Data Fig. 7b–e). The FWHM- and FRC-measured resolution 
values (Fig. 6h) also evince the spatial resolution enhancement. Finally, 
the recording of OMM dynamics during mitochondrial fusion during 
10 min gives us a glance at live-cell SN2N-SOFI SR imaging (Fig. 6i,j and 
Supplementary Video 8).
In fixed-cell experiments, the temporal sampling22,24 (first and 
second 20 frames for SOFI reconstruction) can be applied directly 
(Extended Data Fig. 7f). Interestingly, due to the inconsonant temporal 
fluctuation behaviors, the temporal sampling leads to statistical dif-
ferences between the adjacent frames and produces strong artifacts. 
In contrast, our spatial sampling is impressed with sturdily extracting 
the microtubule structures for requiring no temporal consistency 
(Extended Data Fig. 7f).
SN2N on expanded samples
In addition to the use of fluorescence fluctuations, the ExM29 is another 
system-agnostic SR modality by artificially enlarging the size of samples 
to break the diffraction limit. However, considering the determined 
number of fluorophores, this space extension of samples will result 
in the geometrically decreased SNR according to the expansion times 
(Extended Data Fig. 7g–k). Under a wide-field microscope (Methods), we 
offered the ExM-SN2N enabling high-quality SR imaging of ~110 nm and 
~67 nm resolutions for 2× and 4× expansions (Extended Data Fig. 7g–i), 
respectively. Under different noise levels, the ExM results after SN2N 
denoising exhibited significantly improved signal-to-background 
ratios. Similarly, the SN2N outcomes of the cells expanded by 4.5 times 
yielded (Extended Data Fig. 7j,k) noise-eliminated results, revealing 
the complex ER tubule structures.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
FRC (nm)
SN2N
SOFI
Conf.
100
150
200
250
300
350
100
150
200
250
300
350
FWHM (nm)
Conf.
SN2N
SOFI
a
b
c
d
i
j
e
f
15
3
6
9
12
18
0
0
0.2
0.4
0.6
0.8
1.0
Correlation
Spatial frequency (µm–1)
Wide-field
SOFI-20f
SN2N-20f
2D-SIM
SOFI-SN2N
g
Confocal
4th SN2N 2,000f
2nd SOFI 20f
2nd SN2N 20f
3rd SOFI 50f
3rd SN2N 50f
4th SOFI 100f
4th SN2N 100f
2nd SOFI 20f
2nd SOFI-SN2N 20f
2nd SOFI-SN2N 20f
Steps 1-3. Self-supervised data generation
Training stage
Inference stage
Step 4. Self-constrained learning
Step 1. Nth order
cumulantion (SOFI)
Step 2. Diagonal resampling
& Fourier ×4
Step 3. RL
deconvolution
SOFI
xy1
xy2
SOFI data pair
Net (xy1)
Net (xy2)
Step 1. Direct SOFI
reconstruction
Step 2. SN2N inference
Trained model
Distance (nm)
Intensity (a.u.)
130 nm
65 130 195 260 325
0
390
0
1.0
65 130 195 260 325
0
390
Distance (nm)
Intensity (a.u.)
0
1.0
65 130 195 260 325
0
390
Distance (nm)
0
1.0
Intensity (a.u.)
2D-SIM
2nd SOFI-SN2N 20f
Wide-field
2nd SOFI 20f
h
120
140
160
180
200
220
240
260
280
FRC (nm)
WF
SOFI
SN2N
SIM
FWHM (nm)
WF
SOFI
SN2N
SIM
120
140
160
180
200
220
240
260
280
0
1.0
Intensity (a.u.)
132 nm
Distance (nm)
65 130 195 260 325
0
390
2nd SN2N 20f
2nd SOFI 20f
Confocal
Spatial frequency (µm–1)
15
0
0.2
0.4
0.6
0.8
1.0
3
6
9
12
18
0
Correlation
Confocal
2nd SOFI 20f
2nd SN2N 20f
10 min 40 s
11 min 20 s
12 min 40 s
xy1
xy2
12 min 0 s
12 min 0 s
12 min 0 s
12 min 0 s
7 min 20 s
8 min 0 s
2nd SOFI 20f 
12 min 0 s
Confocal
2nd SOFI
3rd SOFI
4th SOFI
1/7
1/7
Fig. 6 | Integration of SN2N and SOFI massively improves the SR 
reconstruction efficiency. a, Workflow of SOFI-SN2N (Methods). In the training 
stage, self-supervised data generation was applied after nth-order SOFI and 
before Fourier upsampling. b, Cross-validation of SOFI-SN2N. Snapshots of 
microtubules in a COS-7 cell labeled with QD525 under wide-field microscopy 
(top left), 2D-SIM (bottom left), 2nd-order SOFI using 20 frames (2nd SOFI 20 f, 
top right) and its SN2N result (2nd SOFI-SN2N 20 f, bottom right). The intensity 
profiles and multiple Gaussian fitting indicated by the white arrows are provided. 
c, FRC analysis of the images in b. d, Average FWHM (top) and FRC (bottom) 
values (n = 5, measurements). WF, wide-field. e, Results of microtubules in a COS-
7 cell labeled with QD525 under a SD-confocal microscopy reconstructed by 2nd-
order SOFI using 20 frames. f, Zoomed views from the white box in e. First row: 
confocal image (left) and 4th-order SOFI using 2,000 frames denoised by SN2N 
(4th SOFI-SN2N 2,000 f, right); the other rows, from top to bottom: 2nd-, 3rd- and 
4th-order SOFI reconstructions (left) using 20, 50 and 100 frames, respectively, 
and their SN2N results (right). g, FRC analysis of raw SD-confocal image, 2nd 
SOFI 20 f reconstruction, and its SN2N result. h, Average FWHM (left) and FRC 
(right) values of the images in f (n = 5, measurements). i, A representative live 
COS-7 cell labeled with Skylan-S-TOM20 imaged by 2nd SOFI-20f of SD-confocal 
microscopy and its SN2N result. j, Zoomed views of OMM structures. Top three 
rows, from top to bottom: magnified views of the white boxed region in i under 
SD-confocal microscopy, 2nd SOFI 20 f reconstruction, and 2nd SOFI-SN2N 20 f 
result; bottom three rows: montages of a representative mitochondrial fission 
event. The yellow and white arrows highlight the mitochondrial fission and 
before fission, respectively. In the box blots, the center line indicates the median, 
box limits indicate the 25th and 75th percentiles, and the whiskers represent the 
maximum and minimum values; error bars indicate the s.e.m. Experiments were 
repeated three times independently with similar results; scale bars, 2 µm (b, i and 
j), 5 µm (e) and 1 µm (f).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Artifact removal for live-cell SIM
Although SIM is recognized to have a higher photon efficiency than 
other SR modalities, it still requires an adequate SNR for each raw 
image to prevent random reconstruction artifacts10,12,24. Therefore, 
to minimize the artifacts, we included the SN2N module into SIM 
reconstruction (‘raw resampling’, Extended Data Fig. 8a). The gen-
erated twin nine raw images were individually reconstructed by the 
HiFi-SIM54 procedure (high-fidelity SIM; Methods) to create the train-
ing sets, and at inference stage, the SIM images are taken as input by 
using the original nine raw images. First, we systematically evaluated 
the performance of our SIM-SN2N using the BioSR open-sourced 
dataset55 under high (H), medium (M) and low (L) SNR levels. Com-
paring to the GT ultrahigh-SNR reconstructions, our SIM-SN2N effec-
tively disentangled the real features from artifacts and produced 
stable SSIM values for various conditions and samples (Extended 
Data Fig. 8b–g and Supplementary Video 9). Then, the 2D-SIM imag-
ing of mitochondrial cristae exhibited strong background artifacts, 
which was further amplified as the emission fluorescence progres-
sively decreased (Extended Data Fig. 8h,i). The suppression of arti-
facts in SIM-SN2N supports its superior performance in obtaining 
high-fidelity SR images from raw images of low SNR (Extended Data 
Fig. 8j and Supplementary Video 9).
In particular, when meeting ultrafast SIM imaging under 
ultralow-SNR conditions, the resampled dataset might encounter 
reconstruction failures, in which the parameter estimation is highly 
unstable. On the other hand, the frequency-component reassembly 
step in SIM reconstruction is usually accompanied by an artificial 2× 
upsampling, which breaks the pixel independency, and this prevents 
the direct application of SN2N on SIM images. To meet this challenge, 
we conducted a new strategy of spatial sampling by defining 3 × 3 pixels 
as one unit and a new average calculation of four directions followed 
by the Fourier 3× operation (‘SIM resampling’, Extended Data Fig. 8k). 
This one-pixel interval enabled the direct artifact removal on the SIM 
reconstructed images only with a slight drop of SSIM metrics (Extended 
Data Fig. 8l–q). Tested on the ultrafast (188 Hz, ~0.6 ms exposure per 
raw frame) TIRF-SIM experiments, the ‘raw sampling’ failed to offer 
eligible SIM reconstructions, and in contrast, the ‘SIM resampling’ 
provided high-quality images along 6,800 consecutive SR frames 
(Extended Data Fig. 8r–t).
Discussion
In the N2N ecosystem, theoretically, infinite data pairs are essential to 
approach the supervised learning methods’ performance, because of 
the need for averaging training sets to remove the zero-mean noise. 
Using twin images from our self-supervised data generation, the devel-
oped self-constrained learning process further generalizes this N2N 
concept to remove noise with randomness, and also relaxes the need 
for an infinite data amount17. As a result, our SN2N is competitive with 
supervised learning methods but overcomes the need for a large train-
ing dataset and clean GT. Indeed, we showed a single noisy frame is 
feasible for training. We applied SN2N to the photoelectric detector 
directly captured data, including two commercial SD-SIM systems 
with two different types of cameras, one custom-built and two com-
mercial STED microscopes, one ExM under a wide-field microscope 
and one commercial SD-confocal microscope, demonstrating its 
extensive application value and superior performance. We further 
integrated SN2N into the prevailing SR reconstructions, including 
RL deconvolution, SOFI and SIM for artifact removal, enabling effi-
cient reconstructions from limited photons by one-to-two orders 
of magnitude.
A concern of learning-based recovery is that the spatially 
denoised features might distort the temporal signals in a nonlinear 
manner. Recording cells labeled with cytosolic Ca2+ indicators56, we 
found SN2N acted without nonlinearly perturbing the amplitudes 
of different Ca2+ transients, indicating that the SN2N denoising is 
quantitatively accurate (Extended Data Fig. 9a–c). Furthermore, for 
fast imaging applications (Extended Data Fig. 9d,e), although SN2N 
performed superiorly in spatial denoising, DeepCAD’s smaller tem-
poral signal fluctuations led us to integrate its temporal resampling 
method into SN2N (SN2N (temporal)) for further strengthening the 
temporal stability.
Although SN2N was shown to work well overall, it would be appro-
priate to discuss limitations observed in the current version. Here, we 
provided three representative failure cases. The first example is for the 
ultralow-SNR experimental data with strong baseline signal. Without 
percentile normalization (Methods), the resulting predictions will 
exhibit small background fluctuations (Supplementary Fig. 9c). The 
second failure case is the unlimited increasing of noise level and, pre-
dictably, using a fixed amount of data, the SN2N progressively exhib-
ited noisy output (Extended Data Fig. 2i). The last failure case is the 
generalization errors inherited from the mechanism of unsupervised 
learning-to-denoise. The retraining or the transfer learning for the new 
dataset is usually needed. Specifically, the extraction of structures from 
models trained by higher SNR data would exhibit strong background 
artifacts caused by the misleading information learned between noise 
and structures (Supplementary Fig. 9a,b). The outcomes from models 
trained by lower SNR data are free from this failure (Supplementary 
Fig. 9a,b). Furthermore, we directly fed the four-color live-cell imaging 
data (Fig. 3c) to the simulation dataset trained model (Supplementary 
Fig. 9d), and this rough test using data with different features and 
noise levels resulted in stronger hallucinations. Although SN2N can be 
executed without clean data, the internal mechanism of learning-based 
denoising methods is substantially different from the classical ones. 
In an abstract sense, SN2N is learning to extract structures from noisy 
input according to the training set, while the numerical algorithms 
usually intend to remove the noise. We could moderate this issue by 
the iterative execution of the SN2N model (SN2N2) to shrink the gap 
between training and test sets (Extended Data Fig. 10a–f). In addition, 
this network extraction is also influenced by the structural scale of 
input images, and the upsampling/downsampling operation should be 
applied to match the pixel size before SN2N inference (Extended Data 
Fig. 10g–k), otherwise the network would produce erroneous results 
(Extended Data Fig. 10h,i).
Random noise is an unavoidable obstacle in fluorescence 
microscopy, especially for the live-cell SR recording. Our SN2N and 
its extensions provide powerful solutions for routine 2D–5D imaging 
of suborganelle dynamics at ultrahigh spatiotemporal resolution and 
high fidelity for long durations. We anticipate that the elimination 
of noise could benefit precise structure segmentation and facilitate 
automatic multi-parameter analysis, establishing a panoramic view 
of the organelle interaction systems49,50. SN2N is a model-agnostic 
solution. The 2D/3D U-Nets used in this work are simple end-to-end 
baselines, and the extensions to other advanced networks, such as the 
routinely used residual/dense blocks57, adversarial training strategy58 
and transformer-based solutions59, are straightforward. Furthermore, 
the incorporations of SN2N with network-based deconvolution and 
SIM are expected to enable stable and efficient SR reconstructions for 
the real-time potential. Finally, the loss realizations of our SN2N have 
many variants, and the distance calculation by SSIM or in the Fourier 
domain could lead to additional improvements.
Notably, our self-constrained learning process with self-supervised 
data generation have the potential to reduce the data uncertainty of 
learning-based microscopy in general. The predictive fluctuations 
induced by the noise in the input data might be effectively shrunk by 
the integration of our self-constrained learning process. Beyond the 
high-resolution fluorescence imaging showcased in this work, we also 
expect our SN2N can be applied to other sensitive modalities, includ-
ing stimulated Raman scattering microscopy, multiplexed ion beam 
imaging and cryogenic electron microscopy, to further increase the 
imaging throughput and quality in general.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Online content
Any methods, additional references, Nature Portfolio reporting sum-
maries, source data, extended data, supplementary information, 
acknowledgements, peer review information; details of author contri-
butions and competing interests; and statements of data and code avail-
ability are available at https://doi.org/10.1038/s41592-024-02400-9.
References
1.	
Schermelleh, L. et al. Super-resolution microscopy demystified. 
Nat. Cell Biol. 21, 72–84 (2019).
2.	
Schermelleh, L. et al. Subdiffraction multicolor imaging of the 
nuclear periphery with 3D structured illumination microscopy. 
Science 320, 1332–1336 (2008).
3.	
Lawo, S., Hasegan, M., Gupta, G. D. & Pelletier, L. Subdiffraction 
imaging of centrosomes reveals higher-order organizational 
features of pericentriolar material. Nat. Cell Biol. 14, 1148–1158 
(2012).
4.	
Szymborska, A. et al. Nuclear pore scaffold structure analyzed by 
super-resolution microscopy and particle averaging. Science 341, 
655–658 (2013).
5.	
Xu, K., Zhong, G. & Zhuang, X. Actin, spectrin, and associated 
proteins form a periodic cytoskeletal structure in axons. Science 
339, 452–456 (2013).
6.	
Mishin, A. & Lukyanov, K. Live-cell super-resolution fluorescence 
microscopy. Biochemistry 84, 19–31 (2019).
7.	
Godin, A. G., Lounis, B. & Cognet, L. Super-resolution microscopy 
approaches for live cell imaging. Biophys. J. 107, 1777–1784 (2014).
8.	
Valli, J. et al. Seeing beyond the limit: a guide to choosing the 
right super-resolution microscopy technique. J. Biol. Chem. 297, 
100791 (2021).
9.	
Luisier, F., Vonesch, C., Blu, T. & Unser, M. Fast interscale wavelet 
denoising of Poisson-corrupted images. Signal Process. 90, 
415–427 (2010).
10.	 Huang, X. et al. Fast, long-term, super-resolution imaging with 
Hessian structured illumination microscopy. Nat. Biotechnol. 36, 
451–459 (2018).
11.	
Mandracchia, B. et al. Fast and accurate sCMOS noise  
correction for fluorescence microscopy. Nat. Commun. 11,  
94 (2020).
12.	 Weisong, Z. et al. Sparse deconvolution improves the resolution 
of live-cell super-resolution fluorescence microscopy. Nat. 
Biotechnol. 40, 606–617 (2022).
13.	 Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls 
of deep learning for fluorescence image reconstruction. Nat. 
Methods 16, 1215–1225 (2019).
14.	 Weigert, M. et al. Content-aware image restoration: pushing the 
limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 
(2018).
15.	 Chen, J. et al. Three-dimensional residual channel attention 
networks denoise and sharpen fluorescence microscopy image 
volumes. Nat. Methods 18, 678–687 (2021).
16.	 Fang, L. et al. Deep learning-based point-scanning super- 
resolution imaging. Nat. Methods 18, 406–416 (2021).
17.	 Lehtinen, J. et al. Noise2Noise: learning image restoration without 
clean data. In Proceedings of the 35th International Conference 
on Machine Learning (eds Dy, J. & Krause, A.) 2965–2974 (PMLR, 
2018).
18.	 Batson, J. & Royer, L. Noise2self: blind denoising by self-supervision. 
In Proceedings of the 36th International Conference on Machine 
Learning (eds Chaudhuri, K. & Salakhutdinov, R.) 524–533 (PMLR, 
2019).
19.	 Krull, A., Buchholz, T. -O. & Jug, F. Noise2void—learning 
denoising from single noisy images. In Proceedings of the IEEE/
CVF Conference on Computer Vision and Pattern Recognition, 
2129–2137 (2019).
20.	 Eom, M. et al. Statistically unbiased prediction enables accurate 
denoising of voltage imaging data. Nat. Methods 20, 1581–1592 
(2023).
21.	 Zhang, G. et al. Bio-friendly long-term subcellular dynamic 
recording by self-supervised image enhancement microscopy. 
Nat. Methods 20, 1957–1970 (2023).
22.	 Li, X. et al. Reinforcing neuron extraction and spike inference 
in calcium imaging using deep self-supervised denoising. Nat. 
Methods 18, 1395–1400 (2021).
23.	 Lecoq, J. et al. Removing independent noise in systems 
neuroscience data using DeepInterpolation. Nat. Methods 18, 
1401–1408 (2021).
24.	 Qiao, C. et al. Rationalized deep learning super-resolution 
microscopy for sustained live imaging of rapid subcellular 
processes. Nat. Biotechnol. 41, 367–377 (2023).
25.	 Muller, C. B. & Enderlein, J. Image scanning microscopy. Phys. Rev. 
Lett. 104, 198101 (2010).
26.	 Hayashi, S. & Okada, Y. Ultrafast superresolution fluorescence 
imaging with spinning disk confocal microscope optics. Mol. Biol. 
Cell 26, 1743–1751 (2015).
27.	 Hell, S. W. & Wichmann, J. Breaking the diffraction resolution 
limit by stimulated emission: stimulated-emission-depletion 
fluorescence microscopy. Opt. Lett. 19, 780–782 (1994).
28.	 Vicidomini, G. et al. Sharper low-power STED nanoscopy by time 
gating. Nat. Methods 8, 571–573 (2011).
29.	 Sun, D.-E. et al. Click-ExM enables expansion microscopy for all 
biomolecules. Nat. Methods 18, 107–113 (2021).
30.	 Dertinger, T., Colyer, R., Iyer, G., Weiss, S. & Enderlein, J. Fast, 
background-free, 3D super-resolution optical fluctuation imaging 
(SOFI). Proc. Natl Acad. Sci. USA 106, 22287–22292 (2009).
31.	 Zhao, W. et al. Enhanced detection of fluorescence fluctuations 
for high-throughput super-resolution imaging. Nat. Photonics 17, 
806–813 (2023).
32.	 Born, M. & Wolf, E. Principles of Optics, 7th Edn (Cambridge 
University Press, 1999).
33.	 Lequyer, J., Philip, R., Sharma, A., Hsu, W. -H. & Pelletier, L. A fast 
blind zero-shot denoiser. Nat. Mach. Intell. 4, 953–963 (2022).
34.	 Li, X. et al. Spatial redundancy transformer for self-supervised 
fluorescence image denoising. Nat. Comput. Sci. 3, 1067–1080 
(2023).
35.	 Chen, X. et al. Self-supervised denoising for multimodal 
structured illumination microscopy enables long-term 
super-resolution live-cell imaging. PhotoniX 5, 1–22 (2024).
36.	 Stein, S. C., Huss, A., Hähnel, D., Gregor, I. & Enderlein, J. Fourier 
interpolation stochastic optical fluctuation imaging. Opt. Express 
23, 16154–16163 (2015).
37.	 Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional 
networks for biomedical image segmentation. In International 
Conference on Medical Image Computing and Computer-Assisted 
Intervention, 234–241 (2015).
38.	 Yun, S. et al. Cutmix: regularization strategy to train strong 
classifiers with localizable features. In Proceedings of the IEEE/
CVF Conference on ICCV, 6023–6032 (2019).
39.	 Nieuwenhuizen, R. P. et al. Measuring image resolution in optical 
nanoscopy. Nat. Methods 10, 557–562 (2013).
40.	 Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image 
quality assessment: from error visibility to structural similarity. 
IEEE Trans. Image Process. 13, 600–612 (2004).
41.	 Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and 
scalable predictive uncertainty estimation using deep ensembles. 
Adv. Neural Inf. Process. Syst. 30, 6402–6413 (2017).
42.	 Prakash, M., Lalit, M., Tomancak, P., Krul, A. & Jug, F. Fully 
unsupervised probabilistic Noise2Void. In 2020 IEEE 17th 
International Symposium on Biomedical Imaging (ISBI), 154–158 
(2020).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
43.	 Quan, Y., Chen, M., Pang, T. & Ji, H. Self2self with dropout: 
learning self-supervised denoising from single image. In 
Proceedings of the IEEE/CVF Conference on CVPR, 1890–1898 
(2020).
44.	 Pang, T., Zheng, H., Quan, Y. & Ji, H. Recorrupted-to-recorrupted: 
unsupervised deep learning for image denoising. In Proceedings 
of the IEEE/CVF Conference on CVPR, 2043–2052 (2021).
45.	 Richardson, W. H. Bayesian-based iterative method of image 
restoration. J. Opt. Soc. Am. 62, 55–59 (1972).
46.	 Lucy, L. B. An iterative technique for the rectification of observed 
distributions. Astron. J. 79, 745–754 (1974).
47.	 Guo, Y. et al. Visualizing intracellular organelle and cytoskeletal 
interactions at nanoscale resolution on millisecond timescales. 
Cell 175, 1430–1442 (2018).
48.	 Lu, M. et al. The structure and global distribution of the 
endoplasmic reticulum network are actively regulated by 
lysosomes. Sci. Adv. 6, eabc7209 (2020).
49.	 Lu, M. et al. ERnet: a tool for the semantic segmentation and 
quantitative analysis of endoplasmic reticulum topology. Nat. 
Methods 20, 569–579 (2023).
50.	 Sekh, A. A. et al. Physics-based machine learning for subcellular 
segmentation in living cells. Nat. Mach. Intell. 3, 1071–1080 (2021).
51.	 Harke, B. et al. Resolution scaling in STED microscopy. Opt. Express 
16, 4154–4162 (2008).
52.	 Tortarolo, G. et al. Focus image scanning microscopy for sharp 
and gentle super-resolved microscopy. Nat. Commun. 13, 7723 
(2022).
53.	 Liu, T. et al. Multi-color live-cell STED nanoscopy of mitochondria 
with a gentle inner membrane stain. Proc. Natl Acad. Sci. USA 119, 
e2215799119 (2022).
54.	 Wen, G. et al. High-fidelity structured illumination microscopy  
by point-spread-function engineering. Light Sci. Appl. 10,  
70 (2021).
55.	 Qiao, C. et al. Evaluation and development of deep neural 
networks for image super-resolution in optical microscopy.  
Nat. Methods 18, 194–202 (2021).
56.	 Zhang, Y. et al. Mitochondria determine the sequential 
propagation of the calcium macrodomains revealed by the 
super-resolution calcium lantern imaging. Sci. China Life Sci. 63, 
1543–1551 (2020).
57.	 Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J. & Maier-Hein, K. H.  
nnU-Net: a self-configuring method for deep learning-based 
biomedical image segmentation. Nat. Methods 18, 203–211 (2021).
58.	 Mirza, M. & Osindero, S. Conditional generative adversarial nets. 
Preprint at https://arxiv.org/abs/1411.1784 (2014).
59.	 Cao, H. et al. Swin-Unet: Unet-like pure transformer for medical 
image segmentation. In European Conference on Computer 
Vision, 205–218 (2022).
Publisher’s note Springer Nature remains neutral with regard to 
jurisdictional claims in published maps and institutional affiliations.
Springer Nature or its licensor (e.g. a society or other partner) holds 
exclusive rights to this article under a publishing agreement with 
the author(s) or other rightsholder(s); author self-archiving of the 
accepted manuscript version of this article is solely governed by the 
terms of such publishing agreement and applicable law.
© The Author(s), under exclusive licence to Springer Nature America, 
Inc. 2024


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Methods
SN2N framework
SN2N core. In principle, the N2N does not require a clean target to 
train a denoising model approaching supervised learning performance. 
Originally, N2N relies on the assumption that the noise is zero-mean 
and the training set contains infinite noisy data pairs sharing identical 
contents with independent noise components. Our SN2N upgrades 
this to remove random noise with increased data efficiency. First, to 
remove the need for data pairs, we designed a diagonal resampling 
strategy to generate such twin images from one single frame using 
 
the inherent spatial redundancy of SR images. Specifically, we assem-
bled every four pixels (as follows) in an SR image as one unit: 
 
and then we applied a special diagonal binning operation on these four 
pixels to create two new pixels for the N2N data pair. We averaged the 
two diagonal pixels, 
 and 
, as one new pixel; and the other two 
diagonal pixels, 
 and 
, as the other new pixel. Throughout the 
input image, two 2× subsampled images were created. Second, to 
rescale back to their original image size, we adapted a Fourier interpola-
tion method to upsample the resulting image pair by two times. Thus, 
we transformed the two subsampled images into the Fourier domain 
and applied zero padding to extend the borders in the Fourier space 
by half of the original sizes in each direction. Then, the inverse Fourier- 
transformed image was twice the size of the input image, identical to 
the original SR image. Additionally, to prevent boundary artifacts, we 
extended the image (in spatial domain) with mirror-symmetric 
half-copies before the Fourier padding, and the resulting image was 
Fourier upsampled and then cropped back to the original size.
Third, originally in N2N, the infinite training set can execute the 
average of noise to its true zero-mean result. However, in the case of 
limited data for training, there would be inevitable differences between 
these means across the different realizations of noise. Thus, the insuf-
ficient training set will introduce large predictive uncertainty, inher-
ently resulting in limited denoising performance. To minimize this 
uncertainty, we designed a self-constrained learning process while 
considering the identical underlying content from the noisy data pair. 
Specifically, the noisy data pair were successively and individually fed 
into the network, and the two resulting predictions were calculated by 
the following loss function to execute the training stage.
1 =
1
2 + λ ‖ ̃
x1 −x2‖1 +
1
2 + λ ‖ ̃
x2 −x1‖1 +
λ
2 + λ ‖ ̃
x1 −
̃
x2‖1,
where ̃
x represents the network outcome of the input x, and ‖‖1 refers 
to the l1 norm. The first two terms are the conventional N2N loss. 
Because the corresponding two denoised results should have no dif-
ferences, we included the last term as our self-constrained loss with 
constraint weight λ to enforce the consistency of the predictions. 2 + λ 
is a normalization factor that ensures the gradient magnitude remains 
within a stable range when adjusting the weight of λ.
Patch2Patch data augmentation. We developed a data augmentation 
pipeline, namely Patch2Patch, using multidimensional random patch 
transformations to increase the data amount, which is adaptable to 
both 2D (xy) and 3D (xyz) datasets (Extended Data Fig. 1b). Within the 
Patch2Patch framework, there are three modes for augmentation along 
the temporal axis, in a single image and between different experiments. 
These designs will effectively increase variations of noise realizations 
and structural features without altering the intrinsic characteristics 
of data. Mode 1: For image/volume dataset with additional temporal 
dimension (xy-t/xyz-t), Patch2Patch defaults to create augmentation 
along the temporal dimension. The randomly chosen patches at time 
point A (stack A, image A) are exchanged with the patches from ran-
domly selected time point B (stack A, image B) at the same locations. 
Mode 2: For the augmentation of a single image/volume, two distinct 
patches at different positions are randomly swapped. Mode 3: For 
augmentation between different experiments, the randomly chosen 
patches at experiment A (image A) are exchanged with the patches 
from randomly selected experiment B (image B) at randomly picked 
locations. To further increase the data amounts, we also randomly 
applied one of six geometric transformations for each pair of patches: 
no transformation, horizontal flip, vertical flip, 90° rotation to the left, 
90° rotation to the right and 180° rotation.
Network architecture. Because our SN2N can be applied on any 
DNNs, we simply adopted the widely recognized U-Net37 architecture 
to showcase the strengths of SN2N (Extended Data Fig. 1a). The 2D 
U-Net architecture consists of a 2D encoder module (contracting path), 
a 2D decoder module (expanding path) and four skip connections 
bridging the encoder and decoder. Both the 2D encoder and decoder 
modules are structured into four distinct blocks. Each encoder block 
is equipped with two 3 × 3 convolutional layers, followed by a leaky 
rectified linear unit and a 2 × 2 max pooling operation with a stride of 
two in both dimensions. Conversely, each decoder block contains two 
3 × 3 convolutional layers, followed by a leaky rectified linear unit and 
a 2D bilinear interpolation. Batch normalization60 is integrated after 
each convolutional layer. The skip connections serve to concatenate 
low-level and high-level feature maps, enhancing the preservation of 
spatial information.
For denoising tasks involving 3D datasets (xyz), we directly shifted 
the U-Net from the 2D version to its 3D extension to better leverage the 
axial spatial information. With all its internal operations tailored to a 
3D framework, the 2D U-Net was transformed into the 3D U-Net. In this 
work, we changed the convolution operations from 3 × 3 to 3 × 3 × 3, the 
maximum pooling from 2 × 2 to 2 × 2 × 2, and the interpolation opera-
tions from 2D bilinear to 3D trilinear interpolation.
Learning and inference processes. Regarding the training data gen-
eration, the percentile image normalization is first applied before 
training to remove the baseline background and moderate the large 
intensity gap between the bright and dim fluorescence signals, which 
is defined for an image x as:
N(x; Ilow, Ihigh) =
x −perc(x, Ilow)
perc(x, Ihigh) −perc(x, Ilow) ,
where perc(x, I) is the I-th percentile of all pixel values of x, and Ilow and 
Ihigh represent the lowest and highest values, respectively. Notably, for 
some data under ultralow-SNR conditions, a wavelet-based background 
subtraction12 was executed before this percentile normalization. In this 
work, the Ilow and Ihigh were assigned as 0% and 99.999%, respectively, in 
most applications. Specially, for the ultralow-SNR data in Fig. 3c with 
ultrahigh baseline signal and a number of hot pixels, we set Ilow and Ihigh 
as 20% and 99.9%, respectively.
For the small dataset, we first executed the Patch2Patch pre- 
augmentation pipeline on the raw dataset to enlarge the training set 
(step. 1; Extended Data Fig. 1). Then, a sliding window approach was 
used to generate small patches (128 × 128 or 128 × 128 × 16 tiles by 
default) suitable as network input for training (step. 1; Extended Data 
Fig. 1). The interval of the sliding window was customizable to adjust 
different image processing requirements (64 pixels by default). Dur-
ing this step, a background patch rejection was performed on the fly, 
in which the patches with averaged intensity twice lower than that of 
the entire image/volume were filtered. For these small patches, the 
spatial diagonal resampling strategy was applied to produce pairs of 
twice smaller subimages, each sharing identical content but different 


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
noise realizations (step. 2; Extended Data Fig. 1). After that, the Fourier 
upsampling was used to create paired SN2N data to match the original 
structural scale. Additionally, the conventional data augmentation 
strategies such as rotation and flipping could be included to further 
increase the network generalization (step. 3; Extended Data Fig. 1). 
Finally, the self-constrained learning process utilized the basic U-Net 
network and selected either the 2D U-Net or the 3D U-Net based on the 
input data dimensions (step. 4; Extended Data Fig. 1).
We set the constraint weight λ as 1 routinely in most of our experi-
ments. The models were optimized utilizing the Adam optimizer61 with 
a learning rate of 2 × 10−4, and the first-moment and second-moment 
estimates were regulated with exponential decay rates of 0.5 and 0.999, 
respectively. For every training iteration, the batch size is set to the 
maximum number reaching the graphics processing unit memory 
limit, in which the training stage was conducted using NVIDIA GeForce 
RTX 4090 graphics processing units. Finally, in the inference process, 
the raw data after the percentile normalization are directly fed into 
the trained SN2N network for predictive analytics. For input data sizes 
that exceed the memory limit (most of the volumetric data), the inputs 
are automatically cropped into dozens of subvolumes and fed into 
the SN2N network, and the predictions are stitched back together to 
generate the final denoised results.
Integrations of the SN2N framework with SR reconstructions
SN2N-enhanced RL deconvolution. We have integrated RL decon-
volution45,46 with our SN2N framework, namely RL-SN2N (Fig. 3a). This 
approach leverages the contrast and resolution enhancement capabili-
ties of RL deconvolution while mitigating the artifacts that arise from 
deconvolution under low-SNR conditions. To preserve the stochastic 
independence of noise at the pixel level, we perform RL deconvolution 
after the initial self-supervised data generation. We used the acceler-
ated RL deconvolution:
d j+1 = d j ⋅(hT ⋅
g
h⋅d j )
v j = d j+1 −y j
α j+1 =
∑v j⋅v j−1
∑v j−1⋅v j−1
y j+1 = d j+1 + α j+1 ⋅(d j+1 −d j)
,
where y n + 1 is the image after n + 1 iterations; g is the input image; and 
h is the PSF. The d and v are the intermediate variables to help the 
acceleration process. The adaptive acceleration factor α was intro-
duced by Biggs & Andrews62, representing the length of an iteration 
step, which can be estimated directly from experimental results. The 
RL deconvolution was executed by the corresponding theoretically 
calculated 2D/3D PSFs by the Gaussian kernel approximation, and the 
iterations were usually selected by ~10–15 times. During the inference 
phase, the trained RL-SN2N model is fed with the directly deconvolved 
image/volume.
SN2N-enhanced SOFI. We integrated our SN2N solution into the SOFI 
reconstruction pipeline (SOFI-SN2N; Fig. 6a) to increase its imaging 
efficiency30. After the acquisition of the image sequence, we calculated 
the nth order of auto-correlation cumulants, that is, the 2nd-, 3rd- and 
4th-order SOFI, on each pixel along time. Then, we performed the 
self-supervised data generation and RL deconvolution successively. 
To match the improved spatial resolution, in SOFI-SN2N, we applied 4× 
Fourier upsampling on the diagonal resampled data pair rather than 
the routinely used 2× upsampling. Finally, after RL deconvolution, we 
used an intensity linearization step by taking the n-th root directly 
to minimize the nonlinear effects of SOFI. In the inference stage, the 
trained SOFI-SN2N model is fed with the SR image reconstructed by the 
nth order SOFI followed by 2× Fourier interpolation before RL decon-
volution and intensity linearization.
SN2N-enhanced SIM. We designed two strategies for artifact removal 
of SR-SIM images. Mostly, we first performed the self-supervised data 
generation to every modulated raw frame (nine frames) for creating 
the twin SIM sequences. After that, we individually reconstructed 
the paired SR-SIM images with the HiFi-SIM54 pipeline. Finally, these 
two resulting SIM images were considered as the training set of our 
SIM-SN2N. In the test stage, the ordinary SIM reconstruction was 
directly inputted into the SIM-SN2N network.
Beyond that, the self-supervised data generation might influence 
the parameter estimation, especially under ultralow-SNR conditions 
for ultrafast imaging, leading to failed reconstruction. On the other 
hand, the SIM reconstruction brings the inherent artificial upsampling 
(twofold), and hence the adjacent pixels are highly correlated. To create 
independent data pairs from SIM reconstruction, we performed the 
resampling step within 3 × 3 pixels (Extended Data Fig. 8). Specifically, 
we assembled every nine pixels as one unit: 
and then we applied a special binning operation on these nine pixels 
to create two new pixels for the N2N data pair. We averaged the four 
corner pixels, 
, 
, 
 and 
, as one new pixel; and the other four 
intermediate pixels, 
, 
, 
 and 
, as the other new pixel. Through-
out the input image, two 3× subsampled images were created. Followed 
by a 3× Fourier upsampling, we finalized the required data pair directly 
from an SR-SIM image.
SN2N-empowered automated subcellular segmentation  
and tracking
Subcellular segmentation. We used the Otsu63 method to auto-
matically determine the hard thresholds for identifying the corre-
sponding organelle features in images. Additionally, to achieve more 
precise segmentation, we also utilized pretrained models from several 
learning-based approaches, for example, ERnet49 for ER structures 
and Mitonet50 for Mito shapes. Following the segmentation of ER, we 
eliminated isolated pixels in the binarized masks and then extracted the 
skeleton structures for the network topology construction, in which 
the nodes represent intersections in the skeleton graph, and edges 
represent connections between these nodes. After Mito segmenta-
tion, we computed the connected domains within the binary masks 
and identified the skeletons and key points. These key points were 
subsequently categorized into junctions or end points based on their 
respective topological positions (Extended Data Fig. 5d).
Tracking of Lys. Tracking of Lys was performed using TrackMate 
(7.11.1)64. To characterize their dynamic behaviors, we computed the 
MSD for the trajectories across all time points65. The calculated MSD 
curves are approximated with a power-law function65:
MSD(t) = Γ × t α,
where t represents the time interval, Γ is the proportionality factor that 
relates to both particle motion dynamics and the physical properties of 
the system, and α characterizes the different modes of particle move-
ments66. Then, a logarithmic transformation is applied to the MSD 
formula, followed by a linear regression to estimate α:
log(MSD) = α × log(t) + log(Γ ).
In our analysis, the Lys movements were classified into confined 
motion (α < 0.85), free diffusion (0.85 ≤ α ≤ 1.2) and directed move-
ment (α > 1.2)66.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Mito diameter. To estimate the Mito diameter at a selected point, we 
drew the tangent line of the nearest Mito contour. Using the perpen-
dicular line of this tangent line, we obtained the other intersected point 
to the opposite Mito contour. Finally, we approximated the diameter 
by measuring the distance between the initially selected point and this 
intersection point67.
ER–Lys distance. We identified the edges of ER tubules from the 
ERnet-generated masks, and the distances were calculated based on 
the centroid coordinates of Lys and the most adjacent ER edges. Then, 
the final ER–Lys distances were quantified by averaging the distances 
across the entire trajectory.
ER–Mito distance. The ER–Mito contact level was quantified by the 
minimum distance from each Mito to the edges of ER tubules, in which 
the distances of 0 were distributed as ‘contacted’ and larger than 0 as 
‘not contacted’. The Mito diameter was estimated at the point with the 
closest ER–Mito distance.
Lys–Mito interactions. We calculated the MOC to examine the Lys–
Mito interactions. The masks of Mito and Lys were convolved with the 
PSF of the SD-SIM system, and the MOC values were calculated from 
the resulting masks at each time point. Subsequently, we calculated 
the ratio of the mean value and STD of the MOC curve and obtained 
an empirical threshold of 0.26, in which the associated MOC values 
larger than this threshold were identified as functional sites in spe-
cific instances of Lys–Mito contact24 (Extended Data Fig. 5j). With 
MOC values below the threshold, intersection points on the Mito edge 
were identified from the connecting line of Lys and Mito centroids. 
We then used it as the selected point for the Mito diameter estima-
tion. For events of MOC values higher than the threshold, we drew the 
perpendicular line of the connecting line from the two intersection 
points between the edges of Lys and Mito, and the intersection of this 
perpendicular line and the Mito edge was picked as the selected point 
for the Mito diameter estimation.
Performance metrics
Simulations of microtubule filaments. To perform benchmarks 
with simulated ground truth, we created the microtubule-like struc-
tures. Specifically, we utilized the ‘insertShape’ (MATLAB function) 
to sketch multiple lines with random orientations on a blank canvas 
of 4,096 × 4,096 pixels (Supplementary Fig. 1). To simulate the effect 
of incomplete labeling observed in practical experiments, we applied 
a small Gaussian mask (σ as 2 pixels), introducing random notches 
along the lines (indicated by red circles in Supplementary Fig. 1). Then, 
to mimic the curviness of cytoskeleton, an elastic deformation was 
applied to bend the straight lines to curved lines in both x and y dimen-
sions, in which the resulting image has a pixel size of 16.25 nm. The syn-
thetic structures were convolved with a 150-nm PSF and downsampled 
by twofold, for 2,048 × 2,048 pixels with a 32.5-nm pixel size (simulated 
blurred ground truth). Finally, we assigned a number of photons to the 
synthetic structures with intensity emissions of 100 a.u. (level 3), 50 a.u. 
(level 2), 25 a.u. (level 1), 19 a.u., 13 a.u., 7 a.u. and 1 a.u. Then, the Poisson 
noise injection is followed by an addition of the Gaussian readout noise:
y = Poisson(photon × x) + n,
where y is the final noisy image, and x is the downsampled image. ‘Pois-
son’ denotes the Poisson noise injection; ‘photon’ is the photon num-
ber; and n represents the Gaussian noise with a fixed variance value.
Pixel-wise metrics. In this work, the SSIM40, PSNR and RMSE were 
used as metrics to evaluate the pixel-level consistency between recon-
structed images and GT images. For the fixed samples, we directly 
enlarged the exposure time to acquire the high-SNR data. To remove 
potential small baseline background and noise, the GT images were 
created from these high-SNR images by subtracting a constant back-
ground value and subsequently filtering a small Gaussian kernel. To 
qualify data and model uncertainties, we adopted the STD using the ten 
predictions from ten repetitively collected inputs or ten repetitively 
trained models:
STD = ∑
x,y
√
√
√
∑
n
i=1 (Ii −⟨I ⟩1∼n)
2
n −1
,
where n is the sequence length (default as 10); Ii(x, y) represents the 
intensity of the ith image in the sequence, and ⟨I(x, y)⟩1∼n is the averaged 
intensity of the sequence.
FRC resolution. The calculation of FRC resolution requires two inde-
pendent frames of identical contents under the same imaging condi-
tions39. In case of confocal, SD-SIM and STED imaging, we repetitively 
acquired the same content twice. For SOFI, these two frames were 
generated by splitting the raw image sequence into two image subsets, 
for example, the first 20 frames and the last 20 frames, and reconstruct-
ing them independently.
LRQ. To quantitatively evaluate the reconstruction quality of parallel 
lines, we used the LRQ12 metric:
LRQ =
2 × avg(region0)
avg(region1) + avg(region2) .
The ‘region1’, ‘region2’ and ‘region0’ represent the parallel lines and 
the region in between, and ‘avg’ indicates the mean intensity of pixels 
within the corresponding area. To avoid overconfident determination, 
we used a threshold of 0.2 to ascertain the successful separation of 
parallel lines.
Prediction uncertainty estimation. An optimized denoising algorithm 
can be characterized by minimal data and model uncertainties. Data 
uncertainty indicates the algorithm’s ability to efficiently reduce noise 
(minimize errors), whereas model uncertainty reflects the model’s 
efficiency in utilizing data (sustain performance under limited or 
varying data conditions)68. To estimate data uncertainty, we collected 
ten independent frames of identical contents under the same imag-
ing conditions and fed them to the trained SN2N network. The STD of 
the ten resulting predictions was calculated as the data uncertainty. 
Regarding the model uncertainty, we repetitively trained the DNNs 
ten times and inputted the same data into these ten models. The STD 
of the resulting predictions served as a measure of model uncertainty.
FWHM measurements. The FWHM values were estimated from the 
Gaussian fittings of the manually picked intensity profiles. Particularly, 
the profiles and values plotted in Extended Data Fig. 4a were auto-
matically created by using the LuckyProfiler ImageJ plugin69, which 
can autonomously identify and quantify the optimal FWHM locations 
within images. It enabled us to select the necessary regions for FWHM 
calculations and apply Gaussian fitting algorithms.
Compared methods
Supervised learning. For the fixed-sample experiments, we man-
ually created the GT by enlarging the exposure time to acquire the 
high-SNR data, which served as labels. To remove potential low baseline 
background and noise, the GT images were then created from these 
high-SNR images by subtracting a constant background value and 
subsequently filtering a small Gaussian kernel. The network configura-
tions and training hyperparameters of the ‘supervised learning’ used 
in this work were identical to those of SN2N.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Other self-supervised methods. We evaluated the denoising perfor-
mance of the simulation dataset (Fig. 1b and Supplementary Fig. 5a) and 
multicolor SD-SIM live-cell imaging dataset (Supplementary Fig. 6a) 
against five and eight self-supervised denoisers, respectively. The 
tested denoisers include N2V19, PPN2V42, S2S43, R2R44, N2F33, DeepCAD22, 
DeepSeMi21 and SRDTrans34. In particular, for the ultralow-SNR SD-SIM 
live-cell data, we applied percentile normalization before the data gen-
eration step to remove the strong baseline background, with the nor-
malization bounds set to a minimum of 20% and a maximum of 99.9%.
N2V. To control variables and enhance the N2V denoising effect, we 
utilized the data generation code from N2V with a neighborhood radius 
of 5, combined with the SN2N network architecture. We kept other train-
ing settings consistent with SN2N, featuring a learning rate of 2 × 10–4, 
50 epochs, a batch size of 32 and a patch size of 128.
PPN2V. For PPN2V, we used the default settings as published: the noise 
model set to the Gaussian mixture model, a learning rate of 1 × 10–5, 200 
epochs, 50 iterations per epoch, a batch size of 4 and a patch size of 100.
S2S. For S2S, we used the default published settings of 150,000 itera-
tions and a learning rate of 1 × 10–4. According to the final predicted 
results, the optimal iteration numbers for the simulation dataset and 
SD-SIM live-cell dataset were 200 and 2,000, respectively.
R2R. For R2R, we used the published settings of a 1 × 10–3 learning rate 
and 50 epochs. We optimized the denoising performance by reducing 
the batch size from the default 128 to 32. For the simulated microtubule 
dataset, the noise level (noise STD) is set as 25, and for SD-SIM live-cell 
data experiments, it is set as 50.
DeepCAD. For DeepCAD, the only modification was to increase the 
batch size from 1 to 32. The learning rate was set as 1 × 10–3, and the 
number of iterations was set at 50 by default.
N2F. For N2F, the learning rate is set as 1 × 10–3. The network training 
uses a custom loop that includes an early stopping mechanism, which 
halts training when there is no further improvement in PSNR over a 
certain period.
DeepSemi. For DeepSemi, we use the default published settings of a 
1 × 10−4 learning rate, 100 epochs and a batch size of 2.
SRDTrans. For SRDTrans, we adhered to the published settings of a 
1 × 10−4 learning rate, 30 epochs and a batch size of 2.
SD-SIM setup
We used two commercial SD-SIM systems to capture SR confocal images. 
We validated the denoising performance using a commercial fluorescent 
sample (the Argo-SIM slide, Argolight) with GT patterns consisting of 
fluorescing double-line pairs (spacing from 0 nm to 390 nm, λex = 300–
550 nm; http://argolight.com/products/argo-sim/) under the SpinSR10 
system and conducted live-cell imaging tests using the Live-SR system.
SpinSR10 system. The SpinSR10 system is a commercial SD-SIM 
system (SpinSR10, Olympus) equipped with a wide-field objective 
(×100/1.49 oil, APON, Olympus) and an sCMOS camera (ORCA Fusion, 
Hamamatsu). Four laser beams of 405 nm, 488 nm, 561 nm and 640 nm 
were combined with the SD-SIM. The detection optical path adopted a 
further ×3.2 magnification, and the total magnification was ×320. We 
captured SD-SIM images in its SoRa (SR) mode and collected confocal 
images by switching to its conventional SD-confocal mode.
Live-SR system. The Live-SR SD-SIM system is based on an inverted 
fluorescence microscope (IX81, Olympus) equipped with a wide-field 
objective (×100/1.3 oil, Olympus), a scanning confocal system (CSU-X1, 
Yokogawa) and a Live-SR module (GATACA Systems). Four laser beams 
of 405 nm, 488 nm, 561 nm and 647 nm were combined with the SD-SIM. 
The images were captured by either an sCMOS camera (C14440-20UP, 
Hamamatsu) or an EMCCD camera (iXon3 897, Andor).
STED setup
We acquired STED images from two commercial STED systems.
Abberior STED. Long-term activities of mitochondrial cristae (PKMO53 
labeled) were recorded by a commercial STED microscope (STEDYCON, 
Abberior Instruments) equipped with a wide-field objective (×100/1.45, 
CFI Plan Apochromat Lambda D, Nikon). PKMO was excited at a wave-
length of 561 nm, and STED was performed using a pulsed depletion 
laser at a wavelength of 775 nm with gating of 1 ns to 7 ns and dwell times 
of 10 μs. A pixel size of 25 nm was used for STED recording and each 
line was scanned one or ten times (line accumulations). The pinhole 
was set to 0.7 to 1.0 Airy units.
Leica STED. Other live-cell STED images were obtained using a gated 
STED microscope (TCS SP8 STED 3X, Leica) equipped with a wide-field 
objective (×100/1.40 oil, HCX PL APO, Leica). The excitation and deple-
tion wavelengths were 488 nm and 592 nm for the Sec61β-GFP and 
LifeAct-GFP, 594 nm and 775 nm for the Alexa Fluor 594, 635 nm and 
775 nm for the Alexa Fluor 647, and 651 nm and 775 nm for SiR-tubulin. 
The detection wavelength range was set to 495–571 nm for GFP, 605–
660 nm for Alexa Fluor 594, 657–750 nm for SiR and 649–701 nm for 
Alexa Fluor 647. For comparison, confocal images were acquired in the 
same field before the STED imaging. All images were obtained using 
the LAS AF software (Leica).
SOFI setup
Wide-field microscopy. The three phases of structured illumination 
under the same orientation can be averaged to a uniform wide-field 
illumination. Taking advantage of that, we use the SIM setup described 
above to generate the wide-field images by integrating three frames 
(corresponding to three phases of structured illumination) on the 
camera plane, which enables more flexible cross-validation of SIM and 
SOFI-SN2N results. In other words, we use the identical commercial 
inverted fluorescence microscope (IX83, Olympus) equipped with 
an objective (×100/1.7 HI oil, APON, Olympus) and an sCMOS (Flash 
4.0 V3, Hamamatsu) camera to capture the wide-field images for our 
SOFI-SN2N reconstruction.
SD-confocal microscopy. A commercial SD-confocal microscope 
system (Dragonfly SD system, Andor) based on an inverted fluorescence 
microscope (DMi8, Leica) with a wide-field objective (×100/1.3 oil, Plan 
Apo, Leica) is used in this work. Four laser beams of 405 nm, 488 nm, 
561 nm and 647 nm were combined with the SD-confocal microscope. 
The images were captured by an sCMOS camera (Zyla 4.2 Plus, Andor).
SIM setup
The SIM system is based on a commercial inverted fluores-
cence microscope (IX83, Olympus) equipped with an objective 
(×100/1.49 oil, UAPON, Olympus, for 2D-SIM; ×100/1.7 HI oil, APON, 
Olympus, for TIRF-SIM) and a multiband dichroic mirror (DM, 
ZT405/488/561/640-phase R; Chroma) as described previously34. In 
short, laser light with wavelengths of 488 nm (Sapphire 488LP-200) 
and 561 nm (Sapphire 561LP-200, Coherent) and acoustic optical tun-
able filters (AOTF, AA Opto-Electronic, France) were used to combine, 
switch and adjust the illumination power of the lasers. A collimating 
lens (focal length of 10 mm, Lightpath) was used to couple the lasers 
to a polarization-maintaining, single-mode fiber (QPMJ-3AF3S, Oz 
Optics). The output lasers were then collimated by an objective lens 
(CFI Plan Apochromat Lambda ×2 NA 0.10, Nikon) and diffracted by 


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
the pure phase grating that consisted of a polarizing beam splitter, 
a half-wave plate, and the SLM (3DM-SXGA, ForthDD). The diffrac-
tion beams were then focused by another achromatic lens (AC508-
250, Thorlabs) onto the intermediate pupil plane, where a carefully 
designed stop mask was placed to block the zero-order beam and other 
stray light and to permit passage of ±1 ordered beam pairs only. To max-
imally modulate the illumination pattern while eliminating the switch-
ing time between different excitation polarizations, a home-made 
polarization rotator was placed after the stop mask34. Next, the light 
passed through another lens (AC254-125, Thorlabs) and a tube lens 
(ITL200, Thorlabs) to focus on the back focal plane of the objective 
lens, which interfered with the image plane after passing through the 
objective lens. Emitted fluorescence collected by the same objective 
passed through a dichroic mirror, an emission filter and another tube 
lens. Finally, the emitted fluorescence was split by an image splitter 
(W-VIEW GEMINI, Hamamatsu) before being captured by an sCMOS 
(Flash 4.0 V3, Hamamatsu) camera.
ExM setup
We used the commercial inverted fluorescence microscope (IX83, 
Olympus) equipped with a wide-field objective (×100/1.49 oil, UAPON, 
Olympus) and an sCMOS (Flash 4.0 V3, Hamamatsu) camera to capture 
the wide-field images of the expanded samples.
Imaging sample preparation
Cell maintenance and preparation. COS-7 cells (American Type 
Culture Collection, CRL-1651) and HeLa cells (American Type Cul-
ture Collection, CCL-2) were cultured in high-glucose DMEM (Gibco, 
21063029) supplemented with 10% FBS (Gibco) and 1% 100 mM sodium 
pyruvate solution (Sigma-Aldrich, S8636) in an incubator at 37 °C 
with 5% CO2 until ~75% confluency was reached. MCF7 cells were cul-
tured in MEM (Thermo Fisher, 11095072) supplemented with 10% FBS, 
0.01 mg ml−1 human recombinant insulin (Sigma, I9278), and 1% 100 mM 
sodium pyruvate solution. For the SD-SIM/SD-confocal/STED imag-
ing experiments, 35-mm glass-bottomed dishes (Cellvis, D35-14-1-N) 
were used. For the wide-field and 2D-SIM imaging experiments, cells 
were seeded onto coverslips (H-LAF 10 L glass; reflection index, 1.788; 
diameter, 26 mm; thickness, 0.15 mm; customized) coated with 0.01% 
poly-l-lysine solution (Sigma, P4707) for 10 min and washed twice with 
sterile water before seeding transfected cells.
Live-cell samples for SD-SIM and SIM. To label late endosomes or 
lysosomes, we incubated COS-7 cells in 50 nM LysoTracker Deep Red 
(Thermo Fisher Scientific, L12492) for 45 min and washed them three 
times in PBS before imaging. To label mitochondria, COS-7 cells were 
incubated with 250 nM MitoTracker Green FM (Thermo Fisher Scien-
tific, M7514) and 250 nM MitoTracker Deep Red FM (Thermo Fisher 
Scientific, M22426) in HBSS containing Ca2+ and Mg2+ or no phenol red 
medium (Thermo Fisher Scientific, 14025076) at 37 °C for 15 min before 
being washed three times before imaging. To perform nuclear staining 
on COS-7 cells, SPY650-DNA (Cytoskeleton, CY-SC501) was diluted to 
1:1,000 in PBS for ~1 h and washed three times in PBS.
To label cells with genetic indicators, COS-7 cells were transfected 
with LifeAct-EGFP/LAMP1-EGFP/LAMP1–mCherry/Tom20–mCherry/
Sec61β-EGFP/Golgi-BFP/mGold-Mito-N-7/DsRed-ER70. The transfec-
tions were executed using Lipofectamine 2000 (Thermo Fisher Sci-
entific, 11668019) according to the manufacturer’s instructions. After 
transfection, cells were plated in precoated coverslips. Live cells were 
imaged in a complete cell culture medium containing no phenol red 
in a 37 °C live-cell imaging system. For the calcium lantern imaging in 
SD-SIM, the calcium signal was stimulated with a micropipette contain-
ing 10 μmol l−1 5′-ATP-Na2 solutions (Sigma-Aldrich, A1852)56.
Samples for STED imaging. To label the ER-tubule/actin/microtubule 
in live cells, COS-7 cells were either transfected with Sec61β-EGFP/
LifeAct-EGFP, or incubated with SiR-Tubulin (Cytoskeleton, CY-SC002) 
or PKMO53 for ~20 min without washing before imaging.
Immunofluorescence for SOFI. The COS-7 cells were grown in 35-mm 
glass-bottomed dishes overnight and rinsed with PBS, then immedi-
ately fixed with prewarmed 4% paraformaldehyde (Santa Cruz Biotech-
nology, sc-281692) for 10 min. After three washes with PBS, cells were 
permeabilized with 0.1% Triton X-100 (Sigma-Aldrich, X-100) in PBS for 
10 min. Cells were blocked in 5% BSA/PBS for 1 h at room temperature 
(RT). Mouse anti-Tubulin DM1a (Sigma, T6199) was diluted to 1:100 and 
stained cells in 2.5% BSA/PBS blocking solution for 2 h at RT. The cells 
were then washed with PBS five times for 10 min per wash and stained 
with biotin-XX goat anti-mouse IgG antibody (Invitrogen, B2763). The 
cells were then washed with PBS five times for 10 min per wash and 
stained with QD525 streptavidin conjugate (Invitrogen, Q10143MP) 
for 60 min. Finally, cells were washed five times with PBS and imaged.
Live-cell samples for SOFI. To label the mitochondria in live cells, 
COS-7 cells were transfected with Skylan-S-TOM20 (ref. 71). The trans-
fections were executed using Lipofectamine 2000 (Thermo Fisher 
Scientific, 11668019) according to the manufacturer’s instructions. 
After transfection, cells were plated in glass-bottomed dishes. The 
Skylan-S was under sequential illumination with a 405-nm laser (low 
power) when imaging. In addition, live cells were imaged in complete 
cell culture medium containing no phenol red in a 37 °C live-cell 
imaging system.
Sample preparation for ExM
Sample expansion. The sample expansion was performed as previ-
ously described29,72. The labeled cells were incubated with 0.1 mg ml−1 
of Acryloyl-X (AcX, Thermo, A20770) diluted in PBS overnight at RT 
and washed three times with PBS. To prepare the gelation solution, 
freshly prepared 10% (wt/wt) N,N,N′,N′-tetramethylethylenediamine 
(Sigma, T7024) and 10% (wt/wt) ammonium persulfate (Sigma, 
A3678) were added to the monomer solution (1× PBS, 2 M sodium 
chloride, 2.5% (wt/vol) acrylamide (Sigma, A9099), 0.15% (wt/vol) 
N,N′-methylenebisacrylamide (Sigma, M7279) and 8.625% (wt/vol) 
sodium acrylate (Sigma, 408220)) to a final concentration of 0.2% 
(wt/wt) each. Next, the cells were embedded with the gelation solu-
tion first for 5 min at 4 °C, and then for 1 h at 37 °C in a humidified 
incubator. The gels were immersed into the digestion buffer (50 mM 
Tris, 1 mM EDTA, 0.1% (vol/vol) Triton X-100, and 0.8 M guanidine HCl, 
pH 8.0) containing 8 units per ml proteinase K (NEB, P8107S) at 37 °C 
for 4 h, and then placed into double-distilled water to expand. Water 
was changed 4–5 times until the expansion process reached a plateau. 
 
By determining the gel sizes of before and after the expansion, we 
quantified the expansion factor to be 4.5 times. The gels were immo-
bilized on poly-d-lysine-coated cover glass with a thickness of no. 1.5 
for further imaging.
α-tubulin immunostaining. COS-7 cells were seeded in a Lab-Tek 
II chamber slide (Nunc, 154534). Cells were firstly extracted in the 
cytoskeleton extraction buffer (0.2% (vol/vol) Triton X-100, 0.1 M PIPES, 
1 mM EGTA, and 1 mM magnesium chloride, pH 7.0) for 1 min at RT. Next, 
the extracted cells were fixed with 3% (w/vol) formaldehyde and 0.1% 
(vol/vol) glutaraldehyde for 15 min, reduced with 0.1% (wt/vol) sodium 
borohydride in PBS for 7 min, and washed three times with 100 mM 
glycine. Then, the cells were permeabilized with 0.1% (vol/vol) Triton 
X-100 for 15 min, and blocked with 5% (wt/vol) BSA in 0.1% (vol/vol) 
Tween 20 for 30 min. For antibody staining, the cells were incubated 
with monoclonal rabbit anti-α-tubulin antibody (EP1332Y, 1:250 dilu-
tion, Abcam, ab52866) in antibody dilution buffer (2.5% (wt/vol) BSA 
in 0.1% (vol/vol) Tween 20) overnight at 4 °C, washed three times with 
0.1% (vol/vol) Tween 20, incubated with Alexa Fluor 488-conjugated 
F(ab')2-goat anti-rabbit secondary antibody (1:1,000 dilution, Thermo, 


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
A11070) in antibody dilution buffer for 2 h at RT and washed three times 
with 0.1% (vol/vol) Tween 20.
Sec61β-GFP transfection. COS-7 cells were seeded in a Lab-Tek II cham-
ber slide (Nunc, 154534) and cultured to reach around 50% confluence. 
For transient transfection of Sec61β-GFP in a single well, 500 ng plasmid 
and 1 μl of X-tremeGENE HP (Roche) were diluted in 20 μl Opti-MEM 
sequentially. The mixture was vortexed, incubated for 15 min at RT and 
applied to cells. Twenty-four hours after transfection, the cells were 
washed three times with PBS and fixed as described in the α-tubulin 
immunostaining experiment.
Open-sourced datasets
STED images with different excitation/depletion laser powers. We 
used the published dataset from ref. 52 for testing our SN2N’s denoising 
performance on STED images. A custom STED system incorporating a 
SPAD array detector with 5 × 5 elements was used for data collection. A 
series of images (Fig. 5a) were collected from living HeLa cells labeled 
with SIR-tubulin under gradually increased STED power (0%, 5%, 8%, 12%, 
16%, 19%, 30%, 50% and 80% depletion laser powers). Another series of 
images (Supplementary Fig. 7) were acquired from α-tubulin-labeled 
fixed HeLa cells with gradually increased STED power (0%, 10%, 20%, 
25%, 30%, 40% and 90% depletion laser powers). With 0% depletion laser 
power, the system was switched to a conventional point-scanning confo-
cal mode. We used the direct sum and the reassigned sum of signals from 
25 elements as confocal/STED and ISM/STED-ISM images, respectively.
BioSR SIM dataset. We used the open-sourced dataset, the BioSR 
dataset from ref. 55, for evaluating the denoising performance on SIM 
images. The clathrin-coated pits, microtubules and ER data under 
different noise levels were used as SIM images of high-, medium- and 
low-SNR conditions in this work.
Image rendering and processing
We used the ‘biop-12colors’ color map to color code the 3D volumes in 
Fig. 4b,e,f, Extended Data Figs. 4g and 10a–e and Supplementary Figs. 10 
and 11c. The 3D volumes in Fig. 4g–i and Extended Data Fig. 6a,c,e were 
rendered using the Microscape software (https://www.microscape.
xyz/). All data processing was achieved using Python scripts, MATLAB 
and Fiji/ImageJ. All figures were prepared with MATLAB, Fiji/ImageJ, 
Microsoft Visio and OriginPro.
Reporting summary
Further information on research design is available in the Nature 
Portfolio Reporting Summary linked to this article.
Data availability
We provided two representative datasets from Figs. 1 and 3b available 
at https://github.com/WeisongZhao/SN2N/. All other data that sup-
port the findings of this study are available from the corresponding 
author upon request.
Code availability
Videos were produced with Microsoft PowerPoint and our lightweight 
MATLAB framework, which is available at https://github.com/Wei-
songZhao/img2vid/. The percentile normalization method has been 
written as a Fiji/ImageJ plugin and can be found at https://github.com/
WeisongZhao/percentile_normalization.imagej/. The tutorials and 
the updated version of our SN2N can be found at https://github.com/
WeisongZhao/SN2N/.
References
60.	 Ioffe, S. & Szegedy, C. Batch normalization: accelerating 
deep network training by reducing internal covariate shift. In 
International Conference on Machine Learning, 448–456 (2015).
61.	 Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. 
Preprint at https://arxiv.org/abs/1412.6980 (2014).
62.	 Biggs, D. S. & Andrews, M. Acceleration of iterative image 
restoration algorithms. Appl. Opt. 36, 1766–1775 (1997).
63.	 Otsu, N. A threshold selection method from gray-level 
histograms. IEEE Trans. Syst. 9, 62–66 (1979).
64.	 Ershov, D. et al. TrackMate 7: integrating state-of-the-art 
segmentation algorithms into tracking pipelines. Nat. Methods 19, 
829–832 (2022).
65.	 Qian, H., Sheetz, M. P. & Elson, E. L. Single particle tracking. 
Analysis of diffusion and flow in two-dimensional systems. 
Biophys. J. 60, 910–921 (1991).
66.	 Ba, Q., Raghavan, G., Kiselyov, K. & Yang, G. Whole-cell scale 
dynamic organization of lysosomes revealed by spatial statistical 
analysis. Cell Rep. 23, 3591–3606 (2018).
67.	 Damenti, M., Coceano, G., Pennacchietti, F., Boden, A. & Testa, I.  
STED and parallelized RESOLFT optical nanoscopy of the tubular  
endoplasmic reticulum and its mitochondrial contacts in neuronal 
cells. Neurobiol. Dis. 155, 105361 (2021).
68.	 Zhao, W. et al. Quantitatively mapping local quality of 
super-resolution microscopy by rolling Fourier ring correlation. 
Light Sci. Appl. 12, 298 (2023).
69.	 Li, M. et al. LuckyProfiler: an ImageJ plug-in capable of 
quantifying FWHM resolution easily and effectively for 
super-resolution images. Biomed. Opt. Express 13, 4310–4325 
(2022).
70.	 Lee, J. et al. Versatile phenotype-activated cell sorting. Sci. Adv. 6, 
eabb7438 (2020).
71.	 Zhang, X. et al. Development of a reversibly switchable 
fluorescent protein for super-resolution optical fluctuation 
imaging (SOFI). ACS Nano 9, 2659–2667 (2015).
72.	 Tillberg, P. et al. Protein-retention expansion microscopy of cells 
and tissues labeled using standard fluorescent proteins and 
antibodies. Nat. Biotechnol. 34, 987–992 (2016).
Acknowledgements
We thank the assistance of T. Liu from Z. Chen’s laboratory at the 
Peking university for STED imaging of PKMO-labeled mitochondrial 
cristae. This work was supported by the National Key Research and 
Development Program of China (grant no. 2022YFC3400600 to 
L.C.), the National Natural Science Foundation of China (grant nos. 
32422052 to W.Z., 62305083 to W.Z., T2222009 to H.L., 32227802 to 
L.C., 21927813 to L.C., 81925022 to L.C., 92054301 to L.C., 32301257 
to S.Z., 32071458 to H. M.), the Young Elite Scientists Sponsorship 
Program by China Association for Science and Technology (grant 
no. 2023QNRC001 to W.Z.), and the Heilongjiang Provincial 
Postdoctoral Science Foundation (grant no. LBH-Z22027 to W.Z.), 
the Natural Science Foundation of Heilongjiang Province (grant no. 
YQ2021F013 to H.L.), the Beijing Natural Science Foundation (grant 
no. Z20J00059 to L.C.), the Nanyang Assistant Professorship Start-up 
Grant, and National Research Foundation of Singapore (grant no. 
NRF-CRP29-2022-0003 to G.H.) and the Guangdong Basic and 
Applied Basic Research Foundation (grant no. 2022A1515011683  
to J.H.). L.C. acknowledges support by the High-performance 
Computing Platform of Peking University.
Author contributions
W.Z. conceived the research; L.Q. implemented the corresponding 
software; S.Z., X.Y. and K.W. performed the experiments and collected 
the data; Q.L. analyzed the data and prepared the figures; L.Q. and 
Y.H. prepared the videos; X.L., H.M., G.H., W.C., C.G., J.H., J.T., H.L. 
and L.C. participated in discussions during the development of the 
manuscript; W.Z. and L.Q. wrote the manuscript with input from 
all authors; W.Z., H.L. and L.C. supervised the project. All authors 
participated in the discussions and data interpretation.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Competing interests
L.C., H.L., W.Z. and L.Q. have a pending patent application on the presented 
framework. The remaining authors declare no competing interests.
Additional information
Extended data is available for this paper at  
https://doi.org/10.1038/s41592-024-02400-9.
Supplementary information The online version contains supplementary 
material available at https://doi.org/10.1038/s41592-024-02400-9.
Correspondence and requests for materials should be addressed to 
Weisong Zhao.
Peer review information Nature Methods thanks  
Laurence Pelletier, Yide Zhang, Jiji Chen and the other,  
anonymous, reviewers for their contribution to the peer  
review of this work.
Reprints and permissions information is available at  
www.nature.com/reprints.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 1 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 1 | Workflow of SN2N and network architectures.  
a, Detailed flow diagram of SN2N (Methods). Steps 1-3, self-supervised data 
generation. First, the data pre-augmentation (optional) is performed using  
the Patch2Patch (random patch transformations in multiple dimensions) 
strategy. After that, a sliding window approach is employed to generate small 
patches suitable for input into the network for training. Subsequently, the  
spatial diagonal resampling strategy followed by Fourier upsampling is used  
to create paired SN2N data. Additionally, basic augmentations such as rotation 
and flipping (optional) are applied to the generated data pairs. Step 4:  
self-constrained learning process. SN2N utilizes the classical U-Net network 
and selects either the 2D U-Net or 3D U-Net based on the input data dimensions. 
The generated paired images are considered as one training example, and the 
resulting two predictions are used to calculate the loss for back propagation.  
b, Patch2Patch (P2P) pipeline (Methods). It includes three available modes  
for augmentation along the temporal axis, in a single image, and between 
different experiments.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 2 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 2 | Systematical tests of different components of SN2N. 
a, SN2N denoising results under different pixel sizes with same resolution. From 
top to bottom: Raw images, SN2N results, and clean ground truth images. The 
synthetic structures (16.25 nm pixel size) were convolved with a 150 nm size 
PSF and downsampled by 2, 3, 4, 5, 6, 8, and 16 times. SSIM and PSNR values of 
SN2N results are marked on the bottom right corner. b, SSIM (top) and PSNR 
(bottom) values from data under different downsampling rates. c, Ablation 
tests for individual components of SN2N. From left to right: Raw (top)/ground 
truth (bottom), without downsampling process (Raw2Raw, the same noisy 
image as both input and label), with downsampling only, including the Fourier 
up-sampling step, integrating the self-constrained learning process, and 
supplementing our Patch2Patch augmentation. We found that the network 
was not able to execute denoising without the downsampling step, and the 
upsampling step enforced the consistency of the predicted structural scale. 
The self-constrained learning process strengthened the data-efficiency and 
performance, and the addition of Patch2Patch further maximized the data 
efficiency. d, SSIM values of different components in SN2N. e, SN2N denoising 
results under different interpolation methods. From left to right: Raw input, 
SN2N results using data without interpolation, with bilinear interpolation, and 
our Fourier interpolation as training sets, and ground truth image. f, SSIM values 
of SN2N under different interpolation strategies. g, SN2N results under different 
self-constrained regularization weights (values labeled on the top left corners). 
h, Average SSIM values under different self-constrained regularization weights 
(n = 10). i, SN2N denoising results under different photon levels (from several 
hundred to single photon, Methods). j, SSIM values under different photon 
levels. In a, e, g, and i, the models were trained with 50 frames (full data case).  
In c, the models were trained with both 50 frames (full data case) and one 
image (1/50 data case). In a, c, e, and g, the models were trained under noise 
level 1 conditions. Error bars: s.e.m. Experiments were repeated ten times 
independently with similar results; scale bars, 1 µm.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 3 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 3 | Testing results of different noise levels and data 
amounts. a, Denoising results of various methods under three different noise 
levels (Level 3, Level 2, and Level 1, from top to bottom) using the full training set. 
From left to right: Raw input, denoising results of PURE, ACsN, N2V, supervised 
learning ('Supervised'), SN2N without constraint ('SN2N w/o c'), and full SN2N. 
b, Quantitative comparisons of the results shown in a using PSNR (left), SSIM 
(middle), and RMSE (left) metrics (n = 10, mearsurements). c, Denoising results 
of learning-based methods using three different amounts (1/50, 1/5, and full 
data, from top to bottom) of training data under Level 1 of noise. d, Quantitative 
comparison of the results shown in c using PSNR (left), SSIM (middle), and RMSE 
(left) metrics (n = 10, mearsurements). k denotes the slope (red lines) of the 
corresponding metric values along the data increment. e-f, Data uncertainty 
(e) and model uncertainty (f) of neural network models trained by different 
data amounts. Average standard derivation (STD) values calculated from ten 
predictions of ten repetitively acquired data or ten repetitively trained models. 
Centerline, medians; limits, 75% and 25%; whiskers, maximum and minimum; 
error bars, s.e.m. Experiments were repeated three times independently with 
similar results; scale bars, 1 µm.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 4 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 4 | Comparisons of SN2N versus RL-SN2N using SD-SIM 
and applying SN2N on SD-confocal microscopy. a, Zoomed-in views (top) and 
FWHM distribution plots (bottom, calculated by LuckyProfiler) of denoising 
results by different methods (c.f., Fig. 2a). b, Comparison of SN2N and RL-SN2N. 
Top: SD-SIM (left) and its RL result (right); Bottom SN2N result (left) and RL-SN2N 
result (right). c, LRQ values of results in b. d, SN2N denoising results (right) of SD 
confocal image (left) recording the Argo-SIM slide. e, LRQ values of results in d. 
f, Comparisons of SN2N with 2D U-Net (left), SN2N with 3D U-Net (middle), and 
RL-SN2N with 3D U-Net (right) of volumetric data (c.f., Fig. 3b). g, Magnified views 
and their xz and yz cross-sections from white boxed regions in f. Experiments 
were repeated three times independently with similar results; scale bars, 500 nm 
(a, b, d); 1 µm (f, g).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 5 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 5 | SN2N-empowered automated subcellular segmentation 
and tracking. a, Workflow. Step 1, RL deconvolution; Step 2, RL-SN2N inference; 
Step 3, segmentation; Step 4, tracking. Step 5, extraction of motion features. Step 
6, classification. Step 7, topology graph construction. Step 8, specific downstream 
analysis. b, A representative example for dual-color SR imaging of mitochondria 
(Mito, green) and ER (magenta) labeled with Tom20–mCherry and Sec61β-EGFP in 
live COS-7 cells under raw SD-SIM (left) and RL-SN2N (right). c, The white box in b is 
enlarged and shown at seven time points under different configurations. From top 
to bottom: Raw SD-SIM, dual-color RL-SN2N, single-channel (Mito) RL-SN2N, RL SD-
SIM segmentation, and RL-SN2N segmentation results. The yellow and white arrows 
indicate the mitochondrial fission and before fission, respectively. d, e, Results of 
Mito (d) and ER (e) segmentations (first row) using the Otsu hard threshold (first 
column) and Mitonet/ERnet (second column) and their skeletonizations (second 
row) under SD-SIM (left) and RL-SN2N (right). f, Otsu segmentation results for  
Lys (red) and GA (blue) under SDSIM (left) and RL-SN2N (right). g, A representative 
4-color segmentation result under RL-SN2N. h, Spatial distribution of Lys assigned 
with different motion behaviors. i, Distribution of estimated α values of Lys versus 
their temporal average of minimum distances to Mito (n = 46). j, Distribution of the 
Lys-Mito MOCs’ standard deviation (S.D.) versus their mean values. k, Illustrations 
of the MSD curves for different motion behaviors of Lys. Curves are color-coded 
by the corresponding ER-Lys distances. Experiments were repeated three times 
independently with similar results; scale bars, 2 μm (b, c, e).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 6 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 6 | RL-SN2N can suppress noise in undersampled 
data from EMCCD SD-SIM. a, 3D renderings of live COS-7 cells labeled with 
Hoechst (green) and MitoTracker Deep Red (magenta) under raw SD-SIM 
equipped with an EMCCD camera (94 nm pixel size versus <150 nm resolution). 
b, Representative lateral slices from volume in a. c, RL-SN2N results of a with 
additional 2× upsampling before RL deconvolution (47 nm pixel size).  
d, Representative lateral slices from volume in c. e, Another RL-SN2N time point. 
f, Representative lateral slices from volume in e. Experiments were repeated 
three times independently with similar results; scale bars, 5 μm.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 7 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 7 | Full data of SOFI-SN2N results and SN2N-assisted 
expansion microscopy (ExM-SN2N). a, The whole field-of-views of the  
wide-field (top left), 20-frames 2nd order SOFI (2nd SOFI 20f, top right), 2D-SIM 
(bottom left), and 20-frame 2nd order SOFI-SN2N (bottom right) images  
(c.f., Fig. 6b). b, SN2N results of 2nd, 3rd, and 4th orders SOFI (from left to right) 
using 20, 50, 100, 200, 500, 1000 frames (from top to bottom) (c.f., Fig. 6e). 
c-e, Average SSIM values of 2nd (c), 3rd (d), and 4th (e) SOFI-SN2N results (n = 5, 
mearsurements). (f) Comparison of temporal and spatial sampling methods. 
From left to right: SOFI reconstruction, SN2N result using temporal sampling 
(the first 20 frames vs. the second 20 frames), and SN2N result using spatial 
sampling. g, A 2 times-expanded (2×, top) and 4-times expanded (4×, bottom) 
COS-7 cell was immunostained with a primary antibody against α-tubulin and a 
second antibody conjugated with Alexa Fluor 488 under wide-field microscopy 
(left) and its SN2N denoised result (right). Signal-to-background ratios (SBR) 
are labeled. h, Magnified views of the white boxed regions in g under ExM (top) 
and SN2N denoised results (bottom). i, Intensity profiles and multiple Gaussian 
fitting of the filaments indicated by the white arrows in h. Numbers represent the 
distances between peaks; a.u., arbitrary units. j, A 4.5-times expanded (4.5×) COS-
7 cell labeled with Sec61β–GFP under wide-field microscopy (left) and its SN2N 
denoised result (right). k, Enlarged regions enclosed by the white box in j seen 
under ExM-4.5× (left) and its SN2N result (right). Centerline, medians; limits, 75% 
and 25%; whiskers, maximum and minimum; error bars, s.e.m. Experiments were 
repeated three times independently with similar results; scale bars, 2 µm (a),  
1 µm (b, h, j, k), and 5 μm (g).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 8 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 8 | SN2N removes random, non-continuous artifacts in 
low-SNR SIM with two strategies. a, Pipeline of SIM-SN2N using raw image 
resampling strategy (Methods). The self-supervised data generation is applied on 
the 9 raw images followed by the SIM reconstruction. b, d, f, clathrin-coated pits 
(CCPs, b), microtubules (d), and ER (f), recorded by SIM under low-SNR condition 
(SIM-L, left) and their SN2N results (right). c, e, g, SIM reconstructions (left) of 
CCPs (c), microtubules (e), and ER (g) under low (L, top), medium (M, middle), 
and high (H, bottom) SNR conditions and their SN2N results (right). SSIM values 
are labeled on the bottom right corners. h, The mitochondrial cristae structures 
in live COS-7 cells labeled with MitoTracker Green under 2D-SIM (bottom left 
boxed region) and SN2N-SIM imaging at the first time point. i, j, Representative 
montages of 11 time points from yellow boxed region in h under 2D-SIM (i) and 
SIM-SN2N (j). k, Workflow of SIM-SN2N using SIM image resampling strategy 
(Methods). After SIM reconstruction, we apply a SIM-specfic self-supervised 
data generation involving 3 × 3 pixels (1 + 3 + 7 + 9 versus 2 + 4 + 6 + 8) resampling 
followed by a 3× Fourier interpolation. l, n, p, CCPs (l), microtubules (n), and 
ER (p), recorded by SIM under low-SNR condition (left) and their SN2N results 
(right). m, o, q, SIM reconstructions (left) of CCPs (m), microtubules (o), and ER 
(q) under low (top), medium (middle), and high (bottom) SNR conditions and 
their SN2N results (right). SSIM values are labeled on the bottom right corners.  
r, A representative living COS-7 cell labeled with LifeAct–EGFP under ultrafast 
TIRF (left), TIRF-SIM (middle), and SIM-SN2N (right). s, t, Enlarged regions 
enclosed by the white box in r, under TIRF-SIM (s) and SIM-SN2N (t). Experiments 
were repeated three times independently with similar results. Scale bars,  
2 μm (b, d, f, h) and 1 μm (c, e, g, j, r, t).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 9 | See next page for caption.


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 9 | SN2N maintains the linear response of Ca2+ transients 
obtained by the SD-SIM. a, A representative live COS-7 cell was transfected with 
GCaMP6s, stimulated with ATP (10 μM). One snapshot under the SD-SIM (left) 
and after the SN2N (right) were shown. b, Magnified views of regions enclosed 
by white boxes 1-4 in a. c, ATP stimulated calcium traces from corresponding 
macrodomains in b. d, Denoising of fast calcium transients using published two-
photon microscopy dataset from ref. 22. From left to right: Low-SNR recording of 
somatic signals, DeepCAD data, SN2N (spatial) denoising results, SN2N (temporal) 
denoising results by replacing the spatial diagonal resampling with the temporally 
interleaved resampling, and the high-SNR data (tenfold imaging SNR). Magnified 
view of white-boxed regions is shown at the bottom row. e, Fluorescence traces 
extracted from the yellow boxed regions in d. The trajectories’ Pearson correlation 
coefficients (r) were labeled at the bottom right corners. Experiments were 
repeated three times independently with similar results. Scale bars, 5 µm (a),  
2 µm (b), and 50 µm (d).


Nature Methods
Article
https://doi.org/10.1038/s41592-024-02400-9
Extended Data Fig. 10 | Generalization of SN2N across different SNR 
conditions and pixel sizes (structural scales). a-f, Testing generalization across 
different SNR conditions. Color-coded 3D distributions and their xz and yz cross-
sections of all mitochondria (labeled with Tom20–mCherry) in a live COS-7 cell 
(c.f., Fig. 3b). a-c, SN2N results (trained with the first volume) of the first volume 
(0 min) (a) and the last volume (2.5 min) (b), and the SN2N prediction of SN2N 
perdition (SN2N2) from the last SD-SIM volume (c). d, e, SN2N results (trained with 
the last SD-SIM volume) of the first (d) and the last (e) volume. f, Zoom-in views 
from white-boxed regions in a-e. First column: 0 min (top) and 2.5 min (bottom) 
SD-SIM; second column: SN2N and SN2N2 (bottom half of bottom) results (trained 
with 0 min SD-SIM volume) of 0 min (top) and 2.5 min (bottom) SD-SIM; SN2N 
results (trained with 2.5 min SD-SIM volume) of 0 min (top) and 2.5 min (bottom) 
SD-SIM. g-k, Testing generalization across different pixel sizes. g, Nuclear pores in 
HeLa cells were labeled with an anti-Mab414 primary antibody and the Alexa594 
secondary antibody and observed under STED and STED-SN2N configurations.  
h, STED images (left) of 20.66 nm pixel size (top), 7.10 nm pixel size (middle),  
and 20.66 nm pixel size subsampled from 7.10 nm (bottom), and their SN2N 
results (right) from model trained by data of 20.66 nm pixel size. i, STED images 
(left) of 7.10 nm pixel size (top), 20.66 nm pixel size (middle), and 7.10 nm pixel  
size Fourier upsampled from 20.66 nm (bottom), and their SN2N results (right) 
from the model trained by data of 7.10 nm pixel size. j, k, Average FWHM values  
of STED (gray) and SN2N results (yellow) from the model trained by data of  
20.66 nm pixel size (j) and 7.10 nm pixel size (k) (n = 5, measurements). Centerline, 
medians; limits, 75% and 25%; whiskers, maximum and minimum; error bars,  
s.e.m. Experiments were repeated three times independently with similar results. 
Scale bars, 5 µm (e), 1 µm (f-h).




α
α




Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1067
nature computational science
https://doi.org/10.1038/s43588-023-00568-2
Article
Spatial redundancy transformer for self-
supervised fluorescence image denoising
Xinyang Li 
 
 1,2,3,9, Xiaowan Hu2,9, Xingye Chen1,3,4,9, Jiaqi Fan2,5, Zhifeng Zhao1,3, 
Jiamin Wu 
 
 1,3,6,7 
, Haoqian Wang 
 
 2,8 
 & Qionghai Dai 
 
 1,3,6,7 
Fluorescence imaging with high signal-to-noise ratios has become 
the foundation of accurate visualization and analysis of biological 
phenomena. However, the inevitable noise poses a formidable challenge 
to imaging sensitivity. Here we provide the spatial redundancy denoising 
transformer (SRDTrans) to remove noise from fluorescence images in 
a self-supervised manner. First, a sampling strategy based on spatial 
redundancy is proposed to extract adjacent orthogonal training pairs, which 
eliminates the dependence on high imaging speed. Second, we designed 
a lightweight spatiotemporal transformer architecture to capture long-
range dependencies and high-resolution features at low computational 
cost. SRDTrans can restore high-frequency information without producing 
oversmoothed structures and distorted fluorescence traces. Finally, we 
demonstrate the state-of-the-art denoising performance of SRDTrans 
on single-molecule localization microscopy and two-photon volumetric 
calcium imaging. SRDTrans does not contain any assumptions about the 
imaging process and the sample, thus can be easily extended to various 
imaging modalities and biological applications.
The rapid development of intravital imaging techniques enables 
researchers to observe biological structures and activities at microm-
eter and even nanometer scales1,2. As an imaging method with great 
prevalence, fluorescence microscopy has contributed to the discov-
ery of a series of new physiological and pathological mechanisms 
due to its high spatiotemporal resolution and molecular specific-
ity3–5. The fundamental goal of fluorescence microscopy is to obtain 
clean and sharp images containing sufficient information about the 
sample, which can guarantee the accuracy of downstream analysis 
and support convincing conclusions. However, limited by multiple 
biophysical and biochemical factors (for example, labeling concen-
tration, fluorophore brightness, phototoxicity, photobleaching and 
so on), fluorescence imaging is conducted in photon-limited condi-
tions and the inherent photon shot noise severely degrades the image 
signal-to-noise ratio (SNR), especially in low-illumination and high- 
speed observations6.
Various methods have been proposed to remove noise from fluo-
rescence images. Conventional denoising algorithms based on numeri-
cal filtering and mathematical optimization suffer from unsatisfactory 
performance and limited applicability7,8. In the past few years, deep 
learning has shown remarkable performance in image denoising9,10. 
After iterative training on a dataset with ground truth (GT), deep neu-
ral networks can learn the mapping between noisy images and their 
clean counterparts. Such a supervised manner depends heavily on 
paired GT images11–15. When observing the activity of living organ-
isms, obtaining pixel-wise registered clean images is a great challenge 
because the sample often undergoes fast dynamics. To alleviate this 
contradiction, some self-supervised methods have been proposed 
Received: 14 June 2023
Accepted: 7 November 2023
Published online: 11 December 2023
 Check for updates
1Department of Automation, Tsinghua University, Beijing, China. 2Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 
China. 3Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China. 4Research Institute for Frontier Science, Beihang University, 
Beijing, China. 5Department of Electronic Engineering, Tsinghua University, Beijing, China. 6Beijing Key Laboratory of Multi-dimension and Multi-scale 
Computational Photography (MMCP), Tsinghua University, Beijing, China. 7IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, 
China. 8The Shenzhen Institute of Future Media Technology, Shenzhen, China. 9These authors contributed equally: Xinyang Li, Xiaowan Hu, Xingye Chen. 
 e-mail: wujiamin@tsinghua.edu.cn; wanghaoqian@tsinghua.edu.cn; qhdai@tsinghua.edu.cn


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1068
Article
https://doi.org/10.1038/s43588-023-00568-2
or vertically to exploit spatial correlations fully and isotropically. 
 
A simplified implementation of sampling is depicted in Fig. 1b. As the 
noise of adjacent pixels is independent while the signals are closely 
correlated, the substack filled by the central pixels can be used as the 
training input and the other two spatially adjacent substacks are used as 
corresponding targets to optimize the network parameters. Compared 
with other methods19,20,22, our sampling strategy is more effective and 
comprehensive in preserving both spatial and temporal information 
(Supplementary Figs. 1 and 2). In the inference stage, low-SNR stacks 
will be fed into pre-trained SRDTrans models without spatial downsam-
pling. To overcome the locality of convolutional kernels, we constructed 
a transformer network to capture endogenous non-local spatial fea-
tures and long-range temporal dependencies using the self-attention 
mechanism (Fig. 1c). The restoration of each pixel can simultaneously 
integrate the temporal information of all frames and the spatial infor-
mation of all pixels, even if they are far from each other. Besides, the 
network does not contain any spatial downsampling module, allowing 
more high-frequency components to flow through the network and 
avoiding the loss of spatial resolution. Furthermore, as the amount 
of data in fluorescence imaging is often very large, sometimes at the 
petabyte scale, the transformer network was designed to be as light-
weight as possible to relieve the computational burden of large-scale 
data processing. Compared with other transformer networks27–30, our 
architecture can achieve the best performance with more than one 
order of magnitude fewer parameters (Supplementary Tables 1 and 2). 
The lightweight architecture of SRDTrans also makes it easy to train a 
good model even with a small amount of training data (for example, 
500 frames, 490 × 490 pixels each frame), relieving the pressure of 
capturing large-scale datasets (Supplementary Fig. 3).
To demonstrate the predominance of our transformer network 
over CNNs, we generated simulated calcium imaging data (Methods 
and Supplementary Fig. 4) and used them to train a 3D U-Net31, as 
well as the transformer of SRDTrans, using the same spatial redun-
dancy sampling strategy. We term the former as spatial redundancy 
denoising CNN (SRDCNN). The visualized feature maps of deep layers 
intuitively show the superiority of SRDTrans in revealing fine-grained 
patterns (Fig. 1d). As features flow through the network, the limited 
receptive field of convolutional kernels makes CNN-based methods 
focus on only rough features while our transformer architecture still 
has a strong perception of sophisticated structures. We also compare 
the denoised images of the two architectures (Fig. 1e). The result of 
SRDCNN is obviously oversmoothed, especially in regions with sharp 
edges, which is a manifestation of spectral bias that the low-frequency 
information is overfitted while the high-frequency information is 
hardly preserved (Supplementary Fig. 5). This deficiency is largely alle-
viated by SRDTrans, and more subcellular structures such as dendritic 
fibers can be restored accurately. The intensity profile deconstructed 
from the SRDTrans denoised image is more consistent with the GT 
(Fig. 1f). Moreover, lacking the ability to capture long-range temporal 
for more applicable and practical denoising in fluorescence imag-
ing6,16–23. Among them, the first kind of methods rely on the similarity 
between adjacent frames6,16–18. But when the sample changes very 
quickly or the imaging speed is too slow, the time-lapse data cannot 
provide enough temporal redundancy. This is a common problem in 
volumetric imaging as the volume rate decreases proportionally to 
the number of imaging planes. The dissimilarity between adjacent 
frames will result in inferior performance and distorted structures and 
fluorescence kinetics. The other kind of methods learn to denoise only 
from spatially adjacent pixels in two-dimensional frames19–23. However, 
without utilizing endogenous temporal correlations, these methods 
perform poorly on time-lapse imaging. Therefore, to achieve better 
denoising performance, the ability to simultaneously extract global 
spatial information and long-range temporal correlations is essential, 
which is lacking in convolutional neural networks (CNNs) because of 
the locality of convolutional kernels24. Moreover, the inherent spectral 
bias makes CNNs tend to fit low-frequency features preferentially while 
ignoring high-frequency features, inevitably producing oversmoothed 
denoising results25.
Here we present the spatial redundancy denoising transformer 
(SRDTrans) to address these dilemmas. On the one hand, a spatial 
redundancy sampling strategy is proposed to extract three-dimen-
sional (3D) training pairs from the original time-lapse data in two 
orthogonal directions. This scheme has no dependence on the similar-
ity between two adjacent frames, so SRDTrans is applicable to very fast 
activities and extremely low imaging speed, which is complementary 
to our previously proposed DeepCAD that leverages temporal redun-
dancy6,18. On the other hand, we designed a lightweight spatiotemporal 
transformer network to fully exploit long-range correlations. The 
optimized feature interaction mechanism allows our model to obtain 
high-resolution features with a small number of parameters. Compared 
with classical CNNs, the proposed SRDTrans has stronger abilities for 
global perception and high-frequency maintenance, enabling the rev-
elation of fine-grained spatiotemporal patterns that were previously 
indiscernible. We demonstrate the superior denoising performance of 
SRDTrans on two representative applications. The first one is single-
molecule localization microscopy (SMLM) with adjacent frames being 
random subsets of fluorophores26. The other one is two-photon calcium 
imaging of large 3D neuronal populations with a volumetric speed as 
low as 0.3 Hz. Extensive qualitative and quantitative results indicate 
that SRDTrans can serve as a fundamental denoising tool for fluores-
cence imaging to observe various cellular and subcellular phenomena.
Results
Principle of SRDTrans
The self-supervised framework of SRDTrans is shown schematically 
in Fig. 1a. For spatial redundancy sampling, spatially adjacent sub-
stacks are sampled by orthogonal masks from the original low-SNR 
image stack. Each target is adjacent to the input stack horizontally 
Fig. 1 | Principle of SRDTrans and performance evaluation. a, Self-supervised 
training strategy of SRDTrans. The original low-SNR stack of H × W × T pixels  
is sampled by orthogonal masks, producing three downsampled substacks 
(input, target 1 and target 2) of H/2 × W/2 × T pixels. The ‘input’ substack is fed  
into the transformer network, and the corresponding output is compared with 
the ‘target’ substacks to calculate the loss function for parameter optimization. 
b, Simplified schematic of spatial redundancy sampling (H = 4, W = 4, T = 1).  
A 4 × 4 patch is split into four 2 × 2 blocks and three adjacent pixels are randomly 
selected in each block. The central pixel (labeled as ‘2’) is horizontally or vertically 
adjacent to the other two pixels (labeled as ‘1’ and ‘3’). c, The architecture of 
the lightweight spatiotemporal transformer. It consists of a temporal encoder 
module, an STB and a temporal decoder module. Each temporal encoder 
compresses the temporal scale (t) of the input by a factor of r (r = 4 in this work) 
using convolution. In the STB module, the input is divided into small patches, 
and different feature maps of the same spatial position are stitched together 
in the patch flattening layer. The position embedding layer records the spatial 
position of each patch so that it can be mapped back after the global interaction 
in the multi-head self-attention layer. The self-attention mechanism can calculate 
the spatiotemporal correlation between all local patches. The output of the STB 
module will be uncompressed to the original temporal scale by the following 
temporal decoder module. d, Visualizing the feature responses in SRDCNN (the 
last layer of STB) and SRDTrans (the last layer of 3D U-Net). SRDCNN represents 
the method that replaces the transformer network in SRDTrans with a 3D U-Net. 
Scale bar, 60 μm. e, Comparing the denoising performance of SRDCNN and 
SRDTrans on simulated calcium imaging data (30 Hz). Scale bars, 40 μm for the 
whole FOV and 10 μm for magnified views. f, Pixel intensity along the red dashed 
line in e. g, Evaluating the ability of SRDCNN and SRDTrans to capture long-range 
dependencies. Models were trained and validated on simulated calcium imaging 
data (30 Hz) of different input temporal scales (T). All values are shown as 
mean ± s.d. (N = 6,000 independent frames).


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1069
Article
https://doi.org/10.1038/s43588-023-00568-2
correlations also drags down the denoising performance of SRDCNN 
on time-lapse imaging data. For SRDTrans, the output SNR continu-
ously grows as the input temporal scale (T) increases (Fig. 1g). A more 
comprehensive evaluation of the influence of input temporal scale 
indicates that SRDTrans can make full use of the information offered 
by temporally distant pixels (Supplementary Figs. 6 and 7). We also 
investigated the generalization ability of the proposed method by 
cross-dataset and cross-modality validation, which shows that training 
on data with the same SNR and imaging modality can obtain the best 
denoising performance (Supplementary Fig. 8). To verify the practi-
cality of SRDTrans at extremely low imaging speed, we compared the 
performance of methods combining different networks and sampling 
schemes on simulated calcium imaging data sampled at 0.3 Hz (Sup-
plementary Video 1). When the similarity between adjacent frames is 
low, using spatial redundancy sampling is more reasonable (Supple-
mentary Table 3). Succinctly, the synergy between spatial redundancy 
sampling and dedicated transformer architecture endows SRDTrans 
with the ability to restore high-resolution structures and fast dynamics.
High-performance SMLM with SRDTrans denoising
Given N detected fluorescence photons, the lower bound of the preci-
sion of SMLM scales to 1/√N  (ref. 26), which is the mathematical 
 
c
STB
c
Patch flattening
Multi-head self-attention
Position embedding
Spatiotemporal transformer block (STB) 
Temporal encoder
=
=
Temporal decoder
a 
b 
Output
Learning
Learning
Target 2
Training pairs
Images
H
W
Spatial redundancy  sampling
T
T
Orthogonal masks
Target 1
Input
T
f 
g 
Raw data
d 
SRDCNN
SRDTrans
Mask 1 Mask 2 Mask 3
T
W
H
Network
1
1
2
1
3
3 2
3
2
1
1
1
2
2
2
2
3
3
3
1
1
1
1
2 2
2 2
3 3
3 3
Mask 1
Mask 2
Mask 3
3
4 × 4 patch
Orthogonal mask
2 × 2 patches
1
1
2 3
1
Raw
SRDCNN
SRDTrans
GT
t
t
r
t
r
t
W
2
H
2
0
5
10
15
0.2
0.4
0.6
0.8
1.0
Normalized intensity (a.u.)
Distance (µm)
Mask ID
*
*
Raw data
GT
SRDCNN
e 
Normalized intensity
1.0
0
Output SNR (dB)
200
T (slices)
600
1,000
24
25
27
28
26
SRDTrans


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1070
Article
https://doi.org/10.1038/s43588-023-00568-2
formula of the shot-noise limit or the well-known standard quantum 
limit32,33. This indicates that the fundamental physical limit of localiza-
tion precision is shot noise. To investigate the benefits that our denois-
ing method can bring to SMLM, we first applied SRDTrans to simulated 
stochastic optical reconstruction microscopy (STORM) data with GT 
for quantitative evaluation34,35. The noise-free single-molecule-emis-
sion images were synthesized by TestSTORM36 and corresponding 
noisy images with different SNRs were then generated by applying 
different levels of mixed Poisson–Gaussian noise (Methods and Sup-
plementary Fig. 9). For the image stack of each SNR, we trained a spe-
cific model for it and then processed it using the model to obtain the 
denoised image stack. Quantitative comparisons of the visualized 
images (Fig. 2a and Supplementary Video 2) and the extracted inten-
sity profiles (Fig. 2b) show that the results of SRDTrans are highly 
consistent with the GT. Over a wide range of imaging SNRs, including 
some extremely low-SNR conditions, SRDTrans can substantially 
improve the image quality evaluated by the SNR at the pixel-intensity 
level and structural similarity (SSIM) at the perception level (Fig. 2c). 
Compared with other self-supervised methods16–22, SRDTrans can 
better preserve the distribution and intensity of emitters owing to its 
strong ability in exploiting high-resolution features and long-range 
dependencies (Supplementary Fig. 10).
Next, we evaluate the improvement of single-molecule localiza-
tion performance with the enhancement of the image SNR. To recon-
struct super-resolved images, we applied ThunderSTORM37, one of the 
most frequently used localization software with an excellent balance 
between accuracy and execution runtime38,39. The image reconstructed 
from the original noisy data is contaminated by noise and contains 
many misidentified molecules (Fig. 2d). By contrast, the reconstructed 
image from SRDTrans denoised images reveals clear and continuous 
cytoskeletal filaments that are not previously recognizable because 
of suppressed localization error and improved resolution (Fig. 2e and 
Supplementary Fig. 11). For better quantitative analysis, we matched 
the detected fluorescent molecules with the GT using the Hungar-
ian algorithm39. From raw images, few fluorescent molecules can be 
detected and most of them are wrongly localized. After SRDTrans 
denoising, almost all molecules can be detected in high agreement 
with the GT (Fig. 2f and Supplementary Fig. 12). Using the Jaccard 
index and root-mean-squared error (r.m.s.e.) as the metrics to quantify 
the proportion of correctly detected molecules and the localization 
accuracy of those detected molecules, respectively, we found that the 
Jaccard index was improved by ~6-fold (85.7 ± 3.51% versus 14.1 ± 2.76%, 
mean ± s.d.) and r.m.s.e. was reduced by ~3.4-fold (24.86 ± 3.24 nm 
versus 85.94 ± 7.76 nm) after denoising (Fig. 2g). From a more com-
prehensive perspective, we further adopted the metric termed effi-
ciency that combines Jaccard index and r.m.s.e.39. The results show that 
SRDTrans improved the efficiency of single-molecule localization from 
−21.54 ± 5.71 to 71.33 ± 3.38 (Fig. 2h), fully demonstrating the benefits 
of SRDTrans on SMLM.
We further applied SRDTrans to experimental SMLM data of micro-
tubules to validate its ability in revealing subcellular structures. Raw 
frames were captured with low excitation power and short exposure 
time to reduce phototoxicity and emitter density. The experimentally 
obtained single-molecule-emission images and SRDTrans denoised 
images are shown in Fig. 3a. Disturbed by the noise, the reconstruc-
tion algorithm can hardly localize the fluorescent molecules in the 
raw frames. The reconstructed super-resolution image contains 
many erroneous spots and cannot reveal any meaningful structures 
 
(Fig. 3b). In comparison, SRDTrans can effectively suppress the noise 
and remove localization artifacts in the reconstructed image, making 
the distribution and extension directions of microtubules visible. We 
computed the Fourier-ring correlation (FRC)40,41 curve to quantify the 
resolution from the SMLM images. The image resolution is defined as 
the inverse of the spatial frequency at the intersection of the FRC curve 
and the threshold line (~0.143). Benefitting from the removal of noise, 
the resolution of SRDTrans denoised data was improved from 52.4 nm 
to 36.0 nm (Fig. 3c) and the localization uncertainty was reduced from 
8.0 ± 6.88 nm to 5.0 ± 1.34 nm (Fig. 3d). In addition to the data acquired 
by our instrument, we also used SRDTrans to denoise publicly avail-
able SMLM data contributed by other laboratories42 (Fig. 3e). The 
reconstructed super-resolution images indicate that SRDTrans can 
eliminate the artifacts and bring more complete organelle structures 
(Fig. 3f,g). We applied Gaussian fitting to the intensity profile perpen-
dicular to the microtube filaments and measured the full-width at 
half-maximum (FWHM) to quantify the image resolution (Fig. 3h). The 
SRDTrans denoised data show improved resolution as the averaged 
FWHM dropped from 187.89 ± 22.22 nm to 60.96 ± 7.51 nm (Fig. 3i). As 
SMLM is heading towards live-cell imaging and long-term observation43 
our denoising method promises to be a beneficial tool to reduce the 
laser power by resolving fluorescent molecules from very-low-SNR 
frames. For 3D SMLM, as the axial positions of molecules are estimated 
through point-spread-function engineering26, SRDTrans is expected to 
offer great help by resolving single-molecule-emission patterns from 
low-SNR images.
Applying SRDTrans to two-photon volumetric calcium 
imaging
In multiphoton imaging, the volumetric imaging speed decreases 
linearly as the number of scanning planes increases. Thus, the achiev-
able sampling rate for observing neuronal populations with large axial 
ranges is often quite low, making the denoising methods that rely on 
the similarity between temporally adjacent frames infeasible6,16–18. How-
ever, SRDTrans provides an opportunity to restore the highly degraded 
fluorescence signals in large-scale volumetric calcium imaging by 
utilizing the similarity between spatially adjacent pixels. To evaluate 
the denoising performance of SRDTrans on calcium imaging with dif-
ferent sampling rates, we generated realistic calcium imaging data 
with synchronized GT using neural anatomy and optical microscopy 
(NAOMi)44. We started from denoising high-sampling-rate (30 Hz) data 
and found that SRDTrans can effectively remove noise and recover 
previously indiscernible structures such as soma, neurites and vascular 
shadows (Fig. 4a, Supplementary Fig. 13 and Supplementary Video 3). 
 
The enhancement is manifested not only in the visual effect but also, 
more importantly, in the accurate restoration of pixel intensities 
 
Fig. 2 | Validation of SRDTrans on simulated SMLM data. a, Single-molecule-
emission images before and after denoising. Left: raw data. Middle: SRDTrans 
denoised data. Right: GT. Magnified views of boxed regions show the emission 
pattern of a bunch of fluorescent molecules. Scale bars, 2 μm for the whole FOV 
and 0.5 μm for magnified views. The SNR value of the raw data and denoised data 
are noted. b, Intensity profiles along the white dashed lines in a. c, Quantitative 
evaluation of the denoising performance with SNR and SSIM. Left: image SNR 
before and after denoising. Right: image SSIM before and after denoising. Each 
data point shows the statistical result of 24,000 frames. All values are shown 
as mean ± s.d. (N = 24,000 independent frames). d, Reconstructed super-
resolution images of microtubules before and after denoising. Left: the image 
reconstructed from raw data. Middle: the image reconstructed from SRDTrans 
denoised data. Right: GT. Scale bar, 5 μm. e, Merged image of the yellow boxed 
region in d. Magenta, the image reconstructed from raw data; green, the image 
reconstructed from SRDTrans denoised data; red, GT. The overlapping positions 
of red and green appear yellow. Scale bar, 1 μm. f, Consistency analysis of the 
localized fluorescent molecules in raw images (left) and SRDTrans denoised 
images (right) relative to the GT. A magnified view of the boxed region is shown 
at the bottom left of each panel. g, Tukey box-and-whisker plots (Methods) 
showing the localization precision quantified with the Jaccard index (left, higher 
is better) and r.m.s.e. (right, lower is better) before and after SRDTrans denoising 
(N = 5,000 independent molecules). h, Evaluating the performance of single-
molecule localization before (blue) and after (orange) denoising with a more 
comprehensive metric termed efficiency.


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1071
Article
https://doi.org/10.1038/s43588-023-00568-2
(Fig. 4b). Visualization in the frequency domain (calculated by discrete 
Fourier transform) shows that SRDTrans can restore most of the fre-
quency components (Fig. 4c), especially the high-frequency compo-
nents lost by DeepCAD6,18 and DeepInterpolation17, thereby leading to 
high performance in denoising calcium imaging data (Supplementary 
 
Fig. 14). Such a remarkable denoising capability can be maintained over 
a wide range of input SNRs (from −2.08 dB to 17.68 dB), and the average 
SNR improvement is about 22 ± 2.47 dB (Fig. 4d). We also verified the 
performance of SRDTrans on experimentally obtained calcium imag-
ing data with a synchronized high-SNR (tenfold photons) reference6, 
which shows that the neuronal structures and dynamics swamped by 
noise can be restored authentically (Supplementary Fig. 15).
Then we investigate the denoising performance of SRDTrans on 
calcium imaging data sampled at 0.3 Hz, which is 100 times lower than 
the imaging speed demonstrated above. Bilateral assessments in both 
the space domain and the frequency domain reveal that SRDTrans can 
a
GT
Raw data (SNR = -0.05 dB)
SRDTrans (SNR = 26.53 dB)
c
GT
Raw data
SRDTrans
d
GT
Raw data
SRDTrans
Merged
e
f
Normalized intensity (a.u.)
Distance (µm)
Distance (µm)
b
Raw data
SRDTrans
GT
GT molecules
Detected molecules
g
Raw data
SRDTrans
GT
Raw data
SRDTrans
Input SNR (dB)
20
Input SSIM
h
Eficiency = 80
Eficiency = 60
Eficiency = 40
Eficiency = 20
Eficiency = 0
Eficiency = –20
r.m.s.e. (nm)
25
50
75
100
Jaccard (%)
0
20
40
60
80
100
Raw data
SRDTrans
Raw data
SRDTrans
Output SNR (dB)
0
5
10
15
10
20
30
40
0.2
0.4
0.6
0.8
Output SSIM
0.4
0.6
0.8
0.3
0.9
1.5
0.2
0.4
0.6
0.8
1.0
0.3
0.9
1.5
0.2
0.4
0.6
0.8
1.0
1.0
100
Jaccard (%)
20
40
60
80
40
80
120
60
100
20
r.m.s.e. (nm)
0
Raw data
SRDTrans


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1072
Article
https://doi.org/10.1038/s43588-023-00568-2
Raw data
SRDTrans
e
g
Raw data
SRDTrans
(i)
(i)
(ii)
(ii)
(iii)
(iii)
a
b
c
i
Raw data
SRDTrans
50
150
200
100
250
Raw data
SRDTrans
Raw data
SRDTrans
h
Normalized intensity (a.u.)
Distance (nm)
110.32 nm
54.69 nm
(ii)
(ii)
0
100
200
300
0
100
400
115.79 nm
66.70 nm
(i)
(i)
Distance (nm)
0.4
0.8
0.4
0.8
0.4
0
Raw data
SRDTrans
0.8
90.45 nm
Distance (nm)
100
300
Gaussian fitting
(iii)
(iii)
62.29 nm
200
FWHM (nm)
Spatial frequency (nm–1)
Raw data
SRDTrans
Smooth fitting
Threshold
d
0
0.01
0.02
0.03
0.8
1.0
0.2
0.4
0.6
Cut-of = 52.4 nm
Cut-of = 36.0 nm
Uncertainty (nm)
Raw data
SRDTrans
0
10
20
30
f
Normalized FRC
Raw data
SRDTrans


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1073
Article
https://doi.org/10.1038/s43588-023-00568-2
accurately retrieve the fluorescence signals from the original highly 
degraded images without structural blurring and frequency deficiency 
(Fig. 4e and Supplementary Fig. 14). When the sampling rate is much 
lower than the fluorescence dynamics, the large discrepancy between the 
signals in two adjacent frames cannot provide the temporal correlation 
required by DeepCAD and DeepInterpolation, so they are not accurate 
enough to be used in conditions of low imaging speed or fast activity 
(Supplementary Fig. 16). To figure out how SRDTrans works at differ-
ent imaging speeds, we performed an ablation study on different sam-
pling strategies and network architectures (Fig. 4f and Supplementary 
 
Table 3). The results indicate that spatial redundancy sampling performs 
better at low imaging speeds, whereas temporal redundancy sampling 
performs better at high imaging speeds. Almost equally for all imaging 
speeds, our transformer architecture offers an additional improvement 
(~2.05 ± 0.27 dB) over conventional CNNs. In general, the synergistic 
combination of the spatial redundancy sampling and the transformer 
architecture in SRDTrans provides better performance than DeepCAD at 
all imaging speeds (Supplementary Figs. 17 and 18). In the time domain, 
the superior ability of SRDTrans can reveal high-fidelity calcium transi-
tions without distorting fluorescence kinetics (Fig. 4g). Moreover, we 
also simulated fast-moving objects to imitate migrating cells that are 
widely existed in living organisms. The quantitative evaluation shows 
that SRDTrans can preserve the structure of densely distributed objects 
even if they are moving at a speed of up to 9 pixels per frame (Supple-
mentary Fig. 19 and Supplementary Video 4), alleviating the shortage 
of denoising methods for fast-moving cells and organelles.
Finally, we went a step further in denoising calcium imaging data 
by applying SRDTrans to volumetric recordings, which is not achievable 
for other self-supervised denoising methods6,17,18 because of their heavy 
reliance on high sampling rates. We used transgenic mice expressing 
the GCaMP6f genetically encoded calcium indicator45 and imaged 
a brain volume of 500 × 500 × 200 μm3 in the mouse cortex using a 
two-photon microscope. We scanned 100 planes with a frame rate of 
30 Hz, and thus the overall volume rate was 0.3 Hz. For the denoising 
of volumetric calcium imaging data, we extracted all the frames of 
each imaging plane and reorganized them into a separate time-lapse 
(xy–t) stack. The time-lapse stacks of all imaging planes were used for 
network training. A 3D visualization of the neural volume shows that 
the spatial profiles and firing states of the neurons can be revealed 
after denoising, which otherwise would be swamped by severe shot 
noise (Fig. 5a and Supplementary Video 5). For better comparison, 
we present the snapshots of a certain imaging plane at two different 
moments. With the enhancement of SRDTrans, the structure and dis-
tribution of the neurons become clearly observable (Fig. 5b). We also 
extracted the fluorescence traces along the time axis and found that 
a large number of calcium transients can be restored after denoising 
(Fig. 5c). The dramatically improved SNR would propel the decoding 
of underlying neural activity from fluorescence signals. As neural cir-
cuits in the mammalian brain are spatially coordinated and temporally 
orchestrated, deciphering the function of large neuronal ensembles 
requires large-scale volumetric imaging with a high SNR. The superior 
denoising performance of SRDTrans provides an opportunity to imple-
ment high-sensitivity volumetric calcium imaging for investigating 
functionally concerted neurons and recognizing circuit motifs, espe-
cially those distributed across multiple cortical layers.
Discussion
SRDTrans does not rely on any assumptions about the contrast mecha-
nism, noise model, sample dynamics and imaging speed. Thus, it can be 
readily extended to other biological samples and imaging modalities 
(Supplementary Fig. 20), such as membrane voltage imaging, single-
protein detection, light-sheet microscopy, confocal microscopy, light-
field microscopy and super-resolution microscopy46–51. The limitation 
of SRDTrans lies in the basic assumption that neighboring pixels should 
have approximate structures. If the spatial sampling rate is too low to 
provide enough redundancy, SRDTrans would fail. Another potential 
risk is the generalization ability as the lightweight network architecture 
of SRDTrans is more suitable for specific tasks. We believe training 
specific models for specific data is the most reliable way to use deep 
learning for fluorescence image denoising. Therefore, a new model 
should be trained to ensure optimal results when the imaging param-
eter, modality and sample change.
Fig. 3 | Applying SRDTrans to experimental SMLM data. a, Experimentally 
obtained single-molecule-emission images. Left: raw data. Right: SRDTrans 
denoised data. The magnified views of two boxed regions are shown below 
each image. Scale bars, 2 μm for the whole FOV and 0.5 μm for magnified views. 
b, Reconstructed super-resolution images. The microtubules in fixed BSC-1 
cells were labeled with Cy5. Scale bars, 2 μm for the whole FOV and 0.5 μm for 
magnified views. c, FRC curves of the raw reconstructed image (blue) and the 
SRDTrans enhanced reconstructed image (orange). The estimated resolution 
(52.4 nm for raw image and 36.0 nm for SRDTrans denoised image) is the inverse 
of the spatial frequency where the FRC curve drops below the cut-off threshold 
(~0.143). d, Tukey box-and-whisker plots (Methods) showing the localization 
uncertainty before and after denoising (N = 1,048,575 detected molecules for 
raw data, N = 395,908 detected molecules for SRDTrans). The uncertainty was 
calculated by the ThunderSTORM plugin. e, Single-molecule-emission images 
from the open-source platform ShareLoc51. Left: raw data. Right: SRDTrans 
denoised data. Scale bar, 2 μm. f, Reconstructed super-resolution images of 
microtubules (immuno-labeled with Alexa 647). Scale bar, 2 μm. g, Magnified 
views of boxed regions. Scale bar, 0.5 μm. h, Intensity profiles perpendicular 
to the microtubule filaments indicated in g. Blue line, raw data; orange 
lines, SRDTrans denoised data; dashed line, the Gaussian fitting result. The 
corresponding FWHM value is quantified as 2.335σ, where σ denotes the standard 
deviation of the Gaussian fitting result. i, Tukey box-and-whisker plots (Methods) 
showing the FWHM of randomly selected filaments (blue, raw data; orange, 
SRDTrans denoised data; N = 80 independent filament segments).
Fig. 4 | Evaluating the performance of SRDTrans on simulated calcium 
imaging data. a, SRDTrans denoised calcium imaging data sampled at 30 Hz. 
Magnified views show the neural activity of the yellow boxed region in a short 
period (~2 s). Left: the original low-SNR data. Middle: SRDTrans denoised data. 
Right: GT. Scale bars, 60 μm for the whole FOV and 30 μm for magnified views. 
The magenta arrowhead indicates a dendritic fiber and the yellow arrowhead 
indicates two neighboring somas. b, Pixel intensity along the yellow dashed line 
in a. Top left: raw data. Middle left: SRDTrans denoised data. Bottom left: GT. 
Right: plotting the three intensity profiles in one coordinate. The similarity with 
GT is quantified by Pearson correlation coefficients (R). c, Frequency spectrum 
calculated by discrete Fourier transform before and after denoising. Magnified 
views of the boxed regions show the frequency components within the optical 
transfer function. The similarity in the frequency domain is quantified by LFD. 
d, The performance of SRDTrans at different SNR levels. All values are shown as 
mean ± s.d. (N = 6,000 independent frames). e, Comparing the performance of 
DeepCAD and SRDTrans on calcium imaging data sampled at 0.3 Hz. Magnified 
views show the neural activity of yellow boxed regions in a 20 s time window. 
Scale bars, 100 μm for the whole FOV and 40 μm for magnified views. The 
yellow and purple arrowheads point to a firing neuron and a resting neuron, 
respectively. f, Ablation experiments to investigate the effects of different 
sampling strategies and network architectures. SRDTrans (orange) uses spatial 
redundancy sampling and a lightweight spatiotemporal transformer. DeepCAD 
(purple) combines temporal redundancy sampling and a CNN (3D U-Net). 
SRDCNN (green) is the method combining spatial redundancy sampling and 
a CNN (3D U-Net). All values are shown as mean ± s.d. (N = 1,000 independent 
frames for each frame rate). g, Fluorescence traces (F) extracted from 50 
randomly selected neuronal pixels. The similarity with GT is quantified by 
Pearson correlation coefficients (R). Top: traces extracted from raw data. Middle: 
traces extracted from DeepCAD denoised data. Middle bottom: traces extracted 
from SRDTrans denoised data. Bottom: GT.


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1074
Article
https://doi.org/10.1038/s43588-023-00568-2
a
SRDTrans (SNR = 26.51 dB)
GT
Raw data (30 Hz, SNR = 0.91 dB)
5.20 s
6.07 s
4.03 s
4.27 s
4.43 s
5.20 s
6.07 s
166.67 s
156.67 s
166.67 s
156.67 s
150
300
450
600 (s)
176.67 s
176.67 s
R = 0.568
R = 0.782
R = 0.984
R = 1.000
∆F/F (normalized)
1.0
0
4.43 s
Normalized intensity
4.03 s
4.27 s
4.43 s
5.20 s
4.03 s
4.27 s
6.07 s
b
Raw data
R = 0.024
R = 0.996
R = 1.000
SRDTrans
GT
166.67 s
0.5
(normalized)
20 s
176.67 s
Raw data
SRDTrans
0
Output SNR (dB)
10
20
30
40
Input SNR (dB)
0
–5
10
5
20
15
SNR= -0.85 dB
SNR=19.88 dB
SNR=13.32 dB
e
Raw data (0.3 Hz, SNR = –0.83 dB)
1.0
0
SRDTrans (SNR = 19.62 dB)
DeepCAD (SNR = 13.10 dB) 
GT
0.1 (normalized)
30 µm
Normalized intensity
d
SRDTrans
GT
Raw data
LFD = 28.43 dB
LFD = 7.96 dB
c
0
156.67 s
166.67 s
176.67 s
156.67 s
f
g
GT
DeepCAD
Raw data
Imaging speed (Hz)
SRDTrans
Output SNR (dB)
10
–1
10
0
10
1
10
2
14
22
26
18
SRDTrans
SRDCNN
DeepCAD


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1075
Article
https://doi.org/10.1038/s43588-023-00568-2
As the development of fluorescence indicators heads towards 
faster kinetics to monitor biological dynamics at the millisecond 
scale52,53, the imaging speed required to record these fast activities is 
continuously growing. Obtaining adequate sampling rates is becoming 
more and more challenging for denoising methods relying on temporal 
redundancy. Our rationale is to fill the niche by seeking to utilize spatial 
a
Raw data
x
y
x
z
y
z
Raw data
SRDTrans
Time = 273.33 s
c
x
z
y
z
x
y
b
SRDTrans
Raw data
x
z
Time = 243.33 s
150
100
50
150
200
250
300
350
400
450
0
50
100
150
200
250
300
350
400
450
100
50
0
Position Z (µm)
Position X (µm)
Position Y (µm)
y
z
x
y
y
z
x
y
x
z
SRDTrans
1,200
1,600
400
800
0
1,200
1,600
400
800
0
∆F/F (normalized)
1.0
0
Raw data
Time (s)
Time (s)
SRDTrans
0
150
100
50
150
200
250
300
350
400
450
0
50
100
150
200
250
300
350
400
450
100
50
Position Z (µm)
Position X (µm)
Position Y (µm)
Fig. 5 | High-sensitivity calcium imaging of large neural volumes. a, Three-
dimensional visualization of the neural activity of a 510 × 510 × 200 μm3 volume 
(100 planes, 0.3 Hz volume rate) in the mouse cortex. Left: the original low-SNR 
volume. Right: the same volume denoised with SRDTrans. Magnified views of 
yellow boxed regions are shown under each snapshot. Scale bars, 100 μm for the 
whole FOV and 10 μm for magnified views. b, Raw frames and SRDTrans denoised 
counterparts of a single imaging plane at two different moments. The x–z and 
y–z cross-sections of the volume are shown alongside each x–y plane. Magnified 
views of yellow boxed regions are shown at the bottom right of the images. Scale 
bars, 70 μm for the whole FOV and 20 μm for magnified views. c, Fluorescence 
traces (F) extracted from all pixels on the yellow dashed line in b. Left: traces 
extracted from raw data. Right: traces extracted from SRDTrans denoised data. 
Yellow arrowheads point to some representative calcium transients.


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1076
Article
https://doi.org/10.1038/s43588-023-00568-2
redundancy as an alternative to enable self-supervised denoising in 
more imaging applications. Although the perfect case for spatial redun-
dancy sampling is that the spatial sampling rate is two times higher 
than the Nyquist sampling of the diffraction limit to ensure that two 
adjacent pixels have nearly identical optical signals, the endogenous 
similarity between two spatially downsampled substacks is sufficient 
to guide the training of the network in most cases. However, this does 
not mean that the proposed spatial redundancy sampling strategy can 
fully replace temporal redundancy sampling, as the ablation study 
(Fig. 4f) shows that, if equipped with the same network architecture, 
the temporal redundancy sampling can achieve better performance 
in high-speed imaging. The superiority of SRDTrans over DeepCAD at 
high imaging speeds is actually attributed to the transformer archi-
tecture. In general, spatial redundancy and temporal redundancy are 
two complementary sampling strategies to enable self-supervised 
training of denoising networks for fluorescence time-lapse imaging. 
Which sampling strategy is used depends on which kind of redundancy 
is more sufficient in the data. It is noteworthy that there are still many 
cases where neither redundancy is sufficient to support current sam-
pling strategies. Developing specific or more general self-supervised 
denoising methods is of persistent value for fluorescence imaging.
Methods
The spatial redundancy sampling strategy
In SRDTrans, we employed a spatial redundancy sampling strategy 
to produce training pairs. The detailed implementation for generat-
ing training pairs in SRDTrans is shown in Fig. 1b and Supplementary 
Fig. 1d. For each image inside an input training stack with H × W × T 
pixels (H, W and T are the height, width and length of the input image 
stack), we spatially divided it into many adjacent small patches with 
2 × 2 pixels. Next, we randomly selected three adjacent pixels from 
each patch. The central pixel was used to compose the input substack 
and the two spatially adjacent pixels were used to compose the two 
target substacks. After traversing all patches, we can finally obtain 
three downsampled substacks with the size of H/2 × W/2 × T pixels. As 
the fluorescence signals of spatially adjacent pixels are closely corre-
lated, the input substack and each target substack can be considered 
as two independent samples of the same underlying pattern. Thus, 
the input substack and the two target substacks can form two training 
pairs, which can be used for the self-supervised training of denoising 
networks (Supplementary Note 1).
Network architecture and loss function
The transformer architecture of SRDTrans is composed of three parts: 
a temporal encoder module, a spatiotemporal transformer block (STB) 
and a temporal decoder module (Fig. 1c). The temporal encoder module 
is equipped with two temporal encoders. Each temporal encoder can 
compress the temporal scale of the input substack by a factor of r (r is 4 
in this work) using a convolutional layer with 3 × 3 kernels. In contrast to 
the U-Net-type architectures with many upsampling and downsampling 
operations, SRDTrans does not reduce the size of feature maps in these 
encoders (Supplementary Fig. 21). Thus, for an input substack with a 
size of H/2 × W/2 × T pixels, the output size of the temporal encoder 
module is H/2 × W/2 × T/r2. The output from the temporal encoder 
module will be fed into the STB to extract global information both in 
space and time. The STB contains a temporal transformer block and 
a spatial transformer block (Supplementary Fig. 22). Specifically, in 
the temporal transformer block, the input is divided into patches 
with the size of p × p × T/r2 (p is 7 in this work). These patches are first 
flattened into one-dimensional vectors and input into the position 
embedding layer, where spatial concatenation and linear transfor-
mation are performed. Two multi-head self-attention layers are then 
cascaded to extract temporal correlations inside the data. In the spatial 
transformer block, a Swin transformer block27 is adopted to capture 
fine-grained spatial features with high efficiency. Local features flow 
fully in multi-head self-attention layers and densely interact with long-
range global features. Finally, the output of the STB is remapped by 
the temporal decoder module, and its temporal scale can be rescaled 
to T. This decompression operation is implemented by two cascaded 
temporal decoders using convolutional layers with 3 × 3 kernels.
We used a linear combination of L1-norm loss and L2-norm loss as 
the loss function to optimize the parameters of SRDTrans, which shows 
better performance than L1-norm loss and improved convergence 
compared with L2-norm loss (Supplementary Fig. 23). We define the 
input substack filled with central pixels as Sc, the target substack filled 
with its vertically adjacent pixels as Sv and the target substack filled with 
its horizontally adjacent pixels as Sh. The total loss consists of two pairs 
of training losses, which is defined as:
Lver = ‖FSRDTrans(Si) −Sv‖
2
2 + |FSRDTrans(Si) −Sv|1,
Lhor = ‖FSRDTrans(Si) −Sh‖
2
2 + |FSRDTrans(Si) −Sh|1,
Ltotal = Lver + Lhor.
where Lver and Lhor denote the loss of the vertically and horizontally 
adjacent substacks, respectively.
Training and inference
To achieve optimal performance, specific models were trained for 
stacks with different SNRs. One or more training stacks (xy–t or xy–z) 
were divided into a specified number of 3D (xy–t) training pairs (6,000 
by default). The batch size for all experiments was set to the number of 
graphics processing units (NVIDIA GeForce RTX 3090 for most cases) 
being used and the patch size was set to be 128 × 128 × 128 pixels. All 
extracted training pairs were geometrically transformed by random 
flipping or rotation for eightfold data augmentation. The synergy of 
our lightweight architecture and data augmentation eliminates the risk 
of overfitting (Supplementary Fig. 24). The compression factor of each 
temporal encoder was set to 4. In the STB, we set the internal patch size 
to 7, the number of heads in the multi-headed self-attention block to 
 
8 and the embedded feature channels to 128. For model optimization, 
we used the Adam optimizer and the exponential decay rate for the first 
moment was 0.9, the exponential decay rate for the second moment was 
0.999 and the learning rate was 0.00001. PyTorch was used to construct 
the network and implement all operations. In the inference stage, the 
raw noisy data were not spatially subsampled and the model of the last 
training epoch was selected for final processing. The denoised result 
of each image stack was saved as a separate TIF file.
Data simulation
Quantitative evaluations were performed on simulated data because 
noise-free images (GT) are available. To synthesize noise-free two-
photon calcium imaging data, we used NAOMi44, which can generate 
realistic calcium imaging data with high-fidelity tissue characteristics 
and fluorescence kinetics. Then we applied different levels of mixed 
Poisson–Gaussian noise to generate calcium imaging data of differ-
ent imaging SNRs6,18. We also simulated data containing only Poisson 
or Gaussian noise to show the comparable denoising performance of 
SRDTrans on these two types of noise (Supplementary Fig. 25). To gen-
erate calcium imaging data of different sampling rate, we first synthe-
sized images sampled at 30 Hz and 1 Hz, and the data of other sampling 
rates were obtained by extracting frames at different intervals. The 
image size for all simulated calcium imaging data was 490 × 490 pixels 
and the pixel size was 1.02 μm.
To generate realistic SMLM data, we first acquired reconstructed 
super-resolution images from the ShareLoc.XYZ platform (https://
shareloc.xyz/)42. These images were experimentally obtained on a 
Nikon N-STORM microscope and contained densely distributed micro-
tubules immuno-labeled with Alexa 64754. The tracks of all microtubules 
in a selected region of interest were extracted semi-automatically using 


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1077
Article
https://doi.org/10.1038/s43588-023-00568-2
the JFilament plugin of ImageJ55. We then generated synthetic single-
molecule-emission image stacks (GT images without noise) using the 
TestSTORM36 simulator by loading the microtubule patterns from 
JFilament. All fluorescent molecules were linked on the microtubule 
pattern with a radius of 12.5 nm. For imaging parameters, the numerical 
aperture was 1.4 and the frame rate was 200 Hz (5 ms exposure time). 
The image size was 328 × 328 pixels and the pixel size was 30 nm. Noisy 
stacks were generated by applying mixed Poisson–Gaussian noise post 
hoc with a customized MATLAB script6,18.
We synthesized moving objects with different moving speeds 
using the Modified National Institute of Standards and Technology 
(MNIST) dataset56. Each frame was defined as an image of 512 × 512 
pixels with a black background and many bright moving digits. Each 
digit was an image patch (28 × 28 pixels) randomly extracted from the 
MNIST dataset, moved in a random direction, and appeared or disap-
peared only once. The total number of digits in the field of view (FOV) 
was 500. We first generated the data with a moving speed of 0.5 pixels 
per frame. The data with higher moving speeds were then generated 
by extracting frames at different intervals. The total frame number for 
all moving speeds was 5,000. The final experiment was implemented 
on 20 datasets with moving speeds from 0.5 to 10 pixels per frame. 
The sampling interval of the moving speed was 0.5 pixels per frame.
SMLM imaging
The imaging samples (including the buffer solution and the sam-
ple holder) for the SMLM experiments were purchased from 
Standard Imaging Company (https://www.standardimaging.cn/
standardsample?lang=en). The SMLM experiments were performed 
on a commercial microscope (Nikon N-STORM) equipped with laser 
sources of 405 nm and 640 nm, which were used for activation and 
excitation, respectively. A scientific complementary metal-oxide semi-
conductor camera (Hamamatsu Flash 4.0) was placed at the image 
plane to capture the emission signals. To mimic living-cell imaging, we 
used low excitation power to reduce phototoxicity and short exposure 
time to obtain images with low emitter density. The detailed settings 
are summarized in Supplementary Table 4.
SMLM sample preparation
The Biologics Standards-Cercopithecus-1 (BSC-1) cell line purchased 
from Pricella Life Technology was used for SMLM imaging. BSC-1 
cells were cultured in DMEM (Invitrogen, 11965-118) supplemented 
with 10% fetal bovine serum (Gibco, 16010-159). To prevent bacterial 
contamination, 100 μg ml−1 penicillin and streptomycin (Invitrogen, 
15140122) were added into the DMEM medium. Cells were grown under 
standard cell culture conditions (5% CO2, humidified atmosphere at 
37 °C). BSC-1 cells were plated on 1.5 glass-bottom dishes over 48 h 
before sample preparation. For cell passage, cells were washed with 
pre-warmed PBS (Life Technologies, 14190500BT) 3 times and digested 
with 25% trypsin (Gibco, 25200-056) for 30 s. BSC-1 l cell lines were 
tested for potential mycoplasma contamination (MycoAlert, Lonza) 
and all tests showed negative results. For immunofluorescence stain-
ing, cells were grown on 35 mm, 1.5 glass coverslips. We pre-treated 
glass-bottom dishes with fibronectin (Invitrogen, 33016015) for 1 h at 
37 °C to increase cell adhesion. On the day of sample preparation, the 
cell density should be about 50–70%. Cells were fixed with 37 °C pre-
warmed fixation buffer for 10 min, containing 4% paraformaldehyde 
(EMS) and 0.1% glutaraldehyde in PBS. Then the sample was washed 
three times with PBS. For quenching the background fluorescence, 
we incubated the cells with 2 ml 0.1% NaBH4 solution in PBS for 7 min, 
optionally shaking on the shaker (<1 Hz). The sample was washed 3 
times with 2 ml PBS and then incubated for 30 min in PBS containing 5% 
BSA (Jackson, 001-000-162) and 0.5% Triton X-100 (Fisher Scientific) 
at 37 °C. All antibodies were diluted in the 5% BSA + 0.5% triton solution 
described above. Next, we incubated the sample for 40 min with the 
appropriate dilution of primary antibodies: mouse anti-beta-tubulin 
(E7, DSHB) at 25 °C. After primary antibodies incubation, the cells 
were washed 3 times with 2 ml PBS for 5 min. Secondary antibodies 
(AffiniPure Donkey Anti-mouse IgG, 715-005-150, Jackson Immuno 
Research) were incubated for 60 min with the appropriate dilutions 
of secondary antibodies (conjugated with Cy5) at 25 °C. After being 
washed 3 times with PBS, cells were fixed with post-fixation buffer 
for 10 min. The sample was stored at 4 °C in PBS and protected from 
light. Before imaging, we used an imaging buffer (STIBa-031, Standard 
Imaging Company) to replace PBS.
SMLM reconstruction
The super-resolution SMLM images were reconstructed by the Thun-
derSTORM37 Fiji plugin. For our experimentally obtained data, hard 
thresholding was performed to zero out those pixels with values 
smaller than a manually adjusted threshold to suppress the patterned 
noise of the camera. The detailed configuration is set as: the image filter 
was the wavelet filter (B-spline) with an order of 3 and a scale of 2; the 
algorithm for determining the approximate position of molecules was 
the local maximum algorithm; the subpixel localization is performed 
by the integrated Gaussian point-spread-function model with a fitting 
radius of 3 pixels; a fitting method of ‘weighted least squares’, and an 
initial sigma of 1.6 pixels. Both visualization images are generated 
by averaged shifted histogram with a magnification of 5. For better 
visualization, the single-molecule-emission images and reconstructed 
super-resolution images were rendered with pseudo-color and their 
contrast and brightness were manually adjusted.
Mouse preparation and calcium imaging
All experiments involving mice were performed in accordance with the 
institutional guidelines for animal welfare and have been approved by 
the Animal Care and Use Committee of Tsinghua University. All mice 
were aged 8–12 weeks and were housed in cages (24 °C, 50% humidity) 
in groups of 1–5 under a reverse light cycle. Transgenic mice hybridized 
between Rasgrf2-2A-dCre mice and Ai148 (TIT2L-GC6f-ICL-tTA2)-D 
mice expressing Cre-dependent GCaMP6f genetically encoded cal-
cium indicator were used for calcium imaging of neural circuits. Both 
male and female mice were used without randomization or blinding. 
Craniotomy surgeries were conducted to remove the skull and a cov-
erslip was implanted on the craniotomy region for chronic imaging. 
Two-photon volumetric imaging of the mouse cortex was performed 
on head-fixed mice without anesthesia using a standard two-photon 
microscope controlled with ScanImage 5.7. The neural volume being 
recorded was located at the primary visual cortex with a depth of about 
150–350 μm below the dura, and was scanned for 100 planes with an 
axial step of 2 μm. The whole imaging session lasted 30 min with a 
volume rate of 0.3 Hz.
Three-dimensional visualization of neural activity
For volumetric calcium imaging, we used Imaris 9.0 (Oxford Instru-
ments) to visualize the calcium activity of the neuronal population 
before and after denoising. Specifically, we imported the four-dimen-
sional (xyz–t) data into Imaris, applied pseudo-color to the images, and 
then performed 3D rendering using the maximum intensity projection 
mode. The contrast and brightness were adjusted to make structures 
in the volume as clear as possible. All values for gamma correction 
were set to one.
Method comparison
We compared the performance of SRDTrans with six baseline self-
supervised methods: Noise2Noise16, Noise2Void19, Noise2Self20, Proba-
bilistic Noise2Void21, Neighbor2Neighbor22, DeepInterpolation17 and 
DeepCAD6,18. These methods were all implemented by open-source 
codes released by the relevant papers. The denoising model of each 
method was trained and tested on the same datasets. For the methods 
designed for two-dimensional images, we split the time-lapse (xy–t) 


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1078
Article
https://doi.org/10.1038/s43588-023-00568-2
image stack into a series of two-dimensional frames to match the input 
dimension. Training and inference were performed frame by frame. 
We followed the default training settings about network architec-
tures and hyperparameters for all methods. Specifically, the model of 
DeepInterpolation was fine-tuned based on a public pre-trained model 
(pre-trained with 225,000 two-photon images of the Ai93 reporter 
line). Other methods were trained from scratch. The detailed settings 
of each method are listed in Supplementary Table 5.
Evaluation metrics
We used several metrics to evaluate the performance of different 
denoising methods. For an image (or an image stack) Sx and its GT Sy, 
the metrics are defined as follows.
SNR measures the pixel-level deviation between two images using 
the logarithmic decibel scale, which is formulated as
SNR = 10 log10
‖
‖Sy‖
‖
2
2
‖
‖Sx −Sy‖
‖
2
2
.
SSIM measures the similarity between two images on a perceptual 
level, including luminance, contrast and structure. The definition is
SSIM =
(2μxμy + c1)(2σxy + c2)
( μ2
x + μ2
y + c1)(σ2
x + σ2
y + c2)
,
where {μx, μy} and {σx, σy} are the means and variances of Sx and Sy, 
respectively. σxy is the covariance of Sx and Sy. The two constants c1 
and c2 are defined as c1 = (k1L)2 and c2 = (k2L)2 with k1 = 0.01, k2 = 0.03 
and L = 65,535.
The Jaccard index measures the proportion of correctly detected 
molecules in SMLM. The correctly localized fluorescent molecules are 
true positives (TP). The incorrectly localized molecules are false posi-
tives (FP) and the undetected molecules are false negatives (FN). The 
Jaccard index is formulated as:
Jaccard = 100
TP
TP + FP + FN %.
The r.m.s.e. quantifies the mean difference between the local-
ized positions (Px) and GT positions (Py) of all detected fluorescent 
molecules:
r.m.s.e. =
√
√
√
1
TP ∑
TP
‖
‖Py −Px‖
‖
2
2,
Efficiency (E) is a comprehensive metric combining the Jaccard 
index and r.m.s.e. to measure the performance of single-molecule 
localization39. It can simultaneously reflect the ability to detect mol-
ecules from images (measured by Jaccard) and the ability to precisely 
locate molecules (measured by r.m.s.e.), which is defined as:
Efficiency = 100 −√(100 −Jaccard)
2 + α2r.m.s.e.2,
where α = 1 nm−1 controls the trade-off between Jaccard and r.m.s.e.
The Pearson correlation coefficient measures the similarity 
between a variable (images and fluorescence traces) and its GT, which 
is formulated as
R =
E[(Sx −μx)(Sy −μy)]
σxσy
,
where E represents the arithmetic mean. {μx, μy} and {σx, σy} are the 
means and variances of Sx and Sy, respectively.
Logarithmic frequency distance (LFD) quantifies the spectral 
difference between two images in the frequency domain. For images 
with a size of H × W pixels, LFD is formulated as:
LFD = log10 [ 1
HW (
H−1
∑
u=0
W−1
∑
v=0
‖
‖FSx(u, v) −FSy(u, v)‖
‖
2
2) + 1] .
FSx and FSy are the discrete Fourier transform of Sx and Sy, respec-
tively. u and v are the pixel index in the frequency domain.
Statistics and reproducibility
To ensure the reproducibility of the findings, we report the sample 
size and statistics in the legend and text of each experiment. All box 
plots are drawn in the standard Tukey box-and-whisker format. The 
upper and lower quartiles are represented by box bounds, and the 
lines in the boxes indicate the median. The lower whisker represents 
the minimum observed value, equal to the lower quartile minus 1.5× 
the interquartile range. The upper whisker the maximum observed 
value, equal to the upper quartile plus 1.5× the interquartile range. 
Results obtained through experimental or observational studies or 
statistical analysis of datasets can be reproduced with high reliability 
when the study is repeated. Representative images are shown in figures 
and similar results are achieved on all test samples. Experiments in Figs. 
1d,e and 4a were repeated with 6,000 frames. Experiments in Figs. 2a 
and 3a,e were repeated with 24,000, 180,000 and 60,000 frames, 
respectively. Experiments in Figs. 4e and 5b were repeated with 1,000 
and 548 frames, respectively.
Reporting summary
Further information on research design is available in the Nature Port-
folio Reporting Summary linked to this article.
Data availability
Both simulated and experimentally obtained data of two-photon cal-
cium imaging and single-molecule localization microscopy used in this 
work can be found at https://github.com/cabooster/SRDTrans/tree/
main/datasets (refs. 57–60). Source data are provided with this paper.
Code availability
The open-source Python code of SRDTrans is available at the Zenodo 
repository61 and on GitHub (https://github.com/cabooster/SRDTrans).
References
1.	
Royer, L. A. et al. Adaptive light-sheet microscopy for long-term, 
high-resolution imaging in living organisms. Nat. Biotechnol. 34, 
1267–1278 (2016).
2.	
Fan, J. et al. Video-rate imaging of biological dynamics at 
centimetre scale and micrometre resolution. Nat. Photon. 13, 
809–816 (2019).
3.	
Balzarotti, F. et al. Nanometer resolution imaging and tracking of 
fluorescent molecules with minimal photon fluxes. Science 355, 
606–612 (2017).
4.	
Wu, J. et al. Iterative tomography with digital adaptive optics 
permits hour-long intravital observation of 3D subcellular 
dynamics at millisecond scale. Cell 184, 3318–3332 (2021).
5.	
Verweij, F. J. et al. The power of imaging to understand 
extracellular vesicle biology in vivo. Nat. Methods 18,  
1013–1026 (2021).
6.	
Li, X. et al. Real-time denoising enables high-sensitivity 
fluorescence time-lapse imaging beyond the shot-noise limit.  
Nat. Biotechnol. 41, 282–292 (2023).
7.	
Meiniel, W., Olivo-Marin, J. C. & Angelini, E. D. Denoising of 
microscopy images: a review of the state-of-the-art, and a  
new sparsity-based method. IEEE Trans. Image Process. 27, 
3842–3856 (2018).


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1079
Article
https://doi.org/10.1038/s43588-023-00568-2
8.	
Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. Image denoising 
by sparse 3-D transform-domain collaborative filtering. IEEE 
Trans. Image Process. 16, 2080–2095 (2007).
9.	
Zhang, K. et al. Beyond a Gaussian denoiser: residual learning of 
deep CNN for image denoising. IEEE Trans. Image Process. 26, 
3142–3155 (2017).
10.	 Tai, Y., Yang, J., Liu, X. & Xu, C. MemNet: a persistent memory 
network for image restoration. In Proc. IEEE/CVF Conference  
on Computer Vision and Pattern Recognition 4539–4547  
(IEEE, 2017).
11.	
Weigert, M. et al. Content-aware image restoration: pushing the 
limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 
(2018).
12.	 Belthangady, C. & Royer, L. A. Applications, promises, and  
pitfalls of deep learning for fluorescence image reconstruction. 
Nat. Methods 16, 1215–1225 (2019).
13.	 Chen, J. et al. Three-dimensional residual channel attention 
networks denoise and sharpen fluorescence microscopy image 
volumes. Nat. Methods 18, 678–687 (2021).
14.	 Chaudhary, S., Moon, S. & Lu, H. Fast, efficient, and accurate 
neuro-imaging denoising via supervised deep learning.  
Nat. Commun. 13, 5165 (2022).
15.	 Wang, Z., Xie, Y. & Ji, S. Global voxel transformer networks for 
augmented microscopy. Nat. Mach. Intell. 3, 161–171 (2021).
16.	 Lehtinen, J. et al. Noise2Noise: learning image restoration 
without clean data. In Proc. 35th International Conference  
on Machine Learning (eds Dy, J. & Krause, A.) 2965–2974  
(PMLR, 2018).
17.	 Lecoq, J. et al. Removing independent noise in systems 
neuroscience data using DeepInterpolation. Nat. Methods 18, 
1401–1408 (2021).
18.	 Li, X. et al. Reinforcing neuron extraction and spike inference 
in calcium imaging using deep self-supervised denoising. Nat. 
Methods 18, 1395–1400 (2021).
19.	 Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void—learning denoising 
from single noisy images. In Proc. IEEE/CVF Conference on 
Computer Vision and Pattern Recognition 2129–2137 (IEEE, 2019).
20.	 Batson, J. & Royer, L. Noise2Self: blind denoising by self-
supervision. In Proc. 36th International Conference on Machine 
Learning 524–533 (PMLR, 2019).
21.	 Krull, A., Vičar, T., Prakash, M., Lalit, M. & Jug, F. Probabilistic 
noise2void: unsupervised content-aware denoising. Front. 
Comput. Sci. https://doi.org/10.3389/fcomp.2020.00005 (2020).
22.	 Huang, T. et al. Neighbor2Neighbor: self-supervised denoising 
from single noisy images. In Proc. IEEE/CVF Conference  
on Computer Vision and Pattern Recognition 14781–14790  
(IEEE, 2021).
23.	 Lequyer, J. et al. A fast blind zero-shot denoiser. Nat. Mach. Intell. 
4, 953–963 (2022).
24.	 Luo, W. et al. Understanding the effective receptive field in deep 
convolutional neural networks. Adv. Neural Inf. Process. Syst. 29, 
4905–4913 (2016).
25.	 Rahaman N. et al. On the spectral bias of neural networks.  
In International Conference on Machine Learning 5301–5310 
(PMLR, 2019).
26.	 Lelek, M. et al. Single-molecule localization microscopy. Nat. Rev. 
Methods Prim. 1, 39 (2021).
27.	 Liu, Z. et al. Swin transformer: hierarchical vision transformer 
using shifted windows. In Proc. IEEE/CVF International Conference 
on Computer Vision 10012–10022 (IEEE, 2021).
28.	 Zhou H. et al. nnFormer: interleaved transformer for volumetric 
segmentation. Preprint at https://arxiv.org/abs/2109.03201 (2021).
29.	 Hatamizadeh, A. et al. UNETR: transformers for 3D medical 
image segmentation. In Proc. IEEE/CVF Winter Conference on 
Applications of Computer Vision 574–584 (IEEE, 2022).
30.	 Hatamizadeh, A. et al. Swin UNETR: Swin transformers for 
semantic segmentation of brain tumors in MRI images. In 
International MICCAI Brainlesion Workshop (eds Crimi, A. et al.) 
272–284 (Springer, 2021).
31.	 Çiçek, Ö. et al. 3D U-Net: learning dense volumetric 
segmentation from sparse annotation. In Medical Image 
Computing and Computer-Assisted Intervention—MICCAI 2016 
(eds Ourselin, S. et al.) 424–432 (Springer, 2016).
32.	 Taylor, M. A. & Bowen, W. P. Quantum metrology and its 
application in biology. Phys. Rep. 615, 1–59 (2016).
33.	 Nagata, T. et al. Beating the standard quantum limit with four-
entangled photons. Science 316, 726–729 (2007).
34.	 Rust, M., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging 
by stochastic optical reconstruction microscopy (STORM). Nat. 
Methods 3, 793–796 (2006).
35.	 Nehme, E., Weiss, L. E., Michaeli, T. & Shechtman, Y. Deep-STORM: 
super-resolution single-molecule microscopy by deep learning. 
Optica 5, 458–464 (2018).
36.	 Sinkó, J. et al. TestSTORM: simulator for optimizing sample 
labeling and image acquisition in localization based super-
resolution microscopy. Biomed. Opt. Express 5, 778–787 (2014).
37.	 Ovesný, M. et al. ThunderSTORM: a comprehensive ImageJ 
plug-in for PALM and STORM data analysis and super-resolution 
imaging. Bioinformatics 30, 2389–2390 (2014).
38.	 Sage, D. et al. Quantitative evaluation of software packages  
for singlemolecule localization microscopy. Nat. Methods 12, 
717–724 (2015).
39.	 Sage, D. et al. Super-resolution fight club: assessment of 2D 
and 3D single-molecule localization microscopy software. Nat. 
Methods 16, 387–395 (2019).
40.	 Nieuwenhuizen, R. et al. Measuring image resolution in optical 
nanoscopy. Nat. Methods 10, 557–562 (2013).
41.	 Descloux, A., Grußmayer, K. S. & Radenovic, A. Parameter-free 
image resolution estimation based on decorrelation analysis.  
Nat. Methods 16, 918–924 (2019).
42.	 Ouyang, W. et al. ShareLoc—an open platform for sharing 
localization microscopy data. Nat. Methods 19, 1331–1333 
(2022).
43.	 Jones, S. et al. Fast, three-dimensional super-resolution imaging 
of live cells. Nat. Methods 8, 499–505 (2011).
44.	 Song, A., Gauthier, J. L., Pillow, J. W., Tank, D. W. & Charles, A. S. 
Neural anatomy and optical microscopy (NAOMi) simulation for 
evaluating calcium imaging methods. J. Neurosci. Methods 358, 
109173 (2021).
45.	 Chen, T. W. et al. Ultrasensitive fluorescent proteins for imaging 
neuronal activity. Nature 499, 295–300 (2013).
46.	 Zhao, Z. et al. Two-photon synthetic aperture microscopy for 
minimally invasive fast 3D imaging of native subcellular behaviors 
in deep tissue. Cell 186, 2475–2491 (2023).
47.	 Platisa, J. et al. High-speed low-light in vivo two-photon voltage 
imaging of large neuronal populations. Nat. Methods 20,  
1095–1103 (2023).
48.	 Zhao, W. et al. Sparse deconvolution improves the resolution 
of live-cell super-resolution fluorescence microscopcy. Nat. 
Biotechnol. 40, 606–617 (2022).
49.	 Dahmardeh, M. et al. Self-supervised machine learning pushes 
the sensitivity limit in label-free detection of single proteins below 
10 kDa. Nat. Methods 20, 442–447 (2023).
50.	 Li, X. et al. Unsupervised content-preserving transformation for 
optical microscopy. Light. Sci. Appl. 10, 44 (2021).
51.	 Qiao, C. et al. Rationalized deep learning super-resolution 
microscopy for sustained live imaging of rapid subcellular 
processes. Nat. Biotechnol. 41, 367–377 (2023).
52.	 Zhang, Y. et al. Fast and sensitive GCaMP calcium indicators for 
imaging neural populations. Nature 615, 884–891 (2023).


Nature Computational Science | Volume 3 | December 2023 | 1067–1080
1080
Article
https://doi.org/10.1038/s43588-023-00568-2
53.	 Liu, Z. et al. Sustained deep-tissue voltage recording using  
a fast indicator evolved for two-photon microscopy. Cell 185, 
3408–3425 (2022).
54.	 Jimenez, A., Friedl, K. & Leterrier, C. About samples, giving 
examples: optimized single molecule localization microscopy. 
Methods 174, 100–114 (2020).
55.	 Smith, M. B. et al. Segmentation and tracking of cytoskeletal 
filaments using open active contours. Cytoskeleton 67,  
693–705 (2010).
56.	 LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based 
learning applied to document recognition. Proc. IEEE 86, 
2278–2324 (1998).
57.	 Li, X. et al. SRDTrans dataset: simulated calcium imaging data 
sampled at 30 Hz under different SNRs. Zenodo https://doi.org/ 
10.5281/zenodo.8332083 (2023).
58.	 Li, X. et al. SRDTrans dataset: simulated calcium imaging data 
at different imaging speeds. Zenodo https://doi.org/10.5281/
zenodo.7812544 (2023).
59.	 Li, X. et al. SRDTrans dataset: simulated SMLM data under 
different SNRs. Zenodo https://doi.org/10.5281/zenodo.7812589 
(2023).
60.	 Li, X. et al. SRDTrans dataset: SRDTrans dataset: experimentally 
obtained SMLM data Zenodo https://doi.org/10.5281/zenodo. 
7813184 (2023).
61.	 Li, X. et al. Code for SRDTrans. Zenodo https://doi.org/10.5281/
zenodo.10023889 (2023).
Acknowledgements
This research was supported by the National Natural Science 
Foundation of China (62088102, 62222508, 62071272) and 
National Key Research and Development Program of China 
(Project No. 2022YFB36066), in part by the Shenzhen Science and 
Technology Project under Grant (CJGJZD20200617102601004, 
JCYJ20220818101001004). This work was also supported by the 
Chinese Postdoctoral Foundation (BX2021159) and Shuimu Tsinghua 
Scholar Program. We thank H. Hao from Standard Imaging (Beijing) 
Biotechnology Co., Ltd for providing information about SMLM 
sample preparation.
Author contributions
Q.D., H.W. and J.W. supervised this research. Q.D., H.W., J.W.  
and X.L. conceived and initiated this project. X.L., X.H. and X.C. 
designed detailed implementations and performed imaging 
experiments. X.H. and X.L. developed the Python code, performed 
simulations and processed relevant imaging data. J.F. prepared 
samples and provided models animals. Z.Z. gave critical support on 
the two-photon imaging system and imaging procedures. X.H., X.L. 
and X.C. analyzed the data, prepared figures and videos. X.L., X.H., 
X.C., J.F., Z.Z. and J.W. participated in discussions about the results 
and gave valuable advice. All authors participated in the drafting of 
the paper.