GReinSS

Generative Modeling of Discrete Latent Structures via Dynamic Policy Gradients


Stefan Ivanovic and Mohammed El-Kebir
University of Illinois Urbana-Champaign


NCI Spring School on Algorithmic Cancer Biology — Tutorial

Ivanovic et al., ICML 2026

A recurring statistical inference problem in computational biology

States S1:N∼Pr⁡∗(S)S_{1:N} \sim \Pr^*(\mathcal{S})

Measurements X1:NX_{1:N} generated from S1:NS_{1:N}

S\mathcal{S} is typically large and combinatorial — graphs, strings, sets, …

Rather than directly observing the latent state SS we care about, we observe some indirect measurement XX generated from it.

Phylogenies

State: tumor evolution tree
Measurement: DNA-seq

[Ivanovic & El-Kebir, RECOMB/Genome Res. 2023]

CNA profiles

State: copy-number profile
Measurement: read depth + BAF

[Ivanovic & El-Kebir, Genome Biol. 2025]

RNA isoforms

State: spliced transcript
Measurement: aligned short reads

[Ivanovic et al., ICML 2026]

A learning and an inference problem

Approximate Pr⁡∗(S)\Pr^*(S) as Pr⁡(S∣θ)\Pr(S\mid\theta)

Parameters θ\theta: a linear map or neural network

Generative model with given Pr⁡(X∣S)\Pr(X\mid S)

i ∈ [N] θ Si Xi

→

Problem 1 (Learning). Given (i) X1:NX_{1:N} and (ii) Pr⁡(X∣S)\Pr(X\mid S), find θ\theta maximizing Pr⁡(X1:N∣θ)=∏iPr⁡(Xi∣θ){\Pr(X_{1:N}\mid\theta)=\prod_i \Pr(X_i\mid\theta}), where Pr⁡(Xi∣θ)=∑SPr⁡(Xi∣S) Pr⁡(S∣θ)\Pr(X_i\mid\theta)=\sum_{S}\Pr(X_i\mid S)\,\Pr(S\mid\theta)

Problem 2 (Inference). Given (i) X1:NX_{1:N}, (ii) Pr⁡(X∣S)\Pr(X\mid S), and (iii) θ\theta, find S^1:N\hat{S}_{1:N}, where S^i=arg⁡max⁡SPr⁡(Xi∣S) Pr⁡(S∣θ){\hat S_i=\arg\max_S \Pr(X_i\mid S)\,\Pr(S\mid\theta)}

Existing techniques

FamilyMethodWhy it struggles
Exact
inference		Expectation–MaximizationE-step exact only for special structure (e.g. HMMs) — intractable for a combinatorial state space
VariationalVariational inferencemaximizes an ELBO bound, not the likelihood — needs a tractable posterior over combinatorial states
Variational autoencoderslearn artificial continuous latents — not the mechanistic state you want
SearchLocal searchignores the shared model across observations
Reinforcement
learning		Naive policy gradientcollapses to the single highest-reward state
GFlowNetsaim to sample in proportion to a fixed reward — not to maximize a likelihood

Gap: none of these directly maximize Pr⁡(X1:N∣θ)\Pr(X_{1:N}\mid\theta) over a discrete, combinatorial state space.

Primer on reinforcement learning (RL) — fixed rewards

Key question: What actions should an agent take to maximize a reward signal?

Concept In RL
State sts_t current situation
Action ata_t transitions st→st+1s_t\to s_{t+1}
Policy πθ\pi_\theta picks the next action
Trajectory τ\tau path s0→⋯→s∣τ∣s_0\to\cdots\to s_{\lvert\tau\rvert}
Reward r(τ)r(\tau) scores terminal state
Objective max⁡θEτ[r(τ)]\max_\theta \mathbb{E}_\tau[r(\tau)]
sample score update θ Policy πθ Trajectory τ Reward r(τ)
τ a1 a2 a3 s0 s1 s2 s|τ|
a trajectory τ

Policy gradient (REINFORCE):   ∇θ Eτ∼Pr⁡(τ∣θ)[r(τ)]=Eτ[r(τ) ∇θlog⁡Pr⁡(τ∣θ)]\;\nabla_\theta\,\mathbb{E}_{\tau\sim\Pr(\tau\mid\theta)}[r(\tau)]=\mathbb{E}_\tau\big[r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\big] — gradient through log⁡Pr⁡(τ∣θ)\log\Pr(\tau\mid\theta) only, not the fixed reward r(τ)r(\tau).

GReinSS: Generative Reinforcement Learning of Structured States

Policy gradient (REINFORCE):   ∇θ Eτ∼Pr⁡(τ∣θ)[r(τ)]=Eτ[r(τ) ∇θlog⁡Pr⁡(τ∣θ)]\;\nabla_\theta\,\mathbb{E}_{\tau\sim\Pr(\tau\mid\theta)}[r(\tau)]=\mathbb{E}_\tau\big[r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\big]

Concept In RL In GReinSS
State sts_t current situation same as RL
Action ata_t transitions st→st+1s_t\to s_{t+1} same as RL
Policy πθ\pi_\theta picks the next action same as RL
Trajectory τ\tau path s0→⋯→s∣τ∣s_0\to\cdots\to s_{\lvert\tau\rvert} same, terminal S(τ)S(\tau)
Reward r(τ)r(\tau) scores terminal state use Pr⁡(X∣S)\Pr(X\mid S)???
Objective max⁡θEτ[r(τ)]\max_\theta \mathbb{E}_\tau[r(\tau)] max⁡θlog⁡Pr⁡(X1:N∣θ)\max_\theta \log\Pr(X_{1:N}\mid\theta)
τ empty graph A B a A B b A B C τ diploid 2 2 2 2 3 2 2 3 1

actions extend τ\tau; we infer its terminal state S(τ)S(\tau)

Key question: How to set rewards r(τ)r(\tau) such that ∇θ Eτ∼Pr⁡(τ∣θ)[r(τ)]=Eτ[r(τ) ∇θlog⁡Pr⁡(τ∣θ)]=∇θlog⁡Pr⁡(X1:N∣θ){\nabla_\theta\,\mathbb{E}_{\tau\sim\Pr(\tau\mid\theta)}[r(\tau)]=\mathbb{E}_\tau\big[r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\big] = \nabla_\theta \log \Pr(X_{1:N} \mid \theta)}?

Dynamic rewards

Train with a policy gradient using the dynamically rescaled reward r(τ)=∑i=1NPr⁡(Xi∣τ)/Pr⁡(Xi∣θ){r(\tau)=\sum_{i=1}^{N}\Pr(X_i\mid\tau)/\Pr(X_i\mid\theta)}

∇θlog⁡Pr⁡(X1:N∣θ)⏟log-likelihood\nabla_\theta\underbrace{\log\Pr(X_{1:N}\mid\theta)}_{\text{log-likelihood}}

what we optimize

=

Eτ ⁣[r(τ) ∇θlog⁡Pr⁡(τ∣θ)]⏟policy gradient\underbrace{\mathbb{E}_\tau\!\Big[r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\Big]}_{\text{policy gradient}}

how we optimize it

Theorem 1 (Unbiased policy gradient). With the dynamically changing reward r(τ)=∑i=1NPr⁡(Xi∣τ)/Pr⁡(Xi∣θ)r(\tau)=\sum_{i=1}^{N}\Pr(X_i\mid\tau)/\Pr(X_i\mid\theta), the policy gradient Eτ ⁣[r(τ) ∇θlog⁡Pr⁡(τ∣θ)]\mathbb{E}_\tau\!\big[r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\big] is an unbiased estimator of ∇θlog⁡Pr⁡(X1:N∣θ)\nabla_\theta\log\Pr(X_{1:N}\mid\theta).

Intuition behind dynamic rewards — why the denominator Pr⁡(Xi∣θ)\Pr(X_i\mid \theta)?

Pr⁡(X∣S)\Pr(X \mid S) S1S_1 S2S_2 S3S_3
X1X_1 .5.5
X2X_2 .3.3 .2.2

Example: Two measurements X1X_1 and X2X_2. States S={S1,S2,S3}\mathcal{S} = \{S_1, S_2, S_3\}.

Parameters: θ≡(Pr⁡(S1∣θ),Pr⁡(S2∣θ),Pr⁡(S3∣θ))\theta \equiv (\Pr(S_1\mid\theta), \Pr(S_2\mid\theta), \Pr(S_3\mid\theta))

Objective: max⁡θPr⁡(X1,X2∣θ)=max⁡θEτ[r(τ)]\max_\theta \Pr(X_1, X_2 \mid \theta) = \max_\theta \mathbb{E}_\tau[r(\tau)]

Fixed rewards
r(τ)=Pr⁡(Xi∣τ)r(\tau)=\Pr(X_i\mid\tau)

1 0 Pr(τ | θ) 1.0 τ1 τ2 τ3 explains X1 X2 X2

θ⋆=(1,0,0)\theta^\star=(1,0,0): Pr⁡(X1∣θ)=.5, Pr⁡(X2∣θ)=0\Pr(X_1\mid\theta)=.5,\ \Pr(X_2\mid\theta)=0

Eτ[r]\mathbb{E}_\tau[r] is linear in the policy ⇒ all mass on the best, τ1\tau_1
Pr⁡(X1,X2∣θ)=.5×0=0\Pr(X_1,X_2\mid\theta)=.5\times 0=\mathbf 0 ✗

Dynamic rewards
r(τ)=Pr⁡(Xi∣τ)/Pr⁡(Xi∣θ)r(\tau)=\Pr(X_i\mid\tau)/\Pr(X_i\mid\theta)

1 0 Pr(τ | θ) 0.5 0.5 τ1 τ2 τ3 explains X1 X2 X2

θ⋆=(.5,.5,0)\theta^\star=(.5,.5,0): Pr⁡(X1∣θ)=.25, Pr⁡(X2∣θ)=.15\Pr(X_1\mid\theta)=.25,\ \Pr(X_2\mid\theta)=.15

max⁡Eτ[r]=max⁡Pr⁡(X1,X2∣θ)\max\mathbb{E}_\tau[r]=\max \Pr(X_1,X_2\mid\theta) ⇒ balances τ1,τ2\tau_1,\tau_2
Pr⁡(X1,X2∣θ)=.25×.15=0.0375\Pr(X_1,X_2\mid\theta)=.25\times.15=0.0375 ✓

GReinSS training loop

The sample → score → update cycle of RL, run with a reward that changes as θ\theta learns:

r(τ)=∑i=1NPr⁡(Xi∣τ)Pr⁡(Xi∣θ)Pr⁡(Xi∣θ)=Eτ[Pr⁡(Xi∣τ)]r(\tau)=\sum_{i=1}^{N}\frac{\Pr(X_i\mid\tau)}{\boxed{\Pr(X_i\mid\theta)}}\qquad\Pr(X_i\mid\theta)=\mathbb{E}_{\tau}\big[\Pr(X_i\mid\tau)\big]

Dynamic reward — the boxed denominator is re-estimated by sampling each iteration.

sample score update θ Policy πθ Trajectory τ Reward r(τ)

Repeat until convergence:

  1. Sample a batch τ1,…,τM∼Pr⁡(τ∣θ)\tau_1,\dots,\tau_M\sim\Pr(\tau\mid\theta)
  2. Score each: Pr⁡(Xi∣τj)\Pr(X_i\mid\tau_j)
  3. Estimate Pr⁡(Xi∣θ)≈1M∑jPr⁡(Xi∣τj)\Pr(X_i\mid\theta)\approx\frac1M\sum_j\Pr(X_i\mid\tau_j)  (same batch)
  4. Reward r(τj)=∑iPr⁡(Xi∣τj)/Pr⁡(Xi∣θ)r(\tau_j)=\sum_i\Pr(X_i\mid\tau_j)/\Pr(X_i\mid\theta)
  5. Policy-gradient step along Eτ[r(τ) ∇θlog⁡Pr⁡(τ∣θ)]\mathbb{E}_\tau\big[r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\big]

You supply only two things:

  • a generator for SS (action-by-action)
  • the likelihood Pr⁡(X∣S)\Pr(X\mid S)

→ NOTEBOOK · Demo 1: Set reconstruction

Recover binary sets from noisy real-valued measurements. Trains live in ~10 s.

Problem. Si∗⊆US^*_i\subseteq\mathcal U, observe
Xi,j∼N(1,σ2)X_{i,j}\sim\mathcal N(1,\sigma^2) if j∈Si∗j\in S^*_i, else N(0,σ2)\mathcal N(0,\sigma^2).

You'll write:

def log_pr_x_given_g(state, obs):
    return -0.5*np.sum((obs-state)**2)/sigma**2

…then train_model_off_policy(...).

Shared structure: Each Si∗S^*_i = a union of a few reusable subsets, so a shared Pr⁡(S∣θ)\Pr(S\mid\theta) pools across observations.

Watch for:

  • median log⁡Pr⁡(Xi∣θ)\log\Pr(X_i\mid\theta) climbing to ~0
  • recovered sets vs. thresholding the noise
  • where the shared model fixes noisy bits
threshold = ½ 1 0 ½ ⋯ universe U  (one set i of N)

● true set Si∗S^*_i   ● noisy observation XiX_i  —  rounding at ½ flips the ◯ circled elements.

Neural network policy πθ\pi_\theta for the set reconstruction problem

1-hidden-layer MLP — Linear(∣U∣ ⁣→ ⁣64|\mathcal U|\!\to\!64) → LeakyReLU → Linear(64 ⁣→ ⁣∣U∣+164\!\to\!|\mathcal U|{+}1) — run once per step: read the partial set, then add an element or stop.

partial set st hidden · 64 logits 𝒰 𝒰 stop in set not yet softmax → sample at add element j stop → S(τ) autoregressive — grow the set, repeat

∣U∣|\mathcal U| add-logits + one stop; weights shared across steps.
Illegal (already-added) moves masked.  Pr⁡(S∣θ)=∑τ: S(τ)=S∏tπθ(at∣st)\Pr(S\mid\theta)=\textstyle\sum_{\tau:\,S(\tau)=S}\prod_t \pi_\theta(a_t\mid s_t)

→ NOTEBOOK · Demo 2: Off-policy learning at scale

With a larger, dispersed state space: sampling Pr⁡(τ∣θ)\Pr(\tau\mid\theta) rarely hits a terminal state S(τ)S(\tau) explaining any XiX_i, so most trajectories earn no reward and learning stalls.

Off-policy sampling: If the policy rarely samples states explaining any XiX_i, learning stalls — sample where the data says to look:

Theorem 2 (Optimal off-policy proposal). The unbiased, variance-minimizing sampling proposal is q⋆q^\star:

q⋆(τ∣X1:N,θ)=1N∑i=1NPr⁡(τ∣Xi,θ)q^\star(\tau\mid X_{1:N},\theta)=\tfrac1N\textstyle\sum_{i=1}^{N}\Pr(\tau\mid X_i,\theta)

Pr⁡(τ∣Xi,θ)=Pr⁡(Xi∣τ) Pr⁡(τ∣θ)Pr⁡(Xi∣θ)\Pr(\tau\mid X_i,\theta)=\frac{\Pr(X_i\mid\tau)\,\Pr(\tau\mid\theta)}{\Pr(X_i\mid\theta)}

q⋆q^\star is intractable → sample a tractable q≈q⋆q\approx q^\star (for sets: favor element jj by (Xi,j−12)/σ2(X_{i,j}-\tfrac12)/\sigma^2), then reweight by Pr⁡(τ∣θ)/q(τ)\Pr(\tau\mid\theta)/q(\tau).

The off-policy update in full

Exact — unbiased for the data log-likelihood gradient, for any proposal qq:

∇θlog⁡Pr⁡(X1:N∣θ)=Eτ∼q[ w(τ) r(τ) ∇θlog⁡Pr⁡(τ∣θ) ]\nabla_\theta\log\Pr(X_{1:N}\mid\theta)=\mathbb E_{\tau\sim q}\big[\,w(\tau)\,r(\tau)\,\nabla_\theta\log\Pr(\tau\mid\theta)\,\big]

w(τ)=Pr⁡(τ∣θ)q(τ)=∏tπθ(at∣st)q(at∣st)r(τ)=∑i=1NPr⁡(Xi∣τ)Pr⁡(Xi∣θ)w(\tau)=\frac{\Pr(\tau\mid\theta)}{q(\tau)}=\prod_t\frac{\pi_\theta(a_t\mid s_t)}{q(a_t\mid s_t)}\qquad r(\tau)=\sum_{i=1}^{N}\frac{\Pr(X_i\mid\tau)}{\Pr(X_i\mid\theta)}

Set problem — pick an observation ii uniformly, then bias each step by its log-odds:

q(at=add j∣st)=softmax⁡j ⁣(log⁡πθ(add j∣st)+Xi,j−12σ2)q(a_t=\text{add }j\mid s_t)=\operatorname{softmax}_j\!\Big(\log\pi_\theta(\text{add }j\mid s_t)+\tfrac{X_{i,j}-\frac12}{\sigma^2}\Big)

Batch estimate over τ1,…,τM∼q\tau_1,\dots,\tau_M\sim q  (the denominator Pr⁡(Xi∣θ)\Pr(X_i\mid\theta) in rr is re-estimated each batch):

g^=1M∑m=1Mw(τm) r(τm) ∇θlog⁡Pr⁡(τm∣θ)Pr⁡(Xi∣θ)≈1M∑m=1Mw(τm) Pr⁡(Xi∣τm)\widehat g=\frac1M\sum_{m=1}^{M} w(\tau_m)\,r(\tau_m)\,\nabla_\theta\log\Pr(\tau_m\mid\theta)\qquad \Pr(X_i\mid\theta)\approx\frac1M\sum_{m=1}^{M} w(\tau_m)\,\Pr(X_i\mid\tau_m)

On-policy: q=Pr⁡(τ∣θ)⇒w≡1q=\Pr(\tau\mid\theta)\Rightarrow w\equiv1.  Self-normalize (divide by ∑mw(τm)\textstyle\sum_m w(\tau_m)) to cut variance.

→ NOTEBOOK · Demo 3: Graph inference (pre-trained)

Reconstruct latent directed graphs from start/end points of kk random walks. Pre-trained.

Problem. Si∗S^*_i = directed graph (10 nodes, 90 edges); we see only the (v,w)(v,w) endpoints of kk walks. Pr⁡(X∣S)\Pr(X\mid S): shifted-Laplacian.

Shared structure: Each Si∗S^*_i = a random thresholded subgraph of one Erdős–Rényi (p=12)(p{=}\tfrac12) base graph.

Watch for:

  • training likelihood curve (pre-computed)
  • a reconstructed graph S^i\hat S_i vs. true Si∗S^*_i
  • per-graph F1F_1 (median ≈0.97\approx 0.97)

● latent graph Si∗S^*_i   ● GReinSS S^i\hat S_i  —  dashed = observed (start→end); paths never seen.

Simulation results from [Ivanovic et al. ICML 2026]

Latent sets — noisy measurements

Latent graphs — walk endpoints

Method Family Max. ∏iPr⁡(Xi∣θ)\prod_i\Pr(X_i\mid\theta)?
GReinSS RL ✅
Naive policy gradient RL ❌
GFlowNets RL ❌
VAE / autoregressive / diffusion + EM Variational + EM ≈
Local search Search ❌

Conclusion

States S1:N∼Pr⁡∗(S)S_{1:N} \sim \Pr^*(\mathcal{S})

Measurements X1:NX_{1:N} generated from S1:NS_{1:N}

S\mathcal{S} is typically large and combinatorial — graphs, strings, sets, …

Good fit ✓

  • latent state is discrete / combinatorial
  • you can generate it incrementally
  • you know (or can compute) Pr⁡(X∣S)\Pr(X\mid S)
  • observations are indirect & shared structure exists

Reach elsewhere ✗

  • continuous latents → VAE / diffusion
  • tractable exact E-step → classic EM
  • rewards truly fixed & known → standard RL / GFlowNet

Recipe: ① write a generator for SS · ② write Pr⁡(X∣S)\Pr(X\mid S) · ③ train_model_off_policy ·
④ simpleInference. (optional) add an off-policy proposal for hard instances.

Thank you — let's build

Code + notebook: code/README.md, tutorial/GReinSS_demo.ipynb
Recipe: generator for SS + likelihood Pr⁡(X∣S)\Pr(X\mid S) → train → infer


Questions? · melkebir@illinois.edu

GReinSS tutorial — NCI Spring School on Algorithmic Cancer Biology Speaker notes are in HTML comments like this one. Live-demo hand-offs are marked "→ NOTEBOOK".

Laser pointer: replace the mouse arrow with a red dot on every slide of the interactive deck (just preview / marp --html). No effect in the PDF/PNG exports (static images have no cursor).

* Hands-on tutorial on GReinSS. * Problem: infer hidden combinatorial states from noisy, indirect measurements. * Plan: idea → theorem → live training. * Goal: apply it to your own problem.

* Pattern: hidden structure S, indirect obs X, known likelihood Pr(X|S). * Trees, CNA sets, isoforms — all fit. * Self-supervised: measurement known, state unknown.

* Model Pr(S|θ); each S* emits an X. * Problem 1: LEARN θ jointly (shared model couples observations). * Problem 2: INFER best state per observation. * Everything today serves these two.

* Four families: exact, variational, search, RL. * EM: exact E-step only for special structure (HMMs); combinatorial → intractable. * VI: bounds the likelihood (ELBO); needs a tractable posterior. * VAE: latents are artificial ℝ^d, not your state. * Local search: per-observation, no sharing. * Naive PG / GFlowNet: closest cousins — fail next. * All miss the same target: max joint likelihood.

* Generic episodic RL. * Policy builds a trajectory; reward scores it; maximize EXPECTED reward. * Three arrows: sample → score → gradient nudge. * REINFORCE: weight each log-prob gradient by reward; no gradient through reward. * Next slides = this loop, new reward.

> **Key question:** Can we adapt reward function $r(\tau)$ to optimize data likelihood $\Pr(X_{1:N}\mid \theta)$?

* Same diagram, re-labeled — machinery unchanged. * Terminal state S(τ) is the object; policy is a neural net. * Only new piece: the reward (orange); objective = data log-likelihood. * Pivot: max expected reward ≡ max likelihood. * Exact reward → next slide (denominator is the trick).

* What we optimize = how we optimize (policy gradient, rescaled reward). * Numerator Pr(Xi|τ): how well τ explains Xi. * Denominator Pr(Xi|θ): total prob of Xi — rescales each obs. * Theorem 1: unbiased for log-likelihood gradient; reward held constant.

Policy $\theta\equiv\Pr(\tau\mid\theta)$ over $\tau_1,\tau_2,\tau_3$ ($\tau_j$ builds $S_j$); marginal $\Pr(X_i\mid\theta)=\sum_\tau\Pr(\tau\mid\theta)\,\Pr(X_i\mid\tau)$.

> Reward **shrinks as it succeeds** ⇒ the policy covers *every* observation.

* Bars = Pr(τ|θ); each panel is the θ* its reward selects. * Fixed reward: linear → all mass on τ1, θ*=(1,0,0); Pr(X2)=0, L=0. * Dynamic reward: maximizes L; balances τ1,τ2 → θ*=(.5,.5,0), L=.0375. * Denominator = automatic load-balancing. * Reproduced numerically in the notebook's final section.

* RL cycle + our reward: sample → score → update θ. * New vs vanilla RL: denominator Pr(Xi|θ) re-estimated by sampling each iteration. * Covered state → bigger denominator → smaller reward (load balancing). * You supply only: generator for S + likelihood Pr(X|S). * Notebook: write those two, call train.

* SWITCH TO JUPYTER. * Load obs → Pr(X|S) one line → build net → train ~200 epochs → infer → vs thresholding. * Punchline: shared structure denoises, beats per-pixel rounding.

* The "generator" you supply is this small MLP — one hidden layer (64 units). * Each step: read the current partial set (0/1 vector) → U+1 logits: add element j, or stop. * Illegal moves (re-adding an element) are masked to -inf before the softmax → π_θ(a_t | s_t). * No recurrence — same weights every step; the grown set is fed back in (the dashed loop). Rolling out defines Pr(S|θ). * Keep it on sets; the point is how simple the generator is — the reward does the work.

**Watch for:** on-policy median $F_1$ *below* thresholding · off-policy median $F_1=\mathbf{0.938}$ · the same sets recovered.

* Merges off-policy theorem + Demo 2 hand-off. * Concept: sample toward states that fit each obs (Theorem 2 = optimal, min-variance); importance sampling stays unbiased. * Biology: domain knowledge proposes candidates (e.g. CNNaive in CNRein). * SWITCH TO JUPYTER (Demo 2): |U|=1000; on-policy scores WORSE than thresholding (needle in a haystack). * Off-policy proposal (pre-trained) → sets recovered. * Thm 2 if asked: sample where the data points, not blindly. * Pr(τ|Xi,θ) = Bayes: policy prior reweighted by likelihood. * (1/N)Σi: equal effort per observation. * Optimal = samples land where the gradient's mass is; unbiased via importance sampling. * Intractable → approximate: (Xij−½)/σ² for sets, CNNaive in CNRein.

* BACKUP slide (skip in 30 min; good for Q&A). Full off-policy estimator. * w(τ)=Pr(τ|θ)/q(τ): importance weight, product of per-step ratios (free during rollout). * r(τ): dynamic reward (Thm 1), rescaled by Pr(Xi|θ). * Pr(Xi|θ): importance-weighted batch average, re-estimated each iteration. * q=Pr(τ|θ) (w≡1) → collapses to on-policy loop. * Any q unbiased; Thm 2's q minimizes variance. Self-normalize to cut variance.

$(L+I)^{-1}$.

* SWITCH TO JUPYTER (Demo 3). Pre-trained checkpoint (heavier model). * Load model → simpleInference → compare to ground truth → F1 + visualize. * Ground truth: Erdős–Rényi base (p=½), per-graph thresholded subgraph → shared structure. * Obs = (start,end) of k random walks; Pr(X|S) = shifted-Laplacian (L+I)^{-1}. * Cartoon = 6 nodes; real = 10 nodes / 90 edges.

* Same four families, scored on: maximizes joint likelihood? Only GReinSS. * GFlowNets: sample ∝ FIXED reward; we rescale every iteration (Thm 1) → equals data log-likelihood. * Two state types, one method. * Sets: only GReinSS scales to |U|=1000; baselines plateau. * Graphs: GReinSS wins when data is poor (k=10); naive PG → empty graph (F1=0).

* Two requirements: incremental generator + likelihood. * Have those? Four-line recipe — as in the notebook.

* One reward formula, clean theorem. * Beats standard tools on simulations and real isoform data. * Drop-in for discrete latent-state problems in cancer genomics. * Try it on your own Pr(X|S).