An "easy" way to solve counting problems (D-finite functions)

Revision en16, by turmax, 2025-07-25 18:11:20

Introduction

7 months ago Elegia published a blog about D-finite functions (part1, part2) in which he showed some examples of problems that can be solved by D-finite functions.

Here I will explain a fast evaluation algorithm of the coefficients of a D-finite function.

Let $$$K$$$ be some field of characteristic $$$0$$$. Let $$$K[ [x_{1},\ldots,x_{d}] ]$$$ be the ring of formal power series on it. Let $$$K[x_{1},\ldots,x_{d}]$$$ be the polynomials of it. Let $$$K(x_{1},\ldots,x_{d})=\mathop{\mathrm{Frac}}(K[x_{1},\ldots,x_{d}])$$$ be the field of rational functions over it. Let $$$ [ x_{1}^{i_{1}}\ldots x_{d}^{i_{d}} ] f$$$ be the coefficient in $$$f$$$ of the monomial $$$x_{1}^{i_{1}}\ldots x_{d}^{i_{d}}$$$.

Definitions

A function $$$f \in K[ [x_{1},\ldots,x_{d}] ]$$$ is called D-finite if for each $$$i$$$ there exists a positive integer $$$n_{i}$$$ and polynomials (not all zeroes) $$$P_{i,0}(x_{1}, \ldots,x_{d}),\ldots,P_{i,n_{i}}(x_{1},\ldots,x_{d})$$$ such that $$$P_{i,0}f+P_{i,1}\frac{df}{dx_{i}}+\ldots+P_{i,n_{i}}\frac{d^{n_{i}}(f)}{(dx_{i})^{n_{i}}}=0$$$.

A table of coefficients $$$a(i_{1},\ldots,i_{d})$$$ is called P-recursive if there exists $$$k \in \mathbb{N}$$$ such that for each $$$j \in \{ 1,\ldots,d \} $$$ there exists a mapping $$$p: \{ 0,\ldots,k \} ^{d} \rightarrow K[x]$$$ such that at least one $$$p_{v} \neq 0$$$ and for all $$$i_{1},\ldots,i_{d} \geq k$$$ $$$\sum\limits_{v \in \{0,\ldots,k\}^{d}}^{} p_{v}(i_{j}) a(i_{1}-v_{1},\ldots,i_{d}-v_{d}) = 0$$$.

In particular, for $$$d=1$$$ it means that there exist such polynomials (not all zeroes) $$$P_{i}$$$: $$$ \forall n \geq k : \sum\limits_{0 \leq i \leq k}^{} P_{i}(n)a(n-i)=0$$$.

Properties:

1 The definition is equivalent to the "linear space of partial derivatives of $$$f$$$ over $$$K(x_{1},\ldots,x_{d})$$$ has finite dimension".

In particular, every rational function is obviously D-finite

2 They form a linear algebra over $$$K(x_{1},\ldots,x_{d})$$$ (we can multiply them by a scalar, add them and multiply them, and the result will be D-finite)

3 Every algebraic over $$$K(x_{1},\ldots,x_{d})$$$ function is D-finite

4 If $$$u_{1},\ldots,u_{d} \in K[ [u_{1},\ldots,u_{d}] ]$$$ are algebraic, and $$$f$$$ is D-finite and composition $$$f(u_{1},\ldots,u_{d})$$$ is well-defined in some sense, then it is D-finite.

5(Lipshitz, 1988) If $$$f$$$ is D-finite, then $$$\sum\limits_{i_{12},i_{3},\ldots,i_{d}}^{} x_{1}^{i_{12}}x_{3}^{i_{3}}\ldots x_{d}^{i_{d}} ([x_{1}^{i_{12}}x_{2}^{i_{12}}x_{3}^{i_{3}}\ldots x_{d}^{i_{d}}] f)$$$ is D-finite. That is, you can contract the function $$$f$$$ to the diagonal coefficients. (You can do a series of contractions to select only coefficients on the diagonal)

6(Lipshitz, 1989) The function $$$f=\sum\limits a(i_{1},\ldots,i_{d}) x_{1}^{i_{1}}\ldots x_{d}^{i_{d}}$$$ is D-finite if and only if its coefficients are P-recursive.

7(Lipshitz, 1988) If $$$f$$$ is D-finite, then for each $$$i_{1}$$$: $$$\sum\limits_{i_{2},i_{3},\ldots,i_{d}}^{} x_{2}^{i_{2}}x_{3}^{i_{3}}\ldots x_{d}^{i_{d}} ([x_{1}^{i_{1}}x_{2}^{i_{2}}x_{3}^{i_{3}}\ldots x_{d}^{i_{d}}] f)$$$ is D-finite. Moreover, the number of polynomials and their degrees in the D-finiteness relations are uniformly bounded.

Let's prove 7

Proof

Note: Here in the proof we actually have shown that the non-negative (by degree of $$$x_{1}$$$) part of a D-finite function is D-finite.

What functions are D-finite?

Can you guess which of these functions are D-finite (that is, the sequence/table of coefficients is P-recursive)?

(1) $$$a_{n} = [ x^{n+1} ] (1+x+x^{2})^{n}$$$ ($$$f(x)=\sum\limits_{}^{} a_{n}x^{n}$$$)

(2) $$$a_{n}=n!$$$ ($$$f(x)=\sum\limits_{}^{} n!x^{n}$$$)

(3) $$$a_{n,k}=S(n,k)$$$ Stirling numbers of the second kind, number of ways to partition a set of $$$n$$$ objects into $$$k$$$ non-empty subsets.

$$$\sum\limits_{n=0}^{\infty} \sum\limits_{k=0}^{n} S(n,k) \frac{x^n}{n!} y^k = e^{y(e^x-1)}$$$

(It is not so hard to see, that the D-finiteness of the OGF of some sequence/table is equivalent to the D-finiteness of EGF of this sequence/table)

(4) Eulerian numbers $$$a_{n,k}=E(n,k)$$$ is a number of $$$n$$$-element permutations with $$$k$$$ ascents.

$$$\sum\limits_{n=0}^{\infty} \sum\limits_{k=0}^{n} E(n,k)t^{k} \frac{x^n}{n!} = \frac{t-1}{t - e^{(t-1)\,x}} = \left(1-\frac{e^{(t-1)x}-1}{t-1}\right)^{-1}$$$

(5) $$$a_{u,v,w}=\sum\limits_{0 \leq x \leq u, 0\leq y \leq v, 0\leq z \leq w} \frac{(x+y)!(y+z)!(z+x)!}{x!y!z!}$$$.

Answer

The D-finite functions with $$$d=1$$$ are just P-recursive sequences. And we can evaluate the first $$$n$$$ terms of a P-recursive sequence in a stupid way in $$$O(n)$$$, or there is an algorithm to evaluate the $$$n$$$-th term (only one) in $$$O(\sqrt{n}\log(n))$$$.

Algorithm

But how to solve problems using this?

Let us assume that some function $$$f$$$ is D-finite, for example $$$f=\frac{P(x,y)}{Q(x,y)}$$$ where $$$P(x,y)$$$ and $$$Q(x,y)$$$ are fixed polynomials. And we want to compute $$$[x^{n}y^{m}] f$$$ (number of variables actually does not matter).

Let's notice that $$$h(u)=[x^{n}y^{u}] f$$$ is D-finite (this is the property 7) that is equivalent to P-recursive, and we can compute $$$h(m)$$$ in $$$O(\sqrt{m}\log(m))$$$ from some first values of $$$h$$$: $$$h(0),\ldots,h(k)$$$ where $$$k$$$ is a constant.

To compute $$$h(l)$$$ we will do the same, we will notice that $$$g(u)=[x^{u}y^{l}] f$$$ is P-recursive, so we can compute $$$g(l)$$$ in $$$O(\sqrt{n}\log(n))$$$.

So we can compute $$$[x^{n}y^{m}] f$$$ in $$$O(\sqrt{n+m}\log(n+m))$$$.

But when I say "Let's compute the coefficient of this P-recursive sequence", how do I know that the P-recursive relation holds?

One can compute it from the proof of property $$$\textbf{7}$$$, but I think this is a lot of pain. So to find P-recursive relation let's compute the first $$$k$$$ coefficients (for some $$$k$$$) and run a Gauss.

Note: We can also notice from the proof of property $$$\textbf{7}$$$, that the coefficients of P-recursion of $$$h(u)$$$ can be expressed as polynomials of $$$u$$$, and then we can try to evaluate them from some interpolation of a rational function, though I have not implemented it yet.

Example of this algorithm:

Assume we have a function $$$f(n,m)=[x^ny^m] e^{\frac{x+y}{(1-x)(1-y)}}$$$. And we want to compute $$$f(10^{7},10^{8}) \pmod{998\,244\,353}$$$.

This function $$$f$$$ is D-finite.

Let's compute its values for $$$0 \leq n \lt 25$$$ and $$$0 \leq m \lt 25$$$. Then we can extend

$$$f(\cdot,0),f(\cdot,1),f(\cdot,2),\ldots,f(\cdot,24)$$$ to $$$f(10^{7},0),f(10^{7},1),\ldots,f(10^{7},24)$$$ by P-recursion. Then we can extend $$$f(10^{7},\cdot)$$$ from the first $$$25$$$ values to $$$f(10^{7},10^{8})$$$ by P-recursion too. It works in $$$O(25\sqrt{10^{7}}\log(10^{7})+\sqrt{10^{8}}\log(10^{8}))$$$ which takes ~3.1 sec.

Note: for $$$f(10^{8},10^{8})$$$ we can compute it from diagonal.

Implementation:

Here is a code

Code

Code works for all prime moduli. Explanation:

  • findPrecursion finds P-recursion

  • evaluatePrecursion evaluates P-recursion from the initial values if possible

  • evaluatePrecursionfast evaluates the element of a sequence from P-recursion and the initial values

  • sequenceextender is trying to extend the sequence

  • optimalgetvaluebyid is trying to get the element of an extended sequence by its position

  • extendtable extends table A, finding A(que[i].first,que[i].second)

  • getcolumnoftable and getrowoftable get column or row of an extended table

A "small" problem

But there is one caveat in this algorithm. Assume that there is a sequence $$$a_{i}$$$ with P-recursion with relation $$$a_{n}P_{0}(n)+\ldots+a_{n-l}P_{l}(n)=0$$$.

You know $$$a(0),\ldots,a(l-1)$$$ and you want to extend this sequence to $$$a(0),\ldots,a(N)$$$. But what if $$$P_{0}(k)=0$$$ for some $$$l \leq n \leq N$$$? Then we can't extend this sequence. And I don't know what to do with that, but luckily, I suppose that for almost all functions when we can use it, it can't be zero.

Example where it matters

Let's try to compute $$$f(10^{6},10^{6})$$$ with $$$f(n,m)=[x^{n}y^{m}] \frac{1}{1-xy}$$$. We will try to calculate some first columns, and then a row. But even if we managed to somehow guess the first columns, we would be unable to get a row, because our initial values in this row will all be zeroes, and we want to get $$$1$$$ somehow, but we can't.

Examples of the problems

1) Atcoder Regular 202 problem D

Here if you are solving it in $$$O((H+w+T)^{1.5})$$$ using the reflection principle you have a subproblem to calculate $$$(1+x+x^{2})^{a}[x^{a+k}]$$$ for all $$$0 \leq a \leq T$$$. This can be solved by FFT (and even in such a way, that it will not add the logarithm to the asymptotic of the solution), but this also can be solved using the fact that it is P-recursive (because it is а column shift of a diagonal) and find coefficients with $$$a=k,\ldots,k+C$$$, and then extend it to the whole sequence in linear time.

2) We can also solve 1747E - List Generation in $$$O(\sqrt{n}\log(n))$$$ per test (compute the coefficient of $$$(1-x)^{2}(1-y)^{2}(1-2x-2y+2xy)^{-2}$$$), but actually, because there is a multitest and the constraints are big this solution is hard to pass (I have managed, but after a lot of pain and with some strange FFT: 330486092).

I think there are some other examples of problems that can be solved by this method.

Conclusion (Warning)

When I got it at first, it seemed to me that "every" function in combinatorics is D-finite, but it is actually wrong (see before). I think that one should use this algorithm only if one has the proof that some function is D-finite, otherwise it can be a waste of time. So I don't recommend using this algorithm as magic.

Tags counting, p-recursive, d-finite

History

 
 
 
 
Revisions
 
 
  Rev. Lang. By When Δ Comment
en23 English turmax 2025-07-26 10:35:09 3
en22 English turmax 2025-07-25 23:25:18 0 (published)
en21 English turmax 2025-07-25 23:23:59 52
en20 English turmax 2025-07-25 23:04:03 46
en19 English turmax 2025-07-25 22:44:06 22
en18 English turmax 2025-07-25 22:42:31 242
en17 English turmax 2025-07-25 18:12:33 19
en16 English turmax 2025-07-25 18:11:20 66
en15 English turmax 2025-07-25 17:59:08 64
en14 English turmax 2025-07-25 17:49:40 8
en13 English orz 2025-07-25 17:40:40 8
en12 English turmax 2025-07-25 17:36:38 2
en11 English turmax 2025-07-25 17:33:35 90
en10 English orz 2025-07-25 17:28:44 1180
en9 English turmax 2025-07-25 16:45:33 23
en8 English turmax 2025-07-25 16:43:33 12
en7 English turmax 2025-07-25 16:43:10 206
en6 English turmax 2025-07-25 16:41:53 20
en5 English turmax 2025-07-25 16:38:34 534
en4 English turmax 2025-07-25 16:22:13 12
en3 English turmax 2025-07-25 16:20:23 987
en2 English turmax 2025-07-25 16:08:28 6
ru21 Russian turmax 2025-07-25 15:54:52 40
ru20 Russian turmax 2025-07-25 15:54:03 135
ru19 Russian turmax 2025-07-25 13:45:00 2
ru18 Russian turmax 2025-07-25 13:04:07 1
en1 English turmax 2025-07-25 13:00:48 44524 Initial revision for English translation (saved to drafts)
ru17 Russian turmax 2025-07-25 12:55:00 1
ru16 Russian turmax 2025-07-25 12:51:43 104
ru15 Russian turmax 2025-07-25 12:40:48 10
ru14 Russian turmax 2025-07-25 12:03:06 374
ru13 Russian turmax 2025-07-25 11:59:09 570
ru12 Russian turmax 2025-07-25 11:42:55 147
ru11 Russian turmax 2025-07-25 11:33:44 13813
ru10 Russian turmax 2025-07-25 11:24:16 40197
ru9 Russian turmax 2025-07-25 11:15:26 278
ru8 Russian turmax 2025-07-25 11:06:27 117
ru7 Russian turmax 2025-07-25 10:43:01 33
ru6 Russian turmax 2025-07-24 22:38:15 320
ru5 Russian turmax 2025-07-24 22:19:26 110
ru4 Russian turmax 2025-07-24 22:05:25 73
ru3 Russian turmax 2025-07-24 22:01:57 1274
ru2 Russian turmax 2025-07-24 21:50:09 16620
ru1 Russian turmax 2025-07-24 21:36:04 135 Первая редакция (сохранено в черновиках)