Comments on: (Research Thread IV) Determinstic way to find primes http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/ Massively collaborative mathematical projects Thu, 26 Aug 2010 04:34:38 +0000 hourly 1 http://wordpress.com/ By: Polymath again « What Is Research? http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1044 Polymath again « What Is Research? Mon, 26 Oct 2009 22:44:39 +0000 http://polymathprojects.org/?p=115#comment-1044 [...] Terence Tao started a polymath blog here, where he initiated four discussion threads (1, 2, 3 and 4) on deterministic ways to find primes (I’m not quite sure how that’s proceeding — [...] [...] Terence Tao started a polymath blog here, where he initiated four discussion threads (1, 2, 3 and 4) on deterministic ways to find primes (I’m not quite sure how that’s proceeding — [...]

]]> By: Kristal Cantwell http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1042 Kristal Cantwell Mon, 26 Oct 2009 19:28:11 +0000 http://polymathprojects.org/?p=115#comment-1042 That should be BQP and FBQP in the above instead of BPQ and FBPQ. Also the problem is in BQP rather than is is BQP. Sorry about these errors. That should be BQP and FBQP in the above instead of BPQ and FBPQ. Also the problem is in BQP rather than is is BQP. Sorry about these errors.

]]> By: Kristal Cantwell http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1041 Kristal Cantwell Mon, 26 Oct 2009 19:14:15 +0000 http://polymathprojects.org/?p=115#comment-1041 You are right BPQ is for decisions. There is FBPQ see http://qwiki.stanford.edu/wiki/Complexity_Zoo:F I think there should also be a FBPP but I can't find it. I think the problem is is FBPQ. We are given 10^k to 10^k+1 with a quantum number computer we have random numbers and can guess randomly each guess and test is polynomial in k and and has probability of success 1/k so if we repeat this test 2k times we should have probabliity of success 1-e^2 and we should have the problem in FBQP. This looks better than any search involving z^.49 because I think z is exponential in k. If FBPP were defined as FBPQ then something similar should show the problem is in FBPP. You are right BPQ is for decisions. There is FBPQ see

http://qwiki.stanford.edu/wiki/Complexity_Zoo:F

I think there should also be a FBPP but I can’t find it.
I think the problem is is FBPQ. We are given 10^k to 10^k+1
with a quantum number computer we have random numbers and can guess randomly each guess and test is polynomial in k
and and has probability of success 1/k so if we repeat this test 2k times we should have probabliity of success 1-e^2
and we should have the problem in FBQP. This looks better than any search involving z^.49 because I think z is exponential in k. If FBPP were defined as FBPQ then something similar should show the problem is in FBPP.

]]> By: Ernie Croot http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1032 Ernie Croot Sun, 25 Oct 2009 03:31:56 +0000 http://polymathprojects.org/?p=115#comment-1032 I finally got around to looking up the fast multiplication algorithms in Knuth's book ``Semi-numerical Algorithms'' that I had talked about before, and it turns out that one of these is called ``Karatsuba's identity'', which is somewhat similar to Strassen's identity, only it doesn't involve matrices is a lot easier to use. In fact, I think there is a chance of using it in combination with FFTs to get a quick-running algorithm to evaluate our prime generating function. Here is the idea: Suppose that we can write $latex f(X)\ =\ \sum_{i=1}^k \alpha_i(X) \beta_i(X)$, where $latex f(X)$ is our generating function for the primes in $latex [z,z+z^{0.501}]$, and where the $latex \alpha_i$ and $latex \beta_i$ are polynomials such that the total number of terms among all of them is at most, say, $latex z^{0.49}$ or so. What we would like to do is quickly evaluate $latex f(X)$ at, say, $latex X=1,x,x^2,...,x^N \pmod{2,g(x)}\ \ \ \ (*)$, where $latex N \sim z^{0.49}$ or so; and, we'd like to be able to do this using a lot fewer than $latex z^{1/2}$ operations. Now let $latex D$ be some integer (that we choose later), and write each of the $latex \alpha_i(X)$ as $latex \alpha_i(X)\ =\ \alpha_{i,1}(X) + X^D \alpha_{i,2}(X)$, where $latex \alpha_{i,1}(X)\ =\ \sum_{j \leq D-1} c_j X^j,$ and $latex X^D \alpha_{i,2}(X)\ =\ \sum_{j \geq D} c_j X^j$, where $latex c_0,c_1,...$ are the coefficients of $latex \alpha_i(X)$; and, write $latex \beta_i(X)$ similarly -- that is, $latex \beta_i(X)\ =\ \beta_{i,1}(X) + X^D \beta_{i,2}(X).$ Ideally, we want $latex D$ to be such that each $latex \alpha_{i,1}(X), \alpha_{i,2}(X)$ have about half as many terms as $latex \alpha_i(X)$, and we want the analogous to hold for the $latex \beta_i(X)$ -- of course, in order to guarantee that this is possible, we would have to be able to select that decomposition for $latex f(X)$ above carefully. Now, assuming that we can find such a $latex D$ (and have such a decomposition for $latex f(X)$), we then observe the following, which is basically Karatsuba's identity: $latex f(X)\ =\ \Sigma_1 + \Sigma_2 + \Sigma_3$, where $latex \Sigma_1\ =\ (1 - X^D) \sum_{i=1}^k \alpha_{i,1}(X)\beta_{i,1}(X)$, $latex \Sigma_2\ =\ (X^{2D} - X^D) \sum_{i=1}^k \alpha_{i,2}(X) \beta_{i,2}(X)$, $latex \Sigma_3\ =\ X^D \sum_{i=1}^k (\alpha_{i,1}(X) + \alpha_{i,2}(X))(\beta_{i,1}(X) + \beta_{i,2}(X))$. The point here is that we have replaced on sum of products of two polynomials (the $latex \alpha_i(X)\beta_i(X)$) with three sums of such products, but where each polynomial has degree about half what we had before -- each of these sums (forgetting the factors $latex (1-X^D)$ and so on) involves products of two polynomials of degree at most $latex D$. The idea is then to iterate the above process, starting with *these* sums, and then replacing each by three sums of products of polynomials each of degree $latex D/2$, and so on. Eventually, we get down to sums of products of polynomials that we can just expand out by trivial methods, as they will have few terms to begin with, and then apply FFTs to evaluate them at the points (*) above. Unfortunately, it is not always the case that the new polynomials produced at each iteration have fewer terms, since the $latex \alpha_i$ and $latex \beta_i$ may have been sparse to begin with (in fact, likely are). But what we can hope for is that the initial polynomials $latex \alpha_i$ and $latex \beta_i$ can be chosen carefully so that after a small number of iterations, we get ``mixing'' -- that is, the number of terms in, say, each of the polynomials $latex (\alpha_{i,1}(X) + \alpha_{i,2}(X))\ \ and\ \ (\beta_{i,1}(X) + \beta_{i,2}(X))$ is not much bigger than the number in each of $latex \alpha_{i,1}(X), \alpha_{i,2}(X), \beta_{i,1}(X), \beta_{i,2}(X)$. In other words, we will want that sufficiently many iterations into the above process we have that, say, $latex \alpha_{i,1}(X)$ shares many terms in common with $latex \alpha_{i,2}(X)$, and the same for the $latex \beta_{i,j}(X)$'s. I finally got around to looking up the fast multiplication algorithms in Knuth’s book “Semi-numerical Algorithms” that I had talked about before, and it turns out that one of these is called “Karatsuba’s identity”, which is somewhat similar to Strassen’s identity, only it doesn’t involve matrices is a lot easier to use. In fact, I think there is a chance of using it in combination with FFTs to get a quick-running algorithm to evaluate our prime generating function. Here is the idea: Suppose that we can write

f(X)\ =\ \sum_{i=1}^k \alpha_i(X) \beta_i(X),

where f(X) is our generating function for the primes in [z,z+z^{0.501}], and where the \alpha_i and \beta_i are polynomials such that the total number of terms among all of them is at most, say, z^{0.49} or so. What we would like to do is quickly evaluate f(X) at, say,

X=1,x,x^2,...,x^N \pmod{2,g(x)}\ \ \ \ (*),

where N \sim z^{0.49} or so; and, we’d like to be able to do this using a lot fewer than z^{1/2} operations.

Now let D be some integer (that we choose later), and write each of the \alpha_i(X) as

\alpha_i(X)\ =\ \alpha_{i,1}(X) + X^D \alpha_{i,2}(X),

where

\alpha_{i,1}(X)\ =\ \sum_{j \leq D-1} c_j X^j,

and

X^D \alpha_{i,2}(X)\ =\ \sum_{j \geq D} c_j X^j,

where c_0,c_1,... are the coefficients of \alpha_i(X); and, write \beta_i(X) similarly — that is,

\beta_i(X)\ =\ \beta_{i,1}(X) + X^D \beta_{i,2}(X).

Ideally, we want D to be such that each \alpha_{i,1}(X), \alpha_{i,2}(X) have about half as many terms as \alpha_i(X), and we want the analogous to hold for the \beta_i(X) — of course, in order to guarantee that this is possible, we would have to be able to select that decomposition for f(X) above carefully.

Now, assuming that we can find such a D (and have such a decomposition for f(X)), we then observe the following, which is basically Karatsuba’s identity:

f(X)\ =\ \Sigma_1 + \Sigma_2 + \Sigma_3,

where

\Sigma_1\ =\ (1 - X^D) \sum_{i=1}^k \alpha_{i,1}(X)\beta_{i,1}(X),

\Sigma_2\ =\ (X^{2D} - X^D) \sum_{i=1}^k \alpha_{i,2}(X) \beta_{i,2}(X),

\Sigma_3\ =\ X^D \sum_{i=1}^k (\alpha_{i,1}(X) + \alpha_{i,2}(X))(\beta_{i,1}(X) + \beta_{i,2}(X)).

The point here is that we have replaced on sum of products of two polynomials (the \alpha_i(X)\beta_i(X)) with three sums of such products, but where each polynomial has degree about half what we had before — each of these sums (forgetting the factors (1-X^D) and so on) involves products of two polynomials of degree at most D. The idea is then to iterate the above process, starting with *these* sums, and then replacing each by three sums of products of polynomials each of degree D/2, and so on. Eventually, we get down to sums of products of polynomials that we can just expand out by trivial methods, as they will have few terms to begin with, and then apply FFTs to evaluate them at the points (*) above.

Unfortunately, it is not always the case that the new polynomials produced at each iteration have fewer terms, since the \alpha_i and \beta_i may have been sparse to begin with (in fact, likely are). But what we can hope for is that the initial polynomials \alpha_i and \beta_i can be chosen carefully so that after a small number of iterations, we get “mixing” — that is, the number of terms in, say, each of the polynomials

(\alpha_{i,1}(X) + \alpha_{i,2}(X))\ \ and\ \ (\beta_{i,1}(X) + \beta_{i,2}(X))

is not much bigger than the number in each of

\alpha_{i,1}(X), \alpha_{i,2}(X), \beta_{i,1}(X), \beta_{i,2}(X).

In other words, we will want that sufficiently many iterations into the above process we have that, say, \alpha_{i,1}(X) shares many terms in common with \alpha_{i,2}(X), and the same for the \beta_{i,j}(X)‘s.

]]> By: Terence Tao http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1024 Terence Tao Thu, 22 Oct 2009 17:40:25 +0000 http://polymathprojects.org/?p=115#comment-1024 Kristal, complexity classes such as P, BPP, NP, BQP, etc. apply only to decision problems ("Yes/No" questions), not search problems ("Find an X"). Finding a prime is a search problem, not a decision problem, and so does not lie in P, BPP, etc. A relevant decision problem, though, is "Does there exist a prime in the interval [a,b]?". If one can solve this decision problem quickly, one can solve the search problem quickly, by a binary search starting from [x,2x]. It is still open whether this decision problem is in BPP. The problem is that the density of primes in [a,b] could be very low, and so it is not clear even after polylogarithmically many searches that one has a 2/3 chance or more of finding a prime (the needle in the haystack problem). But it seems from Ernie's work that this problem is at least in BP-$latex z^{0.49}$ when $latex a,b = O(z)$, by which I mean that there is a probabilistic algorithm which after $latex O(z^{0.49})$ work will correctly determine whether there is a prime or not in [a,b] with a failure probability of at most 1/3 in either case. Kristal, complexity classes such as P, BPP, NP, BQP, etc. apply only to decision problems (“Yes/No” questions), not search problems (“Find an X”). Finding a prime is a search problem, not a decision problem, and so does not lie in P, BPP, etc.

A relevant decision problem, though, is “Does there exist a prime in the interval [a,b]?”. If one can solve this decision problem quickly, one can solve the search problem quickly, by a binary search starting from [x,2x].

It is still open whether this decision problem is in BPP. The problem is that the density of primes in [a,b] could be very low, and so it is not clear even after polylogarithmically many searches that one has a 2/3 chance or more of finding a prime (the needle in the haystack problem). But it seems from Ernie’s work that this problem is at least in BP-z^{0.49} when a,b = O(z), by which I mean that there is a probabilistic algorithm which after O(z^{0.49}) work will correctly determine whether there is a prime or not in [a,b] with a failure probability of at most 1/3 in either case.

]]> By: Kristal Cantwell http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1023 Kristal Cantwell Thu, 22 Oct 2009 16:56:22 +0000 http://polymathprojects.org/?p=115#comment-1023 If the a quantum computer has a random number generator and can simulate a turing machine couldn't it solve BQP problems? In fact from http://en.wikipedia.org/wiki/BQP BQP contains BPP Furthermore since we have a source of randomness in BPP and the density is 1/log b for primes then using the random source to get a random candidate will give a number that has 1/log b probability of being prime repeating this 2log b times will give success with probability 1-(1-1/log b)^(2log b) or roughly 1-(e^-2) which means probability of failure is less than 1/3. I may be missing seeing something but it looks like the problem is in both BQP and BPP. If the a quantum computer has a random number generator and can simulate a turing machine couldn’t it solve BQP problems?

In fact from

http://en.wikipedia.org/wiki/BQP

BQP contains BPP

Furthermore since we have a source of randomness in BPP and the density is 1/log b for primes then using the random source to get a random candidate will give a number that has 1/log b probability of being prime repeating this 2log b times will give success with probability 1-(1-1/log b)^(2log b) or roughly 1-(e^-2) which means probability of failure is less than 1/3. I may be missing seeing something but it looks like the problem is in both BQP and BPP.

]]> By: Terence Tao http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1021 Terence Tao Thu, 22 Oct 2009 06:38:03 +0000 http://polymathprojects.org/?p=115#comment-1021 One observation following your comment: suppose in the interval [a,b] there were exactly two primes, and they differed by a multiple of $latex 2^m$ for some m. Then $latex \sum_{a \leq p \leq b} x^p \mod 2,g(x)$ would vanish for any g of degree at most m. So it does seem that we need g of moderately large degree (larger than $latex \log_2 z^{0.51}$) in order to deterministically detect primes. One observation following your comment: suppose in the interval [a,b] there were exactly two primes, and they differed by a multiple of 2^m for some m. Then \sum_{a \leq p \leq b} x^p \mod 2,g(x) would vanish for any g of degree at most m. So it does seem that we need g of moderately large degree (larger than \log_2 z^{0.51}) in order to deterministically detect primes.

]]> By: Terence Tao http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1020 Terence Tao Thu, 22 Oct 2009 06:22:59 +0000 http://polymathprojects.org/?p=115#comment-1020 It occurred to me that we might be able to show that the task of determining whether a prime lies in [a,b] could lie in BPP, at least. (This is not a triviality; on the wiki page http://michaelnielsen.org/polymath1/index.php?title=Oracle_counterexample_to_finding_pseudoprimes , there is a reasonable case that one could concoct a set of pseudoprimes for which one could not solve this problem quickly even when P=BPP.) The point is that testing polynomial identities over finite fields is in BPP, because such an identity either holds everywhere or fails for a large fraction of inputs, and so can be placed in BPP by random sampling. And this whole approach is basically polynomial identity testing. Actually, now that I think about it, we won't get BPP, but rather BP-$latex z^{0.49}$, which isn't nearly as impressive, but still non-trivial at least... It occurred to me that we might be able to show that the task of determining whether a prime lies in [a,b] could lie in BPP, at least. (This is not a triviality; on the wiki page http://michaelnielsen.org/polymath1/index.php?title=Oracle_counterexample_to_finding_pseudoprimes , there is a reasonable case that one could concoct a set of pseudoprimes for which one could not solve this problem quickly even when P=BPP.) The point is that testing polynomial identities over finite fields is in BPP, because such an identity either holds everywhere or fails for a large fraction of inputs, and so can be placed in BPP by random sampling. And this whole approach is basically polynomial identity testing.

Actually, now that I think about it, we won’t get BPP, but rather BP-z^{0.49}, which isn’t nearly as impressive, but still non-trivial at least…

]]> By: Terence Tao http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1019 Terence Tao Thu, 22 Oct 2009 06:13:11 +0000 http://polymathprojects.org/?p=115#comment-1019 Hmm, this is interesting; it is the first time I have seen this approach give (or have the potential to give) a run time close to $latex z^{0.5}$, rather than $latex z^{0.49} \times z^{0.49}$ which seemed to be what the previous approaches were giving (each individual evaluation of $latex \sum_{a \leq p \leq b} x^p \mod 2, g(x)$ could be evaluated below the square root barrier, but the number of such evaluations was also close to the square root). I'm still a bit worried though that when scanning an interval [a,b] of length $latex z^{0.51}$ for primes, one may need to evaluate the polynomial at about $latex z^{0.51-o(1)}$ points, because otherwise I don't see how to prevent the polynomial $latex \sum_{a \leq p \leq b} x^p$ from factorising as the product of the minimal polynomials of all the points being tested. (Here I am assuming that the minimal polynomials have degree $latex z^{o(1)}$.) This would keep the run time above $latex z^{0.51-o(1)}$ unless there was some way to test for non-vanishing at these points with sublinear efficiency. It would of course be highly unlikely that this polynomial would be so smooth, but one would need some algebraic miracle or something to rigorously show that this doesn't happen... Hmm, this is interesting; it is the first time I have seen this approach give (or have the potential to give) a run time close to z^{0.5}, rather than z^{0.49} \times z^{0.49} which seemed to be what the previous approaches were giving (each individual evaluation of \sum_{a \leq p \leq b} x^p \mod 2, g(x) could be evaluated below the square root barrier, but the number of such evaluations was also close to the square root).

I’m still a bit worried though that when scanning an interval [a,b] of length z^{0.51} for primes, one may need to evaluate the polynomial at about z^{0.51-o(1)} points, because otherwise I don’t see how to prevent the polynomial \sum_{a \leq p \leq b} x^p from factorising as the product of the minimal polynomials of all the points being tested. (Here I am assuming that the minimal polynomials have degree z^{o(1)}.) This would keep the run time above z^{0.51-o(1)} unless there was some way to test for non-vanishing at these points with sublinear efficiency.

It would of course be highly unlikely that this polynomial would be so smooth, but one would need some algebraic miracle or something to rigorously show that this doesn’t happen…

]]> By: Ernie Croot http://polymathprojects.org/2009/08/28/research-thread-iv-determinstic-way-to-find-primes/#comment-1016 Ernie Croot Thu, 22 Oct 2009 05:33:55 +0000 http://polymathprojects.org/?p=115#comment-1016 Looking closer at the paper I linked to in the previous post, I am not as enthusiastic now that it will help us to evaluate $latex f(X)$ quickly at lots of points. But it gave me an idea that I think might lead somewhere: Suppose that $latex g(X)$ has degree $latex d$, where $latex 2^d \approx \sqrt{z}$, say. Then, the order of $latex x$ mod $latex (2,g(x))$ is about $latex \sqrt{z}$. What this means is that if $latex h_1(X)$ is a polynomial $latex h_1(X) = X^{a_1} + X^{a_2} + \cdots + X^{a_k}$, then if we let $latex h_2(X) = X^{b_1} + X^{b_2} + \cdots + X^{b_k},\ b_i = a_i\ mod\ 2^d,$ then we have that for all integers $latex j$, $latex h_1(x^j)\ \equiv\ h_2(x^j) \pmod{2,g(x)}.$ So, if $latex h_1(X)$ is our prime generating function, then if we can find the corresponding polynomial $latex h_2(X)$ quickly enough (and we can try to do this in $latex C[x]$, say, where things might be easier), which has degree at most $latex \sqrt{z}$, we can use an $latex F_2$ FFT to evaluate it at $latex X=1,x,x^2,...,x^{2^d}$ in time $latex z^{1/2} (\log z)^{O(1)}$. And, since the field $latex F_2[x]/(g(x))$ (we assume $latex g$ was irreducible) contains $2^d$ roots of unity we don't have to even be clever about how exactly to use FFTs for our fast evaluation problem, as these powers of $latex x$ *are* our $latex 2^d$th roots of unity. There is one point to worry about here, concerning how this would fit in with Goal 1: In crafting Goal 1 we had assumed that $latex x$ had quite large order, say order $latex z^2$ or something (see the original note where I introduced Goal 1), yet for the above idea to work we need to work with much smaller values for the order of $latex x$. Well, it's probably ok to do this -- I haven't thought through yet the consequences of using a $latex g(x)$ of such low degree. If a stronger version of Goal 1 could be proved, where one only needs to evaluate $latex f(X)$ at $latex X=1,x,x^2,...,x^{\lfloor z^{0.49}\rfloor}$ or so, then maybe we can even take $latex g(x)$ of degree $d$ satisfying $latex 2^d \approx z^{0.49}$ or so. Looking closer at the paper I linked to in the previous post, I am not as enthusiastic now that it will help us to evaluate f(X) quickly at lots of points. But it gave me an idea that I think might lead somewhere: Suppose that g(X) has degree d, where 2^d \approx \sqrt{z}, say. Then, the order of x mod (2,g(x)) is about \sqrt{z}. What this means is that if h_1(X) is a polynomial

h_1(X) = X^{a_1} + X^{a_2} + \cdots + X^{a_k},

then if we let

h_2(X) = X^{b_1} + X^{b_2} + \cdots + X^{b_k},\ b_i = a_i\ mod\ 2^d,

then we have that for all integers j,

h_1(x^j)\ \equiv\ h_2(x^j) \pmod{2,g(x)}.

So, if h_1(X) is our prime generating function, then if we can find the corresponding polynomial h_2(X) quickly enough (and we can try to do this in C[x], say, where things might be easier), which has degree at most \sqrt{z}, we can use an F_2 FFT to evaluate it at X=1,x,x^2,...,x^{2^d} in time z^{1/2} (\log z)^{O(1)}. And, since the field F_2[x]/(g(x)) (we assume g was irreducible) contains $2^d$ roots of unity we don’t have to even be clever about how exactly to use FFTs for our fast evaluation problem, as these powers of x *are* our 2^dth roots of unity.

There is one point to worry about here, concerning how this would fit in with Goal 1: In crafting Goal 1 we had assumed that x had quite large order, say order z^2 or something (see the original note where I introduced Goal 1), yet for the above idea to work we need to work with much smaller values for the order of x. Well, it’s probably ok to do this — I haven’t thought through yet the consequences of using a g(x) of such low degree. If a stronger version of Goal 1 could be proved, where one only needs to evaluate f(X) at X=1,x,x^2,...,x^{\lfloor z^{0.49}\rfloor} or so, then maybe we can even take g(x) of degree $d$ satisfying 2^d \approx z^{0.49} or so.

]]>