New Decimal Systems - Great Sandbox for Data Scientists and Mathematicians

We illustrate pattern recognition techniques applied to an interesting mathematical problem: The representation of a number in non-conventional systems, generalizing the familiar base-2 or base-10 systems. The emphasis is on data science rather than mathematical theory, and the style is that of a tutorial, requiring minimum knowledge in mathematics or statistics. However, some off-the-beaten-path, state-of-the-art number theory research is discussed here, in a way that is accessible to college students after a first course in statistics. This article is also peppered with mathematical and statistical oddities, for instance the fact that there are units of information smaller than the bit.

You will also learn how the discovery process works, as I have included research that I thought would lead me to interesting results, but did not. In all scientific research, only final, successful results are presented, while actually most of the research leads to dead-ends, and is not made available to the reader. Here is your chance to discover these hidden steps, and my thought process!

The topic discussed is one of active research with applications to Blockchain or strong encryption. It is of interest to agencies such as the NSA or private research labs working on security issues, which explains why it is not easy to find many references; some are probably classified documents. As far as I know, it is not part of any university curriculum either. Indeed, the fear of reversibility (successful attacks on cryptographic keys using modern computer networks, new reverse-engineering algorithms, and distributed architecture) has led to the development of quantum algorithms and quantum computers, as well as Blockchain. 

All the results in this article were obtained without writing a single line of code, and replicable as I share my Excel spreadsheets. 

Content of the article
1. General Framework

• Components of number representation systems
• General properties of these systems
• Examples of number representation systems
• Examples of patterns in digit distribution
• Purpose of this research

2. Defects found in the Logistic Map system

3. First step in designing a new system

• First version of new number representation system
• Holes, autocorrelations, and entropy (information theory)

4. Towards a more general, better, hybrid system

• Faulty digits, ergodicity, and high precision computing
• Finding the equilibrium distribution with the percentile test
• Central limit theorem, random walks, Brownian motions
• Data set and Excel computations

5. Related articles

You can read the full article here. 

Two Beautiful Mathematical Results - Part 2

In Part 1 of this article (see here) we featured the two results below, as well as a simple way to prove these formulas.

Here, we continue on the same topic, featuring and proving the formulas below, which are just the tip of the iceberg.

However cool these formulas might look, the biggest contribution here is a general framework to solve much more general problems of this type. The mathematical level is still relatively simple, accessible to people in their first year of college education, if they attended a solid course on calculus. These results are still easy to prove for people who have been exposed to the basics of complex number theory.

I haven't seen these results published anywhere, but my guess is that they are not new. I encourage readers to post questions on Quora or Stackexchange to find more references on this topic, as Google search is of no use here.The focus here is to get more people interested in mathematics, by featuring fascinating results that are not that hard to prove, even for high school students participating in mathematical Olympiads. Also, it could be an interesting, fresh, original topic for university professors to discuss in their lectures or for exams. Finally, if you have seen many similar formulas on Wikipedia (or elsewhere), and you are wondering how they are derived, you will find the solution in this article.

The last section introduces a new, tough, unrelated problem, still unsolved today, that will be of interest to people with a background in probability and/or number theory.

Read the full article.

I Analyzed 10 MM digits of SQRT(2) - Look at My Findings

This article is intended for practitioners who might not necessarily be statisticians or statistically-savvy. The mathematical level is kept as simple as possible, yet I present an original, simple approach to test for randomness, with an interesting application to illustrate the methodology. This material is not something usually discussed in textbooks or classrooms (even for statistical students), offering a fresh perspective, and out-of-the-box tools that are useful in many contexts, as an addition or alternative to traditional tests that are widely used. This article is written as a tutorial, but it also features an interesting research result in the last section. The example used in this tutorial shows how intuiting can be wrong, and why you need data science.

The main question that we want to answer is: Are some events occurring randomly, or is there a mechanism making the events not occurring randomly? What is the gap distribution between two successive events of the same type? In a time-continuous setting (Poisson process) the distribution in question is modeled by the exponential distribution. In the discrete case investigated here, the discrete Poisson process turns out to be a Markov chain, and we are dealing with geometric, rather than exponential distributions. Let us illustrate this with an example.

Example

The digits of the square root of two (SQRT(2)), are believed to be distributed as if they were occurring randomly. Each of the 10 digits 0, 1, ... , 9 appears with a frequency of 10% based on observations, and at any position in the decimal expansion of SQRT(2), on average the next digit does not seem to depend on the value of the previous digit (in short, its value is unpredictable.)  An event in this context is defined, for example, as a digit being equal to (say) 3. The next event is the first time when we find a subsequent digit also equal to 3. The gap (or time elapsed) between two occurrences of the same digit is the main metric that we are interested in, and it is denoted as G. If the digits were distributed just like random numbers, the distribution of the gap G between two occurrences of the same digit, would be geometric

Do you see any pattern in the digits below? Read full article here to find the answer, and to learn more about a powerful statistical technique.

A Simple Introduction to Complex Stochastic Processes - Part 2

In my first article on this topic (see here) I introduced some of the complex stochastic processes used by Wall Street data scientists, using a simple approach that can be understood by people with no statistics background other than a first course such as stats 101. I defined and illustrated the continuous Brownian motion (the mother of all these stochastic processes) using approximations by discrete random walks, simply re-scaling the X-axis and the Y-axis appropriately, and making time increments (the X-axis) smaller and smaller, so that the limiting process is a time-continuous one. This was done without using any complicated mathematics such as measure theory or filtrations.

Here I am going one step further, introducing the integral and derivative of such processes, using rudimentary mathematics. All the articles that I've found on this subject are full of complicated equations and formulas. It is not the case here. Not only do I explain this material in simple English, but I also provide pictures to show how an Integrated Brownian motion looks like (I could not find such illustrations in the literature), how to compute its variance, and focus on applications, especially to number theory, Fintech and cryptography problems. Along the way, I discuss moving averages in a theoretical but basic framework (again with pictures), discussing what the optimal window should be for these (time-continuous or discrete) time series.

You can read the full article, here. 

DSC Resources

• Subscribe to our Newsletter
• Comprehensive Repository of Data Science and ML Resources
• Advanced Machine Learning with Basic Excel
• Difference between ML, Data Science, AI, Deep Learning, and Statistics
• Selected Business Analytics, Data Science and ML articles
• Hire a Data Scientist | Search DSC | Classifieds | Find a Job
• Post a Blog | Forum Questions

Are the Digits of Pi Truly Random? - Must Read for Math and Data Geeks

This article covers far more than the title suggests. It is written in simple English and accessible to quantitative professionals from a variety of backgrounds. Deep mathematical and data science research (including a result about the randomness of Pi, which is just a particular case) are presented here, without using arcane terminology or complicated equations.  

The topic discussed here, under a unified framework, is at the intersection of mathematics, probability theory, chaotic systems, stochastic processes, data and computer science. Many exotic objects are investigated, such as an unusual version of the logistic map, nested square roots, and representation of a number in a fractional or irrational base system. 

The article is also useful to anyone interested in learning these topics, whether they have any interest in the randomness or Pi or not, because of the numerous potential applications. I hope the style is refreshing, and I believe that you will find plenty of material rarely if ever discussed in textbooks or in the classroom. The requirements to understand this material are minimal, as I went to great lengths (over a period of years) to make it accessible to a large audience.

The randomness of the digits of Pi is one of the most fascinating, unsolved mathematical problems of all times, having been investigated by many million of people over several hundred years. The scope of this article encompasses this particular problem as part of a far more general framework. More questions are asked than answered, making this document a stepping stone for future research.

This article is structured as follows:

1. General Framework

• Questions, Properties and Notations about Chaotic Sequences Investigated Here
• Potential Applications, Including Random Number Generation

2. Examples of Chaotic Sequences Representing Numbers

• Data Science Step
• Mathematical Step
• Numbers in Base 2, 10, 3/2 or Pi 
• Nested Square Roots
• Logistic Map

3. About the Randomness of the Digits of Pi

• The Digits of Pi are Random in the Logistic Map System
• Paths to Proving Randomness in the Decimal System
• Connection with Brownian Motions

4. Curious Facts

• Randomness and The Bad Seeds Paradox
• Application to Cryptography, Financial Markets, and HPC
• Exercises
• Digits of Pi in Base Pi

The full article can be found here.

Four Interesting Math Problems

The level in this article is for college students familiar with calculus, This material will be also of interest to college professors looking for new material to teach, or for original exam questions, as well as for business data scientists with some spare time, interested in refreshing their math skills. The problems cover real analysis, mathematical algorithms and numerical precision, correct visualizations, as well as geometry. The third problem is the most interesting one in my opinion, and could become a subject of active mathematical research with one new great, unsolved conjecture being proposed, of a probabilistic nature. The last problem has many applications in engineering science.

This article is structured as follows:

1. The Simplest Function Defined by an Infinite Product

• Exercise

2. Surprising Series for Powers of Number 2

3. From Continuous Fractions to Nested Square Roots and More

• Algorithm to compute the coefficients
• Problems
• Example: Nested Square Root for the Number Pi
• Conjecture

4. Geometry: Shape Rearrangements and Coverage Problems

Read the full article

Beautiful Number Theory Problem and Sandbox for Data Scientists

The Waring conjecture - actually a problem associated with a number of conjectures, many now being solved - is one of the most fascinating mathematical problems. This article covers new aspects of this problem, with a generalization and new conjectures, some with a tentative solution, and a new framework to tackle the problem. Yet it is written in simple English and accessible to the layman.

I also review a number of famous related mathematical conjectures, including one with a $1 million award still waiting for a solution, as well as Goldbach's conjecture, yet unproved as of today.  Many curious properties of the Floor function are also listed, and the emphasis is on machine learning and efficient computer-intensive algorithms to try to find surprising results, which then need to be formally proved or disproved.

Content of this article:

1. General Framework

• Spectacular Result
• New Generalization of Golbach's Conjecture
• New Generalization of Fermat's Conjecture

2. Generalized Waring Problem

• Definitions
• Main Results
• Open Problems
• Fun Facts (Actually, Conjectures!)

3. Algorithms and Source Code

• Case n = 2: Sums of Two Terms
• Case n = 4: Sums of Four Terms

4. Related Conjectures and Solved Problems

• The One Million Dollar Conjecture

Read the full article here.