Some Fun with Gentle Chaos, the Golden Ratio, and Stochastic Number Theory

So many fascinating and deep results have been written about the number (1 + SQRT(5)) / 2 and its related sequence - the Fibonacci numbers - that it would take years to read all of them. This number has been studied both for its applications (population growth, architecture) and its mathematical properties, for over 2,000 years. It is still a topic of active research.

Lag-1 auto-correlation in digit distribution of good seeds, for b-processes

I show here how I used the golden ratio for a new number guessing game (to generate chaos and randomness in ergodic time series) as well as new intriguing results, in particular:

• Proof that the rabbit constant it is not normal in any base; this might be the first instance of a non-artificial mathematical constant for which the normalcy status is formally established.
• Beatty sequences, pseudo-periodicity, and infinite-range auto-correlations for the digits of irrational numbers in the numeration system derived from perfect stochastic processes
• Properties of multivariate b-processes, including integer or non-integer bases.
• Weird behavior of auto-correlations for the digits of normal numbers (good seeds) in the numeration system derived from stochastic b-processes
• A strange recursion that generates all the digits of the rabbit constant

Content of this article

1. Some Definitions

2. Digits Distribution in b-processes

3. Strange Facts and Conjectures about the Rabbit Constant

4. Gaming Application

• De-correlating Using Mapping and Thinning Techniques
• Dissolving the Auto-correlation Structure Using Multivariate b-processes

5. Related Articles

Read full articles, here. 

New Stock Trading and Lottery Game Rooted in Deep Math

I describe here the ultimate number guessing game, played with real money. It is a new trading and gaming system, based on state-of-the-art mathematical engineering, robust architecture, and patent-pending technology. It offers an alternative to the stock market and traditional gaming. This system is also far more transparent than the stock market, and can not be manipulated, as formulas to win the biggest returns (with real money) are made public. Also, it simulates a neutral, efficient stock market. In short, there is nothing random, everything is deterministic and fixed in advance, and known to all users. Yet it behaves in a way that looks perfectly random, and public algorithms offered to win the biggest gains require so much computing power, that for all purposes, they are useless -- except to comply with gaming laws and to establish trustworthiness.

We use private algorithms to determine the winning numbers, and while they produce the exact same results as the public algorithms (we tested this extensively), they are incredibly more efficient, by many orders of magnitude. Also, it can be mathematically proved that the public and private algorithms are equivalent, and we actually proved it. We go through this verification process for any new algorithm introduced in our system. 

In the last section, we offer a competition: can you use the public algorithm to identify the winning numbers computed with the private (secret) algorithm? If yes, the system is breakable, and a more sophisticated approach is needed, to make it work. I don't think anyone can find the winning numbers (you are welcome to prove me wrong), so the award will be offered to the contestant providing the best insights on how to improve the robustness of this system. And if by chance you manage to identify those winning numbers, great, you'll get a bonus! But it is not a requirement to win the award.

Read the full article

Content

1. Description, Main Features and Advantages

2. How it Works: the Secret Sauce

• Public Algorithm
• The Winning Numbers
• Using Seeds to Find the Winning Numbers
• ROI Tables

3. Business Model and Applications

• Managing the Money Flow

4. Challenge and Statistical Results

• Data Science / Math Competition
• Controlling the Variance of the Portfolio Value
• Probability of Cracking the System

Most Popular Content on DSC

We have been in existence for over 10 years now, with content in many different places, lists, categories, and databases. This is an attempt to put everything together in one place, and help our readers (re-)discover some great articles and resources that were lost on the Internet over the years, but still sit on our web servers. We are making them come back to life. We are in the process of organizing it in a way that is user-friendly.  Some of the resources below are very recent, and some are pretty old, but we only kept what is still relevant and useful today. To not miss this type of content in the future, subscribe to our newsletter.

Technical

• Free Books and Resources for DSC Members
• Learn Machine Learning Coding Basics in a weekend
• New Machine Learning Cheat Sheet | Old one
• Advanced Machine Learning with Basic Excel
• 12 Algorithms Every Data Scientist Should Know
• Hitchhiker's Guide to Data Science, Machine Learning, R, Python
• Visualizations: Comparing Tableau, SPSS, R, Excel, Matlab, JS, Pyth...
• How to Automatically Determine the Number of Clusters in your Data
• New Perspectives on Statistical Distributions and Deep Learning
• Fascinating New Results in the Theory of Randomness
• Long-range Correlations in Time Series: Modeling, Testing, Case Study
• Fast Combinatorial Feature Selection with New Definition of Predict...
• 10 types of regressions. Which one to use?
• 40 Techniques Used by Data Scientists
• 15 Deep Learning Tutorials
• R: a survival guide to data science with R

Non Technical

• Advanced Analytic Platforms - Incumbents Fall - Challengers Rise
• Difference between ML, Data Science, AI, Deep Learning, and Statistics
• How to Become a Data Scientist - On your own
• 16 analytic disciplines compared to data science
• Six categories of Data Scientists
• 21 data science systems used by Amazon to operate its business
• 24 Uses of Statistical Modeling
• 33 unusual problems that can be solved with data science
• 22 Differences Between Junior and Senior Data Scientists
• Why You Should be a Data Science Generalist - and How to Become One
• Becoming a Billionaire Data Scientist vs Struggling to Get a $100k Job
• Why do people with no experience want to become data scientists?

Articles from top bloggers

• Kirk Borne | Stephanie Glen | Vincent Granville 
• Ajit Jaokar | Ronald van Loon | Bernard Marr 
• Steve Miller | Bill Schmarzo | Bill Vorhies

Other popular resources

• Comprehensive Repository of Data Science and ML Resources
• Statistical Concepts Explained in Simple English
• Machine Learning Concepts Explained in One Picture
• 100 Data Science Interview Questions and Answers
• Cheat Sheets | Curated Articles | Search | Jobs | Courses
• Post a Blog | Forum Questions | Books | Salaries | News

Archives: 2008-2014 | 2015-2016 | 2017-2019 | Book 1 | Book 2 | More

Follow us: Twitter | Facebook

Long-range Correlations in Time Series: Modeling, Testing, Case Study

We investigate a large class of auto-correlated, stationary time series, proposing a new statistical test to measure departure from the base model, known as Brownian motion. We also discuss a methodology to deconstruct these time series, in order to identify the root mechanism that generates the observations. The time series studied here can be discrete or continuous in time, they  can have various degrees of smoothness (typically measured using the Hurst exponent) as well as long-range or short-range correlations between successive values. Applications are numerous, and we focus here on a case study arising from some interesting number theory problem. In particular, we show that one of the times series investigated in my article on randomness theory [see here, read section 4.1.(c)] is not Brownian despite the appearance. It has important implications regarding the problem in question. Applied to finance or economics, it makes the difference between an efficient market, and one that can be gamed.

This article it accessible to a large audience, thanks to its tutorial style, illustrations, and easily replicable simulations. Nevertheless, we discuss modern, advanced, and state-of-the-art concepts. This is an area of active research. 

 Content

1. Introduction and time series deconstruction

• Example
• Deconstructing time series
• Correlations, Fractional Brownian motions

2. Smoothness, Hurst exponent, and Brownian test

• Our Brownian tests of hypothesis
• Data

3. Results and conclusions

• Charts and interpretation
• Conclusions

Read the full article, here. 

Fascinating Developments in the Theory of Randomness

I present here some innovative results in my most recent research on stochastic processes. chaos modeling, and dynamical systems, with applications to Fintech, cryptography, number theory, and random number generators. While covering advanced topics, this article is accessible to professionals with limited knowledge in statistical or mathematical theory. It introduces new material not covered in my recent book (available here) on applied stochastic processes. You don't need to read my book to understand this article, but the book is a nice complement and introduction to the concepts discussed here.

None of the material presented here is covered in standard textbooks on stochastic processes or dynamical systems. In particular, it has nothing to do with the classical logistic map or Brownian motions, though the systems investigated here exhibit very similar behaviors and are related to the classical models. This cross-disciplinary article is targeted to professionals with interests in statistics, probability, mathematics, machine learning, simulations, signal processing, operations research, computer science, pattern recognition, and physics. Because of its tutorial style, it should also appeal to beginners learning about Markov processes, time series, and data science techniques in general, offering fresh, off-the-beaten-path content not found anywhere else, contrasting with the material covered again and again in countless, identical books, websites, and classes catering to students and researchers alike. 

Some problems discussed here could be used by college professors in the classroom, or as original exam questions, while others are extremely challenging questions that could be the subject of a PhD thesis or even well beyond that level. This article constitutes (along with my book) a stepping stone in my endeavor to solve one of the biggest mysteries in the universe: are the digits of mathematical constants such as Pi, evenly distributed? To this day, no one knows if these digits even have a distribution to start with, let alone whether that distribution is uniform or not. Part of the discussion is about statistical properties of numeration systems in a non-integer base (such as the golden ratio base) and its applications. All systems investigated here, whether deterministic or not, are treated as stochastic processes, including the digits in question. They all exhibit strong chaos, albeit easily manageable due to their ergodicity.  .

Interesting connections with the golden ratio, special polynomials, and other special mathematical constants, are discussed in section 2. Finally, all the analyses performed during this work were done in Excel. I share my spreadsheets in this article, as well as many illustration, and all the results are replicable.

Read the full article here.

Content of this article

1. General framework, notations and terminology

• Finding the equilibrium distribution
• Auto-correlation and spectral analysis
• Ergodicity, convergence, and attractors
• Space state, time state, and Markov chain approximations
• Examples

2. Case study

• First fundamental theorem
• Second fundamental theorem
• Convergence to equilibrium: illustration

3. Applications

• Potential application domains
• Example: the golden ratio process
• Finding other useful b-processes

4. Additional research topics

• Perfect stochastic processes
• Characterization of equilibrium distributions (the attractors)
• Probabilistic calculus and number theory, special integrals

5. Appendix

• Computing the auto-correlation at equilibrium
• Proof of the first fundamental theorem
• How to find the exact equilibrium distribution

6. Additional Resources

Read the full article here.

How to Automatically Determine the Number of Clusters in your Data - and more

Determining the number of clusters when performing unsupervised clustering is a tricky problem. Many data sets don't exhibit well separated clusters, and two human beings asked to visually tell the number of clusters by looking at a chart, are likely to provide two different answers. Sometimes clusters overlap with each other, and large clusters contain sub-clusters, making a decision not easy.

For instance, how many clusters do you see in the picture below? What is the optimum number of clusters? No one can tell with certainty, not AI, not a human being, not an algorithm. 

How many clusters here? (source: see here)

In the above picture, the underlying data suggests that there are three main clusters. But an answer such as 6 or 7, seems equally valid. 

A number of empirical approaches have been used to determine the number of clusters in a data set. They usually fit into two categories:

• Model fitting techniques: an example is using a mixture model to fit with your data, and determine the optimum number of components; or use density estimation techniques, and test for the number of modes (see here.) Sometimes, the fit is compared with that of a model where observations are uniformly distributed on the entire support domain, thus with no cluster; you may have to estimate the support domain in question, and assume that it is not  made of disjoint sub-domains; in many cases, the convex hull of your data set, as an estimate of the support domain, is good enough. 
• Visual techniques: for instance, the silhouette or elbow rule (very popular.)

In both cases, you need a criterion to determine the optimum number of clusters. In the case of the elbow rule, one typically uses the percentage of unexplained variance. This number is 100% with zero cluster, and it decreases (initially sharply, then more modestly) as you increase the number of clusters in your model. When each point constitutes a cluster, this number drops to 0.  Somewhere in between, the curve that displays your criterion, exhibits an elbow (see picture below), and that elbow determines the number of clusters. For instance, in the chart below, the optimum number of clusters is 4.

The elbow rule tells you that here, your data set has 4 clusters (elbow strength in red)

Good references on the topic are available. Some R functions are available too, for instance fviz_nbclust. However, I could not find in the literature, how the elbow point is explicitly computed. Most references mention that it is mostly hand-picked by visual inspection, or based on some predetermined but arbitrary threshold. In the next section, we solve this problem.

Read full article here. 

Deep Analytical Thinking and Data Science Wizardry

Many times, complex models are not enough (or too heavy), or not necessary, to get great, robust, sustainable insights out of data. Deep analytical thinking may prove more useful, and can be done by people not necessarily trained in data science, even by people with limited coding experience. Here we explore what we mean by deep analytical thinking, using a case study, and how it works: combining craftsmanship, business acumen, the use and creation of tricks and rules of thumb, to provide sound answers to business problems. These skills are usually acquired by experience more than by training, and data science generalists (see here how to become one) usually possess them.

This article is targeted to data science managers and decision makers, as well as to junior professionals who want to become one at some point in their career. Deep thinking, unlike deep learning, is also more difficult to automate, so it provides better job security. Those automating deep learning are actually the new data science wizards, who can think out-of-the box. Much of what is described in this article is also data science wizardry, and not taught in standard textbooks nor in the classroom. By reading this tutorial, you will learn and be able to use these data science secrets, and possibly change your perspective on data science. Data science is like an iceberg: everyone knows and can see the tip of the iceberg (regression models, neural nets, cross-validation, clustering, Python, and so on, as presented in textbooks.) Here I focus on the unseen bottom, using a statistical level almost accessible to the layman, avoiding jargon and complicated math formulas, yet discussing a few advanced concepts.  

Read full article here. 

Content

1. Case Study: The Problem

2. Deep Analytical Thinking

• Answering hidden questions
• Business questions
• Data questions
• Metrics questions

3. Data Science Wizardry

• Generic algorithm
• Illustration with three different models
• Results

4. A few data science hacks