Experiments

At MatMul we follow an experiment-driven framework for any initiatives we pursue.

We believe that the scientific method is our best human-invented tool yet in order to pursue truth in this world. As humans, we may have ideas, insights, beliefs but none of these are foundational truths we can stand upon. Until we do an experiment, collect data and evaluate our hypotheses we do not know for certain that our ideas, insights or beliefs are close to the truth or not. Keep in mind that an experiment only provides a partial view of the whole truth. It does not give us complete certainty. But by doing small experiments and sampling many aspects of the truth, we can hope to gain insights about the landscape in which we aim to operate. Additionally, this frame of mind often allows us to find connections between people, projects and general categories that we might not have stumbled upon had we acted upon our pre-conceived notions about the world.

The output of this process is a set of general meta-learnings that can be synthesized to form a coherent reasoning chain for any decision being made. Because decisions are being made on the basis of experimental results, we can also look at this as a risk mitigation framework.

The Hypothesis

To start the process, we must have a hypothesis that is testable. Any hypothesis that does not have a test is off the table.

The Experiment

The Setup

Once we have a hypothesis, we proceed to devise the experiment. At this stage we estimate the cost and time of an experiment. If there are parsimonious experiments, that are easy, cheap and quick to run, we should always run those first. Hard, time-consuming and costly experiments should not be discarded completely but should only be taken on when we have or can acquire the resources. The experiment should be laid down in writing. It should fit on one page so as not to devise elaborate experiments but simple ones. The experiment should lay out its data-collection procedure. The experiment should be time-boxed. The longer the experiment the less priority it should get. Try low hanging fruit first unless data from your previous experiments have told you to try the longer experiment next.

The Run

We then proceed to perform the experiment. This is pure execution. Be meticulous about collecting data that can prove or disprove your experiment. Do not collect unrelated data just because you can. What we are really focusing on here is data-collection. This means, invest heavily in anything that improves data-collection. Build infrastructure, pipeline, processes that allow you to collect high quality data that can answer your hypothesis. This is where most of our efforts in an experiment should go. The data-collection methods are the artifacts or tools that will be reused for subsequent

The Evaluation

Evaluate your hypothesis against the collected data. Have an evaluation criteria for your hypothesis from the beginning. The data should tell you if your hypothesis was correct or not. It should also tell you in what ways you were wrong and what else you should have considered when forming your hypothesis or experiment.

The Learning

Experiments inherently have an interpretable quality. As long as the experimental setup was correct and we did not collect bad data, there will always be learnings. Why are you doing this experiment? Why this hypothesis? What did you hope to learn and what did you learn from this experiment? How can you devise a better experiment? Here you will learn about connections between problems, hypotheses and see meta-problems. Now you will know which hypotheses to form next and which experiment is the next best one.

If you were correct, Great! If you were partially correct, Great! If you were wrong, Great! Change your hypothesis, devise another experiment. Don't be married to the outcome, be married to the scientific method, until us humans devise something better.