Theory of constraints

Although it is related to management and perhaps production or manufacturing systems, but since in any software code, some parts of the code do the management and have some control over the other operational parts or resources we can use this theory to study software programs and software ecosystems too.

If you have enough of resources like memory, cpu, disk and ...., and your load is low and don't expect more performance perhaps this "theory of constraints" (or TOC for short) can be just a theory for you. But if you are dealing with a lack of resources or low-performance TOC might help you to think about your software design again and perhaps modify some parts to get better performance or better use of available resources.

Stability patterns do exist in our societies too

You may think I'm joking but stability patterns do exist in every society, small, medium or large like
in a family, at work, in city or country and .... Before describing this claim, I have to add if you look for the roots of whatever human has brought or invented in computer science (even in other sciences), you'll surely find something like something in nature and the stability patterns have the same story too.

Let us consider a small company which produces bicycles. The production line has some divisions, to make it simpler we just assume the production line has three parts, P1 is responsible for building whole wheel, P2 is responsible for building the complete frame and there is also P3 which assembles wheels on the frame. And there is also just an inventory for a limited number for complete bicycle and each of P1 and P2 also has a small local inventory space.

Stability patterns in software ecosystem

If you ever have been involved in a big software project in which there are many different binaries should integrate and work together, you should have noticed that sometimes malfunctioning or failure of one of these binaries, causes the whole system get broken, while most of the time such a big software ecosystem should not really get broken because of just malfunctioning of one part.

You know what this mean? Consider a human body as a closed ecosystem, should the whole human body get broken (dead) if suddenly something happens to one of the feet? Of course not, there should be some mechanism doesn't let the fault or error spreads across the ecosystem.

Entropy as a measure of anomaly detection

Let's talk a bit more about the anomaly and then leave it for a while, there are many ways and methods we can talk about these are just some easy practical solution which in fact really works. The thing is I would like to talk about some practical aspects of software engineering too, so this will be my final post on anomaly for a while.

OK, first of all, what is entropy? There are many definitions which more or less all have the same meaning at the end, even when you talk about the entropy in thermodynamics it somehow has the same meaning of the entropy we have in information theory. If you get in depth and reach to a point exactly feel what it means, you'll find that the way entropy describes the behavior of the molecules of gas is not that different from the way we can use entropy to describe the behavior of for example the volume of a network traffic.

Calculating a reference & acceptable distance for anomaly detection

We haven't talked about anomaly detection yet, but we did talk about calculating the distance between a sample data called X and a reference data called R. We talked about a threshold or margin of acceptable distance too (M). In this post, we want to introduce a very simple but effective way of finding the reference and this acceptable distance.

Suppose we have a verified sample set like S which is, in fact, something like:

S = { X | X is a vector like (X1, X2, X3, ..., Xn) , X is not anomaly sample }

Now since these samples are all valid, we can define the R as the point in the middle of the all given X in S as below:

R = { (R1, R2, R3, ..., Rn) |  Ri = (∑ Xij ) / n , where j loops over all elements of S elements }

It is nothing but calculating the average of each dimension of the samples and assume it as the dimension of the R. If it is hard for you to accept that R is exactly in the middle of the points we have in n-dimensional space, try to figure it out in 1, 2 or 3-dimensional space.

Time series as a point in space

We usually like to predict or study the behaviour of variables through the time. Temperature, exchange rate, electricity usage, visitors of a website, traffic usage of the Internet and many more things can be studied during a period of time. For example the average temperature of a city can be studied during each month of a year, the number of the visitors of a site can be studied during each hour of a day ... and here by the study, we mean finding a reference pattern or baseline of how these variables change during their one cycle.

A simple way of calculating anomaly

Suppose you have a set of n-dimensional variables like the following:

S = { (X1, X2, X3, ..., Xn) | Xi € Di , 1 < i < n }

and we also know somehow that the reference value should be something like the following:

R =  (R1, R2, R3, ..., Rn)

And now we want to now if a Xj is an anomaly or not? Note that you can always assume that your sampled values each with n-dimension can be shown in an n-dimensional euclidean space, even if some dimensions are not numeric. For example if samples are specification of a student in grade 12, like final courses score, weight, height, behaviour, age, number of friends, the skin color and ...  you can always map all of this information for a sampled student to a vector X = (X1, X2, X3, ..., Xn) in an n-dimensional space. You just need some transformation functions to transform non-numeric values to numeric values. In other words for each non-numeric Xi , you need a Fi function which accepts some non-numeric values in its domain and transforms them to numeric values in its range.

What is anomaly?

Merriam Webster's dictionary says that anomaly is something which is unusual or unexpected. The word "anomaly" itself comes from the Latin / Greek word "anomolia", meaning "uneven" or "irregular". So what is unusual or unexpected? Something we have learned its behavior and/or physical specifications through the time and now we see it / one of its instances in other way.

You usually go for a coffee around noon on working days, and the barista has been getting used to it. One day you come and ask for exact coffee you usually order, but he gets surprised. He also gets surprised even if you go there on working days at the same time but order something other than a usual thing. So it seems the nature of anomaly just exists if we either study the phenomena in periods of time or study a phenomena or instances of it in many places.

Never ending software projects

I've been working in medium to large scale software projects as a developer, designer, technical  and project manager. Unfortunately, some of them despite of working, never had a chance to be released. I'll try to bring some of the most important things I found as reasons for having never ending software projects. Some of them may also be a reason for project failure too.

Final decision maker
Software development can not be done in complete democratic environment. You can not vote to use a technology or doing this or that all the time, especially when making a decision has time and money costs, there must always be a reason to do something. Even if you are going to make a wrong technical decision, there must be some financial or another reason to make that decision.

If your project doesn't have someone who has the authority or power of saying the last word then, believe me, you are in a big trouble. There are many points in software development cycle which you are facing approaches seem similar and none of them has any advantage over the others. People can't make a decision at these situations, there must be someone with enough power to do this. This guy is responsible for the decisions he makes so it is better for him/her has software background too, not just a manager with administration background.

Giving job to people who think are genius in all aspects 
I never forget a project in which some C++ developers who had just little MySQL experience where arguing on something like writing a query in this or that way. You had to have only a couple of years of SQL writing skills to choose the way the query must be written, but they were arguing and trying to discover how to write a double group joined SQL statement they were arguing about.

Someone who knows all the used technology stack 
There must be someone in the project who knows all the technologies used in the project. The project manager and/or the one who makes the final decisions must trust this guy otherwise project goes to voting or endless arguments most of the time to choose what to do in technical situations or software layer coupling.

Gold plating the software
This one is one of the most important reasons for a project becoming endless. I'll talk about it later, but for now, let's have a brief talk. It means paying more than necessary attention to something. For example focusing on choosing colors of the UI more than it requires, more than the functionality itself. Even when the project has almost finished we are trying to change the colors of the UI again.

Gold plating happens both where the user defines the requirement or developer uses technologies. The team who is responsible for user requirements may always changes the requirements to make the software better and the project manager or developers may also try to change the technologies because after a while they think used technologies are not cutting edge anymore.

Getting used to what the project is
This one mostly is the problem caused by the top managers, managers or anyone else who somehow has power over the project and usually don't have any development duty ... These guys look and check the software all the time, and as they get used to it, they try to make some changes again. Funny, no? They never let the software gets finalized, because while development team tries to close the tasks to complete the current version, they get used to the current version and want some new changes and features.

User/System requirements
Having no exact user or system requirements list which both the developer team and the owner of the software have same understanding on the items will cause problem whenever you want to demonstrate a version to the owner or product manager. Defining these requirements is one of the most difficult parts of the software lifecycle. The problem gets more serious when during the development either part tries to modify the requirements, at this time people are usually busy and the project is not anymore fresh, the other part just accept at that time, but when comes to deliver something, the war begins.

Having no deadline or no money problem
If for any reason the project you are working on it has no deadline or the company has no money problem at all, your project is facing to become one of those never ending projects. The reason is obvious, the human being is a perfectionist and if has time, never stops working on the project and if has money never sell the product to market until the software satisfy him, means never.

Product managers with shallow knowledge of software
It is a huge problem if you have a PM who reads specification sheets and brochures and thinks all they write in these papers are correct or even if they are, expects the software team to do the same in short time. The problem gets worse when they forget that these fact sheets or brochures are from big and old companies and yours is just a child one.

Weak marketing
Yes this is a problem and in fact, the reason is simple, marketing can't sell the product and always brings excuses, like we can't compete in the market, then adds new features, a never ending loop.

Software Evolution

We talked about "last universal ancestor" in one of the posts before, which was the first organism, all of us and other living organisms have diverged from Evolution brings diversity and this diversity is not just at the species level, it goes down to the chemistry of the life too. For example the difference between the human DNA and chimps or plants.

Human, example of an advanced software program

When your parents tell you, for example, go to the grocery store and get some grocery for the weekend. The instructions or orders they give you to go where, do what and ... is an example of a software or better to say a program that runs on you. Yes, they somehow have the ability to run this program on you and get the result. Weird, no? In fact, you are the hardware which runs your parent's software program.

Mathematics and Physics

They are friends, in my point of view physics is the main reason for mathematics development and progress. Remember the Galilei saying that the book of nature is written in mathematics. I read somewhere one of the important difference between human and other animals is the ability of bringing reason. Yes, reasoning and what language better that mathematics for reasoning.

Hierarchy of sciences - 7, Mathematics and Summary

We discussed how we can describe almost every phenomena in the world with just physics. Although we did not bring an exact proof for this claim, instead tried to proof it with common sense and a little bit of science everybody have nowadays. You need to know that we didn't cover the entire sciences exist, just the most important of them to show that the underlying layer of all sciences is physics.

Hierarchy of sciences - 6, Sociology

Although it is strange but we saw we can describe human behaviour with physics only. If you say we didn't exactly prove these things, I'll say yes we didn't, we just wanted to give an idea of if it is possible or not. We didn't do it because we don't have time to do such a thing and don't have detail knowledge, in fact it needs a team of scientists to provide an exact proof for this claim and most of the scientists when reach to a degree of knowledge simply feel that this claim is the only way to exactly describe any phenomena.

Hierarchy of sciences - 5, Psychology

OK, until now we saw that all phenomena in chemistry and biology can be described by physics alone, but in some complex and perhaps longer description manner. To understand these descriptions much easier we need to hide some details and define some new - which I say nonphysical - definitions. These definitions somehow hide physics and create some new science by themselves. And as we saw these abstractions were chemistry and biology by now.

Hierarchy of sciences - 4, Biology and human brain

What is the difference between human brain and an animal brain? I think the most exciting thing about human brain which no other creature is able to do is the ability to deduce or bring reasons. Although there are many other things which scientists think are the difference too, like appreciation of beauty, self-consciousness, morality, understanding of time, love and ... but I don't think animals really have no idea of what these things are. They may barely understand the above-mentioned things like self-consciousness or love, but I'm not sure if we make an accurate measurement system for these definitions we find them zero in animals. We also have to notice the definition itself, what is beauty? We enjoy looking at an art masterpiece, but don't you think a wild animal itself doesn't have more desire to foods that smells better? Doesn't it appreciation?

Hierarchy of sciences - 3, Biology and animals

Yes, animals. Again we are not going to talk deeply to define what animal is or about the hierarchy of the species,  how they have developed and ... We just want to get a feeling of if they all can be explained just by physics or not? Although both animals and plants are made of cells but their cells are not the same, it is because of the way they both evolved through the time. Surely no animal cell needs to have chloroplasts because they don't need to make their own food; instead, they generally digest food by digestion mechanism they have, and by food, we mean other animals or plants.

Hierarchy of sciences - 2, Biology and plants

We saw that chemistry can be interpreted as an abstraction over physics. We don't want to talk about all sciences, just some of them which may not look like as physics at first sight, but believe me, all is physics at their core, now let us talk about the biology. Biology is the study of life, and by life, we mean any living organisms. The study can be done on tiny microorganism level, or complex ecosystem consist of different types of organisms, the study can be done on structure, function, and ... for each organism or on the way organisms affect each other or live and evolve in an ecosystem.