Working paper - Why is there so little impact of system dynamics in the most important social questions?

#21

Hi Jack.

I think that discussing about the world3 is strongly related to the subject of this thread.

Reliable means confidence on the relative correctness of its outcome. I say relative because perfect correctness is impossible to achieve in social science.

Bug free means that every equation associated with a variable delivers the results expected by the name of the variable.

I think that the bug free condition is mandatory to hope reaching a minimum of relative correctness.

And this is not the case for the world3 model. The world3 model is a sophisticated model with a highly dynamic structure that reflects a specific point of view. Any other team would certainly build a different structure. To build such a model, one should build different models with different structures and compare their outcomes and eventually build a model averaging the different structures of the different models. I have a doubt about this having been done.

One should, too, build a very solid system of tests, embedded in the model or not, to test the coherency of the model relative to the reality it wants to describe (like the reality checks of Vensim).

I have not seen such tests. To resume, building a model with a relative reliability would need a considerable effort disproportional with its utility. The model describes an inexorable future with no specific policies impossible to apply due to the multiple policies and interest of the earth countries.

I think that there has been no effort to improve the reliability of the model, doing even partially the necessary actions described. Because if it had been done, I imagine that the authors would have done easier tests discovering eventual equation bugs in the model.

If you download the model, (preferably with any of the 32 bits Vensim versions, I had problems with the 64 ones) and set the model<setting<units equiv to ‘use stricte testing’ and press CTRL U it shows 9 errors that are not shown when stricte testing is off.

When stricte testing is off, Vensim forces a different dimension to DMNL values if it does not fit other values in the expression. For instance, 1 (dmnl) + 1 (person) will assume that 1 (dmnl) is in fact 1 (person) and will not report an error. If strict testing is on, it will report an error.

This test requires no effort and generates immediate results.

On the case of the World3, there are 9 errors reports when strict testing is on.

Sometimes with Vensim, it is necessary to test first Ctrl T before doing the Ctrl U to make the latter work. Everything explained earlier works the same (12 unit errors) with the two Richard’s models, that you can download as explained later on.

But you can use the 64 bit versions.

When I tested one model that was using the same ageing chain than the world 3 model, I did that test, and saw three variables that were in the first view of the model, the population view. It was the three maturation rates of the ageing chain. The error unit came from subtracting a 1/year mortality from a dimensionless unit. When one calculates the number of people maturing, one must calculate the number of people ageing from one cohort to the next. The people ageing are those who are still alive at the end of the time step period, one semester. The people still alive at the end of the period are those alive at the beginning of the period less those who died in the period which is equal to population less (population multiplied by the mortality per semester) and the mortality per semester equal the mortality per year multiplied by year per semester which is ½ years which is the time step.

And having the number of people still living at the end of the semester, one must divide it by the number of semesters of the cohort, to calculate the portion of people leaving the cohort for the next one. And the number of semesters is equal to the number of years multiplied by 2. And the result must be again multiplied by 2 to calculate the rate per year instead per semester. The result gives the equation population * (1 – (mortality of the cohort * time step) / size of the cohort in years. For the 0 to 14, the size is 15 years. The equation is dimensionally consistent). Population(year) * (1 (dimensionless) – (mortality of the cohort * time step (1/year * year ) ) / size of the cohort (year).

The equation is not only correct dimensionally but corrects the wrong result delivered by the original formulation. For instance if one forces with the original formulation, an epidemic on the 0 to 14 cohort, that kills half of the children in a semester, the maturation during this period is equal to 0, because the mortality per year is equal to 1, which makes the mortality per semester equal to 0.5.

But the maturation cannot be equal to 0, because some of the people still alive at the end of the semester are maturing. If one forces the total of people of the cohort dying, which makes the mortality per semester equal to 1, and the mortality per year equal to 2, The people of the next cohort are becoming younger: the maturing is negative. To avoid that error, it would have been better to make all the calculations, considering the mortality per semester or by time step. And the mortality per time step becomes dimensionless, it is a simple percentage. The error comes from a confusion between the mortality per year with the mortality per time step.

One can argue that the error came from changing the time step from one year to half a year. This is partially true, because in the case of a year time step, the results were correct, but the unit error was still there and should have warned the author of a possible bad formulation. To resume my point of view, I consider that the authors were poorly motivated by getting a really reliable model, because if it was the case, they would have made these very simple verifications. I am sure that there are many other equation bugs in the model. I do not even speak about verifying the structure of the model which is much more complicated.

I had recently finished a model and while analysing it, I found a big equation error. I then realized that I was working with a model which I was not 100 percent sure that there was no other bug equation and decided to verify the entire model of about 900 equations heavily subscripted.

I started by verifying the constants then the initials than the variable directly dependant from a constant or an initial etc…. When verifying an equation, I had already verified precedingly the input variables of the equation. That job took me 6 weeks, and I found at least 5 big mistakes changing the results of the model, changed multiple name variables that were badly named, and simplified multiple formulations making them easier to understand and particularly to verify.

In fact, during the months of building the model, I did not make sure that every single equation added or modified was hundred per cent bug free.

I have noticed that in all the public or not models that I took the time to verify, I have always very quickly found equation errors.

To my opinion, this state of the SD field, is damaging for its scientific reputation. For me SD, is a field based on solid scientific foundations, but that is not used scientifically. And it is one of the main reasons explaining the subject of this thread, even if it is not its root cause. The root cause comes to my opinion, from the fact that the authors considering that their models will never be implemented, do not see the interest of increasing considerably the difficulty of the task, by trying to build a really scientific work.

For me the solution, would be to concentrate on less macro topics much easier to solve and implement, before trying to solve more difficult ones.

Anyone interested can go the the Vensim forum www.ventanasystems.co.uk/forum/ and find two models that can be downloaded based on Richard models, and with the bis model, the only change to simulate an epidemic with the time of the epidemic and the level of the mortality and see the results of the maturation when setting the mortality to 1 or 2 per year. the model ter, has the proper equation for the maturation, and one can see the maturation is now correct, when one chooses a mortality.

I tried to upload these models, but the mdl extension is not accepted. My name on the Vensim forum is LAUJJL, if you do not find the post.

Best.

JJ

#22

Thanks for your thoughts on the World3 model.

If you have found an error in someone’s model, then in my opinion the most productive thing to do is discuss this discovery with the creator of that model. Thank them for their work, explain what you found, and review why you feel that making the change would make it a better model. This is how scientists collaborate.

The fifty-year-old Limits to Growth World3 model is an iconic model with enormous impact. Countless negative criticisms have been directed against it. Yours is one more. But the only valid criticisms are those based on the purpose of the model. How well was the purpose achieved?

As stated on page 21 of Limits to Growth, 1972, the World3 model:

“…constitutes a preliminary attempt to improve our mental models of long-term, global problems…. Our world model was built specifically to investigate five major trends of global concern – accelerating industrialization, rapid population growth, widespread malnutrition, depletion of non-renewable resources, and a deteriorating environment. … The model we have constructed is, like every other model, imperfect, oversimplified, and unfinished. We are well aware of its shortcomings, but we believe that it is the most useful model now available for dealing with problems far out on the space-time graph.”

Based on that stated purpose, the model succeeded. It launched and focused a global realization that long-term environmental sustainability can be modeled, that that knowledge provides a firm foundation for policy, and that humanity must act now to avoid catastrophic collapse later. Technically, the model was so accurate (though that was not a goal, but a byproduct of good modeling practice) that Graham Turner, in On the Cusp of Global Collapse? Updated Comparison of The Limits to Growth with Historical Data, 2012, found that “The Limits to Growth standard run scenario produced 40 years ago continues to align well with historical data….”

In other words, have you found “a difference that makes a difference”? That is what science is most interested in. It is the question we must all ask ourselves, as we decide what to pursue and what not to pursue.

#23

Hi

Thank you for your answer.

I have when I first saw that error, followed the right way to deal with that problem.

The date of this discovery is in 2011. During the 2011 Washington SD conference, Kaoru Yamaguchi asked me to help him testing his financial model of the USA exposing his theory of money as debt and gave me its source to be able to test it.

Coming back to Strasbourg, I started the testing. The first thing that I did, was unit testing putting the unit testing in Vensim to strict testing. It is something that does not require any work and can still help find some possible bugs. When doing the test, I found several unit errors and I started to verify the equations that were at the beginning of the model. I found very quickly the origin of the unit error, correcting an equation that was delivering wrong errors and not dimensionally consistent.

I sent an email to Kaoru, explaining very precisely the error and its correction and explaining how an extreme condition test made people in a population cohort become younger instead of getting older.

Kaoru contested the error, probably not making the effort to study my report and told me that it could not be an error, because he had copied the ageing chain in his model exactly from the world 3 model. I stopped testing Kaoru’s model, being disappointed by his reaction and the lack of trust of my work. But I looked at the World 3 model and found the same error in it. Not knowing the author of the world 3 model (probably a team-work) I exposed the error in the SD forum. I very quickly got and answer from Bob Eberlein who recognized my ‘refinement’ as correct, but finally did not recognize an error. Since this time the error has never been corrected in the World 3, even if the correction was very easy to perform: three equations to modify. I recognize that the SD community is not willing to recognize that the World 3 model might be bugged, but this is not my conception of a scientific behaviour.

Of course, the error has probably no effect on the general purpose and results of the World 3 model, but I have a serious doubt on a model where some basic easy tests have not been made and that would have needed much more difficult tests to perform. I could have judged that it is an affair of hazard, but I have observed for 20 years the papers published in the SD conferences and have always found bugs in the models published of a certain size that I took the time to study.

So, I think that I found a ‘difference that makes a difference’, because I do not think that the world 3 model was built correctly and deserves the critics you mentioned. And that if it had been built correctly it would be more recognized by the scientific community and would be less criticized.

I have always thought that before exposing complex theories, the SD community should start first to build models with equations that generate results coherent with the inputs of the equation and with the name of the variables: a variable badly named is as bad as a variable with a wrong equation.

Best.

JJ

#24

Thanks, but you haven’t actually described what the World3 model “error” is and why it makes a significant difference. What nodes are involved? What equations need changing? What units need changing? Does the error affect graphical output or what? Why is error correction important?

The standard procedure in business projects when a discovery like this will be shared with many people is to write up a concise report. Updates can then be added as necessary. Without a report, you are reinventing the wheel every time you talk or write to someone about the error. The relevant knowledge is unorganized. That’s not efficient and wastes time.

Regarding: “Not knowing the author of the world 3 model (probably a team-work) I exposed the error in the SD forum. I very quickly got and answer from Bob Eberlein who recognized my ‘refinement’ as correct, but finally did not recognize an error.”

Bob is a sharp modeler, a wizard programmer, and a really nice guy. I trust his judgement. Without the report I don’t know the details of the situation, but it sounds like Bob, and before him Kaoru, thought the “error” was not that significant in terms of model impact. But yet to you it is significant. That’s the kind of thing the report should describe in detail. That report might be a good opportunity to start a new thread.

Hope this helps!

#25

Hi Richard.
In my preceding posts, that you may have not read fully, I have given the full explanation of the equation errors and why they are important.
But to resume my opinion simply: I do not trust a model with unit errors, even they are strict unit testing errors because they may hide equations or conception errors, which is the case with the world3 model.
Regards.
JJ

#26

Hi
I will resume my opinion about the reasons why SD does not solve big social problems, being myself, an SD user not selling the method to anybody. This is the opinion of a buyer of the method which will not be shared by other SD members who are all method sellers.
I have never seen an SD model about a macro-social problem that I judged as being able to deliver policies worth being applied.
Why?
For four main reasons:
First: SD is based on the principle of endogeneity that supposes that the main factors driving the reality modelled are endogenous in the model. Taking in account the complexity of social problems, finding the right boundaries guarantying this principle is difficult: too large boundaries generate huge models difficult to manage and costly, too narrow ones leave too many factors exogenous.
That choice must be justified by adequate studies, experimentations, taking into account the resources available and the purpose of the model to be sure that the boundaries chosen are the best possible choice, or are at least not too far from it.
I do not remember having ever seen such a study.
Second: SD is based on the principle of the homogeneity of the stocks, because of the general treatment of these stock their outputs in particular. This necessity obliges to split heterogeneous stocks into homogeneous ones. To what extent this splitting must be accomplished must be too justified, by different means, study, experience, taking into account the resources available, the purpose of the model, etc. like the first reason.
I do not remember having ever seen that done.
Third: probably the most important reason is about the structure of the model represented by its CLD. In general, models have a CLD, but the reader is more or less obliged to accept it, even if a lot of different structures can be imagined for the problem and the purpose of the model.
Here again, the CLD chosen must be justified, by different means, more or less of the same nature of the precedent ones. Different structures may generate very different behaviours, and it is critical to chose a good structure averaging all the other possible structures.
Fourth: in the models, there is no consideration with the capacity to implement concretely the policies proposed. This is too an important consideration. Models propose policies that should be implemented but not that can be implemented. The lack of general consideration for concrete implementation explains the difficulty of SD to exist as a practical method.
I have rarely seen models proposing policies to be implemented and in particular looking adapted to a possible implementation.
When one considers the three first reasons, the different possibilities should be studied as are studied the parameters with sensibility analysis, trying to find some central choice adapted to the purpose of the model and the resources available (financial, human etc.).
Respecting these three principles is of course extremely costly in time, money, and human resources and for this reason, is never respected. I think that this is the reason of the very low impact of SD on big social problems. It explains too the difficulty to respect the fourth reason exposed.
The profession, in its desire to solve big social problems, without having the resources needed, has made the choice of delivering models of low quality and level of scientific rigor.
I do not know the resources allocated to the building of the world3 model, and due to this limitation, I cannot blame its authors. They did what they could do. Comparing that model to my experience even if it is not one from an expert, I evaluate the cost of building such a model able to propose real implementable policies at a minimum of 10 million dollars and probably much more. I have too observed over the years, that the same limitation has generated practices that are far from scientific: lack of mathematical rigor for example and finally a kind of acceptance of the impossibility to build models that can be applied: SD is more generally used by researchers, with no purpose of any implementation. Why bother about rigor which is costly and time consuming, when one knows in advance that the model will never be applied?
I do not believe that the present trend towards low quality will change in the near future. I will never see this changing.
I observed too, the increase of influence of qualitative system thinking which is a consequence of the impossibility to achieve correct results with quantitative methods. I personally do not believe much in system thinking alone even if a good system thinking study is better than a bad quantitative one.
System thinking is good to expose a problem, help understanding its nature, difficulties, but this is always qualitative. I do not believe that it can deliver good policies without the help of quantification and the help of the rigor of mathematics.
The solution? Concentrate on problems that can be solved with the resources available, and there are plenty of them, and forget showing models that will never be implemented by anybody.
But due to the inertia and the already long time of existence of the field, I do not see the way to achieve this.
JJ

#27

My point of view about the root cause of the lack of influence of SD on social problems is:
The complexity of the majority of models is far beyond the capacities of modelers and/or end users (clients, decision-takers, problem-owners) to build or use them.
The solution: keep models complexity within the above mentioned capacities.
JJ

#28

Hello Jean-Jacques,

Merry Christmas, Happy New Year, and thanks for thinking your viewpoint over and offering a summary!

Looking back at this long thread, we covered a lot of ground, but did not reach any consensus on next steps. I wonder if we can do better with further discussion, using your “my point of view” summary.

Let me try to help by offering a reframing of your point of view. The purpose is to have a well-described problem to solve. This is often called the problem definition step. How about this:

The inability of analyses based on system dynamics to effectively solve difficult large-scale social problems, such as environmental sustainability, appears to be very much related to problem complexity. Present approaches to applying system dynamics to this class of problem fail due to overwhelming problem complexity.

This is because as models grow increasingly complex in order to mimic and correctly explain problem behavior, that complexity may pass a point where the model is too complex to understand, by modelers and/or model users. On easy, less complex problems, this point is not reached. This allows model examination to lead to effective problem solutions (aka policies). However, on highly complex problems, this point is reached long before the model is capable of offering the insights needed for effective solutions.

Now about your “The solution: keep models complexity within the above mentioned capacities.”

Reframing again, I think this means “The solution is to somehow keep models understandable (for modelers and users), no matter how complex the models become.” This is often called general solution strategy. It’s a starting point for further work. If sound, it can save a lot of time.

Do these two restatements of your viewpoint offer a good, well-focused starting point for productive discussion?

Thanks,

Jack Harich

#29

Hi Jack
Thank you for your answer and happy Christmas and/or New Year for you and all those who will read this thread.
You have pretty well resumed the definition of the problem. I would add that it is not only the complexity of the problem that is the cause of the problem, but the way that the circumstances are taken into consideration during the modeling process. One of the main circumstances is the fact that clients think that not being themselves able to integrate all the factors influencing the reality, they expect modeling to do it effectively with success and think that if it does not do it, it will fail. They want the model to do what they cannot currently do.
The solution is to keep the model understandable for modelers and users but not how complex the model becomes. I think that the solution is to keep the model complexity within the capacities of the modelers and users. These capacities (time, experience, motivation, financial resources, number of people concerned etc.) vary from modelers to users. I think that it is better to understand a model generating policies highly suboptimal because of its simplicity but easy to implement than a model delivering more optimal policies but that are not well understood and more difficult to implement.
About the word of ‘understanding’ I mean intuitive understanding based on the knowledge and experience of reality and not the understanding of the mathematics of the model.
I think that we can start from that point, even if it needs still some clarification.
Best regards.
JJ

#30

Hi JJ,

I appreciate your earnest efforts to tackle a problem that is central to the use of SD: Inability to solve difficult large-scale social problems, apparently due to the effect of high problem complexity on model complexity, understandability, and manageability.

However, I do see a problem. Every time I read one of your posts, I have to translate its words and concepts to correct well-articulated English. Other readers on this forum will probably have the same problem. This may explain why no one besides you and me are involved in this new discussion so far.

For example, you wrote below: “You have pretty well resumed the definition of the problem.” Resumed is the wrong word. It should be structured or articulated. But those two words have very different nuances. I have no idea which you are thinking.

The next sentence is: “I would add that it is not only the complexity of the problem that is the cause of the problem, but the way that the circumstances are taken into consideration during the modeling process.” I have no idea what “the way that the circumstances are taken into consideration during the modeling process” means. Is it your personal relationship to a client? Is it the the client’s unique situation? Is it the influence of the local economy, a nearby war, or what? Or is it perhaps the modeling process that is used?

The next sentence is: “They want the model to do what they cannot currently do.” I have no idea what this means. You need to be explicit about what “do” means.

And so on for most of the the remaining sentences.

May I suggest that if you wish to have an efficient and effective conversation about this problem on this forum, that you form a partnership with someone who knows your native language and knows English fluently, and have them rewrite your posts before posting. They should not just correct wording and grammar problems, but write in an articulate, easily understood manner. This is exactly what the System Dynamics Review requires of submitted papers. Non-native English speakers are suggested to use their English Language Editing service.

Please don’t be offended. I see great value in what you are trying to accomplish here. It would be tremendously helpful if on this forum we could work together and make some large progress on this problem.

Looking ahead, so far your problem solving approach is to start intuitively throwing out ideas on what you think are the causes and solutions. There are several problems with this approach: (1) There are no analytical, evidence-based reasons given for the causes and the solutions selected. (2) There are no concrete examples to clarify and begin to prove various claims, such as why this is a cause and why this solution strategy will likely work. (3) There is no central problem solving process used. There is only intuitive identification of causes and solutions.

Good news! Each of the above three problems is solvable. Once you have solved your non-native English speaker problem, the next step would be to choose a problem solving process to drive the project. This step can be made much easier by writing up a case that illustrates your problem well. Then we can work together to choose/design a process that fits the problem. I’d estimate you can do this in less than ten pages. Working with a case allows us to first solve the specific case, then other cases, and from there generalize to an overall solution that fits the entire class of problems.

The case(s) would contain sections like these:

  • The client problem to solve.
  • The history of trying to solving the problem via SD modeling.
  • Exact description of where in the project history various types of failure occurred.
  • Additional text as needed to completely describe the case.
  • Relevant artifacts, like data, graphs, models.

If you want to proceed in this direction, I’d suggest starting a new thread, where you announce/describe the project and provide a link to the case PDF.

If you don’t want to proceed in this direction, fine. I can understand. But I hope you proceed, as there is lots of potential here.

Warm regards,

Jack Harich

#31

Hi Jack

When using resume I meant summarize. Summarize is translated in French as ‘résumer’.

About the circumstances it is everything that can influence positively or negatively the modeling

process.

About ‘They want the model to do what they cannot currently do’

I meant They want the model to do (take everything into consideration) what they cannot currently

do (take everything into consideration).

But you set the purpose of this thread at a level that I will not be able to reach.

As you said English is not my native language and with so complex topics, it is a severe disadvantage.

You must understand too that I have not the motivation to write a 10 pages report.

I am not an academic used to write papers.

It is then better to stop my participation to this thread here.

Not to mention that I will probably not be able to post comments next year, having decided not

to renew my membership.

Best regards.

JJ

#32

To summarize.
Before saving the world; system dynamicists should first buy a french fries stand and learn to peel potatoes.
JJ

#33

Hi everybody.
One of the reasons of the difficulty to solve large social problems is the number of shareholders concerned with the problem and the fact that these shareholders have generally different point of views.
The solution is to reduce the number of shareholders and reduce their differences of point of view.
And one solution is to give to the regional people the power to decide about their problems. This is true democracy, something that you can see in Switzerland. And things are not so bad there.
Regards.
JJ

#34

Working with Superintendents of Public Education, in order to include System Thinking in cirrculum.

Emphasized 21st century jobs require ability to study comples systems using packages such as Vensim.

any suggestions???

#35

opps forgot the i curriculm