SPC Financial, Migdal Insuran Financial Architects, Inc Financial Arch Vision Capital Interactive Financial Advisors Interactive Fi Cambridge Inve Financial Engi Capital World Investors Capital World Apriem Advisors Apriem Advisor Lazard Freres Gestion S. Lazard Freres Marvel In First City Cap Dana Investment Advisors, Inc. Dana Investmen Truepoint, Inc.
Truepoint, Inc First Heartlan Harel Insuranc Sigma Planning Corp Sigma Planning Apollo Managem Jasper Ridge Partners, L. Jasper Ridge P Tanglewood Wea IMA Wealth, Inc. IMA Wealth, In Wallace Capita Narwhal Capital Management Narwhal Capita Chevy Chase Tr Raymond James Ipswich Invest National Asset YHB Investment Northwest Bancshares, Inc.
Northwest Banc Endurance Weal Johnson Financial Group, Inc. Johnson Financ EMG Holdings, L. EMG Holdings, Diversified Trust Co Diversified Tr Verus Financial Partners, Inc. Verus Financia Sageworth Trust Co Sageworth Trus McKinley Carte Corundum Group, Inc. Corundum Group Norinchukin Bank, The Norinchukin Ba Accuvest Global Advisors Accuvest Globa Charles Schwab Fiera Capital Corp Fiera Capital Decatur Capita American Resea Cohen Capital Management, Inc.
Cohen Capital Retirement Planning Group Retirement Pla Drexel Morgan Empirical Wealth Management Empirical Fina Asset Management Group, Inc. Hall Kathryn A. Hall Kathryn A Brookmont Capital Management Brookmont Capi Financial Advantage, Inc. Financial Adva Accredited Investors Inc. Accredited Inv Mizuho Securities Co. Mizuho Securit Parallax Volat PineBridge Investments, L. PineBridge Inv Liberty Mutual Sompo Asset Ma Institute for Sage Advisory Edgestream Partners, L.
Edgestream Par Berkshire Mone Peregrine Asset Advisers, Inc. Peregrine Asse Safra Asset Management Corp J. Safra Asset Redhawk Wealth Advisors, Inc. Redhawk Wealth Schwab Charitable Fund Schwab Charita Rezny Wealth Management, Inc. Rezny Wealth M Advance Capita Banco BTG Pact Coons Ad Administradora General de Fondos Moneda S. JGP Global Ges Candriam Luxembourg S.
Candriam Luxem Resource Planning Group Resource Plann Westside Inves Crossmark Glob Financial Advisory Group Financial Advi Waratah Capital Advisors Ltd. Waratah Capita Family Management Corp Family Managem United Asset Strategies, Inc. United Asset S Creative Planning Creative Plann New England Pr Ramsay, Stattm Nadler Financial Group, Inc.
Nadler Financi Madden Advisory Services, Inc. Madden Advisor Hudock, Inc. Hudock, Inc Ltd, Zurich Bank Julius Ba Eos Management, L. Eos Management Goelzer Invest Capula Management Ltd Capula Managem Thomasville National Bank Thomasville Na Capital International Investors Capital Intern Pachira Investments Inc. Pachira Invest Desjardins Glo Boltwood Capital Management Boltwood Capit Newman Dignan FDx Advisors, Inc.
FDx Advisors, Sensible Finan Telos Capital Management, Inc. Telos Capital Colonial Trust Advisors Colonial Trust Financial Mana King Wealth King Wealth Mirae Asset Gl Addenda Capital Inc. Addenda Capita M Holdings Securities, Inc. M Holdings Sec Winfield Associates, Inc. Winfield Assoc Beacon Capital Blue Fin Capital, Inc. Blue Fin Capit East Coast Ass Carlson Capital Management Carlson Capita Addison Capital Co Addison Capita TCI Wealth Adv Horan Capital Yakira Capital SRS Capital Ad Blume Capital Management, Inc.
Blume Capital Vontobel Holding Ltd. Vontobel Holdi First Horizon Advisors, Inc. First Horizon Oakworth Capital, Inc. Oakworth Capit Heritage Wealth Advisors Heritage Wealt Retirement Systems of Alabama Retirement Sys Riverview Trust Co Riverview Trus Cordasco Financial Network Cordasco Finan Beta Wealth Group, Inc. Beta Wealth Gr Financial Advi Somerset Trust Co Somerset Trust Verity Asset Management, Inc. Verity Asset M Capital Advisors, Ltd. LLC Capital Adviso Moreno Evelyn V Moreno Evelyn GM Advisory Group, Inc.
GM Advisory Gr JJJ Advisors Inc. JJJ Advisors I Friedenthal Financial Friedenthal Fi Capstone Finan Wunderlich Capital Managemnt Wunderlich Cap Brookstone Capital Management Brookstone Cap Tarbox Family Office, Inc. Tarbox Family Shoker Investm Permanens Capital L. Permanens Capi Under Special Management Mivtachim The Riley Wealt Community Bank of Raymore Community Bank Bellecapital I Bridgewater Advisors Inc.
Bridgewater Ad Riggs Asset Managment Co. Riggs Asset Ma Family Firm, Inc. Family Firm, I Abacus Planning Group, Inc. Abacus Plannin Point72 Asset Management, L. Point72 Asset Johns Inve Mengis Capital BTC Capital Ma Stratos Wealth Fragasso Group Inc. Fragasso Group Garde Capital, Inc. Garde Capital, Elite Wealth Management, Inc. Elite Wealth M Barber Financial Group, Inc.
Barber Financi Summit Financi Gibraltar Capi Ironwood Financial, llc Ironwood Finan Industrial All Dakota Wealth Management Dakota Wealth Northstar Group, Inc. Northstar Grou Leisure Capital Management Leisure Capita Godshalk Welsh Community Bank, N. Community Bank Willow Creek W Trust Co Trust Co Blue Chip Partners, Inc. Blue Chip Part Sunbelt Securities, Inc. Sunbelt Securi Patten Group, Inc. Patten Group, Towercrest Capital Management Towercrest Cap Koshinski Asse Cambridge Advisors Inc.
Cambridge Advi Analyst IMS In Diversified Portfolios, Inc. Diversified Po ETF Portfolio Man Group plc Man Group plc Giverny Capital Inc. Giverny Capita Glassman Wealth Services Glassman Wealt Ayalon Holdings Ltd. Hmm, I was right: I couldn't directly answer your question. I hope this explanation of why is somehow enlightening anyway.
Very interesting, Olivier, thanks for doing this. And let me congratulate both of you for putting your code out to be looked at. That takes great nerve to do. Well done both. Olivier, your solution exemplifies a thing that happens often Namely, the design we create on paper turns out to be more design than we need to solve the problem in hand. TDD has a strength in not causing us to do more design than we need, which is good because it can lead to solutions that are no more complex than they need be.
I'd far rather do too much than too little design. To me, Olivier's design is arguably "more procedural", but it is shorter and simpler and therefore has a lot going for it. This is a common outcome with TDD It can be argued that Ralf's design will be easier to extend. This may even be true. Nonetheless, that extra design is not carrying its weight now, even if at some future time it may.
One last point, which I'll put here rather than in yet another reply to Ralf. This careful paper design may have some good properties. I do not think "didn't need to refactor" is automatically a good property, however, because it depends on how long it took to get there and how long it takes to refactor. But more interesting to me are these facts: 1.
We don't know if Ralf's solution works or not: we do not see the tests. He says there are tests but we didn't get to see them. It sounds like the tests are created after the fact. There may be blind spots when we do that. In any case, we just can't tell. Easily enough corrected, and of course in a real situation we could browse them if we wanted to. Nonetheless, right here right now, we can't tell. We don't know why the solution is what it is: we do not see the progression which we can see in a series of tests.
This is uncorrectable. We can show evolution with TDD much more readily than we can see how the paper design evolved and why. The design as it stands has some rather visible flaws. Perhaps the most important of these is the preponderance of "internal" methods, which often signal an abstraction trying to appear. The most telling of these is the returning of a tuple, which is always a red flag in the code saying "I have an object in mind here but I didn't bother to make it an object".
In this case, if I understand the code correctly, we have a mantissa cleverly named Item1 and an exponent named Item2. These names do not communicate our design intention nor our implementation intention. We then have a separate method that slams values into that tuple. This is a violation of the Law of Demeter. A better design might encapsulate all of that behind some new class with mantissa and exponent, and methods internal to that class that set the components.
The result of all this this is that while perhaps the paper design saved a refactoring, it remains further than one might like to see from a design that correctly presents the abstractions that it contains. Now back to Olivier's program. It is in fact simpler I'm not sure.
On the one hand it has everything embedded in that single convert method but even for a lover of the ternary operator, that method is tricky to understand. It is recursive very cool and therefore harder to understand than Ralf's loop. We see here, I hope, that the question of "which is better" is quite complex.
We get to see a bit more of Olivier's thought process and really none of Ralf's because he did it on paper that we cannot see. Olivier's solution covers exactly what it needs to and no more: there seems to be no excess in it. Ralf's solution seems to me to have more design than was needed to do what it does, but I can't be sure because I can't see the tests. Furthermore, it contains more abstractions, which is not necessarily good, and it contains some very incomplete abstractions around the tuple and the demeter-violating method that hammers on it.
I'd have to say that I prefer Olivier's for being simpler, and agree with Ralf that another reader might prefer the other. More of Olivier's design process is visible and Ralf's design process is hidden. It's a hard question. Partly it is a matter of taste, but I think the partially-implemented abstractions, the violation of Law of Demeter, and the absence of the tests gives Olivier the win in this case, by a small margin, in this referee's opinion.
Your mileage may vary. You're right, it doesn't. And I actually believe that the recursive solution, while it has a certain elegance, is less robust when we realize that we'll have to face it in the future and figure out out again.
The looping solution is simpler in some important ways. Readers of Extreme Programming Installed Jeffries, Hendrickson, Anderson will recall the "Quick Design Session", which is a short period of time when the people working on some feature or features work through a design sketch prior to moving into code This is very important This may be true for you, but it is emphatically not true world wide.
Many individuals and teams understand how to show and manipulate "course-grained functional units" in code. It's not always the best way but sometimes it is. You, Ralf, assert on the one hand that you are very skilled in TDD, and yet on the other hand you state that you cannot work on both design and code at the same time, and that you cannot see the coarse grained design very well in the code. I can think of a few explanations for this.
One would be that your brain is simply not capable of doing what many other brains can do. I think this unlikely. My own model of what would cause you to say those things is that your skill at TDD is not as high as that of people who CAN work on design and code at the same time, and who CAN envision coarse-grained design while working in code.
You seem to take that as a bit insulting. I'm assuming that you could learn how to do this as well as anyone There can be value to this, though it may be quite difficult to bring it out in a written forum like this one. To compare the two approaches, we need to control for quality at least in the sense of running all the tests, and quite possibly for other matters of quality such as having all the abstractions needed and no more, and so on.
I'd like to see it done and I am not sure whether we have the time, the will, or the medium to bring it off. You guys have already gone further than most in displaying your code, and again, I congratulate you. If you can do more that will be great. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.
And then he feels that perhaps there isn't. I'm really not here to explain or justify Kent's words from over a decade ago. I'm not even always able to justify mine from the same era. What he says here is mostly true, I think, depending on what we mean by flattening the curve. The tests provide for increased certainty that changes have not broken things and increase speed of detection when they do.
This reduces the cost of change and the cost of error, flattening the curve. It is the name for the suite of behaviors that cover those topics in XP. Depends how much you want to flatten it. Comprehensive unit tests might do the same level of testing at higher cost. Changing the code without understanding behavior-preserving transformations might still allow us to make changes, but more slowly thus not flattening the curve as much.
Perfect up front design will not do this, from the viewpoint of XP and Agile, because a perfect up front design is not possible, b implementing a perfect design perfectly is not possible, c up front design is done at the moment we know less than we ever will again, and d things change anyway.
That doesn't mean we don't do paper design: we do and it is good. It just doesn't give us the things that TDD does and we need those things. Nothing is sufficient in software and never will anything be sufficient. We are always challenged by harder and harder problems and we must always be advancing our skill. We need to know how to do paper design early, and how to do it late. We need to know how to test early and how to test late.
We need to know how to write code to a design and how to evolve the code to include our new learnings. And a thousand other things. I do not prefer TDD because of faith, nor because of a lack of understanding how to do non-code design. On the contrary, over the half century I've been developing software, I have studied and developed as much skill as I could in every design technique that has attained prominence over that period I have found value in all of those.
I prefer TDD because, like Adan, having developed rather high skill in all the techniques out there, I find that a very liberal dose of TDD lets me go faster, with less stress and fewer mistakes than any approach heretofore. I have every reason to believe that the equation would hold true for anyone of roughly equal skill in all the various forms of design and coding under consideration. I could imagine that some special mental or physical characteristics might play in there.
For example, I would expect that a blind person would find UML design to be much less productive than a sighted person might. And I can at least imagine that someone's unique mental characteristics might leave one ill equipped to do TDD which to be done well MUST include focus on both detailed coding and what you have elsewhere called "coarse" design.
My experience in teaching TDD to many many individuals and teams has led me to believe that most anyone can learn it. Since many people can do in TDD what you say you cannot -- namely deal with detailed and coarse design issues in that "medium" -- my confident expectation is that these people learned something, attained some perhaps small skill, that you, as yet, have not. If there was just one guy standing here saying TDD can do what you say it cannot, it might be a tossup. However, I can do it, Adam can do it, George can do it and the many people we have helped learn it can also do it.
That makes me quite confident that you could do it too. I'm not sure why that seems insulting to you, but it seems that it does, and I regret that. I still think you could also do it if you cared to, and that it would change your preferences between paper and in-code design if you did. I'm perfectly happy if you choose not to. What I'm less happy about is that your description of the universe as being one in which no one can do coarse and fine design in the code is not the universe I'm living in.
If that were the case you'd be saying different things. You'd not be talking about how impossible it is to design this way but how inefficient it is. Yet your argument leans much more on possibility than it does on efficiency. Even then, it would be necessary to explain why, despite reports of TDD experts all around you, saying that it's the best way they know so far, they are all wrong.
The assumption that people who like TDD do so because they are "believers" is I like it because I've been doing it assiduously for 15 years. Prior to those 15 years I have worked just as hard learning every design technique as it came along.
But no one is saying TDD is "the pinnacle of design". We are saying that day in and day out, we find that doing TDD, full-on, with refactoring, gets our software done faster and better than when we do it with up front paper design and then "code slinging", and better than interleaving "code slinging" and paper design.
We still draw pictures of the code, or go chat about it, or get a really good design idea in the shower or on the drive home. And we build that design in a disciplined way that lets us incorporate not just those insights, but the ones that come to us as we produce the code in our incremental disciplined fashion. Doesn't imply that at all.
It happens that I've tried lots of ways and find this one to be substantially more valuable than you do I didn't come to TDD directly from gross hackery. I came to it with 35 years of experience doing all the disciplined design approaches that came along, and now I come to you with a full half century of that plus 15 years of developing my TDD skills in the same way.
Yes, I see that, and it really confuses me. I'm assuming something good about you, namely assuming that you have the innate ability to do what I can do, what Adam can do, what George can do, what many others can do, and use TDD more effectively than you've reported here.
And yet I have confidence in you I'm sorry about your feelings, but not willing to withdraw my confidence in you. It isn't the scientific method. It is more like a report from your future from someone who has been there.
And I've done TDD for a long time as well. I'm reporting that I find that TDD gives me certain advantages. I observe that you are not getting those same advantages in the same proportion. My best guess is that I have stumbled on a way of doing it which you have not. Another possibility, of course, is that you know a way of doing paper design that I don't know, that somehow multiplies its power.
Since you have repeatedly said things about TDD that are not true for me and lots of people I know, I lean toward the theory that there's at least one more TDD learning waiting for you. I could be wrong, and even if there is, you can decide not to seek it.
That's OK. I don't get royalties from people using TDD. I do care that whenever you, or anyone, says "TDD can't do this", and there are people out there like me who know that it can, one or more of us says so. TDD can be used to evolve a design faster than at least some people, highly skilled in paper design, can do paper design and then code to it. TDD can be used in situations where the requirements change, and therefore the paper design initially created needs changing, and it can let us do those changes without a rewrite.
It doesn't make much sense to me for you, or anyone, to suggest that it cannot do that, because people are in fact doing it. The fact trumps the theory. So whenever such a suggestion is made you can be sure that I, and the others who know how to do those things, are going to say "Hey, wait, we can do those things. It's just not true that the only way to go is that other way. Is the first red on paper?
Hmm, I hope not. I am very interested in hearing more about your technique, Ralf. It seems that every time I've tried to model and write clean code up front, I've stalled, stumbled, made mistakes, and bumbled around getting little done. Almost every time I've done a short design session, then TDD, I've finished my task, and been proud enough to share.
If you have a new way to model that I can practice and learn that will make me more efficient at building maintainable code than TDD, I'm all ears. Just saying "do more modeling up front" isn't it, though, because I've done that and then went to TDD. I always try to remember the goal when I write code. The goal is maintainability.
No other measure of good design matters, because the goal is keeping it easy to add new features. Again, it's viscosity working in your favor. And I'll say it again: If you have a new way to build maintainable code better then TDD, please share.
This has been a fascinating conversion, a little tense at times perhaps, but very interesting. Thanks all! Indeed, I had often focused too much on maintainability; I had used too many design patterns for the sake of maintainability when a few functions would have been simpler and sufficient. It probably didn't take me much longer to use those design patterns, but I wasted time.
I was attempting to anticipate changes that would never come. APPP puts it this way, '"Fool me once, shame on you. Fool me twice, shame on me. To keep from loading our software with needless complexity, we may permit ourselves to be fooled once. This means that we initially write our code expecting it not to change. When a change occurs, we implement the abstractions that protect us from future changes of that kind. In short, we take the first bullet and then make sure that we are protected from any more bullets coming from that particular gun" pg I think there are many barriers to doing TDD well, like the above.
How much is too much refactoring, for example? I overdid it, and I recognize that now. I had to learn to trust myself enough to leave the code as-is until a sufficiently strong stimulus came along at which point I could refactor it to a more reasonable design against real "changes of that kind. Ron, Thanks for your comment. I agree with your statement: the code is not easy to understand and needs further refactoring to make it understandable.
In my opinion, this is the kind of refactoring the most difficult to deal with: transform the code so that someone else will find it easy to read. One thing that usually helps me is to think about the model behind the code that comes out of the TDD and check if this model is visble just by reading the code. I should have done that with this program but I did not and the model "the digits in base N for a given number is the concatenation of the digits representing the part of the number grouped in packs of N units and the single digit for the remaining units" remained hidden.
Hi Kaleb! I couldn't agree more with what you just wrote, and it means I misrepresented what I meant. TDD helps me write maintainable code because it helps me keep the design simple. I can see how designing for maintainability could lead to over-designed code, but I would argue that over-designed code is less maintainable than simple TDD code. You've made me start to contemplate the meaning of maintainable, and that's a good thing! Off the top of my head, I meant maintainable in the sense that it's easy to understand, fix, and extend.
I guess that it conforms to the XP rules of simplicity? The reason we care about design is so we don't make a mess. We don't want to make a mess because someday we'll have to go back and fix or extend the code. I guess that's the goal? Don't make a mess? I believe it is. The reason I believe that is that I believe that comprehensive automated testing is necessary to get the level of feedback needed to flatten the curve, and these tests must be run very often At least on the order of a few times per hour.
Much faster should be possible. I have heard many claims that such aggressive testing is possible without TDD, but every time I have had an opportunity to investigate such a claim I have found that the team wasn't doing nearly enough "test-after" to get the benefits. At the very least some sort of aggressive automation is needed so that changes can be tested and shared immediately with the whole team.
Whether this looks like traditional continuous integration, continuous delivery, or something else I am not sure. I also think that having a collaborative work environment where I can simply shout when I need to have a discussion about something I am about to do is a huge benefit to flattening the curve. I have heard of teams managing this adequately when they were distributed, but, again, I have never witnessed it I have witnessed distributed environments where design was shared within hours or days, but not immediately.
Pair programming is also useful. In my experience good pair programming is at least as important to simple design in XP as test-first programming is. Having my pair chime in with a good idea at just the right moment or catch a mistake right when I make it is a huge factor in flattening the curve. What would you do with that answer if you got it? Part of the challenge of adopting XP piecemeal is that the practices support each other very nicely. There is a synergistic effect that happens when you are doing just enough of each of them that doesn't seem to happen when you do one or two in isolation.
That's not to say that everyone does or should do XP the same way, but when it works it works like a stew -- the right stuff in the right proportions, no single ingredient dominates. TDD is one of the more powerful flatteners. ATDD does. Continuous integration does, pair programming does, refactoring sans TDD does The cost of change topic is, in my opinion, mostly a red herring.
A sufficient reason for doing the simplest thing is that doing a more complex thing obviously takes longer. A sufficient reason behind YAGNI is that investing prior to the moment one needs the investment is wasteful. Similarly for LRM. When a team tries to sell "refactoring the code base" they are in very dangerous territory.
First of all the request tells me that the team have crapped up their code base, which is prima facie evidence that the team isn't really very good. Second, the only argument for refactoring after the code is crapped up is that we will go "faster, later". So this team that wasn't smart enough not to crap up their code base is now asking me to throw good money after bad so that they -- demonstrably incompetent though they are -- can work in some invisible way in hopes of doing a good enough job to speed up later.
I don't think so. A manager would, nine times out of ten, be a fool to agree to this. The right strategy when the code is bad is to make it better incrementally, by refactoring areas where new features are wanted, making them a bit cleaner and therefore making the next feature to pass through a bit easier to do , while continuing to deliver features, which is, after all, what the team is paid to do.
So if a team said to me "we need to refactor our code base", I would be pretty sure that they don't know what a good code base looks like, that they don't really know what they are changing or why, and that they don't understand the business's problem, which is to get new features soon. In short, that's a beginner's request and I would not be inclined to support it.
The curve never goes flat. The XP practices, done well, permit a team to produce features at roughly a constant rate over a long period of time. TDD is a big part of this. It isn't all of it, and you have to do it bloody well to get the benefits. I don't follow that but I think it's because I don't buy the cost of change curve argument as a strong motivator in any case.
This is called refactoring. Ralf isn't wrong that it is easier to refactor a paper design than a code design. It's not wrong to think on paper. It is, however, less than ideal to think very long without testing one's ideas in code. Speculative design is easy to get a bit wrong in many different ways. In the example Ralf bravely did, we saw partial abstractions plus over-design. When we combine that kind of thinking with the concrete code, we get faster feedback on the design. When we know how to TDD and refactor well, we don't need to stop thinking and we can make better decisions as to how quickly to move to code.
It is often useful to use a "paper" model to help us see what we have done, or what we should do. My only concern with Ralf's position is not the use of external models. Ralf: Note that I'm answering email randomly. I know you have a reply to me that I've not responded to and I'll do that shortly.
This one precedes that one and therefore does not include whatever you most recently said. I've found it works quite well with the developers communicating during coding. Have you tried that? Code is over-designed to the extent that it contains more classes, methods, or other entities than are required by its present level of function.
A simple example is the existence of unused code, put in because "we're gonna need it". Kent Beck's definition of "Simple Code" is 1. Runs all the tests; 2. Contains no duplication; 3. Expresses all our design ideas; 4. Minimizes the number of classes, methods and other entities. Over-designed code violates 4 by having more entities than are required to express the design, or 3 by having more design expression than required to support the current tests.
An example in the number conversion example would be the insertion of an abstract superclass and a single concrete subclass, before the code actually needs to support multiple numeric bases. Someone designing on paper might will discover the perfectly good design of using a simple concrete subclass for each numeric base, perhaps just containing the list of digits for that base, or any such implementation, inheriting from an abstract superclass. It would be almost impossible for most of us to build a single-base version without putting in the abstract superclass and the single subclass we really need.
We just couldn't resist. Until the day or hour or minute that that code gets the second concrete class, it is over-designed for its capability. Kate might even build the list of digits in line in the conversion class: I believe Olivier's program may have done that.
At some point during the implementation of the original numeric base, she would allow herself to "notice" that inside the conversion class there was a numeric base idea trying to get out. She might, at that point, extract a concrete "Base13" class. Then she would stop. But she doesn't need it now. So she doesn't put it in. Then she writes the first base-seven test.
Quite likely, it runs. Soon she expresses a number bigger than seven in her base seven tests Then Kate probably does one of two things: 1. Kate then goes Oh wow look, this design is starting to suck but she's not really surprised , and she refactors the if statement into the two classes. We should have put it in at the beginning. The TDD ritual is to wait for the code to tell us, whether we know or not. This gives us some things that we like: "First, we never have more code than we need to get to the current green.
This minimizes the code We can ship on any green, as soon as we have enough function. Since much of the code we will encounter in our lives will have design flaws, this skill at recognizing the need, and responding to it with the right refactoring, serves us well. That's just plain waste! With my way, I just type in the correct code to begin with. That's far more efficient. She says: "Perhaps not. If I can spend less time in up front design than I used to, I get my first tests running sooner.
Think of it this way: there is simply less code to type in to get base 13 running. With that, plus the shorter time I spend in up front design, I can get base 13 running faster than I could if I first spent more time up front, and then spent more time typing it in because it's a longer solution. I expect that it will work, and because I'm a good designer, I am usually right.
But I won't know how well that design really fits until I begin to build it, and to use it. If your life has been like mine, you found that those classes were useful, but hard to use and didn't really quite fit your problem. Early on, if your experience is like mine, your use of those classes was a pain. They were too complex to use readily.
Kate goes on: "I have learned that even when I do my own design, if I create the abstractions first, they often do not quite fit my desired usage. They get in the way a bit, slowing down my learning and my implementation.
So I've found that starting with the simple design and evolving it as I need it works better for me. Maybe so. But its not clear, is it? Another perfectly possible outcome is that a programmer using TDD would go faster to any given green bar, because they never write a line of code prior to that moment that they do not need. The TDD pracitioner starts out ahead, by working simply. They might stay ahead forever, by refactoring promptly, always ensuring that they have the minimal code for the moment.
I can design your pretty ass right into the ground. I'm saying that I'm pretty good at design and pretty good at TDD, and that for me, I go faster with a bit less up front design than I used to use. That lets me hit the ground running -- if I were into running which I am not -- and then my simultaneous use of the design and building of the design gives me better results.
I get faster feedback about how good my design is because I use it sooner. But of course our skills aren't quite the same. Depending how quickly you could get the first up front design right, or how quickly you could go back and forth between paper and code, you might come out ahead. This gives me time to do a little more typing. And, just possibly, because I always build the least code I can to do my current job, I might inject fewer stupid coding mistakes, or discover them sooner because my TDD tests tell me.
So I have less switching overhead, and maybe less debugging overhead. I might stay ahead forever because of lower overhead. If that's the case, again I'd stay ahead forever, simply because I start sooner and don't slow down due to under design. I've studied design more than most people and done TDD thoughtfully for a long time. For me, it works best.
I've seen it work best for a lot of other people, and my intuitive feeling is that unless someone designs at some incredibly high level, and is perhaps also really slow at editing I'm just saying that it works for me, and I have a pretty good sense of why. I wanted to share that with you. Do with it as you will. It makes me better to work the way I do, and that's why I do it. But I'm still not convinced TDD is better for me. I am the descendent of a long line of litigators, but I decided to get honest work instead.
And I don't know whether TDD is better for you. Everyone is different. All I know is that I have worked long and hard learning design and becoming facile with TDD, and it's better for me. I know a lot of people who have done the same and had the same discovery.
Or maybe that discovery still lies before you. It may not matter at all. Your work is good, and fast. Learning to shift more of your time from paper design to TDD design might make you better still, or it might not. Maybe you'll find my experience tantalizing enough to push further to find out more about what's true for you, or maybe you won't. Either way, you get to decide. The rest is waiting. The business value of a post-hoc refactoring effort is very difficult to identify, primarily because it delivers no feature value now and only promises or hints at faster feature development later.
The best option is to have the code's design always just as good as the present level of function calls for, neither too much, which reflects wasted not yet paid-off effort, or too little, which slows us down. Of these two, a little over-design is far less harmful than insufficient design, because usually for some definition of usually it will pay off pretty soon.
It's possible to create amazing structures that are unnecessary forever, but a customer's impatience for stories can probably hold that off. If, however, we give our code to the customer with an inadequate design, we are developing the habit of handing it over too soon, and we are slowing ourselves down, step by step, for the future. Cleaning dirty code up is something that we need only do once. Working with dirty code slows down every feature that passes through that area, forever.
So keeping the code clean is cheaper overall than letting it slide. Yes, TDD does that. And the best way that I know to harness the ability to change the code as needed is to clean up the code we're working on. If the design has slipped a bit in some area, then the next time we do a feature in that area, we clean things up a bit. It's like the boy scout rule of leaving the campground cleaner than you found it. Over time, the areas we work in will become quite clean, and if other areas that we never visit never get cleaned up, no harm is done because we're not slowed down by them.
I don't think you misrepresented it at all, I just didn't think it was sufficiently defined to help some of the less-experienced TDDers, like myself perhaps. MinimumNumberOfClassesAndMethods quotes Ward Cunningham: "Better to create the one class needed now and let the future programmer refactor it into the classes needed in the future. I am suspicious of the one responsibility rule because I've seen beautiful code that violates it. Maybe the balance is being able to say, "this is fine for now but we can easily refactor to XX if we ever need to do YY or ZZ.
Thanks for your thoughts Curtis. Responsibility Driven Design Kata. Reply to author. Report message as abuse. Show original message. Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message. Hello all, I am wondering whether anyone can direct me to a kata that would work on the way a developer can use TDD especially to extract certain roles and responsibilities from one class to other collaborators Does that make any sense?
Corey Haines. I like Conway's game of life for this. Thanks a lot Corey :. Matteo Vaccari. It makes a lot of sense IMO. In part, the fact that you use a kata to. Matteo, Thanks a lot for you thoughts! Funny that you say that this is something you can add as a restriction to your kata- that's something I've been thinking of lately. Or am I missing something here? Regards, n. So yes, in fact you are right; when I'm facilitating a dojo I often have to point out that until you remove all duplication you should not write the next test.
The fact is that most TDD beginners, including at times me, don't see how important it is to remove every tiny bit of duplication, even when it seems you're overdoing it! So the reason for the additional rules is to push learners to strive harder. For instance, after doing the OCP Dojo for a while it's become more natural for me to extract functionality to new methods and then to new classes.
Ron Jeffries. Hello, Matteo. On Tuesday, May 10, , at AM, you wrote:. Steve Tooke. Maximise clarity and remove duplication! Ralf Westphal. Curtis Cooley. Peter Gfader. I wouldn't say that I find TDD overrated, but I agree, it is good to step back and think about the problem a bit more before starting to hack away I did the "alpha-end converter" kata in TDD manner. During doing this Kata I got into this "brute force coding" mode, where I didn't think too much, but I just wanted to get the test to green.
It took me a while to get over this hurdle and I couldn't solve the kata in that 1 hour session This annoyed me a lot, so I stepped back, thought about the problem on paper for a couple of minutes and had the solution there A friend of mine had the same experience and he suggested me the "Call the shots" technique for beating this.
Note: By working in pairs that problem wouldn't probably happen, because the observer is already focused about what should happen while the other one is typing. I find that this is a problem with kata that drive towards an algorithm, wrt katas that push me to write a maintainable solution. I don't think that getting folks to recognize that we're asked to convert from base to base is particularly interesting. It gets more interesting if you start asking yourself: can I change base?
Can I also support Roman numerals? Hours and angles base but with more than one symbol per "digit"? Can I support different number systems in the same program? On Friday, May 13, , at AM, you wrote:. I just recently went through a real world example of why refactoring works.
I was tasked with implementing a feature set, that existed in one application, in another application. Not wanting to repeat myself, I decided to model a framework on the whiteboard that could then be reused in other applications. Hi, Curtis! Hello, Ralf. On Saturday, May 14, , at AM, you wrote:. Peter Gfader You said: "I wouldn't say that I find TDD overrated, but I agree, it is good to step back and think about the problem a bit more before starting to hack away Description of the.
Ralf Westphal You said: "refactoring should be avoided altogether if possible. Some believe that multitasking can result in time wasted due to human context switching and apparently causing more errors due to insufficient attention. Here is the gist of Fowler's idea I have turned excerpts of his afterword into bullet points : 1. Programming is hard.
It sometimes feels like trying to keep several balls in the air at once: any lapse of concentration and everything comes tumbling down. TDD helps reduce this feeling, and results in rapid unhurriedness really fast progress despite feeling unhurried. This is because working in a TDD development style gives you the sense of keeping just one ball in the air at once, so you can concentrate on that ball properly and do a really good job with it.
When you are trying to add some new functionality, you are not worried about what really makes a good design for this piece of function, you are just trying to get a test to pass as easily as possible. When you switch to refactoring mode, you are not worried about adding some new function, you are just worried about getting the right design.
With both of these activities, you are just focused on one thing at a time, and as a result you can concentrate better on that one thing. Adding features test-first and refactoring, are two monological flavours of programming. A large part of making activities systematic is identifying core tasks and allowing us to concentrate on only one at a time.
An assembly line is a mind-numbing example of this - mind numbing because you always do the one thing. Perhaps what test-driven development suggests is a way of breaking apart the act of programming into elemental modes, but avoiding the monotony by switching rapidly between those modes. The combination of monological modes and switching gives you the benefits of focus and lowers the stress on the brain without the monotony of the assembly line.
On Saturday, May 14, , at PM, you wrote:. Hello J. XP is meant to be used when requirements are vague or changing 2. XP is most valuable when uncertainty is greatest 3. The key technical premise for XP is a flat 'cost of change' curve 4. The basic problem of software development is risk. Business changes—the software is put into production, but the business problem it was designed to solve was replaced six months ago by another, more pressing, business problem.
XP was invented to address risks. We need to make our software development economically more valuable by spending money more slowly, earning revenue more quickly, and increasing the probable productive lifespan of our project. But most of all we need to increase the options for business decisions. Software project management can be looked at as having four kinds of options: 1. Option to abandon 2. Option to switch 3. Option to defer 4.
Option to grow see the actual book for more details on these options The worth of options is generally dominated by uncertainty in the eventual value of the option: the difference between the cost of the option and the value of exercising the option. XP maximises the value of the project analyzed as options by providing: 1. Accurate and frequent feedback about progress 2.
Many opportunities to dramatically change the requirements 3. A smaller initial investment 4. The opportunity to go faster The greater the uncertainty, the more valuable XP is. One of the universal assumptions of software engineering is that the cost of changing a program rises exponentially over time. The problem is that this curve is no longer valid.
|Rainsberger td direct investing||King Wealth King Wealth Unlike up-front work, which is always insufficient, TDD plus refactoring allows us to apply what we learn, when we learn it. Blue Fin Capital, Inc. North Star Investment Management Corp. Crossmark Global Holdings, Inc. Vision Capital Management, Inc. Dakota Wealth Management Dakota Wealth|
|Hlt global ipo||First of all the request tells me that the team have crapped up their code base, which is prima facie evidence that the team isn't really very good. It shouldn't be that rainsberger td direct investing to work out the intent of the code because the unit test should describe this. When you switch to refactoring mode, you are not worried about adding some new function, you are just worried about getting the right design. There is an award for doing the right combination of practices: success. This is a common outcome with TDD The greater the uncertainty, the more valuable XP is. We can show evolution with TDD much more readily than we can see how the paper design evolved and why.|
|Forex trading platform canada||So the reason for the additional rules is to push learners to strive harder. Addenda Capita Asset Manageme Off the top of my head, I meant maintainable in the sense that it's easy to understand, fix, and extend. Sompo Asset Ma Either way, you get to decide. I don't find this pattern to match the definition of "multitasking".|
|Rainsberger td direct investing||How to forex gold|
|Warren buffett and value investing||We don't know if Ralf's solution works or not: we do not see the tests. Pegasus Partners Ltd. I think that's about me, not about the reality of pictures and code. Aldebaran Fina Hmm, tests and refactoring. So if a team said to me "we need to refactor our here base", I would be pretty sure that they don't know what a good code base looks like, that they don't really know what they are changing or why, and that they don't understand the business's problem, which is to get new features soon. That I should try to overcome.|
|Rainsberger td direct investing||Clark Capital Management Group, Inc. It's also TDD. A friend of mine had the same experience and he suggested me the "Call the shots" technique for beating this. So keeping the code clean is cheaper overall than letting it slide. The bulletproof vest fitting is that most TDD beginners, including at times me, don't see how important it is to remove every tiny bit of duplication, even when it seems you're overdoing it! This is not some amazing new skill that only a genius can attain. Therefore there is value to producing that code so that it is strongly supported by tests, and there is value to knowing how to smoothly improve the design of code so that we can put in our new design ideas safely at any time.|
|Rainsberger td direct investing||Retirement Planning Group Retirement Pla Stockman Wealt I did the "alpha-end converter" kata in TDD manner. When I stare at and empty flip chart and start to feel anxious about what to write, I can often relieve my anxiety by writing the simplest test that fails and going from there. Doesn't imply that at all. Hello all, I am wondering whether anyone can direct me to a kata that would work on the way a developer can use TDD especially to extract certain roles and responsibilities from one class to other collaborators|
|Forex broker top rating||Tall tree investment management|
It was a classic case of market demand outpacing supply, which in turn appeared to help drive higher stock prices and investors getting in the action. When we look at the sector from the perspective of age demographics, it was Boomers who jumped on the Energy train. This investor group had the highest allocation to Energy stocks relative to the other age cohorts. When we look at the sector from a geographic perspective, home bias is apparent. Energy sentiment was pushed higher by investors in the energy producing province of Alberta.
We also saw investors in Ontario positioned in Energy, surprising given that investors in this province typically hold a lower allocation in their portfolios. Nationally, we saw both Active Traders and Long-Term investors bullish in this sector. The largest investor group to exhibit negative sentiment were Boomers , who had the highest allocation to Materials of any age group. Looking at the sector from a trading style perspective, Active Traders, who jumped on the run up in base metals, showed the biggest drop.
B was among the top net sold. The materials sector was also weighed down last month by gold and gold equities. Gold bullion posted its worst monthly value decline September really was a tale of two sectors, each battling to have the most influence on investor sentiment. Energy prices rose on supply and demand factors, while Materials prices dropped on fears of a pull-back on economic growth in China. And the winner is… Energy trumped Materials, winning the title for being the biggest driver of overall sentiment.
October sentiment sits firmly in bullish territory. Likewise, the TSX was up 3. In October, we also saw an interesting trend emerge in investor confidence: the generational divide. There was strong demand for both old and new guard securities, with Boomers rocking it old school, and Gen Z and Millennials getting in their feelings with the next generation of companies.
Energy sentiment continued to lead confidence in all sectors for the second straight month. Energy supplies were still constrained globally and combined with the demand surge on the back of a rebound in global mobility, more pressure was placed on prices. How did the generations respond? AQN — Utilities sector , which has positioned itself in the renewable energy space.
Make no mistake: all age groups pushed this sector and these securities up the ranks. The younger generations were simply more focused on high growth stock in other sectors while the Boomers and Traditionalists focused on these stable, dividend stocks. Geographically, this same trend of 'trade where you live' emerged, with energy stock demand most apparent in the energy-exposed provinces of Alberta, Manitoba, and Saskatchewan.
We would classify the improvement in sentiment as broad-based. In other words, the generations agree: this is a materials market. All investor age groups showed improved sentiment, with Boomers showing the greatest improvement. In terms of trading style, long-term Investors favoured materials, though active traders also rode the wave to a lesser degree. The provincial breakdown was a little bit more obvious, as investors in Ontario, BC, and the Territories showed the greatest improvement in sentiment in this sector.
More interesting correlation not causation is these companies have huge social footprints, which is where younger investors comfortably spend more time than their older investor counterparts. Were older generations more conservative with a focus on wealth preservation and younger generations ready to take risks on innovation?
October's numbers seem to say as much. What was perhaps more interesting were the layers on top of that - some of the factors which may have driven investor decisions, from familiarity buy what you know to home bias buy where you live. And dare we say - to hopes and dreams buy the world you wish for.
Let's start with the most optimistic region. That honour went to Atlantic Canada. That's right. Experiencing the most positive sentiment was Information Technology IT. In fact, demand for stocks in the IT sector was over four times the level of the next most favoured sector in Atlantic Canada Materials. Given PYPL and NVDA are high beta stocks beta measures price swings relative to the market and with the market swings towards the end of the month, those stocks were even more volatile than normal.
We saw Investors in all four provinces embracing IT, though to a much lesser degree than Atlantic Canada. The biggest change from last month came from the sentiment of investors in Ontario and BC. There are all sorts of ways to slice sentiment: age range, trading style, sector, or region.
Use the filters in the charts below to find your people and your family to see how they felt in November. Now we will modestly quote our own paper, Understanding Investor Behaviour :. By looking at self-directed investor trading activity, we can see how people react to economic and financial market events…".
In short, Atlantic Canada felt just fine in November. They liked what was going on and showed it with their trades. It'll be interesting to see if they stay positive. This continues a streak of nine straight months of positive investor sentiment. Money, money, money Our top sector in December was Financials. Canadian banks posted their year-end results at the tail end of the year. There are a few ways to slice this insight. Demographically, Traditionalists and Boomers led the move.
Geographically, home bias played a significant role, as investors in Ontario led the move to the Financial sector. This follows past DII geographic observations, which revealed that there may exist a strong home bias that causes Ontarians to be significantly overweight in bank stocks given most bank head offices are located in Ontario. Feeling confident in the IT and Consumer Discretionary sectors, investors favored companies that were poised to benefit from increased consumer spending.
These stocks were popular amongst Active Traders, more specifically Gen Z, who jumped on recent trends. Spreading holiday cheer were investors in Ontario and BC, who contributed most to the positive sentiment in these sectors. Materials stocks were the least favoured in December. This was apparent mostly with the Boomer generation and investors that live in BC.
The negativity was also apparent with Active Traders who are generally quick with the sell trigger whenever bad news hits, such as, the BC floods, which may have had an outsized influence over the Materials sector this month. Gold stocks such as Barrick ABX and Kinross K were among the top bought as investors added to their gold positioning on rising covid risks.
Houston, we have a problem The Energy sector lost some steam in December as investors exhibited negative sentiment. With Omicron denting travel plans, the expected drop in global mobility seemed to negatively influence energy demand. This is significant since over two thirds of all energy demand comes from global mobility.
Air travel, driving, and shipping are big energy users and stall with lockdowns. Finishing on a fundamentally high note Our year-end felt like a textbook case of macro-economic fundamentals. Big picture topics and events played a clear role in what investors did and how they felt. Outperforming year-end earnings results for banks led to positive sentiment for Financials. Despite a year packed with macro-economic events, Canadian retail investors closed out feeling positive.
Yay us! And a happy belated new year. The question then is how did the DII remain in positive territory? Let's find out. Going off the grid The answer becomes clearer when you look at the bigger picture. January saw investors selling their U. So, while the market news was all about the dip, the DII sentiment stayed north. In particular, U. The massive selloff in Technology came on the heels of growing fear that the then upcoming U. Federal Reserve rate hiking cycle would take the wind out of their sails.
If we look at the sentiment for the Technology sector only, the DII dropped 18 points, to — Apple Inc. The quickest demographic to drop them like hot cakes? Gen X and Boomers. Within the active trader and long-term investor groups, sentiment was most negative for the latter. On a geographic basis, the drop in sentiment was mostly driven by investors in Ontario. Home team advantage While investors were saying salut to their U.
Tech stocks, they were saying bonjour to Canadian Energy and Financials. January saw oil prices skyrocket reaching a 7-year high! This caused a domino effect of improving supply and demand fundamentals—throw in rising geopolitical risks between Russia and Ukraine in the mix and Canadian Energy stocks must have been looking pretty good to investors as the sentiment moved up.
What was the Energy sentiment you might ask? Active traders were clearly chasing trends as they drove sentiment higher. When it came to location, investors in Ontario were most optimistic on Energy, followed by the Energy heavy provinces in the Prairies and Territories. A potential reason for this uptick? The prospect of higher interest rates and, in turn, higher bank profits.
On a demographic basis, Financials were popular across all age groups, led by Boomers and Traditionalists. Just as with Energy, active traders were the ones pushing Financials sentiment higher. With so many financial institution headquarters based in Ontario, this may be why investors from this province were arms and legs above the rest when it came to sentiment towards Financials.
They dropped U. So there you have it. Since the beginning of , the markets have stumbled and soared in reaction to one event after another. We started with the threat of higher interest rates from the U. Federal Reserve, sky high inflation, and Omicron. Then, on February 24 th , Russia invaded Ukraine. With the Russia — Ukraine conflict raising the risk of financial market stress, the question is what could have driven such strong gains in sentiment?
Commodity prices lead The threat of sanctions on Russia caused massive price movements in a host of commodities, ranging from barrels of oil to bushels of wheat. We found that on a demographic basis, Gen X and Boomers were the drivers of positive commodity sentiment. Geographically, investors in Ontario, BC, and the Prairies showed the most positive sentiment for Energy, whereas it was really only investors in Ontario that drove positive sentiment for commodities.
Trading in Tech for Energy The most negative sentiment of the month came in the I. These stocks may have been impacted by expectations of higher interest rates, with the US Fed set to embark on an aggressive tightening cycle. As Tech firms are typically high growth companies, higher interest rates have an outsized impact on the valuation of these firms. We found that Boomers were most likely to sell Tech stocks, followed by Gen X and then Traditionalists. We also found that investors from provinces that bought into Energy stocks Ontario, BC, and Prairies , actively sold Tech stocks.
For some, this implies there may have been a sector rotation strategy afoot. Forging ahead through uncertainty The last few years have been anything but normal, however, as the DII has shown us month over month, this does not necessarily mean Canadian investors are headed to bearish territory — in fact, investor sentiment continues to rise.
It has shown that events, such as geopolitical crises and global pandemics, can affect the market differently and may even be different depending on which country the market resides. When it comes to the current geopolitical crisis, some Canadian stocks have seen growth, Energy and Material stocks in particular, and this is because of an abundance of resources that exist within Canada.
The oldest generation — the Traditionalists — ranked the most positive in sentiment — given their exposure to commodities. The youngest generation — Gen ZY — were the least positive, as tech stocks continue to stumble. What goes up March felt like opposite day in the markets. The sectors and securities we were accustomed to seeing on top continued their downward trajectory while previously sluggish sectors and securities kept going up.
Let's break it down. We could see this played out as the Canadian market outperformed globally. The focus on commodities, from nickel, gold, and copper to wheat and lithium, was in sharp contrast to these securities being outperformed by high-flying Tech stocks. B and Barrick Gold Corp. Tech reboot? Or just Control Alt Delete?
Why is this relevant? When we look at what's happening at the big tech firms, we may be looking at I. Apple or Consumer Discretionary Amazon. Which brings us to the third most popular sector this month: Communications. This sector includes both Meta Platforms Inc.
FB and Alphabet Inc. GOOG , and both made an appearance in the top five bought. Over in I. The best of times, the worst of times The Canadian resurgence was a validation to many investors who had shown confidence in the market in the previous month. But we also saw currency, inflation, interest rates, supply chains, and more lurking in the background— you know something might set off a tumble, but you don't always know what or when.
It is a bit bumpy. So, hold on and enjoy the ride. For the first time in two years, investor sentiment dropped into bearish territory. As we sliced and diced the data, we saw a few glimpses of bullish above neutral sentiment in niche groups. For the most part, however, each sector, demographic, and trading style were bearish. As the Markets Rotate The flight to safety, one of the four cornerstones proxies that make up the DII, had a major role in our bearish sentiment.
For example, there was a rout in growth stocks as investors left behind volatility and rotated to energy or to the sidelines in cash or cash equivalents like fixed income. Specifically, investors took their money out of Technology and put it in smaller sectors like Energy. Because the Technology sector had been such a big part of people's portfolios, this shift was notable: tech heavy Nasdaq, for example, had the worst month since Going Up Now over to Energy.
Supply chain issues and more signs of reopening, such as airline travel, looked to push the demand and sentiment for Energy to the top position. The second highest sector, Consumer Defensive, showcased how investors were still looking for profits. Veru Inc. Copper was down, platinum was down, even gold lost its luster. In value, they're still up year over year, but the pull-back may be related to a retail investor desire to avoid exposure to these materials given the slowdown in China.
Financials, another closely watched sector, retreated from its mid-February peak and was of heightened interest as investors expected the Bank of Canada to continue withdrawing monetary stimulus to tame inflationary pressures. Long-term investors and active traders were negative. The sentiment of all age groups dropped, with Boomers being the most pessimistic. And so we waited, sitting on our piles of cash, wondering when things would go back to normal — and when they do, if it is the same normal or a new one altogether.
Filter your results. Pick up to 3, then press GO. Unless changed, your choices apply to each tab. Change in asset allocation. Change in investments by industry sector 1. Compare our self-directed investing services and choose the one that's right for you. The WebBroker trading platform can make investing easy and accessible for investors with any level of experience. Discover educational material to help you become a more informed investor.
Bullish, bearish, or somewhere in between? Information made easy; insights made effortless. Get a monthly sentiment score to see how self-directed investor behavior trended. See where others invested their money …by filtering for the information that matters to you. And if you want more, learn more, …through daily, live online workshops and videos.
Wonder how other self- directed investors felt about the market? More specifically, it looks at monthly activity of self directed investors on TD's web broker. For example, last month it showed us what these investors were doing to which we can extrapolate how they were feeling The index has a range from being the most bearish to plus one hundred being the most bullish.
But the DII itself is made up of four separate measures and each one of those can be bullish or bearish. So let's take a look. The second measure looks at chasing trends, which shows what was bought on the way up and what was sold on the way down. Third, we look at equities bought at extremes, which measures what was bought at 52 week highs and what was sold at 52 week lows.
And the fourth measure is flight to safety, also known as risk on risk off. And that means people looking at moving into safer investments, things like cash, GICs and fixed income. The overall DII can also be applied to different groups and different sectors. For example, it looks at which sectors investors felt most bullish or bearish about.
It also looks at sentiment by type of investors, those who trade frequently to those who tend to buy and hold and also by age to see which age groups are feeling more bullish and bearish. Finally, every month it will show which securities were most bought and sold in that month. The TD Direct Investing Index DII is designed to be an educational tool that seeks to measure the attitude and behaviour of self-directed investors in the prior month—and present it in a format that is easy to read and understand.
The DII devises how investor sentiment has been trending: Are investors optimistic or pessimistic about recent market conditions? Monthly sentiment is measured by examining four distinct investor behaviours from the previous month. Were investors:. The result of each measure is compiled and averaged to quantify investor sentiment on a scale from very bullish to very bearish—and anywhere in between. See a detailed description of the terms used in the DII. See our report Understanding Investor Behaviour.
Legal disclaimer. Just watch what they do. The actual investment decisions of individuals may be the most honest representation of investor feelings and beliefs. By looking at self-directed investor trading activity, we can see how people react to economic and financial market events. From this we can aim to uncover sentiment patterns of self-directed investors. TD Direct Investing is in a unique position to help further the understanding of Canadian self-directed investor behaviour.
As a market share leader in the Canadian market number of clients and number of trades 0 , our data available is the richest of any Canadian self-directed brokerage firm. The DII is based on aggregated and de-identified trades placed by TD Direct Investing self-directed investing clients, with reference to contemporaneous market data.
We also present a literature review on sentiment indices, compare our index to other sentiment indices, and detail our methodology for determining the index component proxies. The data within the DII that was used to create the index can also be segmented to observe the activity of self-directed investors by different age cohorts and even geographic regions.
By performing these analyses, we reveal the investing behaviour of self-directed investors to so that they can make the most informed investment decisions for themselves. At the end of this paper, we share an example of how we use the rich dataset to present initial findings that contribute to the literature on investor diversification by demonstrating regional bias of Canadians based on their home province or territory.
Following the pioneering research by Baker and Wurgler and referring to the study in Delong, Shleifer, Summers, and Waldmann , investor sentiment is an emotional state of an individual investor. This emotion is tied to the investor's expectation about the future return of their investment assets. As this is a non-physical, intangible psychological state, it is incredibly difficult to measure sentiment directly. An alternative to the challenge of directly measuring the emotions of investors is to define proxies which can infer sentiment.
The simplest way of doing this is to survey investors. Unfortunately, this is an imperfect method as investors may articulate one sentiment at the time of surveying but may act very differently when they actually come to invest. Therefore, researchers and behavioral economists alike approach survey results cautiously Baker and Wurgler In contrast to using the self-reported survey responses to gauge sentiment, we use aggregated trading activity of self-directed investors to reveal sentiment.
Nearly all the sentiment indices found in the literature presented above focus on abstracted market-level information as opposed to the actual transactions of an investor analyzed in aggregate. Rarely is it seen that a study focuses on the actual trading behaviour of the investors to probe elements of behaviour and sentiment. The most probable reason behind this is that analyzing investor actions and behaviour requires a highly granular data with records for every transaction e. Such data is likely difficult to come by.
In cases where transaction data are available, the set of traded securities are different in nature from one study to another i. What's more, published studies focus on aggregate trades and do not make the distinction between trades by individual self-directed investor and institutional investors e.
Crucially, the TD Direct Investing platform is utilized by an investor population composed of only self-directed investors and no institutional investors 1 , and therefore can better represent the behaviour and sentiment of self-directed investors. We are uniquely able to leverage both granular transaction records of self-directed investors and market data to help address the limitations in prior work that focus on aggregate market metrics, specific security markets, and institutional investors.
Finally, it is noted that there is a lack of consensus—in terms of definitions, assumptions, methodologies and results—in the literature describing stock market dynamics. In part, we attribute this disagreement to the breadth and complexity of stock market parameters, for which researchers do not have complete ability to interrogate. All of this makes it difficult to come to a coherent and somewhat universal conclusion about the necessary components of investor sentiment.
In other words, there is no clear consensus on how to determine sentiment—either qualitatively or otherwise. Despite differences, it is possible to generalize principles of investor sentiment to the self-directed portfolio, even if the trader population and dynamics are not perfectly aligned with those in other studies. A fundamental assumption shared by most researchers is that all investors have the common goal of maximizing their return regardless of the degree of rationality of their actions.
Another assumption that we believe would be fair to make is that the amount and quality of information available to the self-directed investor comparable across markets, though it may be utilized differently by investors. There are three main approaches to measuring investor sentiment. The first approach is survey-based. An example of such an approach includes the American Association of Individual Investors AAII Investor Sentiment Survey, which is a weekly poll of its members' opinion of the stock market over the next 6 months.
The drawback of such approach, as mentioned above, is that investors' response may not align with how they will act. The second approach measures sentiment either based on market variables such as stock prices, trading volume i. Perhaps the most widely referred study in the market variable-based approach is that of Baker and Wurgler , In this study, several market variables are proposed in constructing the sentiment:.
The authors apply a principal component analysis PCA, see glossary on the proxies, which have previously been regressed against key macroeconomic factors to remove business cycle effects. The resulting first component is treated as the composite index for sentiment. Note that, some proxies in the above reveal the sentiment in a delayed manner. As such, for these proxies, their lagged values have been considered.
In another study that leverages market variables, Meier suggests utilizing stock price data to measure overconfidence in the stock market. Meier argues that confidence has two components: strength and weight. The former measures the extremeness of the available evidence recent performance versus performance in a benchmark period while the latter measures the actual credibility of the evidence are the gains because of good market conditions or rather due to investor's skills?
Here, evidence is simply the difference in past gains. Meier follows the argument of Griffin and Tversky that investors attribute their recent gains to their own skill rather than other factors good market conditions or luck and therefore characterizes overconfidence by high strength and low weight. In another study, Lemmon and Portniaguina suggest that investor confidence can be measured by considering the variation of small stock prices. This is particularly true for non-institutional investors i.
Four activities, namely: buy-call, sell-call, buy-put and sell-put of options are used to measure the sentiment see glossary. Naturally, buy-call and sell-put contributes positively to the sentiment while sell-call and buy-put negatively. The author also shows that the proposed transaction-based metric has a high correlation with respected market sentiment indices as well as market returns.
This metric is also referred to as the options trajectory. In a separate study that utilizes self-directed investor transaction data, Kumar and Lee propose a sentiment index for self-directed investors based on the Buy Sell Imbalance BSI metric. This metric first calculates the disparity between buy and sell volumes for each stock on a daily basis. The buy-sell imbalance of each stock is then normalized see glossary by stock's volume of transaction.
Finally, the normalized imbalance for each stock is aggregated to give the buy-sell imbalance of the whole market. The third approach measures sentiment based on unconventional, and often unstructured data such as textual data from social media, or factors such as seasonality Kamstra, Kramer and Levi , sporting events Edmans, Garcia and Norli or the occurrence of major events in the news Li et al.
Perhaps one of the most famous such indices is the BUZZ NextGen AI US Sentiment Leaders Index which scrapes the Web to identify the most mentioned stocks by taking into account the network influence ranking of the users talking about the stocks , scans social media to check what is said about these identified stocks and then uses natural language processing in order to determine whether the sentiment expressed on these stocks is positive, negative or neutral. The DII adopts the market-based approach where we consolidate the self-directed investor aggregated trading activity with market variables and leverage the combined data to construct a sentiment index.
Compared to the studies surveyed so far, our approach captures the sentiment of a sample of Canadian self-directed investors rather than that of the overall market, which would include both self-directed and institutional investors. We utilize anonymous, security-specific transactions to build proxies for self-directed investors from different segments such as age group, geographical region and trading style. We aggregate the proxies to build a sentiment index that reflects the sentiment of Canadian self-directed investors by different demographics.
In the construction of the DII, we employ a series of procedures to select the relevant proxies, from which the sentiment index is composed. In the first step, we filter out proxies that are biased to a particular segment of self-directed investors. Specifically, we ensure that the selected proxies reflect the sentiment of the long-term, average self-directed investor. In this preliminary stage, we also consider the availability of the data required to construct the proxies.
In the second step of the procedure, we employed a proprietary statistical process which utilizes dimensionality reduction techniques PCA to further reduce the proxy space. In the final step, we use a qualitative approach where we use expert judgement and effective challenge with subject matter experts to assess and validate the rationale for including various proxies in the sentiment index. Following the three steps, we reduced the pool of candidate proxies 2 down to the four final proxies:.
Once the set of proxies is determined, we construct the sentiment index by combining the proxies. As mentioned previously, investor sentiment is an abstract concept that is difficult to quantify. Therefore, there are multiple approaches to evaluate the validity of the sentiment index. One common way is to compare the sentiment index to a broad benchmark index that represents the market and reflects its sentiment e.
Even if the market is composed of both institutional and self-directed investors, it is expected that some relationship exists between the market return and the sentiment of self-directed investors. This is important because if sentiment doesn't move with the equity markets, the effectiveness of the sentiment index is lessened.
Because we think there is a relationship, we conduct the following tests as evidence. The resulting index can be seen in Figure 1. This is the raw outcome of the model, where positive negative values mean bullish bearish. This is a popular index of Canadian publicly traded securities.
In developing the DII, we noted there is a tremendous wealth of research on the relationship between sentiment indices and stock market returns Brown and Cliff, ; Smales, ; Bathia et al. Here we show the impulse response of the two variables on each other. We also use an Ordinary Least Squares OLS, glossary method for the contemporaneous impact, where the TSX can influence sentiment within the same month rather than with a lag.
Below are the impulse response functions of the VAR estimates. Given the scale of self-directed investors relative to that of institutional investors, we would not expect a strong response of the market to the impulse of self-directed investors' sentiment. In other words, this data suggests that self-directed investors tend to react to the market, but rarely drive the market.
The above charts show the impulse response of the TSX Index and DII Sentiment if one variable moves, how much of an impact does it have on other variables. Bottom right, the impact of DII Sentiment on itself.
Readings above below zero show a positive negative relationship of one variable on another. Given impact of the TSX on the DII sentiment, as demonstrated by the VAR estimation, we further this analysis by conducting an ordinary least squares estimation glossary. The independent variables, lagging sentiment and a contemporaneous TSX same time , are regressed against the monthly sentiment score. Here we find statistical significance: the monthly sentiment score is explained by its lagging score and the concurrent month's market performance.
Taken together, these tests indicate that each month's sentiment is influenced by its historical score as well as past and present market performance. This is consistent with our expectation that a relationship should exist between the monthly sentiment and the concurrent month's market performance; however, the relationship is not always exact given the fact that institutional investors are usually the market movers.
In addition to the sentiment-TSX interdependency validation test, we have also assessed the stability of the model over time each month we look at the output of the model to make sure it is logical. The stability of the model is important as a degraded model can no longer represent the current sentiment accurately.
The results of the stability test suggest that the model performance when trained on the entire dataset is very much aligned with the performance of the model when trained on a fraction of the data on an ongoing basis. A model monitoring plan is also in place to ensure that in the future, we can identify any model drift in order to recalibrate the model in a timely manner. As outlined at the onset of this paper, one of our key objectives was to understand the degree of diversification within self-directed investor portfolios, and whether bias influences investment decisions.
Access to this information goes to the spirit of the creation of the DII, as an educational tool where we share insights with self-directed investors so that they can make the most informed investment decisions for themselves. For the period of December , we use the transactional records of self-directed investors and perform additional analysis to derive insights of self-directed investor behaviour, providing additional context to interpreting the DII.
In Canada, there was an interesting divergence between age categories for certain sectors. In Table 1 we see that Financial stocks were clear favorites for all self-directed investor groups, but we find that the allocation to that sector increased with age. Though we cannot determine the exact cause this, we do know that Financials provide greater dividends than other sectors, and income streams can be an important factor for Canadians in older age categories.
Additionally, Financial corporations typically have been in operation for a long time, and the comfort with such storied companies may impact investor preference. Technology was second highest allocation for Canadians younger than 35 and those to This compares to Energy being the second most popular sector for Canadians 51 years of age and older. This divide may be a result of younger self-directed investors openness to new companies, versus older self-directed investors comfort with historically successful companies.
We can also breakdown the sector allocation by geographic region. Here we found significant geographical differences across provinces and territories. In Figure 2 and Table 2, we show the sectors with the largest differences between regional allocations.
Table 2 shows the complete dataset as of December For the Materials sector, we note that self-directed investors across the Territories and Saskatchewan had much greater exposure than self-directed investors in other locations. Materials companies can include those related to metals such as gold, copper, and iron ore and non-metals such as potash and diamonds.
For Energy, investors in Alberta, Saskatchewan, and the Territories were most exposed. Energy companies largely include those related to oil and gas. For Financials, we saw that self-directed investors in the Maritimes and Ontario had the most exposure. These include companies such as banks and insurance companies. It may suggest that investors from geographic locations where employment and economic production are dependent on a specific sector tend to have an overweight of equity exposure to that sector as well.
This is best exemplified by the Energy sector. Here we saw that self-directed investors who live in the geographic areas with the most economic exposure to Energy were also most overweight Energy in their portfolios. In this paper we use the aggregated and anonymous trading data of TD Direct Investing self-directed investing clients to help build a measure of investor sentiment. This index helps determine how self-directed investors were feeling about equity markets over historical period.
We have also presented details of the dataset to help improve our clients understanding of risk-taking and investor bias. This has direct ties to help our clients be aware of the benefits of portfolio diversification. Here we show how investment exposure to certain higher risk sectors was influenced by age.
We also show that self-directed investors may have a home bias by overweighting sectors that are economically more prominent in the province or territory in which they live. The evidence of age impacting risk-taking and geographic home bias is important in our understanding of self-directed investor behaviour.
Normalization: In statistics and its applications, normalization refers to a process whereby values are adjusted to allow for meaningful cross-comparisons. Normalization may be implemented to bring different measures to a notionally common scale to prior to averaging. Ordinary least squares estimation: This is a statistical technique used to understand the relationship between a dependent variable and one or more independent variables.
The coefficient tells us the estimated magnitude of effect of the independent variable on the dependent variable, as well as the directionality e. On the other hand, standard error tells us the precision in which the coefficient is measured. If a coefficient is large compared to its standard error, then it implies that there is some relationship between the independent and dependent variable.
Principal component analysis: This is a statistical technique used to reduce the dimensionality of data with the goal of increasing interpretability while preserving as much information as possible of the original dataset. This is achieved by creating new variables, or principal components, that contains information of the original variables and maximizes the information or variance. Proxies: In the DII context, proxies are investing behaviour measures based on trade activity that allow us to make inferences about investor sentiment.
Put options: A put option gives the owner the right to sell an underlying security at a specific price until a certain date. When selling a put option sell-put , agree to buy a stock at an agreed-upon price. It's also known as shorting a put.
The seller is anticipating that the stock price will rise in value. Buy-call: a bullish trade that gives the buyer the choice to exercise the option, allowing them to buy the underlying security at the strike price. Sell-call: a bearish trade that if exercised by the buyer, forces them to sell the underlying security at the strike price.
Buy-put: a bearish trade that gives the buyer the choice to exercise the option, allowing them to sell the underlying security at the strike price. Vector autoregression estimation: This is a statistical technique used to capture the relationship between multiple quantities as they change over time.
This technique is useful for understanding how a variable is a function of past lags of itself and past lags of the other variables. The information contained herein has been provided by TD Direct Investing and is for information purposes only. The information has been drawn from sources believed to be reliable. Graphs and charts are used for illustrative purposes only and do not reflect future values or future performance of any investment.
The information does not provide financial, legal, tax or investment advice. Particular investment, tax, or trading strategies should be evaluated relative to each individual's objectives and risk tolerance. The DII is for informational purposes only. Any information provided through the DII should not be considered an investment recommendation, nor is it an offer, or solicitation of an offer to purchase or sell any investment fund, security or other product.
Investors should not take the historical information as an indication, assurance, estimate or forecast of future values or future performance. The DII should not be used as individual financial, legal, investment or tax advice. Information provided through the DII is subject to change without notice. They may also make a market in, issue, and participate in an underwriting of such securities.
A high degree of risk may be involved in the purchase and sale of options and may not be suitable for every investor. The risk of loss in trading securities, options and futures can be substantial. Investors must consider all relevant risk factors, including their own financial situation before trading. A higher level of market knowledge, risk tolerance and net worth is required.
TD Bank Group means The Toronto-Dominion Bank and its affiliates, who provide deposit, investment, loan, securities, trust, insurance and other products or services. We arrive at the most salient proxies using the 3-step proxy selection process. They are subject to change based on future data.
To determine the monthly sentiment of self-directed investors, we analyze these criteria for the last month:. Sold : Measures net equity demand—whether investors were buying more or selling more in a specified month. I'm Anthony Okolie. And this is a look at the TD Direct Investing Index for the month of July, a snapshot of how self-directed investors were feeling based on what they did.
And after two months of bullish sentiment hovering around 50 in the TD Direct Investing Index, keep in mind, it's a range from minus to plus July sentiment dropped 50 points from June to only plus 2 from bullish to neutral. And it's a low sentiment in three months.
A few key points here. Sentiment on materials, particularly commodities, was the lowest of all sectors, pulling the overall sentiment down. When you break things down by activity, long-term investors who have less than 29 trades in the previous quarter were more pessimistic than active traders, the traders that had 30 or more trades in the previous three months.
When you look at age, baby boomers who were born between to were the most pessimistic age group. Now let's take a look at the sentiment proxies or the DI ruler board. This is the total investor activity that makes up the sentiment.
And there was more chasing stocks at 52 week highs, came in at plus 47 score in July, which is interesting, but at the same time, the flight to safety move from investors, as you can see, pulled the overall sentiment down from bullish into neutral territory. Now, in terms of sector activity, investor sentiment on materials was very negative, pulling sentiment down. And health care and energy were also slightly negative, with the exception of gold stocks like B2Gold Corporation and energy giant Suncor, which saw heavy buys.
Meanwhile, technology was by far the most positive sentiment, with the rest of the sectors very close to neutral. Now let's dig into more detail specifically of who was doing what. Here, we have two main categories. In terms of trading styles, active investors were more optimistic than long-term investors. And active traders, their DII sentiment came in at plus 8. When look at the key tickers, most bought was Suncor and AMC.
When we look at long-term investors, the sentiment came in at negative 6, and most bought included Air Canada and Suncor, among the most sold, Brookfield Properties and Apple. And in terms of age, everyone was neutral to slightly bullish, with the exception of boomers, who are comparatively quite negative. And boomers came in with the sentiment of negative In terms of key tickers, most bought was Manulife Financial and Enbridge Among the gen Z and millennials, they were slightly bullish, with a DI sentiment score of plus 3.
Among the most sold was Shopify and Good Food. Now let's break it down by province. Sentiment was the most pessimistic in Ontario last month. Ontario and BC sentiment ended up slightly negative, while the Prairies, Atlantic Canada, and Quebec were slightly positive.
Interesting to see Air Canada was the most traded stock right across Canada, but Suncor was the top bought stock, and Inter Pipeline was most sold in the Prairies. And that's a roundup of what happened with DII in July. For more information on the DI index, please visit the link at the bottom. Hello, everybody, I'm Kim Parlee. And this is a look at the TD Direct Investing Index for the month of August, a snapshot of how self-direct investors were feeling, but based on their activity in the month of August.
There's three key takeaways we want to talk about. First, the DII came in at plus 9 for August. Now keep in mind, the range is minus being the most bearish to plus being the most bullish. Plus 9 for this month of August came in just a couple points higher than we saw in the previous month.
So we're seeing a little more bullishness in the market. But it is well below the plus 50 that we had in August at this time last year. Second takeaway-- for the first time in nine months, long-term investors were more optimistic than active traders. And third of all, traditionalists, who are of course a little older, perhaps a little wiser, were the most optimistic.
And Gen Xers were actually the most pessimistic in the month of August. All right, let's dig into some of the internals in terms of what makes up the DII. These are the four components, a couple of notable pieces we'll pull out here. We're continuing to see more chasing stocks at week highs. As a customer, you can rely on the provision of safety and security from TD Bank itself.
The integrity of the online brokerage is never in question. It has an industry-leading response time to answer client queries, and it offers online live chat functionality. Questrade is the most popular online broker platform in Canada. In my opinion, Questrade is the best platform available. While it may not offer the comprehensive educational and market research tools that TD Direct Investing has to offer, it offers substantially lower costs for you to use the platform.
You can learn more about the platform in my Questrade Review. Wealthsimple Trade is another non-bank online brokerage in Canada that has become vastly popular among self-directed investors. It is one of the most inexpensive discount brokers you can use.
Wealthsimple Trade relies on foreign currency conversion fees to make revenue, and it does not charge you any commissions for trading stocks using the platform. Since it relies on currency conversion fees, it does require you to trade US-listed securities with the platform.
You will essentially be paying 1. You can learn more about the online broker in my Wealthsimple Trade Review. With an overwhelming number of online brokers available in the market, a firm with the backing of a reliable name can be a welcome sight for many self-directed investors. The comfort comes at the cost of higher fees compared to non-banking firms like Questrade and Wealthsimple.
However, it does give you a better chance to learn and become more confident with self-directed investing. A major problem with them is that Webbroker is frequently down. Try search webbroker on Twitter and you will see. Do not use TD. It takes 20 min to log on with the password, then it will freeze. The customer service is also terrible. This was a good product 15 years ago, but not now. Using my desktop computer, online trading and very minor account changes are still barely adequate, but if one needs to make a non-trading phone call 3, 3 , expect to wait at least an hour or 2 over the phone.
Even then you may have to do it all over again when somebody at TD Direct Investing screws up. I hear the big banks have been having problems with that lately. Hi Jerri, I meant that Series A funds are usually around 2. Hi Jerri, Questrade allows Series F mutual funds. Save my name, email, and website in this browser for the next time I comment. Disclaimer: The content on Wealthawesome. Consult a licensed financial expert before making any life-changing decisions with your money.
No content on this website is intended as financial advice. The publisher of this website does not take any responsibility for possible financial consequences of any persons applying the information in this educational content. As an Amazon Associate I earn from qualifying purchases.
Read more. Thanks very much for the additional info! Where can I learn more about series F funds? Do you mean that you pay TD an additional 2. How to Make Money Online. How to Pay Less Tax. How to Start Investing. Retirement Planning Ultimate Guide. How to Save Money. Deals and Special Offers.