Embracing Encapsulation
[This is a bit rambling / repetitive, sorry. I don’t have time to make it shorter, etc.]
Last year at FOWA I had a discussion with Paul Graham about programming and programmers in which we disagreed over the importance of knowing the fundamentals.
By this I mean the importance of knowing things down to the nuts and bolts level, to really understand what’s going on at the lower levels when you’re writing code. I used to think that sort of thing mattered a lot, but now I think it rarely does.
I well remember learning to program in AWK and being acutely aware of how resource intensive “associative arrays” (as we quaintly called them in those days) were, and knowing full well what was going on behind the scenes. I wrote a full Pascal compiler (no lex, no yacc) in the mid-80’s with Keith Rowe. If you haven’t done that, you really can’t appreciate the amount of computation that goes on when you compile a program to an executable. It’s astonishing. I did lots of assembly language programming, starting from age 15 or so, and spent years squeezing code into embedded environments, where a client might call to ask if you couldn’t come up with a way to reduce your executable code by 2 bytes so it would fit in their device.
But you know what? None of those skills really matter any more. Or they matter only very rarely.
The reason is that best practices have been worked out and incorporated into low-level libraries, and for the most part you don’t need to have any awareness at all of how those levels work. In fact it can be detrimental to you to spend years learning all those details if you could instead be learning how to build great things using the low-level libraries as black-box tools.
That’s the way the world moves in general. Successive generations get the accumulated wisdom of earlier generations packaged up for them. We used log tables, slide rules, and our heads, while our kids use calculators with hundreds of built-in functions. We learned to read analog 12-hour clocks, our kids learn to read digital clocks (so much easier!) and may not be able to read an analog clock until later. And it doesn’t matter. We buy a CD player (remember them?) or an iPod, and when it breaks you don’t even consider getting it “fixed” (remember that?). You just go out and buy another one. That’s because it’s cheaper and much faster and easier to just get a new one that has been put together by a machine than it is to have an actual human try to open the thing and figure out how to repair it. You can’t even (easily) open an iPod. And so the people who know how to do these things dwindle in number until there are none left. Like watch makers or the specialist knife sharpeners we have in Barcelona who ride around on motorcycles with their distinctive whistles, calling to people to bring down their blunt knives. And it doesn’t matter, at least from a technical point of view. Their brilliance and knowledge and hard-won experience has been encapsulated and put into machines and higher-level tools, or simply baked into society in smaller, more accurate and easier to digest forms. In computers it goes down into libraries and compilers and hardware. There’s simply no need for anyone to know how, learn how, or to bother, to do those sorts of things any more.
Note that I’m not saying it’s not nice to have your watch repaired by someone with a jeweler’s eyepiece or your knife or scissors sharpened in the street. I’m just noting the general progression by which knowledge inevitably becomes encapsulated.
In my discussion with Paul Graham, he argued that it was still important for tech founders to be great programmers at a low level. I argued that that’s not right. Sure, people like that are good to have around, but I don’t think you need to be that way and as I said I think it can even be detrimental because all that knowledge comes at a price (other knowledge, other experience).
I work with a young guy called Esteve (Hi Esteve!). He’s great at many levels, including the lower ones. He’s also a product of a new generation of programmers. They’re people who grew up only knowing object-oriented programming, only really writing in very high-level languages (not you Esteve! I mean that just in general), who think in those terms, and who instead of spending many years working with nuts and bolts spent the years working with newer high-level tools.
I think people like Esteve have a triple advantage over us dinosaurs. 1) They tend to use more powerful tools; 2) Because they use better tools, they are more comfortable and think more naturally in the terms of the higher-level abstractions their tools present them; and 3) they also have more experience putting those tools and methods to good use.
The experience gap widens at double speed, just as when a single voter changes side; the gap between the two parties increases by two votes. Even when the dinosaur modernizes itself and learns a few new tricks, you’re still way behind because the 25 year-old you’re working with (again, excluding Esteve) has never had to work at the nuts and bolts level. They think with the new paradigms and can put more general and more powerful tools directly into action. They don’t have to think about protocols or timeouts or dynamically resizing buffers or partial reads or memory management or data structures or error propogation. They simply think “Computer, fetch me the contents of that web page!” And most of the time it all just works. When it doesn’t, you can call in a gray-haired repair person or, more likely, just throw the busted tool away and buy another (or just get it free, in the case of Open Source software).
That’s real progress, and to insist that we should make the young suffer through all the stuff we had to learn in order to build all the libraries and compilers etc., that are now available to us all is just wrong. It’s wrong because it goes against the flow of history, because it’s counter-productive, and because it smacks of “I had to suffer through this stuff, walk barefoot to school in the snow, and therefore you must too.”
Some of the above will probably sound a bit abstract, but to me it’s not. I think it’s important to realize and accept. The fact that your kid can’t tie their shoelaces because they have velcro and have never owned a shoe with a lace is probably a good thing. You don’t know how to hunt your own food or start a fire, and it just doesn’t matter. The same goes for programming. The collective brilliance of generations of programmers is now built in to languages like Java, Python and Ruby, and into operating systems, graphics libraries, etc. etc., and it really doesn’t matter a damn if young people who are using those tools don’t have a clue what’s going on at the lower levels (as I said above, that’s probably a good thing). One day very few people will. The knowledge wont be lost. It’s just encapsulated into more modern environments and tools.
I’m writing all this down because I’ve been thinking about it on and off since FOWA, but also because of what I’m working on right now. I’m trying to modify 12K lines of synchronous Python code to use Twisted (an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers). The work is a bit awkward and three times I’ve not known how best to proceed in terms of design. Each time, Esteve has taken a look at the problem and quickly suggested a fairly clean way to tackle it. Desperate to cook up a way to think that he might not be that much smarter than I am, I’m forced into a corner in which I conclude that he has spent more time working with new tools (patterns, OO, a nice language like Python). So he looks at the world in a different way and naturally says “oh, you just do that”. Then I go do the routine work of making his ideas work – which is great by me, I get to learn in the best way, by doing. How nice to hire people who are better than you are.
That’s it. Encapsulation is inevitable. So you either have to embrace it or become a hand-wringing dinosaur moaning about the kids of today and how they no longer know the fundamentals. It’s not as though any of us could survive if we suddenly had to do everything from first principles (hunt, rub sticks together to make fire, etc). So relax. Enjoy it. The young are much better than we are because they grow up with better tools and they spend more time using them. It’s not enough to learn them when you’re older, even if you can do that really fast. You’ll never catch up on the experience front.
But it sure is fun to try.
You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.
June 18th, 2008 at 3:04 pm
What should I say now? Should I ask for a pay rise? :-D Anyway, this reminds me of another post of yours:
http://www.fluidinfo.com/terry/2007/11/14/the-young-are-different/
when I was born, Ollies were already invented (according to Wikipedia, in 1976). So, when I started skating, it was “natural” to learn how to do an Ollie, then a Kickflip and so on. It was a breakthrough when Alan “Ollie” Gelfand landed his first Ollie, but now it’s just a natural thing to do, you don’t amaze anybody. However, you still need to learn how to do an Ollie, you can’t do a Darkslide if you don’t learn the basics.
My point is that, learning how to program with punched cards is pointless, but knowing how threads, mutexes, compilers, etc. work is important. For example, we know there are problems with the GIL in the standard Python interpreter and if we didn’t know how to solve them with lower level libraries (such us Boost, TBB, etc.), we would be doomed.
June 18th, 2008 at 5:04 pm
What should I say now? Should I ask for a pay rise? :-D Anyway, this reminds me of another post of yours:
http://www.fluidinfo.com/terry/2007/11/14/the-young-are-different/
when I was born, Ollies were already invented (according to Wikipedia, in 1976). So, when I started skating, it was “natural” to learn how to do an Ollie, then a Kickflip and so on. It was a breakthrough when Alan “Ollie” Gelfand landed his first Ollie, but now it’s just a natural thing to do, you don’t amaze anybody. However, you still need to learn how to do an Ollie, you can’t do a Darkslide if you don’t learn the basics.
My point is that, learning how to program with punched cards is pointless, but knowing how threads, mutexes, compilers, etc. work is important. For example, we know there are problems with the GIL in the standard Python interpreter and if we didn’t know how to solve them with lower level libraries (such us Boost, TBB, etc.), we would be doomed.
June 18th, 2008 at 3:24 pm
A PAYRISE? How about another complimentary blog post?
I agree with what you’ve written. That’s why I say that it’s only rarely that the lower level skills are needed. Some things really need them, but lots don’t. When we release our stuff the world will be one step closer to not needing that Boost, TBB work that you did. Then one day you can be a dinosaur too! Can’t wait :-)
June 18th, 2008 at 5:24 pm
A PAYRISE? How about another complimentary blog post?
I agree with what you’ve written. That’s why I say that it’s only rarely that the lower level skills are needed. Some things really need them, but lots don’t. When we release our stuff the world will be one step closer to not needing that Boost, TBB work that you did. Then one day you can be a dinosaur too! Can’t wait :-)
June 18th, 2008 at 4:23 pm
Great post. I’ve had these conversations myself, first from Esteve’s shoes and now from yours. Same conclusions. My old boss sounds like Graham, everything should be done in old language/style X.
I think this is just a basic condition of mankind. My father and I would have similar discussions about how to get a job done or if knowing how to change out the transmission in your car is important.
I’d add to this idea that there is a periodic resetting of this encapsulation where the old methods are re-examined and portions re-embraced.
Example: Beer brewing. From 1960 or so on, beer became an industrial homogenized product through massive brewing innovations and distribution. Then about 1990 all these homebrewers start founding breweries and go back to the old methods of producing full flavored beer… and go digging around in Belgian monasteries for inspiration. Now we have a wide variety of beer styles available again.
I think with software the difference is the cycle time is about 4 years max. If a technology isn’t recycled or parted-out within 4 years it will slowly rot away.
Esteve: don’t ask for a raise, ask for more equity! At 25, you’ll just waste the money, while you’ll chase that equity till the job is done.
June 18th, 2008 at 6:23 pm
Great post. I’ve had these conversations myself, first from Esteve’s shoes and now from yours. Same conclusions. My old boss sounds like Graham, everything should be done in old language/style X.
I think this is just a basic condition of mankind. My father and I would have similar discussions about how to get a job done or if knowing how to change out the transmission in your car is important.
I’d add to this idea that there is a periodic resetting of this encapsulation where the old methods are re-examined and portions re-embraced.
Example: Beer brewing. From 1960 or so on, beer became an industrial homogenized product through massive brewing innovations and distribution. Then about 1990 all these homebrewers start founding breweries and go back to the old methods of producing full flavored beer… and go digging around in Belgian monasteries for inspiration. Now we have a wide variety of beer styles available again.
I think with software the difference is the cycle time is about 4 years max. If a technology isn’t recycled or parted-out within 4 years it will slowly rot away.
Esteve: don’t ask for a raise, ask for more equity! At 25, you’ll just waste the money, while you’ll chase that equity till the job is done.
June 18th, 2008 at 4:40 pm
Just wanted to mention that the pay rise thing was a joke. As long as I keep having fun doing all these crazy things, I’m happy :-) Fun is one of the most important motivations that drive hackers.
Anyway, I’ll take Neal’s tip :-)
June 18th, 2008 at 6:40 pm
Just wanted to mention that the pay rise thing was a joke. As long as I keep having fun doing all these crazy things, I’m happy :-) Fun is one of the most important motivations that drive hackers.
Anyway, I’ll take Neal’s tip :-)
June 18th, 2008 at 5:18 pm
I mostly agree with you Terry. But there is one small point, going to higher level some flexibility is always lost. I am usually happy with using high level functions and programs and do not bother about their internals but in very few cases I am not satisfied completely with what is offered for me and I have to write functions from ground, dig into deep implementation details, just to make something working exactly the way I want.
June 18th, 2008 at 7:18 pm
I mostly agree with you Terry. But there is one small point, going to higher level some flexibility is always lost. I am usually happy with using high level functions and programs and do not bother about their internals but in very few cases I am not satisfied completely with what is offered for me and I have to write functions from ground, dig into deep implementation details, just to make something working exactly the way I want.
June 18th, 2008 at 11:07 pm
@Neal
Hi Neal – those are good comments, and I agree. I guess you might say that in a slightly different way the whole food industry got packaging somewhat wrong (esp fast food) and that we’re now seeing the effects and working to re-package.
A difference from the computational world is that, with certain notable exceptions, the people doing the packaging and using the packages are mainly pulling in the same direction. Sure, committees can screw anything up, but their hearts are usually in the right place. And certainly that’s true of open source projects, and a competitive market pushes, in theory, towards better packaging. But when people are mainly trying to make a buck or are focused on the short term (next quarter’s numbers, etc) then I guess mistakes and corrections in packaging should be expected.
I’ll think about your comments in the context of computational systems more though. Certainly, I think, we’ve progressed in how we make APIs. The computational world is, at least in theory, more easily quantified. If you’ve got a better random number generator or compiler, you can either prove it or demonstrate it pretty effectively. So I think advancements there do get baked in. The fly in that ointment, as with lots of optimization, as you know, is that there are many trade-offs and nothing is really the best. So there’s lots of wiggle room.
Anyway, thanks for the comments. Shouldn’t you be writing that dissertation instead of inciting Esteve to riot? :-) Good luck.
June 19th, 2008 at 1:07 am
@Neal
Hi Neal – those are good comments, and I agree. I guess you might say that in a slightly different way the whole food industry got packaging somewhat wrong (esp fast food) and that we’re now seeing the effects and working to re-package.
A difference from the computational world is that, with certain notable exceptions, the people doing the packaging and using the packages are mainly pulling in the same direction. Sure, committees can screw anything up, but their hearts are usually in the right place. And certainly that’s true of open source projects, and a competitive market pushes, in theory, towards better packaging. But when people are mainly trying to make a buck or are focused on the short term (next quarter’s numbers, etc) then I guess mistakes and corrections in packaging should be expected.
I’ll think about your comments in the context of computational systems more though. Certainly, I think, we’ve progressed in how we make APIs. The computational world is, at least in theory, more easily quantified. If you’ve got a better random number generator or compiler, you can either prove it or demonstrate it pretty effectively. So I think advancements there do get baked in. The fly in that ointment, as with lots of optimization, as you know, is that there are many trade-offs and nothing is really the best. So there’s lots of wiggle room.
Anyway, thanks for the comments. Shouldn’t you be writing that dissertation instead of inciting Esteve to riot? :-) Good luck.
June 19th, 2008 at 2:30 am
Hmm…but I still know conceptually what hunting and rubbing sticks for fire is, even if I’ve never done it. I would argue that good programmers should understand that there is such as thing as the nuts and bolts and have an idea of how it works and why it’s important, even if they couldn’t do it themselves at a moment’s notice.
Also there are some fundamentals that must be learned, just like any craft, e.g. electricians, architects, etc. We don’t make mathematicians use slide rules, but they still have to know the basic theory as they type formulas into a computer.
June 19th, 2008 at 4:30 am
Hmm…but I still know conceptually what hunting and rubbing sticks for fire is, even if I’ve never done it. I would argue that good programmers should understand that there is such as thing as the nuts and bolts and have an idea of how it works and why it’s important, even if they couldn’t do it themselves at a moment’s notice.
Also there are some fundamentals that must be learned, just like any craft, e.g. electricians, architects, etc. We don’t make mathematicians use slide rules, but they still have to know the basic theory as they type formulas into a computer.
June 19th, 2008 at 6:16 am
I really like this article, and not simply because I agree with it. As a non-classically trained programmer, I find that most of the work I’ve ever encountered easily relies on the encapsulation of previous work. You can accomplish great things on the backs of the giants that have moved the art forward.
However, I think where Graham is correct is in the hyper-niche that is the startup world. It encompasses a vastly smaller segment of the worlds jobs and mind-share, but is dominated by doing “neat things” with “old tech”. As evidenced by Steve Yegge’s ranting about fundamentals, it is relevant if you’re trying to do something special, especially performance related.
I was lamenting with a friend the other day that, really, very few of my peers know what their computer is doing, even the ones that know how to code, at a basic level. It’s not important for them to build a business/website/widget. They’re not inventing the New New Thing, but they’re successful and happy. Really, this is only a lament because *I* learned all that “basic” stuff that doesn’t really matter on a day-to-day basis. Or didn’t until I got a new job that focuses on performance.
I think the balance of knowledge (or lack of knowledge) is a great and wonderful thing. Computers are slowly replacing many of the old tools that are no longer required to make us successful. And the brain-damaged fringe that really like the details can work at the startups, or become hobbyists :) But, really, there’s better things to do with brain cycles than remember old, worthless stuff like Lisp (j/k).
June 19th, 2008 at 8:16 am
I really like this article, and not simply because I agree with it. As a non-classically trained programmer, I find that most of the work I’ve ever encountered easily relies on the encapsulation of previous work. You can accomplish great things on the backs of the giants that have moved the art forward.
However, I think where Graham is correct is in the hyper-niche that is the startup world. It encompasses a vastly smaller segment of the worlds jobs and mind-share, but is dominated by doing “neat things” with “old tech”. As evidenced by Steve Yegge’s ranting about fundamentals, it is relevant if you’re trying to do something special, especially performance related.
I was lamenting with a friend the other day that, really, very few of my peers know what their computer is doing, even the ones that know how to code, at a basic level. It’s not important for them to build a business/website/widget. They’re not inventing the New New Thing, but they’re successful and happy. Really, this is only a lament because *I* learned all that “basic” stuff that doesn’t really matter on a day-to-day basis. Or didn’t until I got a new job that focuses on performance.
I think the balance of knowledge (or lack of knowledge) is a great and wonderful thing. Computers are slowly replacing many of the old tools that are no longer required to make us successful. And the brain-damaged fringe that really like the details can work at the startups, or become hobbyists :) But, really, there’s better things to do with brain cycles than remember old, worthless stuff like Lisp (j/k).
June 19th, 2008 at 7:27 am
(I apologize in advance for not reading the assuredly fine comments above mine.)
I still think the programming world has a need for high-level application programmers and low-level system programmers for a very very simple reason: while digital clocks replace analog clocks, we don’t have high-performance processors that are able to think in high-level abstractions. Python still needs C because there’s no Python CPU, the same way C must first be compiled in machine code by a good compiler because there is no CPU able to read C and make sense of it.
The maxim goes: write the app and then optimize it — do not optimize prematurely. Which is really something that should read: write the app in a way that is easy to read and follow for an external party and overall makes sense, and then you dive down and spend time with the implementation of your concepts to see if it lives down to your speed requirements. High-level languages that offer some functionality “crystallized” in a library are a barrier to this, as after you have made the app as good as you can, the library then becomes the bottleneck, and if it’s still an unacceptable bottleneck you have to get down a level and start messing around with the library’s internals, if you can (long live FOSS!).
Which, in two words, means: there’s a very fine place for both kinds of programmers in this brave new world. Neither makes the other obsolete.
June 19th, 2008 at 9:27 am
(I apologize in advance for not reading the assuredly fine comments above mine.)
I still think the programming world has a need for high-level application programmers and low-level system programmers for a very very simple reason: while digital clocks replace analog clocks, we don’t have high-performance processors that are able to think in high-level abstractions. Python still needs C because there’s no Python CPU, the same way C must first be compiled in machine code by a good compiler because there is no CPU able to read C and make sense of it.
The maxim goes: write the app and then optimize it — do not optimize prematurely. Which is really something that should read: write the app in a way that is easy to read and follow for an external party and overall makes sense, and then you dive down and spend time with the implementation of your concepts to see if it lives down to your speed requirements. High-level languages that offer some functionality “crystallized” in a library are a barrier to this, as after you have made the app as good as you can, the library then becomes the bottleneck, and if it’s still an unacceptable bottleneck you have to get down a level and start messing around with the library’s internals, if you can (long live FOSS!).
Which, in two words, means: there’s a very fine place for both kinds of programmers in this brave new world. Neither makes the other obsolete.
June 20th, 2008 at 12:28 am
Even a non-programmer’s store of hard-earned learning has been devalued by encapsulation. I’m glad I know what I know, but I’m please to have the new tools, too. BTW, my father, a retired polymer chemist, spent a few years finding ways to implement micro-encapsulation. He put fire-retardant in carpets and textiles, made no-carbon duplicate paper with micro-encapsulated dye, scratch-and-sniff stuff, etc.
June 20th, 2008 at 2:28 am
Even a non-programmer’s store of hard-earned learning has been devalued by encapsulation. I’m glad I know what I know, but I’m please to have the new tools, too. BTW, my father, a retired polymer chemist, spent a few years finding ways to implement micro-encapsulation. He put fire-retardant in carpets and textiles, made no-carbon duplicate paper with micro-encapsulated dye, scratch-and-sniff stuff, etc.
June 20th, 2008 at 7:28 am
Good article, hi Esteve! :-)
I am a big fan of scripting languages, and I work with them for a living, but I think a programmer needs to understand things at a lower level as part of his education. The more languages and paradigms you know the better, but I think you’d need to know some C for example, perhaps not at the level of a professional C programmer, but what C provides would be understood. I am not sure programmers need to understand assembly, memory segments, hardware, etc. The more the better, but for programming I feel that low level is perhaps not that necessary.
On the other hand it goes without saying that we need young C programmers, new interpreters are written, new libraries are developed, people have to evolve operating systems, maintain interpreters, write bindings, etc. In general high-performance software will be needed and a guy with only high-level concepts can’t write it.
Now we’ll see if any of that helps me solve the captcha :-).
June 20th, 2008 at 9:28 am
Good article, hi Esteve! :-)
I am a big fan of scripting languages, and I work with them for a living, but I think a programmer needs to understand things at a lower level as part of his education. The more languages and paradigms you know the better, but I think you’d need to know some C for example, perhaps not at the level of a professional C programmer, but what C provides would be understood. I am not sure programmers need to understand assembly, memory segments, hardware, etc. The more the better, but for programming I feel that low level is perhaps not that necessary.
On the other hand it goes without saying that we need young C programmers, new interpreters are written, new libraries are developed, people have to evolve operating systems, maintain interpreters, write bindings, etc. In general high-performance software will be needed and a guy with only high-level concepts can’t write it.
Now we’ll see if any of that helps me solve the captcha :-).
June 20th, 2008 at 2:17 pm
@thiefhunter
There’s value in encapsulating knowledge and making it available to others. You wrote a book and you have a blog where you bundled up decades of extremely hard won experience. It benefits you to do so, and it benefits the rest of us to learn from you the easy way. And so it goes…
June 20th, 2008 at 4:17 pm
@thiefhunter
There’s value in encapsulating knowledge and making it available to others. You wrote a book and you have a blog where you bundled up decades of extremely hard won experience. It benefits you to do so, and it benefits the rest of us to learn from you the easy way. And so it goes…
June 20th, 2008 at 6:09 pm
Just a quick note about converting synchronous code to Twisted: that’s very tricky :-) I recently interviewed JP Calderone (of Twisted fame) about best practices when attempting to do this, and will post it to my blog soonish (I’m hopng next week).
June 20th, 2008 at 8:09 pm
Just a quick note about converting synchronous code to Twisted: that’s very tricky :-) I recently interviewed JP Calderone (of Twisted fame) about best practices when attempting to do this, and will post it to my blog soonish (I’m hopng next week).
June 20th, 2008 at 10:36 pm
Hi Duncan
I don’t know where your blog is, I don’t think. I’m surprised you know where mine is!
I’d like to read that article. We’ve run into some really interesting things just thinking about how to convert one function (an iterator that periodically goes to the network). It’s fun but also hard, as you say. I’m goin gto post about it to the Twisted list soon. Or maybe I should just blog about it…
Are you going to EuroPytho?
Terry
June 21st, 2008 at 12:36 am
Hi Duncan
I don’t know where your blog is, I don’t think. I’m surprised you know where mine is!
I’d like to read that article. We’ve run into some really interesting things just thinking about how to convert one function (an iterator that periodically goes to the network). It’s fun but also hard, as you say. I’m goin gto post about it to the Twisted list soon. Or maybe I should just blog about it…
Are you going to EuroPytho?
Terry
June 24th, 2008 at 6:03 pm
I think the problem is that you are predicating your views on an implicit assumption that the previous work is good enough, if not great. Thus we can move on. What if all of this modern stuff is just a train that’s gone off its tracks? What if the reason you’re not seeing the solution as being obvious, is because it’s not actually obvious?
We have, in a very real sense gone a long way down a specific technology path, heavily investing in a style of building tools. But are they just more complex because they are, or are they a huge mess? Horribly colored code in a flaky IDE, running on a platform that acts irrationally may look prettier than the old text-based screens, but honestly what is it doing now, that it wasn’t twenty years ago? Sure, the data is massive, but the functionality?
As we stretch the limits of this current technology we are quickly getting to a level beyond which it is too unstable to manage. To get past that threshold we’ll have to go back and refactor our foundations. At that point, either Esteve goes back and re-learns all that was known or he goes forward and just re-invents the whole same mess again (for the 4th time?)
Just a guess.
Paul.
http://theprogrammersparadox.blogspot.com
June 24th, 2008 at 8:03 pm
I think the problem is that you are predicating your views on an implicit assumption that the previous work is good enough, if not great. Thus we can move on. What if all of this modern stuff is just a train that’s gone off its tracks? What if the reason you’re not seeing the solution as being obvious, is because it’s not actually obvious?
We have, in a very real sense gone a long way down a specific technology path, heavily investing in a style of building tools. But are they just more complex because they are, or are they a huge mess? Horribly colored code in a flaky IDE, running on a platform that acts irrationally may look prettier than the old text-based screens, but honestly what is it doing now, that it wasn’t twenty years ago? Sure, the data is massive, but the functionality?
As we stretch the limits of this current technology we are quickly getting to a level beyond which it is too unstable to manage. To get past that threshold we’ll have to go back and refactor our foundations. At that point, either Esteve goes back and re-learns all that was known or he goes forward and just re-invents the whole same mess again (for the 4th time?)
Just a guess.
Paul.
http://theprogrammersparadox.blogspot.com
June 24th, 2008 at 8:07 pm
Hi Paul
I’m not claiming that progress in encapsulation is always (or even often) linear. Neal gave an example above (in beer brewing) in which we’ve gone back to re-examine how things were encapsulated. Also, I guess from your comments that I was thinking (or at least speaking) at a more general level than IDEs and programming languages etc. We do make mistakes, I agree. But the overall trend is towards more being encapsulated, towards good encapsulations gaining mindshare, etc. Very few people will ever need to write their own random number generator, or hashing code, or bitblt code – or you name it – there are hundreds of examples of that kind.
I also don’t like IDEs :-) Emacs is enough for me in that regard…
June 24th, 2008 at 10:07 pm
Hi Paul
I’m not claiming that progress in encapsulation is always (or even often) linear. Neal gave an example above (in beer brewing) in which we’ve gone back to re-examine how things were encapsulated. Also, I guess from your comments that I was thinking (or at least speaking) at a more general level than IDEs and programming languages etc. We do make mistakes, I agree. But the overall trend is towards more being encapsulated, towards good encapsulations gaining mindshare, etc. Very few people will ever need to write their own random number generator, or hashing code, or bitblt code – or you name it – there are hundreds of examples of that kind.
I also don’t like IDEs :-) Emacs is enough for me in that regard…
June 24th, 2008 at 10:50 pm
At that point, either Esteve goes back and re-learns all that was known or he goes forward and just re-invents the whole same mess again (for the 4th time?)
I think the point of the article is not that Esteve doesn’t know about what’s underneath, but that growing up with more powerful tools (OO, patterns, etc.) changes the way you see the world. Take for example the Linux VFS layer, it’s a fine example of OO applied to a project that has to deal with lots of low level things, there’s inheritance, polymorphism, etc. Someone who hasn’t been exposed to OO might have implemented it differently, but since it’s the product of a generation that grew up with objects, Al Viro (the guy who designed the current VFS layer) thought it was natural to design it that way.
Although our primary language at Fluidinfo is Python, we have had to deal with things in C and C++, so I don’t think that a programmer who only knows high-level languages can be a good one.
Since kids have grown up using high level tools, it should be easier for them to think in abstract terms, that’s the key IMHO.
It’s funny to write about oneself in third person.
June 25th, 2008 at 12:50 am
At that point, either Esteve goes back and re-learns all that was known or he goes forward and just re-invents the whole same mess again (for the 4th time?)
I think the point of the article is not that Esteve doesn’t know about what’s underneath, but that growing up with more powerful tools (OO, patterns, etc.) changes the way you see the world. Take for example the Linux VFS layer, it’s a fine example of OO applied to a project that has to deal with lots of low level things, there’s inheritance, polymorphism, etc. Someone who hasn’t been exposed to OO might have implemented it differently, but since it’s the product of a generation that grew up with objects, Al Viro (the guy who designed the current VFS layer) thought it was natural to design it that way.
Although our primary language at Fluidinfo is Python, we have had to deal with things in C and C++, so I don’t think that a programmer who only knows high-level languages can be a good one.
Since kids have grown up using high level tools, it should be easier for them to think in abstract terms, that’s the key IMHO.
It’s funny to write about oneself in third person.
June 24th, 2008 at 11:15 pm
I would say the point was just to highlight the general trend of encapsulation, to assert that it’s an inevitable aspect of “progress” (which is not the same as saying that we are always progressing, or progressing forwards, or progressing linearly), and to suggest that we (i.e., mainly me) embrace it instead of bemoaning the lack of fundamental knowledge in (mainly) the young.
I didn’t think all this out in great depth, and some of you have raised good exceptions that I agree with. As I said to Tim Bray in Twitter, I could have easily argued the other side. But I do believe history shows that that is ultimately the dinosaur’s side of the debate.
BTW, I probably wouldn’t have written the article at all if I hadn’t found such a cool image on Flickr to go with it.
The VFS example is a good one. Maybe Twitter is a good example for discussion. They used very high-level encapsulation (RoR) and it took them a long way, a really long way – much further than 99% of startups get. And yes, they now have to re-engineer a lot of stuff (throw out some encapsulations, and adopt some others that are more reliable). But you might argue that if they hadn’t built their first site in days they might not exist at all.
Amazon web services and Google’s App Engine are another good example. Scaling is really hard. But certain aspects, including some pretty significant ones, are now encapsulated to the point where any programmer who can put together a few HTTP requests can build massive systems that scale (at least in some important ways). I see things like AWS as being huge steps forward. They may not get it all right immediately, but it’s very early days yet for cloud computing.
June 25th, 2008 at 1:15 am
I would say the point was just to highlight the general trend of encapsulation, to assert that it’s an inevitable aspect of “progress” (which is not the same as saying that we are always progressing, or progressing forwards, or progressing linearly), and to suggest that we (i.e., mainly me) embrace it instead of bemoaning the lack of fundamental knowledge in (mainly) the young.
I didn’t think all this out in great depth, and some of you have raised good exceptions that I agree with. As I said to Tim Bray in Twitter, I could have easily argued the other side. But I do believe history shows that that is ultimately the dinosaur’s side of the debate.
BTW, I probably wouldn’t have written the article at all if I hadn’t found such a cool image on Flickr to go with it.
The VFS example is a good one. Maybe Twitter is a good example for discussion. They used very high-level encapsulation (RoR) and it took them a long way, a really long way – much further than 99% of startups get. And yes, they now have to re-engineer a lot of stuff (throw out some encapsulations, and adopt some others that are more reliable). But you might argue that if they hadn’t built their first site in days they might not exist at all.
Amazon web services and Google’s App Engine are another good example. Scaling is really hard. But certain aspects, including some pretty significant ones, are now encapsulated to the point where any programmer who can put together a few HTTP requests can build massive systems that scale (at least in some important ways). I see things like AWS as being huge steps forward. They may not get it all right immediately, but it’s very early days yet for cloud computing.
June 24th, 2008 at 11:24 pm
@esteve
Since kids have grown up using high level tools, it should be easier for them to think in abstract terms, that’s the key IMHO.
Agreed – though I’d say to think in the new paradigms, as I don’t know that they’re necessarily more abstract. In the case of OO, I’d say that it’s more natural and more concrete, not more abstract. But that’s just a matter of words – I agree that that’s key.
You don’t look at problems the way I do. You quickly see approaches and solutions that I find very natural once you’ve pointed them out to me. I’m hoping to change that, but maybe you just are smarter :-)
June 25th, 2008 at 1:24 am
@esteve
Since kids have grown up using high level tools, it should be easier for them to think in abstract terms, that’s the key IMHO.
Agreed – though I’d say to think in the new paradigms, as I don’t know that they’re necessarily more abstract. In the case of OO, I’d say that it’s more natural and more concrete, not more abstract. But that’s just a matter of words – I agree that that’s key.
You don’t look at problems the way I do. You quickly see approaches and solutions that I find very natural once you’ve pointed them out to me. I’m hoping to change that, but maybe you just are smarter :-)
June 25th, 2008 at 11:58 am
I’m hoping to change that, but maybe you just are smarter :-)
C’mon, you know I’m not smarter than you :-) It’s just a matter of getting used to some paradigm/methodology/etc. For example, Python and Ruby make functional programming easy to integrate in a object-oriented language. Once you have gotten used to use functions as first class citizens, you feel that Java imposes you too many limits.
That’s why I like C++ and Python so much:
– traditional procedural language (check)
– object oriented features (check)
– functional programming (check). In C++, if you want to pass around functions, you’ll need to write a class that overloads the () operator (usually called a functor). It’s not as straightforward as in Python, but it’s something you cannot do in Java.
June 25th, 2008 at 1:58 pm
I’m hoping to change that, but maybe you just are smarter :-)
C’mon, you know I’m not smarter than you :-) It’s just a matter of getting used to some paradigm/methodology/etc. For example, Python and Ruby make functional programming easy to integrate in a object-oriented language. Once you have gotten used to use functions as first class citizens, you feel that Java imposes you too many limits.
That’s why I like C++ and Python so much:
– traditional procedural language (check)
– object oriented features (check)
– functional programming (check). In C++, if you want to pass around functions, you’ll need to write a class that overloads the () operator (usually called a functor). It’s not as straightforward as in Python, but it’s something you cannot do in Java.
June 25th, 2008 at 3:52 pm
But, if in order to fully express the problem in a specific technology, one has to learn how to bend one’s perspective to match it, then is it really a good technology?
And if it requires merging together several different mutually exclusive paradigms, such as OO and FP, isn’t this just getting messy for the sake of having too many little moving bits?
In the end, are we looking for intricate little bits to fiddle with or are we actually trying to take something with a massive amount of complexity and reduce it into something that is simple, maintainable and solves (some of) our user’s problems?
So, if after fifty years of progress, the technologies aren’t actually getting more simple or obvious, and are in fact getting way way harder to grok, isn’t that a strong indicator that there is some type of serious problem with them? Do we get more from our modern technologies, or is it that there is just more code and data available to play with?
Paul.
June 25th, 2008 at 5:52 pm
But, if in order to fully express the problem in a specific technology, one has to learn how to bend one’s perspective to match it, then is it really a good technology?
And if it requires merging together several different mutually exclusive paradigms, such as OO and FP, isn’t this just getting messy for the sake of having too many little moving bits?
In the end, are we looking for intricate little bits to fiddle with or are we actually trying to take something with a massive amount of complexity and reduce it into something that is simple, maintainable and solves (some of) our user’s problems?
So, if after fifty years of progress, the technologies aren’t actually getting more simple or obvious, and are in fact getting way way harder to grok, isn’t that a strong indicator that there is some type of serious problem with them? Do we get more from our modern technologies, or is it that there is just more code and data available to play with?
Paul.
December 2nd, 2009 at 12:38 pm
I think with software the difference is the cycle time is about 4 years max. If a technology isn't recycled or parted-out within 4 years it will slowly rot away.