Untangling the Knot

[Symbology]

It hit me this morning as I was working on some non-computer things, that I was yet again working to unravel a metaphorical knot.

 

So many times in my computer science work, and other things outside of work, I find myself being asked to come accomplish a task in an environment that is a great big mess. (Which is to some degree why we get paid more I suppose).

 

I often come in hoping that it will be a rosy environment, all ready for a simple tweak to accomplish the task. But as I talk to people and read up on the documentation, or investigate the code I usually find a tangle of repeated blocks of code and "one offs". I quickly realize that for me to do the task correctly, there are a host of other things that will need updating as well, if any consistency is to be maintained.

 

So I am then left in the quandry:

• Do I just "Git 'er done" and mash in my changes, and to hell with all the other pages that will have to figure it out for themselves? Basically compounding the knot of spaghetti code.
  
• Do I spend the time to (sometimes document and) clean up the other pages that I had nothing to do with, just so that they can remain current and unbroken
  
• Or do I gut the whole system and put in a centralized set of functions, web pages, and/or procedures where changes can be made in one central place?
  
  • this typically involves alot of perceived risk to the existing system, and therefore lots of resistance from entrenched authors and management
    
  • to overcome this perceived risk classically involves lots of personal time to create and present a reasonably solid prototype and/or working model of the new approach
    
  • management is rarely up for additional investment in something that will have no visible gains on the surface (which again is what takes personal time and putting one's reputation on the line to overcome)
    
  • There is a period of follow on time where the bugs are worked out in the central code, and because everything now goes through the central code/process, it can become the scape goat for future problems even though it was not necessarily the cause
    

 

In hind-sight to half a dozen of these major tangles through the years, I can say that in my experience it is always painful, and I am not sure that it's worth it cleaning up other people's messes. It pushes me in the direction of becoming one of the "old guys" and not caring about what is right or wrong but just getting it done and clocking out at 5.

 

I have heard some suggestions through the years that if I were to finally accept some of the offers to move into management, then I would have the authority to take on such things. But the problem there is, would I be lucky enough to have programmers under me that understood the importance of centralized code - because to my experience finding peers that see the point (generally Computer Science majors instead of MIS majors) is extremely rare as well. Plus then I would be the manager taking the risk to again clean up my predecessors mess.

 

I have figured out along the way, that the responsibility to educate peers and management falls to the veteran programmers - to teach enough (through good metaphors etc) that they understand the needs and are willing to take on what is necessary to "do it right". Doing so in a respectful, entertaining, and valuable way is its own set of challenges.

 

This is similar to a gripe I made in the past. But any experience and insight on this repeating issue would be appreciated. I'm posting this separately because I think the bullet points have more universal applicability, but still kept it in Computer Science because I think we experience it regularly.

 

In rereading that thread I can see that LaurieAG suggested consulting. When I think about it, generally consultants are called in to gut the place and put in a new process. Folks rarely like those kind of consultants though - primarily because they tend to create their own set of bloody messes and don't stick around to clean up the consequences. Still its something significant to think about.

 

We offer three kinds of service

GOOD - CHEAP - FAST

*You can pick any two*

GOOD service CHEAP won't be FAST

GOOD service FAST won't be CHEAP

FAST service CHEAP won't be GOOD

 

The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten.

 

Thanks,

Symbology

[LaurieAG]

Hi Symbology,

 

I tend to agree with you about the narrow focus of modern IS degrees. While they seem to contain the latest training for the (MS) products demanded by business they seem to lack the balance for someone who has to cover a lot of ground (like a Computer Scientist who consults) in a lot of different areas.

 

If you get into management you could be able to influence your companies hiring policies to get the people you need to do what you want. It might be better presenting a business plan and outlining the cost benefits of this line of approach in system usability and future staff requirements based on estimates of your past experiences.

 

Also, especially in Australia, since big business and government outsourcing programs flooded the market with highly qualified (in limited areas) ex staff, the people who previously did everything have been pushed out of the market because they lack the latest 'in' programming language or a MCSE.

 

Many of these quite competent people are frozen out of the market and have resorted to other occupations (usually menial) mainly because the next generation have either no respect for their professional skills and qualifications or they actually feel threatened by these same professional skills and qualifications because that is what they actually lack.

 

But there are good and bad in every organisation and I've worked with some young switched on people (usually my Computer Science peers), just look at Dilbert.

[Buffy]

Larry the Cable Guy *is* a Republican.

 

Git-r-done does in fact involve taking into consideration the difference between staying in business or not.

 

I've been one of those hated Product Marketing people most of my career, but just happen to be one of those pointy-haired Development Managers now (albeit one that can code as well as most of my development team).

 

I'll be a mean boss for a second: take a look at your post again. Try to rethink it in terms of:

1. Your users
   
2. The folks that sign your paycheck
   

Unfortunately you sound like every developer I've ever known in not thinking about anything but what causes you pain or how much everything is the fault of those other than yourself.

 

Talk to Pyro. He knows.

 

Fear less, hope more; Whine less, breathe more; Talk less, say more; Hate less, love more; And all good things are yours, :shrug:

Buffy

[Pyrotex]

Part of the solution is in retraining your environment.

 

I have to train people not to step in front of my wheelchair -- without running over their toes.

I have to train them NOT to grab the handlebars and give me a push -- without breaking their wrists.

I have to train them NOT to feel sorry for me, or to be worried about my safety -- without insulting them.

 

Same with all things that are software. Every co-worker and manager automatically becomes one of my "students" -- without me ever telling them or demeaning them.

I encourage having "fun" with software, taking risks, having adventures, becoming better, faster, nimbler -- without asking anyone's permission.

I give all my best ideas away for free -- without demanding that people use them.

I strive to be a force of nature -- without being arrogant or argumentative.

I let others win all the arguments -- because arguments are a wonderful kind of Trojan Horse within which to hide enlightenment.

 

And I'm never as good as you are led to believe by your own optimism and hope.

 

"...And all good things are yours."

[LaurieAG]

Unfortunately you sound like every developer I've ever known in not thinking about anything but what causes you pain or how much everything is the fault of those other than yourself.

 

Hi Buffy,

 

About 20 years ago I had a builder mate who went to the US. He applied for a job at a company that built houses. The boss said 'what do you do, footings, floors, framing, cladding, walls, roofs?'. He replied 'what are you talking about, I build houses'. He was foreman within 2 weeks.

 

Your users want something that actually works and the people who pay you want something (that works too) in return for their payments and if you don't provide something that works within a cost effective time frame you go out of business.

 

Do you have anything to say about specialisation vs generalisation as it appears that you are a specialist generalist? I am similar because, on reflection, I have specialised in fixing/upgrading legacy programs for most of my career (solo and after the stuff ups). I also wouldn't hesitate to say that Pyro is most likely a specialist generalist too (most people in 'science' are).

 

You're not one of those people who think that 'specialist' managers don't actually need to know what the people under them do (like dilberts pointy headed boss), i.e. all they need to know is how to manage, or do you think they actually should know what they are telling those beneath them to do? (Buffy, I know you think the latter so lets get on with some constructive discussion that may help Symbology)

[TheBigDog]

It is important to remember that not all knots are caused by ignorance or shortcutting. Take the simple phone cord as an example; pick it up and hang it up, repeat (hundreds of times) and you have a twisted mess with those inexplicable reversed loops. Nobody made a knot, it just happened from normal use. The same is true of IT. Because of the fast moving nature of the IT world there are always pieces of your structure that are a generation or two older than the others. Even with following industry standards you begin to have to do things to compensate for older equipment which in turn limits the way you are using the newer equipment; maybe not at the time you do it, but a year or two later and suddenly what was a smart decision is a problem for doing something else that you were not previously considering. After several well designed and thought out projects are done you suddenly realize you have a knot. I will give you an example.

 

We had a manufacturing system installed in the 70's. In the 80's we installed an order entry and billing system mated to the manufacturing system. The manufacturing system had limits to its capabilities that needed to be respected by the order entry and billing system; and so they were. These included such things as how many detail lines could be on an order and such. In the 90's we installed a new manufacturing system to replace the one from the 70's. One of the major selling points was the fact that we would be able to move to a much less constrained environment. It took 18 months to roll out the systems across our network, and during the rollout we needed to have both manufacturing systems interfaced to the order entry system. As a result of this we had to limit our use of the new manufacturing system to respect the limits of the order entry system that was built to respect the limits of the first manufacturing system; but only for those 18 months. When the 18 months were over there were new priorities and we never did adjust the order entry system to up its capabilities even though it was no longer constrained by the original system; during the 10 years they had run together almost every report and program had been written based on assumptions of the limits of the first system, so just getting rid of it was not enough, we had a huge expensive knot to undo even after it was gone. What makes things worse on a time frame like this one is that it becomes tribal knowledge that this is even how it happened. Most people in our IT department think that the limitations are caused by capabilities of the order entry system, they do not realize that they are in fact limitation of a system we have not used in ten years still echoing in our business!

 

Could that have been handled better? Hell yeah! But how do you know the right decision at the time? There is much more wisdom in the IT world now than there was twenty years ago in terms of understanding how agile technology needs to be for upgrades, but we will continue to bump into these types of problems simply because technology moves so fast.

 

A favorite story of mine is about why the space shuttle solid boosters are made to the size that they are. You would figure that building the space shuttle the engineering would be based entirely upon best fit for the job, but look deeper at what determined the size. The rocket boosters are manufactured in Colorado and shipped by train to Florida. During the trip they go through a tunnel. The size of a train tunnel was therefore the constraint to the diameter of the boosters. The size of the train tunnel is based upon the stability of a train car on standard tracks. The tunnel only needs to be a few feet wider than the train and this is greatly determined by the span between the tracks. The span of railroad tracks was determined by carriage builders who made the first train cars. They used jigs that they already had for making horse drawn carriages and this quickly became the defacto width for trains. The jigs for carriages dated back to the Roman's and their roads. The chariots of the Romans wore grooves in the road systems of Europe, carriages were made at the same width as the grooves in the roads to prevent wheels from breaking. The Romans had a very practical way of determining the space between the wheels of a chariot; they needed to be able to fit two horses between the wheels without the horses needing to step into the grooves or bump the wheels, so it was this criteria that dictated the standard wheel span on a chariot. So if you follow all of this, the diameter of the space shuttle solid rocket boosters was determined thousands of years ago by the width of a horse's ***.

 

Now that is a knot.

 

Bill

[InfiniteNow]

Lol... TBD must have turned in his last assignment of the semester already. Nice post, Bill. :naughty:

[Buffy]

About 20 years ago I had a builder mate who went to the US. He applied for a job at a company that built houses. The boss said 'what do you do, footings, floors, framing, cladding, walls, roofs?'. He replied 'what are you talking about, I build houses'. He was foreman within 2 weeks.

That's the American Way! :shrug:

 

I have a friend who is a general contractor who went through each of those specialties, taking a huge pay cut every time he switched, then got a contractor's license in each one, and now he's got one of the most successful--low cost and high quality--in his region.

 

Is he the best talent at roofing or plumbing or cabinet work? No, but he understands all of them at a fundamental level, and that's what makes him a good manager.

 

So,

Do you have anything to say about specialisation vs generalisation as it appears that you are a specialist generalist?...You're not one of those people who think that 'specialist' managers don't actually need to know what the people under them do (like dilberts pointy headed boss), i.e. all they need to know is how to manage, or do you think they actually should know what they are telling those beneath them to do?

I guess I don't see a very strong dichotomy between "specialist" and "generalist" as you might be hinting at here.

 

Like my contractor friend, I'm hardly the best programmer around, in fact I've spent the vast majority of my career doing *marketing*, and my work has been primarily focussed on making sure that the development *managers*--specialist generalists for the most part--don't get away with claiming something can't be done: its too hard to do that when the response is "do you want me to show you how?"

 

This is quite a bit different than what I'm doing now, which is *being* the development manager, and my style is--like Pyro describes above--is all about encouraging and suggesting and setting goals, although it being a small startup, I lead by example with the designs I create and the code I write.

 

I think that while it is *invaluable* to know what your staff does at a very concrete and practical level, that the task of *managing* requires you to completely turn your "specialist" skills inside out and start thinking like everyone *except* those specialists that you manage.

 

The notion of the "pointy-haired" Pure Manager though is a falsehood, albeit a very strongly ingrained one left over from Management Theory of the 40s-60s. I personally think that Robert McNamara's bringing the bean-counters in to run the Vietnam War was the beginning of the end of this as far as the academic community was concerned: by the time I went to b-school it was a subject of derision, even though it still has lived on even in high tech, with horrible examples like John Sculley. Sculley was a "specialist" in consumer foods, hardly similar to even "consumer computing" if you want to describe what Apple's business is), but because he had no clue as to how to pick and choose among any of the alternatives, he could hardly "lead" well, and failed.

 

As a general rule, these sorts of Pure Managers have mostly inhabited the middle rungs of management because they quickly hit their level of incompetence, and what you'll notice in the downsizing that's been going on for the last decade or so, its this middle layer that's been decimated.

 

That's no loss in my book, but its also no *worse* a sin than the "specialist" promoted to manager because he does know what his employees do, but simply can't make the huge leap that it takes to actually step back and *empower* people.

 

Larger sizes produce the pathologies of centralization, rising general and administrative costs, rigidity, and bureaucracy. And the only antidote is local unit autonomy: Decentralize or perish, :naughty:

Buffy

[Pyrotex]

Buffy is right on target.

 

In being a software project manager, it is crucial to have been at least a "good" programmer yourself--both as a task programmer and as a member of a programming team. Being a task programmer allows you to be far more creative and experimentive. Typically, it gives you the time to turn out a polished product. Being a team programmer teaches you how to get along with others and establish working interfaces.

 

Through all of this, you have to watch out for psychology. Writing byzantine spagetti code is STILL thought of as "job security" by the naive. The repulsion that many programmers have for ANY kind of "writing" (not just comments in code) is a cultural dysfunction that goes all the way back to the days of punch cards: cards took up a LOT of cabinet space. Carrying more than one modest sized card box across campus at one time was an invite to disaster.

 

Then that dysfunction also fed on the bias against "secretaries" in the 70's. Secretaries "wrote comments"--programmers coded. And if somebody couldn't understand your code, then **** 'em--they're obviously not REAL programmers. The 70's and 80's were the two decades of macho programming. (I may be illiterate, but I'm better than you because I'm a programmer!)

 

The concept of code as a "corporate asset"--something that should be maintainable and adjustable by the programmers that come after you, didn't really catch on until the late 80's, as far as I could see. It was then that software managers were smart enough (ex-programmers) that they could begin to see the killing costs of "bad code" (and bad programmers).

 

It was about then I became a lead programmer and had the luxury of picking my team. I wanted to see examples of their code. If they refused for any reason at all, I dismissed them. Only if they showed some pride in the appearance of their code did I select them. Number of comments was immaterial--consistent neatness and pride in the visual appearance of their work was the main thing I looked for.

 

As a lead/supervisor, I also wrote code, usually the central core. All team members got to see readable, logical, well-laid out, adjustable code (many for the first time!). They got to see a well-thought out variable naming system (most for the first time!!). They got to see a designed-in subroutine debugging system. I expected them to follow by example. When there were tricky bugs, the first thing I did was conduct a code walk-through. If it was too messy, I would give them 24 hours to clean it up so we could walk through it.

 

So, by example and by expectations, most of my programmers rose to the task and performed at levels that they admitted were easily double their prior productivity. Rather than give bad annual reviews to the arrogant plicks who were "too good to comment and use coding standards", I just gave them the little tasks that would have minimum impact on the product. This pissed them off, and I would explain why. Then they were faced with a choice.

[LaurieAG]

Hi Pyro and Buffy,

 

I'm glad that we all agree with regards to what constitutes proper team management and prevention is better than cure but we must admit that things aren't perfect everywhere and somebody has to come along afterwards and clean up the crap when it hits the fan.

 

It looks like, in Symbology's situation anyway, that this is the problem faced. We already know what the ideal situation is so getting there is the most important problem to be solved.

 

So any suggestions about Symbologies repeating problems?

[Buffy]

...but we must admit that things aren't perfect everywhere and somebody has to come along afterwards and clean up the crap when it hits the fan.

Well, even at places like mine where I get to pretty much run the show, its hardly perfect!

 

In my case, the first version of the system was written by a bunch of typical "web developers": they knew how to "script" but the concept of modular code, or coding for supportability were alien concepts (that's why we're in the middle of our 3rd rewrite!).

So any suggestions about Symbologies repeating problems?

If you take Symb's bullets above:

• Do I just "Git 'er done" and mash in my changes....
  
• Do I spend the time to (sometimes document and) clean up the other pages that I had nothing to do with...
  
• Or do I gut the whole system and put in a centralized set of functions, web pages, and/or procedures where changes can be made in one central place?
  

The answer--really between the lines of Pyro's and my posts--is "yes."

 

The specialist sees these options as mutually exclusive: the generalist sees them as pieces of the only practical solution.

 

Obviously you can't carry all three to their logical--but in reality absurd--extremes. Its essential to see the fact that hacking, clean-up, and rewriting each are a piece of the solution, and the cleverness is not in the coding but in balancing the approaches to not just maximize the value of your time, but to make it *look* to the outside observers that you're doing them a huge favor in getting them out of the mess the previous gang of idiots got them into.

 

Forget about what the "perfect" approach is, realize its all about communications, alignment of goals and getting people to understand what they're really getting for their money.

 

This isn't about strictly following some ideal, its about making sausage, and its never pretty, but it sure can taste good with the right combination of "perfection" and "git'r'done," and remembering that those silly garnishes make the difference between fast food and haute cuisine.

 

Art is never finished, only abandoned, :tongue:

Buffy

[LaurieAG]

There are other options Symbology.

 

Alternatively you could do a cost/benefit analysis of all of the available options (there's probably still many more) including the current status quo (just doing things in an ad hoc manner). Work out an estimate of all costs to date (pick a suitable time span, 1-5 years or however long) fixing things under the current process and then project these expenses into the future including a percentage increment each year. Do the same for each potential solution.

 

Once you have justifiable figures on current/projected costs, which are hopefully much more than at least one of your (palatable) solutions, you will know which path to proceed along with management. If it can save money the option will be listened to. I hope this helps.

[Symbology]

There are other options Symbology.

 

Alternatively you could do a cost/benefit analysis of all of the available options (there's probably still many more) including the current status quo (just doing things in an ad hoc manner). Work out an estimate of all costs to date (pick a suitable time span, 1-5 years or however long) fixing things under the current process and then project these expenses into the future including a percentage increment each year. Do the same for each potential solution.

 

Once you have justifiable figures on current/projected costs, which are hopefully much more than at least one of your (palatable) solutions, you will know which path to proceed along with management. If it can save money the option will be listened to. I hope this helps.

 

I figured the correct answer would involve more work for me. And that is ok. Like I said it seems to come down to us to properly "edumacate" our managers and fellow team mates. Creating the cost/benefit analysis requires the time to put some concrete estimates on (some ridiculous) things. But that then lets the authors of the ridiculous things see that they are... well... ridiculous.

 

So thank you. That makes good sense. And I think I would rather spend some personal hours working up the cost benefit analysis and keep the decisions above the table, than instead coding after hours and sneaking in the changes - and deal with the fall out of resentment from programmers having their code replaced, and "I didn't tell you do do that" from the managers, and any blame for reworked code that isn't perfect the first time.

 

Of course then the hazard is they may still decide to cram it all in there, and that there isn't time to do the reworks (typically the opinion). But then I can let it go that I clearly warned them in writing.

 

I'm looking forward to finding the utopian job where none of this is an issue, and everyone believes that doing things right the first time is the most important thing.

[LaurieAG]

I'm looking forward to finding the utopian job where none of this is an issue, and everyone believes that doing things right the first time is the most important thing.

 

Don't forget to do the analysis on this option too, you never know:).

[TheBigDog]

If everything were done right the first time there would be no place for innovation or improvement. I have no problem with redoing things so they are better. I know that I do the best I can with what I am given for each project, and I know that it will be replaced down the line by someone who may well think I was a fool.

 

Bill

[Pyrotex]

Some of the valuable lessons I have learned on the job:

 

1. Learn to write well, and express a coherent thought persuasively -- on paper. People are far more likely to follow you if they understand you.

2. Learn how to use a graphics application (like Canvas) to draw the kinds of diagrams that are worth at least 1,000 words. Suddenly, two hour meetings become 30-minute meetings. Any programmer who can draw intelligent diagrams automatically becomes a lead programmer.

3. Learn the Ladder of Abstraction!! If the only way you know to explain what you did is to give a blow-by-blow description of the sequence of all the individual actions you performed, then I'm talking to YOU. You need to rise above the "sea of details" and be able to describe software, requirements, problems and schedules at an appropriate level of understanding that fits your audience and the time available. So you can give the 1-sentence answer, the 1-page answer, the 15-minute summary, and the 1-hour detailed presentation (all to the same question) -- without bogging your listeners down in petty minutiae.

3a. If while giving a listener an explanation of your software, you see his/her eyes start to flicker shut, you are at too low a level of abstraction. Go up one rung.

3b. Learn to listen to yourself talk while you are speaking.

4. Prioritize functionality! Code the most important features FIRST. Bells and whistles have to wait.

5. Show the programmers a DESIGN before they get caught up in the coding. By definition, a "DESIGN" is on paper. If it's in your head, it is NOT a DESIGN -- it's just an idea.

6. Design your interfaces with utmost care. That way, a poor programmer might produce poor code that can be "isolated" from the rest. As long as the ONLY contact his code has with the rest of the system is, say, four well-defined integers, then unintended consequences can be kept to a minimum.

7. Design the User Interface FIRST!! Write a 10-page User's Guide before the first line of code has been generated. Whoever circulates the first clear User's Guide, controls the design of the system.

8. Design the database SECOND.

9. Learn to perform at least a basic Cost/Function/Risk trade study.

10. Be generous. Give away all your secrets! Give away all your experience! Hold nothing back.

[Pyrotex]

If everything were done right the first time there would be no place for innovation or improvement. I have no problem with redoing things so they are better. ...

Well, there's "right" and then there's "RIGHT". One is a reflection of functionality, and the other is a reflection of quality.

 

In the sense of correct functionality, everything MUST be done right the first time, or the project fails. A compiler that produces execution aborts, or a database that becomes corrupted through normal use, is worthless. Projects that depend on such as these, are in grave peril.

 

Everything should be "right" in that regard. But you may still have a minimum solution on your hands, one that is difficult and cranky to use. Consider the original IBM job control language (JCL). It drove ordinary programmers insane--they quit their jobs and took up water colors and knitting. And yet, it performed exactly as required--it controlled the execution of everyone's jobs exactly as they requested. It wasn't IBM/JCL's fault if people did not know what they were requesting. ;)

 

It is in the arena of being "RIGHT" (quality, performance, flexibility, efficiency) that TheBigDog is correct. We can't always afford to get it RIGHT the first time. We seldom know what RIGHT even is, the first time around. As with JCLs, it took 20 years of experience and innovation to understand what an operating system OUGHT to be doing, and how we should be conversing with it.

 

A database MUST be "right" the first time or you're screwed. But there is plenty of time afterwards for you and your descendents to make it RIGHTER and RIGHTER and RIGHTER.