Hardly you can get a better idea of what a deadlock is than looking at this picture:
(from The daily WTF)
Incidentally this appears to be not just a real life deadlock, but also the results of software bug as most of the deadlocks I’m faced with turn out to be.
I also guess that most of the drivers and passengers involved in the jam would have heartedly approved that extra bit of testing, or that extra money required to hire an experienced professional that would have prevented a deadlock-avoiding device such as a traffic light transformed itself in the deadlock-causing device.
Talking about problem-solving, how would you recover from this situation?
I am very old
It is official, at least according to the young man (he’s 20, I can’t call him a boy) I give private programming lessons to. He said that his class is full of “old people” – most of them is even 30 or older! Scary! Anyway I have been programming for more years of my life than not and most of them have been spent on the C language. Maybe that classifies me as very old nonetheless. After all C had been designed and developed nearly 4 decades ago, when I was 4.
One of the question in an exam attended by my tutored boy was:
Define the result of the following expression given x=N and y=k
x = (y+1>x) ? x++ : y++;
Suddenly a bell rang in my head. Red alert was buzzing at full-volume, but another bell was ringing as well.
The main red alert was labeled as “Undefined Behavior”. Every bad thing could happen when U.B. is invoked by the unknowing programmer. If he is lucky then he just gets something out of the order he expects, otherwise he can blow everything up.
Back in the days where Real Men wrote their own compiler, the C designers decided to relax some constraints in the language semantics so that more aggressive optimizations could be implemented in the compilers. So within a C expression it is not defined the relative order of the side effects. If two or more side effects apply to the same operand, then you have a problem. Moreover the assignment operator (=) in the C language is just an operator with a side effect, so the expression:
x=x++;
is undefined behavior.
The day after I investigated on the C language FAQ and found that the ternary operator (? 🙂 introduces sequence points (a jargon to say that all the side effects have to happen before a given point in a language phrase), so the expression may, after all, be well defined.
The scaring part is that a random C programmer like me, with over 20 years of exercise in the language, could be baffled by such expression and unable to tell apart the result of such a line.
And that’s why the second buzzer triggered. Why young blank minds have to be troubled with such visions? If a program contains one expression like that chances are high that it is a bug. If it is not a bug you may well count it as such because you are going to have a hard time figuring out what’s going on in that code.
There are some values in programming that ought to be taught before putting hands to the keyboard. Simplicity is one of the most valuable principles – keep it simple. You have to understand it, even after months, others have to read and understand it as well.
Simplicity is meant to fight against the common distortion of the young programmer trying to assert himself considering (and unfortunately, writing) “concise, hard to read” for good code. “Hey, I bet you cannot find what this piece of code does”.
I think teachers should make serious effort, especially when dealing with languages such as C, where cryptic and short code could be easily written, to discourage one-liner approaches and striking that the great programmer is the one who is able to tackle complex subjects with simple and clear code.
Writing this could be another sign of my age…
Out of the way
It has been a few days since I have had this thought in and out of my head. There is no particular fact that triggered this off; I think it’s just some random observations. We programmers like computers and software. That’s obvious – we find them entertaining, funny, sometimes even a sort of lifestyle. Wouldn’t it be the case, we would have chosen something different, more economically rewarding (such as the plumber), more socially oriented (e.g., the plumber), or even more fun (water’n’pipes are fun, aren’t they?).
I find that more often than not, our programs have features and behaviors just to amuse the other programmers or even ourselves, forgetting that real users – those who will use our program to do some actual work – aren’t so fond of computers besides having some actual work to do.
I think we should enforce a new paradigm for program development: the computer is actually an obstacle, a barrier between the user and her goal. The software has to clean the road and keep out of the way.
Just ask yourself if that “Error Blah blah” dialog box you are coding is something that helps the user to reach their goal and perform their task, or is just something that will annoy them. Is it really a user’s fault if the file has the wrong name? Are we really sure that the user has to know what a file is?
For another example, consider the need of a user to deliver an amount of data (files) to another user. The obvious choice, since everyone has an email account, is to use it. That is the need for the user and the clear, logical solution she sees.
So, the size limit for attachments is just an obstacle to the activity she has to perform. But the real solution is not to grow the limit, but to take the limit off. If the mail protocol is not suitable for bulk transfers, then the software has to use other means. Why can’t it arrange an ftp transfer? Sure, it’s not as easy as I put it, but it definitively can be done.
On the same issue, some users fall in love with technology and forget any other sensible way of doing things. So their activities could become extremely time-wasting. I needed a reference on how to attach cables to an electronic board for testing. I looked in the usual repository just to find every kind of documentation, but the one I was looking for. I asked the guy who was supposed to provide that kind of information, and he told me that he was unable to provide the board layout because the software is not yet able to do it, and that we need another module for … and so on. I took paper and pen and sketched the layout on a sheet of paper, then scanned it and put it online on the server. It took me about 20 minutes because I had to look up the schematics. The document I produced is not polished, but… who cares?
Software project failures
Yesterday I stumbled upon an article about why software projects fail. Though hardly you can find any surprising claim while reading, it is nonetheless a quick paper you can hand over to your colleagues and/or managers when trying to enhance the chances for a successful project.What I find quite interesting is the table near the beginning, the one titled “Project Challenged Factors”. What draw my attention is that the “other” item weights more than each other single entry in the top ten. In other words the causes for challenging projects are so widespread and so equally important that there is no a main culprit.
We programmers are working on a so complex and delicate mechanism that any one in hundreds of factors could drive the entire stuff crazy. On the other hand the number of variables involved is so high that it is not possible to shot at one target to likely ensure project reliability.
Take software requirements for example, (missing or incomplete accounts for a 12,3%), while I am not advocating about entering a project without them, it is clear that in some circumstances the lack of such documents is not dooming per se. Take for example an expert team that knows all the ins and outs of the domain and has a crystal clear idea of what is needed. In this case maybe the project could get along with no SRS.
So one might be tempted to strictly follow every “best practice” to minimize the risks of a failure. But this is not likely to work either – every practice has its cost and some practices (take the “code review”) are extremely daunting.
Therefore is all a matter of balancing, picking the right decisions, having good reflexes and willing to occasionally work harder to fix what has gone wrong.
Following this line of thought the best insurance you can have for your projects is an experienced and well jelled team with: a) an experienced leader, b) a good success history. This isn’t rocket science either, you can find similar assertions in many software project management books (Peopleware is the first coming to mind).
On the other hand projects keeps failing. This leads me to two considerations, first is that even with the best premises a challenging project could fail – we have to accept this for today’s projects and try to minimize the failure impact. Second how is changing the failure rate in time? The first study quoted in the article is more than 13 years old! That’s a huge time in software industry.
Spaces within
“Unix doesn’t works properly with filenames with spaces”. This assertion from a coworker of mine prompted my harsh reply: “Unix works perfectly with spaces, it’s just programmer sloppiness that prevents it from properly handling blanks”. I was true (as always, I would humbly add) but my pal wasn’t completely wrong, at least when talking about the specific subpart of Unix named shell. Although far superior than MS-DOS batch shell (that’s about the same command line you find in Windows) I bet it originated in more or less the same way – a core with functionalities clustered over during time in response of new needs or new opportunities.
Unix shell (be it bash, ksh, zsh or fish) is nowadays a powerful programming tool allowing the programmer to craft rather complex artifacts. This scope is for sure much broader than the one envisioned by first developers, and this turns out in multiple ways to do the same thing, different ways to do similar things and cryptic ways to do simple things.
The Unix command line conception dates nearly 40 years back in time! Things were pretty different, but I won’t annoy you with details, just leave your imagination wild… likely it was even worse. Fish is a recent attempt to overcome most of the shell problems, but it is not widespread as bash could be. As put by a professional some time ago: emacs may have tons of neat features, but you are SURE you’ll always find vi on any Unix, while you are not certain you’ll have emacs. So better invest your learning time on vi.
Well, back to shell. What’s wrong with blanks? The main problem is that a space is a valid character in a file name and, at the same time, it is a separator for command line arguments. Back to when every single byte could make the difference, it seemed the right thing to do to have optional quotes if the filename doesn’t contain any space. So you can write:
$ ls foo/
To get the listing of directory foo, but you have to write:
$ ls "bar baz/"
if you want the listing of directory “bar baz” (or you could escape the space with a backslash). This could be boring on interactive shells, but is usually overcome by the auto-completion feature (type ‘ba’ then tab and the line gets completed with the available options, in this case: bar baz).
From boring it turns in the range from annoying to irritating in shell scripts, where variables are not real variables like those you are used in high level languages, but just convoluted macros. For example:
a="bar baz" ls $a
is processed and interpreted as:
ls bar baz
As you see the quotes disappears because they are processed by the assignment to put bar+space+baz in the ‘a’ variable. Once ‘$a’ is expanded, quotes are just forgotten memories. In order to write proper shell scripts you have to do something like:
a="bar baz" ls "$a"
Of course this is error prone, because not only the syntax is valid, not only the script is likely to work perfectly in the simple test case the programmer uses to test the script, but also it is likely to work fine most of the times. After all the space character is used only by those Windows naĂŻve users that aren’t aware of the blanks-hating-device they keep hidden under their desks.
Well, I proud myself of writing space-safe shell scripts, at least until I tried to write a script to find duplicated files on a filesystem.
The goal is simple, after many virtual relocations, multiple pet-projects and home-works, I have many files scattered around with the same content. It is not a matter of space saving, rather it is a question of order. Avoid redundant information or make sure that it is really the same stuff.
My design was to have a command similar to ‘find’, something that accepts any number of directories or files on the command line, such as:
$ find_dupes dir1/ dir2/ file …
Shell has two ways for operating this pattern – use the shift command or use one of the special variables $@ and $*.
The first way is useful if you want the shell to process one argument at time, while the latter is handy when you want to relay the command line to a command. In my case I wanted to pass the entire command line to the ‘find’ command, something like:
$ find $@ -type f -exec md5sum {} ;
This line works fine until a filename with space is encountered. In this case, since variables are indeed macros, a single argument with space is expanded into two (or more) distinct arguments. And there is no way to work around the limitation, unless you read the manual J. In this area, the discoverability of bash is quite lacking. The man page states that $* expands in the sequence of arguments separated by the first character of the IFS environment variable. E.g. if IFS is set to dash (‘-‘) and the command line has the following arguments foo bar baz, then $* expands to foo-bar-baz.
Conversely $@ expands to a space separated sequence of arguments, but if you enclose it in quotes, then single arguments are expanded in quotes. E.g. $@ expands to foo bar baz, and “$@” expands to “foo” “bar” “baz”. Eventually this is the solution.
So, basically it is true that Unix has no problem whatsoever with spaces inside filenames, it is also true that shell programming can handle them as well and ultimately is up to programmer sloppiness if the batch script fails, but it has to be recognized that a great effort and investment is required to the programmer to climb out his sloppiness.
Mind if I comment?
When I worked at Ubisoft I conducted some job interviews for programmer positions. As the interviewer, the company provided me with a grid of questions/topics to ease the candidate evaluation. A list of questions, when not taken too rigidly or in a quiz-like fashion, offers the interviewer a good source for firing up some interesting talks.One of these questions was “What are the qualities of a good source code?” In fact this question is open ended enough to let the interviewed express himself (or, in very rare occasions, herself) on what he (she) considers good or bad programming habits, helping the interviewer understanding the programming-wise maturity of the job applicant.
Counter-intuitively, one of the most common answers I got first was not “don’t goto” as you may expect from school fresh programmers, but “comments”.
The virtues of commenting are taught in every entry level programming class and I think that “commenting” is even more agreeable for everyone than “don’t goto”. After all it takes a few days off the code you were writing to recognize the importance of stuffing your work with the code equivalent of post-its.
What actually most teachers fail to convey to their audience is that comments are just a tool, actually quite easy to abuse and thus causing the opposite of the desired effect. Misuse of comments reduces code readability.
I’m going to demonstrate how to actually remove comments while IMPROVING readability. Just don’t misquote me; I am not against comments at all. If in doubt, comment your code! What I’m going to show you is that by removing comments AND carefully crafting your code the readability will improve.
First: terse is beautiful. Consider prose for example, the shorter the sentences the more comprehensible. Comments add repetitions, extra information and dilute what’s going on with redundancy. If we could write good code first, then, maybe, we could leave out most of the comments.
Moreover comments burden maintenance. An old programmers’ saying tells that if a comment mismatches with the commented code then both are wrong. Maybe this is not always true, at least not for the most trivial cases, but it for sure marks a point – comments have to be maintained. An outdated comment may be from disturbing to misleading causing code maintenance problem. In fact rather than helping you in figure out the way the clockwork turns it confuses your understanding.
Let’s start trimming from variables. It is way too common to comment variable declaration:
int b; // number of bits set to one
Why can’t b be named as oneBitCount? By picking the right name not only we avoid the comment, but we’ll make the lines where this variable is used more clear – our brain need not to remember what b is and we don’t have to look it up in the declaration section.
Using b instead of a more meaningful name denotes two levels of laziness – typing laziness and naming laziness. About the first, a good editor could do much. My editor of choice allows me just to type the first letters of any symbols and then cycle through all the available completion in the file, or the included files or the ctags file.
About the second, there’s no excuse, take your time to chose right names, the ones that best describe the purpose of what they identifies. Names are powerful handles to concepts, the right name is not just for better readability, is for better grasping and processing ideas.
This could be the right time for a little digression on Hungarian Notation, but I don’t want to get this post too crowded, better save it ol’ HN for another time.
Another common misuse of comments is to mark the semantic of a block of code within a function. E.g.
// count bits set to 1
b=0;
while( v != 0 )
{
if( v & 1 )
{
++b;
}
v >>= 1;
}
// ... the function continues
This kind of code is easily turned into functions and the block can be reduced to the single, very clear, line:
b = countBitsAtOne( v );
This has also a number of side beneficial effects such as reducing the size of the including function, clearly defining inputs and outputs of the code block, clearly marking where the code block ends.
In other words not only the code is more readable, but it becomes automatically more maintainable because the maintainer has less chance to mess something up.
I have heard two objections to this approach – execution inefficiency and programmer inefficiency. The first is very unlikely since a common optimization operation performed by every compiler is to make an inline copy of short functions where they are called. For long functions the call cost is not relevant. Anyway the old optimization tenet is always to be remembered and applied – measure before optimizing. The programmer judgment is far from fair in most but the trivial programs. (There is another good rule about optimization: Don’t.)
About programmer efficiency, it may be true that a newbie codes like a rocket and feels that spending time in anything but writing code is a waste of time, but the result is usually a write-only source code that no one is willing, nor able to fix or modify. Moreover the time taken to write the good code at first time is much less than writing bad code and come back later to refactor it in something more convenient. In fact for most code blocks is very difficult to pick up inputs and outputs and be sure the no side effect has been missed in refactoring.
This very same technique can be employed to reduce the size and improve readability of large switch-case statements.
There is another use of comments similar to the one just described. An obscure piece of code is commented with the intended functionality. This goes in a quite grey area. I mean, there are times that code has to be obscure because it is handling unusual or complicate stuff. So explanation for, let’s say, floating point math implementations is more than welcome. In all the other cases it is just smoke hiding what’s really going on. There may be several causes (ego, job insurance, pretending being smart…), but the result is always the same unreadable code. Here holds the principle: don’t do anything that may surprise the reader.
Write simple code unless there is a compelling reason to do otherwise.
Friends and namespaces
There are C++ behaviors that may leave you a bit astonished, staring at the lines on the monitor and wondering why the code isn’t compiling, or doesn’t work as expected. Just stumbled in one of these cases.
I usually follow these steps to recover from the puzzled face. First I write a minimal example that reproduces the behavior. It should be a bunch of lines in a single file. Sometimes this could be a daunting task, but I have that it is always worth to grasp the problem.
In fact, once you have the minimal code, you can easily experiment, changing and twiddling bits to see how the behavior changes.
Then you have two options – you can ask your local C++ guru about the problem (if you have one), or you can google the Internet for a clever selection of keywords that describes your problem.
So what happened today?
I decide to move some code I developed into a namespace-constrained library. Everything compiled happily outside the namespace, but failed to do so in the namespace. After some headscraping, I started cutting and shaping a minimum file with the same odd behavior. Here you are:
/** prova.cc
*
* @author Massimiliano Pagani
* @version 1.0
* @date 24/04/2007
*
* @notes
* @history
*
*/
#if defined( USE_NAMESPACE )
namespace NS
{
#endif
class A
{
public:
private:
struct B { int x; };
friend bool fn( B const& b );
};
#if defined( USE_NAMESPACE )
}
using namespace NS;
#endif
namespace NS
{
bool fn( A::B const& b )
{
return b.x != 0;
}
}
Now, if you compile it defining the symbol USE_NAMESPACE (e.g. via g++ -Wall -DUSE_NAMESPACE -c prova.cc), then you get the odd looking error:
prova.cc: In function 'bool fn(const NS::A::B&)': prova.cc:21: error: 'struct NS::A::B' is private prova.cc:31: error: within this context
While if you compile without the namespace everything works as expected. Since the error was quite meaningless to me, I started investigating on friend and namespace. After some mailing list browsing, I figure it out. And it was simpler than what appeared – just a case for a misleading error.
In fact the friend statement declares a function fn somewhere in the NS namespace, while actually fn is defined in the global namespace. In fact there is just a using statement. To fix the problem, just move the fn function into the NS namespace.
Well and I have figure it out alone, without the need of calling my uber-C++-guru friend Alberto.
On a completely unrelated topic, today is the 25th anniversary of the marvelous ZX Spectrum. Happy Birthday Dear Speccy.
Programming in Lua
(cover to the left here, refers to 4th edition of the book, I read the 1st edition). It is somewhat difficult, in writing this review, to distinguish the language from the book. The book teaches about Lua, so my opinions in favor or against this language could interfere with my opinions about the book. Anyway I’ll try to accomplish this hard task.When writing a book about a programming language you could follow either an hands-on approach or a more formal one. Both ways have their pros and cons. The first approach helps language novices to grasp quickly language fundamentals, while latter provide a convenient reference for those already using the language.
For example Bjarne Stroustrup’s “The C++ Language” belongs to the second set. It is very formal and descriptive; learning the language by this book it is definitely the hard way to do that (aside from the fact that C++ is a very complex and huge language). “Programming in Lua” is from the first camp. It features an easy to follow, example-ridden way into the language. It doesn’t pretend to give a thorough reference for the language or the library, demanding other books for this specific purpose.
I have to say that the writer does quite a good job. The text reads smoothly, examples are always fit to the chapter scope, prose is clean and clear. In theory I should have practiced with the language itself in order to state whether the book covers the matter adequately or not, but I have a limited amount of free time and no suitable project for a Lua application, not even a toy one.
Anyway while I didn’t fell in love with the language (quite the opposite as you may have read) I appreciated the book. Also I found an interesting reading the last chapters on how interfacing and extending lua with C code.
What I didn’t like is the author bias toward the language; I feel a vein of naivety. For example Lua supports only floating point values. Not the “float” kind, but the “double” variety. No integers, just doubles. The author advocates that there is no reason for integers since “double” works fine for just everything. Although I have not much experience with doubles (just floats), this appears quite a bold statement, at least from what I have skimmed through. The dumbest consideration I could think of is that doubles still have the usual precision problems that affect floating point arithmetic. It just depends on how many iterations you have to do.
On a whole I found this book a worth reading either if you want to learn a new programming language or you are just curious (like me). Just be sure to buy the 2nd edition that came out just few days after I ordered the book (I ordered the book few days after having read the, now disappeared, slashdot review… maybe you see a connection)… or to read the first one online for free.
Antipatterns at work
It is an odd sensation. I am puzzled about some decisions that are quite unanimously recognized in literature as bad, or, if you prefer the buzzword, as anti-patterns. Forests have been sacrificed for books on the matter. Given the current (poor) state of the planet I would urge management to read them, not just having them to preserving their libraries from dust.
You have this project late, the customer is coming screaming at you because you didn’t made it for the milestone, the product is still alpha quality when it ought be on the shelves. Sounds familiar? If not either you don’t work in any technologically advanced sector or you have been unbelievably lucky (or good… or both). So it is clear that the project needs extra time to achieve the goal set. I understand that there are contracts and politics involved, but for the sake of the project there is just to plan and move the milestone ahead WITHOUT adding new features. Leave them for another project. The customer could be disappointed from this refusal but she will be a lot more disappointed when you’ll told her that you didn’t made it another time.
Disclaimer: I have to be honest, this is not my project, I don’t know exactly what has been agreed on, the only thing I know is what a coworker told me in about a single line: milestone postponed, new features added. So basically I don’t know what I’m speaking of.
The Good, the Bad and the Ugly
Engineers are supposed to deal with systems, usually large and complex devices that are expected to work. They are expected to work at least, but the common expectation is that they keep working reliably under a wide set of circumstances and mostly regardless of the environment. Consider any system that results from an engineering process (say a TV set or a car) from this point of view – it is not enough that the device works fine only at the seller’s location. This should put under the right light one of the most used programmer defenses: “It works on my PC”.
For us, as programmers and programming team leaders, the point of concern is how to deliver such robust products that could reliably work even in non-clean room conditions. A lot of entropy has been sacrificed to this goal and I don’t want to start anything new or propose any existing process methodology, I’d just like to recall what a friend of mine reported from one of his senior coworkers (thanks XtĂ©).
This is a quick analysis that allows you to understand how much in good shape your project (or task) is.
Consider two nearly binary variables – working and understanding. You can get four combinations that identify four states of the system composed of the object the engineer is working on and the engineer itself. Here we go to analyze the four states (I swear, I’ll be brief).
It doesn’t work and you don’t know why. This is the typical state of the project start, you have this black box not working as expected and you are supposed to fix or implement it. This is not so bad if you have enough time to study, ask other people, analyze, and get your knowledge on what you need to do to fix it. Summing it up, this can be fair or bad according to the time you have.
It doesn’t work and you know why. This is quite good. After all, knowledge is power. With the knowledge you can both devise strategies and solutions and figure out the time or the means you need. Moreover, you have plenty of facts to explain the situation to the management and ask for the most suitable resources you need to accomplish the task.
It works and you know why. Perfect, you achieved your goal. You fully understand your system, and why is it working so that you can predict to a good extent when and how it is going to work.
It works, but you don’t know why. This is the worst case of all. Unfortunately, the rush and the wrong assumption that if it works then who cares, can lead to this terrible situation. The real problem here is that you cannot make any assumption for when the system will cease to work, you have no clue how to deal with it both to repair, move, or change. You have no warrant that the system will continue to work.
The fact that it works usually leads the project management to consider it complete and to make pressure to move on. A false sense of security may affect the team cleaning the way for greater disasters.
There are some quick corollaries to this analysis. First, always try to understand what are you doing even if it may seem like a waste of time. Second, always include a learning time in your estimations. Third, poking randomly around to fix things is a dangerous way to further damage a system while creating the illusion of work.
Programmers strongly rely on tools, basically, there is direct contact with the matter we design and develop, our tools are our manipulators and probes into the hidden work of electrons. We have to know our tools by heart. We cannot go away with a rough understanding of the language we use because we cannot afford our ignorance would let something in.
The natural question that could arise is: “What is the extent required for the knowledge?”. Does programming require me to understand OS internals? Digital electronics? Semiconductor physics?
Well, it depends. Gone are the times when a single man could brace the whole human knowledge and art in his lifespan. Nowadays we have to stop at a given interface, taking for granted that what is going on behind us is good enough. I think that you have at least a good knowledge of the first interface you are using, be it OS system calls, environment libraries, or hardware if you are working closer to the metal. Anyway, an average knowledge up to the next barrier could only be good when you are hunting for problems since it could help you to better exploit the environment and to get helpful hints to get you out of trouble… troubles where engineers spend most of their time.