Tag: programming

Java vs. C++, last call

Well, I dragged this for too long. Now I have to pick the language for the next project. My language of choice is C++, but there are some serious motivation for using Java, one is that all of the junior programmers involved has some experience of Java, another is that middle management is biased toward Java. So, I can still chose C++, but I have to put forward strong and sound engineering reasons. Also because this can become a little … political. I mean – should the project be late for “natural” development reason, or whatever other reason, I don’t want to be blamed because I decided for a language perceived as complicated and rough.
The application isn’t going to need machoFlops or be resource intensive, therefore performance-wise there is no reason to avoid Java.
Java has a thorough and complete library, quite homogeneous even if not always well designed.
I consider C++ to be more flexible and expressive, but is there a more demonstrable software engineering property of C++ that I can use in my argumentations to favor the choice of this language?
(Or conversely is there any demonstrable advantage of Java the could convince me to switch my language of choice).

C++ lambda functions, my first time

Some days ago I wrote my first lambda expression in C++.

    int n=0;
    std::for_each( begin, end, [[&n]](std::uint8_t& a){a=n++;});

If you are back at the C++85 age, this could look like any other alien language from a Star Trek episode. If you are updated at the more recent C++98, the syntax would look funny at minimum.
At least, that is what it looked to me before starting to get some information about the new standard (you can find a brief yet comprehensive recap of what changed in C++ on this specific page of Bjarne’s website).
You should read the code above as follows. The square brackets defines the intro for the lambda function. Within the square brackets you should put what the lambda function should capture of the environment.
In the code, I stated that n should be captured by reference (&). I could have replaced the ampersand if I wanted to have the capture happen by value. Or I could have put nothing should I wanted the lambda function to capture everything.
Next the argument comes, not different from standard function. Eventually the function body.
Once you get this quick start, you will easily decode as a way to fill a range with increasing integers. Take a breath, get another look at the code, ok, now it should make sense.
Well, I’m quite with Stroustrup when he says that he has mixed feelings about lambda functions (I am looking for the quote, but I don’t have it handy). For simple function lambdas are a godsend. On the other hand, lambdas can yield a terrific potential of hiding mechanism and causing major headaches should they escape your control.
If you compare the line above with the code you have to write previously, it is obvious that lambda expressions are a giant leap forward.
In the C++98 days you ought to write –

class DisposableNameForClass
{
public:
    DisposableNameForClass() : n( 0 ) {}
    void operator() ( int& a ) { a = n++; }
private:
    int n;
};

//..
DisposableNameForClass disposableNameForInstance;
std::foreach( begin, end, disposableNameForInstance );

And that is very uncomfortable and inconvenient. By looking at code like this it easy to question whether it makes sense to use the std::for_each at all rather than roll your own code.
But, look at the traditional code

    int n=0;
    while( begin != end )
    {
        *begin++ = n++;
    }

This code is pretty clear to anyone has a minimum familiarity with C or one of the derived language (yes, there is the dereference operator which involves pointers, but shouldn’t pose real obstacles to comprehension).
Is it error prone? Possibly as any code longer than 0 bytes. std::for_each saves you at least two errors – messing up the loop termination condition (e.g. forgetting the ! from the comparison, or comparing for equality rather than inequality) and missing the increment of the iterator (this happened to me at least once).
These reasons may not be enough to recommend std::for_each in C++98, but it is hard to argue against if you can use the lambda functions.

A Matter of… Time

As promised, here I am to write about the other bug. This one needs a bit a background to be understood. At the beginning of the current project (that means some 2 years ago) I designed a software timer for providing time services. User code that needs notification about time event may register the software timer specifying the time event. When the time event occurs the user code is called back.
Software timers are identified by a numeric id. Being an 8bit architecture with a severely constrained data memory, I picked an unsigned 8 bit number as timer id. This id indexes a private array containing the software timer parameters.
There are two kind of timers – repeating and one shot. A repeating timer repeatedly calls you back at given intervals, while an one-shot timer just calls you back once when it expires.
Repeating timers need to be explicitly canceled, but when I was designing the class, it felt like a good idea to let one-shot timer self-cancel pending their expiration.
Time passed, Software timer component being tried and tested became trusted. Then I got some mysterious bugs – code that used to work stopped being called back by timer. It took a while to discover what happened. After the one-shot timer executed its callback the user code still had some dangling references to the timer id.
Occasionally user code could inadvertently cancel the timer pointed to by the dangling reference resulting either in an assertion about a non-existing timer (turned into a reset in release code), or in a deletion of an innocent bystander timer.
I thought a while on the matter and I came to the conclusion that it is not good for the timer to self-delete. Rather is must be up to the user code to get rid of the timer. Unfortunately I cannot apply the lesson learned on the current project because the code base is too large and we are releasing, nonetheless I’ll keep a reminder for the next project.
While talking about timer callback, a colleague found another problem – if you perform timer registration and removal during the callback you are in for serious trouble. In fact the code for the timer dispatcher was about like this –

for( id = 0; id < TIMER_COUNT; ++id )
{
    if( timerHasExpired(id) )
    {
        callbackUserCode( id );
        if( timerIsOneShot( id ))
        {
            deleteTimer( id );
        }
        else
        {
            // timer is repeat
            reloadTimer( id );
        }
    }
}

This code appears good as long as nothing happens on the timers repository during callback. Consider what happens if a one-shot timer is canceled and a new timer is created in the callback code…
There are two ways to deal with this – either queue timer creation/deletion operation during the callback and execute them when the above code is completed. The other is to re-consider the current time nature after the callback, i.e. consider the timer repository “volatile” across the callback. This is our choice –

for( id = 0; id < TIMER_COUNT; ++id )
{
    if( timerHasExpired(id) )
    {
        callbackUserCode( id );
        if( timerHasExpired( id ))
        {
            if( timerIsOneShot( id ))
            {
                deleteTimer( id );
            }
            else
            {
                // timer is repeat
                reloadTimer( id );
            }
        }
    }
}

Lessons learned – 1) don’t provide classes that mixes automatic and explicit object disposal – either the user must always delete objects or the user must never delete objects. 2) critical runs are not just about interrupts, but they may occur in callback as well. Calling back code should be programmed defensively expecting that the user code may wreak havoc on the structure. Should this not be possible then a state variable declaring whether the class is in “callback execution” or “normal execution” must be provided and functions that manipulates class state have to be made aware of the state so to act directly or to defer actions at the end of “callback execution” state.

Ehy dude, where did my frame go?

It is nearly three decades since I write bugs code and I am maturing the idea that each new bug I find should inspire a way to avoid it in the future or, at least, to reduce the chance of falling again in the same mistake. During the past weeks my team and I discovered two bugs and here is my ideas for improving the software quality.

The first bug is likely to be the longest-standing bug I had. As you may know, I am working on a proprietary field bus. We found evidence of the problem at least three years ago. Occasionally our devices lose a frame. It is not tragic – it is a frame in 250 – but, since the protocol does not account for transmission acknowledgment, it may result in wrong device behavior.

We got the assembly driver from our customer, therefore the code is granted to be correct. They use the same code in hundreds of devices and they said they never have such a bad figure in frame reception.

This bug had a chance to live so long for two reasons – first, the customer accepted our firmware with this known problem; second we were quite convinced that this problem should have appeared in their devices as well.

An event broke this impasse – first, we had big troubles in an installation and eventually, it turned out that one device has a slightly defective crystal that caused major frame dropout, wreaking havoc on all the system. This led to some investigation by the hardware guys that told me that crystal precision was very critical in device correctness. The crystal component has a tolerance of 0.5% and our device starts behaving weirdly when the frequency goes 0.4% away from the nominal frequency. This was clearly not acceptable and a solution was urged.

I went to some fruitless investigations before discovering that another firmware, mainly developed by our customer, was much more reliable than ours. After much head-scraping and head-banging against the nearest wall, I went through the aging documentation that the customer gave us to integrate the bus driver.

There I found that it was required to set a global variable, let’s call it FLEX with half of the value of a global constant.

Then I turned to the code and found an old comment of mine that went like this /* FLEX and GIZMO seem not to be used anywhere in the code, they must be private variables */

Of course, removing the comment and properly initializing the variable sent the problem the way of Dodos.

How this could have been prevented? That’s a simple shot – avoid the programmer manually setting obscure and arcane magic values in oddly named global variables. Either provide a complete setup function or let the magic value be computed at compile time.

I’ll write about the other bug, tomorrow. In the meantime, your comments are welcome, as always.

Is this the time for Java?

Once upon a time I was a strenuous ZX Spectrum fan, then time changed, technology evolved and, fighting back my feelings, I switched to Commodore Amiga (yes, it took me a detour over the Amstrad cpc, to get ready for change).Then I was fond of the Amiga, but time changed again, Commodore filed bankruptcy and I changed to anonymous PC clone with Windows 3.1.
Sad story, time changes and you must adapt and change as well, saying goodbye to your comfortable little home.
Now I am faced with a hard decision and it seems that it is time for change again. For the incoming project at the company I work for, I will have to lead the development team (and likely to put in some project management as well, but this is another story) and I have been already told that I don’t have to expect we are going to hire top programmer. On the contrary the company is like to get some instances of Joe Slightly Below Average junior programmer.
The project itself is ambitious (as all projects we tackle), there will be Linux, IP communication, complex system, audio and video communication, proprietary and open field bus communication, zig bee localization and the like… So for sure the team will have to find as much as possible from COTS (Components Off The Shelf) and integrate them.
Now I have to select (apparently I am still free of doing this) the language for the main application. Would have been it my project I would have chosen C++ blindly – no discussion. I am also curious about the new C++11 and this would have been a perfect occasion for getting used to the new tools in the language.
But… I have to deal with junior, possibly below the average programmers (and to make things worse, the development team will be far away from me, but this is another story as well).
From my last weekly report:

I am still undecided about the language to use for this project. There are really only two (maybe three) alternatives.

    • Java. Java is quite accessible. It is likely that every college programmer has been exposed (at various degree) to Java. Java prevents you to do simple mistakes on pointers and memory management at the cost of allowing you to enter terrible pains and sufferance by mindlessly making complex mistakes. Java is mature and stable, performance could be sometimes troublesome. Integrating third part components is easy only if they are written in Java or have a Java wrapper.
    • C++. C++ is unfortunately a less obvious choice, many people have it on their resume, just to pair C, but are quite lost at what is C++ today. C++ helps you strongly in avoiding the terrible pains and sufferances of complex mistakes, by letting yourself tie a knot around your neck with simple errors of pointers and memory management. Although C++ is with us since a long time, the latest standard is still young and many programmers still don’t have a clue about the previous one. Performance is not an issue. Integrating third party components should be easy.
    • Python. Ok, I am not really a script language fan, but I reckon that for integration work, these could be the right tools. Write few “glue” code that just works and ignore the rest. Performance are not what you want to look at, but it is up to third party components to be quick. Though resources can be limited on an embedded system to use a scripting language as it was thought to be used.

Despite the silly wording, Java seems to be the reasonable answer, and for sure is the one preferred by management, but is it the right one?

Strcpy blues

Where do you set the notch for a C programmer declaring “good C/C++” in his/her resume? It’s “developpa” hiring time again at the company I work for, so I am conducting some technical interviews with junior programmers to find the best candidate.
Before this point we had many internal discussion about having the candidate to perform some coding during the interview or not, to the point I got a reputation of being the Bad One – a sort of a Torquemada of the C Language.
In fact I fairly agree with many others that if you are looking for a programmer (i.e. someone who writes code for a living) you should see how he/she actually writes code. Some colleagues felt embarrassed in asking this to an experienced programmer or were afraid of intimidating excessively the candidate.
Then we had a couple of hires whose skill claims weren’t exactly matched by their real skills, therefore we eventually agreed on proposing the candidate to write a few lines of code then using those as a base for more profound questions.
With the best intentions of making the candidate at ease we asked to write the implementation of a short standard library function, say memcpy, strcpy or strcmp. In order to have a common base for evaluation I focused my demand on strcpy.
Such functions should be famous enough not to scare anyone and easy enough to implement so that further discussion (improvement, optimization, pitfalls…) could be tackled to know better the candidate’s attitudes.

Well, I was wrong.

The first problem my candidates encountered was well before the implementation, in fact they had trouble in the definition of a string in C. Most of them opted for a function signature employing an undefined type “String”, someone opted for a pair of square brackets at the right end of the name.
That sheds an unsettling light on the candidate’s knowledge both of the standard library and the C string concept.
Anyway I didn’t say anything and left candidates going on. For all of them it was clear that a loop was needed, but most of them panicked on the string length. As I expected from the string type, no one of them was sure about what a C string is. One said that he expected a string length somewhere. So I pointed them in the right direction stating that a C string is more a convention that a language type, consisting in a sequence of characters terminated by 0.
Most of the candidates were able to use the array notation with an assignment. Notably one candidate failed to write the assignment operator pretending that it was the double equal (he said this twice, he was pretty emotional, but I am afraid he’s really convinced).
No one of them attempted the pointer way with post-increment. I am not a fan of the pointer notation, sometimes it is clearer, sometimes is not. For simple cases like this the optimizer is able to convert the source into the most optimized version. But I expected that a candidate could go the pointer way in order to impress the interviewer. Anyway it wasn’t the case. Worse a candidate had a doubt whether he should dereference or not the pointer with the array notation.
Well, we were looking for a junior C programmer, but I didn’t expect such low score in the C knowledge. Maybe in their experience they hadn’t chance of using the string part of the C standard library, but I would say that pointers were quite stranger to most of them.
Given the technical interview I underwent in my career and the technical interview I read about in Internet forums, I would say that in other countries programmers (even junior) are better prepared, while in Italy it seldom occurs in interviews to require the candidate to write lines of code.
I quite agree that the knowledge of the specific language is not everything for evaluating a candidate, but I think it says a lot about his/her interests, will to learn and to work as a programmer. For these reasons I tried to be flexible, ignoring details of C syntax rather focusing on the problem solving abilities. But the lack of a starting point (the code implementation) seriously hampered my options.
Also I recall that when I left the school anyone in my university friends circle could craft a strcpy in a matter of minutes (I would add blindfolded, but that would be an exaggeration). So have the times changed? Or it is my memories going wild? Should I lower the notch or changing entirely the kind of interview questions?

From the management point of view (as they told me) a junior programmer is expected not to be fluent in a programming language and I expect that a junior has no experience of medium/large code base and software engineering can be just abstract rubbish (at best) to him/her. So I have been instructed to look for learning skill and passion. I found a hard time in formulating questions for evaluating the learning skill of a candidate (well I had some ideas about mazes, food and electrical shocks, but they told me that can’t be done with humans… even if they are junior programmers).

PIC18F Software Project Survival Guide 5

This is the fifth installment of a series of posts regarding software programming on PIC18F cpu family. Previous installments: first, second, third and fourth.
Lost in memories
The first problem you are faced with when designing your application is the PIC memory architecture. And saying that this is a problem is like saying that a multi-head nuke missile is a sort of firecracker. You could say that most of the problems you’ll find working with PIC18s can be classified in two sets: those who stem from memory and those who impact on memory.
First problem is size.
PIC 18F if programmed in C (and you want to use C and not assembly to keep you mental sanity for what is left of your life out of the working hours) tends to be quite memory hungry. Code density is low even if compared to the early CPUs developed by mankind.
To make things worse, Harvard Architecture isn’t going to help you – since pointers are implemented differently down into the assembly instructions level whether they point to Ram (file register) or to flash (Program Memory), you will need to code the same function twice (or more). Consider that the standard library strcpy function has four different implementations because of the four combination you get from its arguments (copy ram to ram, ram to flash, flash to ram and flash to flash).

I read about a C compiler that masks these differences away, (with a pointer wrapper, I presume) but according to the website is far from being production level completeness. Also such an approach penalizes execution time when you know where data are located and that their position is not going to change.
If you really need to handle data both in data memory and program memory you can write a wrapper. I need to sequentially read bytes from any source, so I wrote two wrapper classes ByteReader (and ByteWriter with limited functionalities). The additional benefit is that you can adapt the wrapper to read from an external flash memory as well.

When you let your hardware engineers decide which PIC18 to put on board of the device you are developing, take care about some flash memory subtleties.
All PICs 18F can program their own program memory. But there is a sub-family of PICs (18Fx(456)K2x) whose members have a dedicated small (usually 1k or less) data flash attached to the CPU core by a third memory bus.
You may wonder why the Microchip engineers went the chore of adding a third bus and differentiate on chip memory and addressing. Well, they had, indeed, very good reasons:

  1. program memory can be written one byte at time, but needs to be erased one 4k page at time. 4K on 128k is quite a fraction, but worse, you are forced to juggle with data you want to preserve and considering you don’t have 4k of data memory it is going to be quite a juggle.
  2. When you write the program memory the corresponding bus is stalled, since this is the bus where the program instructions are taken and there is no instruction cache, the CPU is stalled. Typical erase/write times for a flash memory causes the CPU to stall for 5-10ms, possibly more and that can be a showstopper for a real time application.

If you need persistent storage and there are no PICs with the feature list you need, you may resort to an external flash memory connected either by I2C or SPI.

Anyway, regardless of the application, always beg for the device with the largest memory (i.e. 128k in the current production)
Since you are going to sell billions of devices there will be some pressures about picking a small memory footprint device. Resist! You must not lose this battle! You can argue that PIC with different memory sizes are pin-to-pin compatible and there’s no need to add risks on the development when you can downsize the memory in pre-production or at the first technical review.
128k of program memory may seem a lot for an embedded system, but given the low code density and the optimizer naivete is not that much.
On some devices of the 18F family (the ones with a high pin count) you can extend program memory with an external memory.
For our application we managed to fit everything in the base memory and we used the extra pins to connect a LCD screen (the parallel port and memory bus share the same pins). Also we employed an external flash for data only connected either via an SPI or an I2C depending on the specific device we were developing.

Harvard Architecture
Although it could seem a good idea from a theoretical point of view, having two distinct addressing spaces for program and data, isn’t a good idea and having distinct instructions, with distinct addressing modes makes things ugly.
In fact you do want store data in non volatile memory – initialization data, constants, lookup tables – would be only for you have at most 4k of data memory.
The compiler trying to favor performance over conformance doesn’t help much.
Let’s start from the void pointer (void*). In C this kind of pointer is just a (temporary) container for any pointer. You take any typed pointer and you can convert it into void* and then, when you need it, you have to convert it back to the original type.
With MCC18 you have two main problems stemming from the default storage chosen for pointers. In facts pointers are data memory pointers by default. Void pointer makes no exception and it is two bytes long. The problem is that it cannot host a program memory pointer when you use the large memory model. Large memory model is needed only when you need to access more than 64k of program memory and implies that program memory pointers are 3 bytes long.
The other problem is that, by this convention, the pointed type it is not enough to tell apart program and data memory pointer. That is be it a uint8_t* or a uint8_t const* you cannot tell anything about the memory region the uint8_t is stored.
For this discrimination MCC18 provides two qualifiers: rom and ram; the latter being optional, since by default everything is stored in ram.
On one side I prefer the qualifier approach when compared to an “intelligent” approach where the compiler decides silently where to put what based on “const”-ness or other consideration. In fact I use const qualifier quite everywhere and not just for data stored in program memory.
On the other side having to explicitly provide several versions of the same function is a plain waste of space. I would like having a flexible approach where by default I have generic pointers, handled by the compiler through an adapter layer, and specialized pointers (rom/ram) on demand where performance matters.

Hardware and Software stack
First PICs were really simple processors featuring a couple of levels for the call stack. 18F architecture has 31levels of call stack, thus enabling the CPUs to power medium sized architecture.
At a first glance 31 levels may seem a lot… what the heck, even Windows stack trace rarely spans over such a distance.
That would be fine, but let’s take a closer look at what happens when you run low on memory and you enable all the optimizations.
One of such optimization is called Procedural Abstraction and does a fine work in trimming down the size of the code. This transformation examines the intermediate code (or maybe the assembly code directly) and creates subroutines for duplicated code snippets. Operating at a lower abstraction level than the C source, the optimization has far more opportunities of applying the transformation.
Although clever the optimization has a drawback – it takes out of programmer’s hands the call stack control. This is generally true for every optimizing compiler (e.g.: when the compiler moves a static function into the calling place), but to a much lesser extent. MCC18 is capable of factorizing every bunch of assembly instructions in the middle of every C statement building up C lines by composing several subroutines. A nightmare to debug, a hell to understand in the disassembly listing and a sure way to eat those 31 call stack entries really quick.
I already wrote about how to recover from a stack over/underflow and restart debugging without having to re-program the chip. Now let’s see how to avoid the overflow at all.
First you can decided the stack overflow handling by setting the STVREN configuration bit. Basically you can chose among the “ignore” and the “trap” policies. Unfortunately, as we’re going to see, they are both rather ineffective.
Ignoring stack overflow means that when the limit is trespassed the execution continues jumping at the requested address without pushing the return address into the call-stack.
This means that at the first return instruction, rather than returning to the caller, execution jumps back last non-overflowing call.
Ideally, since overflow flag STKFUL (in STKPTR register) is set on overflow you could think of a function stub that checks this flag on entry. The trouble is that once you are in the called function you have no way to recover the return address since it is lost.
Changing ignore to trapping may seem more promising. In fact when this mode is selected, on limit trespassing the execution jumps to address 0x0 and the STKFUL flag is set. This acts somewhat like a reset, but since the micro state is not reset you could think of it as a trap.
Shamefully, yet again you don’t have a way to recover a return address, so you can’t do a save/restore call-stack.
After headscraping far too long on this problem, I decided to simplify the problem making the assumption that if the stack overflow occurs, it occurs only within interrupts. That makes somewhat sense since for sure interrupts are going to impact on the stack. So I added some code in the low level interrupt for checking the stack pointer against a threshold and saving all the hardware stack in a data memory region. This allows to reset the stack pointer and continue to execute the interrupt code. Before leaving the interrupt the opposite operation is performed.
Ideally it would have been nice to have the C run time support to handle the issue by adding a prologue/epilogue to every function so that hardware stack could have been “virtualized”. It is not much of a pessimization since you have already to handle the software stack for parameters and you may optimize out prologue/epilogue for functions that do not perform subroutine calls.
As of today no industrial C compiler implements this.

Next time I’ll write about Extended Mode.

PIC18F Software Project Survival Guide 4

This is the fourth installment of a series of posts regarding software programming on PIC18F cpu family. Previous installments: firstsecond and third.
IDE
The IDE, named MPLAB, is not brilliant, but it could be worse. I saw on the Microchip website that they are going to have a new edition – namely MPLAB X – (now in Beta) that is based on Netbeans. That means that what I am writing here may become quite obsolescent in the next months.
At the time of writing MPLAB X lacks of a number of feature (extended mode just to name one) that prevent its use in a professional environment. If you do not need extended mode, then maybe the worse part is that compilation is overly sluggish.
Anyway I commend the choice of Netbeans, which I prefer over the most widespread Eclipse.
Back to the current MPLAB IDE. You can configure the IDE to get it less uncomfortable. For example you can right click on top left corner of the windows and make them dock-able. When you are satisfied with the layout remember to save it because it is not granted that MPLAB restores it at the next run (it is about two years I am using MPLAB, but I haven’t found yet the reason for which often windows and panels layout is lost from a run to another).
You can redefine keyboard shortcut, but there isn’t a keyboard equivalent for each GUI control. E.g. the pretty useful “compile this file” is missing.
There is a “Goto Locator” contextual option that (possibly) brings you to the definition/declaration of the symbol under the cursor. You have to explicitly enable this option and it works only after a successful compilation. You can also enable auto-completion, but it seldom works. Goto Locator option is hidden nearly as the legendary pin in the haystack. Just open any text file, right click in the edit area and select “Properties”. “Editor Properties” dialog box pops up. Click on “General” tab and select “Enable Tag Locators”. Now that you are there you can switch to the “C File Types” tab and add line numbers, set the tab size and have the tab key to insert spaces . You can also have a look at the “Tooltips” tab where you could enable the auto-complete (which is mostly wrong), and at the tab “Other” where you can enable the ruler marker for the 80th column.
One of the most annoying limitation we experienced is the limitation to 32 characters for the find/find in files commands.

About debugging
Debugging is not a nightmare provided you have the ICD3 hardware debugger AND you keep enabling the software breakpoints (not that MPLAB 8.66 doesn’t let you use software breakpoints because of a bug, so don’t install that version).
(Well actually it is not a nightmare until you get short on Program Memory and you need to enable space saving optimization, but that it is another story).
Hardware breakpoints have weird limitations – first and foremost they are just three, and, at least one, is used for single stepping. Next they “skid”, when the debugger stops on a hardware breakpoint it never halts exactly at the line where the breakpoint is set, but a few lines after. If the line after the breakpoint contain a function call, then the execution may stop in that function call, i.e. somewhere else with respect to where you placed the breakpoint.
In the same way single stepping is much like ice skating, especially if you are trying to step over function calls.
Software breakpoints are immune the plagues that pester hardware cousins. Therefore I cannot see any reason, beyond self punishment, to stay with hardware breakpoints.
The only point of attention with software breakpoints is to routinely check that they are enabled since MPLAB seems to forget this option.
In the same way MPLAB is likely to forget the event breakpoints. These are special conditions that can be programmed to halt the debugger. One of such conditions is the processing of a Sleep instruction and the other is the stack overflow/underflow condition.
When running the debug version of a program, I usually #define assert to halt the debugger, so that failed assertion directly bumps up on the PC screen and I am able to examine the whole machine state. So Sleep break event is very handy (when MPLAB remembers about it, otherwise is… surprising).
Halting on stack overflow is welcome since there is no other way to detect call stack overflow. According to the configuration bit you may chose whether to reset or to continue
If you stumble into a stack overflow you are left at address 0, apparently without any chance to run the application again.
This is because the code has to clear the stack overflow/underflow condition, but this is prevented by the debugger that is halted because of such condition. At the same time this is a very clear sign that the stack messed-up. The confirmation for stack over/underflow may be obtained by a look at the STKPTR register (search for it in the View/Special Register File window). If STKPTR has bit 7 or 6 set then the stack exploded.
To run the program, unless you want to re-program the chip, you simply change the STKPTR register value to 0x01.

If you are looking for ram or data memory bear in mind that it is called “File Registers”. While “Program Memory” is the flash memory. Unfortunately program memory can be displayed only word by word (2 bytes) (and, confusingly enough, in little endian format). Conversely ram can be display only byte by byte.
The other serious limitation is that program memory located variables are not displayed neither in the watch window nor in the local variable window. Your only chance is to note down the address of the variable and look its memory content up in the memory dump window.
The feature I miss most is the stack view with the run-to-return debugging option. You cannot peek neither at the hardware stack (return stack) nor at the software stack.
You will find plenty of options that are not disabled when they should be, for example you can always select the stack analysis report, but it works only when the extended mode is selected (and when you don’t employ function pointers which is quite a constraint). Hardware stack is always displayed as filled by zero, and I haven’t found (yet) the conditions under which it reports something meaningful.
MPLAB is so clumsy when compared to any modern IDE (Eclipse, Netbeans, Visual Studio) that you may find yourself more comfortable to develop and maintain the code with one of those IDE and then to resort to MPLAB only for building the firmware and debugging it.

Next time I’ll write about memory.

PIC18F Software Project Survival Guide 3

This is the third installment on a series of post regarding software programming on PIC18F cpu family. You can find the first here and the second here.
Linker
The linker is expected to group together the code from all the different compilation units produced by the compiler and generate the binary code. Since Pic18f architecture is what it is, this is not a trivial task.
Compiler groups data in data sections. These sections may be assigned to a specific region of the data memory via the linker script.
This file is defined for every project and you’d better to familiarize with its syntax.
In fact some troubles may arise by bad linker script and can be fixed by changing the linker script.
For example the compiler uses a temporary data section (named .tmpdata) – to store intermediate expression results – that is free to float around in the data memory. If the linker script is modified without care, then this section may fall across the banks boundary causing wrong computation (in the best case, memory corruption in the worst).
The default project comes with a default linker script that avoids data object to get across bank boundaries. (Note that linker script banks are not data memory banks, but user defined regions of memory, you may want to make linker script banks coincide with data memory banks for avoiding bank switching problems). So, by default, you are protected from this kind of fault (at the cost of some slack space, unless your code is so lucky to fill perfectly all the pages). But when the project size increases your data object will grow as well. So you may be tempted (I was) to merge all the banks into a big one.
I did, then I found many unexpected troubles because of this (see the .tmpdata and C startup problems for example). So I wrote a small awk script to run over the map file to spot these problems:

#!/usr/bin/awk -f

/<[[iu]data>/ {
        len=strtonum($5)
        if( len > 0 )
        {
            lastByte=strtonum($3)+len-1
            if( and(lastByte, 0xFFFFFF00) != and(strtonum($3),0xFFFFFF00))
            {
                print "Warning file " $1 " spans over multiple pages (data "
                    "size=" len ")"
            }
        }
    }

From the results I selected those modules that have large data object. I found three large data objects of 360, 600 and 501 bytes respectively. So I modified the linker script to have 3 multi-page banks – 2 banks composed by 2 pages and 1 spanning over 3.
In this way the linker is forced to put big objects in those multi-pages banks, but it will keep all the other objects within a single bank as required.
The best option you have is to start with a default linker script and then merge together adjacent banks as soon as you discover a large data object (this will be reported by an obscure linker error message pointing to a completely innocent module).
The Linker is also very uninformative about errors, you are allowed only to know that you ran out of memory. To be more precise you are allowed to know it only after a training, because the error message is very obscure, something on the lines of “section <a section name you are unaware of> cannot fit some other section”.

Assembler
Since Pic 18 are basically C-unfriendly, some assembly may be required. If you need a little bit of assembly then you can write it directly in the C source code (at a price we’ll see later). If you need some more assembly you want to the separate assembler. In this case you can take full advantage of specific assembly directives and/or macros, but then you lose integration with C language. In fact the assembler cannot fully understand C preprocessor directives, making it impossible to use the same header file for inclusion in both C and assembly.
There are two ways to workaround this, both not very satisfying. First you can write shared header files with the common subset of preprocessor directives shared both by assembly and C. Just keep in mind that rules for searching header file differs.
The other way is to write a filter (more or less complex according to the complexity of your header files) for converting C headers into assembly includes.
I went the last way because it seemed simpler, just convert C comments into assembly language comments, then I modified the filter to resolve include files. I gave up when I tried to switch from #if defined(X) to the old #ifdef X supported by assembler.
Eventually I decided to opt for very basic header files included both from assembly and integrated in more convoluted header file structure for C. I resorted to this solution only because it would take too much time to write a complete filter. If you do this just keep in mind that although comments are not compatible, you can use #if 0/#endif to bracket away parts from both the assembly and the C part.
When you mix assembly and C in the same source file you may get surprising results. As I wrote before I had defined an assert macro to execute a Sleep instruction in order to halt the debugger. My assert was something like:

#define ASSERT(X__) do { if( !(X__) ) { asm_ Sleep endasm_ } } while( false )

The effect is that this inserts the assembly fragment with the Sleep instruction everywhere you want to assert something. I was running short on program memory so I tried several configuration on debugging and optimization options and I discovered a significant difference in memory usage whether asserts where implemented with the assembly fragment or via a function call.
Apparently the optimizer has a hard time in doing its work when an assembly code block is inserted in a C function, no matter what the content of the block is (the sleep instruction has no side effects that can disturb the C code execution).
I think the assert is one of the rare case where you want assembly language not for performance reason. Therefore it is a sort of contradiction – you use assembly language fragment to improve speed, but you kill the C language optimization.
If you need assembly for performance, put it in a specific .asm source file.

Next time I’ll write about the IDE and debugging.