Gmail spam filter failures

I had come to trust Gmail’s spam filter so much that I never checked the spam folder. In fact, I had not checked it in several months - maybe even a year. But this evening I saw Cliff Hall’s post about Google spam-filtering itself, and decided I had better take a look. Sure enough, there were 4 or 5 non-spam messages in there. The unsettling part of this is that Gmail automatically deletes all spam after 30 days, so if there were any legitimate messages in there from more than 1 month ago, they are now gone forever.

If you sent me a message at some time in the past and didn’t get a response, now you know why. If you are like me and had learned to trust Gmail’s spam filter, then it’s time to quit trusting it.


Pitfalls of responsive development practices

None of the teams I have worked with to date has fully adhered to a formal Agile methodology, but most of them have taken a highly iterative and collaborative approach to the development process. Although “agile” is a good word to describe such an approach, it seems better to avoid its use here since formal Agile methods are not particularly what I have in view. The word “responsive” seems to be a suitable alternative, in that we’re looking at approaches that are responsive to the customer.

While I firmly believe that a responsive approach is best for most types of software, I have also come to learn that it presents a number of hazards. Unfortunately, even though these hazards have been documented numerous times, I have learned to truly appreciate them the hard way: by seeing them go from potential dangers to real fiascoes. It seems worthwhile to put these lessons in writing; the exercise will help me to remember what I have learned, and perhaps it will also help someone else learn from my mistakes. And, if you respond by sharing your own insights, then I will learn even more.

So, here goes…

Bugs

Change is at the heart of any responsive approach. You build a piece of software, let the customer evaluate it, and then change it in response to the customer’s feedback - and you repeat this process over and over again. But it is axiomatic that whenever you change a line of code, you risk introducing new bugs. So it is almost inevitable that your collaboration with the customer will generate new bugs.

The short development cycle is also at the heart of any responsive approach, because you need to move quickly in order to get something into the customers’ hands so that they can provide the feedback you need to continue development. Any time you are in a hurry, you increase your likelihood of making mistakes, whether you’re developing software or making an omelet. So, again, we are almost guaranteed to introduce bugs with this approach.

The need for speed also limits the time available for testing, which reduces the likelihood that our QA team will catch the newly introduced bugs before the customer sees the software. As a result, we not only increase our chances of introducing bugs, we also increase our chances of delivering the bugs to the customer.

So how do we mitigate this? I think that the best solution has three parts: one is technical, and the other two managerial.

The technical part is to write code that is flexible. Keep your classes small, well encapsulated, and loosely coupled. This will greatly enhance your ability to make changes without creating new bugs. (More on this in a later section.)  This is also difficult to do under time constraints, because it is almost always faster to write brittle code. But we need to remember that the shortcuts we take now will come back to haunt us later.

The first managerial part is to give your developers ownership of application components. When an engineer feels like he or she owns a piece of functionality, then pride of ownership has an opportunity to take hold. The engineer cares about the component, and feels more responsible for it, and therefore takes more care to see that it works properly. Also, the engineers become experts on their own components, and this brings real benefits: They are less likely to deliver something that does not function properly, because they know how the components are supposed to work. And, because they know the code, they are more likely to know how to make changes in it quickly without breaking something.

The other managerial part is to regularly perform code reviews. This provides opportunity to monitor and enforce the technical part of the solution, and also keeps the first managerial part from becoming a problem. When an engineer owns a piece of functionality, and knows that no one else is going to be looking at the code, there is a temptation to be less careful about things like coding standards and naming conventions. The code is likely to reflect more of the style of the individual developer and less of the team’s accepted norms, and can end up being almost impossible for anyone but the original developer to understand. Regular code reviews can forestall this, and ensure that the code is maintainable and extensible.

Scope Creep

Very often, when an end user is given a piece of software to evaluate, the response is something like this: “This is good, but it would be really great if it could also do X!” Such a response is entirely natural. Whenever we get a new gadget, our imaginations get sparked, and we begin to think of really cool things that the gadget almost does but not quite, and we wonder how hard it would be to make it do those really cool things.

When we as developers get this kind of feedback from our customers, our instinctive reaction may be to say, “Sure! We can make it do X!” And that instinct is good: it shows that we really want to build software that will help our customers to the greatest extent possible. However, like all of our instincts, it needs to be ruled by our reason.

In order to properly channel our “satisfy-the-customer” instinct, we need to scrutinize customer feedback and carefully evaluate anything that amounts to a request for additional functionality. If we are 100% certain that we can implement it easily, and that doing so will not affect our schedule, then we should do it without hesitation. Otherwise, we need to determine the impact, communicate the cost to the customers, and let them decide whether they want to pay for it or not. And by “pay for it” I don’t mean simply money, but time as well. Even if we are willing to make the change without charging an additional penny, the customers need to know how much longer they will need to wait to get the finished software in their hands. Then they can decide whether the additional functionality is worth the cost. They know their business better than we do, and they know what the change is worth to them, and we need to respect that. It is essential for us as developers to realize that we are not doing our customers any favor if we keep them waiting for weeks or months beyond deadline while we code additional features that they can live without.

Expectation Management

It is easy to take for granted that the people we talk to on a regular basis understand what we are about, where we are coming from, and where we are going. However, common experience tells us that this is not necessarily the case: On the contrary, if we we want to be certain that others understand a particular fact, then we must explicitly communicate that fact to them. We can’t assume that they picked it up from our general conversation.

This is particularly important to remember in the context of responsive software development. Because our customers are also collaborators, and we tend to develop a comfortable working relationship with them, we can easily take for granted that they understand our perspective on the process. But when we allow ourselves to fall into that pattern, we risk not only the success of the current project, but our long-term relationship with the customer as well.

I have seen solid working relationships between customers and developers destroyed by the developers’ failure to properly communicate the impact of changes. On the other hand, I have seen working relationships that survived very difficult projects largely because the developers communicated the impact of changes frequently and clearly, and in so doing they effectively managed the customers’ expectations.

So, it is vital that all significant facts be clearly and explicitly communicated to the customers. Did you decide to implement some new functionality that they requested on the basis that you could do it easily without impacting the schedule? Make sure they know what you did and the reason you did it. This way they will see that you are really working for them, and at the same time it will help them to understand why some change requests cost extra while others do not. Did you agree to make a change that will impact the schedule? Make sure the customers know what that impact will be, so that they know how much of a delay to expect. They may not always like what you have to say, but they will learn to trust you.

Developing for the Customer

This is one of the more paradoxical dangers of responsive development practices. After all, the whole point of taking such an approach to development is to ensure that software not only meets the customer’s needs but actually streamlines the end users’ work flow and enables them to focus on their business rather than on operating their software. So what could be wrong with developing with the customer in mind?

Here is the danger: when you code for a specific customer, and you have taken the time to know your customer’s business needs and work flows, it is easy to code yourself into a corner by thinking such things as, “I know they always do X in this particular way.” Because you know how the customer does things, you unintentionally design your classes with built-in assumptions about how things will always be done, and you end up writing brittle code.

Keep in mind that you may end up with more customers who are interested in the application you’re building. If that happens, each new customer will need some changes. But even if you are 100% certain that you will only have one customer, remember: in the real world it is not unusual for customers to make changes to their work flows or business rules. Whether you have one customer or 100, some features will need to be omitted, others added, and some will need modification.

So, even if you are building an application that will only ever be used by one customer, guard against the complacent attitude that you know how the application will be used. Design and write the code as though you intend to market it to a broad range of organizations, and you have no idea how they will want to use your application. Bake flexibility, scalability, and extensibility into your architecture. Code to interfaces rather than concrete classes. Design and build loosely-coupled components. Maintain clean separation of concerns. Layout your UI’s in such a way that controls can be rearranged, added, and removed easily.

Parting Thoughts

We all live in the real world, and seldom have the opportunity to work in anything resembling ideal conditions. We deal with absurd deadlines. We work on teams that are ridiculously understaffed. Sometimes we feel like we need to just push out the code as fast as we can, and at the end of the day our only concern is that the software actually works. We’re just doing the best we can.

But when we have time to breathe and reflect, we should make the most of it. Maybe we’ll actually come up with the ideas and the resolve that will prepare us for the next time we’re under the gun.


I’m Back

Very few posts on this blog are personal. This is an exception.

I have neglected this website for well over a year now. During that time a number of people have posted questions about things I had written or left comments to inform me of broken links, but I have not responded. I want you to know that I have neither been careless nor have I been intentionally rude. This post is to let you know what has been going on.

About three years ago, my wife began to exhibit some neurological symptoms. First she had one; a few months later she had another; and so forth. The doctors did not know what was going on, although in some instances they thought they knew what the causes were. They actually performed surgery for one of her symptoms - surgery which proved useless and unnecessary, because it did not address the symptom.

Finally, in the fall of last year, her primary care doctor ordered an MRI of her head, and it revealed that there was a tumor on her brain. The neurosurgeon was pretty certain that it was benign, but he was also convinced that it should be removed. He removed it last December, and my wife recovered from the surgery beautifully. Then, in January of this year, we received the biopsy report: it was malignant.

My wife underwent six-plus weeks of radiation therapy, and this took us into April. Since that time, while  recovering from the effects of the radiation, she has been researching ways to help ensure that the tumor does not come back. As a result of this research, she has been able to help herself greatly through nutrition and exercise.

In October she had her first follow-up MRI, and it was fantastic. Not only is there no sign of any recurrence, there is hardly any sign that there was ever a tumor there in the first place. Even the doctors were impressed. Needless to say we rejoiced and offered many thanks to God for that good news.

Earlier this month, my wife and I celebrated our 20th anniversary. We went to Vermont (where we spent our honeymoon) and stayed two wonderful nights at a lovely Bed & Breakfast.

We are thankful for all the blessings of the last 20 years, including the good that has resulted from the difficulties we faced. and we are looking forward to the next 20 years. We also feel that we have good reason to hope for a healthy 2011.

So, I think I am at a point where I can pay attention to this website again. My first task will be to fix broken links, and then I plan to resume writing. I hope that I will be able to contribute some useful things to the community in the future.

Thanks for “listening.”


Flex, Flash, and HTTP Status Codes

This question comes from a *nix sysadmin (and all-around good guy):

“Jim, do you have a way to make flex handle a richer set of HTTP status codes in a portable way? Without that, I’m hesitant to recommend it for REST deployments.”

This is certainly a reasonable concern. The fact that Adobe still has not addressed this issue after all these years seems truly odd. To get an idea of the amount of trouble this causes, go to the Flex Bug and Issue Tracker at http://bugs.adobe.com/jira/secure/QuickSearch.jspa , type “http status code” in the Quick Search box (upper right corner) and see how many results you get.

Having said that, let us now look at what to do about it. First, note that it is not really a Flex issue per se. Remember, Flex is simply a framework (or tool kit or set of libraries or whatever you like to call it) for building Flash applications, so a Flex application is nothing but a type of Flash application. Therefore, Flex applications necessarily suffer from any weaknesses in the Flash VM. And this particular issue, as I understand it, is a weakness in the Flash runtime.

Knowing where the problem lies is half the battle. Since the problem under consideration lies in the Flash Player, we know that we can’t fix it with Flex code, and so we need to work around it some other way.

[Update: James Ward points out in his comments below that, strictly speaking, this issue has its roots in the way the browsers treat their plugins, which is not always very nice. However, from the Flash/Flex developer's perspective, it makes no practical difference: We still can't fix the problem with ActionScript, and the kind of workarounds discussed here are required whether the problem is in the plugin or in the browsers' plugin API. And even it the problem is in the browsers' plugin API, the end result is a limitation in Flash.]

One approach is to catch your errors with the server-side code and output messages that can be read by any client. Depending on the server technology and the developers working with it, this can be a very easy and effective approach to the problem. With ColdFusion, for example, you can do something as simple as this:

<cftry>
  <!---
    Do whatever processing you
    need to do here, such as a
    <cfquery>
             --->
  <cfoutput>
    <result>
        <error>
            <code>0</code>
        </error>
        <!-- Define the XML
             output here --->
    </result>
  </cfoutput>
  <cfcatch>
    <cfoutput>
        <result>
          <error>
            <code>1</code>
            <message>#cfcatch.Message#</message>
            <type>#cfcatch.Type#</type>
          </error>
      </result>
    </cfoutput>
  </cfcatch>
</cftry>

You can do similar things with Alfresco web scripts, JSP, ASP, PHP, etc. This approach will handle the majority of server-side errors in a way that any UI client could work with. However, it does not cover every scenario, and it may not address the portability aspect of the question that we started with. Besides, you simply do not have this option in every situation (for example, when attaching a Flex UI to an existing application). So we need to go a bit further.

One practice I recommend is writing simple HTML forms to invoke the services. These forms are bare-bones, with no styling and very little attention given to layout; they are purely for functionality. I do this regardless of whether I am writing the server code myself or I am just building the UI to someone else’s code. (Side point: One of the many things I like about Rails is that it builds this scaffolding for you.)

These HTML forms are used in two ways. First, use them to test the server code before running it from the Flash/Flex application. If the application doesn’t work correctly this way, then you know it won’t work from your Flash GUI. Once everything appears to be working from the HTML forms, you are ready to start running the application using the Flash GUI.

Now, if you get an error that Flash is not able to handle, you have an easy way to debug it. Simply enter the same data in your HTML form that was entered in the Flash GUI, submit the form, and see what the error message is. In the vast majority of cases, this will point you to the cause of the error, and hence the solution.

It is true that this requires some extra work on the part of the Flash/Flex developer. And this is really where the answer lies: in the developer.

Every technology has limitations. I will go further, and say that every technology has serious limitations. Even if you go down to something as simple as the hammer, and you think about it a little, you will see that it has serious limitations. This does not prevent us from working with the technology. Rather, if we see a benefit in the technology, we learn to work with it in spite of its limitations.

Having worked with this particular technology since the days of Flash 4, I am very pleased with the progress that Macromedia and Adobe have made, and rarely find reason to complain. However, I do realize that some of the remaining limitations can be frustrating - especially to people who are new to Flash, and find there are things which are very simple in other technologies that are not so simple in Flash. What to do about these things? Again, it comes down to the developer.

The best solution to the HTTP status code issue, and other such issues, is to use Flash/Flex developers who have the experience and/or tenacity to make the technology work in spite of its limitations. In the hands of such people, Flash can work wonders, and the non-Flash team members do not feel particularly burdened by the fact that Flash is being used for the UI.

On the other hand, if you are building large, complex applications with inexperienced Flash developers, then your entire team is going to feel the pain. But that is the same with every technology you use in your development and deployment. If you have  inexperienced DBAs, Java developers, Alfresco developers, or sysadmins, everyone suffers.

Twenty-five years ago, when I bought my first 22-oz Estwing , I found it to be unwieldy and tiring. It was much harder to swing than my old 16-oz hammer, and the guys I worked with got some laughs (and some frustration) watching me make my blunders with it. However, I had seen how effective it could be in the hand of an experienced carpenter. So, even though it caused me a lot of pain - to the point of losing a fingernail - I kept using it, and eventually found it to be indispensable. And, in the end, I was much more of an asset to the crew than I had been previously. It is a great tool.

Flash and Flex are great tools, too.


PureMVC vs. Cairngorm revisited

It has been a year since I wrote my first PureMVC vs. Cairngorm post, and in that time I have had the opportunity to build more applications in both frameworks for some pretty diverse organizations. Studying the official Adobe training materials for Cairngorm has also brought some points that I had not previously considered. Perhaps most importantly, I have benefited from conversations with coders on several different development teams regarding the strengths and weaknesses of each framework. All of this experience and interaction has brought additional insight and some modification of the original opinions. That being the case, it seems to be time to post a revised comparison.

First, let me reiterate that I still like both Cairngorm and PureMVC, and that the two frameworks share a number of strengths. Once your development team has conquered the learning curve, either framework will provide the following benefits:

  • Help build complex applications faster
  • Support team development
  • Support code reuse
  • Support maintainability and extensibility

Either framework will help you build modular, loosely-coupled applications using consistent methodologies that are founded on the well-known MVC pattern. However, there are also some significant differences in the frameworks.

One key difference is that PureMVC is not tied to Flex or even ActionScript. The advantage of this that PureMVC can be used for other languages, and in fact has been ported to eleven different languages to date. The disadvantage of this is that PureMVC does not leverage native Flex features; for example, rather than using Flex binding or the native Flex event delegation model, PureMVC uses its own publish/subscribe notification system.

On the other hand, Cairngorm relies too much on Flex binding, and does not provide another way of notifying the view when a relevant event has occurred in the controller or when the model has been updated. For this reason, many experienced Cairngorm developers dispatch Cairngorm events from Commands and listen for these events in the view components - something which Adobe’s training material explicitly discourages.

Another issue with Cairngorm is that it encourages the proliferation of small classes containing boilerplate code (i.e. events and commands). While developers have found various ways around this, most of these workarounds are - again - expressly discouraged by the Cairngorm documentation and training materials.

It might be argued that the framework can’t be blamed for the fact that people misuse it. On the other hand, it could also be argued that if even experienced, senior-level developers find it too onerous to manage without making some illicit modifications, then perhaps the framework has some room for improvement.

There is one issue that I had with Cairngorm when I wrote the original post which I no longer consider to be significant: the monolithic ModelLocator. It seems that my concerns about creating multiple model locators based on business meaning were unnecessary. I have learned that this can be quite successful, even in team development contexts.

Once again, I have outlined the differences in a matrix using the AdvancedDataGrid component.

[UPDATE: I have corrected/revised the content of the matrix after receiving Cliff Hall's helpful comments below. If you have already viewed the matrix, you may need to clear your browser's cache to see the changes. ]

[UPDATE 2: Please note that there are several points listed in the matrix that are not mentioned either in this post or in the comments below. So it is worth your while to take a look at the matrix!]

This post is not intended to be the last word on the subject; rather, it is just a small contribution to the ongoing discussion. In fact, as I write this post, I am in the process of building my first significant application using Universal Mind’s Cairngorm extensions, which appear to address several of the framework’s weaknesses. I will let you know my impressions when I have completed the application. Meanwhile, if this post helps someone to make an informed decision as to which framework to use (or not) then I will be really tickled.


Putting a Flex Face on Alfresco, Part 3

In the second Flex/Alfresco webinar, we showed the Flex side of how to get content from an Alfresco repository and display it in Flex. In this video, we show you the Alfresco side - how to create and deploy a custom JavaScript web script. You can watch the video and download the Alfresco source code here. As with Part 2, you can watch the video directly on YouTube, but you can only get the code on the Rothbury site.

Enjoy!


Bubble chart with labels

In January I had the opportunity to visit the Queen City to provide some Flex training to a great team of developers under the queen’s crown, and to spend time with some dear friends as well. They (the developers, not the friends) had a requirement to produce bubble charts that had labels inside the bubbles. It occurred to me that other people could probably use this feature, too, so I decided to polish it up a bit and make it available. The result is two classes, LabeledBubbleRenderer and LabeledBubbleSeries.

To implement the functionality, simply create a BubbleChart, and then put a LabeledBubbleSeries tag where you would normally put your BubbleSeries. You should never need to explicitly reference the LabeledBubbleRenderer, because the LabeledBubbleSeries class uses it as the default renderer.

<mx:BubbleChart> <mx:series> <rd:LabeledBubbleSeries /> </mx:series> </mx:BubbleChart>

The primary departure from the standard BubbleSeries class is that the LabeledBubbleSeries has a fourth data field: In addition to xField, yField, and radiusField, there is also a labelField property. As the name suggests, this is the name of the field that stores the string that should be displayed in each bubble’s label.

<mx:BubbleChart> <mx:series> <rd:LabeledBubbleSeries labelField="name" xField="leads" yField="sales" radiusField="commission" /> </mx:series> </mx:BubbleChart>

Secondarily, there are also a couple of additional styles, which control the location of the text within the bubble. These values, of course, are passed on to the LabeledBubbleRenderer instances which draw the circles and display the text.

<mx:BubbleChart> <mx:series> <rd:LabeledBubbleSeries textAlign="center" textVAlign="middle" /> </mx:series> </mx:BubbleChart>

I have posted a very simple demo app that shows how this all fits together.

Demo screen shot

If you look at the code, you will see a rather tight coupling between the two classes. I did not view this as undesirable because I could not think of a reason why you would want to use one without the other. However, the code is free for you to modify as needed. If you do make a significant modification that you think might benefit others, please let me know, as I would like to integrate it.

Finally, a disclaimer: As you have probably already noticed, this code is not independent of Adobe’s Data Visualization components. So if you don’t have those, this won’t do you much good. But if you do have them, then you may find this to be a useful little enhancement. Enjoy!

This being the 17th of March, allow me to close with this wish: Erin Go Bragh.


Putting a Flex Face on Alfresco, Part 2

In my first Flex/Alfresco webinar, I showed how you can log in to Alfresco from a Flex UI using Alfresco’s Flex SDK. That was nice, of course, but it was also sort of a tease: it left folks wanting to see the next step. Well, it took a while, but I finally recorded a new screencast that shows how to get content from an Alfresco repository and display it in Flex. You can watch the video and download the Flex source code here. You can also see the video directly on YouTube, but you can only get the code on the Rothbury site. Please let me know if you have any questions, suggestions, frustrations, cogitations, meditations, salutations, etc.

Stay tuned for Part 3, where I plan to show how I created the custom Alfresco web script.

(UPDATED LINKS ON 1/3/2011)


Cairngorm Diagram Explorer update

I was delivering Cairngorm training to a client’s development team, and the team lead mentioned that the old Cairngorm Diagram Explorer really helped him get a feel for how the various moving parts of Cairngorm work together. After browsing through it, everyone on the team agreed that it was a great tool, but that since it was out-of-date (based on Cairngorm 2.0/Flex 2) it could be confusing to new developers. So I decided to bring it up to Cairngorm 2.2/Flex 3, following the best practices outlined in the Cairngorm training materials.

You can click here to view the Explorer and interact with it, or click here to download the zip that includes the Explorer and the Flex application on which it is based.

Let me know if you run into any issues or if you see room for improvement.


Flex and Alfresco on The Flex Show

I’m on The Flex Show this week, talking a bit about Flex/Alfresco integration. Jeff and John were great, and I think I managed to get some useful information across in spite of the fact that I was recovering from the flu when we recorded it (back in December). There are a couple of items that need improving/correcting, but listen to the show first, then come back here for the clarifications.

OK, here we go. For one thing, I have learned since doing the show that in addition to Alfresco, there are other open-source Enterprise Content Management Systems (ECM’s) out there. (When we did the show I knew about projects like Drupal and Joomla, which are great, but they are Web CMS, not enterprise systems.) In any case, Alfresco is the one embedded in LCDS, so it’s going to be the one that has the most potential to impact us as Flex developers.

Another thing I realized in listening to the recording is that I did not do a very good job of describing the Flex/Alfresco implementation that we did for the large hospitality corporation. The application is really a lot cooler than I made it sound. On the Alfresco side, the repository has models and custom content types that map to the corporation’s specific needs. On the Flex side, VOs map directly to these custom content types. The UI presents these as custom item renderers/item editors, with each renderer/editor being specific to a content type. The screens then group these custom renderers/editors based on business logic, not content type. For example, if the user does a search on “Pets”, the application will respond by displaying a list of content items that relate to pets, regardless of the content type of the individual items. It was a bit of a challenge to display renderers/editors for all of these disparate content types in a consistent and user-friendly way, but I think we succeeded.

The application also had alerts to prompt users to appropriate action. For example, many content types had specified periods of time that they were considered to be valid, after which they became “stale”. This period of time varies between content types, and can be configured in the administrative interface. On the Flex side, items that are soon to become stale are highlighted so as to draw the users’ attention to them.

But perhaps the best thing about the application is how it allows different groups of users to interact with the content as a team. For example, the groups that are responsible for entering the content are dispersed around the globe, whereas the groups responsible for reviewing and approving the content are located in the corporate offices. These reviewers/approvers do not have rights to edit the content; they merely review it, and if it needs modification, they return it to the responsible party with notes as to what needs revising. They can also assign tasks to the entities responsible for entering and updating the content.

I could go on with this, but hopefully you are getting the idea that there are a lot of things that you can do when you have an ECM on the backend and Flex on the client.