Best Source for Best Practices

In the enterprise software space, we have historically taken our cues from Enterprise Software Vendors. Vendors such as Microsoft, Sun, Oracle, IBM and BEA have hugely shaped the way we work.

This gave us things like:

• SOAP
• The J2EE App Server
• ASP.NET
• JNDI
• Enterprise Service Buss
• The word SOA Governance
• EJB
• JavaServer Faces

When I was new in the industry my techno-skepticism was not so well honed. I was guilty of spreading and promoting some bad ideas. I accepted everything the Enterprise Software Vendors said without question.

But about 10 years ago I started paying attention to what Google, Amazon, Facebook, Twitter and Netflix were doing. Turns out, many of them use Java. But usually in a different way. Often a simpler way.

If you want to learn enterprise best practices, I suggest you head on over to Google or Amazon and check out their cloud offerings. Study their APIs. Look at their HTTP services. Compare how they are doing things to the above list of Enterprise Mumbo Jumbo.

For example, look at what Java EE considers an appropriate stack for server-side apps. And compare that with Google's offering (Google App Engine Services). You will be surprised.

I saw a John Oliver special the other night that pointed out how much drug companies spend selling to doctors ($28 billion per year).

I suspect there has been something similar going on in enterprise software. If you go up the hill from where I live, to the really expensive houses, you will find a lot of folks with the title: Enterprise Software Sales. It seems to me that the corporate culture at these firms is to pay the Enterprise Software Salesperson the highest salary. The engineers live in the condos at the bottom of the hill, next to the garbage dump.

Dave Recommendation: we should start taking our cues (and best practices) from the great web companies instead of commercial enterprise software vendors.

The Dave Super Duper AJAX Exception Handler

This is a write up I did for a client when they were having a hard time troubleshooting an ajax application. I think this info would be generally useful so I thought i would throw it up on my blog.

Problem

We are observing intermittent errors through the jmeter tests and also through interactive tests. The errors are in a format such that:

1. The AJAX client cannot respond appropriately to the error.
2. They provide no useful information (to the client programmer or the user).
3. Load test failures are hard to interpret
4. Its difficult or impossible for the ajax client to know for certain if there was an error.

Cause

Server-side error handlers that log a message but do not inform the caller that anything went wrong.

Solution

All JSON calls should return one of two things:

1. a valid response or
2. a deliberate json error message, one that that the ajax client can make use of (i.e. try again, inform the user, send an email to cpd, etc). Also, it should aid in debugging.

Unintended AJAX Responses
As it currently stands (in the app we are troubleshooting) the JSON calls sometime return a third thing: an indeterminate, unintended response. A response that was not deliberate but rather accidental. This type of response cannot be handled appropriately by the AJAX client.

The unintended ajax responses that we have observed (using jmeter, interactively with firebug and looking at the code) include:

1. ajax calls returning html
2. ajax calls returning status code 200 and no response body (html, json or otherwise)
3. unintended json: an error occurs server-side, and is only half handled: i.e. its logged to the server logs, but then a half-completed json response is returned to the client.

All three of these make it hard for the the ajax client to respond to errors. They make it hard for the ajax client to even know if there was an error. Its not enough to just log errors, they must be handled as well.

The Ultimate Awesome AJAX error handler

This secret tip will save you 1000s of hours of requesting and dealing with log files, especially when used in conjunction Firebug-like tools or with jmeter. This is one of the most useful tips ever, especially in this client's environment.

Also, its very difficult to associate a message in the error log with the bug detected by a UAT tester. It's even harder to connect an error in a load test with a line in the error log. Even if everyone has perfectly synchronized clocks.

So here is a generic version of the error handler (for use with Spring):

} catch (Exception e) {

model.addObject("errorCode", e.getClass().getSimpleName());

model.addObject("errorMessage", e.getMessage());        model.addObject("stackTrace", Util.serializeStackTrace(e));

}


If you add the error handling changes that I suggested in the last email, UAT testers and keynote test scripts will instantly know "it was a cmllink" problem or "it was a dod problem", without having to:

• request log files
• wait for log files
• spend hours sifting through megabytes of irrelevant information in the log files
• spend hours trying to correlate one of the million log messages to what happened in the browser.

The key point is that the actual error (the relevant error, the one that happened 8 layers down in the app) must bubble to the top level error handler and get send back to the client in a format the client can process.

Multiple Concurrent Versions

Multiple Concurrent Versions is the term I use to describe a practice for deploying multiple versions of a web app at the same time. Technically, the concept applies to any kind of deployment bundle, but Java web apps make a good example.

Here are the basic precepts of the practice:

1. We assume that the thing being deployed, the deployment bundle, is represented as files and folders.
2. Physically moving the deployment bundle to the deployment target (the runtime environment) is a distinct step from activating a particular version.
3. The deployment target will, at any given time, contain multiple versions of the same deployment bundle: 
• old versions that used to be in production 
• old versions that never made it to production 
• the current production version 
• future versions that are very close to production ready
4. Multiple versions are concurrently deployed and accessible at the same time via version specific urls:
   
   http://v8.mycompany.com/myapp
   http://v9.mycompany.com/myapp
   
   The current or default version can be accessed via a version independent url:
   
   http://mycompany.com/myapp
5. Activation is the process of marking a particular version as the default version. 
• Changing the default version from an older to a newer version is called a rollout. 
• Changing the default version from a newer to an older version is called a rollback.
6. Each deployment bundle version should have a cryptographic identity (i.e. sha).
7. Concurrent versions should be isolated. That is, a bug in version 8 shouldn't be able to kill version 7. 

Jar Hell

I recently was called in to help troubleshoot an application. It was a java web app: a war. When I looked inside the WEB-INF/lib directory, here is what I saw:

1. activation-1.1.jar
2. asm-3.1.jar
3. axiom-api-1.2.12.jar
4. axiom-dom-1.2.12.jar
5. axiom-impl-1.2.12.jar
6. axis-1.1.jar
7. axis2-adb-1.6.1.jar
8. axis2-codegen-1.6.1.jar
9. axis2-kernel-1.6.1.jar
10. axis2-mtompolicy-1.6.1.jar
11. axis2-transport-http-1.6.1.jar
12. axis2-transport-local-1.6.1.jar
13. axis2-xmlbeans-1.6.1.jar
14. commons-beanutils-1.7.0.jar
15. commons-beanutils-core-1.7.0.jar
16. commons-codec-1.3.jar
17. commons-collections-3.2.jar
18. commons-configuration-1.2.jar
19. commons-digester-1.7.jar
20. commons-discovery-0.2.jar
21. commons-fileupload-1.2.jar
22. commons-httpclient-3.0.1.jar
23. commons-lang-2.1.jar
24. commons-logging-1.0.4.jar
25. commons-logging-api-1.0.4.jar
26. dac-client.jar
27. dealer-locator-pojos.jar
28. DealerLocatorWebServices-client-1.1.4.jar
29. dom4j-1.4.jar
30. ehcache-core-2.3.0.jar
31. ehcache-web-2.0.3.jar
32. ejb-api-3.0.jar
33. httpcore-4.0.jar
34. isorelax-20020414.jar
35. jackson-core-asl-1.7.1.jar
36. jackson-jaxrs-1.7.1.jar
37. jackson-mapper-asl-1.7.1.jar
38. jackson-xc-1.7.1.jar
39. javax.ejb_3.0.1.jar
40. javax.jms.jar
41. jaxb-api-2.2.2.jar
42. jaxb-impl-2.2.3-1.jar
43. jaxen-1.1.1.jar
44. jaxrpc.jar
45. jersey-core-1.6.jar
46. jersey-json-1.6.jar
47. jersey-server-1.6.jar
48. jettison-1.1.jar
49. log4j-1.2.8.jar
50. mex-1.6.1-impl.jar
51. msv-20020414.jar
52. neethi-3.0.1.jar
53. ojdbc6.jar
54. opensaml-2.2.3.jar
55. pagination-lib.jar
56. quartz-1.6.6.jar
57. quartz-1.7.2.jar
58. rampart-core-1.6.1.jar
59. rampart-policy-1.6.1.jar
60. rampart-trust-1.6.1.jar
61. relaxngDatatype-20020414.jar
62. saxpath-1.0-FCS.jar
63. service-pojos.jar
64. slf4j-api-1.5.11.jar
65. slf4j-jdk14-1.6.1.jar
66. stax-api-1.0-2.jar
67. stax-api-1.0.1.jar
68. tbgv2util.jar
69. TMSFramework-2.2.46.jar
70. TMSIntegrationFramework-0.0.216.jar
71. TMSMessagingFramework-0.0.11.jar
72. TMSUserMgmtFramework-1.0.108.154.jar
73. warranty-maintainence-pojos.jar
74. wiztools-commons-lib-0.2.0.jar
75. woden-api-1.0M9.jar
76. woden-impl-dom-1.0M9.jar
77. wsdl4j-1.6.2.jar
78. wss4j-1.5.12.jar
79. xalan-2.7.0.jar
80. xercesImpl-2.2.1.jar
81. xml-apis-1.0.b2.jar
82. xmlbeans-2.2.0.jar
83. XmlSchema-1.4.7.jar
84. xmlsec-1.4.5.jar
85. xmltooling-1.2.0.jar

I, myself, have never created an application that uses 85 jars.

Making things worse, the app was running on a heavy weight app server (Web Logic) that added many more classes to the classpath, classes that do not show up in the above list. These included the entire stack of J2EE classes. Plus other classes that WebLogic thinks you might need.

Problem: Any app with this many classes in the classpath will surely have some accidental duplicate classes, possibly even multiple versions of the same class. This can be the source of very hard to track bugs.

This was, in fact, the source of the bugs in this particular application.

Moving forward, I would recommend the following:

1. Add libraries and jars to your app very judiciously.
2. Be careful of build tools that might recursively suck in unneeded library dependencies.
3. Beware of any 3rd party libraries that require a whole bunch of other 3rd party libraries to work.
4. When a 3rd party library comes with 10 other 3rd party libraries that it depends on, those other libraries might be optional libraries and not needed for your app.
5. Avoid app servers that litter the classpath with unneeded libraries. 
6. Be especially cautious of putting classes in your app servers global (cross-web-app) lib folder.
7. This entire problem starts to goes away as we move toward a more micro-service style architecture.

The Semi-Swallowed Exception

I am often brought in to help a team track down super-hard-to-find bugs. One situation that results in the super-hard-to-find bug is what I call the semi-swallowed-exception.

Most java developers know that it's a very bad idea (99% of the time) to completely swallow an exception like this:

try{
    ...
    ...
}catch(Exception e){}

That is, you very rarely want an empty catch block.

But almost as troublesome is this:

try{
    ...
    ...
}catch(Exception e){
    log.error(e);
}

The problem is, that the exception probably left the application in an invalid state. For example, some variable wasn't set correctly. Then, sometime later, you get a null pointer exception or worse, some strange hard to explain bug.

Based on the projects I have been involved with, I would say that 90% of time when you find yourself wanting to do this:

try{
    ...
    ...
}catch(Exception e){
    log.error(e);
}
...

Instead, you probably should do this:


try{
    ...
    ...
}catch(Exception e){
    log.error(e);
    throw new IllegalStateException("bla bla",e);
}

Also, if you are just catching an exception from a lower layer, there is not much value in wrapping it in a do nothing exception class. For example, I came across this the other day:


try{
    ...
    ...
}catch(Exception ex{
    log.error(ex);
    throw new MyCompanyNameException("bla bla");
}


Two things to note about this snippet:

1. The MyCompanyNameException class adds no value in this case. This seems to be a popular thing to do. But I can see no point to it.
    
2. This exception handler actually swallows information.The MyCompanyNameException includes a string message ("bla bla"). But it swallows the cause exception. So I would replace the above exception handler with this:

try{
    ...
    ...
}catch(Exception ex){
    log.error(ex);
    throw new RuntimeException("bla bla",ex);
}

Concurrent Maps for lightweight caching

Here is the Dave best practice when using a map for caching purposes in a java server environment:

Don't naively use Java maps as a global cache in a Java Server environment. 

Instead:

If it's a create-once, read-many type cache.

Use a Guava ImmutableMap and ImmutableMapBuilder:

public class DaveMapDemo1 {

   //note: Person is immutable
   private final ImmutableMap<String,Person> map;

   public DaveMapDemo1() {

       ImmutableMap.Builder<String,Person> builder =
               new ImmutableMap.Builder<String,Person>();
       builder.put("123",new Person("Dave","Ford"));
       builder.put("222",new Person("Joe","Blow"));
       map = builder.build();

   }

   public Person getPerson(String id){
       return map.get(id);
   }

}

If it's a get-or-create type cache.
If the cache is populated lazily, use Guava's LoadingCache and CacheBuilder classes (these replace the Guava ConcurrentMap implementations):

public class DaveMapDemo2 {

   //note: Person is immutable
   final LoadingCache<String, Person> map;

   public DaveMapDemo2() {

       map = CacheBuilder.newBuilder()
               .maximumSize(1000)
               .expireAfterWrite(10, TimeUnit.MINUTES)
               .build(
                       new CacheLoader<String, Person>() {
                           public Person load(String id) {
                               return lookupPersonFromDatabase(id);
                           }
                       });

   }

   public Person getPerson(String id) throws ExecutionException {
       return map.get(id);
   }

   private Person lookupPersonFromDatabase(String id) {
       // yada yada yada
   }
}

Summary
Notice that both examples:

1. Have no mutable state
2. Do not use the keyword synchronized

Dart's new template engine

Dart's new template engine seems promising.

I really like the fact that the new Dart template engine generates a *tree* rather than a string. Or more specifically, an html.Element. Other than XSLT, this is the only templating system i know of that does this.

This comes very close to something i have been wanting for a long time, what i call: "typed templates" or "extensible literal types"

http://dave-ford.blogspot.com/2012/02/good-polyglot-bad-polyglot.html#TypedTemplates

Whether we call them XML literals or DOM (typed) templates, the idea is the same.

Suggestions for Improvement

There are four things i think that would improve Dart's template engine:

1. Eliminate the extra compile step. i.e. with something like #source or #import. Maybe:

#template("myTemplates.tmpl");

2. Leave the door open for other template types. i.e. provide some way to specify the returned type and/or the template engine to use. So, instead of this:

template Hello(String to) {

  <div>${to}</div>

}

we would have this:

html.Element htmlTemplate Hello(String to) {

  <div>${to}</div>

}

where html.Node is the return type and html is the template engine.

Or if the template engine type implies the return type you can skip that part:

htmlTemplate Hello(String to) {

  <div>${to}</div>

}

Other examples:

xmlTemplate Hello(String to) {  }

sqlTemplate oldPeople(int minAge) {  }

jsonTemplate Hello(List<Person> people) {}

stringTemplate emailTemplate(Person p) {}

daveDslTemplate(Person p){}

3. An extension mechanism.

For common DSL's (initially just html) the template engine would be built-in. But one could provide a custom template engine like this:

#templateEngine("daveDsl","http://www.smart-soft.com/daveDslTemplateEngine.dart")

4. In-line templates. This one is a stretch, but it would be nice if templates could be defined in a .dart file alongside other dart code:

//normal dart function

Person getPerson(){

        //yada yada

}

//dart template function

htmlTemplate Hello(String to) {

  <div>${to}</div>

}

I know the whole language in a language (island grammars) complicates things from the dart parser/vm perspective. So this is a nice-to-have more than a must-have.