The feedback from the article is about what I expected. People are saying 92 resources aren't that much, such as this tweet:
@AnnieLowrey experts in that story are wrong. 92 requests isn't that many. Reuters' own site makes 329 cc @sxbegle— Andrei Scheinkman (@ascheink) October 5, 2013
The second issue, and I think the more important one in terms of bringing the site down, is how the backend system was accessed. I looked at requests using the Console in Firefox to determine why security questions weren't loading. I noticed an Internal Server Error (error code: 500) being returned for the URL: https://www.healthcare.gov/ee-rest/ffe/en_US/MyAccountEIDMUnsecuredIntegration/fetchAllSecurityQuestions/ffm. It seemed 50/50 in terms of returning an actual response. It returns JSON that contains the security questions. Why this is returned in a separate request and pieced together on the client side I don't know. When the error occurs during a request, the full stacktrace is returned to the client within the JSON. This is another ignored best practice as it gives information on how the code works to unauthorized users and can potentially cause issues.
javax.xml.ws.WebServiceException: Could not send Message.
at $Proxy3761.viewChallengeQuestions(Unknown Source)
at sun.reflect.GeneratedMethodAccessor708.invoke(Unknown Source)
Caused by: java.net.SocketException: SocketException invoking http://10.153.199.69:8663/ChallengeQuestionsService*: Connection reset
at sun.reflect.GeneratedConstructorAccessor4142.newInstance(Unknown Source)
... 49 more
Caused by: java.net.SocketException: Connection reset
... 59 more
It shows a client side resource containing JSON being populated by a web service located on the intranet (Note the
Caused by: java.net.SocketException: SocketException invoking http://10.153.199.69:8663/ChallengeQuestionsService) showing an IP address that is reserved for internal network usage and that the server at that address (or servers behind a load balancer) were overwhelmed causing the frontend to stop working.
My expectation is that these security questions aren't going to change much, if at all, so there is no reason for this web service to be called frequently. Seeing as this failed regularly and I wasn't the only one using the site, obviously these results weren't being cached. There was one resource loaded in a similar way at https://www.healthcare.gov/ee-rest/ffe/en_US/MyAccountEIDMUnsecuredIntegration/fetchEIDMValidations/ffm which just loads regular expressions. Again, why this needs to be loaded in this way is unclear. Since the create account form is the only part people seemed to be able to get through and that these were the only requests that might have been called the backend system (from what I can see as a frontend user). It seems crazy that this could bring the entire system down, but it might just be the case. With basic caching this would have been avoidable. Without caching, this application should have never been let out the door.
Finally, the overall architecture is confusing. When I think of backend systems, I think of batch processes or specific functions being run behind the scenes that aren't integral or synchronous with the frontend system. Submitting a form which posts to a server which then in turn posts to another server, waits for a response from that server, and returns a response from the previous response is an unnecessarily abstracted way for a high-performance application to run. It might not run that way, but that's how it looks based on the stacktrace I received and from the existing discussion of segregated frontend and backend systems.
All in all, the web site's failed launch is unfortunate. As a Democrat I am disappointed in anything that makes it unnecessarily difficult for people to get access to health care. The errors that I've seen, and the many others that I haven't, disappoint me as a developer interested in well-designed code. Of the millions of users to the site, there are undoubtedly some who took time out of their day to go to the library or somewhere with internet access which they can't afford on their own, only to be unable to get through the site. I also have concern for those, as Alex Howard tweeted, that with a proper launch would "have hours of their days back & wouldn't have lost trust in government to provide services online."
Conservative media is having a field day with the "Obamacare glitches" that have nothing to do with Obamacare or the government. These are private contractors that were unable to deliver. The main blame goes to CGI which is implementing many state-level exchanges and has had issues with being behind schedule and over budget. Sharon Begley, who wrote the Reuters article, told me CGI was unavailable for comment. The article's scope seemed to be scaled back from our initial discussion, so it's unfortunate more analysis into the players responsible for the site's failed launch isn't immediately available. I, and others, can only do so much analysis of the code from a web browser when most of the site's functionality is unavailable. So people can critique my analysis, although I'd disagree with it being "wrong."
As I tweeted last week:
If GOP shuts down government on day exchanges launch, they will receive full blame for any issues in Obamacare implementation. #strategery— Matthew Hancock (@matthewhancock) September 28, 2013
In the absence of the shutdown, all of the focus would be on the site's errors so the GOP did Obamacare a huge favor. Despite the shutdown, at least some programmers are deemed critical enough to work on debugging the site. I hope when it re-launches after the weekend that the site is able to provide what the people of America deserve: easy access to affordable health care.
Ultimately, this should never have been contracted out. Those working in the IT department at DHHS would have skin in the game and would be better able to deliver. The downward pressure on government employees makes the government less effective, but that might be the point.
Lesson of http://t.co/6vM9QhIBnn is never contract out work when it's the most important web site your department has ever launched.— Matthew Hancock (@matthewhancock) October 5, 2013