These case studies offer the interested client a fairly balanced idea of the type of work we have done since dataFundamentals started up in 2000.
By 2003, dataFundamentals was becoming fairly well-known for its use of code generation and frameworks within the Dallas area java user community.
Large team, significant multiplier
With a challenging implementation to complete, and the efforts of a very large team to gain from any potential multiplier effect, dataFundamentals was engaged to automate as many of the development team's efforts as it could.
Analysis Required!
Ready to generate the required code on his first day on the team, Pete came up on a new problem that wasn't anticipated. No one seemed to know exactly what it was that created each new component! The app was a J2EE app, but the implementation a highly idiosycratic and proprietary implementation using many non-standard approaches. It took a month of digging and research to come up with the code templates and source files for generating against!
Success beyond expectations - Can you do it again?
If it took longer than expected to come up with the structure to start, the implementation of the generator went much faster than team leaders expected, and working app was deployed in days for initial inspection.
As with any code generation effort, it only started with a 75% complete code base, and hand work was expected for the remainder. Further work, with the developers doing the hand work, clarified further automation that was possible, so additional generating routines were written to automate each additional element as it was discovered.
Easy to use Tools
As soon as the generation routines were at a level where they could be run from the command line when required from the latest source data, the advantages of a graphical tool became apparent for a larger team. Pete was able to spend a few days and convert the existing command lines tools to Eclipse tools that would run in the developers' existing IDE.
Expanded to Testing Tools
Ahead of schedule and brimming with the enthusiasm that sometimes results from things working out well, management pointed dataFundamentals to the web application testing problem.
Another team member found an Eclipse tool that would work perfectly but it had some missing pieces. As such it had the right kind of license, dataFundamentals was able to make some quick modifications in the tool that expanded it's functionality in several key areas, closing the loop on some critical automated testing that was not thought practical weeks before.
DataFundamentals was engaged for a brief staff augmentaion contract to Transplace for a development team with an accelerated schedule.
Staff Augmentation
With experience in web applications and the same Struts, Tomcat, and other technologies that were being used for this project, dataFundamentals was able to start with the team and be productive the first week.
Code Generation
Even with the first few pages, it became clear that much of the code that was being written would be verbose, time-consuming, and prone to typing error and syntactical inconsistencies. I propoed an immediate shift to generated code for the database connectivity beans. By writing some customize templates to match the team's existing coding syntax, and pointing dataFundamentals generators at the team's existing database, the hundreds of new bean classes were generated in a couple of weeks and immediately useable by the entire team.
Frameworks
Experience with frameworks was a bit different than most of the team members who were more accustomed to writing custom code for everything. It is sometimes a challenging sell, but many of the team's practices were eventually migrated to a more framework-oriented approach, including a significantly more effective use of an Object Relational Tool for database access.
Data Migration
Months and months of dataFundamentals experience optimizing data migration routines on previous engagements led Transplace to extend the initial contract engagement to the data migration portion. This period lasted through several deployments, as each of many versions of the app in development was tested on very large data sets from the customer's existing legacy data from several disparate systems.
Tool development
Why pay someone to execute time consuming and technically challenging data migrations every single time, when a tool could do the job faster, and quite often better?
In the last month of the contract, I used the same code generation tools and skills to replicate efforts in a repeatable fashion. Transplace was then left wth a toolset that it could use to create future data-migrations at a lower cost and greater consistency.
Semantra needed to turn its five-year development effort into a deliverable product.
It had a brilliantly written code base that did amazing things, but it was a prototype written by its inventor and chief scientist. It was never intended as a final production code base, as it was neither scaleable nor concurrent.
In April, dataFundamentals came on to re-architect the code base and make it production quality. By August, hundreds of classes had been refactored, an SOA design was in place, and web services running against both clients and stress testing.
Staff was trained in the use of the newer technologies such as Maven, Eclipse, various SOA approaches and an IOC container. Builds were standardized, a production schedule was proposed for the completion of the GUI portion, and a first run at a total database refactoring was in place.
An outsider such as myself can come into a mediocre or poorly performing team and offer many enhancements - new techniques, technolgies, practices, approaches to get allow a team to move at a higher rate of delivery.
But how can I help a team that's already world class ?
In 2008 I got the chance to answer that question first hand. I spent almost the entire year as Product Manager for a team which had been performing top notch work for years, and genuinely did not need (me or anyone else) to show them how to do things better.
The answer arrived on my first month on the job - they were too smart, and too effective. So much so, that they did many tasks through diligence alone, that could be farmed out to automated routines to do for them. For competitive reasons, the techniques used might be better left off of a web page, but the net effect of months of work was a team that was freed up from a huge percentage of their daily workload - and a product that was even more consistent and well tested than when we started. That is saying something, because this is a well regarded product.
My year with this software firm proved to be the most challenging year of my career. I may never know just how effective I was or was not, but it proved to me that even the best teams can use a boost from outside perspectives, even if it's just temporary relief to the pressure of a rocketing success path.
A fortune 500 company had to make some decisions like this for one of it's most important development teams. The product line it has invested in was provided by a large well known firm but it was relatively untested. Should we use it ? Where ? Under what conventions ?
I was called in help with this process after doing a couple other mission critical projects successfully in a related departement. We took the full complement of BPEL, Rules, and SOA processes provided by this vendor and put them through their paces enhancing an existing functionality set with known issues that had to be fixed.
The fallout was quite interesting. In some cases we abandoned the new technologies after a couple of months, as we discoverd it too brittle in it's tooling and support for everyday deployments - at least at that version level. Other parts of the suite worked fabulously well and we included them immediately, and got the resulting immediate boost in productivity.
By providing seasoned and benchmarked experience, and being able to execute the same task in multiple technologies, I was able to help management make sound decisions on which pieces to keep, which to forestall, and even a few other technologies to bring in as alternates which they hadn't even thought of.
By keeping their experimenation narrowly focused and in experienced hands, and splitting it between in-house talent with experience in their business, and external experience to bring in outside standards, we were able to focus expenses on where they could do the most good - and help the company move into the future without cutting itself on the bleeding edge.
The fortune 500 company had already invested millions in a 7 year project to integrate a large vendor's payroll system. 5 years in, they discovered it wouldn't do one mission critical piece, and not only that, but the mammoth, capable vendor had already tried to create this piece of functionality - and had given up after failing.
We had already established a track record of pulling rabbits out of hats for this department, so would we look at the idea of creating a piece of software to integrate on the data side and put this functionality into a separate piece of custom software ?
Tensions were high and the company was already in the throws of re-organization after the 2008 market melt down, so we prototyped at just a data and test level for a couple months with little more than myself and occassional management meetings to establish ever evolving requirements, and help from one of the company's programmers to furnish the external data feeds.
After three months of successful 2 week Agile iterations to prove that would work with real data and real results - we had an established and proven domain model. It was decided to bulk up the team, create a UI and bolt it on to the domain model we had created. Four months later the small team reached substantially feature complete status - with a fairly impressive UI and back end feature set, after dozens of serious requirements challenges.
Three more months in stabilization, securing the app and rolling it out the large corporate user base - and we accomplished for a relatively small budget what the large vendor had not been able to offer.
It was also much prettier and possibly more customized than anything the large vendor's piece was able to offer, but that is just icing on the cake. Even without that, it would have been a huge win for this customer.
[This page was written in August 2009 as future history. The last of the events depicted above have yet to occur.]
In the year 2000, dataFundamentals was engaged by a small engineering company with some unnecessarily high clerical costs. Timekeeping for the dozens of engineers it had on site around the country had created a mini-bureaucracy within its home office, and the owners wanted a more efficient way to get the job done.
DataFundamentals was engaged to design and create a secured web application to centralize all of the records in a database that they could query and report on from within the home office.
Framework utilized to prevent duplication of effort
Expresso, then at version 4.2 was used as the secure web application platform, as it was a strong and well documented J2EE platform that was already in use in thousands of installations worldwide. As Pete Carapetyan had been a core contributor to this and previous releases, it made for an easy choice.
Code generated from previous app
WebAppWriter, also written and provided by dataFundamental's Pete Carapetyan, provided a working application within a few days. Because the company already had a database it had been using (albeit one written in Microsoft Access), it was easy to take this database, make some quick adjustments and data normalizations, and then point the code generator at it to create a new web application written in the Expresso framework.
App customized to business rules
Immediate deployments made it possible for management to come up with requirements that it hadn't been able to predict in advance. They didn't want to have to train field engineers how to enter data correctly, so several enhancements were made to automate data entry and restrict fields to acceptable entries based on project status.
In house admin trained to maintain app
From dataFundamentals first day on site, in-house engineers were involved in the process and brought along with the processes and logic. As the app was deployed, the engineer was trained in its build and deployment, so that he could customize the program at will.
An Emergency Deadline
Engaged in March of 2004 to assist a very small company with a past due deadline, dataFundamentals had to rely on Pete's 20 years in product delivery to even make the resources fit the time available. The project was already seven months past its one-month schedule, and requirements had been so loosely defined as to make failure almost assured if one more misstep were allowed. It took laser sharp focus and strong powers of persuasion to get everyone rowing in the same direction, but we made the deliverable.
Putting a Process in Place
When engaged to carry the project to the next milestone as project Architect, we were lacking almost everything normally found in a mature development process. One at a time, dataFundamentals brought in infrastructure and systems for everything from proper source control procedures to build and test processes, requirements and documentation, and phased development cycles.
From previous projects, Pete found that agile development practices, most notably test-first and small iterative development cycles, were the most effective way to work, so these were the core approaches used to build the processes.
Building a Team
Change is tough on any team, and this team was facing many challenges that more experienced teams didn't have to wrestle with. So in addition to lots of training on the basics, Pete had to slowly and purposefully integrate lots of cooperative processes and trust that is more normal on the many open source projects that Pete had worked with over the years.
The results have been remarkable. A team developed that is cohesive, efficient, and works well within the boundaries set by our roles and task lists. This is an accomplishment that is as satisfying as the quality of the software itself.
Database Design and Normalization
Several previous iterations of this software had become very hard to maintain due to database design problems and it was a hard sell to convince all involved that the investment of time necessary to normalize the database first was worth the time.
Interestingly enough, though this ended up being a significant investment of time on the part of the company, very little time had to be spent by dataFundamentals once the initial period of persuasion had passed. Most of the work was done by existing staff, with Pete providing initial training, objectives, and occasional input and feedback. The existing team knew instinctively what good design looked like, but they had never been forced to take the time to implement it.
Great Software Takes a Good Foundation
DataFundamentals' years' long involvement with the Keel Framework was tapped to bring this in as the foundation for all software. Maintainability and feature sets had always been this company's largest challenges, and this was something that Keel provided in spades.
Doing the math on the cost of converting and maintaining previously built applications, a decision was made to re-architect the suite of applications from scratch rather than attempting to convert a large amount of questionable code. The usable portions of code were identified and then converted into workable portions and wrapped as single components before being brought back into the new development. This allowed for the best combination of new and old development without losing the investment in the old code.
Several months were spent building and tuning each of the relevant components into compliant, cohesive Keel components. Existing staff began to learn the powerful effect of having very consistent, simplified components designed to do one thing very well, and then testing that one thing against a battery of automated tests.
Because of Keel's maturity as a framework, much of what the customer needed didn't have to be done, it was already there. From clustered boxes working together as a set, to failover and role-based security, these were areas we didn't have to focus on.
Automated Code Generation
After a year of developing the undergirding routines, the team was ready to extend the GUIs to the many areas that it had not yet exercised the database. This meant input and retrieval forms for each of the many new tables, hundreds of files all very prone to syntax errors and consistency problems, had to be created.
Pete spent a few weeks automating this process, and ten thousand lines of code appeared with one click of the mouse, providing boilerplate, customizable forms and table views for everything in the database.
Swing Development
Early in the process it became apparent that we needed the deployment flexibility of our browser based web application but that, even with AJAX the browser, wasn't going to cut it as a client.
A move was made to Swing as client. Server side code stayed virtually unchanged, but dataFundamentals used it's experience with Java Swing and Visual Basic to build the kind of functionality at the base level that would extend in a component framework across all applicable views. This allowed the client to use things such as rendering engines as widgets, rather than customizing every line of code in a hard to maintain spaghetti bowl, as with previous versions of the software.
Advanced Component Development
One of the notable issues for this customer are the contrast between the sophistication of its software needs, and the size of its market. It is not uncommon to have to write a component which will only be used by one or two customers, yet it needs to plug in to the system as if it were any other component.
DataFundamentals brought with it high standards for "Black Box" design and insisted that the interface to each component allow for switching out components, much as one might switch out stereo-hi-fi components. It took the organization a while to adjust, but the results have been tremendously successful, often surprising us by helping in situations we had not even anticipated.
One example is the many uses of libraries from the public domain, place in proprietary "Black Box" wrappers within the company's system. By providing this level of abstraction at such a concrete level, the company has discovered that it can safely take chances with libraries that it did not write because they are carefully segretated and maintained as units that can be tested against everything else with automated testing.
When Expresso 4.02 came out in 2000, it was a powerful framework for creating secured web applications for J2EE application servers. But how to write the applications?
A web application to generate web applications!
After spending a couple years a s core contributor to the Expresso framework, we knew we had a great product. But what better way to prove it than provide a web application that writes web applications?
Written by Pete Carapetyan in 2000, WebAppWriter was put on the web and worked as advertised, creating thousands of lines of consistent, compliant, Expresso 4.02 code from an easy to input database schema format. Users input their tables and columns, and received a large zip file by email in a few minutes after hitting the submit button.
Used by thousands.
It worked well enough to gather a few thousand users without marketing, and many users came back time and again to generate applications for themselves or their clients.
Well documented, thorough.
With dozens of pages of very clear documentation furnished with each download, and the clear architecture of Expresso 4.02, users found it possible to implement their code with almost no requests for support over the 5 years it was in operation.
Retired in 2005.
The box running WebAppWriter for 5 years straight was retired at the end of the year. With Expresso moving beyond 4.02 and dataFundamentals no longer working with the Expresso team, we bid it a fond goodbye.