It happens more than you’d expect or hope for: the effort spent on developing a product feature takes much longer than you’d expect. A conversation with the developers leads to a reasonable explanation: “Before we can start with ABC, we need to DEF because of GHI …”. While this explanation is completely valid within the working paradime of the team, it is killing performance.
Simplify essential complexity; diminish accidental complexity
The only way to get out of this catch 22 is to simplify the overall complexity of the way the team works. Easier said than done, so below you find some practical simplifications I find worth trying. Not all of them are possible overnight, the ordering may be a starting point for your simplification journey.
- Squeeze the grey zone. Tasks in the grey zones are “under investigation”, “ready to test”, “maybe worth investigating”, … No developer can focus on it to get it over and done, but these task will pop-up so now and then to claim attention. As a team, get rid of these tasks as soon as you can. Do not let them linger around and eat your time!
- Simplify requirements. This one takes some guts from the product owner, but too many requirements address imaginary problems. Continuously and rigorously ask the question: how does this requirement help my users achieve their goals? Is this the most easy requirement to do so? Are there simpler requirements still giving your users the value they are looking for?
- Automate! Automate! Automate! One of the most complex things in a project or product development is to assess the quality of the current state of development. The only way to give an easy to interpret and consistent answer is to have a very simple metric that is simple to generate. Embedding automated testing in your continuous integration cycle is one of the key elements to get to your goal. New feature development and bug fixing will go much faster once you have the no regression guarantee built in into your development cycle!
- Simplify design. Once you got rid of the grey zones, had your requirements pass the simplification gate and implemented a highly automated development cycle, you can start working on the simplification of your design. Are the implemented processes the simplest possible way to meet the requirements? What parts of your architecture can be refactored to become “the lean and mean” foundation of your application rather than an inextricable tangle of frameworks and third party application? What code can be rewritten so that it is readable for the newcomer in the team? Whenever possible, foresee design simplification effort when adding new features. It is an investment that will pay back!
Step 1 & 3 are mainly focussing on the accidental complexity, 2 and 4 address the essential complexity. Now just do not expect magic :) Steps 1 to 3 will definitely help your development process to become more predictable and free up time from your team. But it is only with simplification of your software that the ultimate goal of developing better software quicker, will become reality…
One of the more fun (sic) aspects of development is package management. Assuring that dependencies are consistently ok, being sure that everybody works with the same versions of libraries and having these dependencies available at all times, … it is an interesting challenge :)
Python has a very handy tool pip, to be used in combination with virtualenv and a central repository full of great packages: Pypi. But sometimes the package you need is not there, or the site is not up… That is when a loca Pypi server comes in handy. I found quite some refrences to Chishop (here and here), but I have the impression is is not really under active maintenance.
So I turned to pypiserver and it worked out rather well.
Installing pypiserver
The following commands did the trick to get the server running in a dedicated virtual environment.
https://gist.github.com/4708189
Adding packages
To add packages, just follow the guide that explains it step by step. The only change I made was replacing the upload command by copying the package tarball to the packages directory
Running the Pypi server as a managed process
Supervisor allows you to run one ore more managed process instances of (several) applications, such as this Pypi server. Installing it goes via pip.
https://gist.github.com/4709415
Adding the program is done by adding a couple of lines to the config file. To take into account the libraries you added to your virtual environment, the bin directory is set as the path (equivalent to activating the virtual environment).
https://gist.github.com/4709358
To have effect, reload the configuration:
https://gist.github.com/4709434
(there are some dirty details, which are nicely documented in this article)
Pointing your package managers to the local Pypi server
The final touch is to point pip and setuptools to your local Pypi Server, as documented in this article under the section configure PIP/Setuptools/Buildout.
Enjoy!
My first exam in >10 years :)
Having a good solution for managing your dependencies (and have them installed) on the target system is a bless for any developer. In an SaaS environment, where auto-scaling should be supported, this becomes a necessity.
Declaring dependencies
The default dependency manager for python, which has also great support on Heroku is pip in combination with virtual environment (read the heroku introduction for python for more info). Works like a charm and takes (almost) no learning curve.
There are however packages that require deep(er) integration in the OS and/or do not have the pip integration yet, such as numpy/scipy/scikit-learn. Heroku buildpacks address this topic, allowing you to compose custom “slugs”, which add the dependencies through customized install scripts.
Adding scikit-learn
While there is a buildpack for python with numpy/scipy (tnx!!), the scikit-learn library was still missing, so I started playing around with the buildpacks :) The result can be found on github, which allows you to run applications with these 3 libraries included.
To use this build pack, add the buildpack parameter to your heroku app:
https://gist.github.com/4653684
Next, you need to change the “requirements.txt” by a setup.py file (if I have time, I’ll put some time to refactor this). An example of a setup.py file can be found in this sample from wyn. Simply add ‘scikit-learn’ to his example to get the dependency installed.
How I made it work
The key to getting it work was adding the correct environment variables (required during the skikit-learn compilation):
https://gist.github.com/4653599
There is more work to do (replacing the setup.py approach inherited from wyn with the requirements.txt approach), perhaps adding R to the build pack …, but that’s for later :)