There are about a hundred billion testing frameworks available, most of which provide some kind of interface to Selenium. So why would someone in their right mind quit a really good job at Magento to build another one?
To explain that I have to go back to the earlier part of the summer in 2015. For some reason I was brought in to work on a project at Magento’s Expert Consulting Group to build out the automated testing for around 200 individual test cases that needed to be run to validate some fairly specific functionality. Let me preface that by also noting that I’ve done some browser test automation in the past, but not that much. So I kind of knew what I was getting into, though not as well as I thought.
Let’s put some numbers to this. 200 test cases by about 30 minutes to build each test (when I’m lucky. Several tests took 2 hours or longer to build). That comes out to about 3-4 weeks of test development. Test development, yes. But there are more to those numbers because this was a Magento library that was being built and as such we had two multipliers.
- Support for Chrome, Firefox, Internet Explorer, and Safari.
- Support for Magento CE 1.8 and 1.9 plus support for Magento EE 1.12, 1.13, and 1.14.
There are now 4000 test runs that need to be made. And, lo and behold, the themes for each version do not work exactly the same and so in the tests I need to take into account differences between all the different iterations.
Did I succeed? No. Not at all. It worked, but not well, and not completely. And on top of that I spent around 4-6 weeks (I can’t actually recall) of working 60-80 hours per week (I can’t recall that exactly either) trying to make it work.
Could there have been better tooling in the community that would have helped? Probably to some degree. But how many browser testing tools allow for the arbitrary switching out of different design packages and run the tests exactly the same way, but with different selectors? None that I could find. And how many allow you to automate the checkout? Zero. You’ll be doing that yourself. And making changes to the checkout? Again, you’ll be doing that yourself.
And so I thought to myself “if there are really are 250k websites running Magento am I the only one who is dealing with this?” The answer would seem to be no. With a quick, completely unscientific, survey I did it looks like almost nobody automates their testing to a significant level. That doesn’t mean that nobody does, but the numbers are so close to zero so-as to consider it zero.
Why not? Because it’s hard and incredibly costly to do it. The learning curve is not quite as sloped as Magento 1 (and DEFINITELY not as sloped as Magento 2) but it is pretty hefty.
So I endeavored to make my contribution to the Magento community by helping companies automate their frontend, browser, user acceptance and whatever other testing that requires a browser as much as possible and in a way that is
- as simple as possible, but not too simple
- easily repeatable across different implementations
- integratable (meaning that third party vendors could distribute Magium tests and they would just work with Magento)
I believe that Magium does, and will, succeed at this.
As Simple as Possible, But Not Too Simple
There are well over 100 individual commands that need to be run to complete a checkout in Magento. What I did with Magium is create a buffer between what the test developer wants to do and what Selenium needs to do it. Does a tester want to think in terms of how Selenium will need to be executed? Not really. They want to think about how they can fullfill the test case requirements.
A big feature of Magium is the separation of concerns between individual components. What this means is (the intention is) that an action should not be tightly coupled to the interface. In other words, the purpose of the action is to execute the functionality. But what happens if there is a design package change? Rather than changing the action, you change the theme configuration.
This should be a boon for Magento extension developers because they can develop a Magium test module for their extension and deploy it as part of a test package. Their customers can just run the included tests as part of the normal test results.
In addition the intension is that there may be third party services that you might want to integrate in with your testing as well. All should be possible with Magium
There is a lot more that can be said, but I tell you truthfully that I want to help Magento merchants automate their testing. This is not just for those who can afford it. In fact, this is not for those who can afford to have full-time test developers.
Magium is for people who are tired of having things wierdly break because something changed in their tests and they didn’t know about it. Magium is for people who don’t have the time to write proper test automation.
Magium is the framework I wish I had had back in July.