My work buddy and I were talking about tests. We wanted something to go through every route in our system and check for any obvious errors. That seemed like a pretty simple way to test an application.
So I wrote a generator to create tests for me automatically at work. I wanted to test every route we have in the system. We have hundreds of routes on this particular project I’m working on today. On our last large project we had over a thousand routes. What can I say? We like our routes.
The test to see if a route works is pretty simple and it looks something like this.
public function test_route()
{
$response = $this->call('GET', '/some/url');
$this->assertEquals(200, $response->status());
}
How to run A LOT of tests in Laravel?
So after running the test code generator I was left with about 300 tests for this project. Some failed, some succeed, others flat out gave me fatal errors and crashed phpunit. This was because we had changed database columns and it totally broke some sections of our the site. After fixing a couple of errors, I ran into a bigger issue. Out of Memory
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 41 bytes) in ../vendor/laravel/framework/src/Illuminate/Database/Connection.php on line 323
Basically, out of the box Laravel integration testing is a memory explosion waiting to happen. Testing will work so long as you have a little application. Don’t be alarmed though, I get emails all the time about pills we can take to increase size. Jokes aside, if you want to test each one of your routes, eventually when you have enough routes, you will run into this same memory problem I ran into.
There’s a quick fix to that problem. Just allow more memory, right?
ini_set('memory_limit', -1);
However, that just traded one problem for another. After running some really slow tests, after about 150 tests I start getting a continious stream of error exceptions.
kelvindorothy:~/space/laravel-tests$ phpunit
PHPUnit 4.8.18 by Sebastian Bergmann and contributors.
..F..FF.F..FFFFFFFFFFFFFF
.FFF.....FS......F..F..FF.FF.FSF.SF.FF 63 / 244 ( 25%)
...FFFF.FFFF.F.F..F.....SF..F.FF.F.FF..FFFFF...SFFFF.F.SF.FF.F. 126 / 244 ( 51%)
F.F.SFFFF.SFSF...FF.FSFFEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE 189 / 244 ( 77%)
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
Time: 1.08 minutes, Memory: 450.00Mb
There were 94 errors:
1) Programming_Session_Confirm_Stage_Update::test_route
PDOException: SQLSTATE[08004] [1040] Too many connections
2) Programming_Session_Course_Row::test_route
PDOException: SQLSTATE[08004] [1040] Too many connections
... yeah, this goes on for a while ...
... then we start getting weird shit like this...
94) Youth_Programs::test_route
ErrorException: include(/Users/K.D./space/jccfs51/vendor/symfony/symfony/src/Symfony/Component/Finder/Exception/AccessDeniedException.php): failed to open stream: Too many open files
... then some failures ...
FAILURES!
Tests: 244, Assertions: 139, Errors: 94, Failures: 78, Skipped: 10.
Well crap. What to do about this problem? My tests are erroring because the framework has created too many database connections. My guess is that Laravel stores it’s entire $app globally and there may be some places where mysql database connections are not closed. This is causing memory leakage and eventually a bunch of PDOException: SQLSTATE[08004] [1040] Too many connections errors.
One way to get around this too many connections problem is to not run so many tests all at once. We can limit our runs to only a subset of tests using phpunit suites. Here is a gist of someone who has done it this way.
However, I decided to tackle the problem head on. The real problem is too many database connections. Thus we will manually close the connections ourself by overwriting the default TestCase::tearDown method as seen next.
class TestCase extends Illuminate\Foundation\Testing\TestCase
{
/**
* The base URL to use while testing the application.
*
* @var string
*/
protected $baseUrl = 'http://localhost';
/**
* Creates the application.
*
* @return \Illuminate\Foundation\Application
*/
public function createApplication()
{
$app = require __DIR__.'/../bootstrap/app.php';
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
return $app;
}
/**
* Close out database connections, so they don't add up and eventually
* cause a bunch of PDOException: Too many connections errors for our
* tests, which otherwise would work normally if ran individually.
*
*/
public function tearDown()
{
if ($this->app)
{
$connectionName = $this->app->make('config')->get('database.default');
$this->app->make('DB')->disconnect($connectionName);
}
parent::tearDown();
}
}
Now when we run our tests, we don’t get a bunch of PDOException errors due to the too many connections. That is because we are closing them. Alright! Should we stop here? Well, we could. And if you’re not that familar with Laravel’s insides, then you might do well to stop here. However, I still have a few bones to pick with these tests.
- They are pretty dang slow.
- They use way to much memory.
After some research I found that other people had ran into this memory issue before too. There is a nice closed github ticket which hasn’t really solved anything to date (even though the ticket dates over 2 years ago). Granted, we got around our memory problem by setting memory_limit to -1 but I want my tests to run faster and use less memory. It seemed to me that the obvious place to first tackle is the TestCase::createApplication() method.
public function createApplication()
{
ini_set('memory_limit', -1); // yeah, I accept that Laravel likes to use lots of my memory... whatever, just do it.
$app = require __DIR__.'/../bootstrap/app.php';
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
return $app;
}
Notice that everytime a new test is ran, Laravel is re-including and bootstrapping the application. However, we discovered earlier that the PHP garbage collection is not removing stuff under the hood of Laravel. There is a ton of stuff behind the curtain of Laravel. Some of which could remain, even after the test is completed. This means, the more tests you create, the more memory is used. I could get through about 20-30 tests before my php started to exploded due to being out of memory.
So how do we really fix this?
So in case you still don’t quite understand the issue. Let’s run a bench mark test. I’ll use the ini_set('memory_limit', -1); hack I mentioned at the beginning of this article. It uses over 450Mb of memory.
One potential fix, re-use the same laravel application as much as possible
class TestCase extends Illuminate\Foundation\Testing\TestCase
{
/**
* The base URL to use while testing the application.
*
* @var string
*/
protected $baseUrl = 'http://localhost';
/**
* [$application description]
* @var null
*/
public static $application = null;
/**
* Creates the application.
*
* @return \Illuminate\Foundation\Application
*/
public function createApplication()
{
if (!static::$application) $this->resetStaticApplication();
return static::$application;
}
/**
* Don't call parent::tearDown()
*
* @return null
*/
public function tearDown()
{
// don't call parent::tearDown() or your application will be screwed up on next test
}
/**
* Resets the application. Reset whenever you do something that might
* affect the global $application
*
* @return null
*/
public function resetStaticApplication()
{
static::$application = require __DIR__.'/../bootstrap/app.php';
static::$application->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
}
}
Now with my changes in place, let’s see how things run. Will we actually be able to complete without the framework blowing up and throwing error exceptions on over half our tests because it used up too many database connections?
kelvindorothy:~/space/laravel-tests$ phpunit
PHPUnit 4.8.18 by Sebastian Bergmann and contributors.
..F..FF.F..FFFFFFFFFFFFFF
.FFF.....FS....F.F..F..FFFFF.FSFFSF.FF 63 / 244 ( 25%)
F..FFFFFFFFF.F.F..F.....SF..F.FF.F.FF..FFFFF.F.SFFFFFF.SF.FFFF. 126 / 244 ( 51%)
F.F.SFFFFFSFSF...FF.FSFFFFFF.SFFFFFFS.FFFF.FSFF..F..FSFFFFFFF.F 189 / 244 ( 77%)
F.FFFFF.F..F.F.FS.FF.FFFFF...F.F.FFF.FFFFFF..F...FF....
Time: 27.96 seconds, Memory: 267.25Mb
There were 145 failures:
1) Account_Edit_Picture::test_route
Failed asserting that 302 matches expected 200.
... this goes on for a while...
FAILURES!
Tests: 244, Assertions: 228, Failures: 145, Skipped: 15.
YAY! We got a lot of failures and some skipped tests, but no weird ass errors. This looks promising. We will try this for a while and see how it turns out. We’ve ran over 600+ tests using a similar method to this on our Devise CMS tests. Hopefully, this will work well.
Also, take a look at the memory and time now. It is down to 27.96 seconds and 267.25Mb. That’s a bit better.
Some Potential Problems with our approach to Integration Testing
There is some merit to the way Taylor setup the TestCase. So let’s try to understand why Taylor is resetting the application every time. If we understand that, then we’ll understand the dangers of the code I just showed you above to get around the memory problems.
The application is reset because Taylor wants us to test in isolation. When we create a new application, we are creating a new IoC container. There are variables stored in this IoC container. If those variables are tampered with because of our test, then it could affect every other test. If you don’t reset those everytime, we run the risk of our application container not being isolated and thus causing certain tests to mess with each other. For example, take these two tests,
class TestMe extends TestCase
{
public function test_me1()
{
URL::shouldReceive('route')->once()->andReturn('awesome');
print URL::route('something-here'); // awesome
}
/**
* @expectedException InvalidArgumentException
* @return [type]
*/
public function test_me2()
{
print URL::route('something-here');
}
}
Above we don’t actually have a ‘something-here’ route defined. We are mocking it with Mockery on the URL facade. I personally don’t use Mockery on Laravel’s facades often because I don’t use facades that often. However, this will show you the problem.
Above we are changing the URL facade to expect the route method. In our second test_me2() method we are just asserting that an exception is thrown because we don’t have a route defined for something-here.
This test should run successfully on the old out of the box TestCase.
kelvindorothy:~/space/laravel-test phpunit
PHPUnit 4.8.18 by Sebastian Bergmann and contributors.
..awesome.
Time: 627 ms, Memory: 29.50Mb
OK (2 tests, 1 assertion)
However, when we change to our modified TestCase we run into a different situation.
Now our results show that mockery has been applied across both testing methods `test_me1’ and ‘testme_2’. See below.
kelvindorothy:~/space/laravel-test$ phpunit
PHPUnit 4.8.18 by Sebastian Bergmann and contributors.
..awesomeFawesome
Time: 425 ms, Memory: 26.75Mb
There was 1 failure:
1) TestMe::test_me2
Failed asserting that exception of type "InvalidArgumentException" is thrown.
/Users/K.D./.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:186
/Users/K.D./.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:138
FAILURES!
Tests: 2, Assertions: 1, Failures: 1.
See how now we no longer get an exception in test_me2 because the mock we applied in test_me1 bled over. This is why testing in isolation is a good thing. How do we deal with this? Here are some simple steps.
A few takeaway tips
-
Don’t bootstrap Laravel on unit tests. Only on integration tests. We use Mockery a lot on unit tests. Unit tests however only require a single class to be loaded. Not an entire framework. If you are using Mockery in unit tests, then you should even be extending
TestCaseandcreateApplication()Feel free to make different classes such as,TestCaseandUnitTestCase. -
Don’t mock things on integration tests.
-
Don’t try to unit test your models. If you really want to see your models working, write some integration tests on classes which heavily use the models. If you use a
Repositorythen often you can just make calls to your repository and ensure that the models are indeed working. -
Use a real database and database transactions to rollback. This will save a lot of time compared to seeding your database over and over.
-
Use different phpunit testing suites for different types of tests.