<– Back to all “Work Diary” entries
I’d created a new Rake task and wanted to add an Rspec test for it. I created said test, pushed to Github, CI passed, and the test was merged into the codebase. Easy.
Fast forward a few weeks and I’d refactored some code and this requires me to tweak the way I’m testing the Rake task I mentioned above. Mind you, I haven’t touched the underlying task code, just some code the tasks calls. Anyway, I make the necessary changes, run the spec locally to ensure that it’s passing, and push the change to Github. CI runs the test suite and…the spec I just changed fails.
This gets me scratching my head, and so I start re-running the spec…
Re-run spec locally, it passes. Every time.
Re-run tests on CI, the spec fails. Every time.
At this point, I’m thinking it’s a race condition or something is off with my environment, so I ask a co-worker to run the spec on his laptop and see what happens for him. It passes, no issues.
He then asks me, “Have you tried running the entire suite?” I hadn’t – I’m running the app in Docker and the UI tests aren’t quite passing yet, so I run specs as one-offs and rely on CI to run the suite for me and tell me if anything is broken beyond whatever code silo I’m working on.
He then runs the entire suite on his machine and gets the same failure as the CI system is getting. Yay?
He then notes that my spec is using class.any_instance to mock some behavior, which can lead to some odd side effects. I follow his advice to instead override the class.new method to return a mock object. I do this, and while it doesn’t fix the failure on the CI system it does make me feel better about the spec 😌
Returning to the bug at hand, since he can run the entire test suite but I can’t, and running the entire suite seems to be the only way to replicate the issue, he starts using pry to step through the code on his machine and discovers that our newly-added class.new isn’t being called once like it should but is instead being called twice!
This then points him in the direction of one line of code that appears in each spec that tests a Rake task: Rails.application.load_tasks.
It turns out, this line gets executed every time it appears, which means that, for the 3 spec files testing 3 rake tasks that all have that line, three copies of the app’s Rake tasks are being loaded. This wasn’t breaking the other task specs as they are idempotent; however, my spec required database interactions and thus assertions that related to the number of database records for a certain table were always failing.
The fix ended up being pretty straightforward. We added a spec support file that only calls Rails.application.load_tasks the first time a Rake-related spec is run, that way the application’s Rake tasks are only loaded once and only when needed.
Key Takeaways
- Try to replicate the failure workflow as much as possible. This is tricky when it relates to tests since, while technically the environment may be the same when running tests, how tests are being run (i.e., in isolation vs. as a suite) may have an impact too, so don’t overlook or assume anything.
- Know your tools. Sometimes info off of the internet is missing vital context. In this case, there are tons of posts all over the internet telling you to use
Rails.application.load_taskswhen creating Rake task specs, but they all don’t understand the tool well enough to know that this behavior is just waiting to bite them. I’m as guilty of this as the next guy, but it goes to show that investing time in learning deeply will always pay off in the long run. - Get a second opinion after a few failed attempts trying to figure it out yourself. Shout out to my co-worker for helping me get to the bottom of this one. Apparently, he’d been bitten by something similar in the past, and it was really nice to be able to piggy-back off of some of his hard-won wisdom.
