On several recent projects I’ve been using VirtualBox and Vagrant to spin up new development machines. The simplicity and power of the two tools together is amazing, and stops me from cluttering up my main machine with tools and packages from random side projects. It hasn’t been all sweetness and light though. I wrote recently about issues with node.js, npm and express, and this week ran in to a far more frustrating issue. It involved nginx and sendfile, with javascript truncated and this developer frustrated.
The Problem: CSS & Javascript truncated
My setup was a very basic one – a VirtualBox running:
- Ubuntu 12.04
- nginx
- PHP-FPM
My code was mounted from a windows host, and initially all was going well. Then suddenly my app started spitting out javascript errors. Digging in to chrome’s error console, it looked like the last 40 or 50 bytes of the file were missing. As these bytes contained the end of a function, them being absent meant the file had invalid syntax so the application didn’t run. It was a similar story with the css – random bits chopped off the end of some files, and in others changes I made not being reflected when viewed in the browser. Javascript truncated, the same for css – bad times all round.
I suspected a cache issue initially. I flushed the browser cache, tried a new browser, and even restarted nginx on the vm – all to no avail. Interestingly, when I added more code to the bottom of an affected javascript file, it was still only the last 40-50 bytes being chopped off – the file itself wasn’t frozen in it’s original, broken state. So, what’s going on there?
Sendfile
Nginx has a sendfile directive, which defaults to on. It allows nginx to use the Linux kernel’s sendfile() operation to read the requested file from disk. This is typically quicker than using read() and write(). So far, so good.
The problem is that sendfile doesn’t work so well in virtual environments, as noted in nginx’s “Pitfalls” section. The fix turned out to be insanely simple – just turn sendfile off. As it’s a development machine, the performance hit incurred by turning off this feature is pretty negligible. (It would want to be a pretty significant performance hit to outweigh the file truncation!)
In the nginx config http server block, I added:
1 | sendfile off; |
Once nginx was restarted, I was getting served full static files, and all was well with the world again!
Thanks kindly for your post, it’s just made my day. I’ve had exactly the same issue and have been scratching my head wondering how to phrase a Google search, let alone find the answer. Wicked stuff.
Holy ever living crap. I just lost 2 hours of my life to this issue, and would’ve kept on losing had I not found this article.
I rarely comment on blogs, but man I just had to say thanks.
Thank you for this post. Saved me a bunch of time. Thanks again!
Thank you so much!!! I have wasted so much time during development because of this bug!!!!!
I wanna kill something. Not you, my friend – you are a sanity saviour. A beer. I need to kill several beers.
Woot! Awsome! Thanks for posting this! This is exactly the issue I was having.
You just save my life!! Great!
excellent! i’m using it in every new project
You just saved a very bad day of mine. Where can I buy you some coffee??
Thanks! That was a big WTF for me.
Thanks! You just saved my… evening.
Many thanks for saving me from frustration
Wow! Amazing!
Thanks so much!
This was a total WTF moment watching files get truncated or null terminating bytes get added.
Thanks for the quick fix!