Varnish cache problem

[crazycatz]

I'm having a problem with the varnish cache, where it is never updating some file-types.

I changed an HTML page a few weeks ago, and was frustrated that it hadn't updated after around 30 minutes. I followed the help topic on disabling it, but ultimately figured "it will update sooner or later."
After trying to update quite a few other files today, and most not updating, I decided to test it by deleting every file, except a single .htaccess containing:

Header set Cache-Control "max-age=0, private, no-cache, no-store, must-revalidate"

A few hours later, I'm still seeing the old files (including the now weeks old HTML file).
I have tried clearing browser cache, using a different browser, using a different computer, etc., I'm fairly confident that the server is still serving these (now nonexistent) files. Particularly, js and php files are returning 404, while others (html, css, svg) are returning content.

Something else, possibly related: I've had a few paths that would return a 404 even while I'm looking at the file in ftp/cpanel. Changing the name ("a.html" -> "aa.html") always fixed the problem, but this probably shouldn't have been happening.

Is there something on my end I can/should do to fix this? Otherwise, could an admin please take a look at my account?

 

[Dead-i]

Hi,

That's really strange. I tried going to your website, and I couldn't see your deleted files (and just an empty directory), but I confirmed that the Cache-Control header was being properly sent. I tried making a test file on your account, accessed it, and deleted it; it seemed to delete fine, and Varnish didn't cache it. Additionally, you mentioned that you saw the old files after a "few hours", but we only cache static content for 5 minutes. Could there be something at your end doing this?

Thank you,

 

[crazycatz]

I'm not sure what exactly it is now. I had a friend try accessing it (out of state, different ISP), and on Firefox he saw the old files (same as me), Chrome saw an empty directory, and IE switched between the two (sometimes loaded the old, mostly got the directory though). Couldn't get all three to be consistent.
On my end, I got Chrome to show the directory index once, but went back to the cache after a refresh.

I also verified I'm not caching it on my end, ran it through Fiddler (HTTP proxy) and got:

Spoiler

GET http://[...].x10host.com/ HTTP/1.1
Host: [...].x10host.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://[...].x10host.com/
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache


HTTP/1.1 200 OK
Date: Wed, 15 Jun 2016 00:03:36 GMT
Expires: Sat, 18 Jun 2016 00:03:36 GMT
Cache-Control: max-age=259200
Last-Modified: Tue, 14 Jun 2016 23:33:57 GMT
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=edge
Content-Length: 991
Content-Type: text/html; charset=utf-8
X-Varnish: 271778950 201985902
Age: 63102
X-Cache: HIT
X-Cache-Hits: 236
Accept-Ranges: bytes
Connection: keep-alive

The first set of numbers in 'X-Varnish' changes and 'X-Cache-Hits' increases by one on every reload, so that should rule out ISP caching as well (I think?).


My theory is that Varnish is using the Cache-Control length (a few days for most files). I assumed it only uses that header to see if it should cache, but used its own time lengths.
Such as: header says to cache for 3 days, Varnish sees that it should be cached and does so for only 5-10 minutes, and client browser does so for specified 3 days; header says to not cache, Varnish doesn't cache at all, and client doesn't. This would explain why some files are returning (mostly html/css) and some aren't (mostly php).
You said it should cache for only around 5 minutes, though, which is what I expected. So not sure if this theory has any merit.

 

[Dead-i]

Hi crazycatz,

Thank you very much for this; I tried from the command line and saw the same issue. While we do typically cache dynamic content for 5 minutes, the "Cache-Control" header you posted had a "max-age" of 259200, which is 72 hours, and so Varnish was using this copy of the webpage. I forced a purge, and it looks like this isn't happening anymore. Please can you check again?

Thank you,

 

[crazycatz]

I'm getting the directory listing now, so seems to be working. Some other sub-pages are still returning, but if it is just the cache-control I can wait it out.
I wasn't aware varnish followed this directive, or at least it didn't at one point. I should be able to work with it knowing this.

Thank you