Graphite

Whisper filesystem path permissions flipping unknowingly

Asked by Sumeet Grover

Hi.

We have Graphite running in our company for over a year, and we capture large scale stats. ServerG that receives the metrics stores the metrics here:

/opt/graphite/storage/whisper/

Recently, we noticed that a metric that was being sent into Graphite was inaccessible on the Web App, and we found that the whisper path to it had gotten owned by root, in a very strange permission set u=-wx i.e. 300. We do not have any process other than carbon tha touches the whisper directories.

Yesterday, I installed stats collection on a new server, and half the stats are accesisble on front end, i.e. they have correct, read permissions, and the other half have no read permissions to the whisper directories.

How is this possible? Especially, only for few metrics?

We have a single Carbon daemon, running under the root user.

I'd appreciate if you have come across this before and solved it.

Thanks.

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Jason Dixon (jason-dixongroup) said : 		#1

Something is changing your file permissions but it wouldn't be caused by Carbon. Some other process is mucking with your files.

Jason

Revision history for this message
Dave Rawks (drawks) said : 		#2

https://github.com/graphite-project/carbon/issues/82 < seems that some weirdness with file/directory permissions and umask might have been caused by a race condition in twistd. I haven't seen reports of this in quite some time though. I'd try updating your graphite stack to 0.9.12 and also bump your twistd modules to a recent supported version to see if it helps.

Can you help with this problem?

Provide an answer of your own, or ask Sumeet Grover for more information if necessary.

To post a message you must log in.