Snapshots are generated weekly during scheduled maintenance and when nodes are
(un)deployed. A snapshot consists of hard links to SSTable files; so, a snapshot takes up
little disk space. But when an SSTable is deleted during compaction, the snapshot remains
and does consume disk space. This can add up over time. There is currently nothing in
place for managing snapshots. Here are a few possible options,
1) Move snapshots older than X to a specified location
2) Move all snapshots to a specified location
3) Delete snapshots older than X
4) Move N snapshots (from oldest to youngest) to a specified location
5) Delete N snapshots (from oldest to youngest) to a specified location
This could be done as a reoccurring operation. We could also introduce some new metrics to
monitor snapshot disk usage similar to what we already have for the data directories. If
the disk usage exceeds a threshold, we can fire an alert and perform one of the above
actions. I think that this is another good step we can take for providing storage node
disk management.
Thoughts?
- John
Show replies by date