Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
L
lecture materials
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
generic software skills
lecture materials
Commits
2f5d8fc8
Commit
2f5d8fc8
authored
10 months ago
by
Florian Ziemen
Browse files
Options
Downloads
Patches
Plain Diff
raid and erasure coding
parent
953f8b7f
No related branches found
Branches containing commit
No related tags found
Tags containing commit
1 merge request
!11
File and Data Systems
Pipeline
#71419
passed
10 months ago
Stage: test
Stage: build
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
lectures/file-and-data-systems/slides.qmd
+26
-5
26 additions, 5 deletions
lectures/file-and-data-systems/slides.qmd
with
26 additions
and
5 deletions
lectures/file-and-data-systems/slides.qmd
+
26
−
5
View file @
2f5d8fc8
...
@@ -320,16 +320,37 @@ mv same_same_but_different slides.qmd
...
@@ -320,16 +320,37 @@ mv same_same_but_different slides.qmd
# Redundancy
# Redundancy
## RAID
Can protect against
* Accidental deletion
## Erasure coding
* Protection against data loss
* High Availability
## Backups
* Keep old states of the file system available
* Need at least as much space as the (compressed version of the) data being back-uped.
* Ideally at different locations.
* Automate them!
## RAID
* Combining multiple harddisks into bigger / more secure combinations
* RAID 0 distributes the blocks across all disks - more space, but data loss if one fails.
* RAID 1 mirrors one disk on an identical copy.
* ...
* RAID 5 is similar to 0, but with one extra disk for (distributed) parity info
* RAID 6 is similar to 5, but with two extro disks for parity info (levante uses 8+2 disks).
p
## Erasure coding
* Similar to raid, but more flexible with the numbers of disks (more than two *parity* disks are possible).
* Used in object stores
* Usually, data is distributed across independent servers for higher availability.
# Lustre as a parallel file system
# Lustre as a parallel file system
*What if you are not the only one controlling the FS?*
*What if you are not the only one controlling the FS?*
. . .
. . .
The file system becomes an independent system.
The file system becomes an independent system.
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment