LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-15-2018, 05:54 AM   #856
brodo
Member
 
Registered: Jan 2004
Location: Poland, Poznan
Distribution: Slackware current 32 / 64
Posts: 406

Rep: Reputation: 30

How can I make a copy of the HDD in a form of a multiple archive volumes placed on the other USB connected HDD ?
2...3 GB pieces would be optimal.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 04-15-2018, 01:04 PM   #857
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Original Poster
Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
You could use the skip and seek parameters.

Code:
$ dd if=/dev/sda of=/mnt/sdb/volume1.bin bs=1M count=1000 
$ dd if=/dev/sda skip=1000 of=/mnt/sdb/volume2.bin bs=1M count=1000 
$ dd if=/dev/sda skip=2000 of=/mnt/sdb/volume3.bin bs=1M count=1000 
$ dd if=/dev/sda skip=3000 of=/mnt/sdb/volume4.bin bs=1M count=1000
edit, didn't really need the 'seek' parameter.

Last edited by AwesomeMachine; 04-15-2018 at 02:34 PM.
 
1 members found this post helpful.
Old 04-15-2018, 02:11 PM   #858
brodo
Member
 
Registered: Jan 2004
Location: Poland, Poznan
Distribution: Slackware current 32 / 64
Posts: 406

Rep: Reputation: 30
How your skip and seek parameters relate to a 250GB disk to be cloned ?
Is it practical ?

If talking about directory archiving this is useful:
Code:
tar cvzf - /home/user/something/ | split --bytes=2000MB - something.tar.gz.
Can this idea be arranged with DD to clone HDD in this manner ?
 
Old 04-15-2018, 02:20 PM   #859
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,552

Rep: Reputation: 872Reputation: 872Reputation: 872Reputation: 872Reputation: 872Reputation: 872Reputation: 872
With dd you will also archive the empty spaces though...
 
Old 04-15-2018, 02:45 PM   #860
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Original Poster
Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
Quote:
Originally Posted by brodo View Post
How your skip and seek parameters relate to a 250GB disk to be cloned ?
Is it practical ?

If talking about directory archiving this is useful:
Code:
tar cvzf - /home/user/something/ | split --bytes=2000MB - something.tar.gz.
Can this idea be arranged with DD to clone HDD in this manner ?
Skip skips the number of blocks in the input. The drive you're reading from is the input. Seek you don't actually need in my example, so I edited it out. You can add compression to a dd command by piping and redirect or double piping. It's in the OP.

No, dd has no split switch. Usually when a user wants to make separate volumes with dd, it's to fit each volume to a DVD disk.
 
Old 04-16-2018, 03:54 AM   #861
qrange
Senior Member
 
Registered: Jul 2006
Location: Belgrade, Yugoslavia
Distribution: Debian stable/testing, amd64
Posts: 1,061

Rep: Reputation: 47
er, would it be possible to make 'smart dd', so that it reads partition table first, and, if it recognizes the (checked) filesystem, it reads only used blocks if user desires so?
so that later it writes less.
I think clonezilla does that?
 
Old 04-17-2018, 08:03 PM   #862
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Original Poster
Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
The excellent feature of dd for back-ups is that it will create a bootable copy of a drive. If you don't need it to be bootable, there are other tools much more suited to the task, like clonezilla, rsync, bacula, etc.
 
Old 04-19-2018, 05:17 AM   #863
qrange
Senior Member
 
Registered: Jul 2006
Location: Belgrade, Yugoslavia
Distribution: Debian stable/testing, amd64
Posts: 1,061

Rep: Reputation: 47
dd will also clone partition table. if you are upgrading drive capacity, it will take some time to resize partitions if needed.
tar is also nice option, probably used by some of aforementioned tools.
 
Old 04-20-2018, 12:41 AM   #864
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Original Poster
Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
Dd will copy everything on the drive. That's not technically true, but it will copy everything in the user-data storage areas of the disk.

Tar simply puts multiple files into one bigger file. Linux file-compression programs don't compress multiple files, so tar is necessary to compress a directory. I would estimate that tar is seldom used for it's original purpose, which was to make files ready for tape archives.
 
Old 04-20-2018, 04:52 AM   #865
qrange
Senior Member
 
Registered: Jul 2006
Location: Belgrade, Yugoslavia
Distribution: Debian stable/testing, amd64
Posts: 1,061

Rep: Reputation: 47
tar is better than most other similar utils, because, iirc, it can preserve linux file attributes, permissions..
one advantage over dd would be defragmentation on hdd.

Last edited by qrange; 04-20-2018 at 04:54 AM.
 
Old 09-19-2018, 06:22 PM   #866
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Original Poster
Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
dd tar rsync

Quote:
Originally Posted by qrange View Post
tar is better than most other similar utils, because, iirc, it can preserve linux file attributes, permissions..
one advantage over dd would be defragmentation on hdd.
Dd preserves file attributes. Tar is ok, but you need room for the tar files. In this day and age I prefer rsync and then if it needs to be bootable, just boot with a live system and install grub to it.


Disks are getting too big to use dd for typical backups. But dd still has a lot of uses.
 
Old 01-30-2019, 08:45 AM   #867
tschenk
LQ Newbie
 
Registered: Oct 2008
Distribution: Salix 13, Linux Mint Sylvia
Posts: 2

Rep: Reputation: 0
Post When you want to know how far dd has gotten ....

This line at the begin on page 1 does nott work on my linux station:

`ps aux | awk '/bin\/dd/ && !/awk/ {print $2}' | xargs kill -s USR1 $1`,

since `ps aux | awk '/bin\/dd/ && !/awk/ {print $2}'` is an empty string.
A solution would be to run:
`pidof dd | xargs kill -s USR1 $1` since I tested `pidof dd | xargs echo $1` is not empty.
In the end I use this twoliner found on this page :
(https://www.cyberciti.biz/faq/linux-...-while-coping/)
in pasting the result (the PID) of
`pidof dd` ### <--- say pid is 21145
in the next line :
`sudo kill -USR1 21145`

Further an interesting remark discussing the exact same extension to the dd-command you present later on your page 1
(reading: "Note that sending a SIGUSR1 signal to a running 'dd' process makes it ... ")
is that this will only work on (some) Linux, (see this discussion :
(https://unix.stackexchange.com/quest...d-mean-exactly)

-USR1 works like this only in the GNU version of 'dd' (AFAIK). A SIGUSR1 to 'dd' for other OSes (eg AIX, HP-UX & Solaris) will terminate the process, by default.
And instead of SIGUSR1 use SIGINFO (or INFO) on OSX (probably the same on BSD)

And like the last time I posted something here: Great page, thank you
 
Old 02-03-2021, 12:25 PM   #868
rincon
LQ Newbie
 
Registered: Mar 2020
Posts: 20

Rep: Reputation: Disabled
Wink

awesome great article,


a great rundown on a very important topic.

im in the mid of a huge process of expanding my partition and
this thread just helps to clear a lot of current questions,


thanks alot. I will bookmark this thread!
 
Old 02-03-2021, 04:07 PM   #869
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,753

Rep: Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983Reputation: 7983
Quote:
Originally Posted by rincon View Post
awesome great article,

a great rundown on a very important topic.

im in the mid of a huge process of expanding my partition and this thread just helps to clear a lot of current questions,
thanks alot. I will bookmark this thread!
Oh? Never had a 'huge process' of repartitioning a drive before, it's always been pretty simple. And you've been asking about drive partitioning/re-partitioning now for EIGHT YEARS, under your other user ID, sayhello_to_the_world:
https://www.linuxquestions.org/quest...to-4175463188/

What questions did this thread clear up for you? Which posts? Be great if you 'posted all of your findings', so that others with those questions could locate the posts in this thread easily.
 
1 members found this post helpful.
Old 02-16-2023, 06:24 PM   #870
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Original Poster
Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
DD in 2023

The needs for dd have changed radically since about 2015. SSDs parallelize writes, sdo the new way to wipe drives is with the SECURITY_ERASE_UNIT ATA command. But that only works up to SATA drives.


NVMe drives have a different command set. Basically the same command set--ATA--was used for 20 years. Now along comes NVM command set which is quite robust, quite flexible and powerful, fully accessible to the general public, but designed with SSD technology in mind.


A sequential bitstream copy program such as dd may not work predictably with the newer NVMe controllers. With wear concerns, a SSD won't write /dev/zero to an entire drive.



And, each manufacturer kind of rolls their own command set out of the immense NVM set of command sets. NVMe drives will be used in many consumer, government, military, institutional and industrial devices the likes of which cannot be fully imagined yet. Therefore the drive controllers do much more, and require a more robust command set that dd can't adequately exploit presently



Dd was made with a much simpler model in mind. Although I'm only aware of one area where it fails with SSDs, and that is writing sequential zeroes to an entire drive. That model is now obsolete.


Also, duplicating massive swathes of empty drive space is no longer rational, so at the very least, for mirroring, use clonezilla. Dd still has it's uses, but sequential mirroring is on the way out.
 
1 members found this post helpful.
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 04:22 AM
The best way to learn? iz3r Programming 7 02-06-2005 11:00 PM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 03:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 01:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 01:02 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:33 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration