Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
OK, the best way to do what you want is to partition the new drive before you copy everything. You can make the partitions any way you want as long as there are enough of them. Sector 1 of drive 0, sda, hda contains the MBR. The MBR is 446 bytes, but there are signature bytes at the end of sector 1, "55 AA" that have to be there for anything to read the partitions. So, the exact mix of partitions is not the important thing, that is primary and logical. If you install MS Windows first it will install to an extended partition if you tell it to. I always put Windows in a logical drive on an extended partition, along with swap. Then I make two primary partitions for /home and /. I actually make / first, then /home, then the extended parttition, and then MS Windows partition, and finally swap. A DOS boot disk with a Partition Magic second emergency floppy disk works wonders. You just boot with the DOS floppy, or make a floppy image with dd and use cdrecord to make a cd out of it. Boot, take out that disk, and put in the second floppy of Partition Magic and type:
PQMAGIC
and there's you fully featured partition program, MS Windows and linux. PM 8.0 works for everything.
Since you want to keep the partition tables separate:
dd if=/dev/hda1 of=/dev/hdb1
These should be the / directories for linux.
Then do every partition. When you get done, boot from the linux install CD into rescue mode and tell it to fix everything, esp GRUB. Unless you use NTLDR to boot, but please don't do that. The rescue mode should fix the the first 429 bytes of sector 1, or boot strap code, without touching the partition table.
If you get hung up on doing an NTFS partition, which needs to be type "07" for MS Windows to even work, first write zeroes to all of the new disk:
dd if=/dev/zero of=/dev/hdb
Then partition. You can make the MS Windows partition linux type "83" and fix it later. Heres how you fix it so it's NTFS:
If it's in an extended partition:
dd if=/dev/hdb | hexdump -C | grep '00 83'
Look where it finds that on the terminal screen. Go there with dd:
But figure out, by the offset in hexdump, where dd has to go. In the above example, dd is going to start at decimal offset 5,120,000. That's the default bs=512 times skip=10000. You're skipping 10000 blocks of 512 bytes, then recording a single sector, count=1. Hex offset is calculated with a calculator in linux, 5,120,000 d = 0x4e2000 h. Your screen is going to give you hex values, but dd works in decimal. You have to convert the hex offset where "00 83" are found to decimal so you can make dd go to the right spot. When you capture file.bin, open it with a hex editor and change "00 83" to "0007", then:
dd if=/directory/file.bin of=/dev/hdb seek=10000
That writes the new value to the extended partition.
If MS Windows is in a primary partition, is it partition 1, 2, 3, or 4?
0x1c2
0x1d2
0x1e2
0x1f2
Contain the partition type indicators for the drive. Do:
dd if=/dev/hdb of=/directory/file.bin count=1
That will record sector 1, the MBR, which contains the primary partition. Type "05" is extended, type "83" is linux, type "82" is linux swap, "01" is DOS, "07" is NTFS/HPFS. So, if your partitioning program doesn't make NTFS partitions, and you made a linux partition for MS Windows, change the corresponding type indicator:
location hex offset in sector 1:
0x1c2
or
0x1d2
or
0x1e2
or
0x1f2
with a hex editor. Screw it up afew times. Use fdisk -l in linux to see the aprtition is NTFS, and to see all your partitions. Now don't freak, just take it slow, easy, like you would with a really attractive, classy woman on a date. Give her what she wants, and she'll give you what you want. That's what linux is all about.
just one of them, depending on which primary partition, 1-4 you want to be NTFS, "07". Then, write the file back to the HDD. The partition is now NTFS type
Dear AwesomeMachine,
Thanks for your help - The the method I used to attempt the cloning. I'm using a desktop computer to do the cloning.
I have a 40.0 GB HD installed as the primary on the first IDE controller of the desktop computer. Its running RedHat Linux 9.0.
The 40.0 GB HD from my HP laptop computer is installed as the slave on the first IDE controller.
The 100.0 GB is the primary on the second IDE controller
I issues this command to start the process: dd if=/dev/hdb of=/dev/hdd bs=4096 (sometimes is bs=2048)
The following diagram is the state of the system after many cloning attempts using differnt "bs" values
[root@bangkok ~]# fdisk -l
********* HP Laptop computer's HD ************
Disk /dev/hda: 40.0 GB, 40007761920 bytes =====> this is the HD from the HP Laptop
255 heads, 63 sectors/track, 4864 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 902 7245283+ 7 HPFS/NTFS
/dev/hda4 903 4864 31824765 f W95 Ext'd (LBA)
/dev/hda5 903 915 104359+ 83 Linux
/dev/hda6 916 2062 9213246 83 Linux
/dev/hda7 2063 3082 8193118+ 83 Linux
/dev/hda8 3083 3337 2048256 83 Linux
/dev/hda9 3338 3592 2048256 83 Linux
/dev/hda10 3593 3719 1020096 82 Linux swap / Solaris
/dev/hda11 3720 3900 1453851 83 Linux
/dev/hda12 3901 4283 3076416 7 HPFS/NTFS
/dev/hda13 4284 4864 4666851 7 HPFS/NTFS
omitting empty partition (7)
Disk /dev/hdb: 100.0 GB, 100030242816 bytes ======> I was never able to get further than this!!
255 heads, 63 sectors/track, 12161 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hdb1 * 1 902 7245283+ 7 HPFS/NTFS
/dev/hdb4 903 4864 31824765 f W95 Ext'd (LBA)
/dev/hdb5 903 915 104359+ 83 Linux
/dev/hdb6 916 2062 9213246 83 Linux
[root@bangkok ~]#
Jun 28 12:33:23 Lab-linux kernel: end_request: I/O error, dev 03:40 (hdb), sector 22352652
Jun 28 13:14:24 Lab-linux gconfd (root-1215): Received signal 1, shutting down cleanly
Jun 28 13:14:24 Lab-linux gconfd (root-1215): Exiting
*************************************
Some questions:
I have tried this cloning procedure using "dd" many times and the program always stops at the same place, between partition (boundaries) six(hda6) and seven(hda7) - I would think that a file system bit stream duplicator should not be concerned with partition boundaries - is this statement true??
By the way, if I clone the PC's HD(80.0GB) to the 100.0GB Target HD, the cloning works and I'm able to boot from the 100.0 GB HD using both the PC and the Laptop computers. (the PC's HD only has linux RedHat 9.0 installed.)
Since the OSs and the data on the HP HD are less than 40.0 GB, what value should "bs" be in order to use a minimum amount of space on the 100.0GB HD? (speed is an issue-see the last question below)
Does the program stop when the destination drive is much larger than the source drive and all the data on the source drive has been transferred or does it try to use the entire destination drive?
In reference to partitioning and installing the OSs on the new HD, this is not an option they are already installed on the 40.0GB HD. I need to transfer everything to a larger HD drive first.
My strategy is to clone the 40.0GB HD first and fix the partition after the cloning is successful - even fdisk program says that the partition are in the wrong order. I can't afford to lose any data. The system works fine, right now.
In reference to "I always put Windows in a logical drive on an extended partition, along with swap.", Why?
Partition Magic is a useful program, but its unforgiving when it is checking a system with partition errors, it wont proceed. Many times I use it just for a quick error check of a mixed HD (OSs) environment. In my opinion, the Linux fdisk and sfdisk are better programs for identifying and fixing problems in a mixed OS system environment, .
Let say I had no problem with the partitions on the 40.0GB HD, what command(s) would you suggest to clone this drive to the 100.0GB HD using minimum space and maximum speed?
I'm looking forward to your response /Bennett
Click here to see the post LQ members have rated as the most helpful post in this thread.
Having trouble copying large (mostly empty) disk to smaller disk
This thread is extrememly helpful. thanks for the effort.
To make a long story short, I have an old 20gig ide drive that is working fine, but probably getting tired. It is the boot disk and houses the centos linux os. Most of it is empty; only about 7 gigs is being used. I am trying to put everything on an 18.2 gig scsi drive. At first i tried just dd the whole drive /dev/hda to /dev/sdd, this gave me No space left on device errors. Then I partitioned /dev/sdd to have a boot and ext3 partiion. I dd the boot parition and it did clone.
I can't seem to get the /dev/hd2 partion to copy over. I have read this thread and tried several things.
Any ideas on how to solve this problem? Below is my output.
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4k conv=notrunc,noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4096 conv=notrunc,noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
[root@centos ~]# man dd
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4096 conv=noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
This thread is extrememly helpful. thanks for the effort.
To make a long story short, I have an old 20gig ide drive that is working fine, but probably getting tired. It is the boot disk and houses the centos linux os. Most of it is empty; only about 7 gigs is being used. I am trying to put everything on an 18.2 gig scsi drive.
<snip>
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4096 conv=noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
Anybody?? I tried again. Reminder that hda is 20g with about 11 empty gigs and sda is 18.2g This time I used:
dd if=/dev/hda of=/dev/sda conv=noerror
I got the error message above.
Just to see what happened I rebooted with both hda and sda in the machine. I got a warning that there were identical paritions and that linux was going to use sdd1 and sdd2 (from the new drive) and not the hda partitions (I'm not sure why). Centos (linux) started up and I coudl see all the files, but nothing mounted correctly, mdadm, fstab, etc. I'm not sure what was going on?
I'm warry about continuing until i get the dd error resolved. Anybody know how to get a good copy under these conditions?
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Quote:
Originally Posted by bennettr
********************************************************
I have tried this cloning procedure using "dd" many times and the program always stops at the same place, between partition (boundaries) six(hda6) and seven(hda7) - I would think that a file system bit stream duplicator should not be concerned with partition boundaries - is this statement true??
It is, and not. It depends what you are doing with it. If it doesn't work the first time, you don't have to keep trying the same thing over and over. The only time dd is not concerned with partitions is when the disk is not partitioned. There aren't really partition boundaries on the disk. All the partition info is in the MBR, except if you have an extended partition, of which you are allowed two, I think.
Quote:
Originally Posted by bennettr
By the way, if I clone the PC's HD(80.0GB) to the 100.0GB Target HD, the cloning works and I'm able to boot from the 100.0 GB HD using both the PC and the Laptop computers. (the PC's HD only has linux RedHat 9.0 installed.)
Since the OSs and the data on the HP HD are less than 40.0 GB, what value should "bs" be in order to use a minimum amount of space on the 100.0GB HD? (speed is an issue-see the last question below)
You really haven't taken advantage of what is already written in this thread. leaving out 'conv=notrunc' is the only thing which will reduce an image size.
Quote:
Originally Posted by bennettr
Does the program stop when the destination drive is much larger than the source drive and all the data on the source drive has been transferred or does it try to use the entire destination drive?
I asked you to partition both drives before you use dd, and leave out 'conv=notrunc'. That is the only way this will work.
Quote:
Originally Posted by bennettr
In reference to partitioning and installing the OSs on the new HD, this is not an option they are already installed on the 40.0GB HD. I need to transfer everything to a larger HD drive first.
My strategy is to clone the 40.0GB HD first and fix the partition after the cloning is successful - even fdisk program says that the partition are in the wrong order. I can't afford to lose any data. The system works fine, right now.
In reference to "I always put Windows in a logical drive on an extended partition, along with swap.", Why?
Because MS Windows is a spyware trojan, first and foremost, and then an operating system. With MS Windows XP there is an underlay filesystem, which is built by libraries of MIL code, a map, and a c++ compiler during boot. It is then destroyed before the gui appears. This filesystem is all metadata, keeping track of you. When you use MS Windows XP, there is a file called NTUSER.DAT. There is also a mirror copy of this file, which is inaccessible to the user. NTUSER.DAT is little known to be a registry file, but it is. It contains literally everything about the user. Picture this: Your overclocking, and your system crashes 17 times in a row because of corrupted memory contents. So, you tune it down a little, and it works. But, MS Windows XP keeps track of all those system crashes and profiles you as an overclocker. Guess what MS already knows? If you have expensive hardware, and overclock, you don't buy software. You steal it. So, because NTUSER.DAT keeps track of how you use your machine, and you can also be assumed to be male because women don't overclock, MS knows you are a thief. Well, MS isn't really concerned with small time thieves, but you are being labeled. Verisign Corporation is hard wired into MS Windows XP. Every time your IP makes any transaction, Verisign Corporation records the exact time, in microseconds, source IP, destination IP, and computer finger print, which is hal.dll.
MS Windows error reporting works like this:
You're busy, so you don't report 50 errors in a row. Then, you're not so busy, and you report 1 error. You just reported 51 errors because MS has put a delay loop in the error reporting interface, so people think it takes a really long time to report an error, but it is actually enough time for thousands of error reports to be transmitted. Now, MS can not collect any personally identifying info. That means your full name, birthdate, Social Security number, street address, and telephone number are off limits. MS is legally free to collect every bit of other info on your whole machine. IP address was ruled in federal court to 'NOT' be personally identifying information. Most isp's also have public lists of all their IP's in use, who is using them, with full names, addresses, and telephone numbers. You usually need to opt out of being listed manually, over the telephone, with someone at your isp in order not to be listed.
That is only the tip of the iceburg. If people knew what MS Windows does to them they would smash their hard drive with a sledge hammer and never touch a computer again. But, you can finesse the system. If your smart you can appear to be someone totally different than who you are. Like if you've got a souped up box, and the bios is set really wrong(dork). Or, everytime you use a program, you close it before opening anything else(dork). Or, you use pink baby pigs as wallpaper(girl). Or, whenever you leave your machine, you always leave your browser on the yahoo stock quotes(rich). Or, when you type in the serial number for Windows, you have it totally memorized, and you've practiced entering it, so when the time comes it only takes you one second to enter the whole 25 digit number that you are supposed to be reading as you type(phenom, accountant, clerical worker).
I might write a book on how to fool MS Windows, but if you're going to look at porno, no one is going to be convinced with the pink wallpaper. The point of all this is social engineering. With gobs of data about everyone, corporations can socially engineer people so they consume more goods and services. Taking The Name of Jesus Christ away from television, which has been a long time now, is social engineering to get people to buy more stuff. God satisfies, but sellers want people who consume. So, you get rid of God, and the people can't get enough stuff. That was discovered through similar, but much smaller scale information gathering.
Quote:
Originally Posted by bennettr
Partition Magic is a useful program, but its unforgiving when it is checking a system with partition errors, it wont proceed. Many times I use it just for a quick error check of a mixed HD (OSs) environment. In my opinion, the Linux fdisk and sfdisk are better programs for identifying and fixing problems in a mixed OS system environment, .
Let say I had no problem with the partitions on the 40.0GB HD, what command(s) would you suggest to clone this drive to the 100.0GB HD using minimum space and maximum speed?
There is not one command. You need to do each partition separately.
Quote:
Originally Posted by bennettr
I'm looking forward to your response /Bennett
Well, there it is.
Last edited by AwesomeMachine; 07-08-2006 at 01:30 PM.
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
I'm glad you liked my post
Quote:
Originally Posted by cruiserparts
This thread is extrememly helpful. thanks for the effort.
You are quite welcome
Quote:
Originally Posted by cruiserparts
To make a long story short, I have an old 20gig ide drive that is working fine, but probably getting tired. It is the boot disk and houses the centos linux os. Most of it is empty; only about 7 gigs is being used. I am trying to put everything on an 18.2 gig scsi drive. At first i tried just dd the whole drive /dev/hda to /dev/sdd, this gave me No space left on device errors. Then I partitioned /dev/sdd to have a boot and ext3 partiion. I dd the boot parition and it did clone.
I can't seem to get the /dev/hd2 partion to copy over. I have read this thread and tried several things.
Any ideas on how to solve this problem? Below is my output.
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4k conv=notrunc,noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4096 conv=notrunc,noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
[root@centos ~]# man dd
[root@centos ~]# dd if=/dev/hda2 skip=1 of=/dev/sdd2 seek=1 bs=4096 conv=noerror
dd: writing `/dev/sdd2': No space left on device
4417875+0 records in
4417874+0 records out
Leave notrunc out of the command. If it still won't work, fill the free space with zeros
dd if=/dev/zero > /home/sam/bigfile.file
When dd says no more room left on device
rm /home/sam/bigfile.file
then dd without notrunc Notrunc writes all the free space. Leaving out notrunc compresses zeroed sectors.
I've noticed that a lot of the dd examples here use a block size of 4096 (dd ... bs=4096 ...).
Is this for performance reasons? Does this block size have anything to do with the current system's page size? If I were using a different architecture with a different page size would I use that specific page size instead for the block size?
I'm running Debian stable and want to try upgrading to testing or unstable. If things don't work out, I want to have a snapshot of stable I can roll back to like nothing happened. From this thread I know how to make an image of my root dir. What else do I need to image? I'm guessing the mbr. Is it always just 446 bytes? And what about the loader code, partition table and whatever else is at the start of a disk? In the future I'll test one of the BSDs on this partition so this'll be useful info then as well.
Thanks AwesomeMachine for writing such an extensive tutorial!
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Quote:
Originally Posted by doofus
I've noticed that a lot of the dd examples here use a block size of 4096 (dd ... bs=4096 ...).
Is this for performance reasons? Does this block size have anything to do with the current system's page size? If I were using a different architecture with a different page size would I use that specific page size instead for the block size?
Cool thread BTW!
bs=4096 or 4k seems to work much faster than 2048 or 8192, with modern equipment, copying hard drive partitions. It's called: a sweet spot at bs=4k.
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524
Original Poster
Rep:
Quote:
Originally Posted by dtz
I'm running Debian stable and want to try upgrading to testing or unstable. If things don't work out, I want to have a snapshot of stable I can roll back to like nothing happened. From this thread I know how to make an image of my root dir. What else do I need to image? I'm guessing the mbr. Is it always just 446 bytes? And what about the loader code, partition table and whatever else is at the start of a disk? In the future I'll test one of the BSDs on this partition so this'll be useful info then as well.
Thanks AwesomeMachine for writing such an extensive tutorial!
First of all, You're Welcome. The man and info documentation for dd was so bad compared to how good of a program it is, as I found more and more uses for dd, I gathered together enough info for a tutorial. God really helped me, too.
Ok, for people reading this thousands of years from now, I'm going to make it clear sector 63 starts partition 1. You can do:
fdisk -lu
Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 63 16787924 8393931 83 Linux
/dev/sda2 16787925 147557024 65384550 83 Linux
/dev/sda3 147573090 163959389 8193150 7 HPFS/NTFS
/dev/sda4 163959390 234420479 35230545 f W95 Ext'd (LBA)
/dev/sda5 163959453 168152354 2096451 82 Linux swap / Solaris
/dev/sda6 168152418 234404414 33125998+ 83 Linux
This is one of my partition tables. Yours will be different.. But lets say we're using this one. Then, the root parttition ends at sector 167 879 24. I put spaces in the number so it can be read. Here's the command:
to copy the existing partition, and the first 63 sectors of the drive.
simply reverse the arguments for if and of to put the image file back. /home should always be a separate partition from /, and much bigger than /. That way you can image the / partition to /home, format /, install a new os, and if you get messed up you can boot with:
Okay just to make sure I have this straight since I don't want to destroy the world:
Code:
# fdisk -lu
Disk /dev/hda: 40.0 GB, 40000000000 bytes
255 heads, 63 sectors/track, 4863 cylinders, total 78125000 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/hda1 63 80324 40131 6 FAT16
/dev/hda2 80325 610469 265072+ 82 Linux swap / Solaris
/dev/hda3 610470 11100914 5245222+ 83 Linux
/dev/hda4 11100915 78124094 33511590 83 Linux
hda3 is the root, hda4 is /home.
I'd use
dd if=/dev/hda3 of=/home/me/root.image bs=4096 conv=notrunc,noerror
dd if=/dev/hda of=/home/me/mbr.image count=62
to backup the root and the boot portion respectively.
Correct?
This thread continues to become better and better with time. Cheers, Awesome!
I have another question. Here goes:
I'm trying to clone Windows 9x partitions, and, as I mentioned earlier in the thread, I don't have any large hard disks available so I'm trying to backup my partitions to files instead of to the raw disk itself. The disk I want to clone is 120GB in capacity, but only has 18 or so GB of used space. I've run into a problem, however. I'm using the 'count=x' option to control the size of the files I'm making so that I can split the partitions into pieces that will fit nicely onto DVDs... But instead of truncating the empty space on the partition, dd just obediently copies as many blocks as I tell it to, even if they're empty. I know this is true, because dd didn't stop cloning even after it had passed all the used space on the partition, and all the files I made were exactly the same size--none were smaller. I definetly am not using the 'conv=notrunc' option... So my question is, if I use the 'count=x' option, does dd simply not truncate empty space? And if so, how do I use dd (or is there a better way) to clone my partitions but split the clones up into files of an arbitrary size that I specify? Here's the exact command I'm using:
I am trying to copy my old 40Gb disk to a brand new 80Gb one, so as to keep my old OSes (winXP & Mandrake9.1) and add a brand new Ubuntu.
Here is the process I followed:
1- I partitionned the new disk with Partition Magic (PM) to make sure the partitionning is the same for the copies of old partitions.
Note: I tried to copy partitions with PM, but it fails (error 510 whatever the partition).
2- Then I install Ubuntu on the free space.
3- From Ubuntu, I use dd to copy the old partitions to my new disk.
It seems to have worked properly (good number of blocks read, and files are accessible), but each time dd was ending with a segmentation fault.
Any idea why dd was segmenting ?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
