i must say i found this thread quite useful
but i have a problem as i'm trying to copy an image of a disk to a fat partition.. which, as you may know, has a 4gb file size limitation. and the partition that i want imaged is sth like 17,9gb big.. although only 4,3 is actual data.

so is there a way to write the image to, let's say... a few different files?
or am i forced to use a fs that has no such limitations (at least not that low )?

Dd will copy the format with the data. I'm not sure if dd will change the partition type word, which is 2 bytes in each partition table entry, if you copy to a partition of a different file system type. So, if you copy an NTFS partition to a FAT volume you'll wind up with a NTFS format and a type 06 partition, probably. These don't match, and the file system won't end up FAT, but the partition table will probably say it is FAT.

See this link for partition types:

http://www.win.tue.nl/~aeb/partition...n_types-1.html

1 members found this post helpful.

To make the backup, gzip it and split it in 4GB files:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

To restore this backup:
cat /mnt/hdc1/backup.gz.* | gzip -dc | dd of=/dev/hda1

FDHIN, or anyone, I noticed you have both /mnt/<drive> and /dev/<anotherdrive>. This was confusing me a lot when trying to run DD on a live Knoppix 4.02 live cd.
I'm working with a terminal window open to the DDing drives on my live cd booted from external usb/dvd.

I end up right clicking the shortcuts to mount/unmount or just changing mnt to dev in window, vice versa, etc, and eventually the drives see each other.

Is there a thread explaining mounting, unmounting, permissions etc? In the case of using DD?

And Awesome, thanks for the howto wish I saw it weeks ago.

I don't know what this post pertains to, but it is wrong. The way it is written, dd would write to the screen, and even if it did write to a drive, it still won't work right. You cannot use the "cat" command for anything but text. It doesn't work with bitstreams. Fortunately, nothing bad will happen even if you do enter these commands. The first one will fail. So, the second one has to fail.

1 members found this post helpful.

Dd uses device files. It doesn't matter if they are mounted or not. You don't have to worry about mount with dd. Dd will make a bitstream duplicate regardless of whether or not the file system is mounted. I usually do not mount a file system when using dd. If it happens to be mounted already, so be it. But I don't intentionally mount a device's file system in the linux file tree just to use dd.

I have a question for you. What are you doing with dd, and a Knoppix CD in a usb drive. You can't do an installation to a hard drive that way. You cannot dd the CD to the hard drive and have it be a Knoppix hard drive after that. There is a way to install knoppix to a hard drive, but dd isn't it.

Your welcome for the tutorial.

1 members found this post helpful.

1. Interesting about the dev/mnt info looking at my notes I am sure I used /dev/somedriveorfile to dd to /dev/anotherdriveorfile, but would only work when I used the combination of mnt and dev. Anyway I am learning. I'm use dmesg or just seeing which item lists its files, then I go from there.

2. Not installing Knoppix, just using a livecd do do the drive xfer operations, mark them as read only, permissions, mount umount or not. I'll learn the cli for all this eventually, but I am also able to check here and elswhere on the web without running to my other computers.

3. I am backing up an irreplacable (softwarewise) seagate drive, 8.6gig from 1999 in my HDTV broadcast simulator. This generates an off the air signal from large HDTV bitstream files to attach to an HDTV tuner/tv. It uses w98. Not even SE I think. I already cloned to a spare drive, it works ok, but I want to dd everything to cdr/dvd including boot info for a permanent off site backup.
After mastering this, I want to use DD fluently to back up to safe media, xp, mac, old dos, etc. Also my HDTV off air for free recordings are huge, 20-30gigs for 1 olympic broadcast day in 1080i for example. So need to learn splitting them to files for xfer to DVD, just split .nuv data files... DVD for tv burning is another matter that I have solutions for.

I am sick of ghost, Acronis and similar costing me a lot so far, and peeing their marks all over my drives. I have pre Norton Ghost also that was the bees knees for years, but it is limited in todays world. Acronis would not do my fedora core 3 mythtv system either. So its Linux to stay. I like cli anyway, being a dos veteren.
Also all this copyright and DRM b.s. is so fragile to back up or transfer around legally on my own stuff.

Awesome, just noticed your reply to fdhin regarding cat command...This is what I have been using to put split files back to my drives. They are not text obviously, so what do I use to put my split files of drive imaged by dd split gzip back together again? This may be answer to my problem.

Nope, you are plain wrong on all accounts. Now instead of talking about it, try this example out. It is proof of concept so to say.

Make two files with random bytes in them:
dd if=/dev/urandom of=~/sourcefile bs=200M count=1
dd if=/dev/urandom of=~/targetfile bs=200M count=1

Load the loopback kernel driver and associate each file with a loop device. They will from here on function as a block device:
modprobe loop
losetup /dev/loop0 ~/sourcefile
losetup /dev/loop1 ~/targetfile

make an ext2 filesystem on loop0 (which is the sourcefile):
mke2fs /dev/loop0

Make a mountpoint and mount the sourcefile:
mkdir /mnt/sourcemountpoint
mount -t ext2 /dev/loop0 /mnt/sourcemountpoint

Now copy 200MB of assorted crap to /mnt/sourcemountpoint. I copied some MP3 files.

Make the backup with the commands I suggested:
dd if=/dev/loop0 | gzip -c | split -b 1m - /tmp/testbackup.gz
You will have a lot of 1MB testbackup.gz files in /tmp now.

Now put these on loop1 device (the target file)
cat /tmp/testbackup.gz* | gzip -dc | dd of=/dev/loop1

Let's check now if it worked:
mkdir /mnt/targetmountpoint
mount -t ext2 /dev/loop1 /mnt/targetmountpoint

So.. I can see and play MP3 files now from the /mnt/targetmountpoint, so the commands work. Both of them.

However there was one little flaw in the first set of commands I gave. split does not put a . in the file, so it is cat /tmp/testbackup.gz* and not cat /tmp/testbackup.gz.*

Thus the commands will be:
dd if=/dev/blockdevice | gzip -c | split -b 1m - /tmp/backup.gz

And to restore this backup
cat /tmp/backup.gz* | gzip -dc | dd of=/dev/blockdevice


And finally, to clean up after this example if you try it:
umount /dev/loop0
umount /dev/loop1
losetup -d /dev/loop0
losetup -d /dev/loop1
rm /tmp/testbackup.gz*
rm ~/sourcefile ~/targetfile
rmdir /mnt/targetmountpoint/
rmdir /mnt/sourcemountpoint/

This is quite different than you original post. If you go through all that trouble it will work, but why would anyone ever do that? You don't have to mount a file as loopback device to use dd. Dd will make a file you can later mount as loop device, and people have been doing backups with dd since the 1970's. If you are going to backup you are never going to do it on the same drive. Furthermore, in the other person's post, he wasn't trying to do a backup. If he was trying to do a backup he would have just made a file in any file system type. It wouldn't matter.

I know you think you are really special, and you are. But, read my post again and learn how to use dd. There are all sorts of weird, useless things you can do with anything. I'm trying to teach constructive methods to solve real world problems with dd. I compliment you on your astute ability. It is impressive that you figured out that command. But, it was totally impertinent to what you were replying to. You are needlessly confusing people in an otherwise quality thread. I don't mean to get down on you, but if you go find me some good info on dd, I'll edit the post and put in it what you turn me on to about dd. But, post a link. Do not post the whole web page in a reply. I will look at the link and edit the post. Then I will post a post thanking you. Would that be ok?

What I posted above is adequate proof that this will work:

To make the backup, gzip it and split it in 4GB files:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

To restore this backup:
cat /mnt/hdc1/backup.gz* | gzip -dc | dd of=/dev/hda1

I posted the above as a method to prove it, without anyone having to risk any data in the proces.

I'll happily analyze it now, so that everyone understands why it works. So, the first command:
dd if=/dev/hda1 | gzip -c | split -b 4096m - /mnt/hdc1/backup.gz

AwesomeMachine writes "dd would write to the screen". Actually not so. dd will write to stdout which as default happens to be the screen. We can pipe the stdout somewhere else with the | character, so that the dd output is sent on to the gzip commands stdin instead.
So what happens here is that block for block dd is reading hda1, passing it on to gzip which is set to pipe the compressed blocks to stdout instead of a file with the -c switch. From gzip the compressed blocks are passed from its stdout to the stdin of the split command, again with the | character. Split does just what its name implies: It splits files up into smaller fragments. With split -b we determine the size of the chunks and set them to be 4096MB, and the second - (minus) tells split that it should get its input from stdin. If we did not put that then split would start looking for a file called /mnt/hdc1/backup.gz and split that, which is not what we want here.

If you some day desire to restore this backup on hda1, then you can use this command:
cat /mnt/hdc1/backup.gz* | gzip -dc | dd of=/dev/hda1

This command takes the backup.gz files in the correct order and pass them on to gzip. The gzip command is set to decompress with -d and again set to pipe to stdout. The stdout is piped on to the dd command, which then uses hda1 as outfile and writes the contents.

AwesomeMachine writes "You cannot use the "cat" command for anything but text. It doesn't work with bitstreams." - In fact cat can do that. Try to cat any character device in /dev/, and cat also works for the above.


Anyway, I dont want to discuss anymore whether this works or not. I use it, and it works.

So awesome, before I confuse myself with the cat process discussed above, I would like the equivalent of using cat but using dd to recombine split files. Do I do dd if=splitfile.gz* of=recombinedfilename to that effect?
I will go back and read your tutorial now and if I find answer, I'll come back here soon and edit/delete this post.

"If your only tool is a hammer, everything looks like a nail"

Yes, eventually you CAN do anything with DD. I guess that using NETCAT and DD in a smart way, you can even share files over the network. BUT... NFS was invented for that.

DD is an awesome tool, and awesomemachine did the best job I ever saw to describe it.

However, DD is not the only tool in Linux.

I am still convinced that if you do want to copy N bits from here to there in the broadest sense then use DD. If it is more complicated, you should look for a more appropriate solution. If you want to split into multiple volumes, compress, and copy to a different file system on a different sized medium, it is insane to insist to use DD. Or you do it just for fun, but then it is no use to confuse other people in this thread.

So, if you want to split into multiple volumes, compress, and copy to a different file system on a different sized medium, use the appropriate tool. TAR or STAR is one of them.

jlinkels

Program names in CAPS are actually lowercase names.

So can you answer my question I asked at 5:34pm today? how to use DD to recombine files split using dd and pipe to split. Not using cat. I am not looking for shortcuts. I want to learn all the proper dd stuff. thanks

Is there anyone out there?
repeat:So can you answer my question I asked at 5:34pm 2 days ago? how to use DD to recombine files split using dd and pipe to split. Not using cat. I am not looking for shortcuts. I want to learn all the proper dd stuff. thanks
clip from the past:So can you answer my question I asked at 5:34pm today? how to use DD to recombine files split using dd and pipe to split. Not using cat. I am not looking for shortcuts. I want to learn all the proper dd stuff. thanks