Comments on: Splitting files with dd

By: Alex Forbes

Alex Forbes — Sat, 01 Aug 2020 07:37:58 +0000

In reply to Christian Rakotondratsima. Hi Christian, thanks for the contribution!

By: Christian Rakotondratsima

Christian Rakotondratsima — Sat, 01 Aug 2020 06:19:15 +0000

Hello Alex,
We used it a lot, thank you.
Here is our variant, we faced some issue sometimes with the previous version.
I added also the merging, or concatenation.
It works for both, splitting and merging, if one uses the same name for the 2 parameters
– the splitting
===========================
#!/bin/bash
#From Al4, Alex https://blog.al4.co.nz/2011/03/esxi-and-splitting-a-file-with-dd/
#Improved by Chad Jay Harrington
#Improved by CRA Miro K.E.
FILE=$1
NEWFILE_STUB=$2
#How big we want the chunks to be in bytes, here is for 256 mb
CHUNKSIZE=$(( 256 * 1024 * 1024 ))
#Block size for dd in bytes
BS=$(( 8 * 1024 ))
echo “block size: “$BS
#Convert CHUNKSIZE to blocks
CHUNKSIZE=$(( $CHUNKSIZE / $BS ))
echo “chunksize: “$CHUNKSIZE
# Skip value for dd, we start at 0
SKIP=0
#Calculate total size of file in blocks
FSIZE=$(stat -c%s “$1”)
echo “file size: “$FSIZE
SIZE=$(( $FSIZE / $BS ))
#Loop counter for file name
i=0
echo “Using chunks of “$CHUNKSIZE” blocks”
echo “Size is “$FSIZE” bytes = “$SIZE” blocks”
while [ $SKIP -le $SIZE ]
do
NEWFILE=$(printf “$NEWFILE_STUB.part%05d” $i)
NEWFILE=basename $NEWFILE
i=$(( $i + 1 ))
echo “Creating file “$NEWFILE” starting after block “$SKIP””
dd if=$FILE of=$NEWFILE bs=”$BS” count=”$CHUNKSIZE” skip=$SKIP
SKIP=$(( $SKIP + $CHUNKSIZE ))
done
===========================
– the merging
===========================
#!/bin/bash
SPLITTED=$1
MERGED=$2
FLIST=$(ls -tr1 $SPLITTED*)
OUTPUT=$MERGED
for F in $FLIST
do
cat $F >> $OUTPUT
done
===========================
All the best.
Christian

By: Chad Jay Harrington

Chad Jay Harrington — Tue, 20 Sep 2016 15:38:55 +0000

In case of premature errors where all the pieces are not generated, perhaps we could provide an option that identifies that some of the parts are there already, and if we would like to regenerate them all or just the missing parts. and that way we don’t waste any time if not necessary

By: Chad Jay Harrington

Chad Jay Harrington — Tue, 20 Sep 2016 15:31:08 +0000

In reply to Chad Jay Harrington.

Here’s what I did to resolve my concerns… Also, in your post your files indicate that you ran another program as well (tgz)….

I also had problems with the printf of the FILE name putting quotes in the filenames at the beginning and end of every part…

I am trying to get an image of a 240GB SS HD and a 1 TB SS HD, split into pieces so I can burn the pieces onto ~25GB Bluray discs so i can do the Autopsy later when I have more money.

Here’s my code variant:

#!/bin/bash
FILE=$1
NEWFILE_STUB=$2

#How big we want the chunks to be in bytes
CHUNKSIZE=$(( 4096 * 1024 * 1024 ))
#Block size for dd in bytes
BS=$(( 8 * 1024 ))
#Convert CHUNKSIZE to blocks
CHUNKSIZE=$(( $CHUNKSIZE / $BS ))
# Skip value for dd, we start at 0
SKIP=0
#Calculate total size of file in blocks
FSIZE=`stat -c%s “$1″`
SIZE=$(( $FSIZE / $BS ))
#Loop counter for file name
i=0
echo “Using chunks of “$CHUNKSIZE” blocks”
echo “Size is “$FSIZE” bytes = “$SIZE” blocks”
while [ $SKIP -le $SIZE ]
do
NEWFILE=$(printf “$NEWFILE_STUB.part%03d” $i)
NEWFILE=basename $NEWFILE
i=$(( $i + 1 ))
echo “Creating file “$NEWFILE” starting after block “$SKIP””
dd if=$FILE of=$NEWFILE bs=”$BS” count=”$CHUNKSIZE” skip=$SKIP
SKIP=$(( $SKIP + $CHUNKSIZE ))
done

By: Chad Jay Harrington

Chad Jay Harrington — Tue, 20 Sep 2016 15:09:07 +0000

Almost feel like you should separate the FILE and NEWFILE into two separate variables… had a little trouble with that one

By: Alex

Alex — Sat, 06 Sep 2014 09:24:22 +0000

In reply to fuchur. This makes the order correct when you ls and sort by filename, good addition thanks :-)

By: fuchur

fuchur — Fri, 05 Sep 2014 20:19:50 +0000

that script really helped me – thanks!

Just a little improvement for the split:
NEWFILE=$(printf “$FILE.part%03d” $i)

By: Henry

Henry — Thu, 30 Jan 2014 23:44:47 +0000

Very good. This helped me put my VM backups on Blu-Ray DVDs.

By: Alex

Alex — Wed, 25 Jan 2012 12:12:37 +0000

In reply to anonymoose. Glad I could help!

By: anonymoose

anonymoose — Tue, 24 Jan 2012 18:20:24 +0000

Cheers for the tip. This came up on a search and was useful for recovering some simulations that I had accidentally appended to, rather than overwritten