U3 Howto #1

So here's how you do it: 

First, I run linux.  These commands are all available under Windows, but I don't care.  If you are looking for a Windows how-to, go here first: WUBI

Now once you have a nice sane Ubuntu install here's how you do it:

Supplies:
1 U3 crapware thumb drive (preferably with nothing important on it)
1 bootable iso (Anything will do, but if you want to point and click, HERE YOU GO.)
1 copy of u3-tool  (It builds just fine on Ubuntu 9.10.  Follow the directions.  I'll probably throw it in my PPA for people who are lazy.)

Syntax:
>> A command you can copy and paste
[something you need to fill in for yourself]

Howto:
Insert your thumb drive.

Keep things clean:

>>mkdir customu3
>>cd customu3

Figure out what drive letter it was:

>> sudo fdisk -l

 ...
    Device Boot      Start         End      Blocks   Id  System
/dev/md0p1               1   122095984   488383934   83  Linux

Disk /dev/sdd: 7605 MB, 7605321216 bytes
255 heads, 63 sectors/track, 924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         976     7839698    b  W95 FAT32

If you just plugged it in, chances are it's the last thing in the list.  Also, quickly verify that the size is about right.  My drive is an 8GB drive do 7605MB is about right.  If you have trouble remembering /dev/sdd1, write it down, but beware that it may change in subsequent steps.

If you want the small iso I recommended above (for a first try) you can run this:

>> wget -c http://distro.ibiblio.org/pub/linux/distributions/tinycorelinux/2.x/release/tinycore_2.7.iso

or

>>cp [../myiso.iso] .

Next, find out how big the iso is:

>> ls -l *.iso

-rw-r--r-- 1 myuser myuser 10614784 2009-12-30 22:09 tinycore_2.7.iso

The tinycore 2.7 iso is 10614784 bytes.  I typically add 1000 bytes just to be on the safe side.


Next, make room for the custom iso image:

>>sudo [path to]/u3-tool -p  10615784 /dev/[sdd1]

Ubuntu will remount things for you.  Make sure the drive location didn't change.  Notice that my thumb drive changed size slightly.  Yours should have too or else something didn't work.

>>sudo fdisk -l

...
Disk /dev/sdd: 8055 MB, 8055029248 bytes
255 heads, 63 sectors/track, 979 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         976     7839698    b  W95 FAT32

Now write the new image and you're done.

>> sudo [path to]/u3-tool -l tinycore_2.7.iso /dev/[sdd1]

That's it.  You should have a new virtual cdrom:

>> ls -l /dev/scd*
lrwxrwxrwx 1 root root 3 2009-12-27 15:04 /dev/scd0 -> sr0
lrwxrwxrwx 1 root root 3 2009-12-30 22:16 /dev/scd1 -> sr1

I like to then check that it worked right using a virtual machine.  It's faster than rebooting.  If this works, you have a bootable, virtual CDROM running from your thumb drive.  Congratulations!

>>kvm -cdrom /dev/scd1

I used to hate U3...

U3 is that annoying software that comes pre-installed on your Sandisk thumb drives.  It mounts a virtual CD on your PC when you plug it in and auto-runs a piece of crappy software when all you want is to drag and drop files onto your thumb drive.  I hate that.

But wait - I USED to hate U3. U3 was a closed, windows-only, back-door, auto-run, BS that Sandisk forced me to borrow a Microsoft PC to rip out of my thumb-drive...

I was so wrong. 

I discovered I could hack it. And that CD partition on the U3 drive... looks like a real CD to the computer... and it's bootable! The CD acts like a separate device on an USB hub!

I now have a bootable, 8GB Sandisk Cruzer thumbdrive with all the useful boot CD's I use - all selectable from a little menu. I haven't perfected it yet, but I have half a dozen of them already on there. I want to work on it some more and then post it so all zero of you can easily do this too. :-p

For now, here are some links to the bits and pieces I used (both are cross platform Windows-Linux-Mac-whatever that has mkisofs):
u3-tool - Allows customization of the u3 partition.
UBCD - Has a bunch of boot images pre-installed and easy directions for customizing.


So far, I left all the stuff that came on the UBCD with a few tweaks:

• I upgraded to the latest memtest86+, and DBAN.
• I added SpinRite. (The only piece of software I've bought in a decade.)
• I added the following Live CD's:

1. TinyCore v2.7 - 10MB, runs almost everything useful, boots into ram 5-10 seconds. Enough said.
2. Puppy Linux v4.3.1 - A little bigger, also runs in RAM, less customization needed.
3. Slax 6.3.1 - Most polished Live CD I ever used. Only 200 MB base image. Online customization. Check it out.
4. BackTrack 4 - 1.3GB of evil hacker tools
   "computer security tools". Mwahahaha.
   

Really this was mostly a no-brainer with just the two links above. I had to do some hacking editing of the isolinux configs (which requires a bit more than monkey skills) to get things just right, but it was really pretty easy.

The coolest thing is that the live CD's are read only and behave just like a giant CD. The thumb-drive still shows up as a thumb-drive too. It acts like a separate device to the PC. (An U3 thumb-drive acts like a hub with two devices plugged into it.) This has some really cool implications.

Now I say... U3 ROCKS!

Didier Stevens Tools

I have created a little branch for Didier Stevens programs. He makes some really useful little gadgets like XORSearch. XORSearch parses a file for plain strings, but also trivially scrambled strings such as ROT13 et al. It's neat.

Anyway, you can get the code like this:

bzr branch lp:ds-tools

I have added a makefile that builds all the tools I could get to work on Linux and copies them into the base directory. From there ./ --help should be enough to get you going. Have fun.

Defrag for Linux!!!

So I've heard that we don't have a defrag for linux because we don't need it. I understand that the ext filesystem leaves a litte bit of room at the end of files and does other bits of magic that keep fragmentation down. However, a 90% full volume is going to get fragmented. And there's no good way in linux to undo that, even if you unfill that volume. That's a bummmer. Also, windows power users switching to Linux aren't going to buy the whole "you don't need defrag" bit.

In comes John Dong's python based defrag utility. It uses the rather simple method of creating a copy of the file and then renaming the new file to the old filename. This allows the filesystem to put the entire file in the best possible location and then frees the old file. In theory it will work on any file system.

I have defraged my home directory and it made a noticable difference in performance. (I have had this same volume in continuous use for about 4 years. I have added this utility as a jaunty package to my PPA repository so everyone else can have linux defragmenter goodness without building the package themselves.

PPA:
https://edge.launchpad.net/~brywilharris/+archive/ppa

I plan to clean up the code a bit and then begin work on a GUI. (It's a work in progress so please don't sue me if this trashes your data! However, it worked for me.)

I will post some benchmarks once I've done a little more testing.

bmpfs

This is really cool! File systems you can SEE!

Python code:
#!/usr/bin/python
# create_image.py
import Image
im = Image.new('RGB', (8192,8192),(255,255,255))
im.save('new.bmp', 'BMP')

Put the above in a file and run:
python create_image.py

This creates a blank, white bitmap. Nothing special yet.

Next do this:
sudo losetup /dev/loop0 new.bmp -o 100
sudo mkfs.ext2 /dev/loop0
sudo losetup /dev/loop0 -d

Now look at the bitmap. There's a thin bar along the bottom. That's your file allocation table! You now have an ~190 MB combination disk image-bitmap image!

Now lets stick something in our new "disk image":
mkdir temp
sudo mount -o loop,offset=100,user new.bmp temp
sudo chown 1000:1000 temp # replace 1000 with you user and group id respectively

Now copy something respectably sized into temp. I copied a 32mb video into there.

Unmount the image and you can look at it:

sudo umount temp
eog new.bmp

also try this (imagemagick required):

convert new.bmp new.png

Note the size of the file(s) you put in there. I put a 33799296 byte video in there and the png came out to 33939231 bytes. (Videos are basically not compressible.) Not bad. Since png uses lossless compression, you can convert back and the bitmap still mounts! Try it!

Why are windows executables so big?

So I found a cool search tool by Didier Stevens: http://blog.didierstevens.com/programs/xorsearch/

I goes through a file and searches for a string, and also several common transpositions of that word. It included a windows executable :-(, but it built fine with gcc and appears to work! :-) I will definitely use this.

But, it got me thinking. I could use my new tool! I could look at the difference gross between a Windows a Linux executable with bin2bmp. (Cut me some slack, I like pictures...) Here are the results:

Linux Binary (gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3):




Windows Binary (Borland C++ - Copyright 1999 Inprise Corporation):








To be fair, his windows binary is digitally signed, but that can't be all the difference? Is it statically linked maybe?

Linux Statically Linked Binary (gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3):










Maybe not. I don't have a way to generate a Windows statically linked binary for comparison though... anyway this was mostly just an excuse to play with my new tool. But I am curious.

BTW, Latest version of bin2bmp here: http://sourceforge.net/projects/bin2bmp/

Making pictures of files

Hello world!

I made a little python tool and wanted to share it. It's a neat little tool to visualize binary data in a graphical form. It's really interesting to look at different types of files.


#!/usr/bin/python
#"Copyright 2009 Bryan Harris"
#
#This file is part of bin2bmp.py.
#
# bin2bmp.py is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# bin2bmp.py is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with bin2bmp.py; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

import Image
import os
import sys
import array

if len(sys.argv)<3:
    print "usage:", sys.argv[0], "filename width"
    exit(-1)


try:
    width=int(sys.argv[2])
except:
    print "The second argument, "+sys.argv[2]+", does not appear to be an integer!"


tmpfile = sys.argv[1]
filename = os.path.split(sys.argv[1])[1]
try:
    fileobj = open(tmpfile, mode='rb')
except:
    print "Can't open "+sys.argv[1]+" for some reason."
    exit(-1)

size=os.path.getsize(tmpfile)
buffer=array.array('B',fileobj.read())
buffer.reverse()
print filename+':',size,"bytes"
chunks=size/3
black=(0,0,0)
white=(255,255,255)
im = Image.new("RGB",(width,int(chunks/width)+1),black)
for i in range(chunks):
    x=i%(width)
    y=i/(width)
    RGB=(buffer.pop(),buffer.pop(),buffer.pop())
    #if i<20:print RGB
    im.putpixel((x,y),RGB)
im.save(filename+'.bmp',"BMP")