Defrag for Linux!!!

So I've heard that we don't have a defrag for linux because we don't need it. I understand that the ext filesystem leaves a litte bit of room at the end of files and does other bits of magic that keep fragmentation down. However, a 90% full volume is going to get fragmented. And there's no good way in linux to undo that, even if you unfill that volume. That's a bummmer. Also, windows power users switching to Linux aren't going to buy the whole "you don't need defrag" bit.

In comes John Dong's python based defrag utility. It uses the rather simple method of creating a copy of the file and then renaming the new file to the old filename. This allows the filesystem to put the entire file in the best possible location and then frees the old file. In theory it will work on any file system.

I have defraged my home directory and it made a noticable difference in performance. (I have had this same volume in continuous use for about 4 years. I have added this utility as a jaunty package to my PPA repository so everyone else can have linux defragmenter goodness without building the package themselves.

PPA:
https://edge.launchpad.net/~brywilharris/+archive/ppa

I plan to clean up the code a bit and then begin work on a GUI. (It's a work in progress so please don't sue me if this trashes your data! However, it worked for me.)

I will post some benchmarks once I've done a little more testing.

bmpfs

This is really cool! File systems you can SEE!

Python code:
#!/usr/bin/python
# create_image.py
import Image
im = Image.new('RGB', (8192,8192),(255,255,255))
im.save('new.bmp', 'BMP')

Put the above in a file and run:
python create_image.py

This creates a blank, white bitmap. Nothing special yet.

Next do this:
sudo losetup /dev/loop0 new.bmp -o 100
sudo mkfs.ext2 /dev/loop0
sudo losetup /dev/loop0 -d

Now look at the bitmap. There's a thin bar along the bottom. That's your file allocation table! You now have an ~190 MB combination disk image-bitmap image!

Now lets stick something in our new "disk image":
mkdir temp
sudo mount -o loop,offset=100,user new.bmp temp
sudo chown 1000:1000 temp # replace 1000 with you user and group id respectively

Now copy something respectably sized into temp. I copied a 32mb video into there.

Unmount the image and you can look at it:

sudo umount temp
eog new.bmp

also try this (imagemagick required):

convert new.bmp new.png

Note the size of the file(s) you put in there. I put a 33799296 byte video in there and the png came out to 33939231 bytes. (Videos are basically not compressible.) Not bad. Since png uses lossless compression, you can convert back and the bitmap still mounts! Try it!

Why are windows executables so big?

So I found a cool search tool by Didier Stevens: http://blog.didierstevens.com/programs/xorsearch/

I goes through a file and searches for a string, and also several common transpositions of that word. It included a windows executable :-(, but it built fine with gcc and appears to work! :-) I will definitely use this.

But, it got me thinking. I could use my new tool! I could look at the difference gross between a Windows a Linux executable with bin2bmp. (Cut me some slack, I like pictures...) Here are the results:

Linux Binary (gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3):




Windows Binary (Borland C++ - Copyright 1999 Inprise Corporation):








To be fair, his windows binary is digitally signed, but that can't be all the difference? Is it statically linked maybe?

Linux Statically Linked Binary (gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3):










Maybe not. I don't have a way to generate a Windows statically linked binary for comparison though... anyway this was mostly just an excuse to play with my new tool. But I am curious.

BTW, Latest version of bin2bmp here: http://sourceforge.net/projects/bin2bmp/

Making pictures of files

Hello world!

I made a little python tool and wanted to share it. It's a neat little tool to visualize binary data in a graphical form. It's really interesting to look at different types of files.


#!/usr/bin/python
#"Copyright 2009 Bryan Harris"
#
#This file is part of bin2bmp.py.
#
# bin2bmp.py is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# bin2bmp.py is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with bin2bmp.py; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

import Image
import os
import sys
import array

if len(sys.argv)<3:
    print "usage:", sys.argv[0], "filename width"
    exit(-1)


try:
    width=int(sys.argv[2])
except:
    print "The second argument, "+sys.argv[2]+", does not appear to be an integer!"


tmpfile = sys.argv[1]
filename = os.path.split(sys.argv[1])[1]
try:
    fileobj = open(tmpfile, mode='rb')
except:
    print "Can't open "+sys.argv[1]+" for some reason."
    exit(-1)

size=os.path.getsize(tmpfile)
buffer=array.array('B',fileobj.read())
buffer.reverse()
print filename+':',size,"bytes"
chunks=size/3
black=(0,0,0)
white=(255,255,255)
im = Image.new("RGB",(width,int(chunks/width)+1),black)
for i in range(chunks):
    x=i%(width)
    y=i/(width)
    RGB=(buffer.pop(),buffer.pop(),buffer.pop())
    #if i<20:print RGB
    im.putpixel((x,y),RGB)
im.save(filename+'.bmp',"BMP")