[dev] UBIFS?

[Guest wbaw]

http://elinux.org/File_Systems#UBIFS_vs._YAFFS2_comparisons_for_2.6.31.1

http://en.wikipedia.org/wiki/UBIFS

It's an alternative file system (to yaffs2, currently used) for raw nand memory. It supports write caching, so it could mean over a 10x increase in write performance. It also supports on-the-fly compression.

So, how do we make it work?

Edited July 28, 2011 by wbaw

[Guest wbaw]

What I've found so far...

CONFIG_MTD_UBI=y

CONFIG_MTD_UBI_WL_THRESHOLD=4096

CONFIG_MTD_UBI_BEB_RESERVE=1

CONFIG_UBIFS_FS=y

CONFIG_UBIFS_FS_LZO=y

CONFIG_UBIFS_FS_ZLIB=y

is needed in the kernel config. kernel command line may need to be changed. need tools to create ubifs partition. need to edit init.rc to mount userdata as ubifs. need clockworkmod to support it too.

[Guest hedgepigdaniel]

how does write caching make it so much faster even with large file sizes? I would only have expected it to help significantly with small writes.

[Guest wbaw]

It uses less cpu time for writes & the write caching should make sure that it's always sending data at full speed to the nand, so it performs better than if it sends data in smaller chunks. The benchmark may be a bit flawed too.

These look like better benchmarks: http://elinux.org/Flash_Filesystem_Benchmarks_2.6.38

It should have some significant performance increase for writes, maybe not 10x, but noticeable.

[Guest wbaw]

Actually, looks like the results for large files are all about cpu speed, at 1ghz yaffs2 is a bit faster, at 200mhz ubifs is faster. The writeback cache should help for small writes though, which is what our phones probably do more of.

It looks like it doesn't use space as efficiently as yaffs2 either, without compression.

It makes more sense than doing this anyway: http://events.linuxfoundation.org/slides/2011/abs/abs2011_chen.pdf

Edited July 28, 2011 by wbaw

[Guest Tom G]

Any performance increase is always good, but have you thought about how much gets written to the yaffs file systems? Sure there is a heap written to /data at first boot when everything is cached, but that should only happen once and after that there isn't really a lot. Most data is written to SD card (assuming an SD card is available), so efficient use of space may be more beneficial for internal storage. There would probably be more benefit in replacing the SD card FAT filesystem with something more efficient (and that is probably already supported considering internal SD cards are formatted as ext3 for some partitions).

If one of the desired results is faster boot time you might be interested in this. http://review.cyanogenmod.com/6886

I'm not sure if that is fixing a slow boot issue only in CM7 or if all roms have the same issue.

[Guest wbaw]

I'm thinking there are a few small writes to /data to keep databases up to date & iirc sqlite doesn't have very efficient writes, nand has poor performance for small random writes, so write caching could speed that up a lot. The lower cpu use bit is interesting too.

FAT is actually a good choice for the main sd card partition, it's compatible with everything & no permissions to worry about.

I also think ext2 would be better than ext3 or ext4 for the sd-ext partition.

Edited July 28, 2011 by wbaw

[Guest fonix232]

UbiFS may be a good solution as we do not work with huge files, but I think ext4 (with noatime and disabled journaling) would be better. Although both method requires such high modification of the boot process (not kernel but Android-side), that I would think twice before beginning it. RFS and ext4 has many similarities, that's why it was easy to do on the Galaxy series phones, but this is a bit harder.

For UbiFS:

- rewrite the mount part totally to support Ubi partitions

- reformat the internal NAND to MTD partitions containing Ubi support (it's possible to add an MTD yaffs partition as a UbiFS one)

For Ext4:

- rewrite the mount part (just replace yaffs mount with ext4)

- reformat internal NAND to ext4 (I guess it would be easier than adding every partition to a ubi controller, and managing them).

Fortunately, if we can reformat the device, and MAYBE rewrite the bootloader (u-boot is promising for this), THEN, maybe we can handle partition layout by kernel and such, every ROM could handle it's own NAND layout, WITHOUT TPT.

[Guest Paul]

I looked into ubifs, the biggest problem I could see was not having a mkubifs binary to try. :)

P

[Guest brickornot]

So do we need newest kernel source to compile it into it ?

[Guest wbaw]

UbiFS may be a good solution as we do not work with huge files, but I think ext4 (with noatime and disabled journaling) would be better. Although both method requires such high modification of the boot process (not kernel but Android-side), that I would think twice before beginning it. RFS and ext4 has many similarities, that's why it was easy to do on the Galaxy series phones, but this is a bit harder.

For UbiFS:

- rewrite the mount part totally to support Ubi partitions

- reformat the internal NAND to MTD partitions containing Ubi support (it's possible to add an MTD yaffs partition as a UbiFS one)

For Ext4:

- rewrite the mount part (just replace yaffs mount with ext4)

- reformat internal NAND to ext4 (I guess it would be easier than adding every partition to a ubi controller, and managing them).

Fortunately, if we can reformat the device, and MAYBE rewrite the bootloader (u-boot is promising for this), THEN, maybe we can handle partition layout by kernel and such, every ROM could handle it's own NAND layout, WITHOUT TPT.

We can't use ext4 on it's own on raw nand because it doesn't provide wear levelling, it'd have to be a ext4 .img file loop mounted on top of yaffs2, which is stupid. You're better off with ext2 if you're going to do that, ext2 is faster, they already tested in that pdf I linked to above. Anyway loop mounting is a bit of a dumb idea, makes it needlessly complicated & inefficient.

Such high modification of the boot process? It looks like one line needs changing in init.rc & we need to make sure that ubifs is compiled into the kernel. As Paul says it looks like the biggest problem is not having the binaries to create the file system, but those could be easily compiled. The partitions are already mtd, I just want to try formatting them as ubifs instead of yaffs2. Clockworkmod would need hacking a bit too.

Edited July 28, 2011 by wbaw

[Guest wbaw]

So do we need newest kernel source to compile it into it ?

It could be compiled into the current .32 kernel source used by cyanogenmod, but it probably can't be used on any roms based on the leaked .35 kernel until there is source code for that.

It might not even be faster, or only be a bit faster in certain situations, I think it might be worth testing for /data though.

[Guest wbaw]

If one of the desired results is faster boot time you might be interested in this. http://review.cyanogenmod.com/6886

I'm not sure if that is fixing a slow boot issue only in CM7 or if all roms have the same issue.

I think that's just a CM7 issue, it does seem to boot slower than other roms. Better get that patch merged if it works properly.

I'm not too concerned about boot time, I just think it might greatly speed up small NAND writes while in use if we use writeback caching on /data, which isn't possible on yaffs2. LZO compression might be beneficial too. It'll probably be a similar speed to yaffs2 for large sequential writes, but the writeback caching should help to turn the small random writes into larger sequential writes done in the background, so they should be much faster.

It should work better & be less hassle than the 'god mode' rubbish I've seen on other roms for other devices, which use the method described in that pdf I linked to above, loop mount a ext2 .img file on top of yaffs. The only performance improvement that seems to give is writeback caching & stupid quadrant scores as a result of that. Then theres the problem of a fixed size .img file, a bit smaller than the yaffs2 partition & the overhead of running 2 filesystems on top of each other.

Edited July 29, 2011 by wbaw