Running android in ram?

[Guest wbaw]

but a simple hack like not mounting /cache in init.rc would make it end up in RAM which would be much simpler than linking it into /data. Unless of course /cache contains stuff that need to survive a reboot?

In android /cache is only used for downloaded apks from the market & they don't always get removed once the app is installed & it's no longer needed. If we can fix that issue, so that the apks are removed from /cache as soon as they've been installed, then it'd work well, otherwise it risks wasting ram if someone downloads or updates a few large apps between reboots. not mounting cache, linking /cache to data & clearing it on reboot is simple enough.

It is also used for storing recovery logs, but a minimum size of 1.5mb that only gets mounted by the recovery program is fine for that.

Edited February 7, 2011 by wbaw

[Guest wbaw]

Is there much in /system that gets changed? I thought things like the settings get stored somewhere else, otherwise they would have had problems with this too on the build for the HD2 . If there are no changes while Android is running it would be no problem that the RAM isn't permanent. Only problem would be the need of a(ext2 or 3, I'd guess) partition for /system on the SD.

system doesn't normally get changed, unless you use something with root access to change it, uninstall apps with titantium backup, or flash an 'addon' to a rom, or do something like that. it does seem like quite an extreme hack & i'd say the performance benefits are questionable. you'd probably want to use swap & mess about with the android task killer settings too, so that you could get better performance. it quickly gets complicated, for a hack that might not even work well at all.

And what if the device has a spontaneous reboot...?

If dalvik-cache was in ram, it's no big problem to lose it on reboot, it just means that the next boot will take longer when it's regenerated.

Preloading anything into ram shouldn't really provide any performance benefits overall, it'll cost ram & the time it takes to preload it.

It'll get cached in ram if you use it anyway, I think it's better to just let the kernel decide what to do with the ram.

[Guest FelixL]

Okay, thank's kallt for trying it, and thank's for the rest for the detailed explanations.

In unrelated news, this forum needs a "thank's" button :P

Would reduce the spam in most threads.

[Guest kallt_kaffe]

In android /cache is only used for downloaded apks from the market & they don't always get removed once the app is installed & it's no longer needed. If we can fix that issue, so that the apks are removed from /cache as soon as they've been installed, then it'd work well, otherwise it risks wasting ram if someone downloads or updates a few large apps between reboots. not mounting cache, linking /cache to data & clearing it on reboot is simple enough.

It is also used for storing recovery logs, but a minimum size of 1.5mb that only gets mounted by the recovery program is fine for that.

/cache in RAM didn't work well. It seems Market checks the free space before starting to download so it aborts and logcat shows it thinks there is no free space. The ramdisk seems to be dynamic and you can put stuff there until you run out of ram but i suppose diskfree calls doesn't work the normal way with it.

[Guest 0x90]

I dont think the magic is to run the system partition in ram. Most of the framework files and data gets loaded in to ram and stays there anyway. To get a performance increase i think you have to run /data in ram. One approach would be to use a compination of tmpfs and unionfs that stores the changes into the disk but still runs mostly of the ram.

Btw. Have anyone tried formatting the partitions to ext3/ext4? I know that its a quite common approach the get better IO performance on a lot android devices.

[Guest rjm2k]

/cache in RAM didn't work well. It seems Market checks the free space before starting to download so it aborts and logcat shows it thinks there is no free space. The ramdisk seems to be dynamic and you can put stuff there until you run out of ram but i suppose diskfree calls doesn't work the normal way with it.

Googling suggests RAM disks can be fixed size, how were you creating yours?

[Guest 0x90]

Googling suggests RAM disks can be fixed size, how were you creating yours?

Ramdisks can be a fixed size. Tmpfs mounts cant be. So my guess is that he is using tmpfs..

Edited February 8, 2011 by 0x90

[Guest ztedd]

In fact, the dir /dev is a ramdisk. has about 200 M free space but only uses as much ram as needed.

[Guest oh!dougal]

I dont think the magic is to run the system partition in ram. Most of the framework files and data gets loaded in to ram and stays there anyway.

That was the impression I was under.

To get a performance increase i think you have to run /data in ram.

Mostly its just storage, isn't it? But isn't there one thing in there that COULD make a big difference - putting only the dalvik cache into ram.

Booting would take longer (though shutdown and startup scripts could potentially save and restore it, even from sdcard, to avoid rebuilding it.)

But during runtime, having the dalvik cache in a ramdisk rather than nand ought to lead to a noticeable performance improvement, oughtn't it?

AFAIK our /cache is ONLY used during market downloads.

Wbaw did some good work on partitioning to minimise /cache's permanent nand footprint. Apart from the partitioning, he was using a boot-time script to redirect /cache usage to a directory in /data. This does reduce wasted nand space, but does require that boot-script in any/every rom you use with that partitioning scheme.

Edited February 8, 2011 by oh!dougal

[Guest Dr. Mouse]

Hi

I am not an Android expert, but I thought I might share a few experiences I have had with running things from RAM under Linux.

The first time I ever tried this, I was actually exporting a disc from Linux to use to boot a Windows XP machine. All I did for this was run a loop containing 'cat /dev/sdb1 &>/dev/null'. Quite simply, it continually read the HDD I was interested in, thereby keeping it in the cache. It resulted in lightning reads (coming straight from RAM), but normal speed writes (as it was still writing directly to disc). This option I found to be the safest, simplest way.

The next method I have used is to modify the 'init' script, copying the required files to a tmpfs and using them for the system. Gives excellent performance (so long as you have enough spare RAM in the system for it), but it does have a major downside (or upside, depending on your usage): Writes go only to RAM, so all changes are lost on a reboot/shutdown/power loss/crash. In my case, this was for a system I was using as a set-top box, so it didn't matter to me (I had scripts I could run to manually sync files if I needed to save changes), but you would need to sync regularly and/or on shutdown, or find a way to hook into system events to know when to sync.

The next way I tried was for a more mainstream system, so the above approach was not good enough. What I did was to start with the method above, then use unionfs so changes were saved to a HDD partition. This worked quite well, but we are back to writes being the same speed as they would otherwise, and changed data must either be accessed from disc or merged into the ramdisk.

Another way I tried, mainly to fit more into RAM than I had available, was a variant on what I believe is known as Casper. Basically, for the filesystem you wish to load into RAM, you create a "squashfs" image (read-only compressed filesystem), load that into ram, mount it, then use unionfs to disk or RAM, depending on whether you want to keep changes (basically as above, but using squashfs to reduce RAM use). The CPU use from decompression is tiny (at least on a desktop, this was on a single core Atom with 2GB RAM for a Mythbuntu box at the time), and is more than compensated for by the speed of access to RAM. This method also speed boot times, as it lakes less time to load the smaller compressed image than it would the full one.

And here's the final method I've tried (god, I've spent far too much time on this, didn't realise till I started writing this). Use the Linux Software RAID driver. Build a RAID1 (mirror) using the disk storage and a file on tmpfs, with the disk set to be write-mostly (and write-behind if you can stomach data loss in a power cut or crash). This will then start off syncing the disk to RAM, reading "misses" from disk. Once this is synced, all reads and writes will go to RAM, but writes will also go to the disk.

Phew!

I don't know how much can easily be used on a Droid device (as I don't know what modules are available, probably varies between devices), but in this case I would say using SquashFS and UnionFS is probably best (due to limited RAM), but it's something which would need to be experimented with.

Wish you all well with this.

[Guest ninzor]

I wonder how much of boost the dalvik cache would give when it was on ram...

Thank you for your another contribution to the ZTE Blade community, kallt_kaffe!

Edited February 8, 2011 by ninzor

[Guest richardtkemp]

That essay was very informative, thanks Mouse! Also kallt for your quick test.

[Guest Phoenix Silver]

Hi

I am not an Android expert, but I thought I might share a few experiences I have had with running things from RAM under Linux.

The first time I ever tried this, I was actually exporting a disc from Linux to use to boot a Windows XP machine. All I did for this was run a loop containing 'cat /dev/sdb1 &>/dev/null'. Quite simply, it continually read the HDD I was interested in, thereby keeping it in the cache. It resulted in lightning reads (coming straight from RAM), but normal speed writes (as it was still writing directly to disc). This option I found to be the safest, simplest way.

The next method I have used is to modify the 'init' script, copying the required files to a tmpfs and using them for the system. Gives excellent performance (so long as you have enough spare RAM in the system for it), but it does have a major downside (or upside, depending on your usage): Writes go only to RAM, so all changes are lost on a reboot/shutdown/power loss/crash. In my case, this was for a system I was using as a set-top box, so it didn't matter to me (I had scripts I could run to manually sync files if I needed to save changes), but you would need to sync regularly and/or on shutdown, or find a way to hook into system events to know when to sync.

The next way I tried was for a more mainstream system, so the above approach was not good enough. What I did was to start with the method above, then use unionfs so changes were saved to a HDD partition. This worked quite well, but we are back to writes being the same speed as they would otherwise, and changed data must either be accessed from disc or merged into the ramdisk.

Another way I tried, mainly to fit more into RAM than I had available, was a variant on what I believe is known as Casper. Basically, for the filesystem you wish to load into RAM, you create a "squashfs" image (read-only compressed filesystem), load that into ram, mount it, then use unionfs to disk or RAM, depending on whether you want to keep changes (basically as above, but using squashfs to reduce RAM use). The CPU use from decompression is tiny (at least on a desktop, this was on a single core Atom with 2GB RAM for a Mythbuntu box at the time), and is more than compensated for by the speed of access to RAM. This method also speed boot times, as it lakes less time to load the smaller compressed image than it would the full one.

And here's the final method I've tried (god, I've spent far too much time on this, didn't realise till I started writing this). Use the Linux Software RAID driver. Build a RAID1 (mirror) using the disk storage and a file on tmpfs, with the disk set to be write-mostly (and write-behind if you can stomach data loss in a power cut or crash). This will then start off syncing the disk to RAM, reading "misses" from disk. Once this is synced, all reads and writes will go to RAM, but writes will also go to the disk.

Phew!

I don't know how much can easily be used on a Droid device (as I don't know what modules are available, probably varies between devices), but in this case I would say using SquashFS and UnionFS is probably best (due to limited RAM), but it's something which would need to be experimented with.

Wish you all well with this.

i used of casper method some times too

people often use it to run things from a usb stick

[Guest Dr. Mouse]

i used of casper method some times too

people often use it to run things from a usb stick

Yep, I got the idea from Ubuntu, then modified it for my own use (this one was a boot-from-network media PC). Involved some dirty hacking about of the initrd and init scripts, but it was very fast (once it had loaded the image into RAM)

[Guest Phoenix Silver]

Yep, I got the idea from Ubuntu, then modified it for my own use (this one was a boot-from-network media PC). Involved some dirty hacking about of the initrd and init scripts, but it was very fast (once it had loaded the image into RAM)

yes i made the same thing :P

i have 2 or 3 ready systems in a 8 gigs stick (unetbootin is my friend hehe)

anyway with actual computers not a problem 2 or 4 gigs memory is standart

for me i prefer archlinux it's not like ubuntu a take you by the hand system

Edited February 9, 2011 by Phoenix Silver

[Guest Luke2642]

Can anyone see any problems making a TPT zip without a system or cache partitions all together? I've slimmed down and tar.gz'd the system partition image to about 60mb... I'll make the boot partition 60mb bigger add the system image, and mount it in ram like you've been discussing. A few meg in ram for cache & dalvik-cache.

I bet we can get somewhere near to 450mb space in /data for apps! And by keeping system as small as possible, it'll not use much ram or take much time to boot either :P

[Guest IronDoc]

Can anyone see any problems making a TPT zip without a system or cache partitions all together? I've slimmed down and tar.gz'd the system partition image to about 60mb... I'll make the boot partition 60mb bigger add the system image, and mount it in ram like you've been discussing. A few meg in ram for cache & dalvik-cache.

I bet we can get somewhere near to 450mb space in /data for apps! And by keeping system as small as possible, it'll not use much ram or take much time to boot either :P

Cache is needed to download apps. They go in there temporarily.

[Guest Luke2642]

Cache is needed to download apps. They go in there temporarily.

Perhaps I didn't explain that clearly... cache will be mounted in ram too. :P

It's great that ADB works even when you bugger everything up, eh! I should probably practice on the emulator first... hehe!

[Guest IronDoc]

I don't really see the benefit of putting cache there. You'd need to reserve it some space so that would be wasted wouldn't it? Wbaw's method of putting cache in data seems better.

Edited February 10, 2011 by IronDoc

[Guest Luke2642]

I don't really see the benefit of putting cache there. You'd need to reserve it some space so that would be wasted wouldn't it? Wbaw's method of putting cache in data seems better.

Yep, much better. Must investigate how often it gets automatically cleared too... or stick it on the SD card so it doesn't matter.

The blade's 512mb internal storage... does anyone know how it's used? Adding up the hex sizes given in cat /proc/mtd and converting to decimal I still end up ~60mb short... where's the rest? Is it accessible or is it used internally (wear levelling etc) by the flash controller?

[Guest IronDoc]

Yep, much better. Must investigate how often it gets automatically cleared too... or stick it on the SD card so it doesn't matter.

The blade's 512mb internal storage... does anyone know how it's used? Adding up the hex sizes given in cat /proc/mtd and converting to decimal I still end up ~60mb short... where's the rest? Is it accessible or is it used internally (wear levelling etc) by the flash controller?

Someone said there was like a hidden or unnamed portion iirc.

[Guest oh!dougal]

The blade's 512mb internal storage... does anyone know how it's used? Adding up the hex sizes given in cat /proc/mtd and converting to decimal I still end up ~60mb short... where's the rest? Is it accessible or is it used internally (wear levelling etc) by the flash controller?

The boot process to get linux running seems to be quite complex.

There are various partitions created that are not accessible to linux.

Also, there are various different bits of (nothing to do with, not visible to, linux) boot code for handling the plural different boot modes, including the update-flash stuff for the TPT install (which at least needs to mount the fat-formatted sdcard, and then access - or run - the data in /image). All that code has to be somewhere ...

[Guest Phoenix Silver]

The boot process to get linux running seems to be quite complex.

There are various partitions created that are not accessible to linux.

Also, there are various different bits of (nothing to do with, not visible to, linux) boot code for handling the plural different boot modes, including the update-flash stuff for the TPT install (which at least needs to mount the fat-formatted sdcard, and then access - or run - the data in /image). All that code has to be somewhere ...

surely something like the bios in PC

[Guest oh!dougal]

surely something like the bios in PC

Well, a bit like some bits of the BIOS ... but an important difference is that once AndroidLinux is running, it doesn't have the capability of accessing that stuff. Its done its job, and won't be needed until the next boot. I don't think its much like BIOS routines being called by MSDOS, or MSDOS programs ...

[Guest wbaw]

/cache in RAM didn't work well. It seems Market checks the free space before starting to download so it aborts and logcat shows it thinks there is no free space. The ramdisk seems to be dynamic and you can put stuff there until you run out of ram but i suppose diskfree calls doesn't work the normal way with it.

Yeah, I just tried it, didn't work well at all, market wont download anything.

I'm going to experiment with tmpfs next.

[edit] that doesn't work either - if i mount cache /cache on it's own 42m tmpfs, market still fails ... i'm sticking to symlinking to /data/cache for now.

I might have a go at repacking the kernel ramdisk, not mounting /cache & symlinking it in init.rc

Edited February 14, 2011 by wbaw