linux tools coming to MCR

[Guest freeasabeer]

(for reference, topic previous name was: screen coming to MCR and have also merge the "mc coming to MCR" topic)

8-feb: Major update in the way these additional toys are handled (now with ipkg feed).

I will now publish the linux tools packaged as ipk files and one will have to use the opkg tool to install / uninstall / udate these ipk packages.

For reference, opkg is a lightweight package management system based on Ipkg. Ok, but what is ipkg ? Easy: ipkg re-implements for embedded systems the functionality of Debian tools dpkg, apt, and apt-get. For example, it uses much less disk space for the package metadata.

Believe me, this will make your life as end-user quiet easier and as a side effect it will make my life as developer & packager quite easier as well B)

So, having said that; the first action is to have opkg installed on your phone. To achieve this, download this specific opkg bootstrap opkg_bootstrap_v2.tar.bz2.zip.

Unzip it to your SD card, open a shell with adb and run:

cd /

tar xjf /sdcard/opkg-bootstrap-v2.tar.bz2

/data/xbin/opkg install /sdcard/opkg_0.1.7+svn519-2_arm.ipk

You now have installed opkg as an ipk package. This means: 1) you are now able to easily install other ipk 2) opkg can now also be easily updated if required ! One of the benefitis of opkg is the ability to set-up an internet feed so as to be able to update or install packages while being online. Edit opkg configuration file /data/local/opkg.conf in this way:

src/gz common http://freeasabeer.byethost33.com/android/feed/common

src/gz mcr http://freeasabeer.byethost33.com/android/feed/mcr

#src/gz cyanogen http://freeasabeer.byethost33.com/android/feed/cyanogen


dest root /


#option verbose-wget

#

# Proxy support:

#

#option http_proxy http://localhost:5865

#option ftp_proxy http://proxy.tld:3128

#option proxy_username <username>

#option proxy_password <password>

Note: If you're using Paul's MCR, then those settings are OK. If you're using one of the 2.1 cyanogen based ROM, then comment out the second line and uncomment the third one. You are now ready to update the package list and to install from the feed:

 opkg update

Note 2: If you already had opkg installed from the previous bootstrap, then you will have to do the following (as there is a bug in former opkg package that prevent it to actually find the configuration file):

opkg -f /data/local/opkg.conf update

opkg install opkg

Some quick instructions on how to use opkg: Always run "opkg update" before to be sure to get the latest updates ;) Install the rsync package:

opkg install rsync

Want to remove the package, easy:

opkg remove rsync

Want to list installed packages:

opkg list_installed

Want to list files belonging to a given package (e.g. rsync):

opkg files rsync

You don't need to reboot your phone in recovery mode, neither to just reboot it. Just install things live.

PS: you don't need to uninstall or remove any of the previously installed tools: they will (should) be nicely overwritten.

Forewords:

these linux tools are mainly intended for geeks that like top play with the internals of their Hero from a shell (either adb shell, ssh connection, or a terminal application). Regular users won't find any benefit by installing these tools: they won't make their hero faster or work better, they will just increase the ROM memory consumption.

1. mc (Midnight Commander)

If you don't know what mc is then you most probably don't need it. But you may be interested to discover what it is.

For the geeks that do want it to easily browse the phone file system from a ssh or adb shell connection, I've done a uclibc statically linked binary of mc that you can install with opkg.

From what I've seen, it mostly works ok from a linux xterm connected with adb shell: browsing files is ok and I've not tested any advanced features of mc. I have no idea if it is usable from a windows cmd terminal or a cygwin one.

Once installed, open a shell on your phone and start mc this way: mc

2. GNU screen

As a midnight commander companion, we can now enjoy screen !

Don't know what screen is ? Check Wikipedia.

Basically screen allow to open several shell connections in parallel from one physical connection to the phone (either adb shell or ssh) and to switch from one to the other one at will. It also allow to close the physical connection without losing the ongoing shell connections so that you can resume them once you have restarted the physical connection.

An interesting side effect I've noticed is that vi works great within a screen shell where as is was unusable from a direct shell !

I have however to raise a warning here: in order to get screen to work I had to setup a system/etc/passwd file with the solely purpose to declare the root user and I've done it with an empty password. So I'm not sure yet of the security implication that this could have on the phone... On a regular unix system the security breach is obvious but here on an anyway rooted phone I'm not so sure. So, you've been warned.

Once installed, you can start screen with: screen and resume an ongoing session with screen -r

3. tree

Tree is a recursive directory listing program that produces a depth indented listing of files, which is colorized ala dircolors if the LS_COLORS environment variable is set and output is to tty.

4. rsync

Due to popular demand, here is also rsync...

Note: installing rsync will also install a new version of dropbear.

Some hints on how to use rsync with the Hero:

1) From the Hero to a remote host

In this scenario, you run rsync on the hero itself and connect to a remote host.

There are no oddities here, just use rsync as you normally would.

2) From a linux/mac host to a remote Hero

In this scenario, you rsync on linux/mac host and connect to a remote hero.

In this case, the rsync invocation is a bit tricky, for example:

rsync -av -e 'ssh -p 2222' --rsync-path=/data/xbin/rsync <some local files/dir> root@<YOU_HERO_IP>:/data/tmp

5. busybox

This is an updated busybox as compared to what Paul currently ships with MCR (even current betas). This one do include my ip resolv tweak !

6. wget

Don't ask for wget, it is already provided by busybox :)

7. htop

First release for htop... However I don't really see how to use the Fx keys... So pretty useless ?

8. zip

The famous zip binary is now available !

Enjoy !

Edited March 11, 2010 by freeasabeer

[Guest freeasabeer]

Changelog:

5-feb-2010: Major update

- Make use of opkg tool and of the .ipk package format to handle the tools

3-feb-2010: screen 0.3

- Regression fix in screen wrapper script

3-feb-2010: busybox 0.3

- Bug fix for the update script of busybox 1.16.0

29-jan-2010: busybox 0.2

- busybox 1.16.0 packged for modaco with the ip name resol fix.

29-jan-2010: zip 0.1

- first release for the famous zip binary

25-jan-2010: htop 0.1

- first release for htop. (Within ConnectBot, the Fn key is emulated with: Trackball + n)

21-jan-2010: screen 0.2

- made it more MCR friendly: recompiled to use /data/screen as its "data" folder where the global screenrc file can be found

- addition of the uft8encodings in /data/screen

- uses now /data/local/tmp as the temporary folder instead of /data/tmp

18-jan-2010: busybox 0.1

Modifications to Paul's busybox

- crond: spool folder is set to /data/cron by default

- ls color enabled by default

- fixed the ip name resolve bug. Requires a valid /etc/resolv.conf file.

The update package also make /etc/resolv.conf a symlink to /data/local/resolv.conf and provides a valid /data/local/resolv.conf file (Google DNS).

A simple script that reads net.dns1 property and updates /data/local/resolv.conf accordingly is also provided in /data/xbin/set_resolconf.

15-jan-2010: rsync 0.3

- fixed issue in the update-script that prevented installation in the case rsync was already installed

- lesson to learn: don't publish an update too late in the night :)

15-jan-2010: rsync 0.2

- dropbear rebuild to fix non working scp

- /system/bin/dropbear.sh script updated to allow use of public keys in /data/dropbear/.ssh/authorized_keys

Edited February 5, 2010 by freeasabeer

[Guest kendon]

this is so f***ing awesome! thanks!

do you take requests? i'd be interested in tree aswell... B)

and i am sure i can find more, but i don't have the skills to make them work...

edit: needed to get a superlong hyperlink on my phone, abused the forum for this :)

Edited January 13, 2010 by kendon

[Guest freeasabeer]

Done !

Now thinking of the best way to provide it without running off topic from the 4th post on :)

Not sure opening a new topic is the best one ? I'll attach it to the first post of this topic. 

Edited January 13, 2010 by freeasabeer

[Guest Paul]

Is this a static compilation?

I can include these in future MCR builds (along with nano, busybox) etc.

I have a request... rsync... :)

P

[Guest freeasabeer]

Yes they are all static binaries against uclibc.

rsync, that's a pretty good suggestion ! I'll have a look at it and keep you informed.

And yes, you can include then in MCR!

[Guest kendon]

Yes they are all static binaries against uclibc.

rsync, that's a pretty good suggestion ! I'll have a look at it and keep you informed.

And yes, you can include then in MCR!

woohoo this is getting fancy :) want rsync too :)

Edited January 13, 2010 by kendon

[Guest pulser]

woohoo this is getting fancy :) want rsync to :)

This is getting pretty interesting. So our phones are becoming feasibly linux terminals right now (well at least via. adb/ssh)

I take it none of these would work through an on-device emulator like Better Terminal or ConnectBot (the latter obviously being an SSH client)?

[Guest kendon]

Now thinking of the best way to provide it without running off topic from the 4th post on :)

Not sure opening a new topic is the best one ? I'll attach it to this post for this time but definitively not the best way though.

i'd suggest renaming this thread or creating a new one named "unix tools for android"...

[Guest kendon]

This is getting pretty interesting. So our phones are becoming feasibly linux terminals right now (well at least via. adb/ssh)

I take it none of these would work through an on-device emulator like Better Terminal or ConnectBot (the latter obviously being an SSH client)?

just tested tree in connectbot, works great.

[Guest kendon]

next one: wget. not necessary, but could come in handy...

sorry for hijacking this thread, but this is way to cool to let it die :)

[Guest intronauta]

i'd suggest renaming this thread or creating a new one named "unix tools for android"...

+1!

Awesome :)

Love these tools!

[Guest freeasabeer]

if we add too many linux tools then at some point we may ask ourself:

- why not go the "debian on android" route ?

- as currently all these binaries are statically linked to avoid any dependencies to other libs, wouldn't it be preferable to follow a similar way to the "debian on android" route and set up a chroot with all our beloved linux tools which can then be dynamically linked and hence save some space ?

I do not have the answer *yet* and do not want to push one particular option at this stage. But we need to think about it.

If we keep with a reasonable number of tools (let say not more than 10) then I think we can keep the static binaries approach.

[Guest Paul]

I'm not expert enough to understand the implications of this... elaborate?

P

[Guest pulser]

I'm not expert enough to understand the implications of this... elaborate?

P

+1.

I appreciate these are good things, but other than fooling about with on a quiet afternoon, I can't really figure out a use of this.

Not that I'm criticising this, in fact I'm actually about to start downloading and giving these a shot :) So, don't take this as negative. I'm pretty impressed we're getting these kind of things running on the Hero.

Just that I don't quite know why these things will be useful. I'm sure someone can enlighten us :)

[Guest steveh2]

The point he's making is that with no shared library use each one can be large, both on disk and in memory.

However, even with a large list available to the cooker, most people will just use a few of them in a rom I'd guess.

[Guest]

An interesting side effect I've noticed is that vi works great within a screen shell where as is was unusable from a direct shell !

vi doesn't work on non visual sessions. Visual mode gives you the ability to mark text for cut, copy (yank) and paste.

ed and ex are both non visual editors. ed spawned sed (which are both part of busybox) but ex isn't included in busybox for some reason (its usually part of vi) so I've emailed the busybox team to ask why. :)

Edited January 13, 2010 by Guest

[Guest]

next one: wget. not necessary, but could come in handy...

sorry for hijacking this thread, but this is way to cool to let it die :)

wget is actually included in busybox, so there won't be any need for a separate binary. :)

Open up a shell and type busybox and you will be presented by the number of compiled in programs ready for use.

[Guest Paul]

vi? Pah! Nano ftw! :)

P

[Guest]

Its amazing what is in busybox.

If you want to run an FTP server or a web server, you can from busybox. It will require a change to the init scripts so that inetd starts (also part of busybox) and an entry in /etc/inetd.conf

I know what would be nice would be a PHP binary or even Perl, so I can write a web app that uses sqlite database so I don't have to rely on one running on a distant server.

[Guest kendon]

wget is actually included in busybox, so there won't be any need for a separate binary. :)

Open up a shell and type busybox and you will be presented by the number of compiled in programs ready for use.

lol, i could've come up with this myself :) should check busybox features more often when fooling around with the phone... thx for the tip.

as for the debian idea: i had both debian and ubuntu on my phone, even ubuntu with gnome running :)

it is just not that comfortable to use. you have to start it somehow via bootdeb, you need to shutdown before you can restart your phone and all this. imho this is too much trouble if you just want some simple tools at hand. for example i have an alias as='adb shell', so whenever i want a command executed on android i just put as in front of it.

i'd prefer to have them statically linked ( :) whatever that means, i guess something like standalone executable), as they are easy to include in roms this way. correct me if i am wrong, more than typing make to compile something and i am so out of there...

fwiw here is my list of aliases i use when fooling around with android, remount is a script i created in /system/bin, you can guess what it does:

alias as='adb shell'

alias asll='adb shell ls -alh'

alias aslog='adb logcat|tee logcat.txt'

alias asro='adb shell remount ro'

alias asrw='adb shell remount rw'

alias assd='adb shell mount /sdcard'

alias cdhero='cd /media/data/hero'

alias vnchero='vncviewer hero:5901'

alias sshero='ssh root@hero -p 2222'

[Guest]

Once installed, you can start screen with: /data/xbin/screen and resume an ongoing session with /data/xbin/screen -r

Enjoy !

I copied screen and screen.real into /system thinking that I wouldn't have to type the path, but it then asks for files in /var/run/utmp. :)

screen still runs though, bizarre! :)

[Guest Paul]

/data/init.sh is run at first boot, so you could run inetd from there.

P

[Guest stevenz]

I realise this is probably a stupid question, but do you guys have any examples of what you're wanting to achieve by running this level of shell functionality on the Hero? As much as I'd like to have a fully functional remote Linux shell to ssh into from my Windows PC at work, I'm not sure it'd be of a great deal of use.

Please prove me wrong, I might stick Ubuntu onto it just "because I can", but I doubt I'd get a lot of use out of it, pretty sure I've seen someone's (Kendon's?) screenshots of remote X11 VNC sessions which would be amusing, but is vnc practical on such a "low-end" device? (this from someone who used to run Slackware on a PC with a crusty DX/2 66Mhz Cyrix CPU o/clocked to 80mhz years ago).

Suggestions of uses would be interesting :)

@Kendon - Statically linked binaries have any required library code embedded, dynamically linked ones call the external libraries (smaller executables). Portability vs. size.

Edited January 14, 2010 by stevenz

[Guest pulser]

I

just tested tree in connectbot, works great.

Cheers, kendon. Must have missed this post during a merge, either that or I'm going blind. :???