Guest Scanno Posted January 25, 2013 Report Share Posted January 25, 2013 @EJTagle, Thanks for the updates on the kernel. I am building a new version of CM10.1 that includes the kernel with the SIO schedular added and set as default schedular. As soon as it is ready i will post it as an experimental version so people can try it out. Did not add the other two additions yet. Link to comment Share on other sites More sharing options...
Guest Scanno Posted January 27, 2013 Report Share Posted January 27, 2013 Eduardo, I have added the SIO scheduler to the kernel and it seems to speed things up. I have also added the new governor and that seems to work also but I have not tested it very well yet. I am not able to set it as the default governor. If I do I an not able to boot. The over clock I have not tested added yet. Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 27, 2013 Report Share Posted January 27, 2013 (edited) Eduardo, Thank you very much for the help with the out of memory error. Increasing our GPU memory from 85 to 128 solved that issue. Was hoping we could run 4.2 with 85M because that really helped on 4.1.2 with free memory. We did run into 1 more big issue with 4.2. Our tablet uses LP1 because LP0 is not supported due to the processor revision. Basically the tablet will go to sleep then after like 0.090 seconds it will snap back awake. It will basically loop this over and over until the tablet is "woken" up by the power button. From our previous releases the tablet will go into LP1 and stay there like it should. I'm trying to figure out what is waking it up. I do not believe this is a wakelock issue because the tablet DOES actually go into LP1 mode, its just something kicking it out of the mode. Here is a dmesg of the tablet sleeping and waking. http://pastebin.com/1C8UsFH0 Starting around line 1031 you can see the "sleep loop." Any help on that would be greatly appreciated. EDIT: I have tried this on both AOSP and CM10.1 The issue is on both roms. -fosser2 Edited January 27, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 27, 2013 Report Share Posted January 27, 2013 (edited) Eduardo, Thank you very much for the help with the out of memory error. Increasing our GPU memory from 85 to 128 solved that issue. Was hoping we could run 4.2 with 85M because that really helped on 4.1.2 with free memory. We did run into 1 more big issue with 4.2. Our tablet uses LP1 because LP0 is not supported due to the processor revision. Basically the tablet will go to sleep then after like 0.090 seconds it will snap back awake. It will basically loop this over and over until the tablet is "woken" up by the power button. From our previous releases the tablet will go into LP1 and stay there like it should. I'm trying to figure out what is waking it up. I do not believe this is a wakelock issue because the tablet DOES actually go into LP1 mode, its just something kicking it out of the mode. Here is a dmesg of the tablet sleeping and waking. http://pastebin.com/1C8UsFH0 Starting around line 1031 you can see the "sleep loop." Any help on that would be greatly appreciated. EDIT: I have tried this on both AOSP and CM10.1 The issue is on both roms. -fosser2 wakeup wake lock: power-supply It is a wake lock. I have been fooled by the same behaviour a lot of times.. Probably the wake lock system is cancelling the LP1 sleep... Regarding the lack of ram... zRam will be your friend in a short time ;) Edited January 27, 2013 by ejtagle Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 28, 2013 Report Share Posted January 28, 2013 wakeup wake lock: power-supply It is a wake lock. I have been fooled by the same behaviour a lot of times.. Probably the wake lock system is cancelling the LP1 sleep... Regarding the lack of ram... zRam will be your friend in a short time ;) Ya we did get zRam set up and it does help a ton. Kinda sucks only having 512M ram but we're making do. As for system cancelling sleep do you have any more info on this? Would this be a rom or kernel related issue? I know the kernel provides wakelocks but I'm not really sure where to start looking.... Thank you. Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 28, 2013 Report Share Posted January 28, 2013 Ya we did get zRam set up and it does help a ton. Kinda sucks only having 512M ram but we're making do. As for system cancelling sleep do you have any more info on this? Would this be a rom or kernel related issue? I know the kernel provides wakelocks but I'm not really sure where to start looking.... Thank you. Seems to be kernel related. The definition of that wakelock is in drivers/power/power_supply_core.c. And the function that takes it is power_supply_changed(struct power_supply* psy) That function is called by your battery/charger driver to inform linux the battery stats changed, or the charger status changed. Maybe you are calling from a timer, and you are forgetting to stop the timer when the kernel suspends. When the battery/charger driver is suspended, no calls to the power_supply_changed function should be done. In fact, you should NOT call this function unless there is a change in some value. Otherwise, you would be preventing the system to suspend by continuously holding a wakelock... Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 28, 2013 Report Share Posted January 28, 2013 (edited) Seems to be kernel related. The definition of that wakelock is in drivers/power/power_supply_core.c. And the function that takes it is power_supply_changed(struct power_supply* psy) That function is called by your battery/charger driver to inform linux the battery stats changed, or the charger status changed. Maybe you are calling from a timer, and you are forgetting to stop the timer when the kernel suspends. When the battery/charger driver is suspended, no calls to the power_supply_changed function should be done. In fact, you should NOT call this function unless there is a change in some value. Otherwise, you would be preventing the system to suspend by continuously holding a wakelock... I disabled our battery code and charger code (smba-battery & gpio-charger.c). It is still looping through sleep mode, so I don't think that power_supply_changed is causing the issue. Here is a new paste of what is going on... http://pastebin.com/FpNHgB4j The only thing that I can really see in that log is this: <6>Wake24 for irq=362 <6>Disabling wake24 <6>gpio bank wake found: wake24 for irq=119 <6>Disabling wake24 IRQ 362 is "GPIO power" according to the IRQ dump that I made. Do you see anything else in that log that I may be missing? Thanks for the reply. Edited January 28, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 28, 2013 Report Share Posted January 28, 2013 (edited) I disabled our battery code and charger code (smba-battery & gpio-charger.c). It is still looping through sleep mode, so I don't think that power_supply_changed is causing the issue. Here is a new paste of what is going on... http://pastebin.com/FpNHgB4j The only thing that I can really see in that log is this: <6>Wake24 for irq=362 <6>Disabling wake24 <6>gpio bank wake found: wake24 for irq=119 <6>Disabling wake24 IRQ 362 is "GPIO power" according to the IRQ dump that I made. Do you see anything else in that log that I may be missing? Thanks for the reply. By disabling the battery driver, you removed one of the wakelock that was causing trouble... But, now, there is another one: wakeup wake lock: event1-354 ... That name seems to be generated from something. We have to figure out the culprit here... Buyt, still the problem is a wakelock... What I usually do is to grep though all kernel sources looking for usages of wake_lock_init(), looking for that particular wakelock name to figure out the culprit... If not found, it could be the android framework or an application... Also, do echo 15 > /sys/module/wakelock/parameter/debug_mask echo 1 > /sys/module/kernel/parameters/initcall_debug And increasing printk verbosity to all is not a bad idea here... Edited January 28, 2013 by ejtagle Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 28, 2013 Report Share Posted January 28, 2013 (edited) By disabling the battery driver, you removed one of the wakelock that was causing trouble... But, now, there is another one: wakeup wake lock: event1-354 ... That name seems to be generated from something. We have to figure out the culprit here... Buyt, still the problem is a wakelock... What I usually do is to grep though all kernel sources looking for usages of wake_lock_init(), looking for that particular wakelock name to figure out the culprit... If not found, it could be the android framework or an application... Also, do echo 15 > /sys/module/wakelock/parameter/debug_mask echo 1 > /sys/module/kernel/parameters/initcall_debug And increasing printk verbosity to all is not a bad idea here... Here is a dump with the 2 echo commands run. I will start grepping the kernel source looking for the wake lock. http://pastebin.com/gvRSK5xU Thanks for the help so far! Edited January 28, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 28, 2013 Report Share Posted January 28, 2013 Here is a dump with the 2 echo commands run. I will start grepping the kernel source looking for the wake lock. http://pastebin.com/gvRSK5xU Thanks for the help so far! Several wakelocks are preventing the tablet to enter suspend... Not all of them at the same time. I suggest to disable wlan ... There is an option to keep tablet awake while wlan is active. Make sure it is not checked. Sometimes, the alarm wakelock prevents the tablet to enter suspend. That is perfectly normal, as android wakes periodically to check for emails (gmali, dropbox sync, etc). There is also the PowerManagerService sometimes preventing the tablet to enter suspend. My suggestion is to test suspend without gapps installed, with wlan disabled.... Without the charger plugged in and without usb plugged in... Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 28, 2013 Report Share Posted January 28, 2013 (edited) Several wakelocks are preventing the tablet to enter suspend... Not all of them at the same time. I suggest to disable wlan ... There is an option to keep tablet awake while wlan is active. Make sure it is not checked. Sometimes, the alarm wakelock prevents the tablet to enter suspend. That is perfectly normal, as android wakes periodically to check for emails (gmali, dropbox sync, etc). There is also the PowerManagerService sometimes preventing the tablet to enter suspend. My suggestion is to test suspend without gapps installed, with wlan disabled.... Without the charger plugged in and without usb plugged in... Here is a pastebin with wlan, power & gpio charger disabled at kernel level. The power and usb was also unplugged when i took the following serial dump. The build does not have gapps installed either. http://pastebin.com/ZCXzNfDh Edited January 28, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 29, 2013 Report Share Posted January 29, 2013 (edited) Here is a pastebin with wlan, power & gpio charger disabled at kernel level. The power and usb was also unplugged when i took the following serial dump. The build does not have gapps installed either. http://pastebin.com/ZCXzNfDh Sometimes, the answer is so obvious, we fail to see it... <6>wakeup wake lock: event1-353 This wakelock is created in /drivers/input/evdev.c, line 322... What input device do you have as /dev/input/event1 ? ... That is the culprit... It is creating continuous events that prevent the tablet to go to sleep... That device fails to stop sending events while the system is suspended... Perhaps accelerometer ? I had similar issues with the p10an01 accelerometer... You need to port that driver to the latest suspend infraestructure. You can see what i did to make it work on the bma150.c accelerometer driver ... Edited January 29, 2013 by ejtagle Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 29, 2013 Report Share Posted January 29, 2013 (edited) Sometimes, the answer is so obvious, we fail to see it... <6>wakeup wake lock: event1-353 This wakelock is created in /drivers/input/evdev.c, line 322... What input device do you have as /dev/input/event1 ? ... That is the culprit... It is creating continuous events that prevent the tablet to go to sleep... That device fails to stop sending events while the system is suspended... Perhaps accelerometer ? I had similar issues with the p10an01 accelerometer... You need to port that driver to the latest suspend infraestructure. You can see what i did to make it work on the bma150.c accelerometer driver ... ejtagle, I actually looked into this a bit yesterday. I ran cat /sys/class/input/event1/device/name and pulled event1 which happened to be the capacitive keys on our tablet. Then I disabled them and took another serial dump. Event1 then became our touchscreen and the error persisted. I actually DID try disabling the touchscreen also and the error was still there. So I am at a loss why event1-353 would be causing this... Latest Pastebin: http://pastebin.com/aTGk1F4T (wifi, capacitive buttons, gpio-charger, battery all disabled usb unplugged) In the log above i also notice there is a event3 at the end. This relates to "gpio-keys" and is most likely because I pushed the power button to get the device out of the sleep cycle. Edited January 29, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 29, 2013 Report Share Posted January 29, 2013 ejtagle, I actually looked into this a bit yesterday. I ran cat /sys/class/input/event1/device/name and pulled event1 which happened to be the capacitive keys on our tablet. Then I disabled them and took another serial dump. Event1 then became our touchscreen and the error persisted. I actually DID try disabling the touchscreen also and the error was still there. So I am at a loss why event1-353 would be causing this... Latest Pastebin: http://pastebin.com/aTGk1F4T (wifi, capacitive buttons, gpio-charger, battery all disabled usb unplugged) In the log above i also notice there is a event3 at the end. This relates to "gpio-keys" and is most likely because I pushed the power button to get the device out of the sleep cycle. Let's try to add to the kernel command line no_console_suspend=1, and also add the timestamps to the kernel logs... Something is waking up instantly the system as soon as it tries to go to LP1... If you have disabled accelerometer, and this problem continues... Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 29, 2013 Report Share Posted January 29, 2013 (edited) Let's try to add to the kernel command line no_console_suspend=1, and also add the timestamps to the kernel logs... Something is waking up instantly the system as soon as it tries to go to LP1... If you have disabled accelerometer, and this problem continues... Ok, I added no_console_suspend=1 to the command line. I also checked and as for the timestamps they should be on... This was enabled. PRINTK_TIME [=y] Here is the latest dump with the new command line option. http://pastebin.com/b3tarEcS EDIT: in this pastebin event0 is "tegra-alc5623 Headphone Jack" This is what our Command Line looks like from the dump: <5>Kernel command line: zcache no_console_suspend=1 mem=448M@0M nvmem=64M@448M vmalloc=192M video=tegrafb console=ttyS0,115200n8 usbcore.old_scheme_first=1 cpuid=200102 devicetype=1002 bspversion=0202100907 btmac=9c5ed6131a00 tegraboot=nand mtdparts=tegra_nand:16384K@12032K(misc),16384K@62208K(recovery),16384K@79104K(boot),358400K@96000K(system),68480K@455296K(cache),4096K@7424K(bootbmp),32768K@28928K(logodata) Edited January 29, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 30, 2013 Report Share Posted January 30, 2013 Ok, I added no_console_suspend=1 to the command line. I also checked and as for the timestamps they should be on... This was enabled. PRINTK_TIME [=y] Here is the latest dump with the new command line option. http://pastebin.com/b3tarEcS EDIT: in this pastebin event0 is "tegra-alc5623 Headphone Jack" This is what our Command Line looks like from the dump: <5>Kernel command line: zcache no_console_suspend=1 mem=448M@0M nvmem=64M@448M vmalloc=192M video=tegrafb console=ttyS0,115200n8 usbcore.old_scheme_first=1 cpuid=200102 devicetype=1002 bspversion=0202100907 btmac=9c5ed6131a00 tegraboot=nand mtdparts=tegra_nand:16384K@12032K(misc),16384K@62208K(recovery),16384K@79104K(boot),358400K@96000K(system),68480K@455296K(cache),4096K@7424K(bootbmp),32768K@28928K(logodata) I have followed all the suspend procedures in kernel... By looking at the logs, seems the processor is actually going to LP1... And then it is woken up ... probably by an external source... Seems not to be a wakelock... otherwise, the process would not go so far... Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 30, 2013 Report Share Posted January 30, 2013 (edited) I have followed all the suspend procedures in kernel... By looking at the logs, seems the processor is actually going to LP1... And then it is woken up ... probably by an external source... Seems not to be a wakelock... otherwise, the process would not go so far... So do you think its actually something rom based that is waking it up? Personally I think it is... Here is a logcat from during the sleep loop http://pastebin.com/PG7ZZ51s Edited January 30, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 31, 2013 Report Share Posted January 31, 2013 No, if you mean Android rom based, i don't think so... There is something hw related that is actually waking the system from LP1... When the message <6>Entering suspend state LP1 is shown, there is no turning back: The system actually goes to LP1, and then, the only way to wake it up is by hw means. Android usually programs the hw alarm clock to wake up the system if required at a given point in time. Of course it can be waken by other sources if enabled> (UART activity, keyboard presses, interrupt sources). (taken from nvidia conversations regarding LP modes on tegra2 a long time ago) Tegra supports three low power modes that involve powering down the CPU. LP2 powers down both CPU cores and the GICs, but leaves the core peripherals, including the memory controller and the legacy interrupt controller, enabled. The legacy interrupt controller is used as the wakeup source, and any interrupt can wake the device. LP2 can be used in idle. LP1 is the same as LP2, but in addition turns off the memory controller and puts the DDR memory in self-refresh. Any interrupt can wake the device. LP1 could be used in idle if no peripherals are doing DMA. LP0 turns off everything in the SoC except the RTC and a power management controller, both of which run off a 32 kHz clock. The power management controller has 32 wake sources, all other interrupts can not be used to wake from LP0. LP2 idle mode power-gates the main CPU complex, requiring a full processor state save and restore from a reset vector. Processor context area is allocated during platform initialization from the kernel, and mapped into the hotplug page tables (which also serve as the initial page tables for the LP2 main processor reset). Platform-specific data (power good times, PMU capabilities, etc.) must be specified when registering the suspend operations to ensure that platform power sequencing restrictions are maintained. Since all device interrupts (except timers) are disabled in the suspend path, the wakeup interrupts need to be manually unmasked before entering into a suspend state or the processor will never wake up; these forced-unmask interrupts are re-masked immediately in the resume path to prevent the kernel from live-locking prior to driver resume. In both LP0 and LP1, SDRAM is placed into self-refresh. in order to safely perform this transition, the final shutdown procedure responsible for * turning off the MMU and L1 data cache * putting memory into self-refresh * setting the DDR pads to the lowest power state * and turning off PLLs is copied into IRAM (at the address TEGRA_IRAM_BASE + SZ_4K) at the start of the suspend process. In LP1 mode (like LP2), the CPU is reset and executes the code specified at the EVP reset vector. Since SDRAM is in self-refresh, this code must also be located in IRAM, and it must re-enable DRAM before restoring the full context. In this implementation, it enables the CPU on PLLP, enables PLLC and PLLM, restores the SCLK burst policy, and jumps to the LP2 reset vector to restore the rest of the system (MMU, PLLX, coresite, etc.). The LP2 reset vector is expected to be found in PMC_SCRATCH1, and is initialized during system-bootup. In LP0 mode, the core voltage domain is also shutoff. As a result, all of the volatile state in the core voltage domain (e.g., pinmux registers, clock registers, etc.) must be saved to memory so that it can be restored after the system resumes. A limited set of wakeups are available from LP0, and the correct levels for the wakeups must be programmed into the PMC wakepad configuration register prior to system shutdown. On resume, the system resets into the boot ROM, and the boot ROM restores SDRAM and other system state using values saved during kernel initialization in the PMC scratch registers. Resuming from LP0 requires the boot ROM to supply a signed recovery codeblob to the kernel; the kernel expects that the length and address of this blob is supplied with the lp0_vec= command line argument; if not present, suspend- to-LP0 will be disabled For simplicity, the outer cache is shutdown for both LP0 and LP1; it is possible to optimize the LP1 routine to bypass outer cache shutdown and restart. To save power, SMP tegra SoCs place non-boot CPUs in reset when they are removed from the scheduling cluster using CPU hotplug. Slave CPUs save their contexts (incl. CP15 and VFP state) out to a reserved memory region, cancel SMP operation, and write to the SoC reset controller to disable themselves. This is done with caches and MMU enabled, so care is taken to ensure that all the dirty context cache lines are cleaned out to the PoC before shutting down. When re-enabled, slave CPUs execute a hotplug boot routine which mirrors the initial configuration performed by secondary_startup, but after enabling the MMU "return" to __cortex_a9_restore which restores the saved state from the context area, and returns to platform_cpu_die. So, any interrupt source not properly masked at suspend time could be the culprit here... I have a question here... Is the 3.1 kernel the first one you did for this device? ,,, Because, if it is not, and the previous ones suspended to LP1 as they should, maybe we could compare them... :D (kernel 2.6.32 does not count! ;) ) Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 31, 2013 Report Share Posted January 31, 2013 So, any interrupt source not properly masked at suspend time could be the culprit here... I have a question here... Is the 3.1 kernel the first one you did for this device? ,,, Because, if it is not, and the previous ones suspended to LP1 as they should, maybe we could compare them... :D (kernel 2.6.32 does not count! ;) ) The last 2 kernels we have run were .39 and 3.1. We have had lp1 working since the ICS days, our latest 3.1 kernel and android 4.1.2 were working with sleep just fine. As soon as we upgraded to 4.2.1 we had the sleep issues. Someone also took the older .39 kernel and tried it on 4.2.1. It also had the sleep issue! If we flash a 4.1.2 rom to the tablet, the tablet sleeps fine... My mind is blown here. Link to comment Share on other sites More sharing options...
Guest ejtagle Posted January 31, 2013 Report Share Posted January 31, 2013 If the same kernel with the 4.1.2 works fine, and with the 4.2.x aosp does not sleep, the problem then lies in the Framework... Maybe is the framework holding a wakelock, after all ? ... Maybe the values in config.xml ? Link to comment Share on other sites More sharing options...
Guest fosser2 Posted January 31, 2013 Report Share Posted January 31, 2013 If the same kernel with the 4.1.2 works fine, and with the 4.2.x aosp does not sleep, the problem then lies in the Framework... Maybe is the framework holding a wakelock, after all ? ... Maybe the values in config.xml ? Here is our config.xml. Nothing really sticks out to me that we did wrong in here that would be causing a wake... https://github.com/redeyedjedi/android_device_smba_common/blob/CM-10.1/overlay/frameworks/base/core/res/res/values/config.xml Link to comment Share on other sites More sharing options...
Guest ejtagle Posted February 1, 2013 Report Share Posted February 1, 2013 Before I forget, posting here the latest Overclocking kernel patches for the Advent Vega. This version (at least, for me) stabilizes the system and completely avoids system freezes.oc to 1.7Ghz v2.rar Link to comment Share on other sites More sharing options...
Guest ejtagle Posted February 1, 2013 Report Share Posted February 1, 2013 Here is our config.xml. Nothing really sticks out to me that we did wrong in here that would be causing a wake... https://github.com/r...lues/config.xml Maybe installing a 4.1.x rom, getting the dmesg and logcats when it enters sleep with all the debug options we have already enabled, then doing the same with the 4.2.x rom and them comparing both dmesgs and logcats could give us a clue on the differences that are making the 4.2.x keep awake instead of going to lp1 sleep... Link to comment Share on other sites More sharing options...
Guest fosser2 Posted February 1, 2013 Report Share Posted February 1, 2013 (edited) Maybe installing a 4.1.x rom, getting the dmesg and logcats when it enters sleep with all the debug options we have already enabled, then doing the same with the 4.2.x rom and them comparing both dmesgs and logcats could give us a clue on the differences that are making the 4.2.x keep awake instead of going to lp1 sleep... With those debugging options enabled here are the 2 serial dumps. 4.1.2: http://pastebin.com/hBAf850J 4.2.1: http://pastebin.com/yzkW0NiQ In 4.2.1 I see <6>gpio bank wake found: wake24 for irq=119 <6>Enabling wake24 This little bit seems to be the biggest difference I noticed... EDIT: I did use the EXACT same kernel on both roms. There were no tweaks to it at all. Edited February 1, 2013 by fosser2 Link to comment Share on other sites More sharing options...
Guest ejtagle Posted February 2, 2013 Report Share Posted February 2, 2013 Posting for scanno: Latest power hal thatpreserves the maximum speed set by the CPU master app (or any other that setsthe maximum possible frequency for CPU)power.rar Link to comment Share on other sites More sharing options...
Recommended Posts
Please sign in to comment
You will be able to leave a comment after signing in
Sign In Now